Engineered

Academic Analytics

Today's post is about the company Academic Analytics, which describes its mission on its LinkedIn profile as follows: "Academic Analytics partners with research institutions to provide data that offer comprehensive information on faculty research activity as well as strategic solutions drawn from analyses of national disciplines and custom comparisons at the institutional, department, program, and individual faculty levels. The data have tremendous potential, supporting strategic investment decisions, benchmarking research activity, reviewing academic units, and developing the university research enterprise." 

This is apparently not the message the company used to have, according to an October 2016 article in Chronicle of Higher Education. "Just a few years ago, Academic Analytics, an upstart company providing data on faculty productivity, talked of helping cash-strapped universities save as much as $2 billion by identifying their lowest-performing professors." The article explains that Academic Analytics toned down its message after Georgetown University cancelled its subscription (in 2016). Georgetown's provost Robert Groves "explained a decision to drop the university’s subscription by questioning whether Academic Analytics’ data are comprehensive, accurate, or consistently valuable." Readers can learn the details about the study Georgetown performed in the summer of 2016 to investigate the accuracy of the Academic Analytics information here, although Provost Groves took pains to state, in a follow-up post dated July 2017 available there, that: "Academic Analytics does a good job of collecting and making available the data that it purports to collect, and [the] data largely are accurate. For that reason Academic Analytics can be a useful tool for universities, and it has the potential to become even more useful as it expands the nature and types of data it collects." 

The American Association of University Professors posted a "Statement on Academic Analytics and Research Metrics" on its website back in March 2016. It mentioned: "[Academic Analytics] now claims 385 institutional customers in the U.S. and abroad, representing about 270,000 faculty members in 9,000 Ph.D. programs and 10,000 departments. Some of the firm's metrics are without any qualitative dimension: per capita production of books and articles, for example. Other figures, such as per capita citations and "awards" per faculty member, may be said to introduce some qualitative dimensions, but seem to produce puzzling results." It described concerns at Rutgers (a contract worth $492,500), including some about data accuracy stemming from faculty members' inability to check their data for correctness, since the portal is only accessible to administrators. (It seems that universities can now choose to have their faculty members see and perhaps edit their data.)

The statement also explains, "The faculty of the university's School of Arts and Sciences [at Rutgers] voted overwhelmingly to forbid use of the firm's data in tenure and promotion decisions or "in decisions affecting the composition of the faculty, graduate and undergraduate curricula, and grant‐writing." The resolution also called on the university to distribute personal data collected by the firm to each faculty member of the school."

It goes on to say: "Most faculty members have some sort of direct experience of metrics used to assess performance... There is, however, good reason to doubt the utility of such metrics in tenure and promotion decisions and/or in judgments affecting hiring, compensation or working conditions. A 2015 study by the Higher Education Funding Council for England,2 where use of research metrics is now required at public institutions... found that indicators can be misused or "gamed," [...and] concluded that "carefully selected indicators can complement decision‐making," but expert judgment and qualitative measures that respect research diversity remain essential elements of a representative process of peer review, which should remain "the primary basis for evaluating research outputs, proposals and individuals.""

I have found more information on the Northeastern University's data access website, which provides a User Guide to Faculty Insight, a key module among the offerings of Academic Analytics that "enables faculty and academic staff to find targeted funding opportunities and experts at their institution". The Faculty Insight portal is open to all Northeastern faculty. Of greater concern is the Benchmarking Suite, which the Northeastern website says is used to perform departmental program reviews, identify vulnerabilities, strengths and weaknesses for strategic planning and analyze faculty career profiles and trajectory by discipline. 

(Screenshot taken of this website on 10/10/20)

    

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

While data has its place in higher ed, situations where evaluating key components of the academic enterprise, from departmental programs to faculty members, would potentially be farmed out to a non-academic, for-profit entity are particularly worrisome. This approach also completely fails to recognize each university's brand, departments' brands (especially in doctoral programs), signature strengths, and all the qualitative measures that explain why certain students decide to enroll in a university over others. 

Georgetown and Rutgers are not the only universities where concerns have been raised about Academic Analytics. Another famous, more recent example is UT Austin. In 2018, UT Austin faculty joined a campaign against Academic Analytics, as described in another Chronicle article, which refers to Academic Analytics as a "faculty-productivity company". (Readers will notice that Academic Analytics's self-description on LinkedIn does not use the word "productivity".) The opening paragraph states: "The University of Texas at Austin this week became one of the most prestigious research institutions to join a faculty rebellion against Academic Analytics, a data company that promises to identify low-performing professors." UT Austin's Faculty Council (their equivalent to the Faculty Senate) passed a resolution about Academic Analytics in January 2018 in which it recommended that UT Austin "not adopt Academic Analytics, LLC as a faculty management tool" (the term "faculty management tool" already sounds quite ominous). Further, it urged the administration, should it adopt Academic Analytics anyway, not to use that data to allocate resources among faculty nor in decisions affecting the composition of the faculty, nor in tenure and promotion decisions, nor in raise decisions, nor in undergraduate and graduate curricula decisions. Further, it urged the administration to make all personal data available to faculty members for their review, correction and ratification. Follow-up to the administration and their response are here.

As a data scientist, I am always interested in ways to use data to make better decisions, especially in unusual applications, such as faculty research. More broadly, data can help foster conversations about the right metrics to evaluate one's field and opportunities to leverage one's strengths. There are, however, clear pitfalls associated with letting a for-profit company wield such outsized influence over so many in academia and in so many universities focusing on the same Academic-Analytics-infused thinking to improve their academic standing. Universities already compete aggressively for the most meritorious students, spending extensive amounts of aid to bring in always better students (as measured by ACT and SAT scores), and the result of that has been a recent national refocus toward need-based financial aid and an update by US News of the criteria they use in ranking schools to put less emphasis on "student excellence" (this WaPo article is a great analysis of some of the recent changes and what remains wrong with the US News rankings). It often seems that U.S. universities try to do everything they can to shine in whatever indicators that US News uses in its rankings and, although data is valuable, I don't think it would be a compliment to higher education if they now fell in line behind Academic Analytics to evaluate their own faculty and their research productivity. 

Here is, however, one possible useful way to leverage Academic Analytics. Deans and Provosts know their peers and aspirational peers at the school/college or university level, but it is very hard for them to know how individual departments in disciplines other than their own measure up, except by looking at the department-level (graduate) US News rankings. A department head may have in mind five peers and five aspirational peers, but how can a Dean from other discipline, let alone a Provost from a totally different field, augment that thinking or come up, as part of a brainstorming session with the department, with other peer/aspirational peer departments that would be more aligned with the strategy the Dean or the Provost has for the school or university? Under those circumstances it could be valuable to have access to a database like Academic Analytics. But it is not clear that it would be then be worth its hefty price tag to many administrators who use it.

 

What is a Tier 1 university?

After my last post, I received some questions on the difference between Research 1 (R1) and Tier 1 universities. R1 status is formally defined in the Carnegie Classification as Doctoral universities: very high research activity. As for a Tier 1 definition, a 2004 interim report of the Texas Senate Finance Committee explains that: "Tier 1 status is not formally defined, but is intended to reflect excellence at an institution of higher education. Common characteristics of Tier 1 institutions include: high research expenditures and a large number doctoral degrees awarded in various fields. Higher Education Commissioner Raymund Paredes urged the Legislature to define Tier 1 broadly and in a way that makes the most sense for the greatest number of institutions in Texas." 

The Texas Research Alliance admits that: "There is no official federal system to designate National “tier 1” institutions. However, The Center for Measuring University Performance creates an annual report of the Top American Research Universities that are typically considered nationally “Tier 1” in the higher education field. HB 51 in 2009 established programs to increase the amount of Tier 1 institutions in Texas."

The interested reader can find an overview of HB 51 in the 81st Texas legislature here. Key points include requiring long-term strategic plans for each research and emerging research university and establishing funding criteria as well as incentive grants for excellence programs at universities. Three funding streams were established:

• Research University Development Fund
• Performance Incentive Funding (allocated in proportion to the increase in the average number of degrees awarded in the two most recent fiscal years compared to the previous two fiscal years and in proportion to the average number of degrees awarded by each institution)
• Texas Research Incentive Program (matching a certain percentage of gifts received by an emerging research university) 

Further, the National Research University Fund "provides funding to emerging research universities that meet critical benchmarks for achieving national research university criteria such as certain levels of endowments, certain numbers of Ph.D. degrees awarded, etc."

The University of Houston explains Tier One designation in its Tier One FAQs: "“Tier One,” “Top Tier” and “Nationally Competitive Research University” are terms used interchangeably to refer to universities known for world-class research, academic excellence, an exceptional student body, and the highest levels of innovation, creativity and scholarship. Because of these accomplishments, these universities enjoy a national “brand,” recognition and prestige."

Further, "Within the academic community, there are three organizations that are generally accepted as national arbiters of an institution’s rank as a Tier One institution.  They are the Association of American Universities [AAU], the Carnegie Foundation for the Advancement of Teaching and the Center for Measuring University Performance, which issues Top American Research University (TARU) reports.  Recognition by any of these three is taken as an indication of Tier One status." (Looking at the list of AAU members, I couldn't find one that was not R1 but I may have missed it. I still have to analyze the TARU reports. The top institutions may be also R1. Note that I found SMU in several of the data tables! Hopefully I'll find the time to write a blog post about that soon. But unfortunately not today.)

Back in 2016, the Texas Tribune equated the Carnegie Classification Research One (R1) with Tier One (T1) status. Baylor also, in its own FAQs about R1/T1, sounds like it believes Carnegie's R1 is interchangeable with Tier-1 status. Its other answers to the FAQs are excellent, especially "how does a research focus fit with Baylor's historic strengths in classroom teaching?" I'll let you read the full answer on the website, but here is an excerpt: "High-level research enhances the classroom experience by providing students with cutting-edge, real-world examples that advance knowledge in the very disciplines they study. As students are equipped by Baylor faculty with foundational understanding in their chosen field, research opportunities provide meaningful applications to invigorate and advance that knowledge." 

I'll leave my readers with wise words I found in the UH's Tier One FAQs: "Texas has lagged behind other states in the number of Tier One universities – California has nine, New York seven, for example. Increasing the number of Tier One institutions allows Texas to compete more successfully for the best and brightest students and faculty, to spur economic growth, to sustain an educated workforce and attract innovative research."

Current and Future R1 Universities

One of my undergraduate students is doing a research project on R1 institutions this semester (Research One: those designated in the Carnegie Classification as doctoral universities having Very High Research Output). There is a lot to study, whether nationwide or at the Texas level. 15 universities achieved R1 status from R1 (High Research Output) in the latest update of the classification in 2018: Auburn, Binghamton, Dartmouth, Drexel, Mississippi State, Montana State, Oklahoma State, Rensselaer (RPI), University of Alabama, UT El Paso, UC Denver / Anschutz Medical Campus, U. Nevada - Las Vegas, U. Nevada - Reno, University of New Hampshire and University of Southern Mississippi.

Locally, UT Dallas, UT Arlington and University of North Texas are all R1, and all three achieved that designation for the first time in 2015 - only five years ago. (In Texas, the R1 universities are Rice, UT Austin, UT Arlington, University of Houston, U. North Texas, Texas A&M, UT Dallas, UT El Paso and Texas Tech.) The research landscape of the Dallas-Fort Worth metropolis is being transformed at lightning speed and it is exhilarating to be part of an area with such vibrant interest in research and innovation. Maybe I'll write a post about how they managed that feat some day soon, but for now I thought it would be best to focus on private universities that have achieved or aspire to achieving R1 status. I am particularly interested in universities like Rensselaer Polytechnic Institute, Drexel and Dartmouth, which have recently received the R1 designation, but also in local universities like Baylor, that publicly announced their commitment to R1 goals last year, only months after Baylor alumna Nancy Brickhouse became provost and two years after Linda Livingstone became President ("Baylor officially announces pursuit of Research 1/Tier 1 goals".) 

In its Illuminate fundraising campaign, Baylor articulates four pillars (Christian environment, transformational education, research & scholarship and arts & athletics) and five initiatives for the university (health, data sciences, material sciences, human flourishing, leadership&ethics, and Baylor in Latin America). What I like about this strategy is that it allows donors to support broad themes while avoiding too narrow projects that would fail to build momentum for the university as a whole or too vague endeavors that would not galvanize would-be donors. For instance, I think it is very smart from Baylor to focus an internationalization strategy on a specific part of the world where it believes it can make a difference. Many universities have launched campuses abroad, but given Texas's current and projected demographics, building links with Spanish-speaking parts of the world seems very wise.

I haven't studied R1 universities to be able to judge whether those goals are tightly coupled with R1 status or if other goals would have been more appropriate first, so I can't say whether this approach will lead Baylor to R1 or not, and if it does, in what timeframe. But there is no doubt that a clear plan will always help motivate donors to contribute money, because it gives them confidence that various scenarios have been carefully considered and the donors are now being presented with (what people in the know view as) the best one. I don't know when Carnegie plans to publish the next update to its classification, and there are certainly pitfalls to a university's announcing so publicly it is going after R1 goals if that goal won't be achieved for many more years, because of the risk of losing momentum. So I suppose Baylor has a timeline to achieve R1 status that is, as far as academic timelines go, on the short side. (I don't know anyone at Baylor. This is just my assumption.) I can't measure Baylor's chances of making good on its promises, but there is no doubt few things can rally the troops more effectively than a bold, inspiring project presented by a fairly new, enthusiastic leadership team. It will definitely be fun to watch Baylor's progress toward R1 in the years to come!

Review of "No Rules Rules: Netflix and the Culture of Reinvention"

Today's post is about "No Rules Rules: Netflix and the Culture of Reinvention" by Reed Hastings and Erin Meyer. One-sentence summary: this is a really good book, very insightful, that makes you think about the processes in your own company and whether you could, or could not, implement what Hastings advocates. In other words, it is not just a puff piece designed to laud Hastings and Netflix, but a business book with genuine value for readers everywhere.

I'll admit that at first I found the book gimmicky, because it is split in sections by either author within each chapter and the reader will of course be more interested in Netflix CEO Hastings's perspective rather than INSEAD business professor Meyer's. Each segment by either author is preceded by a little drawing of their face to better set them apart and over the first chapter or so I couldn't shake the impression that the book lacked substance  - but after a while I got used to the setup, and since Netflix is primarily a provider of visual entertainment I guess the little face drawings made sense for the Netflix brand. Perhaps I should have expected even more visual add-ons. Or maybe a book trailer! (Totally a book trailer!) 

But once I got used to the setup, I really enjoyed this book. It is structured around nine core principles:

1. First build up talent density... A great workplace is stunning colleagues. This is the most important because none of the rules below will work if you don't have high-caliber colleagues you can trust to act in the company's best interest. Being surrounded by top colleagues elevates your own thinking and pushes you to do your best." In Hastings's words: "For top performers, a great workplace isn't about a lavish office, a beautiful gym, or a free sushi lunch. It's about the joy of being surrounded by people who are both talented and collaborative. People who can help you be better." It is particularly important in a place like Netflix that delegates a lot of the decision-making to mid-level managers and lower.  
2. Then increase candor... Say what you really think (with positive intent). The concept of candor in the workplace deserves its own book, and thankfully someone has already written it: Kim Scott, author of Radical Candor. I absolutely love that book and recommend it any chance I've got. 
3. Now begin removing controls... Remove vacation policy, remove travel and expensive approvals. This is obviously not possible if you don't have the best of the best working for you, as you risk ending up with situations where mediocre people have made less-than-advisable purchases and straddled the company with the cost. You have to trust people's judgement first. In Hastings's words: "Set context up front and keep an eye on spending out back" and "Once you have a workforce made up nearly exclusively of high performers, you can count on people to behave responsibly... The trust you offer will in turn instill feelings of responsibility in your workforce, leading everyone in the company to have a greater sense of ownership." 
4. Fortify talent density... Pay top of personal market. (Surprising insight: "Bonuses are bad for flexibility." In more details: "Don't pay performance-based bonuses. Put these resources into salary instead. Teach employees to develop their networks and to invest time in getting to know their own - and their teams' - market value on an ongoing basis. This might mean taking calls from recruiters or even going to interviews at other companies. Adjust salaries accordingly.")
5. Pump up candor... Open the books. This gave rise to interesting discussions in the book regarding whether employees need to be told about a possible restructuring if their jobs may be at stake - but may not be. Another insightful situation was debated regarding post-firing communication. Netflix always goes for transparency.
6. Now realize more controls... No decision-making approvals needed. (1) Farm for dissent ("socialize" the idea, aka, get feedback) (2) For a big idea, test it out (3) As the informed captain, make your bet (4) If it succeeds, celebrate. If it fails, sunshine it.
7. Max up talent density... The keeper test. This chapter described the team vs family metaphors of working for a company and the behaviors that align with the prevailing metaphors (hint: Netflix is not a family). There is an interesting paragraph about Netflix as a professional sports team. "These players are really close... But they are not a family. The coach swaps and trades players in and out throughout the year in order to make sure they always have the best player in every position." Hastings explains: "At Netflix, I want each manager to run her department like the best professional teams, working to create strong feelings of commitment, cohesion, and camaraderie, while continually making tough decisions to ensure the best player is manning each post." And now, for the keeper test: "If a person on your team were to quit tomorrow, would you try to change their mind? or would you accept their resignation, perhaps with a little relief? if the latter, you should give them a severance package now, and look for a star, someone you would fight to keep." (This is a question worth asking of the people about you too, although obviously you would not be in a position to fight to keep them. But if you think that your direct reports will pop up a bottle of champagne and cry tears of joy once you've left, it is worth asking if you are a good fit for the position.) The Keeper Test, in its shortest version, is: "Which of my people, if they told me they were leaving, for a similar job at a peer company, would I fight to keep at Netflix?" "Penguins abandon those in the group that are weak or struggling, while elephants rally around them and nurture them back to health." Hastings speaks out against Performance Improvement Plans, but only because Netflix's culture of candor has allowed employees to receive extensive feedback separately. He also speaks out against stack-ranking, with a set percentage of employees being shown the door. "Stack ranking sabotages collaboration and destroys the joy of high-performing teamwork." Some Netflix employees spoke about a culture of fear that they will lose their jobs at a moment's notice. 
8. Max up candor... A circle of feedback. Live 360 dinners with 25 percent positive and 75 percent developmental. 
9. Eliminate most controls. Lead with context, not control. The story of the Oscar-winning Icarus documentary alone is worth reading that chapter! There is also an interesting discussion about loosely vs tightly coupled systems, the goal of highly aligned, loosely coupled organizations and alignment as a tree, not a pyramid. 

Most valuable in the book, besides Hastings's insights, were the vignettes with real Netflix employees illustrating this or that principle with an anecdote that had happened to them. You can argue whether Netflix's principles can be applied elsewhere (not every company will be able to hire and keep top-of-the-market) but the case studies brought home those principles in impressive fashion.

In summary, an excellent book.  

Supplemental readings: Radical Candor by Kim Scott and Creativity, Inc by Ed Catmull.

The college that said no to rankings

Since early fall is US News rankings season, I figured I'd write a post about a college that has dared to say no to the rankings since 1995. That college is Reed College in Portland, Oregon. The decision of this private liberal-arts college not to participate in the US News rankings was the focus of an Atlantic op-ed at the time. (Reed College also discusses this decision in a September 2018 post here. Key quote: "We believe that the value of an education is directly related to the depth of intellectual engagement in the classroom—something that most guidebooks do not and cannot measure.") In July 2019, Reed students investigated the "penalty" Reed College pays for refusing to play along and found that "[i]f USN faithfully followed its own formula in the 2019 rankings, Reed would be ranked at #38, rather than its assigned rank of #90." Further, the graph about financial resources estimated and actual rank is quite jaw-dropping.

What I love about what I have read regarding Reed College is its unapologetic focus on the intellectual life of the university - what its new-ish President Audrey Bilger refers to as "the intellectual odyssey of a lifetime." From its website: "The curriculum includes a yearlong humanities course, broad distribution requirements, and a senior thesis. A 10:1 student-to-faculty ratio and small conference-style classes allow faculty members to truly mentor students and engage with them in individual discussions." Reed is also noted for the large percentage (about 25%) of its alumni who go on earning a PhD. 70% of its alumni earn a graduate degree.

Here are some features of the Reed education that caught my attention (from this page, but the emphasis in bold is mine, because in this overcrowded field of higher education where every institution clamors for more high-caliber applicants, distinctive academic experiences are needed to attract top students):

• "The educational program’s pedagogies are characterized by close interaction of students and faculty in an atmosphere of shared intellectual and scholarly concern and active learning. Small conference and laboratory classes are the norm, providing students the opportunity to demonstrate intellectual initiative and creative engagement."
• "The program in a student’s major area of study is an intensive examination of the objects, literature, theoretical concerns, and research models characteristic of an academic discipline. Among the requirements for the major are successful performance on a junior qualifying examination, completion of a yearlong senior thesis based on original research or artistic expression, and a successful oral defense of the thesis before an interdisciplinary faculty board." (No wonder so many graduates go on earning a PhD after this!) 
• "The Reed education is distinguished by a yearlong interdisciplinary humanities course, required for every first-year student. The college further encourages interdisciplinary study through upper-division humanities courses, as well as established interdisciplinary majors representing areas of research and scholarship that span traditional disciplines."
• "To ensure the highest quality education for its students, the college supports and encourages scholarly research by the faculty and the application of such scholarship throughout the teaching program."

The operating principles are also worth a read, especially the part where Reed keeps a transcript of grades but professors focus on giving qualitative feedback to students. (Specifically, on ctcl.org : "While Reed maintains a traditional transcript for students, feedback from professors is qualitative and does not include letter grades." From the Reed website: "To emphasize learning—not striving for a grade—exams, papers, and lab exercises at Reed are returned with extensive feedback from faculty instead of letter grades; although grades are recorded for all classes, they are not routinely reported to students unless students ask to see them.") Reed has 23 departments and 40 majors, including 15 interdisciplinary majors, such as Chemistry-Physics, Classics-Religion, Dance-Theatre, History-Literature, Literature-Theater, Mathematics-Computer Science, Mathematics-Physics and Religion-Political Science. Reed also has dual-degree programs: the one in engineering is with Caltech, Columbia University and RPI, the one in Forestry and Environmental Sciences is with Duke and the one in visual arts is with Pacific Northwest College of Art.

Of course Reed has sometimes been the focus of controversies, such as this. But the overall theme in the articles I have read about Reed is best summarized in the "Colleges that Change Lives" website ctcl.org : "If you’re a genuine intellectual, live the life of the mind, and want to learn for the sake of learning, the place most likely to empower you is … the most intellectual college in the country — Reed, in Portland, Oregon."

I love the description Reed provides of its students on its website.

 

 

 

 

 

 

 

 

Another description I liked was: "Reed College students are independent, curious, highly intellectual, and full of character. While no two are entirely alike, they all share one characteristic—a love of intellectual inquiry."

The accomplishments of Reed's faculty and students include: 3 MacArthur Fellows, 174 National Science Foundation Fellows, 103 Fulbright Students, 32 Rhodes Scholars and 68 Thomas J. Watson Fellows.

Interestingly enough, Reed does not have any fraternities or sororities.  

Back to blogging

It's been a while since I've updated this blog and it is high time I went back to blogging. As many of you know, I became Faculty Senate President at SMU back in May, and having that role in the middle of a pandemic has certainly been interesting. Colleagues always comment that I got a lot more work than I expected from the role when I ran for President-Elect back in the spring of 2019, and that is true, but I have also gotten a lot of joy and reward from advocating for my fellow faculty members and I keep saying that, if someone has to be Faculty Senate President in the times of COVID-19, I'd rather have it be me, because if I'm going to do that job, I might as well have an impact. So as I said in my remarks to faculty at the university-wide faculty meeting in late August, I am so glad I am Faculty Senate President this year. 

Also, there is simply no way I would have met so many of my talented colleagues both on the faculty and on the staff, or learned how many phenomenal things they are doing, without being involved in the Faculty Senate, and I'll leave you with a couple of books my colleagues have written recently.  

 

• 
• 
• 
• 
• 
• 

 

The Rise of a Prairie Statesman: The Life and Times of George McGovern by Thomas Knock

Figurines in Hellenistic Babylonia by Stephanie Langin-Hooper

How to get tenure by Michael Harris

(Coming soon) Reconstructing the corporation by Grant Hayden (and Saint Louis University professor Matthew Bodie)

Art and Vision in the Inca Empire by Adam Herring

(Coming soon) Decoding the digital church by Stephanie (Sam) A. Martin