Engineered

Analytics for Decision Support

I often get questions on how I teach business analytics, so I figured it'd be as good a blog topic as any, especially since I had a great conversation this morning with a graduate student who took that class her first semester and now has included me on her dissertation committee. The analytics course I teach (or taught until last year, since I've been teaching something else this year) is an overview to analytics and has no pre-requisite. It also has an undergraduate section (senior elective) and a graduate section. While the undergraduate version of the course is taken by students in our major, who are therefore well prepared by that time in their studies, the graduate version of the course is open to any graduate student in the school of engineering, which creates a much wider distribution of backgrounds and preparedness. Oh, and there is a distance (graduate) section too, and before the pandemic it was asynchronous, so ideally you have to make the course engaging for asynchronous viewing. 

I start by providing an introduction to the analytics cycle, also known as CRISP-DM (Cross-Industry Standard Process for Data Mining) and go over select articles from Harvard Business Review, especially Competing on Analytics by Thomas Davenport, which got the analytics field started back in 2006 (fifteen years ago!) and the spin-off articles he has co-authored, such as Data Scientist: The Sexiest Job of the 21st Century, from 2012. If I teach the course again, I will probably update the readings with one of his most recent articles such as 4 Ways to Democratize Data Science in Your Organization, from 2021. Also I think it is good for students or recent grads to read publications that the people they aspire at becoming read, and Harvard Business Review is definitely high on that list, although it is a little expensive if they can't get a relative to buy them a subscription as graduation gift (although perhaps HBR gives students and young alumni special rates). 

After the high-level introduction to data science, I usually dive into R and cover linear regression, logistic regressions, classification and regression trees (including random forests for the graduate students) before I take a little break from all the math and cover visualization in Tableau and R's ggplot and then return to clustering (hierarchical and k-means) before finishing with linear, integer and (my favorite topic) nonlinear optimization. I do tell the students that in practice visualization or descriptive analytics is done first, but in the past I've liked putting it halfway through the semester because it gives the students a nice little break at a time where they often have midterms. Maybe if I teach the course again I will just cover the topics in the normal sequence (descriptive analytics first, then predictive, then prescriptive) because it is closer to the real data science process. (Interestingly enough, I started teaching Tableau because a student of mine at Lehigh had just gone back from a summer internship at a consulting company where they had learned Tableau and told me I should teach it to my next cohort of students. Always keep in touch with alumni of your courses! They may have great information to share with you.)

My main book is The Analytics Edge, by my PhD advisor Dimitris Bertsimas along with his co-authors Allison O'Hair and William Pulleyblank. What I like even more than the book is the "Analytics Edge" course website on edX,org, which was instrumental in helping me understand how to pace a course that basically everyone can enroll in. While the book itself consists mostly of case studies without code, the appendix has a fantastic introduction to R for beginners and the book itself also has several good assignments for each chapter, solved in the instructor's manual. Sadly, I have taught the course for so many semesters that I am reluctant at using those assignments again, due to previous students' answers floating around the Internet.

So what I do instead is to use CSV datasets from kaggle.com. In an ideal world, we would have the data in databases and we would extract it to CSV in SQL queries so the students would get to see the entire process, but the semester has only so many weeks, and it's unrealistic to try to do too much. So what I do is I go over the datasets that have received many up-votes and look for something that would illustrate well whatever topic I covered. (I change the datasets every semester, because of websites like coursehero.com where students like to dump their assignments for the next crop of students to look at.) Kaggle.com is such a wonderful thing to happen to data science, just like GitHub.com . I also use a dataset from kaggle.com for the final project.

Somewhere along the way, I ask students to write an essay on ethics in data science, using a list of readings I provide to show them the dangers of data science being misused, and then also asking them to add a few more papers on an angle that is particularly interesting to them. This is where the double majors try, when appropriate, to tie the different parts of their curriculum together. I vividly remember a double major in management science and philosophy who was just so happy to use his entire training to write a particularly meaningful essay. 

Now the Analytics Edge book provides a great introduction to R but it is only an introduction, and to meet the needs of all the students in the class including the more advanced ones, a few semesters ago I started also using the R for Marketing Research and Analytics textbook by Chris Chapman and Elea McDonnell Feit. Elea was once the chair of an Analytics conference while I was on the conference committee, which is how I got to know her and learn about her book, and if you don't have her book, you should get it. The book website is tremendously useful with lots of codes and slides for all the chapters. What I like about the book and the related code is that it introduces students to more advanced features and packages in R, so students who are further away on the learning curve when the semester starts still get to extract something out of the course. Many of the advanced students have also told me that the course, in the way I teach it, provides a nice summary of multiple dimensions of management science, so it is a nice way to wrap up one's graduate degree too.

Somewhere toward the end of the course, I also give a test (I don't have a final exam, only a final project, but I do have one exam in the course). What always amazes me is that you can distinguish most easily the students who are at the top from those who are not from the optimization (or prescriptive analytics) questions. Most students can learn the basics on predictive analytics, so they know, say, you put about 70% of the data in the training set and the rest in the testing set, or you use linear regression to predict continuous variables and logistic regression to predict 0-1 variables. But the weaker students have tremendous difficulty in modeling optimization problems correctly, and I think creating good optimization models from data will be the next frontier that will distinguish the top data scientists from the others, especially given the fact that few Master's programs in data science have time to cover optimization in any depth, due to the amount of material they have to teach the students in only a few semesters.

At the end of the course, I try to give one lecture on big data. This way, if the students have to interact with employees who specialize in big data, they will have some idea of how it is done. (The main course objective is to give the students the background to contribute meaningfully to such conversations, rather than turning them into data scientists overnight, which no single course can do.)

A final piece of advice I give my students, and if you have read thus far (let me know in the comments or @ me or DM on Twitter) you deserve to have it too, is that if you are interested in this field it is good practice to check out job offerings periodically for the required and recommended competences. This is in fact a piece of advice I give in the context of career management. Once a student has a job, it is useful to remain competitive on the job market by keeping track of what the companies are asking for. This may help students identify gaps or software they had never heard of that they would benefit to learn asap so that their skills are in top shape. So when they see in the list of required skills something they are not good at or have never heard of, they need to develop a strategy on how they can become better at it, so that they can be given a chance if the opportunity arises. I'll try to write a blog post on a couple of job ads as an example, when I am done grading all my end-of-semester papers!

More analytics in a Big Data world, Part 3

Here comes Chapter 4: Descriptive Analytics of this book. In the analytical process, descriptive analytics usually comes before predictive analytics that we saw in the previous chapter, because we want to get a sense of the data we have before making predictions with it, but as one of the many peculiarities of this book, here the author covers descriptive analytics after predictive analytics, perhaps because predictive analytics is more advanced and more interesting, and many readers don’t read books in full, so it may make sense to put more of the interesting stuff upfront. Anyway, this chapter focuses on association rules, and the key metrics of support and confidence. Association rules are basically “if X happens then Y happens in this % of the cases”. The support of a rule is the percentage of total transactions in the data set that contains X or Y. The rule “X implies Y” has confidence (c) if 100c% of the transactions in D that contain X also contain Y (meaning Y indeed happens when X happens). 

Then association rule mining is about:

1. identifying all item sets that have support above a prespecified threshold called minsup (called “frequent item sets”)
2. discovering all derived association rules having confidence above a prespecified threshold minconf. 

The lift, measured as support of X or Y divided by the product of the support of X and the support of Y, quantifies whether those association rules better predict what happened than a baseline model. Wikipedia has a good example of lift for market basket analysis. It also explains that the lift is the ratio of the observed support to that expected if X and Y were independent. If the lift is less than 1, the items are substitutes to each other. 

The author then moves on to discussing sequence rules. Association rules simply investigate which items appear together at the same time but sequence rules study which items appear at different times. We create sequences from transaction data and then calculate support, although in the “X implies Y” rule we have to distinguish whether Y should happen immediately after X or rather at any subsequent in the sequence. 

We are then treated to a discussion of hierarchical clustering, various distances definition (Euclidean and Manhattan for distance between points and single, complete, average and centroid method for distance between clusters). Hierarchical clustering leads to a sequence of clustering schemes, starting with every instance being its own cluster and instances being progressively grouped together until there is only one big cluster. The next page is about k-means clustering and that one page includes two big graphs, only one about k-means clustering. I would have wanted to see some guidelines on how to select the number of clusters and the fine-tuning of parameters, since the outcome k-means clustering can depend on the initialization, which is random.   

Finally, the author discusses self-organizing maps (SOMs), in three pages. This was quite interesting, because SOMs are about visualizing high-dimensional data on a low-dimensional grid of neurons using feedforward neural network with two layers. SOMs are then visualized using a U-matrix, which is the map of the grid of neurons color-coded by the average distance between the neuron and its neighbors, and a component plane visualizing the weights between each input variable and its output neurons. I have enjoyed reading this Wikipedia page about it. The blog towardsdatascience.com has some interesting posts about it, which are written by data science students and thus may be more accessible to other students, but there is always the possibility they misunderstood something or coded it wrong. 

The following chapters are on survival analysis, social network analytics, “analytics: putting it all to work” and then example applications (although no code), but I will let you read those for yourself.

So again, the main interest of this book is to provide names of techniques and quick descriptions so that the interested reader can go and read some descriptions of it online. It's like an appetizer to data science. The author has enough real-life analytics experience that the reader can trust he knows which techniques matter more than others. I do really wish his descriptions of the techniques were more extensive but in data science it is difficult to hit the sweet spot between too short and too long because if you want to give more than cursory descriptions you have to introduce a lot of notations, mathematical equations, and give more detailed small examples throughout, which perhaps the author didn't have time for, and it also occurs to me the book may have had a different audience in mind. Also, I know authors don't necessarily pick their book title, but there shouldn't be "Big Data" in a title if you are not ever going to mention the challenges specific to Big Data. That's not his fault, though. All in all, maybe not a book I will consult frequently but not a book to send to the recycling bin either. A good introduction for MBAs who have to manage data scientists, who will definitely want this book on their shelves.

Analytics in a Big Data World, Part 2

Continuing on my previous post on Analytics in a Big Data World! 

Chapter 3: Predictive analytics

Here is an example of why I am less than enthusiastic about this book. Chapter 3 begins with: “Two types of predictive analytics can be distinguished: regression and classification. In regression, the target variable is continuous.” Well, no. That is true of linear regression but not logistic regression. But the author clearly knows this, since he talks about logistic regression later. This is just sloppy writing at its most obvious. 

In response modeling, the author makes the distinction between gross response (customers who purchase after receiving the marketing message) and net response (customers who purchase because they received the message). He provides a simple definition of Customer Lifetime Value and goes back to his favorite application of credit risk by discussing loss given default and the need to “check the robustness and stability of the target definition” doing for instance roll rate analysis. He then quickly defines linear and logistic regression and drops names like Bernoulli distribution (“the errors/target are not normally distributed but follow a Bernoulli distribution.”)   

Moving on to decision trees, he has some good graphs and examples for this topic, which lends itself well to visual aids. Splitting of the tree is determined using criteria such as Gini’s criterion or the Information Criterion for discrete variables or the mea squared error for continuous variables. Another way to measure the quality of spots is do calculate a F-statistic, where good splits have high F-value. 

Then, neural networks, where I found the author’s description to be likely unintelligible for the everyday scientific reader that his book is supposedly targeting, but perhaps I am underestimating his readers. There are many superior introductory treatments on the topic, including for instance this excellent post on the blog towardsdatascience.com  Those two treatments (of the book and of the blog post on towardsdatascience.com ) are so vastly different in quality, with the blog post being exceedingly better, that it is almost like they are from two different planets. The blog post has examples in Python too. And if you have a lot of time to read, you can read this book too, or that one, although they are a bit old.

Because neural network modeling is known as a black-box technique, meaning the manager doesn’t know what happens inside the neural network to motivate the final prediction, it is good to extract rules from the model to make it more understandable for the manager. Rules should be evaluated in terms of accuracy, conciseness and fidelity (ratio of correct predictions over incorrect predictions, i.e., ratio of sum of diagonal elements over sum of non-diagonal elements in the confusion matrix). 

Neural networks have two shortcomings: the objective function is nonconvex and the computational effort needed to tune the number of hidden neurons can be substantial. The author goes on to describing support vector machines to address those shortcomings. SVM problems are about finding a hyperplane separating a class of points from the other class of points (for instance the people who pay back their loans vs the people who don’t). Of course, perfect separation between the payers and the defaulters may not be possible and we need to account for misclassification. The algorithm will use some transformation of the data that doesn’t need to be known exactly; instead, what needs to be known is a certain derivative mapping of the data known as the kernel function. For a good introduction to SVM, I recommend this tutorial.    

The author then moves on to ensemble methods, that s, collections of models. Predictions are then made using, for instance, majority voting. The author goes very quickly over bagging, boosting and random forests. The paragraph about bagging starts with: “Bagging (bootstrap aggregating) starts by taking B bootstraps from the underlying sample.” Well, that explains everything then. In the next sentence, the author bothers to add “Note that a bootstrap is a sample with replacement (see section on evaluating predictive models)” [that section is later in the book.] The whole book is like that. More linear writing would have served the author well. 

At least the boosting section is well explained. “Boosting works by estimating multiple models use a weighted sample of the data. Starts from uniform weights, boosting will iteratively reweight the data according to the classification error, whereby misclassified cases get higher weighs. The idea here s that difficult observations should get more attention.” One of the first boosting algorithms is known as AdaBoost (short for adaptive boosting). 

Another ensemble method is random forests, i.e., a collection of decision trees. This was recently extended into rotation forests, which combine random forests wth principal component analysis (not something the author bothers to define, but a transformation that extracts uncorrelated components from a set of possibly correlated variables.)  

Then the author goes over multi class classification techniques: multi class logistic regression, multi class decision trees, multi class neural networks and multi class support vector machines. 

Mind you, by that point in the book, we are only at p.71, so that gives you an idea of how cursory the treatment of everything is. But if you want to learn names of data science techniques you can learn about later, it is not a bad book. Maybe a little expensive but not bad. 

Then we move on to evaluating predictive models. This is where the author talks about splitting the data in a training and a testing sets. I think that the author’s statement “note that in case of decision trees or neural networks, the validation sample should be part of the training sample because it is actively being used during model development” is not completely true: you create a model using the training sample and if there are two models you are hesitating between (say, with and without an independent variable), you take a piece of the testing set, call it validation set, try your two models on that, pick the better one, and then implement it on the remaining part of the testing set (“true testing set”). You didn’t know ahead of time you would hesitate between two models so you had your testing set ready but you then had to split it between validation and testing set to be able on which model to keep. So in that sense, the part of the initial testing set that became the validation set is part of the training set because it helps design the model, but I am not sure the reader will pick up on that. Of course, if you know in advance you will have multiple models to evaluate, you could split into training/validating/testing set before you even start. The author also explains cross-validation, although the explanation in the Analytics Edge edX course is better. 

Performance measures for classification models include classification accuracy, sensitivity and specificity (just like I tell my students in class). Classification error is one minus the accuracy. The author also discusses the Receiver Operating Characteristic (ROC) curve and the Area under the Curve (AUC), again better explained in the Analytics Edge edX course. A good model generally yields an AUC of 0.8 or higher.  

Another important performance metric is the lift curve. The data set is sorted in deciles from low score to high score of the dependent variable (what we are trying to predict). The lift curve computed, for each decile, the ratio of the percentage of what we are predicting using our model over the percentage of the dependent variable in the entire population. If the model isn’t better than random guessing, the lift would be 1. The cumulative accuracy profile (CAP) computes the cumulative percentage of what we are trying to predict (the “1” in a binary dependent variable) for each decile. Then the CAP curve can be summarized by an accuracy ratio or Gini coefficient.

Other performance metrics are the Kolmogorov-Smirnov distance (maximum distance between the cumulative score distribution of the good vs the bad customers in credit scoring), the Mahalanobis distance between score distributions and the divergence metric.   

Performance of regression models is evaluated using R^2, Mean Squared Error (MSE) and Mean Absolute Deviation (MAD). 

Coming up: Chapter 4 on Descriptive Analytics in my next post. 

Analytics in a Big Data World, Part 1

I bought this book to see how practitioners implement analytics in real life, where the data sets are usually very large, but the book does not cover Big Data at all. It is just something in the title to draw sales. The book, in fact, aims at being some sort of reference book (no exercises, no code, no data sets) but the concepts are presented so quickly and defined so perfunctorily as to make the book almost useless, except to name-drop key concepts that the interested reader can then go and look up online. Based on my reading of the beginning of the book, I expected the author to be a self-made consultant who had written a book to market his consulting services and was surprised to find out he is a faculty member. This is definitely not geared toward teaching analytics to his audience. When you teach something you make sure to define all the terms clearly when you first introduce them and increase the degree of complexity of the material as you go along. This is rather a reference textbook but is too think to do a good job of it.

However, there are many nuggets of wisdom in the text so it is still useful. In my opinion, the intended audience for this book consists of business managers of data scientists who want to know what names to throw around when they talk with their subordinates and have a one-line idea of what the topic is about. (And I don’t always agree with his one-line descriptions of concepts, for instance he says clustering is part of descriptive analytics and for me descriptive analytics is strictly visualizations and dashboards, that sort of things, but the equations he provides of the quantities he defines are correct.) Although the book is too vague to be useful and doesn’t involve any sort of computer codes, the author clearly knows his topic. There is value in knowing those topics so here is a list of ideas and concepts I found useful and some quick explanations of them (not necessarily from the book.) 

Chapter 2 Data collection, sampling and preprocessing

I liked that he talked about checking the signs of the coefficients in a regression model as an example of expert-based validation (“do the signs make sense?”) because that’s something I always tell my students do. He also has a great example of sample bias in credit scoring, where we don’t know if the people who did not get the credit would have paid the loan back and it is important to remember that the people who did not get credit before the analytical approach was implemented were refused credit because the company followed some sort of policy although we don’t know what it was. 

For outliers the author makes the distinction between valid and invalid observations and then goes on quantifying outliers with box plots, z-scores (computed by subtracting the mean of observations from a specific observation and then dividing by the standard deviation). He talks about truncation/capping/winsorizing valid outliers to bring them back into a more reasonable range and then standardizing data. (He has a tendency to drop names without defining them or defining them later.) Coarse categorization, i.e., placing instances into buckets, can for instance be done with equal interval binning or equal frequency binning, or through more sophisticated analysis techniques such as the chi-squared method. (Wikipedia has a far clearer example of what the method does for categorical data than what the author writes in his book.)

In linear regression with categorical variables, the computer must estimate coefficients for each of the factor’s levels minus one. This can introduce a lot of factors. It would be better (more manageable) to have only one new coefficient per feature and have some number associated with each of the levels, if possible (this will not always be possible but could be for instance when the categories are about age groups, when a clear ordering exists). Weight-of-evidence coding is a transformation that achieves that goal and is widely used in credit scoring (the application most familiar to the book’s author). For it to work well you have to bin the levels into a manageable number of categories. Then the WOE for one of the bins is computed as ln(Event%/Non-Event%) to use the definition on a more helpful Medium.com page. 

To perform variable selection for an analytical model, we can consider correlations between each variable and the target, such as Pearson correlation (the correlation most engineers think about first), Fisher score (which generalizes into analysis of variance or ANOVA, which he assumes the reader knows about), Information Value or IV (which uses the WOE weights mentioned earlier and is considered medium predictive for IV>0.1 and strong predictive for IV>0.3) and Cramer’s V, which is the square root of “the chi-square divided by the number of observations” and should be higher than 0.1. 

Chapter 3 will be discussed in my next post!

GrAIt Expectations: AI in Business

 

I recently came across this special report on AI in Business, entitled "GrAIt Expectations", in an old issue of The Economist. Overall I found the special report thoughtful and well documented, with lots of interesting examples. Some are better known than others, for instance it is no surprise that big companies such as Johnson & Johnson and Accenture are now using Artificial Intelligence (AI) to sort through job applications and pick the best candidates, but AI can also be used in more cutting-edge and perhaps disturbing ways: the Chinese insurance company Ping An thinks it can spot dishonesty by having prospective borrowers answer questions by video.    

I am not sure if I agree with the definition of AI: "AI and machine learning (terms that are often used interchangeably) involve computers crunching vast quantities of data to find patterns and make predictions without being explicitly programmed to do so." I tend to think that artificial intelligence is the ability of computers to make decisions, rather than predictions, without being explicitly programmed to do so. Or we can adopt the definition in the Oxford Dictionary, which defines Artificial Intelligence as "the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages." According to a report by MIT's Sloan Management Review and the Boston Consulting Group, "around 85% of companies think AI will offer a competitive advantage, but only one in 20 is "extensively" employing it today."

This got me interested in reading the MIT/BCG report itself. In fact there are three reports, since this is a yearly effort that started in 2017. The authors give the example of Airbus, which uses AI to match about 70% of the production disruptions to solutions used previously, shortening the amount of time it takes to deal with disruptions by more than a third. Because the report is already about 2 1/2 years old, we are nearing the time where we can check the validity of predictions such as "within the technology, media and telecommunications industry, 72% of respondents expect large effect from AI in five years." They split their survey's respondents into 4 groups: pioneers (19%), investigators (32%) [referring to organizations that are deploying AI at the pilot stage only], experimenters (13%) [organizations that are piloting or adopting AI without deep understanding, which frankly sounds like a bad thing] and passives (36%) [organizations with no adoption or much understanding of AI]. The 2017 report particularly emphasized the need for good data in order to create good AI models. Apparently a lot of companies didn't have the historical data required. Some applications such as drug development need negative as well as positive data (what didn't work as well as what did work), while only the positive data tends to get published. The report's authors make an interesting point about the make-or-buy problem applied to AI capabilities. They say that the leaders in using AI for business develop their own in-house AI capabilities, while the followers tend to work with independent AI vendors. But it is going to be important for organizations to have their own AI in-house capabilities. 

The MIT-BCG report lists several managerial challenges: (1) developing an intuitive understanding of AI, (2) organizing for AI (especially having the organizational flexibility that enables collaboration between humans and machines on project teams, taking into account whether AI resources should be managed centrally or locally or in a hybrid model) and (3) re-thinking the competitive landscape. Among the authors' recommendations (ensure customer trust, perform an AI health check, brace for uncertainty, adopt scenario-based planning, add a workforce focus), I found the scenario-planning one the most intriguing. "Companies need to think even more expansively about their businesses, build cohesive future scenarios, and test the resilience of their directional plans against such scenarios."

In a 2018 update, the MIT-BCG authors identify four major patterns in their survey and interview data:

• Pioneers are deepening their commitments to AI ("fully 88% of Pioneers invested more in AI than in the previous year")
• Pioneers are eager to scale AI throughout their enterprise ("85% [of Pioneers] agree that they have an urgent need for an AI strategy and 90% say that have a strategy in place already.")
• Pioneers prioritize revenue-generating applications over cost-saving ones because "a much higher penalty is incurred by missing an opportunity in the external market" rather than internally. Examples include "complex pricing applications, churn analysis in the customer portfolio, new delivery forms that significantly speed up [the company's] ability to meet customer demand" as well as "wholly different business model[s]."
• AI is creating both hope and fear among workers due to the possibility of workforce reduction.

Many respondents also expect change to their business model. Also, it is important not only to get more data science resources but also train more businesspeople in the potential of AI to think about potential use cases.

The 2019 update started on a less upbeat note: "Many AI initiatives fail. Seven out of 10 companies surveyed report minimal or no impact from AI so far." The authors argues that "companies that capture value from their AI activities exhibit a distinct set of organizational behaviors. They integrate their AI strategies with their overall business strategy, take on large, often risky AI efforts that prioritize revenue growth over cost reduction, align the production of AI with the consumption of AI, through thoughtful alignment of business owners, process owners and AI expertise to ensure that they adopt AI solutions effectively and pervasively, unify their AI initiatives with their larger business transformation efforts, invest in AI talent, data and process change in addition to (and often more so than) AI technology. They recognize that AI is not all about technology." As an example, Aetna already uses AI to "design provider networks, prevent fraud and discover overpayments - traditional applications of analytics within the insurance industry" but "is now pursuing strategic initiatives to create more customer value with AI. In one Medicare-related offering, product designers used an AI-based method to customize benefit design." Interestingly enough, the percentage of Pioneers + Investigators remains at 50% of respondents, with 20% of Pioneers. Risk from AI is particularly strong in highly regulated industries like banking, which are being disrupted by new entrants such as Apple Card and Amazon Cash. "Out of three major possible applications of AI - cost reduction, revenue generation and new product development - most of the Pioneers are applying AI in at least two ways." The report's authors also emphasize the need to focus on the consumption of AI, not just its production. This means "developing a fertile environment in which producers can develop, champion, and implement AI solutions in strong collaboration with the business" and "developing sufficient expertise among business users so they can properly use the probabilistic nature of many AI solutions." Basically, it is about cultivating nontechnical leaders and business users.

AI is "not just a corporate race but an international one," especially between the United States and China. The MIT-BCG authors even started to conduct a separate survey with Chinese executives in 2018. "Chinese AI Pioneers are investing more aggressively and report a greater focus on business model transformation."

The MIT-BCG authors argue, based on their written questionnaire and interviews, that the key benefit of AI is not cost cutting but revenue generating, in particular in areas such as new product design. This actually seems to contradict The Economist, which claims that "in private, many bosses are more interested in the potential cost and labour savings than in the broader opportunities AI might bring." A consultant insists: "If you just cut costs and don't increase value for customers, you're going to be out of the game." Both The Economist and the MIT-BCG teams agree that AI pioneers may develop an unsurmountable competitive advantage, leading to corporate concentration and monopoly power.

The best known application of AI is for operations, specifically early maintenance. "Companies can combine data on past performance with those generated by smart sensors on machinery (part of the much-hyped "internet of things") to predict when a jet engine or a wind turbine is likely to fail," says The Economist. Other improvements will occur in demand forecasting and inventory management, leading to decreases in overstocking cost (estimated at $470bn worldwide) and understocking cost ($630bn). Packages are routed more efficiently - and INFORMS long-time member Jack Levis is even cited in the article, saying that "for every mile that [UPS] drivers in America are able to reduce their daily route, the firm saves around $50m a year." There are also enormous opportunities in customer service, where text mining (for emails) and voice analysis (for phone calls) can triage the most urgent service requests from the others. 

In human resources, a novel application will be to identify potential candidates for new positions from the pool of past applicants, and re-direct candidates who are not a good fit for the position they applied to toward positions that better match their skill set. But companies need to monitor their algorithms regularly to make sure they are free of bias, in particular for groups that are under-represented in the training set. The workplace of the future, though, may be much "creepier" than what we are used now due to employee monitoring. The example of startup Humanyze was particularly telling. Employees wear an "ID badge the size of a credit card and the depth of a book of matches. It contains a microphone that picks up whether they are talking to one another; Bluetooth and infrared sensors to monitor where they are; and an accelerometer to record when they move." This is supposed to provide a "full picture of how they spend their time at work." 

The potential corporate market for AI software, hardware and services is estimated at around $58bn by next year. Cloud providers such as Amazon and Microsoft have a clear advantage, while consultancies have been strengthening their capabilities. For instance McKinsey bought QuantumBlack in 2015. It remains to be seen whether AI wizards go to work for conventional consultants. The Economist argues that even IBM "finds it hard to get hold of the best talent. None of the top doctoral candidates in AI goes to work for IBM."

Personalized recommendations and faster deliveries sound like good things, and creating jobs that only exist because of AI or developing new drugs thanks to the power of machine learning is very exciting. But according to the McKinsey Global Institute, 14% of the global workforce could have their jobs automated away. This will require workers' retraining, which companies are reluctant to pay for. The second issue is of course data privacy. A third concern is a winner-take-all situation where the companies that win the AI race put their competitors out of business, increasing concentration in the tech sector and beyond.

How to win at online learning

About a year ago UT Austin announced the demise of its Project 2021 after only a two-year attempt at implementation (which for universities is rather short, given that only a few iterations of a course can fit into one academic year). I thought it'd be useful to revisit it as we in higher education are constantly talking about online education, both as a tool to help undergraduates master the curriculum and as a way to attract graduate students who cannot relocate near the university. 

When Project 2021, as it was called, was launched in 2017, The Daily Texan described it as "based on research aimed at enhancing students’ experience and strengthening their social ties with peers and faculty." The lead faculty member said: “I frequently ask alumni the three most formative experiences they had in college and I feel lucky if one of them is something that happened in the classroom", while an economics senior "said his social experience in college has been far more memorable than academics and that teachers should focus more on the interaction aspect of education to make classes more engaging." This was perhaps not an auspicious beginning and in hindsight it may be no surprise that the project failed. A recent study suggests that the quality of video productions positively affects knowledge retention, making online learning an expensive value proposition. 

UT Austin still seeks to provide more experiential learning to its students. "Launched this January, the Experiential Learning Initiative is a pilot program housed by the Faculty Innovation Center that seeks to accomplish Project 2021’s goal of providing students with real life situations that develop problem solving skills related to their field." This raises its own set of challenges as projects in fields like computer science need to be designed in a way that removes the incentive to hire someone else to write the code, for instance through Upwork. You'd think students want to learn and most do but a few may have other priorities, especially if the course is difficult and they have a full-time job that also needs their attention. It's a problem I was confronted with last semester (note that I don't work for UT Austin), although the vast majority of the students were hard workers. There has to be a way to design experiential learning like hackathons so that the students who submit the work are the ones who truly did it. 

I am reminded of what Clayton Christensen said about products, which is (I paraphrase) that companies had to understand what "job" customers were "hiring" the product to do when they purchased it, otherwise any attempt at innovation was bound to fail. I recall he gave the example of enhanced textbooks that had failed to gain traction among students, because it turned out most students weren't particularly looking to master all the nuances of the material and go above and beyond what was expected of them, but really just wanted to pass the course. Graduate students don't just pay tens of thousands of dollars to get another piece of paper at Commencement but also to benefit from a network of peers and alumni and the atmosphere of discovery on college campuses. Videotaping the lectures and putting them online is better than nothing, but I think the university that will master the "soft" (social) aspects of online education will have an advantage over its competitors. edX attempted to do something like that when it encouraged the creation of local communities of learners, but if anyone can register for a M.O.O.C., then students have no idea whether they have anything to learn from other students enrolled in the course or if their peers are hopelessly behind. One possibility is to use local companies in the vicinity of the university (something that is far easier to do in Austin or Dallas than in other parts of the country), as they might send their employees for an online Master's degree as part of their employee benefits, which helps with recruitment and retention. Those employees may benefit from knowing each other, as professionals seeking the same educational goals.

When it comes to giving a taste of the university's intellectual atmosphere, too often it is all or nothing: either a lecture isn't videotaped, or it's videotaped and posted on YouTube for everyone to watch. But there should be a middle ground with a website that is accessible only to current students (and perhaps dues-paying alumni) that allows them to watch content others cannot see and also, when possible, to submit online questions to the speaker and interact with fellow attendees. If universities only treat online Master's students as cash cows to meet their tuition goals, they may not get much loyalty from such students once they are turned alumni, depriving current students from the opportunity to interact with rising stars and benefit from alumni's insights and advice.

Online education at the undergraduate level, in my opinion, makes the most sense to provide refreshers of prerequisites and in the flipped classroom model. "Refreshers of prerequisites" are particularly important these days where universities are bracing for a change in demographics and are trying to attract new segments of the population, including transfer students and non-traditional students. Those students may have completed their Associate's Degree several years back and most likely will not have the money to retake a course they have already passed even if it would make their path to graduation smoother. Universities need to take into account both that those students want and often need to graduate as fast as possible for financial reasons, and that their success may require support that traditional students, who entered college at 18 and stayed a straight 4 (or 5 or 6) years there would not have needed. As for the value of the flipped classroom environment, it has been subject to debate, with some researchers noticing benefits in learning and others discerning no effect at all or a positive effect only for white high-achieving males (admittingly at West Point, which attracts a very specific type of students and is not known for its large number of female students, but since the white paper is from MIT, this report will probably get more attention than it should based on its contents alone). 

When it comes to online learning, it might be that the burden of producing professional-quality videos should be on textbook publishers rather than universities. Of course all universities want to showcase their experts, and M.O.O.C.s have been a good marketing tool for that, but ultimately universities may want to put their resources elsewhere (although of course that might just provide textbook publishers with another incentive to increase their already exorbitant prices). I particularly like what McGraw-Hill is doing with its Connect approach. A smart book "personalizes learning to individual student needs, continually adapting to pinpoint knowledge gaps and focus learning on concepts requiring additional study. For instructors, SmartBook tracks student progress and provides insights that guide teaching strategies and advanced instruction, for a more dynamic class experience." The online resources also place students in immersive real-life scenarios as they apply course concepts and include videos to help with challenging topics. Further, Connect Master replaces narrative-driven chapters with 2- to 4-minute-long professionally-produced videos on 250 topics. So perhaps the future of online learning is through connections with textbook publishers that will produce and disseminate high quality videos and supporting tools to a far bigger segment of the student population than would take the course at a given university. That also means that the universities whose instructors are being videotaped for such textbooks may seem more knowledgeable or prestigious than those who aren't, and may increase disparities between universities, with some producing novel content and taking a more creative role and others assigned the task of communicating content developed elsewhere to their own students. It is perhaps not very glamorous but, with the tuition pressures looming ahead, fewer and fewer universities will have the resources to spend on high-quality videos of their lectures. The other thing that may happen is that universities may cut out the middleman  altogether and directly purchase online-augmented courses from each other. Either way, it is a fascinating time to teach at the university level. 

What place for optimization in data science?

 I was looking at SMU's online Master of Science Program in Data Science the other day and unsurprisingly - given the discussions we've had last year at the meeting for program directors of Master's in Analytics - there isn't any time to squeeze a course on optimization among 30 credits on applied statistics, database management, network security, data mining and the like.

Yet a key takeaway from last year's meeting, which took place right before the INFORMS Analytics conference, was that while companies now were very excited about getting insights into the data they have (this is known as predictive analytics), the next frontier is to use insights to improve decision-making through optimization (or prescriptive analytics). But we analyze data to create models to make better decisions and in today's complex environment, in many cases it is not possible to think from the top of one's head of only 2 or 3 potentially optimal strategies that we could then compare to select a winner.

If you only care about, say, predicting customer demand for a product, you're missing the point. The reason why you care about the demand is because you want to know how many items to produce now and in which warehouses to stock them so that our customers won't be dissatisfied and go to a competitor if they come to our store and find the product they want out of stock. So a big part of data science is to estimate accurately the data inputs that go into mathematical formulations. For instance you can create a linear regression that will predict demand as a function of price, but then you do want to find the optimal price to maximize revenue. 

Data science is still a new field, so you can find plenty of applications where managers are currently very happy limiting their analytical work to predictive analytics and using those insights to develop a few action plans. For instance, in trying to predict health care costs, a health care company may place patients in "buckets" and assign a specific strategy for each bucket, from nothing (hands-off approach for the healthiest patients) to case management (for the sickest patients). But in the future we could evolve toward a greater customization of health insurance plans so that patients would be offered personalized prices for various health insurance plans, and the question would be to determine what prices to offer to drive the patient's behavior toward health. Or we can cluster users but then we want to provide each group with specifically tailored incentives. For instance, should a loyalty program send a gift card to certain users? for which amount? how long should the customer have to use the card? Right now many companies are still trying to understand the data they have and grabbing the low-hanging fruit. But as a glut of Master's in data science come on the market, programs will seek the next competitive advantage and that advantage will be optimization, especially the optimization of complex systems that can't be done by the human brain alone. 

These concerns above may well have played a role in MIT's development of a new joint major at the intersection of economics, computer science and data science, dubbed 6-14 (at MIT Electrical Engineering and Computer Science is Course 6 and Economics is Course 14). From the MIT News Release: "The new major aims to prepare students to think at the nexus of economics and computer science, so they can understand and design the kinds of systems that are coming to define modern life. Think Amazon, Uber, eBay, etc. “This area is super-hot commercially,” says David Autor, the Ford Professor of Economics and associate head of the Department of Economics. “Hiring economists has become really prominent at tech companies because they’re filling market-design positions.” Because these companies need analysts who can decide which objectives to maximize, what information and choices to offer, what rules to set, and so on, “companies are really looking for this skill set,” he says." Needless to say, this all falls sharply within the domain of optimization. You can read more about the degree programs here or here.

Another way optimization affects data science is in the underlying analytical problems, which are solved using nonlinear solvers (for instance logistic regression is solved by maximizing log-likelihood). While it might be more important today to use optimization (a rigorous quantitative framework that incorporates many possible feasible strategies that the human mind can't all evaluate individually) to improve managers' decisions, there is no doubt that existing techniques aren't computationally tractable on the large-scale data sets that are becoming available today and extracting information from such examples of Big Data will require special-purpose algorithms that are adapted to the problem at hand. Hence, it is always good to understand what goes on behind the scenes when you call a specific function in your favorite programming language. In that case aspiring data scientists can take Optimization Methods in Data Science at Northwestern, Large-Scale Optimization in Data Science at Princeton, Discrete Optimization and Scalability for Data Science at Kansas State, Optimization for Machine Learning at EPFL or at least go through these informative slides by Stephen Wright or those from WikiStat. I also recommend this tutorial on stochastic optimization for big data analytics. What the courses in optimization for data science above hint at is that, in the workforce you might need to adapt existing off-the-shelf algorithms to deal with a large data set, so you may not be able to get any output at all to your data science analysis if you don't know optimization. The programs that fail to include an optimization component in their training may well prepare their students only for small-data data science. 

There are many good things to be said about the SMU online Master's in Data Science. I particularly liked the 3 immersion programs for the introductory weekend and the 2 capstones, and the fact that the immersion conference coincides with one of the capstone experiences, allowing new students to judge for themselves what they will be able to do in a few months. First of all, for a degree in data science it seems paramount that the students test their skills in a capstone format. Although it is harder on faculty to supervise capstones, the best students will gravitate toward programs that provide the sort of high-touch environment that allow students to distinguish themselves. Also, I liked the fact that the capstone is divided in 2 parts where the second part is aimed at giving time for students to revise their project. I believe low-residency programs have a major role to play in educating tomorrow's workforce, by providing the flexibility to study remotely and opportunities to expand one's network and connect face-to-face with one's peers. Face-to-face interaction is also facilitated by live classes delivered online every week. From the website, there seems to be a mix of synchronous and asynchronous teaching. If we are to attract more and more graduate students, we have to free the time to answer their questions and advise them properly. This might be achieved by faculty recording segments of the lectures and checking in with the students in-between modules through short live sessions for clarifications, so that time spent going over the same material from one semester to the next is freed for advising. In fact when I see the screenshots of the app I wish I could develop my analytics course using that format too. 

Ultimately it may well be difficult to include optimization in a Master's program in Data Science due to time and credit constraints, which raises the question: should students start their training in data science earlier, as part of a Bachelor's degree? Or, using Johns Hopkins' summary of the differences between D.Eng and PhD here, should we create a new professional degree, for instance a Doctorate of Engineering in Data Science, to give students time to cover all the intricacies of data science, including optimization? What is clear is that optimization is already part of the tool set of the highest-performing data scientists, and as we move toward always bigger and more complex data sets, optimization will only become more critical to companies' competitive advantage in extracting value from their data and acting on those insights.

A 7:30am session

Lo and behold, that session on robust optimization organized by Dan Iancu of Stanford University was very well attended, showing that early sessions on the right topic can attract good audiences, although I hope INFORMS never experiments with a 7am start. Best of all was that Dimitris Bertsimas himself showed up to listen and stayed the whole time.

Peng Xiong talked about the adaptive robust optimization software AROMA (that was joint work with (Zhi Chen and) Melvyn Sim, whom I was thrilled to see again after many years. Back in the days we were pioneers of robust optimization. It is fun to think how much robust optimization has become mainstream now.) Dan Iancu of the Graduate School of Business at Stanford University talked about dynamic robust optimization and discrete convexity. Louis Chen talked about distributionally robust linear and discrete optimization with marginals (joint work with David Simchi-Levi and others). 

I talked about some of my recent work on multi-range robust optimization. (I guess you can say it was a MIT-alumni session.) Multi-range RO is a topic dear to my heart since I started it some years ago with my then PhD student Ruken Duzgun. It so happens that after our paper got into a bad reviewing situation and was ultimately rejected by EJOR, a European team investigated very similar concepts under a closely related name. Ultimately Ruken's and my paper got published as conference papers. Ruken has done such outstanding research that I don't think has received the visibility it deserves, but she has a great job at Microsoft now so I suppose her career turned out alright.

It made me happy that DB had positive words about my presentation and I'm looking forward to presenting my new results on multi-range RO very soon. Definitely at the next INFORMS annual meeting in Seattle if not before!

"Is 'fake' data science a threat?"

I finally got around to reading the June 2018 issue of OR/MS Today, with its catchy headline "Is 'fake' data science a threat?" Can anyone say no to that? The cover admits sheepishly: "Yes. How should the O.R. profession respond to 'charlatans'?" When I read the article itself, I was surprised to note that it seemed directed toward data science itself as a potential threat to operations research and makes sweeping statements such as "O.R. has always focused on decisions or actions", "data science often stops at having found interesting phenomena", "'charlatans' claim to be doing traditional statistics with little evidence for their claims" and "data science appears to be the latest and possibly most threatening manifestation of a desire by people from other disciplines to get the benefit of OR/MS and applied probability without having to learn much mathematical statistics." More reasonable is the comment that practitioners should have to "consider the theoretical justifications to [their] calculations" and know how to "check whether the assumptions underlying the analysis are defensible."

The article attempts to tackle two different issues at once: (1) the greater name recognition of data science compared to operations research, and (2) the fact that with the hype for data science has come the danger that under-qualified people will get hired as data scientists and end up doing more harm than good in their organizations. The name "operations research" has come to mean much more than research on operations for, say, the military facing logistical challenges during World War II, encompassing research from finance to health care. This has made it hard to communicate what we do to a lay audience. On the other hand, "operations research" sounds obscure enough that it conveys easily the difficulty of the techniques involved. These days, everyone generates data through Internet browsing and smartphone use, and uses data, from Amazon.com ratings to sports statistics, so data science seems familiar enough, for better or for worse. 

As interest for analytics and data science has grown after the 2011 McKinsey report on Big Data (with a 2016 followup), universities have launched analytics programs by the dozens where they try to cram as much information as possible on descriptive, predictive and prescriptive analytics in the meager 30 credit hours they have to transform students into analytics experts. This puts pressure on faculty to cover a lot of topics fast. I don't think anyone working in data science today wants to view himself as a charlatan, and I would rather stay away from the name-calling. But what to do with students who may be weak in certain areas of analytics or have forgotten the assumptions underlying the models? Or consultants who try to generate follow-up projects at any cost?

It might be hard for academics to admit it, but a lot of people in industry aren't really after mathematical rigor. Academics are after mathematical rigor because that shows good teaching and because rigor is necessary to get their papers published. Practitioners are after improving their current business situation, and the mathematical rigor is helpful to generate buy-in for whatever idea they come up with. If someone has an insight that saves money, practitioners may well implement it, even if, say, the assumptions for linear regression weren't satisfied. This is very upsetting for academics to think about, and hopefully nowadays we have enough success stories about the proper use of advanced analytics at Amazon.com or Uber to motivate more practitioners to rigorously implement data science. But maybe some of the people viewed as charlatans are, in some circumstances, pragmatists and realists. (Not all, though. Some might well be incompetent. See paragraph after next.)

Perhaps the best solution is to give other graduates the tools to recognize people whose understanding of data science is very superficial, and in that sense it is beneficial to graduate as many would-be data scientists as possible so that they can challenge co-workers' assumptions and methodologies. About six or seven years ago I interacted with a company that was trying to predict college loan defaults and the person who had created the model had used a linear regression to predict 0/1 variables. In those days many students only ever saw linear regression as a predictive analytics technique. Nowadays, I like to think more students are familiar with logistic regression, and soon that will mean more managers too. This is an example of situation where it is best if managers have the same technical background as the employees they supervise.

We also should recognize how the rise of social media platforms such as LinkedIn and Twitter have pushed some individuals to try to brand themselves a certain way without having the formal qualifications to do what they say they do in any sort of depth. Which is not to say that they are not necessarily capable of doing it, since some might have learned from studying books by themselves. Just that, given the sheer number of people who state they do analytics but have no formal, rigorous qualifications in their resume, the odds are that many of them either don't know what they're doing or are sticking to simple tasks that don't nearly represent the expertise of data scientists. 

But I also think we in academia have to pause and ask ourselves about the role we collectively might have played in creating this situation. Everyone wants to jump on the analytics bandwagon. Business schools create programs of varying quality on "business analytics", which might well be, in some programs, analytics watered down so that the non-quantitative crowd can get some training in data science that is neither rooted in statistics departments nor in operations research departments. Universities want to answer the needs of the workforce but they also want to generate abundant tuition money. In data science in particular, it is tempting (as one continuing education program I know of plans to do) to hire many adjuncts practitioners in the field supervised by one faculty member to teach as many students as will be willing to enroll. This has the advantage of scale and keeps costs down. But if those adjunct practitioners have superficial knowledge of data science, such programs will simply graduate more practitioners with superficial knowledge. 

So, what to do? The most advisable course of action for a company would be to hire enough PhDs in their data science teams (whether PhDs in math, statistics, operations research). This would allow a critical mass of PhDs who can interact with Master's graduates and identify flaws in their reasoning as well as offer mentoring. A Master's may be great in providing enough tools that the graduate can interface intelligently with more technical staff. But to know enough about all of data science and a lot about some specific area of it, a longer degree than the Master's might be needed.