Two Analytics Success Stories

Here are two success stories about analytics that initially appeared in the Winter 2011 issue of the MIT Sloan Management Review.

Helping students graduate in the Gwinnett County School District

The first one is related to education. It dates back from 2002, when Gwinnett County in the Greater Atlanta public school system decided to analyze the troves of data it had to improve high school graduation rates. Because data is currently used nationally to rate teachers and decide on tenure decisions, and also because people who understand analytics only superficially can generate analytical rules of dubious quality (see the linear regression with 32 decision variables to rate NY teachers I wrote about here), analytical approaches can have a bad reputation in the educational system.

But properly designed analytical studies can generate valuable insights too. The researchers found out that the single best predictor of high school completion was completion of Algebra I, which is a ninth-grade course. The issue was a lot of high-risk students had struggled in math courses before reaching Algebra I. But when they performed a follow-up analysis, the researchers realized that the single best predictor of Algebra I completion was... completion of creative writing in eighth grade. Creative writing. So the school district focused on helping students succeed at creative writing, and in turns that helped them succeed at algebra and finish high school, for reasons I'm not sure anyone truly understands. But it worked. In 2010 Gwinnett County School District receive the $1million Broad Prize for its work toward closing the achievement gap.

For those of you interested in learning more, this MIT SMR article is currently available for free view on the Forbes website.

Retaining customers at Assurant Solutions

The other analytics success story is about a company that sells credit insurance and debt protection products, and wants to improve its customer retention rate from its original 16% rate, which is said to be in line with the rest of the industry (meaning that when customers see the $10 or more they have to pay each month, 5 out of 6 change their mind and decide they don't need protection anymore).

The analytics team that was brought in did not consider current metrics such as average speed of answering phone calls, and instead focused on the end result: was the customer retained after the call (success) or not (failure)? What they found out is that, in the words of the author, "some customer service reps are extremely successful at dealing with certain types of customers. Matching each specific in-calling customer to a specific CSR made a difference." [I think that in-calling customers have to type in an identifiant or an account code so that the company is able to identify who they are and what premium they are paying right away.]

People don't really understand why some CSRs are more successful with more segments of their customer base than others - in particular customers with low premium vs customers with high premium - but understanding why was not important. In the past the customer center had routed the calls based on "expertise", but expertise, as the vice president of targeted solutions at the company explains, "is a subjective term" and is estimated based on anecdote. What was taken in consideration in the new study was the output: the success rate in convincing callers not to cancel. So the company developed an evidence-driven affinity-based routing that significantly improved its retention rate.

The affinity-based assignment was hampered by the fact that "the best matches are almost always not available... because that CSR was on the phone" which resulted in the creation of another complementary predictive model to estimate time to availability. This was driven by the fact that customers were willing to wait on the line a lot longer than the company had thought, balking at the 60-second mark rather than the 20- or 25-second mark.

The interviewee only makes a good point regarding why more people are not doing this: there is little appetite to adopt a new technology if the old system has been working fine. On the other hand, the program that IBM developed to help Assurant, called Real-Time Analytics Matching Platform, is now available to other companies. This made me happy to think that IBM hires so many of our IE graduates at Lehigh. Hopefully they get to play some role in the development of high-impact analytical solutions that help companies throughout the country in their chosen line of business - not only for the sake of improved customer experience, but also for the sake of the employees who benefit from working at a successful, prosperous company.

Comments

[Ilyaquant.wordpress.com]

Credit protection? Debt insurance? What the heck?! How would that even work?

Either way...I don't recall IBM hiring IE grads for analytical work so much as consulting paper pushing and silly Excel spreadsheets. That, or I would have had a couple of years of experience by now >.>...

[Aurelie Thiele]

Hi Ilya - Actually, IBM has a history of hiring our best grads. Maybe you should give them another try!