Engineered

Analytics in Advocacy

The Spring 2013 issue of the Stanford Social Innovation Review has the first article on the use of analytics that I have seen in the magazine so far – specifically, on the use of analytics to guide and inform advocacy efforts.

The authors provide a systematic method that their employer, the Redstone Strategy Group, helped design to “plan, monitor and evaluate advocacy investments”. Such a method can help overcome the subconscious decision biases identified by Princeton’s Professor Emeritus Daniel Kahneman and winner of the 2002 Nobel Prize in Economics in his seminal work.

The analytical assessment framework is built around nine conditions that are viewed as “essential to a successful policy campaign.”

1. Functioning venue(s) for adoption (i.e., functioning regulatory institutions and the like)
2. Open policy window (demand for the solution)
3. Feasible solution (a solution has been “shown to provide the intended benefits”)
4. Dynamic master plan (“pragmatic and flexible advocacy strategy”)
5. Strong campaign leader(s)
6. Influential support coalition
7. Mobilized public
8. Powerful inside champions
9. Clear implementation path.

These nine conditions can be used as (i) a checklist, to help “grantmakers and campaign leaders consider the full range of influential factors in creating a successful campaign”, (ii) a rubric, which “allows grantmakers and advocates to take a deeper look at the conditions and score them from 1 to 5 on a spectrum from “not at all” present to “fully” present”, or (iii) a quantitative estimator for likelihood of success, which “helps funders and campaign leaders judge returns on financial investments”. Method (iii) is the most quantitative approach, because costs of planned activities and performance measures have to be estimated with and without the contribution of the grantmaker to assess the grant’s potential impact.

The authors point out that “[a]ny formula that provides quantitative estimates of uncertain values, such as the likelihood of achieving policy change, is a decision-making aid, not a scientific truth”, a caveat that is useful to remember when faced with the temptation to blindly follow the conclusions of a mathematical model having required a number of hard-to-check assumptions. They further divide policy change into three stages and estimate likelihoods of success for each: agenda-setting (conditions 1 to 3 above), adoption (conditions 4 to 8) and implementation (condition 9). Because successful policy will need to go through the three stages, the cumulative likelihood of success is the product of the likelihoods in each individual stage.

Conditions 3 to 8 are labeled “campaign conditions”, i.e., conditions that the campaign can influence in the near term. Conditions 1, 2 and 9 are called “context conditions” and “reflect forces that advocates have little ability to influence via the campaign, especially in the near term.” The campaign’s contribution is estimated by comparing the estimated likelihood of success with and without intervention. The philanthropic return on investment for a campaign is then the potential social benefit of a successful policy multiplied by the percentage-point increase in the likelihood of success, divided by the cost of the advocacy campaign.

The framework is valuable at each of the four stages of an advocacy campaign: (a) evaluating pathways (deciding on a strategy), (b) incorporating contribution (deciding on a tactical approach), (c) monitoring progress and (d) assessing results. For each of these four stages, the authors explain how their nine-condition Advocacy Assessment Framework can be used as a checklist, as a rubric and as a quantitative estimator.

The Hewlett Foundation in particular implemented the framework with great benefits when it had to identify five states to give support to for “broad-based advocacies of clean electricity policies.” The Western Energy Project is also using the framework to guide its advocacy efforts to restrict oil shale leasing. Finally, the framework can help analyze completed efforts such as those to protect a remote Wyoming range from further oil and gas exploration, which culminated in the 2009 Wyoming Range Legacy Act. The authors conclude: “The early applications described in this article suggest that it can help grantmakers determine whether they are choosing the right topics and investing in the right grantees to reach their goals.”

Decision-making under uncertainty using robust optimization

Decision-making in most applications involves dealing with uncertainty, from random stock returns in finance to random demand in inventory management to random arrival and service times in a service center. This leads to the following two questions:

1. how should this randomness be modelled?
2. how should system performance be evaluated, and ultimately optimized? 

When assessing possible answers, it is fair to ask what makes a decision-making methodology more appealing than another. Intuitively, an “ideal” approach exhibits strengths in both dimensions outlined above, in the sense that (1) it allows for a modelling of randomness that does not require many additional assumptions, especially in settings where the validity of such assumptions is hard to check in practice, and (2) it allows for efficient optimization procedures for a wide range of problem instances, which in these days of “big data” most likely includes instances of very large scale. In other words, an “ideal” approach should build directly upon the information at hand – specifically, historical observations of the randomness – without forcing this information into rigid constructs that are difficult to validate even with the benefit of hindsight, and while preserving computational tractability to remain of practical relevance for the decision maker.

Interestingly, the method of choice to model uncertainty has long scored poorly on not just one but both criteria above. Indeed, probability theory assigns likelihoods to events that, when defined as a function of several sources of uncertainty, may occur in many different ways (think for instance of the sum of ten random variables taking a specific value, and all the possible combinations of those ten random variables that achieve the desired value as their sum). To assign a probability to this complex event, it is then necessary to resort to an advanced mathematical technique called convolution, made even more complex when random variables are continuous – and this is just to compute a probability, not optimize it. Additional hurdles include the fact that the manager observes realizations of the random variables but never the probability distributions themselves, and that such distributions are difficult to estimate accurately in many practical applications.

So why have probabilities become the standard tool to model uncertainty? A possible answer is that they were developed to describe random events in the realm of pure mathematics and have since been used beyond their intended purpose to guide business managers in their optimization attempts. This might have happened because academicians and practitioners alike lacked other tools to describe uncertainty in a satisfactory manner, or because the systems under early consideration remained simple enough to fit easily within the probabilistic framework. But as the size of the problems considered by today’s managers increases and today’s business environments are defined more and more by fast-changing market conditions, it seems particular urgent to bring forward a viable alternative to probabilistic decision-making that exhibits the attributes outlined above. This is precisely what researchers in a field called “robust optimization” have done. (Disclosure: robust optimization is also one of my key research areas.)

By now robust optimization has been a hot topic in operations research for almost two decades, and I won’t go over its distinguished contributions to fields as varied as portfolio management, logistics and shortest path problems, among many others, in this post – for that purpose, the interested reader is referred to one of the several overview papers available on the Internet.

For today’s post I will focus on a specific aspect of robust optimization that is currently gaining much traction, specifically, the use of robust optimization as a “one-stop” decision-making methodology that (a) builds directly upon the data at hand, (b) remains tractable, intuitive and insightful in a wide range of settings where probabilistic models become intractable and (c) leads to the axioms of probability theory as a consequence of the modelling framework.

While (a) and (b) have been standard arguments in favor of robust optimization for at least a decade, the connection with the world of probabilities provided in (c) is a novel development that should appeal to many decision makers. The resulting framework is at the core of a paper by Chaithanya Bandi and Dimitris Bertsimas – my former dissertation adviser – published last summer in Mathematical Programming.

They suggest modelling sources of uncertainty not as random variables with probability distributions but as parameters belonging to uncertainty sets that are consistent with the limit laws of probabilities (which bound the deviation of a sum of independently and identically distributed random variables from their mean, using their standard deviation and the square root of the number of random variables). Other asymptotic laws can be incorporated to the uncertainty set in specific circumstances.

The objective of the problem then becomes to optimize the worst-case value of some criterion, instead of its expected value as would have been the case in stochastic optimization.

Bandi and Bertsimas discuss:

• Using historical data and the central limit theorem,
• Modeling correlation information,
• How to use stable laws to construct uncertainty sets for heavy-tailed distributions that have infinite variance,
• Incorporating distributional information using “typical sets”, first introduced in the context of information theory, which exhibit the properties that (i) the probability of the typical set is nearly 1 and (ii) all elements of the typical set are nearly equiprobable, and are defined in the Bandi and Bertsimas paper in terms of uncertainty sets with examples drawn from common distributions.

The rest of the paper shows the results obtained by applying this framework to three applications.

Performance analysis of queueing networks. (Some of the work described in this section is joint work between Bandi, Bertsimas and Nataly Youssef - download their paper here.) The authors introduce the concept of a robust queue where arrival and service processes are modelled by uncertainty sets instead of assigning probability distributions and derive formulas on the worst-case waiting time for the n-th customer in the queue. They connect their results with those obtained in traditional queueing theory and derive results for systems with heavy-tailed behavior such as Internet traffic that are not believed to have been previously available.

Further, they analyze single-class queueing networks using their framework and present computational results where they compare their approach, dubbed the Robust Queueing Network Analyzer (RQNA), with results obtained using simulation and the Queueing Network Analyzer (QNA) developed in traditional queueing theory. Their observation is that RQNA’s results are often significantly closer to simulated values than QNA’s.

Optimal mechanism design for multi-item auctions. (Full paper.) In this problem, an auctioneer is interested in selling multiple items to multiple buyers with private valuations for the items. The auctioneer’s goal in the robust optimization approach is to maximize the worst-case revenue, with his beliefs on buyers’ valuations modelled through uncertainty sets. His decisions are the allocation of items and the payment rules, which should satisfy the following properties:

• Individual Rationality: Bidders do not derive negative utility by participating in the auction, assuming truthful bidding, i.e., bidding their true valuation of an item.
• Budget Feasibility: Each buyer is charged within his budget constraints.
• Incentive Compatibility: the total utility of the i-th buyer under truthful bidding is greater than the total utility that Buyer i derives by bidding any other bid vector.

Bandi and Bertsimas provide a robust optimization mechanism (ROM) that solves the problem and consists of (1) an algorithm to compute the worst-case revenue before a bid vector is realized, and (2) an algorithm to compute allocations and payments afterward, which also uses the worst-case revenue as input.

In addition, they investigate the special case where the buyers do not have any budget constraints, for which they compare the resulting algorithms in their model with a mechanism called Myerson auction. They argue that their method in that setting exhibits stronger robustness properties when the distribution or the standard deviation of the valuations is misspecified.

Pricing multi-dimensional options. In this finance problem, Bandi and Bertsimas (along with their collaborator Si Chen) propose to model the underlying price dynamics with uncertainty sets and then apply robust optimization rather than dynamic programming to solve the pricing problem. They illustrate their approach in the context of European call options and reformulate the problem as a linear problem.

The approach has the flexibility to incorporate transaction costs and liquidity effects. It also captures a phenomenon known in finance as the implied volatility smile, which can be explained in the context of robust optimization using risk aversion arguments. Bandi and Bertsimas further give examples of the accuracy of their method relative to observed market prices. You can download that paper here.

The authors’ central argument is that “modelling stochastic phenomena with probability theory is a choice” and that “given the computational difficulties in high dimensions, we feel we should consider alternative, computationally tractable approaches in high dimensions.” They provide compelling evidence that robust optimization is well-suited for that purpose.

Do you know your cost of capital?

The July/August issue of Harvard Business Review had an outstanding article ("Do you know your cost of capital?") on how the assumptions that leaders make can affect their cost of capital. In a nutshell, the article explains that "companies differ widely in the assumptions built into the financial models they use to evaluate investment opportunities... Not a single question about such assumptions received the same answer from a majority [>50%] of respondents."

Here are some highlights:

• What's your forecast horizon? 46% answered 5 years, 34% 10 years, 6% 15 years and 14% other.
• What's your cost of debt? 37% answered current rate on outstanding debt, 34% forecasted rate on new issuance, 29% average historical rate.
• What's the risk-free rate? 46% answered 10 years, 16% 90 days, 12% 5 years, 11% 30 years, 6% other, 5% 1 year and 4% 20 years.
• What's the equity-market risk premium? 49% answered 5%-6%, 23% 3%-4%, 17% 7% or greater and 11% less than 3%.
• What's your beta period? 41% answered 5 years, 29% 1 year, 15% 3 years, 13% 2 years and 2% other.
• What's your debt-to-equity ratio? 30% answered current book debt to equity, 28% targeted book debt to equity, 23% current market debt to equity and 19% current book debt to current market equity.

The article provides insights into what the correct answers should be, and hopefully corrects some misconceptions. Readers can visit this HBR-sponsored website (hosted by forio.com, which builds custom simulations and sells simulation development software to businesses, universities and government agencies) to test out their own data and calculate their cost of capital.

INFORMS Future Practitioner Colloquium

I got back from the INFORMS Annual Meeting in Phoenix, AZ yesterday night and figured I'd write a quick blog post about something my doctoral student E.C. mentioned, in addition to my regular blog post for the week which was published on Tuesday. Here is what prompted the extra post: my student participated in the Future Practitioner Colloquium, a one-day event before the annual meeting geared toward senior doctoral students that focuses on "career paths for OR/MS professionals, challenges that new practitioners experience, discussions concerning choosing between an academic and a practice career and more", and she said that, although the Future Academician Colloquium was very well attended, there were only twelve students in the Future Practitioner Colloquium, including nine from the University of Florida and herself. Only twelve.

Now, I don't know how many doctoral students in operations research or analytics have completed their basic course of study (one of the few requirements to be nominated, see here) and are interested in industry careers, but I'm very sure there are more than three in the country outside the University of Florida. With all the talk about the current analytics wave - or tsunami? - and interest from top companies in "data scientists", as well as the number of companies recruiting at the annual meeting, it seems certain that a whole more students would have benefited from, and should have been interested in, attending the colloquium.

E.C. was actually happy at the small number because it gave her more opportunities to interact with the students who did attend, but from the perspective from someone who enjoys preparing tomorrow's practitioners, I wish more students had been able to benefit from the colloquium for several reasons. One, it covers important topics that we professors can't really teach to students who want to join the real world. And of course it takes more than a one-day colloquium to teach those topics, but at least the colloquium is a start. This is particularly relevant because the whole concept of doctorate involves spending several years making one's own contribution to the field in a specific area, and at the end there is only one name on the dissertation. In contrast, industry involves significantly reduced time scales and teamwork, among other things, which also has implications in terms of students' ability to interact with others, contribute to a large project, deal with difficult team members, and so on.

Intuitively, students who pursue doctoral studies are people very comfortable in spending a lot of time by themselves with their computer, equipment or textbooks, even if they have many friends outside the lab. We teach our undergraduate students to work in teams for some courses or capstone projects, and graduate students in some courses also do team projects, but as doctoral students gain seniority and focus on research, the whole emphasis shifts to that of individual performance over a time horizon of years. Hopefully students attracted to industry will already have the skills required to succeed in the real world - but maybe they'd benefit from identifying early on areas that are particularly important for them to work on in order to be successful, and from hearing about the experience of other O.R. professionals who made the choice before them.

Some students get that exposure through industry summer internships, but not every doctoral student is able to get one, perhaps because his/her adviser prefers that he/she focuses on working on his/her dissertation, and perhaps because industry summer internships for PhD students are not nearly as common as undergraduate internships - besides, finding one that meshes well with a student's expertise is not always easy. A one-day colloquium won't be the solution to everything, but not attending guarantees students won't get any exposure at all to those topics.

Two, given the rising importance of analytics, data science and O.R. in industry, it seems particularly relevant to create a network of young, up-and-coming practitioners who can stay in touch after graduation - those are the people who will call upon each other when senior analytics opportunities present themselves later on. Again, a one-day colloquium won't forge everlasting bonds, but if the students don't attend, then they most definitely won't know about each other. INFORMS tries hard to remain relevant for practitioners, and hopefully there will be a time in the not-too-distant future where the practice conference in the spring (rebranded "analytics conference") will attract as many attendees as the more academic one in the fall, but for now membership does seem to be heavily tilted toward faculty members, and helping future practitioners see a clear benefit in INFORMS membership could contribute to INFORMS's continued relevance in that respect.

Maybe an issue is that professors don't bother telling their department chair that they want to nominate their students for the practitioners colloquium - maybe they are not sure of the value of attending, and while the first student per department goes for free (i.e., the colloquium fee is waived), the department does have to shoulder some of the costs for the subsequent ones. So whoever organizes the colloquium might consider distributing the agenda from the previous year and testimonials from students when he/she calls for nominations. Maybe the timing, in the middle of the summer, is bad for the students or professors who are away; maybe it'd be easier to move the nomination date forward to match abstract submission. But if we want to ensure the continued relevance of analytics in industry, we academics should offer those among our students who are inclined to pursue industry careers more opportunities to meet like-minded students and learn about the specific challenges related to industry work. The University of Florida, whose ISE department is headed by a former Lehigh faculty member, certainly seems to have understood that. The rest of the O.R. departments in the country might want to take notice.

HBR's "Selling into Micromarkets"

Today's post is about an article that appeared in the July/August issue of Harvard Business Review: "Selling into Micromarkets: How to use big data to uncover lucrative new hot spots." It starts with an example at an unnamed chemical and service company, which "diced its seven US regions into 70 micromarkets and zeroed in on those with the greatest potential," leading to a twofold increase in the sales growth rate "without an increase in marketing or sales costs."

The article continues: "The key to the firm's remarkable turnaround was its new ability to combine, sift and sort vast troves of data to develop a highly efficient sales strategy." Apparently, "B2B sales organizations have only recently begun to use big data to both inform overall strategy and tailor sales pitches for specific customers in real time" and the thesis of the article is that "micromarket strategy is perhaps the most potent new application of big-data analytics in B2B sales."

The strategy it describes involves three steps: (1) find new pockets of growth, (2) make it easy for the sales team (align sales coverage with opportunity, create sales plays for each type of opportunity, support the sales force in executing the plays) and (3) put data at the heart of sales (through performance management, cross-functional collaboration and talent development).

Other articles I enjoyed in that HBR issue include:

• "Expanding the entrepreneur class"
• "Dupont's CEO on executing a complex cross-border acquisition"
• "The view for the field: six leaders offer their perspectives on sales success"
• "Disrupt yourself"

and if you must only read one article from HBR, make it "Do you know your cost of capital" (more about that in my post next week).

Analytics and Linchpins

I recently finished reading Linchpin by Seth Godin, which I found to be a far better book than I expected in spite of its many disjoint thoughts. One thing that struck me was Godin’s attitude that “the easier it is to quantify, the less it’s worth” (p.96 of paperback edition). When I interact with business practitioners outside the crowd of analytics faithful I encounter at conferences, it sometimes seems that a few of them really don’t want to use mathematical models to, say, set their inventory. They cling instead that they have some magical, intuitive knowledge of future product demand that can’t be inputted in a computer. Of course, it makes at first their job more secure by giving the illusion that they are indispensable, until someone at their company or one of their competitors decides to implement analytical techniques.

The question this raises is: how can we convince such practitioners that analytics aren’t a threat (representing the big bad world that is continually replacing humans by computers) but instead an opportunity (of said humans to focus on higher-value tasks)? Or, what retraining do these people have to receive to be able to take advantage of the opportunity?    

Back from Huntington Beach, CA!

Yesterday evening I got back from the INFORMS 2012 Analytics conference, which was held in Huntington Beach, CA. I wanted to post this earlier, but things got hectic at Lehigh with my meeting with the project sponsor for the analytical finance projects, the Spencer Schantz luncheon and lectures, which honored Lehigh ISE alumnus and Stanford emeritus professor Arthur (Pete) Veinott, and the annual ISE banquet, which honored both Veinott as 2012 Distinguished Alum for Excellence in Academia and Lehigh ISE alumnus Tim Wilmott as 2012 Distinguished Alum for Excellence in Industry.

(I also have two Master theses to edit before tomorrow evening/Saturday morning, at which point they have to be in my department chair's inbox for reading and approval before next Friday's deadline in the Registrar's Office... the last problem of the quiz in my financial optimization course to grade... the project and the last homework to upload... an overdue review... several papers that are past their revision deadlines... and those are only the things that need to happen before Monday. Oh, the end of the semester.)  

Anyway, I know I owe this blog a "GoodSemester, Part 2" blog post, but that will have to wait for next week. For today, I'll write about the Analytics conference. I've been to this conference once before, in 2009 when it was called the "Practice Conference" and was held in Phoenix. I wouldn't have attended this year - it is a really good conference but it is really expensive too and the timing in the semester wasn't great - if the ISE department at Lehigh hadn't been named finalist for the UPS George D Smith Prize on the strengths of the application I wrote.

Prof Terlaky, Prof Gustafson and Rob Rappa '11 '12G came along to the conference and presented the department case with me to the judging committee. Our application was for the whole ISE department and included our bachelor's, master's and doctoral programs. The other finalists were Cornell IEOR's M.Eng program and the certificate program of the Tauber School for Global Operations at the University of Michigan. Michigan won. Many thanks to the judges for the time they spent evaluating the submissions - for this award and others, as some did double duty on the Edelman or INFORMS Prize Committees. Their commitment to INFORMS is truly inspiring.

The conference was a resounding success from beginning to end, even if I had to stop by Lehigh on my way to the airport to make copies of my quiz solutions for my teaching assistant, so that she could proctor the exam and grade her part while I was gone (just-in-time operations at their finest). I landed in Orange County late on Saturday, after a "flight" from ABE to EWR that is actually a trip down Route 78 on a tiny bus [the type of the bus you take at the airport to go to the economy parking lot - it is apparently more reliable than operating a flight into EWR for such a short distance, and it leaves outside the airport at ABE so you and your carry-on luggage only go through security at EWR, although your checked-in luggage is screened at ABE] and a flight from EWR to SNA, aka John Wayne Airport.

Because I had made my hotel reservation late to the conference hotel - the Huntington Beach Hyatt Resort and Spa - due to learning late that Lehigh was a finalist for the award, the conference block had been sold out for the first and last nights of my reservation and I was supposed to change rooms 3 times in 4 nights. I showed up Saturday night and started explaining to the young employee at the check-in counter how it would be a great plus in my life if by any chance he could find a way to make me stay in the same room for all four nights... The teenage kid raised an eyebrow and said: "oh, we fixed your reservation. Of course we're not going to have you change rooms in the middle of your stay!" That's customer service to you.

My enthusiasm for the Hyatt hotel chain further skyrocketed when I saw the enormous room I'd been put in. As some of you know, my family used to be very poor so even now that we've pulled ourselves out of that, I can get excited like a little kid for expensive things at bargain prices. The room rates weren't cheap to begin with, but I could tell immediately that the rate of that room was much higher than what I was paying for. I glanced outside and saw palm trees outside my balcony, a courtyard and a fountain (it was dark so I couldn't see much). I thought the view was nice and forgot about it until the next morning, when I opened the curtains and realized I actually had a view of the beach and the Pacific Ocean, and I don't mean far in the distance. From that point on, that conference could be nothing but a success in my eyes! The staff at the Hyatt was extremely courteous and professional; the food was delicious. The hotel has received extremely high ratings on Google Reviews and I can say with full confidence that it deserves every single one of them.

But back to the conference, since that's why I had come all the way to California for. It provided us (the Lehigh team) with great opportunities to connect with companies, and I also enjoyed the excellent talks I attended,

• from Intel's entry as a finalist to the Edelman competition,
• to the winning presentation in the Innovation in Analytics competition (US Immigration and Customs Enforcement with Booz Allen Hamilton - a fascinating topic),
• to "Smarter Analytics in Insurance" presented by the head of the IBM group on advanced analytics and optimization (our Lehigh student Rob Rappa was thrilled to talk to him afterwards, since that group would be a perfect match for his skills),
• to the work Lehigh's recent graduate Ana Alexandrescu '10 '11G did for her Master's thesis on "Comparison of Risk Prediction Models for Home Health", which was presented by her supervisor at Bayada Home Health Care since she couldn't attend due to her new job.

I also enjoyed the keynote by eBay's Vice President for Analytics Platform, the informative lunches and the great networking opportunities. Lehigh's recent graduate Nick Kastango '09 was part of the Memorial Sloan Kettering team that got recognized with the INFORMS Prize and I'm looking forward to hearing his presentation next year. I'm also looking forward to hearing TNT Express present at the annual meeting in the fall the work they received the prestigious Edelman award for. Since they were recently bought by UPS (another company that values analytics), it is nice for them to show they bring significant value to their new parent company.

The conference was extremely well-organized and ran very smoothly thanks to the tireless efforts of our dedicated INFORMS staff Paulette Bronis, Sandi Owens and many others. Thanks to you all for another great event. See you in Phoenix in the fall!

Business and Social Media

For today’s post, here are a few articles on social media.

The first one, “Decoding Social Media” in the November/December 2011 issue of Technology Review, explains how discovering patterns in tweets will affect TV, ads and even politics. An example of “social-media analytics firm that attempts to track comments on shows and ads”, including comparing amount of tweets with Nielsen ratings, is Bluefin Labs. While traditional surveys only represent respondents’ opinion at specific times, social-media data can be analyzed continuously, which represents a fascinating opportunity for start-ups.

This has piqued the interest not only of TV companies such as CBS, but also of conglomerates such as Procter & Gamble, which oversees brands such as Tide, Duracell and Pringles. The journalist comments: “While Procter & Gamble carefully vets ads with consumers before airing them, it has never known whether the same viewers would respond differently to an ad depending on what show surrounded it.” Bluefin Labs hopes to elucidate precisely that sort of issue. This would allow the placement of TV ads to be tailored to viewers’ responses, similarly to what is happening to online ads today. Other companies tracking social-media conversations include:

• Radian6, (from their website: "[Radian6's] social media listening, tracking, monitoring and engagement tools allow organizations to successfully employ a social media strategy and understand the impact the Social Graph and Social CRM have on their success." They also say that half of all Fortune 100 companies use their software.)
• General Sentiment, (they seem good too, but I found Radian6's website more compelling.) 
• Sysomos, (also offers a demo video on its website, has more details than General Sentiment on what it is offering customers, states on its website that customers include Microsoft, Coca-Cola, IBM and Reebok.)
• Converseon, ("an award-winning, full-service social media agency that helps brands harness the power of social to meet business objectives." Was founded in 2001, long before Facebook, Twitter or even Myspace existed.)
• Trendrr, (Catchy slogan: "Unlock the true potential of the real-time web", also the only site with a "We're hiring" banner prominently displayed on top, but a little light on the discussion of its products' features, which can be said of many other sites above. Hence, it is hard to determine what distinguishes these companies from each other.)

The TR article really focuses on Bluefin, which is headquartered in the Kendall Square section of Cambridge, MA near MIT. The company was co-founded by MIT professor Deb Roy, currently on leave from the famed Media Lab, and his former doctoral student Michael Fleischman. (Among many things, the article provides a great background story on how analyzing TV-related social media came about for the company.) Today, Bluefin’s clients include Pepsi, Mars, Comcast, CBS, Fox Sports, A+E Networks and about ten others. Its business model hinges on selling subscriptions to its custom analytics, generated by uploading raw TV data obtained from their data center to Amazon’s cloud computing service. (This also seems to be the business model of the companies listed above.)

Bluefin generates two key metrics based on all the social-media responses: response level and response share, which refer to the number of tweets or public Facebook posts on a given TV show and the share of these tweets compared to the total volume of TV-related tweets and public FB posts. Much remains to be done to extract as much information as possible from this social-media goldmine, maybe generating one day the social-media equivalent to the Nielsen ratings.

Another interesting article is “What’s your social media strategy?” in the July-August 2011 issue of Harvard Business Review. It reveals that companies’ social media strategies can be divided into four groups: the “predictive practitioner” (connecting with customers to ask them what feature they’d like to see included in a new product, for instance), the “creative experimenter” (listening to customers on Facebook and Twitter in a less scripted way than the predictive practitioner), the “social media champion” (involves joint efforts of various functions at a company, for instance asking customers with large social-media following to discuss their experiences with a new car) and the “social media transformer” (allowing large-scale interactions extending to external stakeholders).

The HBR special report on “Social Media and the New Rules of Branding” in the December 2010 issue was particularly fascinating. It consisted of four articles:

1. “Branding in the digital age”, which builds upon the fact that “customers today connect with brands in fundamentally new ways” and the “funnel metaphor” of buying (first researching many brands, then fewer and fewer until only one is left) has been replaced by a “consumer decision journey.” As a result, marketers now have to “target stages [rather than allocating across media] in the decision journey”, taking into consideration when customers are best influenced. The article has convincing examples and also discusses new roles for marketing.
2. “Reputation warfare”, about small-scale antagonists targeting the business of a larger company. Advice includes “empower your team to help tell your organization’s side of the story” and “find sympathetic third parties to serve as “force multipliers”.
3. “Why you need a new-media ringmaster”: here, “ringmaster” refers to “a new type of executive who has digital savvy and is skilled at coordinating a variety of marketing and customer-facing activities.”
4. “The one thing you must get right when building a brand”, which is “deliver[ing] on four basics: offering and communicating a clear customer promise; building trust by delivering on it; continually improving the promise; and innovating beyond the familiar.”

But my favorite article of the lot remains the Technology Review one. Extracting insights from the enormous, shapeless mass of tweets and Facebook posts is one more reason the field of analytics has a bright future ahead.

This will be my last post for 2011. Look for a new post on January 9, 2012. Happy New Year!    

On Impact Factors

The International Federation of Operational Research Societies, or IFORS, recently reposted an article that first appeared in The Chronicle by Richard Monastersky. When I looked up the original article, I realized it was now 6 years old, but many of the issues remain very timely. The article describes the use and over-use of impact factors, and the techniques that some less scrupulous people employ to game the system.

First, my non-academic readers might ask: what are impact factors? They are the brainchild of a Philadelphia-based researcher named Eugene Garfield, who came up in the late 1950s with "a grading system for journals, that could help him pick out the most important publications from the ranks of lesser titles." The system relies on "tallying up the number of citations an average article in each journal received." Fair enough. But the tool now plays an important role in tenure and hiring decisions, because universities want a simple metric to evaluate the quality of the research, and papers accepted by journals with high impact factors seem likely to be of high quality. This, predictably, has led to some abuse.

The calculation of impact factors, by the company ISI (owned by Thomson Corporation), goes as follows: "To calculate the most recent factor for the journal Nature, for example, the company tallied the number of citations in 2004 to all of the articles that Nature published in 2002 and 2003. Those citations were divided by the number of articles the journal published in those two years, yielding an impact factor of 32.182 — the ninth-highest of all journals."

But trying to draw conclusions about average quality from an average number is bound to generate misconceptions and errors. "For example, a quarter of the articles in Nature last year drew 89 percent of the citations to that journal, so a vast majority of the articles received far fewer than the average of 32 citations reflected in the most recent impact factor." If a paper is accepted by a top-ranked journal, does its impact on the scientific field reflect the importance of the journal, or is it on the contrary one of the 50% accepted by the journal whose quality will be below the median?

The following caught my attention: "In China, scientists get cash bonuses for publishing in high-impact journals, and graduate students in physics at some universities must place at least two articles in journals with a combined impact factor of 4 to get their Ph.D.’s." Is putting so much pressure on scientists really a good idea? Maybe such high stakes explain some egregious cases of plagiarism over the past few years, such as the one Prof Alice Agogino of UC Berkeley was recently a victim of. You can find her paper (from 2004) here, and the other paper (from 2007) there.

But what should impact factors really blamed for? I do find the claim that they are "threatening to skew the course of scientific research" and that "investigators are now more likely to chase after fashionable topics" because they want to get published in journals with high-impact factors frankly outlandish. Investigators chase after fashionable topics because they need grants to support their research group and get promoted. Journals in my field (operations research) publish contributions in a variety of fields, from theory to logistics to transportation, and no one needs to know impact factors to be aware that the hot topics right now are in health care and energy, but the fact that your research focuses on a hot topic doesn't mean you'll make the sort of meaningful contribution that will warrant publication in a top journal.

Another dynamic that was left unanalyzed in the article is that between journals and senior researchers. The article spends a lot of time and space discussing the idea that editors might reject a paper they feel won't be cited enough and thus would be detrimental to the journal's impact factor, but if the paper is co-authored by a top researcher and falls within that researcher's core area of expertise, it is hard to imagine that the paper would not be cited many, many times.

So a more relevant question would be: among the authors published by those top journals, how many are authored by junior researchers without senior researchers co-authoring? how many by junior researchers with other junior researchers? how many by junior researchers with students? If many highly-ranked journals publish mostly the work of senior researchers, then it is not reasonable to ask junior researchers - who are supposed to demonstrate their own research potential without the guidance of senior staff - to publish in those journals too in order to benefit from the golden aura of the high impact factor. (Some junior researchers certainly will be able to, but that shouldn't be a widespread standard.) This reminds me of those statistics about average salaries per profession, where journalists forget to mention that the averages are computed for employees ranging from 25 to 64 in age, thus giving the impression that any college graduate can hope to make $80k a year within a few short years out of college. Maybe the high impact factors are due to the predominance of articles by well-established scientists with decades of experience in the profession.

This being said, a lot of the article rang true. For instance: "Journal editors have learned how to manipulate the system, sometimes through legitimate editorial choices and other times through deceptive practices that artificially inflate their own rankings. Several ecology journals, for example, routinely ask authors to add citations to previous articles from that same journal, a policy that pushes up its impact factor."

I had heard of the practice before, although I've never experienced it. I have, though, been in a situation where the anonymous reviewer requested that a citation be added to my (and my student's) paper, and when I looked up the citation there didn't seem to be anything in common with the area of our paper. On the other hand, if the reviewer was one of the authors of the other paper, as seems likely, then having his paper cited increases his number of citations, which is a metric universities take into consideration at promotion time. I could have complained and have my paper left in limbo for several more weeks or months. I decided I didn't have the time to wage that fight and simply made the citation as inconspicuous as possible.

Self-citations, where previous papers published in a journal are cited in that same journal, are certainly a potential issue. The article mentions the case of a researcher who "sent a manuscript to the Journal of Applied Ecology and received this e-mail response from an editor: “I should like you to look at some recent issues of the Journal of Applied Ecology and add citations to any relevant papers you might find. This helps our authors by drawing attention to their work, and also adds internal integrity to the Journal’s themes.”"

The editor who sent the email later said that "he never intended the request to be read as a requirement," which is disingenuous, given that the paper had not been accepted yet and researchers can't submit their paper to other journals until it has been withdrawn or rejected, thus putting them at the editors' mercy.

Recently ISI has developed an alternative methodology to address the flaws of the impact factor metric, called eigenfactor, in which citations from high-impact journals are weighted more heavily. To evaluate a scientist's value to his field, people should use the H-index rather than drawing inferences from journals' metrics, although the H-index also suffers from drawbacks. And, as it becomes more used, it is sure to be more misused too.