|
JUDGING from the condemnation, Facebook just “friended” Satan. The company has been pilloried for academic research it published, showing that by adjusting its users’ news feeds so that a greater proportion of positive or negative items appeared, it could nudge people to post slightly sunnier or gloomier items themselves. For sociologists, the study provided evidence of “social contagion”—the spread of ideas and moods through imitation and conformity—without face-to-face contact.
That Facebook runs similar tests all the time for commercial reasons—to get users to linger longer, or to click on ads—was largely lost on the critics. And the most telling bit went unnoticed: the difference in posting behaviour was between 0.04% and 0.1%. It sounds paltry, and is. But to Facebook’s data scientists, it amounted to a smashing victory. “Given the massive scale,” they noted, “even small effects can have large aggregated consequences.”
This may indeed be the unspoken secret of big data. The much-hyped term captures the idea that it has become possible to collect vastly more data than before, and to process it far more cheaply and quickly. The result, typically, is to identify the right combination of tweaks capable of bringing about marginal changes that, when multiplied by a huge number of instances, or allowed to work over a long time, produce a significant effect.
“It is about building a mountain with pebbles,” says Jim Manzi, the boss of Applied Predictive Technologies, which sells data-crunching software. Internet firms like Google and Amazon have built their entire businesses on the back of repeated incremental gains. For example, Google’s search algorithm is designed to learn from users: if slightly more people click on the ninth link of a given set of search results than one higher up, then the algorithm learns from this, and moves the lower item upwards. Amazon’s boss, Jeff Bezos, insists that all decisions in his company are based on statistical analyses. Every pixel on the company’s home page has had to justify its existence through repeated testing of alternative layouts, to identify the combination that maximises sales. Each individual change may make the product only a tiny bit better. But the cumulative effect over time can be big.
This sort of constant experimentation and rapid implementation has long been relatively easy for internet firms. But the expense of data collection and processing had forced offline businesses to stick to measuring just the big stuff, and thus only to value big wins rather than examining the fine grains of their activities for small but useful improvements. Companies might easily have missed out on changes that, spread across a large customer base, would reliably produce a gain in output of just 1% or so. Now they can make such changes, and even if the aggregate improvement is only of this modest order, it can over time help them to advance steadily on their rivals. It is a bit like evolution: just a tiny mutation in physiology that makes a creature marginally quicker in escaping predators can, over many generations, mean that it outlives those which lack it.
Take, for instance, UPS’s 60,000 delivery vans in America: the company crunches the data on the 100 miles or so a day that each one covers, and uses its findings to perfect the software in the navigation devices installed on board the vehicles: it reckons that cutting each route by just one mile would save it $50m in fuel and other costs a year. Big Lots, a large American discount retailer, tried out numerous variations on its in-store financing offer, to identify the most profitable one to roll out. It altered such things as the size of the downpayments and the repayment schedules, and tried a variety of ways of marketing the scheme. Juggling all the permutations of these, it says, produced a combination that increased furniture sales by 9%.
The approach can even be used to tell businesses how better to organise their employees. QuantumBlack, a data-analysis firm, says a client, an engineering multinational, measured how the output of its teams of workers varied as a result of dozens of differences in their composition. It found a small step-change in productivity when the teams had more than seven members, whereas efficiency fell steadily with each additional time-zone in which team members were based. How well the members already knew each other turned out to be especially important, yet the managers doing the scheduling had not thought of this. Most individual improvements were in the range of 0.5% to 1%, says Simon Williams of QuantumBlack, but together they added up to a 22% rise in the teams’ overall productivity.
Sweating the small stuff
Big-data techniques are making businesspeople question the management theories they have long relied on. They have been accustomed to apply the “80-20 rule”, an assumption that of all the things they could change, to make their organisation perform better, just a handful of them (the “20%”) would achieve almost all (“80%”) of the possible improvement. Now it has become far easier to do endless trial-and-error testing of small things in order to squeeze out the remaining 20% of improvement. Companies used to doing things the old way may find it hard to recognise that it is worth investing in all this testing and data analysis—it is cheaper than it was, but still costs money—and to accept that there will henceforth have to be more experiments that “fail” on the road to success.
Stimulated by all the talk from consultants and sellers of data-crunching software about the transformative potential of big data, managers may have been misled into hoping it will give them massive, instant, Holy Grail solutions. But such discoveries are rare; and if they do exist, they have probably been made already. The reality is that big data produces lots of small advances—and that is good enough.