Technology

Big Data: A Revolution That Will Transform How We Live, Work, and Think

Viktor Mayer-Schönberger, Kenneth Niel Cukier

Which paint color is most likely to tell you that a used car is in good shape? How can officials identify the most dangerous New York City manholes before they explode? How did Google searches predict the spread of the H1N1 flu outbreak?


Big Data: A Revolution That Will Transform How We Live, Work, and Think

Publisher: Houghton Mifflin Harcourt
Price: $27.00
Author: Viktor Mayer-Schönberger, Kenneth Niel Cukier
Length: 242 pages
Format: Hardcover
Publication date: 2013-03
Affiliate
Amazon
Excerpted from Big Data: A Revolution That Will Transform How We Live, Work and Thing by Viktor Mayer-Schönberger and Kenneth Niel Cukier. Published by Houghton Mifflin Harcourt Copyright © 2013. All rights reserved. No part of this excerpt may be reproduced or printed without permission in writing from the publisher.

NOW

In 2009 a new flu virus was discovered. Combining elements of the viruses that cause bird flu and swine flu, this new strain, dubbed H1N1, spread quickly. Within weeks, public health agencies around the world feared a terrible pandemic was under way. Some commentators warned of an outbreak on the scale of the 1918 Spanish flu that had infected half a billion people and killed tens of millions. Worse, no vaccine against the new virus was readily available. The only hope public health authorities had was to slow its spread. But to do that, they needed to know where it already was.

In the United States, the Centers for Disease Control and Prevention (CDC) requested that doctors inform them of new flu cases. Yet the picture of the pandemic that emerged was always a week or two out of date. People might feel sick for days but wait before consulting a doctor. Relaying the information back to the central organizations took time, and the CDC only tabulated the numbers once a week. With a rapidly spreading disease, a two-week lag is an eternity. This delay completely blinded public health agencies at the most crucial moments.

As it happened, a few weeks before the H1N1 virus made headlines, engineers at the Internet giant Google published a remarkable paper in the scientific journal Nature. It created a splash among health officials and computer scientists but was otherwise overlooked. The authors explained how Google could “predict” the spread of the winter flu in the United States, not just nationally, but down to specific regions and even states. The company could achieve this by looking at what people were searching for on the Internet. Since Google receives more than three billion search queries every day and saves them all, it had plenty of data to work with.

Google took the 50 million most common search terms that Americans type and compared the list with CDC data on the spread of seasonal flu between 2003 and 2008. The idea was to identify people infected by the flu virus by what they searched for on the Internet. Others had tried to do this with Internet search terms, but no one else had as much data, processing power, and statistical know-how as Google.

While the Googlers guessed that the searches might be aimed at getting flu information—typing phrases like “medicine for cough and fever”—that wasn’t the point: they didn’t know, and they designed a system that didn’t care. All their system did was look for correlations between the frequency of certain search queries and the spread of the flu over time and space. In total, they processed a staggering 450 million different mathematical models in order to test the search terms, comparing its predictions against actual flu cases from the CDC in 2007 and 2008. And they struck gold: their software found a combination of 45 search terms that, when used together in a mathematical model, had a strong correlation between their prediction and the official figures nationwide. Like the CDC, they could tell where the flu had spread, but unlike the CDC they could tell it in near real-time, not a week or two after the fact.

Thus when the H1N1 crisis struck in 2009, Google’s system proved to be a more useful and timely indicator than government statistics with their natural reporting lags. Public health officials were armed with valuable information.

Strikingly, Google’s method does not involve distributing mouth swabs or contacting physicians’ offices. Instead, it is built on “big data”—the ability of society to harness information in novel ways to produce useful insights or goods and services of significant value. With it, by the time the next pandemic comes around, the world will have a better tool at its disposal to predict and thus prevent its spread.

Public health is only one area where big data is making a big difference. Entire business sectors are being reshaped by big data as well. Buying airplane tickets is a good example.

In 2003 Oren Etzioni needed to fly from Seattle to Los Angeles for his younger brother’s wedding. Months before the big day, he went online and bought a plane ticket, believing that the earlier you book, the less you pay. On the flight, curiosity got the better of him and he asked the fellow in the next seat how much his ticket had cost and when he had bought it. The man turned out to have paid considerably less than Etzioni, even though he had purchased the ticket much more recently. Infuriated, Etzioni asked another passenger and then another. Most had paid less.

For most of us, the sense of economic betrayal would have dissipated by the time we closed our tray tables and put our seats in the full, upright, and locked position. But Etzioni is one of America’s foremost computer scientists. He sees the world as a series of big-data problems—ones that he can solve. And he has been mastering them since he graduated from Harvard in 1986 as its first undergrad to major in computer science.

From his perch at the University of Washington, he started a slew of big-data companies before the term “big data” became known. He helped build one of the Web’s first search engines, MetaCrawler, which was launched in 1994 and snapped up by InfoSpace, then a major online property. He co-founded Netbot, the first major comparison-shopping website, which he sold to Excite. His startup for extracting meaning from text documents, called ClearForest, was later acquired by Reuters.

Back on terra firma, Etzioni was determined to figure out a way for people to know if a ticket price they see online is a good deal or not. An airplane seat is a commodity: each one is basically indistinguishable from others on the same flight. Yet the prices vary wildly, being based on a myriad of factors that are mostly known only by the airlines themselves.

Etzioni concluded that he didn’t need to decrypt the rhyme or reason for the price differences. Instead, he simply had to predict whether the price being shown was likely to increase or decrease in the future. That is possible, if not easy, to do. All it requires is analyzing all the ticket sales for a given route and examining the prices paid relative to the number of days before the departure.

If the average price of a ticket tended to decrease, it would make sense to wait and buy the ticket later. If the average price usually increased, the system would recommend buying the ticket right away at the price shown. In other words, what was needed was a souped-up version of the informal survey Etzioni conducted at 30,000 feet. To be sure, it was yet another massive computer science problem. But again, it was one he could solve. So he set to work.

Using a sample of 12,000 price observations that was obtained by “scraping” information from a travel website over a 41-day period, Etzioni created a predictive model that handed its simulated passengers a tidy savings. The model had no understanding of why, only what. That is, it didn’t know any of the variables that go into airline pricing decisions, such as number of seats that remained unsold, seasonality, or whether some sort of magical Saturday-night-stay might reduce the fare. It based its prediction on what it did know: probabilities gleaned from the data about other flights. “To buy or not to buy, that is the question,” Etzioni mused. Fittingly, he named the research project Hamlet.

The little project evolved into a venture capital-backed startup called Farecast. By predicting whether the price of an airline ticket was likely to go up or down, and by how much, Farecast empowered consumers to choose when to click the “buy” button. It armed them with information to which they had never had access before. Upholding the virtue of transparency against itself, Farecast even scored the degree of confidence it had in own predictions and presented that information to users too.

To work, the system needed lots of data. To improve its performance, Etzioni got his hands on one of the industry’s flight reservation databases. With that information, the system could make predictions based on every seat on every flight for most routes in American commercial aviation over the course of a year. Farecast was now crunching nearly 200 billion flight-price records to make its predictions. In so doing, it was saving consumers a bundle.

With his sandy brown hair, toothy grin, and cherubic good looks, Etzioni hardly seemed like the sort of person who would deny the airline industry millions of dollars of potential revenue. In fact, he set his sights on doing even more than that. By 2008 he was planning to apply the method to other goods like hotel rooms, concert tickets, and used cars: anything with little product differentiation, a high degree of price variation, and tons of data. But before he could hatch his plans, Microsoft came knocking on his door, snapped up Farecast for around $110 million, and integrated it into the Bing search engine. By 2012 the system was making the correct call 75 percent of the time and saving travelers, on average, $50 per ticket.

Farecast is the epitome of a big-data company and an example of where the world is headed. Etzioni couldn’t have built the company five or ten years earlier. “It would have been impossible,” he says. The amount of computing power and storage he needed was too expensive. But although changes in technology have been a critical factor making it possible, something more important changed too, something subtle. There was a shift in mindset about how data could be used.

Data was no longer regarded as static or stale, whose usefulness was finished once the purpose for which it was collected was achieved, such as after the plane landed (or in Google’s case, once a search query had been processed). Rather, data became a raw material of business, a vital economic input, used to create a new form of economic value. In fact, with the right mindset, data can be cleverly reused to become a fountain of innovation and new services. The data can reveal secrets to those with the humility, the willingness, and the tools to listen.

Viktor Mayer-Schönberger is Professor of Internet Governance and Regulation at the Oxford Internet Institute, Oxford University. A widely recognized authority on big data, he is the author of over a hundred articles and eight books, of which the most recent is Delete: The Virtue of Forgetting in the Digital Age. He is on the advisory boards of corporations and organizations around the world, including Microsoft and the World Economic Forum.








Kenneth Cukier is the Data Editor of the Economist and a prominent commentator on developments in big data. His writings on business and economics have appeared in Foreign Affairs, the New York Times, the Financial Times, and elsewhere.
Music

The Best Metal of 2017

Painting by Mariusz Lewandowski. Cover of Bell Witch's Mirror Reaper.

There's common ground between all 20 metal albums despite musical differences: the ability to provide a cathartic release for the creator and the consumer alike, right when we need it most.

With global anxiety at unprecedented high levels it is important to try and maintain some personal equilibrium. Thankfully, metal, like a spiritual belief, can prove grounding. To outsiders, metal has always been known for its escapism and fantastical elements; but as most fans will tell you, metal is equally attuned to the concerns of the world and the internal struggles we face and has never shied away from holding a mirror up to man's inhumanity.

Keep reading... Show less

In Americana music the present is female. Two-thirds of our year-end list is comprised of albums by women. Here, then, are the women (and a few men) who represented the best in Americana in 2017.

If a single moment best illustrates the current divide between Americana music and mainstream country music, it was Sturgill Simpson busking in the street outside the CMA Awards in Nashville. While Simpson played his guitar and sang in a sort of renegade-outsider protest, Garth Brooks was onstage lip-syncindg his way to Entertainer of the Year. Americana music is, of course, a sprawling range of roots genres that incorporates traditional aspects of country, blues, soul, bluegrass, etc., but often represents an amalgamation or reconstitution of those styles. But one common aspect of the music that Simpson appeared to be championing during his bit of street theater is the independence, artistic purity, and authenticity at the heart of Americana music. Clearly, that spirit is alive and well in the hundreds of releases each year that could be filed under Americana's vast umbrella.

Keep reading... Show less

Two recently translated works -- Lydie Salvayre's Cry, Mother Spain and Joan Sales' Uncertain Glory -- bring to life the profound complexity of an early struggle against fascism, the Spanish Civil War.

There are several ways to write about the Spanish Civil War, that sorry three-year prelude to World War II which saw a struggling leftist democracy challenged and ultimately defeated by a fascist military coup.

Keep reading... Show less
8

Beware the seemingly merry shades of green and red that spread so slowly and thickly across the holiday season, for something dark and uncertain, something that takes many forms, stirs beneath the joyful facade.

Let's be honest -- not everyone feels merry at this time of year. Psychologists say depression looms large around the holidays and one way to deal with it is cathartically. Thus, we submit that scary movies can be even more salutary at Christmas than at Halloween. So, Merry Christmas. Ho ho ho wa ha ha!

1. The Old Dark House (James Whale, 1932)

Between Frankenstein (1931) and The Invisible Man (1933), director James Whale made this over-the-top lark of a dark and stormy night with stranded travelers and a crazy family. In a wordless performance, Boris Karloff headlines as the deformed butler who inspired The Addams Family's Lurch. Charles Laughton, Raymond Massey, Gloria Stuart, Melvyn Douglas and Ernest Thesiger are among those so vividly present, and Whale has a ball directing them through a series of funny, stylish scenes. This new Cohen edition provides the extras from Kino's old disc, including commentaries by Stuart and Whale biographer James Curtis. The astounding 4K restoration of sound and image blows previous editions away. There's now zero hiss on the soundtrack, all the better to hear Massey starting things off with the first line of dialogue: "Hell!"

(Available from Sony Pictures Home Entertainment)

2. The Lure (Agnieszka Smoczynska, 2015)

Two mermaid sisters (Marta Mazurek, Michalina Olszanska) can summon legs at will to mingle on shore with the band at a Polish disco, where their siren act is a hit. In this dark reinvention of Hans Christian Andersen's already dark The Little Mermaid, one love-struck sister is tempted to sacrifice her fishy nature for human mortality while her sister indulges moments of bloodlust. Abetted by writer Robert Bolesto and twin sister-musicians Barbara and Zuzanna Wronska, director Agnieszka Smoczynska offers a woman's POV on the fairy tale crossed with her glittery childhood memories of '80s Poland. The result: a bizarre, funy, intuitive genre mash-up with plenty of songs. This Criterion disc offers a making-of and two short films by Smoczynska, also on musical subjects.

(Available from Criterion Collection / Read PopMatters review here.)

3. Personal Shopper (Olivier Assayas, 2016)

In the category of movies that don't explain themselves in favor of leaving some of their mysteries intact, here's Olivier Assayas' follow-up to the luminous Clouds of Sils Maria. Kristen Stewart again plays a celebrity's lackey with a nominally glamorous, actually stupid job, and she's waiting for a sign from her dead twin brother. What about the ghostly presence of a stalker who sends provocative text messages to her phone? The story flows into passages of outright horror complete with ectoplasm, blood, and ooga-booga soundscapes, and finally settles for asking the questions of whether the "other world" is outside or inside us. Assayas has fashioned a slinky, sexy, perplexing ghost story wrapped around a young woman's desire for something more in her life. There's a Cannes press conference and a brief talk from Assayas on his influences and impulses.

(Available from Criterion Collection / Reader PopMatters review here.

4. The Ghoul (Gareth Tunley, 2016)

The hero (Tom Meeten) tells his therapist that in his dreams, some things are very detailed and others are vague. This movie tells you bluntly what it's up to: a Möbius strip narrative that loops back on itself , as attributed to the diabolical therapists for their cosmic purposes. Then we just wait for the hero to come full circle and commit the crime that, as a cop, he's supposedly investigating. But this doesn't tell us whether he's really an undercover cop pretending to be depressed, or really a depressive imagining he's a cop, so some existential mysteries will never be answered. It's that kind of movie, indebted to David Lynch and other purveyors of nightmarish unreality. Arrow's disc offers a making-of, a commentary from writer-director Gareth Tunley and Meeten along with a producer, and a short film from Tunley and Meeten.

(Available from Arrow Video)

​5. The Illustrated Man (Jack Smight, 1969)

When a young man goes skinny-dipping with a mysterious stranger (Rod Steiger) who's covered with tattoos, the pictures comes to life in a series of odd stories, all created by Ray Bradbury and featuring Steiger and Claire Bloom in multiple roles. Nobody was satisfied with this failure, and it remains condemned to not having reached its potential. So why does Warner Archive grace it with a Blu-ray? Because even its failure has workable elements, including Jerry Goldsmith's score and the cold neatness of the one scene people remember: "The Veldt", which combines primal child/parent hostilities (a common Bradbury theme) with early virtual reality. It answers the question of why the kids spend so much time in their room, and why they're hostile at being pulled away.

(Available from Warner Bros.)

6. The Hidden (Jack Sholder, 1987)


In one of my favorite action movies of the '80s, a post-Blue Velvet and pre-Twin Peaks Kyle MacLachlan plays an FBI agent who forms a buddy-cop bond with Michael Nouri while pursuing a perp -- a bodiless entity that plugs into the human id. In the midst of slam-bang action comes a pivotal moment when a startling question is asked: "How do you like being human?" The heart of the movie, rich in subtext, finds two men learning to embrace what's alien to them. In pop-culture evolution, this movie falls between Hal Clement's novel Needle and the TV series Alien Nation. On this Warner Archive Blu-ray, Sholder offers a commentary with colleague Tim Hunter.

(Available from Warner Bros.)

7. Twin Peaks: Fire Walk With Me (David Lynch, 1992)

Speaking of Twin Peaks, here we have a textbook example of a movie that pleased almost nobody upon its release but has now generated such interest, thanks in large part to this year's Twin Peaks revival, that it arrives on Criterion. A feature-film prequel to David Lynch and Mark Frost's original TV serial that answered none of its questions and tossed in a raft of new ones, the film functions as one of cinema's most downbeat, disruptive and harsh depictions of a middle-class American teenage girl's social context. Sheryl Lee delivers a virtuoso performance that deserved the Oscar there was no way she'd be nominated for, and she wasn't. The extras, including a 90-minute film of deleted and alternate takes assembled by Lynch, have been available on previous sets.

(Available from Criterion Collection)

8. The Green Slime (Kinji Fukasaku, 1968)

Incredibly, Warner Archive upgrades its on-demand DVD of a groovy, brightly colored creature feature with this Blu-ray. As a clever reviewer indicated in this PopMatters review, what director Kinji Fukasaku saw as a Vietnam allegory functions more obviously as a manifestation of sexual tension between alpha-jock spacemen competing for the attention of a foxy female scientist, and this subconsciously creates an explosion of big green tentacled critters who overrun the space station. While we don't believe in "so bad it's good," this falls squarely into the category of things so unfacetiously absurd, they come out cool. There's a sublimely idiotic theme song.

(Available from Warner Bros.)

If the idea is that earth, water, fire, air and space constitute the core elements of life, then these five songs might seem as their equivalents to surviving the complications that come from embracing the good and enduring the ugly of the Christmas season.

Memory will never serve us well when it comes to Christmas and all its surrounding complications. Perhaps worse than the financial and familial pressures, the weather and the mad rush to consume and meet expectations, to exceed what happened the year before, are the floods of lists and pithy observations about Christmas music. We know our favorite carols and guilty pleasures ("O Come All Ye Faithful", "Silent Night"), the Vince Guaraldi Trio's music for 1965's A Charlie Brown Christmas that was transcendent then and (for some, anyway) has lost none of its power through the years, and we embrace the rock songs (The Kink's "Father Christmas", Greg Lake's "I Believe In Father Christmas", and The Pretenders' "2000 Miles".) We dismiss the creepy sexual predator nature in any rendition of "Baby, It's Cold Outside", the inanity of Alvin and the Chipmunks, and pop confections like "I Saw Mommy Kissing Santa Claus".

Keep reading... Show less
Pop Ten
Mixed Media
PM Picks

© 1999-2017 Popmatters.com. All rights reserved.
Popmatters is wholly independently owned and operated.

rating-image