The truth about Jeff Dean appeared on April Fool’s Day 2007.

Somewhere inside Google, a private website served up a list of facts about Dean, one of Google’s earliest employees and one of the main reasons the web giant handles more traffic than any other operation on the net. The site was only available to Googlers, but all were encouraged to add their own Jeff Dean facts. And many did.

“Jeff Dean once failed a Turing test when he correctly identified the 203rd Fibonacci number in less than a second,” read one.

“Jeff Dean compiles and runs his code before submitting,” read another, “but only to check for compiler and CPU bugs.”

“The speed of light in a vacuum used to be about 35 mph,” said a third. “Then Jeff Dean spent a weekend optimizing physics.”

No, these facts weren’t really facts. But they rang true. April Fool’s Day is a sacred occasion at Google, and like any good April Fool’s joke, the gag was grounded in reality. A Google engineer named Kenton Varda set up the website, playing off the satirical Chuck Norris facts that so often bounce around the net, and when he mailed the link to the rest of the company, he was careful to hide his identity. But he soon received a note from Jeff Dean, who had tracked him down after uncovering the digital footprints hidden in Google’s server logs.

Inside Google, Jeff Dean is regarded with awe. Outside the company, few even know his name. But they should. Dean is part of a small group of Google engineers who designed the fundamental software and hardware that underpinned the company’s rise to the web’s most dominant force, and these creations are now mimicked by the rest of the net’s biggest names — not to mention countless others looking to bring the Google way to businesses beyond the web.

Time and again, we hear the story of Xerox PARC, the Silicon Valley research lab that developed just about every major technology behind the PC revolution, from the graphical user interface and the laser printer to Ethernet networking and object-oriented programming. But because Google is so concerned with keeping its latest data center work hidden from competitors — and because engineers like Jeff Dean aren’t exactly self-promoters — the general public is largely unaware of Google’s impact on the very foundations of modern computing. Google is the Xerox PARC of the cloud computing age.

“Google did a great job of slurping up some of the most talented researchers in the world at a time when places like Bell Labs and Xerox PARC were dying,” says Mike Miller, an affiliate professor of particle physics at the University of Washington and the chief scientist of Cloudant, one of the many companies working to expand on the technologies pioneered by Google. “It managed to grab not just their researchers, but also their lifeblood.”

These Google technologies aren’t things you can hold in your hand — or even fit on your desk. They don’t run on a phone or a PC. They run across a worldwide network of data centers.

They include sweeping software platforms with names like the Google File System, MapReduce, and BigTable, creations that power massive online applications by splitting the work into tiny pieces and spreading them across thousands of machines, much like micro-tasks are parceled out across a massive ant colony. But they also include new-age computer servers, networking hardware, and data centers that Google designed to work in tandem with this software. The idea is to build warehouse-sized computing facilities that can think like a single machine. Just as an ant colony acts as one entity, so does a Google data center.

While Silicon Valley stood transfixed by social networks and touch screens, Google remade the stuff behind the scenes, and soon, as the other giants of the web ran into their own avalanche of online data, they followed Google’s lead. After reinventing Google’s search engine, GFS and MapReduce inspired Hadoop, a massive number-crunching platform that’s now one of the world’s most successful open source projects. BigTable helped launch the NoSQL movement, spawning an army of web-sized databases. And in so many ways, Google’s new approach to data center hardware sparked similar efforts from Facebook, Amazon, Microsoft, and others.

To be sure, Google’s ascendance builds on decades of contributions from dozens of equally unheralded computer scientists from many companies and research institutions, including PARC and Bell Labs. And like Google, Amazon was also a major influence on the foundations of the net — most notably through a research paper it published on a file system called Dynamo. But Google’s influence is far broader.

The difference between it and a Xerox PARC is that Google profited mightily from its creations before the rest of the world caught on. Tools like GFS and MapReduce put the company ahead of the competition, and now, it has largely discarded these tools, moving to a new breed of software and hardware. Once again, the rest of the world is struggling to catch up.

Google’s Twin Deities

Kenton Varda could have targeted several other Google engineers with his April Fool’s Day prank. Jeff Dean just seemed like “the most amusing choice,” Varda remembers. “His demeanor was perhaps the furthest from what you’d expect in a deity.”

The obvious alternative was Sanjay Ghemawat, Dean’s longtime collaborator. In 2004, Google published a research paper on MapReduce, the number-crunching platform that’s probably the company’s most influential data center creation, and the paper lists two authors: Dean and Ghemawat. The two engineers also played a major role in the design of the BigTable database. And Ghemawat is one of three names on the paper describing the first Google File System, a way of storing data across the company’s vast network of data centers.

Even for Varda, who works on the team that oversees Google’s infrastructure, the two engineers are difficult to separate. “Jeff and Sanjay worked together to develop much of Google’s infrastructure and have always seemed basically joined at the hip,” says Varda. “It’s often hard to distinguish which of them really did what.

“All code changes at Google require peer review prior to submission, but in Jeff and Sanjay’s case, often one will send a large code review to the other, and the other will immediately ‘LGTM’ it, because they wrote the change together in the first place.” LGTM is Google-speak for “looks good to me.”

Varda means this quite literally. Over the years, Dean and Ghemawat made a habit of coding together while sitting at the same machine. Typically, Ghemawat does the typing. “He’s pickier about his spacing,” Dean says.

The two met before coming to Google. In the ’90s, both worked at Silicon Valley research labs run by the Digital Equipment Corporation, a computing giant of the pre-internet age. Dean was at DEC’s Western Research Lab in Palo Alto, California, and Ghemawat worked two blocks away, at a sister lab called the Systems Research Center. They would often collaborate on projects, not only because Dean had a thing for the gelato shop that sat between the two labs, but because they worked well together. At DEC, they helped build a new compiler for the Java programming language and a system profiler that remade the way we track the behavior of computer servers.

They came to Google as part of a mass migration from DEC’s research arm. In the late-’90s, as Google was just getting off the ground, DEC was on its last legs. It made big, beefy computer servers using microprocessors based on the RISC architecture, and the world was rapidly moving to low-cost machines equipped with Intel’s x86 chips. In 1998, DEC was acquired by computer giant Compaq. Four years later, Compaq merged with HP. And the top engineers from DEC’s vaunted research operation gradually moved elsewhere.

“DEC labs were going through a bit of rocky period after the Compaq acquisition,” Dean says, “and it wasn’t exactly clear what role research would have in the merged company.” Some engineers went to Microsoft, which was starting a new research operation in Silicon Valley. Some went to a Palo Alto startup called VMware, whose virtual servers were about to turn the data center upside-down. And many went to Google, founded the same year DEC was acquired by Compaq.

It was a time when several of the tech world’s most influential research labs were losing steam, including Xerox PARC and Bell Labs, the place that produced such important technologies as the UNIX operating system and the C programming language. But although these labs had already seen their best days, many of their researchers would feed a new revolution.

“At the time of the bubble burst in 2001, when everyone was downsizing, including DEC, the main two high-tech companies that were hiring were Google and VMware,” says Eric Brewer, the University of California at Berkeley computer science professor who now works alongside Dean and Ghemawat. “Because of the crazy lopsidedness of that supply and demand, both companies hired many truly great people and both have done well in part because of that factor.”

Like Dean and Ghemawat, several other engineers who arrived at Google from DEC would help design technologies that caused a seismic shift in the web as a whole, including Mike Burrows, Shun-Tak Leung, and Luiz André Barroso. At the time, these engineers were just looking for interesting work — and Google was just looking for smart people to help run its search engine. But in hindsight, the mass migration from DEC provides the ideal metaphor for the changes Google landed on the rest of the world.

DEC was one of the first companies to build a successful web search engine — AltaVista, which came out of the Western Research Lab — and at least in the beginning, the entire thing ran on a single DEC machine. But Google eclipsed AltaVista in large part because it turned this model on its head. Rather than using big, beefy machines to run its search engine, it broke its software into pieces and spread them across an army of small, cheap machines. This is the fundamental idea behind GFS, MapReduce, and BigTable — and so many other Google technologies that would overturn the status quo.

In hindsight, it was a natural progression. “The architecture challenges that arise when building a data system like Google’s that spans thousands of computers isn’t all that different from the challenges that arise in building a sophisticated monolithic system,” says Armando Fox, a professor of computer science at the University of California, Berkeley who specializes in large-scale computing. “They problems wear very similar clothing, and that’s why it was essential to have people with experience at places like DEC.”

Jeff Dean Follows His Uncle to Google

Jeff Dean was the first to arrive from DEC. He came by way of his “academic uncle,” Urs Hölzle.

Hölzle was one of Google’s first 10 employees, and as the company’s first vice president of engineering, he oversaw the creation of the Google infrastructure, which now spans more than 35 data centers across the globe, judging from outside sources. He joined Google from a professorship at the University of California at Santa Barbara, and before that, he studied at Stanford under a prof named David Ungar, developing some of the core technologies used in today’s compilers for the Java programming language.

Dean’s academic adviser also studied with Ungar, and this made Hölzle his academic uncle. In 1999, with DEC in its death throes, Dean left the company for a startup called MySimon, but when he saw Hölzle turn up at Google, he sent an email looking for a new Google job of his own. He was soon hired by the same man who hired Hölzle: Google co-founder Larry Page.

At first, Dean was charged with building an ad system for Google’s fledgling search engine. But after a few months, he moved onto the company’s core search technologies, which were already buckling under the weight of a rapidly growing worldwide web. He was soon joined by Ghemawat, who made the move to Google in large part because Dean and other DEC researchers — Krishna Bharat and Monika Henzinger — were already on board.

“It’s fairly likely that I might never have interviewed at Google if Jeff hadn’t been there,” Ghemawat says. They quickly picked up where they left off at DEC. Over the next three or four years, together with an ever changing group of other engineers, the two engineers designed and built multiple revisions of the company’s core systems for crawling the web, indexing it, and serving search results to users across the globe.

Yes, they would often code at the same machine — while drinking an awful lot of coffee. Cappuccino is their drug of choice. Their partnership works, Dean says, because Ghemawat is more level-headed. “I tend to be very impatient, thinking about all the ways we can do something, my mind and hands spinning at a very fast rate. Sanjay gets excited, but in a more subdued way. He corrects my course, so that we end up moving in the right direction.”

But Ghemawat says Dean’s approach is just as important. He keeps them moving forward. “I often get down, thinking about all the different ways of doing something, worrying about the right way,” Ghemawat says. “It’s good to have someone with the energy and excitement needed to get to the end goal.”

The big breakthroughs came with the creation of the Google File System and MapReduce, which rolled out across Google’s data centers in the early part of the last decade. These platforms provided a more reliable means of building the massive index that drives Google’s search engine. As Google crawled the world’s webpages, grabbing info about each, it could spread this data over tens of thousands of servers using GFS, and then, using MapReduce, it could use the processing power inside all those servers to crunch the data into a single, searchable index.

The trick is that these platforms didn’t break when machines failed or the network slowed. When you’re dealing with ten of thousands of ordinary servers as Google was, machines fail all the time. With GFS and MapReduce, the company could duplicate data on multiple machines. If one broke, another was there to step in.

“The scale of the indexing work made it complicated to deal with machine failures and delays, so we started looking for abstractions that would allow for automatic parallelization across a collection of machines — to give higher performance and scalability — and could also make long-running computations that ran on thousands of machines robust and reliable,” Jeff Dean says, in describing the thinking behind MapReduce. Once these tools were in place on the search engine, he explains, Google realized they could help run other web services too.

BigTable arose in similar fashion. Like MapReduce, it ran atop the Google File System, but it didn’t process data. It operated as a massive database. “It manages rows of data,” Dean says, “and spreads them across more and more machines as you need it.” It didn’t give you as much control over the data as a traditional relational database, but it could handle vast amounts of information in ways you couldn’t with platforms designed for a single machine.

The same story appears again and again. As it grew, Google faced an unprecedented amount of data, and it was forced to build new software.

“What do you do when your job is to take the entire internet, index it, and make a copy of it — and not do it in a way that the copy is the same size as the internet? That’s a pretty interesting technical challenge,” says Jason Hoffman, the chief technology officer at cloud computing outfit Joyent. “Very often the hammer swinger knows how to make the hammer. Most things that are innovative come from a forge. They come from those points where you’re facing failure.”

The Data Center Empire Built on Crème Brûlée

Luiz André Barroso followed Jeff Dean and Sanjay Ghemawat from DEC to Google. But he almost didn’t.

Barroso had worked alongside Dean at DEC’s Western Research Lab, and in 2001, he was weighing job offers from Google and VMware. After visiting and interviewing with both companies, he put together a spreadsheet listing the reasons to join each. But the spreadsheet ended in a dead heat: 122 reasons for Google, and 122 for VMware.

Then he talked to Dean, who asked whether the spreadsheet included the crème brûlée served by executive chief Charlie Ayers the day he visited Google. “Crème brûlée is his absolute favorite,” Dean remembers. “I asked if he had factored it into his 122-point list, and he said: ‘No! I forgot!'” Barroso accepted Google’s job offer the next morning.

Barroso was unusual in that he wasn’t necessarily a software engineer. At DEC, he helped pioneer multicore processors — processors that are actually many processors in one. But after Barroso briefly worked on Google software, Hölzle put him in charge of an effort to overhaul Google’s hardware infrastructure, including not only its servers and other computing gear, but the data centers housing all that hardware. “I was the closest thing we had to a hardware person,” Barroso remembers.

Hölzle, Barroso, and their “platforms team” began by rethinking the company’s servers. In 2003, rather purchase standard machines from the likes of Dell and HP, the team started cutting costs by designing their own servers and then contracting with manufacturers in Asia to build them — the same manufacturers who were building gear for the Dells and the HPs. In short, Google cut out the middle men.

Uniquely, each Google machine included its own 12-volt battery that could pick up the slack if the system lost its primary source of power. This, according to Google, was significantly more efficient that equipping the data center with the massive UPSes — uninterruptible power supplies — that typically provide backup power inside the world’s computing facilities.

Then the team went to work on the data centers housing these servers. At the time, Google merely leased data center space from other companies. But Barroso and crew started from scratch, designing and building their own data centers in an effort to save money and power, but also to improve the performance of Google’s web services.

The company began with a new facility in The Dalles, Oregon, i.e. the rural area where it could tap into some cheap power — and some serious tax breaks. But the main goal was to build an entire data center that behaved like a single machine. Barroso and Hölzle call it “warehouse-scale computing.”

“Large portions of the hardware and software resources in these facilities must work in concert to efficiently deliver good levels of internet service performance, something that can only be achieved by a holistic approach to their design and deployment,” Barroso and Hölzle in their seminal 2009 book on the subject, The Datacenter as a Computer. “In other words, we must treat the data center itself as one massive warehouse-scale computer.”

They designed the facility using a new kind of building block. They packed servers, networking gear, and other hardware into standard shipping containers — the same kind used to transport goods by boat and train — and these data center “modules” could be pieced together into a much larger facility. The goal was to maximize the efficiency of each module. Apparently, the notion came to Larry Page in 2003, when he saw the Internet Achieve give a presentation on its plans for similar modules — though Barroso doesn’t remember where the idea came from. “Other than it wasn’t me,” he says.

The company’s facility in The Dalles went live in 2005. Over the years, there were rumors of data center modules and custom servers, but the details remained hidden until 2009, when Google held a mini-conference at its Silicon Valley headquarters. In the data center, Google isn’t content to merely innovate. It keeps the innovations extremely quiet until it’s good and ready to share them with the rest of the world.

The Tesla Effect

Larry Page has a thing for Nikola Tesla. According to Steven Levy’s behind the scenes look at Google — In The Plex — Page regarded Tesla as an inventor on par with Edison, but always lamented his inability to turn his inventions into profits and long-term recognition.

Clearly, the cautionary tale of Nicola Tesla influenced the way Google handles its core technologies. It treats them as trade secrets, and much like Apple, the company has a knack for keeping them secret. But in some cases, after a technology runs inside Google for several years, the company will open the kimono. “We try to be as open as possible — without giving up our competitive advantage,” says Hölzle. “We will communicate the idea, but not the implementation.”

In 2003 and 2004, the company published papers on GFS and MapReduce. Google let the papers speak for themselves, and before long, a developer named Doug Cutting used them to build an indexing system for an open source search engine he called Nutch. After Cutting joined Yahoo — Google’s primary search rival at the time — the project morphed into Hadoop.

A way of crunching epic amounts of data across thousands of servers, Hadoop has long been used by the other giants of the web, including Facebook, Twitter, and Microsoft, and now, it’s spreading into other businesses. By 2016, according to research outfit IDC, the project will fuel a $813 million software market.

History repeated itself with BigTable. In 2006, Google published a paper on its sweeping database, and together with an Amazon paper describing a data store called Dynamo, it spawned the NoSQL movement, a widespread effort to build databases that could scale to thousands of machines.

“If you look at every NoSQL solution out there, everyone goes back to the Amazon Dynamo paper or the Google BigTable paper,” says Joyent’s Jason Hoffman. “What would the world be like if no one at Google or Amazon ever wrote an academic paper?”

Google’s hardware operation is a slightly different story. We still know relatively little about the inside of Google’s data centers, but the company’s efforts to design and build its own gear has undoubtedly inspired similar efforts across the web and beyond. Facebook is now designing its own servers, server racks, and storage equipment, with help from manufacturers in Asia. According to outside sources, the likes of Amazon and Microsoft are doing much the same. And with Facebook “open sourcing” its designs under the aegis of the Open Compute Foundation, many others companies are exploring similar hardware.

What’s more, modular data centers are now a mainstay on the web. Microsoft uses them, as do eBay and countless others. Mike Manos, Microsoft’s former data center guru, denies that Google was the inspiration for the move to modular data center, pointing out that similar modules date back to the 1960s, but it was Google that brought the idea to forefront. As Cloudant’s Mike Miller points out, GFS and MapReduce also depend on ideas from the past. But Google has knack for applying these old ideas to very new problems.

Google’s Past Is Prologue

The irony is that Google has already replaced many of these seminal technologies. Over the past few years, it swapped out GFS for a new platform dubbed “Colossus,” and in building its search index, it uses a new system known as Caffeine, which includes piece of MapReduce and operates in a very different way, updating the index in realtime rather than rebuilding the thing from scratch.

Google may still use data center modules in The Dalles, but it seems they no longer play a role in its newer facilities. We don’t know much about what the company’s now uses inside these top secret facilities, but you can bet its a step ahead of what it did in the past.

In recent years, Google published papers on Caffeine and two other sweeping software platforms that underpin its services: Pregel, a “graph” database for mapping relationships between pieces of data, and Dremel, a means of analyzing vast amounts data at super high speeds. Multiple open source projects are already working to mimic Pregel. At least one is cloning Dremel. And Cloudant’s Miller says Caffeine — aka Percolator — is sparking changes across the Hadoop and NoSQL markets.

These are just the some latest creations in use at Google. No doubt, there are many others we don’t know about. But whatever Google is using now, it will soon move on. In May of last year, University of California at Berkeley professor Eric Brewer announced he was joining the team building Google’s “next gen” infrastructure. “The cloud is young,” he said. “Much to do. Many left to reach.”

Brewer — one of the giants of distributed computing research — is yet another sign that Google is the modern successor to Xerox PARC. But the company also takes the PARC ethos a step further.

You can trace Google’s research operation through DEC, all the way back to PARC’s earliest days. The DEC Systems Research Center was founded by Robert Taylor, the same man who launched the computer science laboratory at PARC.

Taylor started the SRC because he felt that by the early 80s, PARC had lost its way. “A lot of people who I worked with at PARC we as disenchanted with PARC as I was,” he says. “So they joined me.” He worked to build the lab in the image of the old PARC Computer Science Lab — even in terms of its physical setup — and in some ways, he succeeded.

But it suffered from the same limitations as so many corporate research operations. It took ages to get the research into the marketplace. This was also true at the DEC Western Research Lab, where Jeff Dean worked. And this is what brought him to Google. “Ultimately, it was this frustration of being one level removed from real users using my work that led me to want to go to a startup,” Dean says.

But Google wasn’t the typical startup. The company evolved in a way that allowed it to combine the challenge of research with the satisfaction of instantly putting the results into play. Google was a research operation — and yet it wasn’t. “The Google infrastructure work wasn’t really seen as research,” Ghemawat says. “It was about how do we solve the problems we’re seeing in production.”

For some, the drawback of working on Google’s core infrastructure is that you can’t tell anyone else what you’re doing. This is one of the reasons an engineer named Amir Michael left Google to build servers at Facebook. But, yes, there are times when engineers are let loose to publish their work or even discuss it in public.

For Google, it’s balancing act. Though some are critical of the particular balance, it’s certainly working for Google. And there’s no denying its methods have pushed the rest of the web forward. PARC never had it so good.

• Amazon
• caffeine
• Cloud Computing
• data
• databases
• Dremel
• Enterprise
• Facebook
• google
• Hardware
• History
• Infrastructure
• maps
• Microsoft
• networks
• platforms
• secret servers
• Servers
• software