09 August 2011

Facebook's new data centre is home to numerous innovations

It is reasonable to assume that anyone reading this article has dealings, in some way, with a data centre. Whether it's through the use of social networking sites, buying goods and services online or accessing 'cloud computing', the data centre is an integral part of our lives.

At the top end of the scale, data centres are huge operations; large buildings crammed with unbelievable numbers of servers and vast amounts of data storage. And they consume huge amounts of power; not only to drive the servers, but also to remove the heat they generate.

Nobody knows for sure how many data centres there are around the world. But an educated guess by those 'in the know' suggests that data centres consume between 2 and 3% of all electricity generated in the US. It's no surprise that energy efficiency is rising very quickly to the top of the agenda.

Social networking pioneer Facebook is leading efforts to address data centre energy consumption with the launch of the Open Compute project - www.opencompute.org . While the goal of the project is to develop an open forum to drive data centre efficiency, the initiative has grown out of Facebook's desire to be as green as it can.

Until recently, Facebook's website has been hosted in a number of leased facilities. But it has just opened its own data centre in Oregon, which it has designed from a 'clean sheet of paper'.

Open Compute is the result of two year's development work by a team at Facebook's Palo Alto headquarters. Hardware engineering manager on the team is Amir Michael. He said his main focus has been to drive the Open Compute effort. "My job has been to reduce the overall infrastructure cost. We looked at a variety of ideas, but all revolved around the data centre – and servers in particular." In fact, the project had a wide remit: everything from the power grid to gates on the processors was up for examination.

According to Michael, the best optimisations could only be achieved by modifying the way the data centre and the servers work together. "The 'big win' comes when you modify both," he said. "We had two teams working on the project. One was designing a custom data centre, the other was designing custom servers." And he said there were a number of overlaps between the two teams.

Along with innovative server design, Michael believes there have also been 'big wins' in the way the servers are powered and in how they are cooled. "The environment within which the servers operate has a big impact," he asserted.

The two year project split nicely into a year of research and a year of execution. "In the first year," Michael noted, "we began to understand what the design would look like. Getting the whole project done in two years is quick for this industry and we went from breaking ground at the Oregon site to powering up the servers in about a year."

Facebook is currently hosted at four sites: along with its Oregon data centre, leased facilities are used in California, Virginia and North Carolina. A project is now underway to construct another Facebook data centre in North Carolina. An indication of the scale of these operations is the fact that Facebook can host all traffic from its West Coast and Asian users at the Californian facility.

While the design of the servers is innovative, the processors that feature are not. For the moment, Facebook has specified what Michael calls 'same source processors' for the servers. "We started out with devices from Intel and AMD and each has its own strengths and weaknesses." Intel processors are used for front end tasks, while AMD processors are associated with memory operations. "Our operations require a lot of memory," Michael noted, "and the AMD parts have an extra memory channel."

Michael accepted that Intel and AMD were 'safe choices'. "We were familiar with the parts and we can bet on them to have competitive products when we want to evaluate new devices." Having said that, he noted that new products were reviewed constantly.

Along with ARM based devices and those from Tilera (see NE, 13 July 2011), Facebook is also evaluating Atom based devices and processors aimed at desktop applications. "Whatever we can get our hands on to see if it makes sense," Michael observed. "We see how they perform and plug the results into a spreadsheet to provide a definitive view."

The team developed a tray based design for the servers at Oregon. Michael said the design was driven not only by the quest for greater power efficiency, but also by the need for easier maintainability. "Many data centre technicians spend their days swapping things in and out. We wanted to make their job as easy as possible." A time and motion study on the new design showed a factor of ten improvement. "Every hour a server is out of commission costs money," he noted, "and our aim was that less than 1% of servers would be offline at any time."

Not only are the trays easier to remove, they have fewer parts and weigh 3kg less than a 6U server.

Trays are then grouped into what are called triplets, racking systems designed specially for the job. "We looked at 19in racks," Michael noted, "but decided to simplify them while keeping the form factor."
Previously, there were 40 servers per rack. Now, the triplet scheme boosts this to 45. "A typical network card supports up to 48 servers," he added, "so we don't use all the capacity and this adds cost. Although we're now putting 45 servers, with extra capacity should we decide to expand."

In building its own data centres, Facebook has looked at the entire cost of ownership; and power is, without doubt, one of the largest costs. Michael said: "How much work can we get out of a data centre? How much does that cost? And how many dollars does it take to run the data centre?".

One of the first design considerations was the power consumed by the data centre. A top line indication of how efficiently this is used is given by the Power Usage Effectiveness ratio, or PUE, which relates total power supplied to a facility and the power consumed by the servers. The US Environmental Protection Agency has a target figure of 1.5; which implies 33% of the power supplied is consumed by air conditioning and similar services. Michael said: "Our target was 1.15 and, as we got further down the design road, we revised that to 1.05. We have ended up with a figure of 1.07, where some large data centres typically have a figure of 2."

The Oregon facility has a series of buildings, each consuming 'tens of MW'. Plans show the company is looking to expand this with the construction of a power transmission line that could supply 120MW. The computing efficiency is complemented with an energy efficient evaporative cooling system, led lighting and a tracking photovoltaic solar array generating 100kW.

Power supply efficiency has also been targeted. "Most devices are more than 94% efficient," Michael claimed, "and some next generation parts will be more than 95% efficient."

Facebook has also looked to eliminate uninterruptible power systems (UPS) where possible. "There's a number of reasons," Michael pointed out, "including conversion losses and the sheer cost of UPS. If you look at a data centre UPS, it costs around $2/W. Our target is 25c/W." The Oregon facility is also powered by a 480V supply, rather than 208V, to reduce losses.

"Rather than convert ac to dc, store the power in a battery, then convert it back to ac," he said, "we're taking batteries and putting them closer to servers." The triplets include one lead acid battery for every six servers.

Open to all
In what might seem to be an unusual move, Facebook has made all this information available via Open Compute. John Kenevey, program manager with Facebook, explained. "The project resonates with Facebook's basic idea of sharing. But we also felt the technology itself is not a differentiator for us; we compete at the product level. The idea of Open Compute is to open up data centre innovation.

Most innovation over the last couple of decades has come in the consumer space, so one thing we're focused on is developing a community and a contributions model. And we already have contributors lining up to share their IP with us." Alluding to the success of the Linux software distributor, Kenevey claimed that Open Compute wants to be the 'Red Hat' of the data centre industry.

The two year Oregon data centre project appears to have been a success. The facility consumes 38% less energy to do the same work as existing sites, while costing 24% less. According to the company, it's 'vanity free'.

Server boasts 384 Atom processors
Low power server pioneer SeaMicro has launched what it says is the most energy efficient 64bit x86 server. The SM10000-64HD combines SeaMicro's server architecture with 384 Intel Atom N570 dual core processors, boosting compute density by 150% and its compute/W metric by 20%. All processors are housed in a 10 rack unit.

Optimised for 64bit internet data centres, SeaMicro's SM10000-64HD system comprises:
• 384 x86 dual core 1.66GHz Intel Atom processors,
• 1.536Tbyte of DDR3 dram,
• up to 64 SATA solid state or hard disk drives,
• 8 to 64 1Gbit Ethernet uplinks.

Keeping your cool
Increasing energy costs mean there is no way data centres can continue to operate with air cooling models that have been in place for more than a decade, says Jason Preston, director of 2bm. "It is time to reconsider the way data centre cooling is delivered and explore opportunities for reducing the primary source of energy consumption."

Preston says a compelling technology, is fresh air evaporative cooling. The approach combines external air with mechanically generated hot air from the data centre to create the right temperature – typically 21°C. Once the external temperature exceeds 19°C, the system switches to evaporative cooling, passing the fresh air over water saturated pads and mixing it with hot data centre air. The result allows companies to use fresh air, even in ambient temperatures as high as 38°.

"The payback on this technology is also compelling: most organisations will save up to 90% of their cooling costs and get a return on their investment within two years," said Preston. The approach also qualifies for the Enhanced Capital Allowance, which allows the entire investment to be claimed against tax in the first year.

Graham Pitcher

Supporting Information


This material is protected by Findlay Media copyright
See Terms and Conditions.
One-off usage is permitted but bulk copying is not.
For multiple copies contact the sales team.

Do you have any comments about this article?

Add your comments


Your comments/feedback may be edited prior to publishing. Not all entries will be published.
Please view our Terms and Conditions before leaving a comment.

Related Articles

'Power supply in a package’

Intersil has introducted the ISL8216M, an 80V, 4A non isolated DC/DC step down ...

Brick modules updated

The PFE-SA series of AC/DC full brick power modules from TDK Lambda is said to ...

€55m power electronics project

Described as one of the most important European energy efficiency research ...

Power electronics challenges

There isn't a sector of the electronics industry that operates without the use ...

A/D converters get it right

Successive approximation A/D converters, with resolutions of up to 18bit and ...

DC transmission on a comeback

The pressure throughout the energy supply chain to deliver electrical power ...

Power electronics in EVs

This whitepaper presents a review of power electronics systems in electric ...

Power management ics

This whjitepaper presents various application scenarios where high voltage ...

Power saving techniques

Wireless telecommunications has given rise to a host of new portable ...

6W DC/DC converters

XP Power has announced the ITX series of 6W DC/DC converters. Available in an ...

Low voltage power supplies

TDK has added 12 and 15V models to its ZWS300BAF series of single output power ...

1W DC/DC converters

Murata has announced the NCS1 family of single output, 1W DC/DC converters.

PCIM Europe 2014 Conference

20th – 22nd May 2014, Nuremburg, Germany

BEEAs 2013

9th October 2014, 8 Northumberland, London

Engineering Design Show 2014

22nd-23rd October 2014, Jaguar Exhibition Hall, Ricoh Arena, Coventry, UK


In this video product marketing engineer, Rich Nowakowski, discusses a family ...

TI power design tools

Discover and get to market faster with TI Power Design Tools; easy-to-use ...

3E EPM Hands on Digital Power

Introduced by Ericsson in 2008, 3E* digital power products and solutions became ...

Packing a powerful punch

Power is back when it comes to industrial system performance and efficiency. ...

Powering ahead

Too often in the past, electronics industry initiatives have been announced ...

Communication breakdown

Last week's Anti Counterfeiting Forum, organised by UKEA and The Electronics ...

Roger Rogowski, UKEA

Last week's Anti Counterfeiting Forum saw electronics industry leaders convene ...

Patrick Le Fèvre, Ericsson

Patrick Le Fèvre, marketing director, Ericsson Power Modules, talks with Chris ...

Andy Gales, vp, Vicor

Andy Gales, vice president, international sales, Vicor, talks with Chris Shaw