In order to balance this traffic better, more and more edge and fog server farms are emerging – compact server configurations between the central clouds and the end devices. In the telecom sector, they are called cloudlets or industrial clouds (see fig 1). Such decentralised data centres, not only operated by carriers, but also by industrial providers, are a first step to enable better network load distribution.
This decentralisation continues to be essential; but operators cannot expand these centres endlessly in order to meet rising performance requirements. On one hand, there is often not enough space for further expansion – particularly true for cloudlet server installations on mobile phone towers. On the other, it makes no sense to replace existing equipment after a few years since – at least in the carrier segment – the earnings per gigabit of bandwidth are decreasing. As a consequence, it takes longer to get a return on investment.
Operators of such high-availability server farms are therefore looking for ways to optimise their investments in new rack performance. The solution needs to be standardised as it is the only way to guarantee that it will last for decades. Since 19in rack servers are most common, it is advisable to stick with this design in order to maintain the infrastructure. But replacing individual rack servers requires a reinvestment of approximately 80%. It would be much better to reverse this ratio, so the incurred upgrade costs are only 20%. How can this be achieved?
On the technology side, it is generally processors that enable new performance levels. But it hasn’t been an easy job to replace a processor. Intel and AMD have, so far, failed to develop a socket technology for servers that reliably lasts longer than one tick-tock generation. The board, and often the entire workings of a server, have to be exchanged. The housing, invariably the smallest cost item, is then disposed of.
However, there is now a vendor-independent standard at the module level that enables upgrades of existing equipment over several socket generations. In many cases, it is now possible to use two to three processor and performance generations without having to replace rack servers or entire systems. While some manufacturers may already offer this using servers with processor modules, these are proprietary and tie the customer to the manufacturer.
Figure 1: COM Express Type 7 provides a modular approach for upgrading cloudlets and industrial fog and edge servers
A vendor-independent standard for server on modules (SoM) – COM Express Type 7 – was developed by PICMG under the editorship of congatec. Officially adopted in April 2017, this standard is designed specifically for carrier-grade rack servers and robust industrial servers. It differs from computer-on-modules in that the feature set is entirely designed for server technology. Essentially, this refers to the connection of the servers in the rack using up to 4 x 10GbE links and the connection of storage via fast native PCIe. Dedicated interfaces are executed exactly where they are needed on the carrier board and comprehensive design guides, as well as free circuit diagrams, make the developer’s job easier. Scalability translates into a high degree of investment security, which results in cost advantages and a maximum payback of NRE costs over decades. An extensive community ensures competitive pricing and provides an ecosystem for off-the-shelf accessories. In addition, PICMG ensures continued vendor-neutral development of the standard.
In this respect, computer-on-modules are distinctly different from the modular solutions offered by the server manufacturers, which are generally not sufficiently standardised to allow swapping modules between different manufacturers. COM Express Type 7 is therefore perfect for the standardisation initiatives of major global server operators, such as Facebook and Google. However, this standardisation is not just an advantage when upgrading large installations; smaller batch sizes also benefit from this standard.
Technically speaking, the first processors installed on the COM Express Type 7 are designed for high parallelism with an optimal performance per Watt ratio and – at least from a server point of view – are most suitable for low-power applications and micro-servers. Thanks to the lower power envelope, they are also suited to applications in harsher environments at the edge of carrier networks.
The modules are available with a choice of six Xeon processors, ranging from the 16 core D-1577 to the quad core D-1529, all specified for use in temperatures ranging from -40 to 85°C. As an industry first, the modules also offer server features, including two 10GbE interfaces, 32 PCIe lanes and 48Gbyte of DDR4 ECC RAM.
Future modules could take the COM Express Extended format – 155 x 110mm – which would allow them to host more memory in order to integrate more powerful processor technologies, allowing them to be used in the backbones of larger server farms.
Martin Frederiksen
If these developments are not enough, PICMG could specify larger modules because there is no upper limit.
Another vital feature for servers is remote management, which must also be enabled for switched off systems. Out of band management allows server connection and control, even when the processor and operating system are not running. For this purpose, the COM Express Type 7 pinout integrates dedicated pins for the Network Controller Sideband Interface (NC‑SI). This defines both the protocol and the electrical interface for connecting a baseboard management controller (BMC) on the carrier board. As a result, module-based server designs offer exactly the same remote management functions available in ‘classic’ servers. Recognising this, congatec has integrated a BMC on its latest evaluation board, available either as a telco variant, with 4x10GbE interfaces, or as an industrial version for the extended temperature range.
Current congatec SoMs also support virtualisation technologies that carriers need for software defined networks and network function virtualisation. For example, it has developed application-ready SoM configurations that implement hypervisors from Real Time Systems. The implementation of further virtualisation technologies is possible at any time on request.
Computers at the edge of carrier networks and servers in industrial data centers must cater for the ever increasing workloads of higher and higher resolution streaming services and data intensive IoT applications. COM Express Type 7 modules can provide the required leaps in performance in a cost effective manner.
Author profile
Martin Frederiksen is sales director, northern Europe, with congatec.