Linux is clearly a popular solution. In fact, it has been reported that Linux is used on every supercomputer in the Top500 project. Thanks to its tie-in with Android, it is also one of the most widely deployed general-purpose operating systems. Its open nature means that anyone can take Linux and configure it for use on a particular hardware platform, which has also made it popular for embedded applications.
While the level of adoption varies across industries, the underlying trend is that Linux is present across them all. In data centres the adoption is almost complete and, now, they are focussed on increasing the efficiency of Linux-based systems. In the medical market, the pace of adoption is once again picking up after some false-starts. In automotive and transportation, Linux is well established in the infotainment part of the market but is rapidly spreading to other areas. Of course, the consumer sector is very actively adopting Linux, and increasingly in resource-constrained devices such as the IoT and wearables.
There are also opportunities to use open source during the development of safety-critical systems. These systems need to undergo certification processes for whatever software they implement, and they need to do this each time new software is added. While Linux won’t be the right fit for every application, especially for these types of systems, it very much can be helpful to use it as a starting point to quickly prototype and work out initial ideas.
The openness of Linux often creates forks in the road for developers, which effectively forces them to make an important decision at a very early stage in the project. They can choose to partner with a software company that offers an enterprise Linux distribution, often with their own hardware platform that they can support. Or they can start with a flexible distribution builder, such as the Yocto Project, develop their own hardware platform and manage the entire process themselves. An engineer’s instinct may well be to opt for the latter; it preserves choice, after all. Others may see the benefit in choosing a fully supported design flow, particularly if software development is outside the core strengths of the development team. Both have merits and, as we will discover, both have consequences.
You don’t know what you don’t know
One of those consequences is the cost. Open source software is, of course, available without any upfront cost and this can be a powerfully persuasive reason to use it. But teams will discover that most projects in general, and embedded projects in particular, will require the additional cost of maintenance and support.
It is very difficult, even for experienced teams, to accurately budget for the cost of maintaining and supporting a system based on technology that isn’t developed in-house. That is one of the realities of using any open source technology, whether that is open source software or, as is also common today, open source hardware. More importantly, for inexperienced development teams, the implications of maintaining and supporting open source technology for their own product may not even feature in the technical requirement document, at least not the first time they go through the developer journey.
The reason for this hidden surcharge is simple, an open source project is always in development, an innovation engine that never stops. With many millions of lines of open source code, engineering teams need to decide when and if they are going to follow that development path, or freeze, branch off and maintain their own version of the distribution. In order to take advantage of the (typically security) patches that will be made available, they may need to follow the project.
Choosing a vendor-supported enterprise Linux can mitigate the impact of this, however it comes with some level of lock-in to that vendor: while it may still be possible to customize the underlying operating system, this may invalidate the support that comes with choosing that vendor.
The Yocto Project and You
Remaining portable yet compatible is an important aspect of open source Linux. The Yocto Project is a Linux distribution builder intended for embedded systems that is inherently customizable and portable between the leading instruction set architectures (ISAs) used in embedded devices today.
It is provided with the kernel, essential middleware and a tool chain. The target architecture can be selected at build, so the same distribution can be ported by the development team to any supported ISA; the decision doesn’t need to be made until further down the development process. This allows other aspects of the design to influence the choice of most appropriate processor, rather than being saddled with the device provided as part of a supported package.
Just like Linux in general, the Yocto Project has become extremely popular, forming the basis of a software platform for embedded devices. This would comprise the kernel itself and the minimum amount of middleware necessary to support the application code. This so-called micro OS, or minimum OS, is where the Yocto Project excels today. The ability for Linux to support containers to compartmentalize other middleware is also an important aspect but one that can quickly lead to fragmentation of the underlying OS.
Getting started with the Yocto Project is incredibly accessible. The project is cloned from GitHub by downloading the latest branch. Then, using BitBake, which is the Yocto Project build system, an image can be made. Typically, the first time, this will be an image for an Intel target. This takes perhaps half a day and your experience is positive. This will be very encouraging for developers and perhaps all the positive reinforcement they need in order to settle on using the Yocto Project.
Based on this initial experience tasks can be allocated and a team put together. The estimate is it will require one person on the OS build, two developers on the User Interface, one on connectivity middleware, one more on data analysis algorithms and two on testing and usability.
From this project team it is clear that the majority of the effort is focused on what makes this product different, while the one thing that doesn’t differentiate it – the OS – receives the least resources. For many manufacturers, this may be the most alluring reason to use open source software.
Reality Bytes
This scenario describes perfectly what the Yocto Project is intended to provide, a fast route to product development. What isn’t clear from this initial experience is what the Yocto Project isn’t intended to provide, and that is mitigating your risk of long-term stability. After all, like all open source projects, the Yocto Project is an innovation engine that never stops.
Looking purely at support costs, based on an average worldwide developer’s salary being around $100,000, maintaining the product for five years with the equivalence of one full time developer will be $500,000. The cost of development based on seven people for eight months is another $500,000, making the total project cost $1 million between development and maintenance. The cost of maintenance is equivalent to 50% of the project cost, and this is only a very optimistic estimate that more times than not proves to be wrong. Experience would suggest that when it comes to Linux projects in the embedded and IoT markets, the cost of maintenance sets in at 5x the initial cost of development over a five year period of time. In fact, with tens of millions of lines of code in the Yocto Project, and many developers working on it, it is inevitable that bugs will be introduced and fixed, making it a constantly changing platform. This means the cost of maintenance can rapidly overtake the cost of development; the budget allocated to differentiating the product is consumed by supporting the open source software, which may require some of the engineers working on the UI and connectivity to move over to OS support.
If the kernel has been modified, which is often the case with an embedded system, then the patches issued to fix generic issues may or may not fix the issues with a branched kernel. This compounds the support issue and contributes to the escalating cost of development and support. In addition, if the product needs to be certified for its end market, changing the kernel or any part of the software may invalidate that certification, meaning it becomes even more difficult to keep patching.
Another important point to consider is the legal requirement to supply licenses for any software that is shipped. This would fall on the shoulders of the development team or team leader and would cover any and all middleware. In a well supported project like Linux there is a huge library of middleware solutions to choose from, but not all will be viable for international markets and may be subject to trade embargoes.
The cost of support quickly escalates, and that is even before the product has shipped. Ongoing support will not necessarily be simpler, so the cost of maintenance and support is really an unknown for any manufacturer following the in-house route. Enterprise commercial Linux deployments exist to mitigate this but come with vendor tie-in and lack of flexibility, so is there a better way?
The Yocto Project... with a difference
For embedded applications, choosing a supported enterprise Linux distribution can come with vendor tie-in. If the distribution is oriented towards an Enterprise environment it is probable that any changes made to the distribution – which are normally desirable for embedded projects, in order to strip away the unnecessary middleware – will likely void the support package. Similarly, choosing a distribution that has been ported to a reference design could very well shackle your project to the same processor, which removes choice at a hardware level.
For good reasons, the Yocto Project has become a popular choice for embedded development. But as mentioned earlier, while choosing open source offers many technical benefits, any product development comes with other obligations that go far beyond the platform. For manufacturers targeting the industrial and medical sectors, products will need to be supported for between five and ten years after shipping, as a minimum. That support must cover both hardware (so continuity of supply of the processor is important) and software, which makes being able to patch the kernel equally important.
Wind River is a founding contributor to the Yocto Project, its Wind River Linux Yocto Project-compatible product is technically indistinguishable from the Yocto Project yet comes with full support that can be applied at any point in the design process. Changes to the kernel will not invalidate this support, and there is never a need to upgrade, because keeping a Wind River Linux subscription maintains a version of your specific distribution and there is effectively no vendor tie-in. It is on these points that commercial Linux distributions are really differentiated. In addition, every Wind River Linux build comes with its own Openchain Compliance Envelop, making it much simpler to regulate its distribution.
The developer’s journey when first using open source software often starts with a positive experience. However, the realities of product development do not disappear when choosing an open source platform. As discussed here, challenges can actually increase and, at least for the first-time developer, be somewhat of a shock. Choosing a robust commercial Linux partner like Wind River can make the difference between having a good developer experience or a bad one, between a project that fails and one that thrives.
Author: Davide Ricci, Director, EMEA Open Source Business, Wind River