Author: Nobuhiro Iwamatsu, Toshiba
Introduction
CIP (Civil Infrastructure Platform) was launched in 2016 under the Linux Foundation and
has been jointly developing a Linux platform for civil infrastructure such as railways, power grids, and factory control systems that operate for decades.
CIP comprises several working groups (WG), one of which is the Kernel WG.
In this blog post, we will introduce the achievements of the Kernel WG over the past 10 years.
Looking Back at Our Milestones
1. Provision of Super Long-Term Support (SLTS) Kernels
The Kernel WG’s greatest achievement has been the development of SLTS kernels, which are supported for at least 10 years, significantly longer than standard Linux LTS kernels.
Based on the Linux community’s LTS kernels, the WG continuously backports security patches and critical bug fixes to its maintained versions. To date, the WG has provided the following series, all of which aim for at least 10 years of support:
- linux-4.4.y (started in 2017, supported until 2027)
- linux-4.19.y (started in 2019, supported until 2029)
- linux-5.10.y (started in 2021, supported until 2031)
- linux-6.1-cip (started in 2023, supported until 2033)
- linux-6.12-cip (started in 2025, supported until 2035)
Taking into account the Debian Linux kernel version used in the corresponding root filesystem, CIP members democratically decide every two years which upcoming CIP kernel version will be supported during the SLTS period.
Although linux-4.4.y and linux-4.19.y have reached EOL (End of Life) in the Linux LTS project, the WG continues to maintain these versions through a self-maintenance model, limiting them to a subset of architectures and member-provided configurations.
2. Continuous Contributions to LTS and mainline kernel
CIP not only maintains its own kernel but also places a high priority on contributing to the Linux community.
- Reviewing fix patches for LTS
We are reviewing patches that have been backported into our based LTS kernel. We do not review every single patch in the base LTS kernel; instead, we focus on reviewing the latest kernel version we support (currently 6.12.y). This is because many patches merged into older kernels are also merged into newer kernels, which helps us avoid duplicating patch reviews. And we have reviewed tens of thousands of LTS patches.
- Testing support
We run build and boot tests on all LTS, LTS-rc (LTS release candidates), and stable kernels, and we report and share the results with the LTS community. Over the past 10 years, we have tested more than 600 LTS kernel versions.
The table below shows the number of entries extracted from “Tested-by” tags in the LTS commit logs; in reality, testing is performed on a larger number of LTS kernels.
| LTS | Test count |
| linux-4.4.y | 54 |
| linux-4.9.y | 28 |
| linux-4.19.y | 141 |
| linux-5.4.y | 13 |
| linux-5.10.y | 182 |
| linux-5.15.y | 27 |
| linux-6.1.y | 99 |
| linux-6.6.y | 14 |
| linux-6.12.y | 34 |
| linux-6.18.y | 12 |
- Bug reports and backport
We have been backporting vulnerability fixes and bug fixes, and we have contributed to the community by backporting about 100 fixes.
| LTS | backported patches |
| linux-4.4.y | 22 |
| linux-4.9.y | 17 |
| linux-4.19.y | 21 |
| linux-5.4.y | 18 |
| linux-5.10.y | 12 |
| linux-5.15.y | 4 |
| linux-6.1.y | 3 |
| linux-6.6.y | 2 |
| linux-6.12.y | 0 |
| linux-6.18.y | 0 |
- Contribute to mainline
Reviewing patches that have been backported to LTS or CIP kernels may reveal new issues. Since we develop based on the “Upstream first” philosophy, we follow a process where we propose fixes to the mainline kernel and then incorporate the accepted fixes into the LTS and CIP’s kernels. And we have contributed several patches to the mainline.
While this number is small compared to the number of patches merged into the mainline, it represents part of the results of our contributions.
We continuously carry out these activities and share the insights gained through the maintenance process with the entire Linux community.
3. Expanded Hardware Support for Long-Lifecycle Products
In the Linux kernel, there is a time lag between when new hardware becomes available via the upstream and when an LTS kernel – specifically an SLTS kernel – incorporating that support is finally released. This gap can extend up to two years, particularly when using SLTS.
To minimize this gap and reduce maintenance costs, we adhere to an “Upstream First” philosophy rather than incorporating proprietary vendor patches.
Therefore, it is necessary for older kernels to be able to handle new hardware. To this end, the WG has established policies to accept the following proposals from project members, enabling the practical maintenance of older kernels over the long term.
- Driver backports that do not break the existing driver framework
- Porting support for an SoC variant to an existing SoC
- Adding features required for industrial applications
When accepting these proposals, the WG conducts reviews by kernel maintainers and performs testing using LAVA and KernelCI.
4. Establishing a Security Maintenance Framework
The WG has established a framework to ensure long-term kernel security and carries out the following tasks:
- Monitoring CVEs
- Evaluating security patches
- Backporting patches and proposing them to linux kernel community
The WG also manages the CVE status for each CIP kernel and its underlying LTS kernel within the cip-kernel-sec repository. Additionally, we developed a tool called kernel-cve-triage to identify CVEs affecting specific kernel configurations, establishing a system that allows us to focus our efforts on CVEs that impact the reference boards and kernel configurations we maintain.
As these data and tools are designed for general use, they are available to other users as well.
5. Supporting Realtime Linux
Since real-time performance is critical for civil infrastructure, the WG participates in activities related to the PREEMPT_RT patch and CIP joined the Realtime Linux Project in 2017, supporting its merge into the mainline kernel and keeps on funding it to ensure keeps and further improves its quality and features.
6. Continuous Testing Using KernelCI and LAVA
Maintaining quality is a key challenge in long-term maintenance. The WG utilizes
to continuously perform automated builds, automated testing, and verification on actual hardware.
And CIP is a founding member of the KernelCI project. It supports quality assurance for the mainline kernel and leverages KernelCI also for its own SLTS kernels.
7. Bridging the “Longevity Gap”
The CIP was established in response to the significant disparity between the support period for Linux and the lifespan of infrastructure equipment.
– Linux LTS: 4 to 6 years
– Civil infrastructure such as railways and power generation facilities: 20 to 50 years
To bridge this gap, a common, long-term maintenance kernel was developed across the industry, reducing the burden on individual companies of maintaining their own proprietary maintenance kernels.
The Future of the Kernel WG
1. AI-Powered Patch Review
The Kernel WG is setting up an environment to conduct patch reviews supported by AI. This will enable more efficient patch handling while their numbers continue to increase and allow us to address issues and fixes that might have remained undetected previously.
Conclusion
In the last 10 years, the Kernel WG has developed a framework that enables the Linux kernel to be used safely for civil infrastructure for over 10 years.
Through the provision of the SLTS kernel, contributions to upstream, long-term security maintenance, real-time support, and continuous testing, the group has been working to bridge the “Longevity Gap”. And CIP demonstrates how collaboratively funded Open Source long-term support can even be achieved for something as complex as the Linux kernel.
This was not achieved by CIP only; it was made possible by collaborating with the kernel community through our “Upstream first” development policy and by maximizing the use of the OSS ecosystem to reduce maintenance costs and ensure compatibility with future versions. Demonstrating the effectiveness of “Upstream first” development in OSS-based projects is certainly one of the major achievements of our efforts.
Many of the civil infrastructure products we target are in use for over 10 years. Going forward, kernel support spanning more than 10 years will likely become a key focus.
We will continue to develop and maintain a Linux kernel that can be used with confidence over the long term.
We encourage you to adopt the kernel we support in your products and participate in its development.