Author: Jan Kiszka, CIP Kernel Team Chair, Siemens
We, the Civil Infrastructure Platform project, have just announced to select the new 6.12 kernel as our 5th SLTS (super long-term stable) kernel. This means that — after 4.4, 4.19, 5.10 and 6.1 — 6.12 will also receive up to 10 years support. If your update/testing/roll-out plans are not yet aligned to the 2 years that the latest LTS kernel guarantees by now, your problems are now solved once again by CIP, right? As often, things are a bit more complex. But let’s first reflect on why there is a kernel with such long support, where it can help and where it may not be needed.
The reasons for having stable branches besides the development head should be obvious: Ensuring that complex systems with hundreds or thousands of components work smoothly can be challenging. This is not only true for Linux. Eventually, you have to switch from integrating the latest and greatest versions to eliminating problems for a feature-wise frozen set of components. Those problems may be functional issues or performance gaps, including timing constraints when running real-time workloads. However, this way of consuming a dependency via stable releases does not yet answer the question of how long to stay on a specific stable series.
Let’s assume you are on kernel version 6.12. That release is going to be maintained as an LTS kernel for 2 years now. After one year 6.18 may be selected as the next LTS, and there may be 7.3 another year later, exactly when 6.12 is discontinued. But that means you must pick up version 6.18 as a replacement of 6.12. Otherwise, if you were to wait for 7.3 and it wouldn’t work immediately for you, you could end up stuck in the field without support for the discontinued 6.12.
Are you prepared for annual major kernel updates? Are you on a mainline kernel with no, or at least very few, extra patches so that updating will be little effort? Or do you depend on a chip or board supplier who requires you to use a large out-of-tree patch set? If so, is that supplier ready to provide you with an annual update early enough so that your own integration and qualification will be ready in time? What does your continuous test strategy for upstream kernels look like? Does it allow you to identify, report and possibly hunt down non-obvious regressions early?
Updating the major kernel version every year may be too ambitious for many projects using Linux for now, particularly in the embedded space. Every additional customization step and transfer point in your supply chain between the upstream kernel and the final deployment will make it even harder. Ideally, everyone would be working upstream first to avoid such hand-over points, but we are not yet living in an ideal world.
The CIP kernels with up to 10 years of support provide you way more flexibility regarding when you do a major update. If your product requires support for less than 10 years and you were quick to release it right after a CIP kernel was started, you may get away without any major version update in the field. More realistic is that it takes about a year, in some cases even longer, to develop a product based on a newly released CIP kernel, specifically if the supply chain is long or certification work is required. This reduces the usable support span. And as a new CIP kernel is generally selected every 2 years, you may be left with about 6-7 years of staying exclusively on the major version selected at the product launch.
No one says that you need to use the full support length of the CIP kernel, though. The wider the version jumps are, the harder it will be to find regressions and their root causes when you hit one. Instead, you may consider the CIP support length as an insurance that kicks in when you cannot switch at some earlier point, e.g. because the development and/or testing teams are fully booked or something else delays an update.
So, just use the CIP kernel and you are done? Not quite.
First of all, the extended support of CIP kernel is built on focusing its scope on a reduced set of kernel subsystems and features. You might be lucky, and your kernel configuration is already covered by the CIP project. If not, you have your first reason to join CIP: bring in your requirements and support the project in handling the additional effort this may entail. Joining also allows you to influence which SoCs and boards are actively tested, a second reason to become a member.
But even if both your configuration and your hardware are covered already, there is a third reason why you should support the CIP project if you want to use its kernel (or already do so): By the end of 2026, support for regular LTS kernels 5.10 and 6.12 will end according to current schedules. One year later LTS kernel 6.1 will reach end of life. This means the CIP project will have to maintain 4 kernels independently of LTS instead of the current 2 (4.4 and 4.19). We know how to do that, but it is clear that this task will require more hands on deck. And these hands are needed earlier in order to expand the team smoothly, give everyone a chance to get familiar with workflows and improve them before the peak workload is reached.
We need more eyes reviewing stable patches, ideally before they hit regular LTS. We need more people identifying backport candidates that get stuck between LTS versions or require extra effort to merge them into CIP kernels. And, last but certainly not least, we need more ideas and contributions to improve, together with KernelCI, the test coverage of I/O drivers in regular as well as CIP kernels so that regressions remain few and are found quickly. You can find the CIP kernel source code at https://git.kernel.org/pub/scm/linux/kernel/git/cip. Accompanying repositories containing supported kernel configurations, CVE tracking, patch review results and other bits and pieces can be found at https://gitlab.com/cip-project/cip-kernel. The CIP kernel community is exchanging patches and discussing issues on the cip-dev mailing list at https://lists.cip-project.org/g/cip-dev. In addition, there is also a weekly IRC meeting, watch out for its announcements on the mailing list.