Linux Kernel 2.4 Or 2.6 In Embedded System?

snikulin writes "My 6-year-old embedded software happily runs on kernel v2.4 on an XScale CPU. The software gets a bunch (tens of megabytes) of data from an FPGA over a PCI-X bus and pushes it out over GigE to data-processing equipment. The tool chain is based on the somewhat outdated gcc v2.95. Now, for certain technical reasons we want to jump from the ARM-based custom board to an Atom-based COM Express module. This implies that I'll need to re-create a Linux RAM disk from scratch along with the tool chain. The functionality of the software will be essentially the same. My question: is it worth it to jump to kernel 2.6, or better to stick with the old and proven 2.4? What will I gain and what will I lose if I stay at 2.4 (besides the modern gcc compiler and the other related dev tools)?"

Re:GCC 2.95? Seriously?

[Brane2]

2.95 is/was regarded as a "golden" version for its maturity and stability.

I'm not certain that newest 4.3x is that much better on small embedded system without SSE and FPU units to be worth a swap...

Premption, responsiveness

[Thomas Charron]

The largest benny for an embedded system with 2.6 is timing, really. The kernel is now, for the most part, 'almost' totally preemptable, bring sort real time to the kernel. Additionally, using RTAI Fusion, you can get hard real time. RTAI extensions have phased out support for the 2.4 kernel.

But one would have to ask, I suppose. If your just replacing a legacy component with the same thing, why change the code?

Constraints

[LS]

Others in this thread will adequately cover the feature differences between 2.4 and 2.6, though it sounds like 2.4 already covers your needs when it comes to functionality. This makes your question more of a management one than an engineering one.

With these types of decisions you need to look at what your constraints and requirements are, whether they be time, developer resources, product lifetime, estimated lifetime of leveraged technology (kernel 2.4 in this case), cash, etc. It sounds like you'll be doing the development yourself, but otherwise I can't tell what the rest of cycle looks like, so you need to clarify these things before making a decision.

Those are major considerations, but it gets more subtle when you consider things like how much time you'll save with future updates due to better development tools and support with a new kernel, etc., so you need to estimate also whether the time you spend up front will be saved down the line.

LS

What do you want?

[RedWizzard]

So far all the positively moderated posts have advocated 2.6. Here's a slightly contrary view. You know 2.4. It seems to satisfy your needs. Why exactly are you considering change? Is there something 2.4 doesn't do that you want? I realise you might be asking in case there is some improvement that 2.6 may possibly provide that you've missed, but if the current setup does what you need then why would you even consider a change? My advice: stick with 2.4 unless 2.6 provides something additional that you definitely need.

Re:Why Linux?

[miknix]

Exactly, I would not say it better.

As a member of the gentoo embedded team I would recommend the use of crossdev to generate the toolchain.
By emerging crossdev-wrappers and setting up some gentooish cross-compiler environment, it is possible to cross-compile (by simply emerging them) a lot of packages on portage.
Emerge will take care of most things leaving the most ugly cross-compile errors for you.

http://www.gentoo.org/proj/en/base/embedded/cross-development.xml [gentoo.org]

Regarding the guide, don't use the xmerge script. Just emerge crossdev-wrappers instead.
Feel free to join #gentoo-embedded on irc.freenode.net

Happy xcompiling.

Yes if you have time and patience

[Anonymous Coward]

I constantly port the kernel and distributions for custom embedded designs for new and old hardware. It is really painful.

To go to 2.6 you have to rewrite ALL your custom drivers and the board configuration files. Yes is nicer in 2.6 - but it still has to be done. Just like we did in 2.2, 2.4 and probably again for >= 2.8.

It is a real pain that the kernel and apps change so rapidly. You regularly run into compatibility problems with:

1. Host GCC supporting target tools
2. Host Linux distribution supporting target tools
3. Libraries for the target
4. Dependencies - bloody dependencies

I recommend:

1. Get a new harddisk with a fresh distro for the build machine.
2. Different login for every board type with its own CROSS_COMPILE exported in .bashrc. Stops accidental bad ROM builds.
3. Use TFTP and NFS root - they are your friends.

The port from 2.4 to 2.6 can take 2 days to 2 weeks. If you can afford 2 weeks go for 2.6.

Re:GCC 2.95? Seriously?

[AaronW]

I have had issues with the VxWorks GCC 2.95.3 on several occasions when the compiler generated incorrect code resulting in crashes and lockups. The C code was correct, but the resulting MIPS assembly was incorrect. Each time, making slight changes to the C code would fix it, i.e. replace a for loop with a while loop. I say good riddance to VxWorks. The memory management in VxWorks 5.4 was atrocious and had to replace malloc with DLMalloc plus add a method of tracking memory usage on running systems (marking each memory block with the task and caller PC) so we could find and fix memory leaks on systems out in the field. The networking was also pretty bad in it.

Linux 2.6 is a likely candidate...

[Svartalf]

As someone facing a similar situation at his day job, 2.4 is painful in some regards. In my case, 2.6 allows me to do a non-standard initramfs (the stock is minimal and then you can load other initrd or initramfs images from the kernel options...) so that I can tapdance around the three differing hackish bootloaders they did in 2.4. This allows me to do major cleanups in what they did for doing NFS rootfs on the IXP2800 blades and on the X86 ones with minimal pain.

Most of the people commenting on 2.6 being too big are thinking of the whole size with everything loaded up. Minimal kernels with just your drivers loaded and only your drivers in the module build, you end up with only about 5-10% increase in footprint in memory and store space, with the ability to provide modern device support for things. In the case of what you mention, you're moving to an Atom based machine board. Given that you're moving to a modern board, the odds of things being "nicely" supported is lower with the 2.4 kernel.

Since you're manipulating large volumes of data over GigE, you're going to want to switch, probably even with the old ARM stuff if you can manage it. 2.6 provides much more responsive networking performance (so long as you do your network code right and don't dink with the scheduler (heh...let's just say I corrected a not so good idea there recently...)).

You may have to port a few custom drivers over to 2.6, but in the end, it'll work better since the driver architecture is better in 2.6.

I'd suggest doing both, one at a time.

[wtarreau]

Well, it's not wise to change both the hardware and the software at the same time. You think it will reduce your time to market but it might increase it instead due to the numerous changes that will have to happen in your toolchain before getting anything barely working again.

From what I understand, you have a big experience in 2.4 and Xscale. 2.4 Also works on x86, so you'll not have to re-learn everything from scratch by just changing the architecture. All your toolchains, boot scripts, packaging scripts, etc... will still work as they did before. Then, only once you get familiar with your new architecture and the minor changes you might observe in the boot sequence, build process etc... it will be the right time to evaluate a migration to 2.6. Once you put your finger there, you'll quickly discover that you need to upgrade your gcc, glibc, replace modutils with module-init-tools, experiment with hotplug and sysfs, maybe switch to udev, etc... Step by step you'll notice a big number of changes, but you will be able to proceed one at a time, which is not possible if you change the soft at the same time as the hardware.

Also there are other aspects to consider. 2.4 has been maintained for a very long time, and you're probably used to backport some mainline fixes to your own kernel from time to time. 2.6 is not maintained that long (avg 6 months), and changes so fast that you will not be able to backport fixes for many years. I'd strongly recommend to start with 2.6.27, because Adrian Bunk will maintain it for a long time, as he did with 2.6.16. Once 2.6.27 is not maintained anymore (in about 2 years) you'll have to decide whether you stick to 2.6.27 and try to backport fixes yourself or switch to 2.6.36 (just a guess).

Also, 2.4 accepts almost no new hardware nowadays. If your new platform works well, that's fine, but how can you be sure that next year your GigE NIC will not change to something not supported anymore ?

I would say that the only case where 2.4 would make sense for a long term starting from now is if you don't have the time to revalidate 2.6 or to wait for 2.6.27 to stabilize, and need to quickly release something which will sit at your customer's in a place where it cannot be upgraded. Something like "install and forget". But I don't feel like it's what you're looking for.

So, to summarize :
      1) switch your architecture
      2) switch your kernel

Whether an official release of your product exists between 1 and 2 is just a matter of your time constraints and customer demand.

Last, to show you you're not alone, I'm too considering switching our products to 2.6, but next release will still be 2.4. Too many changes for a short-term release, and 2.6.27 not ready yet to reach years of uptime (but it's getting better though). 2.6.25 was particularly good but not maintained anymore.

Hoping this helps,
Willy