Core Linux developers are finding themselves managing and checking, rather than coding, as the number of kernel contributors grows and the contributor network becomes more complex.

That is the view of Greg Kroah-Hartman, maintainer of USB and PCI support in Linux and co author of online book Linux Device Drivers.

In the latest kernel release the most active 30 developers authored only 30% of the changes, while two years ago the top 20 developers did 80% of the changes, he said. Kroah-Hartman himself is now doing more code reviewing than coding. "That's all I do, is read patches these days," he said during a discussion at the Linux Symposium in Ottawa last month.

In theory the kernel development process involves changes going from the original author through a file or driver maintainer, to the maintainer of a major subsystem such as PCI or SCSI, to Andrew Morton for testing and finally to Linus Torvalds for a kernel release. But Kroah-Hartman said the process was much more complicated: "I tried graphing that, and that's not what happens. It's a mess. There's routing all over the place."

A graph of all the developers involved in the upcoming 2.6.22 release, as well as of the relationships of who reviewed whose patches, extends to a 40 foot print out with names in small type. The graph is on display at the Ottawa event.

The "mess" results in innovative features becoming integrated into Linux distributions much more quickly, according to Jonathan Corbet, author of the camera driver for One Laptop Per Child and another writer on Linux Device Drivers.

Previously, when developers maintained both "stable" and "development" kernels, it could have been two to three years before a feature made it from development to mainstream users. Today, by contrast, the newly released Fedora 7 distribution has the power saving tickless kernel functionality, which came out in the 2.6.21 kernel in April.

Enterprise Linux distributions that pick a single release and maintain it for five to seven years are another reason for the complexities. Instead of waiting for a stable upstream release and then modifying it to include new functionality from development kernels, an enterprise distribution can support any of the 2.6 releases, which come out every two and a half months. Corbet said: "The patch loads carried by the distributors have shrunk quite a bit."

Kernel release 2.6.11 of March 2005 had 475 developers, and the upcoming 2.6.22 release this month has 920 developers, Kroah-Hartman said.

The number of kernel changes is also growing. While 2.6.11 had about two changes per hour, 2.6.22 is at four changes per hour. And even when Linus Torvalds took a month off to write the git revision control system in the spring of 2005, the rate of change was faster than the entire 2.5 "development" tree. "We are going up. We are going faster than we have before,” Kroah-Hartman said. “This scares the heck out of traditional computer scientists."

Despite Red Hat's dominance of commercial Linux sales, the spread of actual development contributions is much more diverse, with Novell, where Kroah-Hartman works, punching above its weight at 9.7% of contributions to Red Hat's 11.8%. The next three companies in the rankings are IBM, Intel and SGI, which has installed a 4096-processor supercomputer, the largest single system image Linux box yet.

But just above SGI in the listings are what Kroah-Hartman calls "amateurs". Although the great independent Linux hacker has been thought extinct, known amateurs still count for 3.9% of kernel changes. Reflecting the large number of small changes, the top category is still "unknown", but everyone with 10 or more changes in the kernel is accounted for, Kroah-Hartman said.

The large contributor base, huge code base (8.2m lines) and rapid change means that it is impossible to maintain quality kernel space code for Linux outside the development process. Kroah-Hartman said: "You're never going to be able to catch up. It's just physically impossible. If your company isn't showing up here and you need to affect how Linux is developing in the future, you need to get involved."

An audience member asked about kernel size comparisons to Microsoft's Vista operating system, but the two projects are not directly comparable, because Linux maintains its device drivers "in the tree" and available for all developers to see. 52% of the size of the kernel is in drivers, and less than 5% is "core", Kroah-Hartman said. However, changes happen in the core at the same rate as in other parts of the kernel. APIs and core data structures change regularly.

Kroah-Hartman said he monitors bug reports from several distributions. "I've seen a flat rate over the past two years," he added. Constant bugs and growing changes implies fewer bugs per change, but there is always room for improvement. Linux needs a person to coordinate bug tracking, Corbet added, and Google has offered to hire someone to do the job.

Users cannot really blame the kernel hackers for moving things around. Many users and vendors are using Linux for diverse requirements, from high performance computing to mobile phones, and users' hardware and application needs drive kernel changes.

"We are changing based on external stimuli," Kroah-Hartman said, adding that Linux is the only operating system that runs on very small embedded systems and on large NUMA systems from SGI and others. "That's an architecture scalability that nobody has ever achieved. We're achieving that thanks to our rate of change," he said.

In the past two years, 3,200 developers have contributed at least one patch, with half contributing two or more, one quarter contributing three or more, and so on in a "long tail" exponential curve. Every patch will have at least one author and one reviewer. "It is a web of trust. Linus trusts 10 to 15 people, I trust 10 to 15 people," Kroah-Hartman said. "I trust that they'll be around to fix the problem. And that's more important than getting it right in the first place."

Now read

Microsoft - Linux patent wars

Linux news roundup