Android and Linux communities forge communication ties

History of a free software collaboration fraught with misunderstandings

One of the key achievements of the computing community over the past five years has been the elimination of rancor between the Android and Linux development groups. There’s less of throwing changes over the wall and having them rejected (often with nasty language) and more resort to regular meetings with an exchange of ideas about solutions. This evolution also signals a modest culture change among Google developers of the Android operating system, who were long noted for their reluctance to engage with outsiders.

The very structure of modern software, where one layer depends on another, inevitably leads to inter-team disputes. Free and open source communities routinely look for pre-existing projects on which they can base new ones, so Byzantine dependencies develop between teams. (Just look at the dependencies you need to install whenever you download new software.) The Android run-time depends on the Linux kernel, so friction sometimes comes up along the interface. But the Linux community also realizes that Android has brought Linux to a gigantic market and set of users it had never reached before (there have been a good billion shipments of Android devices) and must be taken seriously.

The arguments caused by this friction reminds me of the distrust that for a long time existed between the Ubuntu and Debian distributions of GNU/Linux, which I explored in another article. Ubuntu, like many other distributions, chose to enhance the Debian distribution instead of starting out from scratch. And this choice had clear benefits to the Debian community, because the Ubuntu distribution is incalculably more popular than the Debian original. But they sniped at each other over questions of whether changes to Ubuntu had been adequately communicated to Debian, and whether Debian was accommodating Ubuntu's needs.

It's important to remember that disputes between free software communities are much less destructive than those between proprietary companies, where the company with the underlying platform is watching the users of that platform with eagle eyes to detect signs of success, whereupon it often snaps up the company or re-implements a competing product of its own. I'll take the Android/Linux controversy over that situation.

The Android project is also unique from a free software perspective. Although released as open source, it is developed by a tightly controlled team. Many community patches are accepted into Android (Serban Constantinescu mentions in particular the contributions of Linaro, an open-source ARM team, to the Android Runtime), but direction of the project rests with Google. That development model has plenty of precedents (including the variants of the BSD operating system), but it may have led to the difficulties Google had getting approval from the Linux community for changes it made to the Linux kernel.

Chief among these disputes was Android's power management system. Android developers naturally wanted to put devices to sleep as often and as long as possible, given that battery life is one of the dominant criteria by which mobile users judge their phones. From the very first release, therefore, they tried to improve battery life by changing the Linux kernel to support a power management system called wakelocks and later suspend blockers. The most accessible description of this kernel change probably comes from LWN.net, a widely trusted source of information not only for GNU/Linux but for many related open source technologies.

In LWN.net, Jonathan Corbet also chronicles the unsatisfactory information exchanges between the Google developers and the Linux kernel community. After coming up with the wakelock solution--which was not ideal, as we shall see--the Android team thought they could submit it to the mainstream version of Linux with minimal friction. Rom Lemarchand, staff software engineer at Google, ruefully describes the original attitude of the Android team as, "We implemented this feature and drivers out in the field are using it, so mainline it."

In his article, Corbet explains why the kernel developers wouldn't allow the change in for several years, and even uses the debate as a poster child for poor relations between embedded systems developers and the Linux development team. The last point was an interesting rhetorical pivot, because Android is not a typical embedded project. The breadth of its software and scale on which it ships undermine comparisons to any other project. Corbet says that the Linux community rejected wakelocks on technical grounds, but my sense it that they were reacting as well to the Android developers’ flawed communication pattern.

I won't describe the technical reasons for which the Linux team rejected wakelocks (you can find 18 pages about these issues) but I’ll just say that the requirements for the feature made it very hard to implement successfully. Essentially, the Android team wanted:

• to put the processor to sleep as often and as long as possible, but
• to let the Android framework and even user apps to wake up the processor when they need it, and yet
• to prevent buggy or rogue apps from waking up the processor too often.

I'll also note that complaints were not limited to the Linux developers. A study discovered errors and abuses in applications using wakelocks, performance problems were widely reported, and someone even created an app that allowed users to disable wakelocks. Unsatisfactory replacements were suggested.

Lemarchand defends wakelocks against the accusations, and attributes the backlash against them among Android users to two factors. First, wakelocks radically altered the familiar suspend/resume model known to laptop users. The second factor was the Android team's silence on the reasons for their solution, which was based on the controversial idea that "applications know better than the kernel when they need the system to be awake." Lemarchand says the team has gotten much better at explaining themselves in public, having learned from the wakelocks arguments.

LWN.net reported some improvement in communications between the communities in February 2009. Karim Yaghmour, who straddles the Linux and Android development communities, told me that a real breakthrough came at a collaboration summit in 2010. Kernel developers Greg Kroah-Hartman and Ted Ts'o worked with Android developers there.

At the 2011 Linux kernel summit, key leaders of the kernel community grudgingly decided that, since wakelocks had been in Android devices for so long, they ought to be accepted into the kernel repository. The reluctance and backhanded language that accompanied this historic decision can be seen in the LWN.net report. But the ending is a happy one: a enhancements to the wakelocks concept evolved, such as the addition of a queue, and the Android solution has converged with the mainline kernel. The communities are working in a much healthier fashion on a more recent controversy about how to distribute processes among CPUs. Corbet says, “The important thing with the current scheduling work is that the developers did it with the plan to upstream the code from the Android team to the Linux kernel quickly from the outset. They knew it couldn't be a single-device solution and would have to pass muster in front of the whole community.”

More important, ambassadors from both sides have worked hard to bring together the Android and Linux developers to avoid such long-lasting disputes. Yaghmour organized the first Android microconference at the 2012 Linux Plumbers Conference; it has reconvened each year since then and has doubled in size. An article on its recent November 2016 session appears on LWN.net. Some of the key peace-makers in the past several years include Brian Swetland, Colin Cross, Rom Lemarchand, Todd Kjos, and Dmitry Shmidt on the Android side, along with John Stultz and Serban Constantinescu on the Linux side.

Android made another key advance in 2014, when its binder driver (a key piece of technology supporting the main form of interprocess communication) was accepted into the mainline Linux kernel. A number of other Android changes have made it into the mainline kernel over the years, while others have not.

What we can learn from this contentious history is that communication is a key skill for computer developers, whose poor communication skills have been historically lamented. Great solutions can rarely be found by a single person, or even a single team. Although brilliant developers have created solutions that work for the moment, the solutions that last over time need to poked and prodded by communities. Many developers for embedded and mobile systems need to learn to work more closely with the kernel developers, according to Corbet--currently they add millions of lines of code to Linux and run idiosyncratic kernels on their devices. Conversely, the Linux developers are trying to broaden their viewpoints to make sure they can respond to needs from downstream developers such as Android.

Lemarchand summarizes the current approach that the Android team takes as a blending of rapid innovation and community involvement, where the Linux kernel is concerned. Their timelines don't allow for leisurely discussion, because they typically have a year or less to deliver on market needs. So they post their changes to appropriate public forums, along with discussions of their needs and why they chose their solutions. This allows the kernel community to come up with improvements even as the Android team makes its changes. What eventually end up mainlined in the kernel is a synthesis of the best that the Android team and the wider community have come up with. For instance, after implementing the Atomic Display Framework (ADF), they had many discussions with Rob Clark of Linux's Direct Rendering Manager project to address their needs to reduce fragmentation and standardize on a single interface between kernel and userspace.

Developers can make great progress just by reaching out, saying, "What do the rest of you think of this"?" and being sincerely open to answers. Even stronger collaboration can be achieved by participating regularly in forums and face-to-face meetings, and cultivating leaders comfortable in both communities as Android and Linux kernel developers are doing.