I spent the latter half of last week at the Linux Plumbers Conference in downtown Portland. And despite being a new conference, I found it to be one of the best conferences I've attended. I think this might be in part to its small size, there was a much better average developer to community leader ratio than at other conferences, indeed some of the key leaders don't even attend some of the larger conferences. The fact that LPC piggy-backed on Kernel Summit had a lot to do with the turnout. So what was exciting?

"If you want to do it right, do what IBM does, do what Intel does. -- Greg KH"
Well, maybe that was only funny to me...

While there was a lot of great content, the things that stood out to me were:

From Naught to Sixty in 5 Seconds

Arjam van de Ven and Auke Kok shared their work on getting Fedora and Ubuntu to boot in under 5 seconds on a netbook. While the SSD drives were key here, they still could do 10 seconds with rotating platters. Key bits including Super Read Ahead, disabling everthing you don't use, and picking on everyone that wasted boot time :-) Canonical took another hit here for GDM taking 3 seconds to start (60% of their total schedule!) due to storing massive pngs that it scaled down at each run to fit the current resolution. Definitely plan to look at my boot process and see if I can knock a few 10s of seconds off.

Git Tutorial

So was this cool only because it was Linus presenting? Yes and no. Linus is actually a great presenter. He is bright (duh) but also witty and engaging. He had no slides and simply walked the room through some git usage. While most everyone in the room had used git before, no-one seemed ruffled by it. It was nice to hear first-hand why git is the way it is. Did I come away from it loving git - uh... no. But it did clear up some of the issues I had with it - mostly by correcting my perspective on how it should be used.

Graphics Drivers in the Kernel: 20 Years Late

Dave Ailie and Jesse Barnes (among others) discussed the work being done to move video drivers into the kernel (from xorg userspace where they are now). Specifically memory management and mode setting. There was some good debate between the speakers and Linus, which ended with three of the speakers simultaneously stating "Well, you're wrong!" to the chief penguin. There was a good chuckle from the crowd, things usually go the other way. The presenters knew their work, they knew the interactions of the various pieces, and they were able to defend their approach which Linus didn't accept at first. His pragmatism was very apparent during the discussion and his drive toward simplicity made very clear. An excellent discussion. So what does this mean for users? hopefully those of us with the right chipsets will not have to watch our screens struggle through resolution changes 3 times during a single boot :-)

Hallway Dialog

While the formal talks and discussions were great, the best part of any conference is the one on one time with the people you work with over email and irc the rest of the year. For me this was a lunch with Thomas Gleixner, Steven Rostedt, and Gregory Haskins. We discussed the problems pthread condition variables have with priority inheritance, and formed a game plan to get the solution moving forward. It's a tough problem, and largely logistical in nature rather than technical. Changing APIs (and ABIs) at the glibc level is a painful process (for good reason).

All in all, it was a great conference. A lot of good discussion between the developers of various systems was had (which was of course the intent). Thanks to IBM for letting me attend, and thanks to the organizers who donated so much of their time to the event.

I picked up a 1 TB Western Digital My Book World from Fry's today - that's a 1 Terabyte Gigabit Network Attached Storage box - for $220. I've seen a few consumer technical appliances that run Linux under the covers, and haven't been terribly impressed with many of them. So far however, this box is slick. While WD doesn't support Linux officially (bad WD!) it is pretty trivial to get the box into a techie-friendly state.

With only an hour or so of tinkering, I was able to enable ssh, disable the java server, add my own users, and mount my partitions via sshfs. The info on doing this is already covered by "Paul" at his How to setup My Book World Edition II page so I won't duplicate that here. Should also probably link to Martin Hinner's My Book Howto directly as he appears to have done the hard work. I'll spend the next few days figuring out how to best make use of this device; most likely looking into something like rdiff-backup.

Something they don't mention are the specs of this little beauty, for those of you who might care, read on...

Today at the Red Hat Summit in Boston, Red Hat announced the official release of Red Hat Enterprise MRG V1 (Messaging Realtime Grid) [1]. A couple snippets of note:

"The Realtime component of Red Hat Enterprise MRG comprises numerous kernel enhancements that provide deterministic performance for time-critical and latency-sensitive applications."

"IBM has worked together closely with Red Hat on the development of the real time Linux kernel and has optimized both WebSphere Real Time and BladeCenter servers on Red Hat Enterprise MRG. We are delighted that IBM and Raytheon have been recognized by Red Hat for this innovation which has led to the largest deployment of real time technology in the next generation of US Navy Destroyers." --Jeff Smith, vice president, Open Source and Linux Middleware at IBM

As enterprise distributions continue to roll out their respective real-time offerings, the benefits (and drawbacks) of real-time are becoming available to the masses. The expectations for these distros are often a mixture of general purpose computing performance and real-time determinism and low-latency. Unfortunately, meeting one of these often comes at the expense of the other. Deterministic behavior requires more overhead in the OS, including some lock serialization and additional context switching, which naturally degrades throughput. Two real-time Linux developers, Steven Rostedt of Red Hat and Gregory Haskins of Novell have been actively working on two distinct approaches to minimizing the effects of determinism on throughput.

Steven's rwlock-multi patches [1] (2.6.24.X-rt8) attempt to reduce the bottleneck of the conversion of the rwlock to an rtmutex in the PREEMPT_RT [2] tree. In order to provide Priority Inheritance [3] support, the rwlock was converted to an rtmutex, limiting it to one owner. This bottleneck hit some subsystems harder than others, networking was particularly affected. The rwlock-multi patch relieves the serialization by allowing a fixed number (configurable) of readers to hold the lock at any one time, bounding the wait time to a higher priority writer. As would be expected, if a high priority writer is blocked on an rwlock, it will boost the priority of all the readers, and prevent any new lower priority readers from getting the lock (as soon as the existing readers release the lock, the writer gets it). This approach certainly increases the latency a high priority writer may see when acquiring a contended lock, but the increase is bounded, and therefor still deterministic. The benefit is seen by all subsystems that leveraged the rwlock for additional throughput, bringing the rt kernel closer in performance to the mainline kernel, while maintaining the necessary deterministic behavior.

In recent years it seems commodity desktop systems have outstripped the demands of software, with the possible exception of demanding video games. With enough RAM to run multiple operating systems easily, more diskspace than most users can even comprehend (let alone use), and dual-core processors running at equally incomprehensible speeds, one might expect to see richer and richer applications and user interfaces becoming the norm. While Vista and Compiz/Beryl certainly polished up the desktop, they certainly haven't done so on a scale to match the growth in raw power over the last several years.

Instead, we are seeing more and more of "Web 2.0" type applications. Gmail marked the start of this era for me, followed closely by Google Calendar, and Google Docs. I was just today introduced to Zimbra (an open-source messaging and collaboration suite with a fully AJAX based user interface). Rather than seeing richer clients to take advantage of all these unused clock cycles, the industry seems to be going back to the model of the server and the thin-client. Back when computing resources were costly, it made sense to deploy inexpensive dumb-terminals, which just rendered what the server told them to, and let all the computing occur on the server or mainframe. Interestingly enough we are returning to this model, even though modern desktops far outstrip the servers of this previous age.

Priority inversion is a common and well documented situation known to any real-time system or application developer. The mechanisms in place to avoid it are similarly well known: priority inheritance and priority ceiling. For the uninitiated, I'll briefly review these topics.

Priority inversion occurs when a high priority task is prevented from executing because a low priority task holds a lock it requires. Unbounded priority inversion occurs when that low priority task is preempted by a medium priority task, preventing it from completing its critical section and allowing the high priority task to continue (since there is no way to know how long the medium priority task will run for). POSIX pthreads provide two mechanisms for bounding priority inversion (not preventing it): priority ceiling and priority inheritance mutexes.

Well I couldn't resist the tempation to try Linux on the Playstation 3 any longer. I held out longer than Mel Gorman did (I played the games first, _then_ hacked it ;-) While Yellow Dog seems to be the officially endorsed distribution, I am accustomed to Ubuntu, aptitude, and all the Gutsy goodness that comes with the Debian based Ubuntu Gutsy. PSUbuntu has some pretty simple to follow instructions and convenient installation media that made the process pretty painless. Note that the otheros.bld included on the alternate installation media has a "USB Disconnect" bug that prevents the installation from starting, so you will need to grab the otheros.bld file separately and put it on a USB stick (the instructions do tell you to do this - but don't explain why). The installation itself took a long time (I forgot to time it), but several of the dialogs would sit at a given percentage for what seemed like several minutes before continuing.

Once installed, all I had to do was configure my network (I use wired ethernet) and setup the screen resolution (documented on the PSUbuntu Setup page). The ubuntu login on my 40" 1080p HDTV was glorious :-) The system runs pretty slow, due to the 128 MB RAM the system has to work with. I'm hoping to tinker around with it and come up with an interface that is both light-weight, and more convenient for use from the couch 12 feet away. I am considering running a VNC client on it, rather than a full Desktop, and let my mostly idle server do the heavy computing. I think I'll also take a look at getting the -rt kernel running and experiment with the IBM Cell Development kit.

I have received some questions lately about how to get started with Real-Time, and how to make your application take advantage of Real-Time. In particular with regard to process scheduling. I'd like to share some of the basics here, with some references for more reading.

Real-Time Scheduling Overview

The Real-Time Linux kernel provides more control over what gets run when. With the proper privileges, you can specify the priority of not only your application threads, but also the critical system service threads, like hard and soft interrupt handlers. With this power, comes great responsibility! It is very easy to mis-configure a real-time system and render it unresponsive.

Real-Time processes run under either SCHED_FIFO or SCHED_RR policies. These processes are treated differently by the scheduler. SCHED_FIFO processes have a real-time priority, and do not have a time-slice. This means that they will run until they are either preempted by a higher priority real-time process or they voluntarily release the CPU, either by explicitly sleeping, or blocking on a lock or some other synchronous blocking call. SCHED_FIFO tasks are globally scheduled, meaning the scheduler ensures that the NR_CPUs highest priority runnable SCHED_FIFO tasks are running on the CPUs at any given time. For example, if a priority 90 process is preempted on a CPU by a priority 91 process, the system checks to see if the priority 90 process would preempt any processes on the other CPUs, if so, it is moved and continues running. SCHED_RR processes have a real-time priority, but they also have a timeslice. The scheduler schedules these processes in a round-robin fashion out of the local runqueue. This may produce unexpected results if you were hoping for global round-robin behavior, but round-robin'ing tasks from all the system runqueues would require a lot of extra logic (rather than relying on the FIFO nature of the runqueue priority arrays). A final key difference for both policies: real-time processes are never placed on the expired array. This affects the behavior of things like sched_yield(), which is basically a no-op on the highest priority real-time process on the system. In my experience, SCHED_FIFO is a much more common real-time scheduler policy.

We've been struggling with some pretty nasty bugs relating to predictable runtime. The logs the testcases produced are enormous, and we were struggling to visualize what was actually going on. I spent some time whipping up a visualization tool using python, numpy, scipy, and pylab. I haven't used scipy since my Signals and Systems course at the university, it felt great to dust it off and make use of it again. With only a meager understanding of signal theory and statistical analysis you can easily create scatter and line plots, histograms, and even fast fourier transforms. One of the most useful things about the pylab plotting interface is that it is interactive. You can zoom in and out and pan the original plot, increasing the amount of discernable information. Kudos to the python guys for creating such an excellent language for rapid tool development.