Kernel Development

For the kernel hacking class that I’m in it looks like I’m going to be adding support for turnstiles to the Linux kernel.  Turnstiles can be found in Open Solaris and they are used in Semaphores instead of wait queues in Linux.  Turnstiles are different from wait queues because they also take priority into account when they are selecting the process that will receive the lock.  This helps solve the priority inversion problem.  I’ll also need to come up with a way to prevent process starvation.

Solaris Locking Primitives

Mutexes – There are two types of mutexes that are currently implemented in the kernel. They are spin and adaptive mutexes. Adaptive mutexes are designed to either spin or block when a lock is held; depending on what state the holder of the lock is in. Spin locks have to be used in high-level interrupt handlers, because blocking means you are forced to have a context switch.

Conditional Variables – Conditional variables are kernel thread level synchronization primitives. These are implemented as a structure, and they also use a mutex to provide the locking ability. The mutex is passed into the functions as necessary, it is not part of the conditional variable structure.

Semaphores – The semaphores that are included in Solaris are standard counting semaphores that we’re used to using. They should be used in situations that the speed of mutexes and rwlocks aren’t necessary. The fact that they are a counting semaphore makes it very natural for them to be used for resource allocation and deallocation. They also use standard sleep queues, unlike mutexes which use turnstiles when the processes block on a given resources.
Contracts – These are a much more complicated synchronization primitive. There main purpose to help enrich the relationship between a process and the system resources. These system resources can include pages, threads, and they can also be used for asynchronously reporting errors to a process. This primitive depends on the contracts file system to provide an interface to user level processes. Contract types are used to implement the different relationships between processes and system resources.

Turnstiles – Turnstiles are another synchronization primitive that provide blocking and wakeup support, and priority inheritance for synchronization primitives, which include mutexes and rwlocks. This is very similar to the linux kernel wait queue, but the difference is that this helps prevent priority inversion. It does this be supporting priority inheritance and not being a standard FIFO data structure like the linux kernel wait queue.

Dispatcher Locks – This is a type of kernel locking primitive that is specific to the Solaris scheduler. There are two types of these locks they are spin locks and locks that also have the ability to raise the priority level of a process. They are used almost exclusively in the scheduler, they see extremely limited use in other parts of the kernel. This is needed because you cannot handle interrupts when you are modifying the dispatch queues. This is very similar to Linux spin locks that also disable interrupts on the system. This is a one byte data structure that also has the ability to disable the irqs and is implemented in assembly language all platforms.

Authors problem

Nate,

Are you referring to the author part that was in the date tag on each post before?

Mike

Nexenta Review

Install:

The installation of Nexenta isn’t exactly the easiest thing in the world, but it’s not the hardest. It’s on par with a more advanced Debian installation. My only real gripe with the installer was it was slow. It took 2 hours to install under VMWare with no other programs running. That’s just way to much, especially when I can install Fedora in about an hour and a half and install my complete development environment.

User Interface:

The user inteface is the quite similiar to Ubuntu. It is based on it after all. The only other thing that annoyed me was the fact that they still included all the KDE applications in the Gnome menus and they didn’t bother to put them in another submenu. It just seems to clutter everything up and really bother me.

Package Management:

Nexenta uses apt for its package management.  This is much better and easier to use than the regular Solaris tool, pkg-add which doesn’t fetch any of the dependencies when you go to install a package.  One thing that I found disappointing is that they don’t have a package of the Sun Studio Compilers.  You’ve got to go and download it yourself, they have gcc as the default which doesn’t easily build open solaris kernels.  You’ve got to have some extra tools and packages.

Overall Nexenta is a pretty decent Unix distribution.  It is a huge step up from Open Solaris and the Java Desktop System.  It just can’t really compare to most of the Linux distributions that are available, especially in the installation process.

A quick test run

I installed one of the Fedora Core 5 betas in the lab today.  The new installer’s pretty slick, but it doesn’t support an upgrade from Fedora Core 4 currently.  I find it’s a bit easier to use, and it seems to be faster even when I’m installing from dvd, instead of the usual network based installation. It took twenty minutes.

We’re using one system in the lab to test it so hopefully we can be ready to roll out FC5 relatively quick, it’s got some new cool features in it and hopefully kickstart is fixed so we can stop using g4u, as thats a bit to slow for me.

Finally some hack time

Finally managed to get in some quality hack time this morning on my CS450 project. Managed to get the build system setup, and I designed and implemented most of the common virtualization interfaces, so if Xen 3 ever manages to stabilize we’ll be able to implement support for that to. We’ve also got a good chunk of our project plan completed. It’s a bit aggresive but we should be able to get everything accomplished.

Also managed to start working on my Tuxmasters in the past couple of days. It looks like I will have to implement a specialized filesystem for Xen overcommit support, the Linux swap support doesn’t look like it will be fast enough. I think I’ve found the code path that I’d like to have the balloon driver replace so the OOM Killer doesn’t actually get executed. It’ll be interesting to see how effective my approach is.

It’s that time of year again (A small lab update)

It’s almost time to head back to the North Country and get ready for the semester. This semester is going to be pretty interesting as we’ve got new desktops from IBM that we’ve got to get setup in the first couple of weeks. We’re also going to be getting some new servers at some point during the semester, including the ES7000 from Unisys.

Hopefully we’ll have some interesting projects on deck. The web page is going to be redone, and hopefully a lot more work is going to be done with Xen.

On a bit of a random note, how many people would be interested in doing an Alumni weekend/event at some point during this semester? If you are drop me an email.

Well no one else said anything

Clarkson won the University Prize for the OSDL Data Center LinuxIniative Tuxmaster’s Invitational. We’ll be getting an 8 way Itanium 2 ES7000 running SLES 9 (and Xen of course) at some point this fall. Todd and Patty’s entry took first and Ed, Jeremy, Justins, and mine took 2nd place. (The PSP rocks by theway). On the contest website, www.tuxmasters.com it looks like they’re gearing up for another contest already.

OLS Day 4

OLS Day 4

After a morning of hacking on random stuff, we went to the sysfs filesystem. Sysfs is a filesystem that is based off of ramfs and is used to display information about the system devices. He went over why the sysfs was created and how he created it. He then explained some of the deficiencies that are in the current implementation of the sysfs. They are currently looking to build a new userspace library for sysfs.

The closing keynote was by Dave Jones, one of Redhat’s kernel leaders. He had some very interesting points on how tools need to be improved in order to make developers much more effective. He showed examples of how Static Code analysis has reduced the number of bugs greatly. He also explained why it is so difficult to regression test the Linux kernel. Jeremy and I are probably going to do tutorials at some point during the semester on doing static code analysis and using code profiling tools. He also had some great ideas about a solution to problems with bugs not getting in the upstream bugzilla’s. I’ll post more on my thoughts on a solution later. Hopefully someone can pick this up for a COSI project this semester.

After the keynote Mark, Sean, Ed and I headed off to dinner before Mark had to catch his flight to England. Tomorrow or next week will be the report on the closing party. Apparently it’s usually pretty wild and out of hand, so this should be very interesting.

OLS 3 (Virtualization Day)

OLS 3 (Virtualization Day)

Went to the Linux Virtualization on Virtual Iron VFE talk. It turned out to basically be and advertisement for Virtual Iron, a company that has written their own virtualization layer that is targeted at corporate data centers. They had some pretty interesting ideas, but they did not seem like they were looking to scale their product up to more than tens of machines and a large portion of their product was closed source with no plans to open source it. I also spent a decent amount of the morning hacking on various little projects with Xen, which should be released shortly.

I was looking forward to going to the talk on Beagle by Jon Trowbridge, but United Airlines screwed up his flight, so he couldn’t make it in, and he couldn’t find someone else to give the talk.

Ed had a pretty amazing comment today. He was surprised at the number of women that are actually attending the conference. There’s probably around ten women here, in a crowd of approximately seven hundred and fiftey developers. Ed’s definitely got a very weird view on women in Computer Science.

In the afternoon the intense Xen presentations started. The first one was presented by Mike Day from IBM on using Xen to Supercharge OS Deployment. He went over how virtual machines make deploying instances of Linux faster because if you create a standard hardware configuration you can automate deployment of your Linux systems. This talk seemed to be geared more towards a system administrator, as he went over several different ways you can normalize your hardware to reduce installation time. One of the problems with reusing filesystems in deployment is that each time it is reused you have to customize parts of the filesystem unless you are using a read only filesystem.

He had some very interesting points on the weaknesses of Xen. These are some of the things that we were starting to address this past Spring with Xenophilia and I am hoping to address even further this fall in some of my work in CS450 and possibly my independent study. Currently there is no mechanism for customizing images as they are deployed, and there is no integration of images with the Xen domain configuration tools. He was thinking about adding scripting support into the domain configuration files to allow the customization of images. They developed a proof of concept, Xen Container Syntax that allowed users to add domain specific configuration information into the domain configuration files. This idea has several problems and is probably going to be reworked from the ground up with support for Xen 3.0. It was written using bash scripting. It’d be interesting if you could use some sort of template or inheritance system to create the domain configuration files. After seeing this presentation I guess I will have to rethink the way I was going to generate customized disk images for when a domain comes up.

The second talk that I attended was a case study of using Usermode Linux to test multiple Linux distributions that was performed by engineers at Intel. It started out with a brief introduction to the LSB. They validated distributions using the runtime tests for the LSB to make sure that the distributions are LSB compliant. He gave a brief introduction to UML, and the guest distribution actually runs as a process in a host. They ran their testing on a 4 Gigahertz Hyperthreaded server with one gigabyte of ram, and two 15,000 RPM SCSI disks.

The next talk was “Testing the Xen Hypervisor and Linux Virtual Machines” by Paul Larson from the LTC. When testing the Hypervisor they were able to reuse many of their tests from testing of the Linux Kernel. They did have to write new tests for the testing of the hypercalls into the hypervisor and they also wrote tests for the tools interface. They created a test interface called Xenfc that is not a systematic test, it randomly generates data to test the hypervisor. This approach has been much more useful in helping developers find corner cases. It was extremely interesting to here some of the ways that they have automated the integration testing of the kernel and hypervisor.

At the Common Virtualization Infrastructure BOF Rik Van Riel discussed some of the new memory management techniques, included page hinting for swapping paging. Problems with over committing pages and having the hypervisor swap the pages to disk was also discussed, but apparently not all architectures can do that at a page granularity level. Currently the hypervisor on s390 has the ability to swap pages. Paravirtualization is also in development at VMWare, and is going to be implemented in their VMWare ESX Server. They are currently working on SMP optimizations, split drivers, and 64 bit support along with a couple of other things. Their system is a non cooperative paravirtualization system, so it’s performance isn’t the best but, it can run transparently.

I also learned of a new addiction Frozen Bubble from Mark Williamson. It’s written in Perl using the SDL module and I’m finding that very scary. There’s also a java port if a game programmed in perl is to scary for you. This is going to make it much more difficult to actually get real work done on xen-get.

The Fedora Foundation is currently being setup after the announcement in the beginning of June. The Foundation is not going to be a subsidiary of Redhat it is probably going to be a Not For Profit Organization with a Board of Directors.

The release schedule was extended due to the non Redhat employees that are now very involved in the release project. They have several big projects on tap for Fedora Core 5 which include:

A rewrite of Anaconda using yum and supporting Xen

A new Fedora Extras build system

A new update notification

Xen 3.0 is possibly on the schedule

A Live CD is a possibility (it has to be created in a sustainable fashion)

GDM early login for FC5

Most of the Stateless Linux changes will be merged for FC5, but it looks like the lab is going to be the test monkeys as their hasn’t been a real deployment yet.