Posted by: Akutz
Andrew Kutz, Linux and virtualization, VMware, Xen
How many of you believe that Apple’s 1.1.1 iPhone update accidentally bricked modded iPhones? Personally, I try to air on the side of optimism, but there are certainly many people out there that think Apple intentionally went after those individuals who took it upon themselves to jailbreak and unlock their shiny gadget-of-the-moment.
Here we are again, not even a month later, and the new Linux Kernel, 2.6.23 was released on 2007/10/09. The latest product of the world’s greatest hackers includes a bevy of new features, including increased support for Xen and KVM and two open source virtualization solutions. Users of those products are probably very happy today, eagerly awaiting the adoption of the new kernel by their favorite distribution in order to take advantage of the increased guest support that comes with it.
VMware Server users on the other hand are getting the proverbial shaft. Kernel 2.6.23 has one MAJOR change and one minor change that completely break VMware Server.
For purposes of dramatic effect, I will detail the minor change first. VMware Server inserts a driver module into the kernel called vmnet. It provides magical networking gnomes that help shuffle bits in and out of VMs to the wide world of webs. In one of its source files, driver.c on line 522, the vmnet driver makes a function call to “unregister_chrdev”, a function defined in the Kernel source file “fs/char_dev.c”. Prior to Kernel 2.6.23 the function “unregister_chrdev” returned an integer value; a return value that the vmnet driver keys on in order to determine whether or not to issue a warning. Kernel 2.6.23 changes the function signature of “unregister_chrdev” to return void instead of and integer. This really hoses the vmnet module source file since it expects an integer value to be returned, and thus the vmnet module will not compile when the “vmware-config.pl” script is run. Luckily there is an easy fix. It seems that the function “unregister_chrdev” has actually returned a value of “0″ despite what transpires in the function as far back as 2.6.20, a Kernel that VMware Server runs fine on. Thus the easy fix is to just edit the vmnet driver.c source file and re-run the VMware Server configuration script.
That is the minor problem that the new Kernel creates.
The major problem is a bit more cumbersome, since the fix involves either redacting a change that Linus (Torvalds) has approved for the 2.6.23 Kernel or lying and declaring that the vmmon module is GPL licensed.
But I’m getting ahead of myself. Let’s start at the beginning. A memory structure called mm_struct is defined in a Linux Kernel header file “linux/sched.h”. Prior to 2.6.23 this structure included a field called “dumpable” that would determine how memory was dumped, securely or not. Kernel 2.6.23 removes this field and lets two functions defined in “fs/exec.c” take its place: set_dumpable and get_dumpable. VMware Server uses the dumpable property in its memory management module vmmon: in the file driver.c to be exact. Since the dumpable property is no longer in the 2.6.23 kernel the vmmon module will not compile.
One might think that a quick fix would be to simply edit the vmmon source file to use the new set_dumpable function. In fact, this action will result in a vmmon module that compiles; however, it will not insert into the Kernel, and an error will occur that says the module contains an unknown symbol. A quick check of dmesg reveals that the unknown symbol is indeed set_dumpable. ‘What, what, whattttt,” you say. But the set_dumpable symbol IS in the kernel. That is verifiable by peeking in /proc/kallsyms.
Heh, heh. Hold on to your seats. This is where it gets fun.
The function set_dumpable is exported in 2.6.23 with the new EXPORT_SYMBOL_GPL, meaning that only modules that are GPL licensed can use it. More can be read about this decision on the Kernel mailing list.
VMware Server’s vmmon module cannot use set_dumpable because it is not GPL licensed. There are two solutions to this problem. The first solution is to edit the Kernel source file “fs/exec.c” so that “set_dumpable” is exported with EXPORT_SYMBOL instead of EXPORT_SYMBOL_GPL and compile a custom Kernel. Then, the vmmon module source file “driver.c” still needs to be edited such that the “dumpable” property is no longer used in favor of “set_dumpable”. The second solution is to edit the vmmon module source file the same way as in the first solution, but also using the macro “MODULE_LICENSE” to indicate that the vmmon module is licensed under the GPL.
Neither solution is nice, because the first one involves maintaining a custom Kernel and custom vmmon module, and the second solution involves changing the vmmon module license without permission. A long-term solution is needed where either the Kernel developers change set_dumpable to be exported out from underneath the aegis of the GPL, or VMware could license the vmmon module under the GPL or create some type of GPL-compatible shim module that in turn calls the proprietary code in vmmon.
Perhaps most interesting of all is the timing. The same Kernel that provides extended support for Xen and KVM also breaks VMware Server. Coincidence? Like I said, I try to err on the side of optimism. How about you?