VMware was initially considered a likely candidate to buy at least some of Novell’s IP because of the way the two companies have tightened their alliance in recent years. Last June, VMware said it would standardize its virtual appliances on Novell’s SUSE Linux, which had been modified by Novell a year earlier to run faster on VMware virtual machines.

After the initial reports of talks between VMware and Novell, some users said such a deal would fit VMware’s pattern of expansion into a software stack beyond the hypervisor, which has included acquisitions of SpringSource and Zimbra as well as a partnership with Salesforce.com.

Reuters followed the WSJ story with a report Sept. 22 which said those talks had stalled, citing a “valuation gap” between Novell and its suitors when it came to products outside the company’s SUSE Linux operating system unit. Still, some industry experts expressed hope at that time that VMware would evaluate acquiring at least some of Novell’s virtualization management IP, particularly within its PlateSpin portfolio.

Meanwhle, “as many as 20 companies initially expressed interest in Novell,” according to the September 15 WSJ report.  This detail takes on a new wrinkle when combined with a line in Novell’s press release about the acquisition today, which says that as part of the $2.2 billion deal, some of its IP will be sold to “CPTN Holdings LLC, a consortium of technology companies organized by Microsoft Corporation, for $450 million in cash.”

We know now that VMware did not become one of Novell’s principal buyers, but if you connect all the dots, it is possible VMware took a look at Novell two months ago, and passed. Also, very little is known about CPTN Holdings at this point, whether it’s what specific patents were bought or which technology companies are included. Thus, VMware at least theoretically could have gained access to some Novell IP today, in a roundabout way (which would also add new meaning to the term ‘coopetition’), even if it didn’t pick up the whole enchilada.

On the other hand, if VMware isn’t a part of the consortium, it will be interesting to see if Microsoft and friends are able to use these patents to disrupt the market, given VMware’s past coziness with Novell and SUSE.

The paper has just reported that Novell “is in advanced talks” to sell its SUSE Linux operating system business to VMware and the rest of its assets to Attachmate, a “private-equity backed software company.” Contrary to what was said in yesterday’s New York Post report (which did not name VMware, only a “strategic buyer”), the Journal says the sides have not finalized any agreements and that VMware and Novell “still differ on valuation.” The Journal’s report is attributed to “people familiar with the matter.”

After yesterday’s Post report, I wrote a blog sizing up all of Novell’s potential suitors, and VMware was at the top of the list. Here’s what I said about the potential VMware/Novell acquisition:

After VMware announced its exclusive virtual appliance deal with Novell in June, the acquisition rumors started circling. That has VMware as the favorite among many observers. … But there are a few major points that don’t make VMware a sure bet. Most importantly, Novell doesn’t fit the traditional VMware acquisition strategy, which is to pick up smaller growth companies in an “opportunistic” fashion. Plus, some VMware execs have privately intimated that the virtual appliance deal means much more to Novell than it does to VMware.

Now let’s take a slightly deeper look at a potential VMware/Novell acquisition. The virtual appliance deal put the spotlight on closer integration between the virtualization layer and the OS layer. But virtual appliances are just one way to combine the two. By acquiring Novell SUSE Linux, VMware would have significantly more options — like, to bundle its hypervisor with the OS, a la Microsoft. That could help VMware in the SMB market, where it’s facing its biggest challenge from Hyper-V.

More importantly, acquiring Novell SUSE Linux would completely alter the face of VMware. With VMware’s recent acquisitions — SpringSource and Zimbra, in particular — there has been a lot of talk about the company moving beyond virtualization. But most people still viewed VMware as a virtualization vendor that just happened to own a development platform and an email provider. That wouldn’t be the case anymore. VMware would establish itself as a full-blown infrastructure and software vendor.

Is that a good thing? Depends who you ask. There are many who viewed VMware’s acquisitions of Zimbra and, to a lesser extent, SpringSource as needless distractions. “Why does VMware need an email solution?” these people asked. “With the challenge from Hyper-V, shouldn’t VMware be focusing strictly on virtualization?”

Then there are others who say the challenge from Hyper-V — and the commoditization of the hypervisor in general — is the exact reason why VMware should be expanding beyond virtualization. Virtualization alone won’t keep VMware growing; offering a full infrastructure stack will. And an OS is a major part of that equation.

Which side are you on? Let us know in the comments.

Novell, the “strategic buyer” and the private equity firm have all agreed to the deal and will sign it in three or four weeks, according to the Post. In the meantime, let the speculation begin!

Who’s buying Novell? Let’s take a look at some possible candidates and the odds they’re the “strategic buyer”:

VMware (3 to 1): After VMware announced its exclusive virtual appliance deal with Novell in June, the acquisition rumors started circling. That has VMware as the favorite among many observers, including Gartner’s Chris Wolf, who says it’s not a coincidence that VMware is training Novell’s SUSE Linux Enterprise Server organization as part of that deal.

But there are a few major points that don’t make VMware a sure bet. Most importantly, Novell doesn’t fit the traditional VMware  acquisition strategy, which is to pick up smaller growth companies in an “opportunistic” fashion. Plus, some VMware execs have privately intimated that the virtual appliance deal means much more to Novell than it does to VMware.

Oracle (7 to 1): Unlike VMware, Oracle definitely doesn’t shy away from large acquisitions of major companies. Larry Ellison and co. have made it clear they want to control the entire IT stack, and while the Sun Microsystems deal gave them pretty much everything they need to meet that goal, Novell’s SLES would definitely provide a boost when it comes to the server OS. (Novell has about 16% of the market, and Oracle Enterprise Linux another 11%, according to SearchDataCenter.com’s 2010 Data Center Purchasing Intentions Survey. Add that to the 27% market share held by Solaris, and Oracle would control more than half of the server OS market.)

Also, Oracle’s virtualization strategy focuses heavily on VM templates; adding Novell to the fold would strengthen those offerings, while at the same time weakening VMware’s virtual appliances.

Red Hat (9 to 1): You’re the market leader. Your top competitor is vulnerable and for sale. Why wouldn’t you swoop in? No, I’m not talking about Vince McMahon’s WWF and Ted Turner’s WCW in 2001, I’m talking about Red Hat and Novell today. There have been no indications Red Hat is interested — execs may be content to see Novell split up and see what happens under new ownership — but the deal would make a lot of sense.

Citrix (12 to 1): There’s a clear trend toward tighter integration between the OS and the virtualization layer. Hyper-V comes with Windows Server, and in the ultimate example, KVM is part of the Linux kernel. Acquiring Novell would help Citrix out a lot in this area. The interesting subtext to this speculation is the feud between Citrix and Red Hat: As Red Hat tries to woo Xen customers to KVM, Citrix CTO Simon Crosby has suggested that Red Hat Enterprise Linux customers should move to Oracle or SLES.

Microsoft (15 to 1): These two companies have a long history. Microsoft crushed Novell’s NetWare business in the ’90s, and there was significant proprietary-vs.-open source friction for much of the past decade. But after their 2006 patent agreement, the two companies have grown closer, and now acquisition speculation pops up occasionally. The Novell-VMware virtual appliance deal sure wouldn’t last long if this acquisition happened.

Intel (20 to 1): Just a few months ago, Intel wouldn’t belong on this list. But after acquiring McAfee and stating its intentions to offer a “complete stack,” the chip manufacturer can’t be ruled out. A server OS is a pretty important component of a “complete stack,” after all.

Field (6 to 1): There are plenty of other companies that would make sense as the “strategic buyer,” from IBM to Amazon. And as some of the recent head-scratching IT acquisitions have shown, no company can be completely ruled out. Guess we’ll just have to wait three or four weeks to find out for sure.

One of the big questions around the announcement: Why Novell? As News Director Alex Barrett wrote in her story, “Red Hat still leads Novell in terms of Linux market share by a wide margin, leading some to wonder why VMware didn’t partner with that company instead.”

VMware isn’t the only virtualization vendor to spurn Red Hat lately. In fact, this latest news makes you wonder if Red Hat’s virtualization strategy is backfiring.

VMware said it chose Novell because of SUSE’s broad ISV support, particularly with SAP. But there’s more to it than that. As Barrett pointed out, “Red Hat has been pursuing its own virtualization strategy — first with Xen, and more recently with KVM — putting it directly in VMware’s crosshairs.”

The virtualization tide first started to turn against Red Hat back in April, when Citrix went on the attack. (Citrix, with its flag firmly planted in Xen soil, is especially threatened by Red Hat’s KVM push.) So CTO Simon Crosby countered by encouraging Red Hat Enterprise Linux (RHEL) customers to move to SLES or Oracle Enterprise Linux.

I wrote at the time that Red Hat can’t afford to turn off its core customers — admins and engineers who work with RHEL — in its pursuit of the virtualization market. Well, the company is already turning off the core vendors that it may have been better off partnering with instead. Customers may not be far behind.

It’s a serious issue for the company as its annual Red Hat Summit approaches. The show begins Tuesday, and I’ll be there covering this topic in depth.

The Xen vs. KVM debate is not new, but it has definitely been picking up steam as of late. Red Hat Enterprise Linux 6 has given Xen the “let’s just be friends” speech and moved in with KVM. Several major hosting providers are switching their platforms from Xen to KVM. And many Linux diehards say Xen is a huge pain to manage.

The battle lines are drawn, and the fate of the open source virtualization market hangs in the balance. (How’s that for overdramatic? I think I’ve been watching too many “Lost” commercials.)

Anyway, the ramifications of this potential shift may also affect the open source OS market. Leading the charge on this side of the battle is Citrix CTO Simon Crosby, the co-founder of XenSource. Red Hat is trying to move customers off Xen, and now he’s trying to move customers off Red Hat.

“If you approach your virtualized world with a Linux/RHEL based mindset, then I recommend you consider switching to Oracle Enterprise Linux,” he blogged last week. “It is a superior, enterprise class version of RHEL. … Alternatively, if you’re wary of giving Larry more control than he already has over your environment, Novell SUSE Linux offers a superb enterprise Linux platform.” (Note that Crosby linked Oracle CEO Larry Ellison’s name to a picture of his massive yacht.)

I’ve previously said that Red Hat has nothing to lose by switching from Xen to KVM. I meant that in terms of Red Hat’s standing in the virtualization market. VMware, Microsoft and Citrix are way ahead of Red Hat there. (And as Crosby blogged, “Having failed to capitalize on Xen, Red Hat needs a ‘differentiated’ story in virtualization in order to regain credibility.”)

But Red Hat has a ton to lose in the enterprise Linux server OS market, where it’s the leader. Sure, the Linux community may be in love with KVM, but Red Hat butters its bread thanks to the IT admins and systems engineers who work with RHEL. The company can’t afford to turn off these core customers in its pursuit of the virtualization market.

The company is developing a KVM hypervisor called AlacrityVM, as virtualization.info points out. The move follows in the footsteps of Red Hat, Novell’s open source rival, which moved from Xen to KVM with its latest release, Red Hat Enterprise Linux 5.4.

KVM is still a relatively unproven enterprise technology with a very small user base. Its biggest advantage over Xen, the leader in Linux virtualization, is that it is built into the Linux kernel. And that’s just not enough of a reason to switch for most people.

The proprietary virtualization platforms, VMware and Hyper-V, are far and away the market leaders. Behind them are the Xen platforms, led by Citrix XenServer but also including Oracle VM and others.

Red Hat and Novell are even further behind. They really have nothing to lose, so they both can afford to take a shot on KVM. If the technology catches on, they can ride the wave and prosper. If not, they won’t be much worse off.

For more on Linux virtualization trends, check out this Xen vs. KVM face-off between experts Andi Mann and Sander van Vugt.