Server 2008 is Experimental for ESX 3.5, but it is fully supported by ESX 3.5 Update 2. Here’s the OS compatibility link.
Microsoft doesn’t support clustering under ESX. However I’ve done it in production, and it works fine for me. The catch is that Server 2008 doesn’t support clustering under SCSI disks which is what VMware presents, so you have to present iSCSI disks directly to the guest OSs.