Is this local storage or shared storage? If it is shared storage make sure all hosts that have the datastore configured have it removed from their configuration. For both make sure all VMs are shutdown. If this is local storage then after you have the VMs shutdown (or deleted) try restarting the host to remove any disk locks and then remove the volume.
ESX 3.5.0 has a bug that will cause the delete of the datastore to fail. You can see the error if you look in /var/log/vmware/hostd-*.log and see something like:
<pre>Error : Error: The partition table on /vmfs/devices/disks/vml…. is inconsistent.
the most likely reason is that Linux detected the BIOS geometry for /vmfs/devices/disks/vml….incorrectly. GNU Parted suspects the real geometry should be 46208/64/32 (not 5890/255/63).</pre>
You can just directly reformat the datastore without deleting it and creating it.
<b>This will still DESTROY everything on the datastore:</b>
<pre>vmkfstools -C vmfs3 -b <i>blocksize</i> -S <i>datastore_name device</i></pre>
<pre>vmkfstools -C vmfs3 -b 2m -S storage1 /vmfs/devices/disks/vmhba1:0:0:3</pre>