WTF!? - Powering on a virtual machine fails with the error: A general system error occurred

This can by far be one of the most frustrating error messages you will run across in a vSphere environment.  I was at a customer site yesterday and the customer ran across it while I was working through some other things.  He asked if I could fix it for him.  :)

I immedieatly dove into the issue and started looking at the problem he was having and the error message was very generalized in nature.  "Powering on a virtual machine fails with the error: A general system error occurred"  I wasn't able to change any settings on the VM or power it up.  I was able to remove it from inventory and re-add it no problem.  So I Googled the error message and ran across this KB article: Link.

The issue sort of pointed to a corrupt VMX file, so I attempted to create a new VM and point it at the disks from the old VM folder.  The problem with that was the fact that while I could browse the datastore and see the VMDK files, I wasn't able to "see" them when creating a VM.  I was able to download the files from the datastore but, wait for it, I was not able to upload anything.  I know, it seems like a read only datastore right?  Well it was, but vSphere didn't think it was.  I wasn't able to unmount it from the GUI, but was able to via command line.  I could even remount it after I had removed it.

So it turns out that the customer had changed the VMKernel IP address that he was using to access the datastores on his VNX 5300.  If you've never used a VNX before, if a datastore has been mapped previously then vSphere will still be able to see the datastore even though it hasn't been given explicit permissions to do so.  It essentially is a read only file system even though vSphere thinks it's not.  As soon as you try and power on your VMs (which writes files to the datastore) you see the general error message from the KB article.  Fun huh?

Hopefully this helps some of you to quicker resolution should you run into this.