Thursday, August 15, 2013

SQLing my way around the "Authorization Exception" Domain Logon Error in vSphere 5.1

Here's the situation. A customer had some AD issues related to botched DCs that were incorrectly P2V'd for whatever reason. As a result the customer had to open a trouble ticket with Microsoft and forcibly remove the DCs from AD as DCPROMO was not working properly.

All then seemed to be well, there were some residual DNS and DHCP quirks to be worked out but things were improving. When suddenly any attempt to log into vCenter resulted in an "Authorization Exception" error. This is vSphere 5.1, which of course means the dreaded Single Sign On. Here is one (of many) problems with VMware's SSO - it statically binds to two domain controllers, and we had just removed them. The user was able to log on with local accounts, but they didn't have the permissions needed to make the change outlined here that would have bound two new domain controllers. I tried a hosts file redirecting the hostnames for the old DCs to the IPs of a new DC on the SSO server but unfortunately that didn't work.

 To compound the problem the user had forgotten his SSO admin@System-Domain password and all of the password reset procedures require you to know the original SSO admin password. At this point it was pretty much looking like a rebuild at this point, not fun, especially considering we thought the problems had just ended. Then I found this thread. Basically it told me a few things:


  1. C:\Program Files\VMware\Infrastructure\SSOServer\webapps\ims\WEB-INF\classes\krb5.conf can be opened in notepad and lists the DCs used to authenticate currently.
  2. Editing that file does nothing, its just for reference.
  3. The actual config is stored in the RSA database in the "IMS_CONFIG_VALUE" table.
Now we were getting somewhere. I threw on my DBA hat for a few moments and downloaded SSMS then logged into the SSO database, luckily the domain admin account was a sa. If you are not lucky, see here for how you can recover sa rights using single user mode, and don't you dare make fun of Pinal Dave, that guy is a freakin' genius. Anyhow, I got into the db and found the values for the LDAP servers that SSO was using, there are 4 columns we need to update here are the queries, keep in mind my db was named rsa and my schema was the default dbo schema, wrap them in transaction statements if you are a good DBA, on this day I was not.


UPDATE rsa.dbo.IMS_CONFIG_VALUE
SET VALUE = 'ldap://newdc1name.doman.local:3268'
WHERE VALUE = ''ldap://olddc1name.doman.local:3268'

--the above should update two rows


UPDATE rsa.dbo.IMS_CONFIG_VALUE
SET VALUE = 'ldap://newdc2name.doman.local:3268'
WHERE VALUE = ''ldap://olddc2name.doman.local:3268'

--the above should update two rows

Then just start => run => services.msc and restart the vCenter Single Sign On Service and bam, you should be able to authenticate as a domain user again. Down side to this situation is the customer still does not know his SSO admin password. I have a suspicion I could take the hash of a known password out of another SSO db and update the appropriate tables (there is a principal table and a password history table as far as I can see) with it, then log on using that password. I am going to try it in the lab and there may be a blog post to follow with the unofficial method for reseting a SSO password.

Thanks for reading!

1 comment: