VMware VirtualCenter Management Webservices won't start - vCenter 5.1
I ran into an interesting issue at the tail end of a project not to long ago. Updated a customer's vCenter and his Hosts from 4.1 to 5.1, which amazingly enough went off without a hitch. We were about to head out when I realized that the Management Webservices weren't started. Although not a big deal I definitely wanted it sorted out so that he was able to view all his performance graphs and whatnot within the vSphere client. If you are too short on time for my horror story, just read the TLDR bullets at the end of this post.
I went in to services.msc and tried to start the service, but it immediately stopped... strange. I decided to see what event log had to say. As usual, I had a very not-so-helpful error message staring back at me:
So I did some Googling. Turns out that the Webservices are a based Java application, which contains a core component known as . Neither the underpinnings of Tomcat nor Catalina were of major importance to us, what was relevant is that the Catalina logs might give us some insight as to where the service was bombing out.
After some digging I landed in the C:\ProgramData\VMware\VMware VirtualCenter\Logs directory where I found, wouldn't you know it a bunch of files named catalina.YYYY-MM-DD.log. So I popped open the most recently modified log and scrolled to the end.
As with most Java logs there was a bunch of stack trace type stuff that I can't personally read in it's entirety, but some of the lines did stick out to me, of particular interest were the following lines:
Failed to initialize component [Connector[HTTP/1.1-8080]] -** This told me the port it was trying to use - 8080 - which is also listed in this**
Caused by: java.net.BindException: Address already in use: JVM_Bind - Aha! Something else is using the port!
I thought I had it, even with the service stopped, something was on the port, I verified with netstat -a
Victory was within my grasp, now it was easy. Just download TCPView
Unfortunately for me I had never done this, I tried a couple of command line tools and an app called JConsole but apparently none of these work for Tomcat apps because it uses a non-default home directory. After more digging than I would like to admit, I found out you can do it with my old friend Process Explorer
The management console was no longer needed on the vCenter server, so we uninstalled it, and the Management Webservices started right up. Hopefully this little story is helpful here are a few bullet points for the TLDR crowd:
- If the Management Webservices wont start, use TCPView to look for something blocking port 8080
- If whatever is blocking port 8080 is Java based, use Process Explorer to find the jar file being launched
- Don't install the Compellent console on your vCenter server, better yet, just Buy EMC