Tuesday, March 25, 2014

Varrow Madness 2014 - Auto Deploy, Image Profiles, and Host Profiles Slide Deck

Another Varrow Madness is behind us, and the feeling is bittersweet. So much fun and such great content, but I would be lying if I didn't say that we all look forward to it every year. It was a huge success with a great speaker, Shawn Achor, check out his Ted Talk if you havent. Also, we had some AMAZING speakers including but not limited to Scott Lowe, Chris Colotti, Jim French, Mike Foley, Chad Sakac, Rick Scherer, and of course, The Jason Nash. In addition to these big names there were a host of other industry experts from both inside and outside of Varrow that also contributed amazing content. I attended what sessions I could given the need for me to help out with the conference and I was blown away by the quality of the content. As always, highly impressed by what Varrow has done given the size of our organization.

Anyhow... While at madness I did a session on deploying stateless hosts with Auto Deploy, Image Builder, and Host Profiles. It covers a few of the VCAP-DCA objectives so I figured I would post the deck here. If I have time later, I will make a video of the demo portion and post it as well. Hope its helpful!

Friday, February 28, 2014

PernixData FVP 1.5 Reinstall

One of my favorite things to do is try to squeeze as much performance out of my home lab without buying more expensive gear or generating excessive amounts of heat since I already have space and cooling issues in my 15x15 office at home (yay townhouse). To that end, PernixData's FVP seemed like a perfect fit. These guys are awesome and one of their Engineers, Charlie Gautreaux has been really helpful as far as aiding me in getting this set up in our Greensboro lab and my home lab.

Smarter people than me (for instance, Nash and Wahl) have already blogged in depth on the architecture and performance gains offered by FVP so I wont dive too deep here. Suffice it to say the technology leverages local flash drives in your hosts as an intelligent read/write cache to accelerate storage performance. There is also a write back option that will mirror your writes to other hosts before acknowledgement rather than flushing to the datastore immediately further improving performance and ensuring data integrity if you lose a host.

Instead, what I am going to blog here real quick is an uninstall/reinstall of 1.5. please note there is an upgrade process for the management server, but I prefer to just dig it all out and start fresh. Refer to Pernix's documentation for an in place upgrade. This will give you an idea of how easy the product is to use in addition to the massive performance gains it offers up. With that, lets dive in.

Flash Cluster Removal

Before we start messing around with the managment server I am going to disable the acceleration on my datastore, remove the flash devices from the cluster, and delete the cluster.

So lets hop into our flash cluster from the web client and start working:

We will need to add the security exception to manage FVP since I am using Chrome:

And now we can mange our flash cluster, first I am going to remove acceleration from my iSCSI datastore:

Next, I am going to remove my flash devices from the cluster:

Once that is all done we can delete the flash cluster without issue, you may be able to delete the flash cluster outright and have it clean up for you, but I like to control all these mechanics. Also ignore the fact that my screenshot shows 3 flash devices in the cluster, you should see 0. I was just too lazy to take another screenshot:

This should be the end result:

Host Extension Removal

Next we need to uninstall the host extension from each of the hosts, so go ahead and throw one in maintenance mode, SSH into the host, then remove the host extension using Pernix's uninstall script: 

Once this is done, reboot the host, exit maintenance mode, vMotion some VMs, wash rinse and repeat.

Management Server Uninstall

Now that all the host extensions are uninstalled, we can uninstall and reinstall the management server. So lets start the uninstall:

Management Server Reinstall

Sweet, now we can install the latest management server, download the executable, and kick it off:

Accept the EULA:

Select an install type, I've always done complete:

Installation location:

Point it at your vCenter server and give it credentials to connect, the checkbox is for fairly niche use cases where you need to run the mgmt server as local admin rather than a domain account:

Give it credentials to the DB, I am am using my sql SA account b/c it is a lab, this is not a good security practice. Windows integrated auth works fine too:

Select the name used to identify this management server, I am using local hostname:

Away we go:

All set!

Host Extension Reinstall

First stick one of your hosts in maintenance mode. Then, upload your new host extension to one of your datastores, and kick off the extension installation using the following command:

esxcli software vib install -d /<PATH TO VIB>/PD-host-extension-version# --no-sig-check 

Where path to vib is the path to the folder location of the vib on your datastore and version# is the appropriate version# string for your extension. No sig check is necessary for now but eventually you can do this all through VUM once it is signed.

Once this is done, you can pull that host out of maintenance mode, and move on to the next. Continue until all of your hosts have the new extension on them.

Flash Cluster Recreation

Now that the management console and the host extensions are reinstalled we can rebuild our flash cluster. You will need to log out of, then back into the vSphere web client in order to get started.

Once logged back in you should see Flash Clusters listed under your vCenter server:

Click on the flash cluster and then click the add icon to create a new flash cluster:

Give it an appropriate name and select the cluster you are working with:

Once it's creation is complete select it, then head to the manage tab, then flash devices, and click add devices to bind your host flash devices to the FVP cluster:

Next, Select datastores/vms and add either the datastores you want to accelerate or the specific VMs you want to accelerate, make sure you choose write back if you want peak performance, and select the write redundancy you are comfortable with. I usually power off one of my hosts and this is a lab so I am only doing 1 network flash device for redundancy:

Voila! In short order you will be able to use the monitor tab to see your individual VMs consuming flash cache, make sure that your write policy matches what you had selected earlier:

Hopefully this is helpful, FVP is awesome and I really hope it gains some real traction out there. Hit me on twitter if you have questions!

Thursday, February 27, 2014

PEX Recap - Time for VCDX!

On the 14th I returned from VMware PEX in San Francisco, California and man, what a trip. First of all if you haven't been to San Francisco, the city itself is worth the trip. Extremely hilly, but full of delicious food, great coffee, wine, the arts, you name it. I consider myself a bit of a pizza snob being from very near New Haven, CT which is basically Mecca for the pizza lover. However, my friend and co-worker Tom Queen outshines me in this discipline to an exponential degree, and he had nothing but positive things to say about the pizza in San Francisco. I agree with him 100%, check out 00 Pizza if you get the chance.

VMware PEX

As much as I wish it was, my trip to San Francisco was not for pizza alone. The main purpose of my trip was VMware Partner Exchange, a conference dedicated to VMware partners where they share roadmap information, conduct technical breakout sessions, offer Hands on Labs, and give us as engineers the opportunity to network with some really sharp folks. I am extremely grateful that my employer afforded me such a great opportunity.

Mirage Bootcamp

The first couple days I spent in a Horizon Mirage boot camp, such a great product and fairly simple to use. Basically it decouples the base OS from the Application and User data via a layering technique, then uses an agent to apply the appropriate layers to the corresponding VM or physical workstation. The tool can then be used as a backup and recovery mechanism, even supporting file level restores initiated by the user via a web portal. Additionally, Mirage offers a smooth migration path to Windows 7 by allowing the administrator to apply a new OS layer and then reapply the application and user data layers back on top of the OS. There are already some integration points with Horizon View, and I foresee this increasing eventually giving View admins real flexibility in how they handle user and application data as well as how they manage their base images. Mirage makes heavy use of compression and block de-duplication, meaning that after the initial sync bandwidth utilization back to the data center is kept to a minimum.

General Sessions

Monday I spent in an all day session by EMC. Lots of great roadmap info in here that unfortunately I cannot share due to NDA. Suffice it to say that the Software Defined Datacenter is where things are going. We all need to get our automation hats on and prepare to interact with technology in a very different way. Speaking of software defined, I also sat in on some great sessions on VMware NSX and VSAN. The NSX course was introductory but it's great to see the focus shifting away from designing for the network and towards designing for the application. NSX is going to offer a lot of opportunity for rapid network configuration and application deployment. This includes stuff like load balancing, firewalling, etc. Which has traditionally been a bit of a hurdle for application developers to get over when spinning up new instances. VSAN is also very cool, leveraging local host storage (SSD and Spinning Disk) to create an elastic datastore that can scale out or scale up. Really, looking forward to getting my hands deeper into these technologies.

On top of the technical sessions I sat in on some really good process oriented sessions. The first of which was a Mirage based session focused on where Mirage compliments competing technologies and where it competes head to head. The other session on this track that I attended was an EUC experts panel focused on View. The big takeaway here was that you need to take a strategic approach to EUC engagements and sometimes try to get the customer to do things that they may not want to do initially in order to deliver the best overall solution. Plan and Design sessions as well as Environmental Assessments are crucial to a successful VDI deployment because there are so many factors that go into the end user experience. Aside from properly sizing the environment from a compute, network, and storage perspective, we have to ensure that the proper applications installed and working, that there is no extra bloat in the base disk, and that we are designing both for the administrator and for the end user. I have pages of notes on this stuff that I won't bore you with, but it's a hot topic and I really enjoyed the session.


On top of all the great speakers and sessions VMware offers up the opportunity to take a discounted exam at PEX so I figured I would take a go at the VCAP5-DCD. I am in a Master's program focused on IT management, have a good amount of vSphere experience including my VCAP5-DCA, had spent a bit of time watching Scott Lowe's course on the DCD, as well as watched a few of the #vBrownBag podcasts on the topic but really hadn't put any real time into studying or preparing. Well, imagine my surprise when I actually passed the thing, and what a bear it was. This exam is not particularly fun like the DCA exam. It's 100 questions focused around translating business requirements into VMware solutions. If you are going to take this exam you will definitely want to tap some of the resources I mentioned prior, as well as possibly attending one of the VMware design workshops if you are able. Stuff like RTO, RPO, Constraints, Risks, Requirements, Assumptions, Resource Pool sizing, LAN and SAN architectures, CPU and Memory sizing should all be second nature by the time you take this exam if you want any chance at passing it. Also be sure to watch the VCAP-DCD Exam Demo on myLearn ahead of time because the design tool can be pretty confusing at first glance.

What's Next?

I found that I was asking myself this question a lot after getting the DCD, after all that was my next goal and I wasn't really mentally prepared for actually passing it this time around, I was planning on getting a feel for the exam then retaking it back home in a few weeks. Given the progress and momentum I have in the VMware Data Center cert track right now, the next logical step is VCDX. Just typing at this point seems surreal, Derek Seaman said it best in his post describing his 180 day VCDX journey, "...I mean, those people are book authors, world renowned bloggers, and levitate at will, right?" In all seriousness, it's a very intimidating prospect, but I think I am ready. Right now I am searching for a design opportunity as I do not have a past design that I feel will meet the submission requirements. Once I find a good fit, I will begin the design process, and hopefully defend in October. I have watched the vBrownBag bootcamp series, picked up the required reading materials, and to be frank, I am pretty pumped. Lets see where this roller coaster goes...

Wednesday, February 19, 2014

vSphere 5.5 Web Client Console - Works in Windows not in OSX - vCenter Storage Appliance

Quick post cause my googling didn't point me in the right direction but as usual Jason Nash did.

Had an issue with the vSphere 5.5 web client on the VCSA in my lab where it would launch from a Windows machine but not from an OSX machine even with the plugin installed on the Mac. Here's what I saw when it was broken.

Turns out it's am issue with the environment variables for JAVA_HOME on the VCSA. This KB article walks you through fixing it, but doesn't mention the case where it works in Windows but not in OSX so it can be a little misleading. Once you restart the web client and log back in you will be able to use the HTML5 client on OSX in all it's glory.

Hopefully this helps people find the fix they are looking for. Thanks for reading!

Monday, February 3, 2014

My VCAP-DCA experience

On December 3rd of last year I took the VMware Certified Advanced Professional 5 - Data Center Administration exam. For those of you that know me, I thought for sure I had failed it for sure. I didn't finish all the questions, I wasn't 100% sure I had done the ones I did finish completely and correctly. Essentially, I had moved on and assumed I was going to have to retest. On December 12th I received an email notification that I had passed. Needless to say I was ecstatic and so was the rest of my office because they no longer had to listen to me drone on about how I was sure I had failed. I guess I will go ahead and do the obligatory logo here so lets get that out of the way, boomshakalaka!

As hard as it may be to believe, the purpose of this post isn't for me to gloat. I am actually somewhat disappointed in the fact that I didn't knock this one out of the park, I sort of scraped by. Instead, this post is here to advise those of you planning on taking the VCAP5-DCA on what approach to take when preparing for and when taking the test. Hopefully I'll be able to prevent a couple of you from going through the stress and anxiety that I went through right after walking out of Pearson Professional Center in Winston-Salem.

About the Exam

First things first, nothing I am posting here is going to be direct information off of the exam. I work for a VMware partner and posting exam content is most definitely against the terms and conditions of the test. What I can tell you about is strategies that I feel will help your prepare for the exam, as well as those that will help you when testing.

The exam consists of 26 tasks that you have to accomplish in three and a half hours. You are presented with an environment that will allow you to access a few different servers via RDP as well as putty and the vSphere client so you have all the tools of the trade. Additionally you have access to all of the documentation listed in the ESXi and vCenter section as well as the Command Line Interfaces section here: https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html (as of this posting the VCAP is still using 5.0 but that is likely to change very soon). Lastly you have the tasks window that you can toggle back and forth to. This window allows you to navigate forwards and backwards through the tasks you are to accomplish. Essentially what you are doing here is sitting in the seat of a Sr. vSphere Admin who has a bunch of change tickets pending, which is what I believe makes this exam such an excellent test of practical skills.

Preparing for the Exam

There are so many good resources out the that you can leverage to prepare for this exam, however as with any cert exam the best place to start is with the blueprint. Currently you can find the VCAP5-DCA blueprint here. This brings me to my first piece of advice going into this exam. Know everything on that blueprint, no joke. There are a lot of exams out there where you can scrape by knowing the major components or features, this is not one of them. This test is built to test your ability to implement an end-to-end configuration of a vSphere environment, and a lot of the configuration later in the exam is contingent upon your ability to complete the initial configuration first.

Reading the blueprint obviously isnt that much fun, so here are some folks who have done the legwork for you and put the blueprint into easy to reference guides:

I personally used Josh's and Chris' guides and they were of immense help.

Ok so great, thats what to study, but how do I study? Well first if you haven't already get a Pluralsight subscription and check out Jason Nash's Course as well as his blog. This course is a full fledged walk of the entire blueprint, and if you go through it all and feel like you could teach the course, you are probably good to go. Additionally, the vBrownBag series is an awesome podcast that has subject matter experts from nearly every area within VMware present on various topics as guest speakers. They even run a specific series dedicated to prepping for the VCAP-DCA, definitely check this out. When you really feel ready, hit Josh Andrews up on twitter and see if he will let you use his practice environment. If you follow my advice, you will be ready the day of.

Taking the Exam

I have quite a few tips for you here, I am lazy so incoming bulleted list:
  • Get some rest if you can, I didn't and I regretted it
  • Don't use the vSphere client from the launcher, RDP to the vCenter server and use it (EDIT: Apparently this is not the consensus, in fact most people have found the opposite, including people much smarter than myself, test it for yourself!) 
  • esxcli command list - nuff said
  • PowerCLI can be a big help if you know how to script your way through the scenarios
  • Don't get bound up in the documentation, if you dont know how do to something skip it and go back you will fail or come very close if you keep digging through the docs blindly, time management is key
  • Be ready to extrapolate configs based on feature requirements. For example if they ask you to create a VM so that it supports snapshots, you should be able to determine that a physical mode RDM is a bad choice
  • Know the CLI, if you think you can pass this test without knowing esxcli, PowerCLI, or the vMA you are mistaken
That's pretty much it, I hope this short post proves to be helpful to someone, I know there are a lot out there. Just to make sure I cement the idea in all you potential VCAPs minds, time management will make or break you on this exam. Best of luck, I'll try to drop some DCD knowledge on everyone if I ever manage to get that one done. Thanks for reading!

Saturday, October 26, 2013

VMware VirtualCenter Management Webservices won't start - vCenter 5.1

I ran into an interesting issue at the tail end of a project not to long ago. Updated a customer's vCenter and his Hosts from 4.1 to 5.1, which amazingly enough went off without a hitch. We were about to head out when I realized that the Management Webservices weren't started. Although not a big deal I definitely wanted it sorted out so that he was able to view all his performance graphs and whatnot within the vSphere client. If you are too short on time for my horror story, just read the TLDR bullets at the end of this post.

I went in to services.msc and tried to start the service, but it immediately stopped... strange. I decided to see what event log had to say. As usual, I had a very not-so-helpful error message staring back at me:

So I did some Googling. Turns out that the Webservices are a Tomcat based Java application, which contains a core component known as Catalina. Neither the underpinnings of Tomcat nor Catalina were of major importance to us, what was relevant is that the Catalina logs might give us some insight as to where the service was bombing out.

After some digging I landed in the C:\ProgramData\VMware\VMware VirtualCenter\Logs directory where I found, wouldn't you know it a bunch of files named catalina.YYYY-MM-DD.log. So I popped open the most recently modified log and scrolled to the end.

As with most Java logs there was a bunch of stack trace type stuff that I can't personally read in it's entirety, but some of the lines did stick out to me, of particular interest were the following lines:

Failed to initialize component [Connector[HTTP/1.1-8080]] - This told me the port it was trying to use - 8080 - which is also listed in this VMware KB

Caused by: java.net.BindException: Address already in use: JVM_Bind - Aha! Something else is using the port!

I thought I had it, even with the service stopped, something was on the port, I verified with netstat -a

Victory was within my grasp, now it was easy. Just download TCPView, look for port 8080 and find out what is using it, then we can remediate this and go home. To my disappointment it was Java.exe using 8080, but which app?!

Unfortunately for me I had never done this, I tried a couple of command line tools and an app called JConsole but apparently none of these work for Tomcat apps because it uses a non-default home directory. After more digging than I would like to admit, I found out you can do it with my old friend Process Explorer. I took the process ID listed in TCPView and found it in process explorer, it was still listed as Java.exe but I could right click, select properties and see the command line used to launch Java, which included the jar file in use! The screenshot below is not the actual process causing the issue, because I had to use powershell at home to duplicate the port blocking. In the case of the customer environment it was actually a management console for a Compellent SAN (boo, hiss).

The management console was no longer needed on the vCenter server, so we uninstalled it, and the Management Webservices started right up. Hopefully this little story is helpful here are a few bullet points for the TLDR crowd:

  • If the Management Webservices wont start, use TCPView to look for something blocking port 8080
  • If whatever is blocking port 8080 is Java based, use Process Explorer to find the jar file being launched
  • Don't install the Compellent console on your vCenter server, better yet, just Buy EMC :)

That's it! Hopefully I helped you guys out, thanks for reading!

Saturday, August 17, 2013

You guys are gonna love this - vCenter 5.1 SSO Password Recovery Using SQL

Hey all, as per my last post I tested altering the password in the SSO DB and it worked! Watch the below video for instructions on how to get it done:

vCenter 5.1 SSO Password Reset from Bill Gurling on Vimeo.

Always back up your database before modifying it directly if you are inexperienced in SQL!