Using Commercial Certificates in vSphere 5.5

I would like to share my experience and findings when working with commercial certificates in vSphere 5.5.  I’ve been working on a customer project where there is a requirement to replace the self-signed certificate of the Web Client with a commercial one. This particular project involved using a Verisign certificate.

I initially used the Certificate Automation Tool to generate the CSR (rui.csr) and the private key (rui.key). Oddly, Verisign is giving the Error 4824 below with the generated CSR:

The error points to the Subject Alternative Name (SAN) and specifically because there is a Shortname and Private IP in the SAN field of the CSR which Verisign does not allow. I remember when using the Certificate Automation Tool, the shortname and IP address are fields that must be populated. With the help of my colleague Frank Buechsel (http://fbuechsel.eu/), he recommended to generate the CSR’s manually using KB 2044696.

Following the KB article, I created an OpenSSL configuration file for the Web Client using the following format:

To remove the shortname and IP address, edit the subjectAltName and remove the ServerShortName and IP. It should look like this afterwards:

subjectAltName = DNS:server.domain.com

We can now generate the CSR using the modified configuration file. Following the same KB article, we generated the new CSR using the below commands (openssl is available inside the extracted Certificate Automation Tool):

After performing the above steps, we were able to successfully submit the CSR to Verisign.

After receiving the certificate in X.509 format, we are ready to install it using the Certificate Automation Tool. I renamed the certificate to rui.crt and attempted the replacement but encountered an error in the tool with a message that the chain cannot be validated. I then checked the downloaded certificate and saw that it does not contain the Intermediate CA in it. So it means that the tool won’t let you install the certificate if the chain to the intermediate CA is missing. We then proceeded to  download the Intermediate certificate from Verisign and again referencing the same KB article, created the chain.pem file in order to have the certificate chain.

Using the Certificate Automation Tool with the new chain.pem file, the certificate replacement successfully went through.

To cut the long story short, the key takeaways when using commercial certificates are:

• Ensure that only FQDN is in the SAN field of the CSR else CSR submission will fail
• Once you receive the certificate from your commercial CA, verify that the chain to the intermediate CA is present before proceeding with replacement

 

 

 

NATting Multiple Subnets on DD-WRT

Wanted to share an issue that hit my homelab setup the past two weeks. I’ve completed my network setup using my new Cisco SG300-20 L3 switch. My setup is very similar to Vladan’s post where we used the same L3 switch and placed a DD-WRT router (mine’s a Buffalo WZR-HP-G300NH2) in between the Internet router and the Cisco SG300 switch. The reason being my Internet router does not support static routes. So the plan is to use NAT to allow internet access from within the management and VM subnets in the Cisco SG300 switch. My network looks like this:

Cisco SG300-20 L3 switch:

VLAN_10 (Management network):10.10.10.x/24
VLAN_20 (Storage network): 10.10.20.x/24
VLAN_30 (vMotion network): 10.10.30.x/24
VLAN_40 (VM network): 10.10.40.x/24

DD-WRT Router:

WAN_Side (To Internet router): 192.168.11.28/24
LAN_Side: 10.10.10.2/24

I only needed internet access on my  management and VM networks, so I have set up the static routes from within the DD-WRT to allow traffic to route back to the VLAN_interfaces of the Cisco SG300 switch. Initially I have tested with the VM’s inside the management network and internet access worked fine so I thought I’m good already. Then when I started running VM’s inside my VM network, this is when I found out that it can access all other configured networks except the internet.

Now I am very new to DD-WRT and I don’t have prior experience using it but I suspect something is wrong with it and why it only translates my management network. But I can say that persistence always pays off as I searched and searched and try to find out similar issues from within the net and finally found this article at around the 10th page of my Google search. Turns out that by default, DD-WRT only translates traffic from the first network. This is the reason why my management network is working fine. To be able to allow NAT for my VM network, I pasted the command in DD-WRT’s gui to edit the firewall rule as suggested by Patrik’s blog:

iptables -t nat -I POSTROUTING -o `get_wanface` -j SNAT –to `nvram get wan_ipaddr`

This resolved my issue and now all my VM’s are able to access the internet.

Homelab Upgrade

I run my current homelab off of VMware Workstation and it has served me well in the past. I have 2 whiteboxes with 32GB of RAM and everything else is virtual (virtual ESXi, virtual freenas, virtual Vyatta Router). I have used this setup for quite some time now and it has helped me with my VCAP-DCA as well as playing around and testing other VMware products (I have vCloud Director, SRM, VCOps, Log Insight). But in the past months I have noticed slowness as I add more and more VMs and products, particulary NSX.

So I have decided to upgrade my lab and instead of using Workstation, I am now planning to convert my whiteboxes to baremetal ESXi’s. A few items I have added so far:

1. Synology DS412+ NAS – managed to grab a used NAS from a local web forum. I thought of getting a brand new DS415+, but settled for the older model as it is more than enough for my requirements.  This will allow me to fully deploy an ISCSI storage and will also allow me to test RDM’s which is the limiting factor in my current lab.

2. Intel PRO Dual Port NICS – The changes in ESXi 5.5 removed the Realtek drivers which is used by my motherboard’s Onboard NIC. I have tried to inject the Realtek drivers from ESXi5.1 (using ESXi Customizer), but I noticed that the NIC card will get recognized, but no traffic is flowing. This has been reported in the Communities as well so I guess it is a hit and miss (some worked and some has not). Anyways I am adding 2 dual port NICs on each whitebox which gives me enough ports to separate traffic (Management, vMotion, ISCSI Storage)

3. Cisco SG300-20 20 Port Gigabit Managed Switch – I just ordered this switch from Amazon. What got me enticed is that this switch can do static L3 routing on top of L2 functionality such as VLANs.

I am also planning to add another whitebox in the mix for added compute power. Hopefully I can get everything setup early next year so I can continue with what I love to do best.. which is to play around with the coolest products in the virtual world 🙂

VCAP5-DCA PASSED!

I wrote an article Iast January (article here) about my goal to continue on with my VCDX journey. Last Sunday, I took the plunge and sat the VCAP5-DCA (VDCA510) exam and I am happy to anounce that I passed it.

My Experience and Strategy:

As other bloggers have mentioned, time is the enemy in this test. The lab is slow so you really need a strategy to do the lab tasks as quickly as possible. There are 26 lab questions which are really spot on with the Exam Blueprint. My strategy was to do all the install and configure tasks first and skip the troubleshooting tasks until I reached the last question. Do all those things that you know first and quickly to gather enough points. Then I went back to the troubleshooting questions and tried to answer them until I ran out of time. After 4 brain draining hours, I’m done and was reminded that I will receive the result in 15 days. On my way home, exactly two hours after the test, I received an email that I passed.. I was quite surprised on how fast I got my result but was also relieved to have passed it.

How I Prepared:

Lab work! You cannot pass this exam without studying and doing hands on work. My experience with vSphere helped but there are areas in the blueprint that even a seasoned Admin do not perform very often. So I checked the blueprint, assessed my skills, and focused more on to the areas that I’m weak at. I only managed to seriously study for this exam after I booked it. It really forced me to prepare! I guess it is a good motivating factor 🙂

I relied heavily on Jason Nash’ VCAP-DCA Pluralsight/Trainsignal course and the free Unofficial Study Guide for VCAP5-DCA by Jason Langer and Josh Coen. Also, I highly recommend that you try Joshua Andrews’ Test Track lab (link here) a week or two before your actual exam. It will give you a good feel of what the exam will be like. You can hook up with Joshua by following him on Twitter (@SOSTech_WP).

Final Notes:

It was extremely rewarding to validate your skills and the VCAP5-DCA is very good certification to do this. Be aware that now there are two DCA exams, VCDA510 which is based in vSPhere 5.0 and the new VCDA550 based on vSphere 5.5. Passing either of them will give you VCAP5-DCA certification. I decided to go for the older one due to the vast amount of resources available online as well as “exam experience” of other bloggers.

vCenter Server Install/Upgrade Gotcha when using Custom Install

Starting to prepare now for an upcoming vSphere 5.0 to 5.5 upgrade project and was building my thought process on upgrading the vCenter Server. Should I propose Simple Install or Custom Install? VMware’s recommended method is Simple Install where all components are installed in a single machine/VM. This method puts all components  in the default location in C:\ drive.

But we know that there are customers who have an OS policy that disallows use of C:\ drive for any application other than the OS itself. In this scenario, you will be forced to use the Custom Install method.

But there is one major gotcha with this as I have experienced in my vSphere 4.1 to 5.1 upgrade last year. I was actually surprised that this is still the case in vSphere 5.5. Look at KB 2044953. There is known issue when installing Web Client in a different directory other than the default. Web Client will not work and will throw an HTTP 404 error. The workaround is to re-install web client back to the default directory. There is another alternate workaround stated in the KB to put Web Client in a directory that does not contain spaces. I have not tested this though.

A very short post but I hope this helps vSphere Admins who are planning to upgrade to vCenter 5.5.

 

 

 

 

Removing Unwanted Plugins in vSphere

I always use my lab to prepare for projects and to learn different VMware products. Of course I don’t have the privilege of keeping all products running due to limited amount of compute resources . As time goes, I have to remove some of the appliances like VDP, vSphere Replication and vShield Manager.

Now that I am introducing NSX into my lab, I have noticed that logging into the Web Client now becomes painfully slow. It now takes about 3 to 4 minutes before I can get into the Web GUI. Once inside, everything is normal. Since NSX can only be configured in the Web client, I need to get this issue resolved.

Searching for similar issues in the web, I saw this one which is the closest to what I’m experiencing. I remember that I have lots of plugins that I no longer use: VDP, vSphere Replication Management and vShield Manager. To remove those plugins, I used this KB article.

Below, I captured the screenshots when I removed my VDP 5.1 plugin.

Login to http://<vcenter_name_or_IP>/mob and click content

Click ExtensionManager

Select and copy the extension that you are removing. In my case I’m removing the VDP 5.1 extension which is com.vmware.vdp

Click UnregisterExtension

Paste the name of the plugin and click Invoke Method to remove the plugin

You should get the Method Invocation Result: void message which tells you that the plugin has been removed.

I did the same procedure for my vSphere Replication Management (com.vmware.vcHms) and vShield Manager (com.vmware.vShieldManager) plugins and tested login again. And what have you, my web client login is back to normal.

vSphere Replication 5.8 and Site Recovery Manager 5.5

vCloud Suite 5.8 had just been released with new product releases like vCenter 5.5 Update2, ESXi 5.5 Update2, SRM 5.8, vSphere Replication 5.8 to name a few. For a complete list, you can refer to this link.

Itching to try out the new vSphere Replication 5.8 in conjunction with SRM 5.5, I went to the download site to get it. But looking at the dowload page for VR 5.8, it looks like vSphere Replication is not supported in SRM 5.5.

And verifying the VMware Product Interoperability Matrix, it is confirmed that only SRM 5.8 is supported.

But for standalone vSphere Replication without SRM, VR 5.8 is supported down to ESXi 5.0

 

 

Invalid Configuration for device ‘0’ Error Message

First time I’ve encountered this issue in my environment after a DRS initiated vMotion of a VM. I’ve noticed that the VM’s network card is disconnected and whenever I tried to connect it, the Invalid Configuration for device ‘0’  error pops up. From the looks of it, vMotion had a bandwidth issue but it did migrate successfully after another attempt.

This exact issue is documented in KB 2014469 and although vSphere 5.5 is not in the product list, I gave it a try to see if it will resolve this issue.

I did Option 1 of the KB article which is to move the VM into a different and unused dVPort. From the screenshot below, my VM Win2K8R2 is using port 104 of distributed switch vDS-DMZ.

To change this, edit the VM setting and highlight the network adapter. Under network connection, click Switch to advanced settings. In the Port ID box, change the Port to a new unused port. In the screenshot below, I changed the port from 104 to 105

Click OK then go back to Edit Settings and connect the NIC card. This resolved my issue and the VM is back into the network.

Forwarding vCenter Server logs into Log Insight 2.0

Im loving Log Insight!!! Ever since I’ve installed this product in my lab, there is virtually no need to go over logging to each ESXi host and go through important log files like vmkernel.log, hostd.log, vpxa.log. For me, gone are the days where you have to go over these logs, using linux tools and commands to peak over log files when troubleshooting an issue. Its so easy to do query in the Interactive Analysis page of Log Insight and it presents it to you in a graphical manner. Filtering for keywords is a breeze.

 The builtin vSphere Contect Pack will surely get you started pretty fast as it contains all the important queries in your vSphere environment. See below screenshot.

In the GA version of Log Insight,  in order to forward vCenter Server (Windows version) log files such as vpxd.log, you have to install a 3rd part Syslog agents like Syslog-NG or Datagram Syslog agent and configure them to forward the log files to Log Insight. This is because Windows does not natively support Syslog

With the release of Log Insight 2.0, a Log Insight Windows agent has been included that allows collection of Windows events and log files from a Windows machine and forwards them into Log Insight. There is also a Content Pack for Windows where the important queries for Windows events are pre-created. More information about the Log Insight Windows agent can be found here.

Back to the subject of this post 🙂 So now with the Windows agent included, how do we forward the vpxd.log file into Log Insight. Easy! Just install the Log Insight agent into your Windows vCenter Server and edit the liagent.ini file. Add the highlighted lines from the screenshot below. Restart the Log Insight service and  you are good to go!

For those interested or are using in Log Insight, I highly recommend following Steve Flanders blog at http://sflanders.net/. I’ve been constantly reading his excellent posts about Log Insight ever since I started using this product.

Deploying an Additional vSphere Replication Server

Depending on a customer’s requirement, there may be a need to deploy additional vSphere Replication servers. A few use cases are for availability and load balancing. Another one I can think of is if you want to deploy a vSphere Replication Server on a remote site that is managed by a vCenter Server on the main site. With vSphere Replication 5.5, you can add up to 9 VRS for a total of 10 including the first VR appliance which holds both the VRMS and VRS functionality. For a complete detail on vSphere Replication/SRM limits, see KB Article 2034768

In this article, I’ll show how easy it is to deploy an additional vSphere Replication Appliance.

1. In the web client, go to Manage–> vSphere Replication–> Replication Servers. Click on the OVF deployment icon to deploy the VRS ovf.

2. The file to select is vSphere_Replication_Addon_OVF10.ovf. This OVF is only 512MB in RAM, all other resources (CPU, disk) is the same as the vSphere Replication appliance OVF.

3. Review the OVF details and click Next.

4. Select the name of the VRS and folder to place it to, then click Next

5. Select the cluster as a resource to run the VRS. Click Next

6. Select the datastore where the VRS will be located. If needed, you can also change the virtual disk format. Click Next

7. Select the portgroup that will be used by the VRS. Under IP Allocation I selected Static-Manual and configured the DNS, Netmask, and Gateway settings. Click Next

8. Provide a password for the VRS root account as well as the IP address of the VRS. Click Next

9. Click Finish to start the OVF deployment.

Once the VRS is powered on and initialized, we now have to register this new VRS as an additional VRS. To do so, perform the following:

1.  In the web client, go to Manage–> vSphere Replication–> Replication Servers. Click the middle icon (Register a virtual machine as vSphere Replication Server) and select the newly deployed VRS.

2. Once registered, you will now see the new VRS under the vSphere Replication tab

You can now select this VRS or use Auto-assign when protecting a VM with vSphere Replication:

If you decided you don’t need the additional VRS, you need to unregister the VRS before removing/deleting it.

That’s it. Never gets easier than that..