VCSA 6.5 with subCA Gotcha’s

One of my projects recently was migrating from a vSphere 5.5 enviroment, running on Widows OS, to vSphere 6.5 running on VCSA.
VMware did a good job with the tools for migrating to VCSA 6.5.

In vSphere 5.5 it was a challenge to replace the self-signed certificates, especially if they where expired…. From 6.0 on VMware introduced certificate managed for the vSphere enviroment. And where the PSC would be the CA for the vSphere enviroment.
With this, you can choose to let vCenter be a subCA in your PKI infrastructure.

You can find enough how-to’s for configuring the VCSA as a subCA. Like¬† here:

I just want to add some Gotcha’s, maybe you think ‘Yeah, of course…. duh….. why don’t you know that…’ well… then that Gotcha’s wasn’t meant for you ūüôā

Gotcha’s

  1. file access to VCSA
  2. certificate names
  3. issuing the subCA certificate on a windows offline root CA

Gotcha 1  РFile access to VCSA

At some point you need to download the CSR (certificate request) and copy it in your PKI enviroment. The most common method is to use SCP for file access… but the shell of the VCSA doesn’t support this by default.
This is what you need to do:

  1. allow SSH and bash access to VCSA
  2. Login as root into the VCSA with a ssh-client
  3. access the shell
    >shell
  4. change the shell for root to bash shell
    >chsh -s “/bin/bash” root
  5. Run winscp (or any other SCP client) and connect to the VCSA

To change the shell setting back to its original configuration, run
>chsh -s “/bin/appliancesh” root

Gotcha 2 – certificate names

This was a, a-yes-off-course moment…. but it did cost me some headache….

When running the steps to configure the VCSA as a subCA in your PKI, you get a few times the question

Enter proper value for ‘Name’ [Default value : CA] : ……

Make sure you use unique names here. This is the name of a certificate that is going to be generated for you. The certificate manager doesn’t check these settings, and in the end it will make a rollback…. a.k.a. waste of time.

My suggestion is for a naming convention: <hostname>-<file name being configured>
For instance, when your VCSA hostname is: VCSA-01 and your busy with the questions for the vpxd.cfg file, then the value you would enter would be ‘VCSA-01-vpxd’
But you are (off course) free to choose your own. Just make sure that these values are unique.

The certificate manager is asking values for the following files:

  • MACHINE_SSL_CERT.cfg
  • machines.cfg
  • vsphere-webclient.cfg
  • vpxd.cfg
  • vpxd-extensions.cfg
  • certool.cfg (used to creating the subCA CSR)

Gotcha 3 – issuing the subCA on a windows offline root CA

As a good practise, the CA is installed on a windows server that is not part of the AD domain and normally is powered off.

When requestion a certificate you normaly would use a browser and go to the url <hostname subca>/certsrv. But if this isn’t available on a offline root, you can use the cmd-line.

The cmd is:
> certreq -submit -attrib CertificateTemplate:<name of CA template> <filepath of CSR>

Steps to issue a the subCA certificate

  1. create the CSR with the certificate-manager in the VCSA
  2. copy the CSR to the offline root server (see gotcha 1 for scp file access)
  3. request the certificate with the cmd-line command
  4. open the certificate authority
  5. find the requested certificate under ‘pending requests’
  6. issue the certificate
  7. export the certificate to a file (export Binary data -> Binary Certificate )
  8. import the new certificate under local machine in the personal location
  9. export the imported certificate as a base64 certificate
  10. export the rootCA certificate as a base64 certificate
  11. create a chain file by adding first the subCA file content and then the rootCA file content into the chain file
  12. copy the chain file to the VCSA
  13. (shutdown  the offline root server)

Restore experience vCenter appliance

Until a few months ago I worked with vCenters that were running on a Windows OS. Because that was the common practise. The appliance was promising but not ready for production use. When vSphere 5.5 was introduced it became an interesting discussion,should you use the appliance or still go the windows way.
I think one of the points holding people of from going to the appliance was (and maybe still is) the unknown.
I was mostly familiair with a windows OS.Yes the Windows patching was irritating, and if you wanted to upgrade VMware vCenter ….. that also was annoying. But not anymore… (well I guess for over a year that is…)

The appliance in 6.0 is great. I wrote in another article about my experience of migrating to 6.0 appliance with an update manager deployed. (https://vblog.bartlievers.nl/2017/01/17/update-manager-6-0-with-vcenter-6-0-after-migration-wizard) And I’m loving it. I was impressed by the ease of migrating from vCenter 5.5 running on Windows to 6.0 VCSA.

But there was one, tiny little, issue, called backup and restore.
How should that work when you are running an appliance.
For testing purposes we did a restore of the appliance but ran into a problem. The network adapter changed. The appliance OS didn’t recognize it anymore as eth0 but as eth1.
And our first conclusion was… well…. that is a problem…restoring a VCSA.
Diving into some vmware documentation, I ran into info that restoring a VCSA should work… (see question 29 in https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146439)…. What did we do wrong….

After examening the events of the restored VCSA we saw an error. vCenter was saying that there was a duplicated MAC address…. so it changed the MAC address of the restored VCSA… hence the eth1.

And yes… that was the issue…. in default vCenter will change a MAC address if it already exists. (that is the short version)…. so I tried it again…. But with a small change.

I created a virtual portgroup on a vswitch, not on the dvswitch. Taking care of the duplicated MAC address in a dvSwitch… I changed the MAC address of the restored VCSA back to the original setting. And I started the VM…. voila….
The VM booted…. and after waiting for the boot process to complete…. I had a restored VCSA, fully functional.

Summary:

  • restoring a VCSA does work… just keep in mind if you replace the existing VCSA or restoring it next to it (duplicated MAC address)
  • Updating the VCSA is a breeze . You need connection to the internet, and the appliance will take care of the update. No seperate OS updates and vCenter updates.
  • (side note) the webclient interface is faster than the one in 5.5

Update manager 6.0 with vCenter 6.0 after migration wizard

While I was busy at at customer upgrading their VMware environment, I ran into a strange issue.
The upgrade involved migrating from windows vCenter 5.5 to a vcsa 6.0 Update 2. I used the migration wizard for this solution. (6.0 Update 2M).
The old vCenter was running also the update manager. Meaning I had to move the update manager to a new windows server and after the vCenter was migrated I had to upgrade the update manager.
This sounds simple.
And it was ….. for a while…
I successfully moved the 5.5 update manager installation to a new windows server.
The migration to the VCSA also went successfully (loving it)
but then…..
The iso for 6.0Update 2M doesn’t contain the update manager software. So I downloaded the iso for windows installation. From the same page as I downloaded the previous iso. But when I install update manager, I get the error that the vCenter server is incompatible with the update manager…. (what !!!!)
After some (soul) searching…. I found a forum thread that the Update Manager for vCenter 6.0.0b does work. (really…..)

And yes … after downloading that iso, starting the update manager installation, the upgrade was succesfull.

I only wonder… what is the difference between 6.0Update2M and the windows version. Why won’t the update manager from that iso work with the 6.0Update2M version….

TIP:

If you want to migrate from a 5.5 enviroment to the latest version  on 6.0 (currently update 2a), then after using the migration tool, go to the vCenter Appliance administration site and start from there the update. (https://<fqdn&gt;:5480)
After which you can upgrade vCenter Update Manager to the latest version.