Setup SSL Certificate on Cisco 3000 Series VPN Concentrator

In this case we went with a SSL Certificate from GoDaddy.com because of price. The problem arose from the fact that the CSR (Certificate Signing Request) wasn’t generated from the Cisco VPN Concentrator itself. However, in retrospect, this was probably the only way to do it properly because there doesn’t seem to be a way to export the private key from the Cisco. This would have posed a problem in our case because we purchased a Wildcard SSL Certificate (*.domain.tld) that can be installed on unlimited servers in our domain, and secure any host server within our domain.

Required Resources: The following software was used to complete this project:

• OpenSSL installed from cygwin
• Windows IIS Server
• Text Editor
• Web Browser

Resolution: These are the general steps necessary to set this up, and should work for most major Certificate Authorities (CAs):

1. Certificate Signing Request (CSR) generated on IIS Server. The CSR contains the Requested Public Key. The Private Key is left on the server.
2. CSR submitted to Certificate Authority (CA). They generate the SSL Certificate, and provide the Cert, along with their CA Cert and Intermediate Cert.
3. The CA Cert and Intermediate Cert are installed as CA Certs on the Cisco VPN Concentrator 3000.
4. Here’s the tricky part. The Cisco Concentrator requires the SSL Cert to be in PKCS8 format, and contain the Private key and SSL Cert.
5. I exported the Private/Public Key pair from the IIS Server, using the Windows Certificate Export Wizard; selecting to export both keys, and saving withOUT ‘high security’, and a password.
6. This generates an encrypted PKCS12 file.
7. At the unix command line (I used CYGWIN), I used OpenSSL (thanks to this site for OpenSSL basics)to first convert the PKCS12 file to standard format:
   

   openssl pkcs12 -in CERTIFICATE_NAME.pfx -out CERTIFICATE_NAME.pem

   The command prompts for the password used to export the key file from the IIS server. Then asks for a new password.
8. I then converted the standard file to PKCS8 format for the Cisco:
   

   openssl pkcs8 -in CERTIFICATE_NAME.pem -topk8 -out CERTIFICATE_NAME.pk8

   Again the command prompts for the ‘New Password’ from the last export, and asks again for a newer password.
9. Back on the Cisco Concentrator, I import a SSL certificate manually with Private Key for the Private Interface. Use the ‘copy and paste’ method.
10. One other issue we have is that our CA uses an ‘Intermediate Certificate’. Thus creating a ‘chain’ of 3 trusted certificates: Ours, the ‘intermediate’ CA and the ‘root’ CA.
11. In a text editor open both the CERTIFICATE_NAME.pk8 Private Key file you generated, along with the CERTIFICATE_NAME.cer SSL Certificate file provided to you by the CA.
12. Copy and paste the Private key into the text box on the concentrator.
13. Then immediately after, copy and paste the SSL Certifcate. Avoid any excess spaces, or blank lines.
14. Then copy and paste the Issuing certifacte after the SSL Certificate.
15. Finally copy and paste the Root Certificate at the end.
16. The whole thing should look something like this:
    

    -----BEGIN ENCRYPTED PRIVATE KEY-----
MIICiyuqweiSDuryiGquweryiDFuqweyGrqour

9bgt3ouiiDnmbweFmnriorGuweioruu8u=
-----END ENCRYPTED PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIEZkjasdlkDFajljkFyasdf8kGlyysfklysk

asklfjsadkyy7DuklasyFdjfklyaGsdfjld=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEQkjasdlkDFajljkFyasdf8kGlyysfklysk

asklfjsadkyy7DuklasyFdjfklyaGsdfjld=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC5jasdlkDFajljkFyasdf8kGlyysfklysk

asklfjsadkyy7DuklasyFdjfklyaGsdfjld=
-----END CERTIFICATE-----

17. Enter the same sequence of data again for the Public interface.

How To Backup ESXi Configuration – The Missing Piece

This came up on #VMware on Freenode this weekend. Basically the concern was “How do I Backup my ESXi USB Key?” Other than ripping the USB key out of a production machine… how was the user to do this? Well, vMA and the vCLI provide a method for this:

Backing up your ESXi Configuration:

To backup your ESXi configuration you’ll be using the vicfg-cfgbackup.pl command as follows:

• Download either the vMA or vCLI
• Launch vicfg-cfgbackup.pl:
  C:\Program Files\VMware\VMware vSphere CLI\bin>vicfg-cfgbackup.pl –save –server 192.168.15.253 –username root –password password backup.bak
• Note: The backup will be stored relative to your user “AppData” path:
  C:\Users\Username\AppData\Local\VirtualStore\

Restoring your ESXi Configuration:

Restoring your ESXi config can be done after you have the host up and responding over the network again by using the following:

C:\Program Files\VMware\VMware vSphere CLI\bin>vicfg-cfgbackup.pl –load –server 192.168.15.253 –username root –password password backup.bak

Note: You will be asked to reboot the host on restore.

Backing up multiple hosts! – There is a script to backup multiple ESXi hosts on the VMware communities site here. Also in PowerCLI here!

[Edit: Added link to backup multiple ESXi hosts from William in the comments. Thanks William!]
[Edit 2: Added PowerCLI link from NiTRo. Site is in French, PowerCLI is not]

ESXi and USB failure?

Interesting Article

In recent years, servers with embedded USB storage have become common practice. Today, all major hardware vendors deliver servers with embedded ESXi. Even in my home lab, servers are equipped with an onboard USB connector, USB stick and ESXi. Recently, on one host, the USB stick was moved to an external connector.  I was wondering, what would happen with an ESXi host with USB stick failure. Or even worse, pulling the USB stick.

So, after booting up my 2 node cluster, I made a fresh backup of a few important VMs and checked the vCenter Service Status. Now, it is time to remove the USB stick from one host. And this is what happened:

• VMs on the affected host are still running.
• Task & Events of the affected host shows this message “Lost connectivity to storage device mpx.vmhba32:C0:T0:L0. Path vmhba32:C0:T0:L0 is down. Affected datastores: “Hypervisor1”, “Hypervisor2”, “Hypervisor3”.”.
• Followed by 3 Alarms “Cannot connect to storage”.
• Another message in Tasks & Events is “Boot partition /bootbank cannot be found (0:02:33:03.304 cpu1:30722)”.
• Time for some testing, all these actions do work: Power On a VM, Migrate a VM, host in Maintenance Mode, Exit Maintenance Mode (HA Agent is configured correctly).
• Also the ESXi console is doing fine, System Customization is in place, and so are the System Logs.
• From time to time above messages are repeated and in some occasions while migrating VMs “The Operation is not allowed in the current state” messages are received.
• After 24 hours, the host is still running, and performing. So finally, I decided to enter the host in Maintenance Mode and shut it down. The power down took about 10 minutes ( less then 2 minutes is normal).
• After insertion of the USB stick, the host was powered on and was automatically reconnected to the cluster.

At this time, my tentative conclusion is that failure, or even an missing USB stick does not have much impact on a ESXi host. Thanks for reading and I’m very interested in your experience and opinions concerning this subject.

P.S. A few days after posting, I stumbled onto this post, written by Alan Renouf. In the first part it is explained why ESXi keeps running without USB boot device.

Script from Alan to backup ESXI host.

############################

$RootFolder = "C:\Support\"

Get-VMHost | Foreach {

Write-Host "Backing up state for $($_.Name)"

$Date = Get-Date -f yyyy-MM-dd

$Folder = $RootFolder + $Date + "\$($_.Name)\"

If (-not (Test-Path $Folder)) {

MD $Folder | Out-Null

}

$_ | Get-VMHostFirmware -BackupConfiguration -DestinationPath $RootFolder

# Next line is a workaround for -DestinationPath not working correctly

# with folder names with a - in them.

MV ($RootFolder + "*") $Folder -ErrorAction SilentlyContinue

########################################