vCenter


Upgrade vom vSphere 5.0 to 5.5 – 512 bit certificate issue 10

This week I upgraded the vSphere 5.0 environment to 5.5 Update 1, which is usually not a big deal. I really can’t complain about the upgrade process itself, it’s more the result which I didn’t expect.

Once all components were up to date, I launched the vSphere Web Client which was working fine but at the top I saw the following error message:

Failed to verify the SSL certificate for one or more vCenter Server Systems: https://vCenter_FQDN:443/sdk

I was able to login via the C# client which showed the vCenter as usual, so it seemed to be a problem between the vCenter server and the Web Client.

After I spoke to the VMware Support it turned out that the vCenter Server doesn’t support the old 512 bit certificates. This problem is mentioned in the release notes:

After you upgrade vCenter Server 4.x to 5.5 Update 1, vCenter Server is inaccessible through vSphere Web Client*
When you upgrade from vCenter Server 4.x to 5.5 Update 1, the vCenter Server is not visible through vSphere Web Client. The issue occurs as vCenter Server 4.x supports SSL certificates with 512 bits but vCenter Server 5.5 supports only SSL certificates with greater than or equal to 1024 bits.

Workaround: To resolve this issue, replace the vCenter Server certificates with greater than or equal to 1024 bits

I wasn’t aware of this issue and even if so I would’t have recognize it, since I upgraded a 5.0 environment. The actual problem was that the environment has been upgraded from 4.1 to 5.0 before which is the cause why there still was the 512 bit certificate in use. To see if you are affected by this problem simply open the the rui.crt file (C:\ProgramData\VMware\VMware VirtualCenter\SSL) before upgrading vSphere:

512BitCert

The funny thing is that none of the installation wizards recognizes that the certificate is unsupported so the upgrade went through without any errors.

However there is a way to fix it which I outline below:
1. Backup your vCenter Server / database / old SSL certificates

! This process will cause some downtime to certain vCenter services !

2. Generate new Certificates (KB2037432)

a) Create a temporary directory like c:\certs

b) Create a file called vcenter.cfg with the following content:

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

 

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: HOSTNAME, IP: xxx.xxx.xxx.xxx, DNS: FQDN

 

[ req_distinguished_name ]
countryName = DE
stateOrProvinceName = YOURPROVINCE
localityName = YOURCITY
0.organizationName = YOURORGANIZATION
organizationalUnitName = vCenterServer
commonName = FQDN

 

3. Start to create new certificates

The openssl utility can be found in C:\Program Files\VMware\Infrastructure\Inventory Service\bin\openssl.exe

openssl req -new -nodes -out c:\certs\rui.csr -keyout c:\certs\rui-orig.key -config c:\certs\vCenter.cfg

openssl rsa -in c:\certs\rui-orig.key -out c:\certs\rui.key

openssl req -text -noout -in c:\certs\rui.csr

openssl x509 -req -days 3650 -in c:\certs\rui.csr -signkey c:\certs\rui.key -out c:\certs\rui.crt -extensions v3_req -extfile c:\certs\vCenter.cfg

//Update: I just saw that it might be better to add the -sha256 option to use a more secure algorithm instead of the default SHA1

openssl.exe pkcs12 -export -in c:\certs\rui.crt -inkey c:\certs\rui.key -name rui -passout pass:testpassword -out c:\certs\rui.pfx

openssl pkcs12 -in c:\certs\rui.pfx -info

openssl x509 -text -noout -in rui.crt

4. Create a file called chain.pem in C:\certs and then open the rui.crt file with an editor and copy the content into the chain.pem file and finally save it.

5. Use the SSL Certificate Automation Tool 5.5 (for vSphere 5.5 only!) to plan the actions and the order in which they should be performed:PlanSSLUpdateSetps

Take a screenshot of the list or write it down, you will need it in a second.

6. Now replace the certificates just for the vCenter server. The tool will ask you for the certificate chain which is located in C:\certs\chain.pem and the private key c:\certs\rui.key and some credentials:UpdatevCenterCerts

7. Once this is done, you will need to re-establish the trusts between vCenter server and it’s components like the Inventory Service, SSO and so on. When performing these steps, follow the order depicted on the list you have written down. The following screenshot shows the process generic, because it’s pretty similar with all other components:TrustUpdateManager

That’s it. After that I went to the vSphere Web Client and logged in as usual. No errors left, the vCenter server connected to the Web Client correctly and I was able to manage it. So overall this certificate replacement was easier than expected and it fixed the issue as required. I hope this helps!


Veeam v7.0 – My favorites – Transparent vCenter Backup

This post is dedicated to another minor enhancement which makes your life much easier.

As you might know, pre Veeam v7.0 if you wanted to backup your virtual vCenter server with full guest processing (I/O quiescing), you had to use some kind of a workaround to make it work.

You had to add an ESXi host via its management IP address to the Veeam inventory and then within the backup job you had to select the vCenter VMs via the host and not via the vCenter itself. With this workaround you are able to perform a consistent backup the vCenter server VM. Otherwise the job would fail because the I/O quiescing causes that the vCenter doesn’t receive the required confirmation of the snapshot creation to be able continue I/O.

Veeam described the problem in KB1051

But with Veeam v7.0 all that belongs to the past, now you can easily add your vCenter VM as any other virtual machine to a backup job, without the need of a workaround.

“Transparent backup of vCenter Server VM. The vCenter Server VM can now be backed up without having to resort to workarounds, such as adding the vCenter Server’s host by IP address and then configuring a separate job for it.”

Quote “veeam_backup_7_whats_new_en.pdf”

I simply love it 😉


VMware vSphere 5.1 – Some tips around Single Sign On

When I did my first installations or upgrades to vSphere 5.1 I struggled with the new single sign on server. Here I will provide you some tips to prevent some pitfalls.

 

SSO database (connection)

SSO requires its own database but this is not the problem. Before your start the SSO installation you should have already prepared the database and the SQL user(s).

In the first step you need to run a SQL script which will create a new database called “RSA” and insert the required table spaces. This script can be found in:

VMware-VIMSetup-all-5.1.0-880471\Single Sign On\DBScripts\SSOServer\schema\mssql\ rsaIMSLiteMSSQLSetupTablespaces.sql

You can do so by using the  SQL Server Management Studio, just open the file and execute it.

Later on the installer will ask you for two users to access the database. The funny thing is, this only works with SQL users! No support for Windows authentication.

VMware recommends creating the following two users: RSA_DBA + RSA_User

This can be also done manually or via another predefined script:

VMware-VIMSetup-all-5.1.0-880471\Single Sign On\DBScripts\SSOServer\schema\mssql\ rsaIMSLiteDBNameSetupUsers.sql

If you run the scripts in the order I mentioned above you should be fine. If you create the users manually, make sure the RSA_DBA gets the sysadmin rights.

Quote out of the SetupUsers script:

“The DBA account is used during installation and the USER account is used during operation” 

The RSA_USER needs rights to access the “RSA” SSO database. I guess the most of you know this procedure from creating a vCenter or an Update Manager database.

 

SSO configuration

When you login to the web client you can add single users or groups to you single sign on server. But you should know these will only exist within the SSO “System-Domain”. SSO won’t create local Windows/AD users for you.

Users you add/create here will only have permissions to login into the web client and view or edit SSO settings. But what you really need is an identity source, which can be added (if not  happened automatically):

This identity source will be queried every time you login to your vSphere or web client. The reason why I highlighted the “CN=Users” part in the Base DN for groups line, is that at the moment queries can time out if your active directory contains many groups/OUs. This is a workaround to limit SSO to search for objects only within a particular group or OU.

And don’t forget to set the alias. Otherwise you will run into problems when you want to use your windows session credentials. Your session will send ALIAS\Username but if you don’t set a domain alias your SSO will wait for home.local\Username. At least we had some issues without the proper alias; feel free to provide some feedback if I’m wrong here.

When you don’t use your current Windows session you will need to provide the domain that SSO is able to query the right identity source. The same can be necessary if you use local user accounts <computername>\<local user>

I hope this will help you to a bit then you set up your single sign on server. And don’t forget to check the vSphere Installation and Setup Guide.

 

*UPDATE* (21.05.2013)

Today I stumbled upon a problem where the login in general worked fine but some specific users I got the following error message:

Web client:

“The authentication server returned an unexpected error: ns0:RequestFailed: Internal Error while creating SAML 2.0 Token. The error may be caused by a malfunctioning identity source.”

vSphere client:

WrongUserPW

I checked the imsTrace.log located in <vCenter Installation path>\SSOServer\logs, where I   received the following error every time I attempted to login:

“… Failed group search, unexpected interrupt …”

This information led me straight to KB2050941

All I had to do, was to remove the second domain / identity source within the SSO server, which wasn’t needed at all (luckily) but that fixed it!

*UPDATE* (23.05.2013)

vCenter Server 5.1 Update 1 has been released