A while back, I was noticing some strange behaviors with some parts of the vSphere web client. Primarily, it was related to vSAN details.
Quickstart configuration wouldn’t load, status on a couple of the vSAN services would get stuck at loading, an error message about failed to extract requested data.
Turned out to be related to SSL certificates in some of the underpinnings.
Came across a Reddit thread:
https://old.reddit.com/r/vmware/comments/cxbk24/vsan_67u3_error_failed_to_extract_requested_data/
The solution was a python script for checking trust anchors.
https://web.vmware-labs.com/scripts/check-trust-anchors
Using the -cml switch to do a live check on machine certificates and colorize the output.
For some reason, one of the endpoint certificates being used was the original self-signed certificate. This should have been replaced, but it seems it had not.
root@vcenter [ /tmp ]# ./check-trust-anchors -cml No 'lstool.txt' file found in this directory. Dumping service registrations to /tmp/lstool.txt... -----Endpoint Certificate 1----- Certificate Info: Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcenter.incendiary.local, OU=VMware Engineering Validity Not Before: Aug 11 22:57:30 2019 GMT Not After : Aug 5 22:57:29 2029 GMT Subject: CN=vcenter.incendiary.local, C=US SHA1 Fingerprint=51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D -------------------------------- -----Endpoint Certificate 2----- Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51 -------------------------------- -----Machine SSL Certificate----- vcenter.incendiary.local Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51
You can use the -e switch to display all the individual endpoints that are being used by each certificate. It’s a long list, so I won’t paste it here.
Then in conjunction with the -cml switch, use -f to fix.
It will prompt for the SSO password, and then ask which certificate fingerprint needs updated.
In my instance, it’s endpoint certificate 1.
51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D
So we provide the creds and that fingerprint.
root@vcenter [ /tmp ]# ./check-trust-anchors -cml -f
No 'lstool.txt' file found in this directory. Dumping service registrations to /tmp/lstool.txt...
-----Endpoint Certificate 1-----
Certificate Info:
Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcenter.incendiary.local, OU=VMware Engineering
Validity
Not Before: Aug 11 22:57:30 2019 GMT
Not After : Aug 5 22:57:29 2029 GMT
Subject: CN=vcenter.incendiary.local, C=US
SHA1 Fingerprint=51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D
--------------------------------
-----Endpoint Certificate 2-----
Certificate Info:
Issuer: CN=incendiary-ROGUE-CA
Validity
Not Before: Oct 11 19:44:16 2019 GMT
Not After : Oct 8 19:44:16 2029 GMT
Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local
SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51
--------------------------------
-----Machine SSL Certificate-----
vcenter.incendiary.local
Certificate Info:
Issuer: CN=incendiary-ROGUE-CA
Validity
Not Before: Oct 11 19:44:16 2019 GMT
Not After : Oct 8 19:44:16 2029 GMT
Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local
SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51
---------------------------------
{CYAN}SSL Trust Anchor Repair
---------------------------------
Enter SSO admin [[email protected]]:
Enter password for [email protected]:
Enter fingerprint of trust anchor(s) to update: 51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D
Enter the FQDN of the node to update: localhost
Get site name
Lookup all services
Get service default-site:3a3b9e34-bc7c-4089-aee8-21f29c262ec7
Update service default-site:3a3b9e34-bc7c-4089-aee8-21f29c262ec7; spec: /tmp/svcspec_m5zme_pb
Get service default-site:0ed7813d-b31a-47f0-9c30-b8c1d05ada25
Update service default-site:0ed7813d-b31a-47f0-9c30-b8c1d05ada25; spec: /tmp/svcspec_qof9h8zl
Get service default-site:cdddfeca-7765-4ba2-acf4-b25e1bd2d26a
Update service default-site:cdddfeca-7765-4ba2-acf4-b25e1bd2d26a; spec: /tmp/svcspec_i39mfvuc
Get service f4335c81-ce40-4d03-8f83-d043bbb64e8e
Don't update service f4335c81-ce40-4d03-8f83-d043bbb64e8e
Get service be370880-e1e9-4bf6-9af2-4e7c5d756908
Don't update service be370880-e1e9-4bf6-9af2-4e7c5d756908
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vrops
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vrops
Get service 18360c0f-d11c-450e-967a-446beef6cbe0
Don't update service 18360c0f-d11c-450e-967a-446beef6cbe0
Get service 2fdd948a-fb14-49cc-a27d-7c4cae7e3e01
Don't update service 2fdd948a-fb14-49cc-a27d-7c4cae7e3e01
Get service ba3e37d4-26b9-47ab-9007-f3d8dc6a88cb
Don't update service ba3e37d4-26b9-47ab-9007-f3d8dc6a88cb
Get service 7a68d318-a9ca-4e2c-adbc-27873b7a7cb1
Don't update service 7a68d318-a9ca-4e2c-adbc-27873b7a7cb1
Get service 081405da-1dc1-43c9-adb0-885fabb1c325
Don't update service 081405da-1dc1-43c9-adb0-885fabb1c325
Get service 21504f6a-e383-4510-96e2-e2a814535cde
Don't update service 21504f6a-e383-4510-96e2-e2a814535cde
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.nsx.ui.h5
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.nsx.ui.h5
Get service 23994053-eef3-456b-b0ef-ae6d9f91d7a9
Don't update service 23994053-eef3-456b-b0ef-ae6d9f91d7a9
Get service 35a5d0cd-b2dd-46e7-81b3-59bc1da10f79
Don't update service 35a5d0cd-b2dd-46e7-81b3-59bc1da10f79
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vShieldManager
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vShieldManager
Get service 5bceba88-ee1a-46a2-a52a-2f42462ae3f3
Don't update service 5bceba88-ee1a-46a2-a52a-2f42462ae3f3
Get service da147e1a-09d6-4cc3-b784-9a762ca1694e
Don't update service da147e1a-09d6-4cc3-b784-9a762ca1694e
Get service 70f41eaa-5fbf-4c3e-ab4f-f7de3d0f3aa1
Don't update service 70f41eaa-5fbf-4c3e-ab4f-f7de3d0f3aa1
Get service f1d36406-546f-43e2-8ae3-b942c5439c54
Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54
Get service 9fb71a11-ffd9-4872-8ce5-b4034f3d40a7
Don't update service 9fb71a11-ffd9-4872-8ce5-b4034f3d40a7
Get service abb20cc9-1261-467e-aafe-33b042775026
Don't update service abb20cc9-1261-467e-aafe-33b042775026
Get service d1f68783-4dd7-498e-8051-0080c746f254
Don't update service d1f68783-4dd7-498e-8051-0080c746f254
Get service a672b45e-7896-42cf-917b-cea8f75c0b71
Don't update service a672b45e-7896-42cf-917b-cea8f75c0b71
Get service 946bdda0-c2e3-4001-b7c3-244788312349
Don't update service 946bdda0-c2e3-4001-b7c3-244788312349
Get service bb159ac3-79df-4983-9556-290c945e1c9f
Don't update service bb159ac3-79df-4983-9556-290c945e1c9f
Get service 0c43747b-aac8-45cd-9428-56d06e800aef
Don't update service 0c43747b-aac8-45cd-9428-56d06e800aef
Get service a3cfb721-605f-4bf0-9e9a-d7ae0c605bac
Don't update service a3cfb721-605f-4bf0-9e9a-d7ae0c605bac
Get service f1d36406-546f-43e2-8ae3-b942c5439c54_kv
Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54_kv
Get service 28aff66c-2429-425f-8b58-775b577d790b
Don't update service 28aff66c-2429-425f-8b58-775b577d790b
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vic
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vic
Get service 109d7960-21b1-4d99-a0bd-697e3550dbab
Don't update service 109d7960-21b1-4d99-a0bd-697e3550dbab
Get service 24a6452f-2c0b-4e56-a450-1fbee7243041
Don't update service 24a6452f-2c0b-4e56-a450-1fbee7243041
Get service bcdcb8b6-3782-419e-b826-ecece024d7c5
Don't update service bcdcb8b6-3782-419e-b826-ecece024d7c5
Get service 122bff4a-443d-4094-9035-be4e244286cd
Don't update service 122bff4a-443d-4094-9035-be4e244286cd
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsphere.client
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsphere.client
Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsan.dp
Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsan.dp
Get service f1d36406-546f-43e2-8ae3-b942c5439c54_authz
Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54_authz
Get service b57f2a19-ce5b-40d2-aff7-dcad57518729
Don't update service b57f2a19-ce5b-40d2-aff7-dcad57518729
Get service 0f889511-99db-427b-983a-bdc8a308fcf5
Update service 0f889511-99db-427b-983a-bdc8a308fcf5; spec: /tmp/svcspec_hmbi6gma
Updated 4 service(s)We can see that it updated the 4 services that were using the wrong certificate, and then the strange behaviors I had were resolved.
Note that there is also a KB about a similar issue for vCenters that started life as vCenter 5.5, and issues began in vCenter 6.7. This was resolved in 6.7U3a.
https://kb.vmware.com/s/article/74731
Also note that VMware has a new tool called lsdoctor for “addressing issues in the PSC database, as well as data local to vCenter.”
This can also be used to correct certificate related issues.