Error during secondary validation

[Nepherim]

Version: 2.49
OS: Debian

Started receiving this error when trying to renew a domain cert:

Upgraded to v2 (changed https://acme-staging.api.letsencrypt.org to https://acme-staging-v02.api.letsencrypt.org)
Registering account
Verify each domain
Verifying DOMAIN.com
copying challenge token to /PATH-TO/DOMAIN.com/.well-known/acme-challenge/-xvcbRT5qoVusdKnQDOXfPoYvjvGAXMQDTpQfs4XntM
sending request to ACME server saying we're ready for challenge
checking if challenge is complete
Pending
checking if challenge is complete
getssl: DOMAIN.com:Verify error:    "detail": "During secondary validation: 2a06:98c1:3121::1: Invalid response from http://DOMAIN.com/.well-known/acme-challenge/-xvcbRT5qoVusdKnQDOXfPoYvjvGAXMQDTpQfs4XntM: 403",

The well-known file is viewable from a browser.

Detail log extract:

...
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 21 Apr 2024 15:19:47 GMT
Content-Type: application/json
Content-Length: 194
Connection: keep-alive
Boulder-Requester: 6163656
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Link: <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12092680974>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12092680974/ERokzg
Replay-Nonce: _O0fw7ZkbdMMIHWGmg6pWlxlilztJkhzSerLoCba8EAIF8dy4xo
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

 
response {  "type": "http-01",  "status": "pending",  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12092680974/ERokzg",  "token": "kXlWl32Yw3zGqylFJ3y3Y6uEXWG2ZpC-vRTRfQoOLbY"}
 
code 200
 
response status = pending
Pending
 
sleep 5 secs before testing verify again
checking if challenge is complete
 
url https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12092680974/ERokzg
 
using KID=https://acme-staging-v02.api.letsencrypt.org/acme/acct/6163656
 
payload = 
 
responseHeaders HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 21 Apr 2024 15:19:52 GMT
Content-Type: application/json
Content-Length: 1018
Connection: keep-alive
Boulder-Requester: 6163656
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Link: <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12092680974>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12092680974/ERokzg
Replay-Nonce: _O0fw7ZkyWZK2p_O0Chn-5vpsYv_dFSLOfE8x4ltVNAWh7lzZwU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

 
response {  "type": "http-01",  "status": "invalid",  "error": {    "type": "urn:ietf:params:acme:error:unauthorized",    "detail": "During secondary validation: 2a06:98c1:3120::1: Invalid response from http://DOMAIN.com/.well-known/acme-challenge/kXlWl32Yw3zGqylFJ3y3Y6uEXWG2ZpC-vRTRfQoOLbY: 403",    "status": 403  },  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12092680974/ERokzg",  "token": "kXlWl32Yw3zGqylFJ3y3Y6uEXWG2ZpC-vRTRfQoOLbY",  "validationRecord": [    {      "url": "http://DOMAIN.com/.well-known/acme-challenge/kXlWl32Yw3zGqylFJ3y3Y6uEXWG2ZpC-vRTRfQoOLbY",      "hostname": "DOMAIN.com",      "port": "80",      "addressesResolved": [        "172.67.172.39",        "104.21.47.196",        "2606:4700:3034::6815:2fc4",        "2606:4700:3036::ac43:ac27"      ],      "addressUsed": "2606:4700:3034::6815:2fc4",      "resolverAddrs": [        "A:10.0.32.88:27567",        "AAAA:10.0.32.89:20459"      ]    }  ],  "validated": "2024-04-21T15:19:46Z"}
 
code 200
 
response status = invalid
getssl: DOMAIN.com:Verify error:    "detail": "During secondary validation: 2a06:98c1:3120::1: Invalid response from http://DOMAIN.com/.well-known/acme-challenge/kXlWl32Yw3zGqylFJ3y3Y6uEXWG2ZpC-vRTRfQoOLbY: 403",
 
Traceback
 
 main() line 3468 called
 
  fulfill_challenges() line 1525 called
 
   check_challenge_completion() line 546 called
 
    error_exit() line 1304 called traceback

[githubRover]

Do you block access by geographic region or countries? Because Let's Encrypt recently added two additional remote validation server locations. The "secondary validation" points to a problem with one of the 4 secondary sites (the 5th validation center is in the USA)
This has been a common topic on the Let's Encrypt community forum since this change
https://community.letsencrypt.org/t/lets-encrypt-is-adding-two-new-remote-perspectives-for-domain-validation/214123

[Nepherim]

That was the exact problem. I have spent days trying to track this down, and there is zero chance I would ever have considered this as the issue. Thanks so much for responding!

[Nepherim]

That did resolve the specific error, but now I'm getting:

- The certificate could not be installed on the domain “DOMAIN.com”.
- Certificate verification failed!  The system did not find the root certificate that corresponds to the supplied Certificate Authority Bundle’s intermediate certificate. Please supply a full Certificate Authority Bundle with the root certificate included.

[githubRover]

Where do you see that error?

If your last good cert was before Feb8 of this year I would guess that the system reporting the error does not have ISRG Root X1 certificate in its CA store. Is it an older system? On Feb8 the default chain from Let's Encrypt no longer includes the cross-signed DST Root CA X3 and so systems must trust ISRG Root X1.

Temporarily you can request the older "long chain" but this will soon be gone anyway. If this sounds possible see below.
https://community.letsencrypt.org/t/shortening-the-lets-encrypt-chain-of-trust/201580

[Nepherim]

The expired certificate was issued in January 2024.

"system reporting the error does not have ISRG Root X1 certificate in its CA store."
Is something the webhost needs to do?

Using the long chain option below didn't change the getssl output:

FULL_CHAIN_INCLUDE_ROOT="true"

That's an error from getssl. Full text below:

DOMAIN.com: remote cert expires sooner than local, attempting to upload from local
reloading SSL services
[2024-04-21 13:44:22 -0500] warn [uapi] Cpanel::Wrap::send_cpwrapd_request adminbin Cpanel/ssl/ADD: exit 5: namespace=[Cpanel] module=[ssl] function=[ADD]: raw_response=[{"mode":"full","statusmsg":"adminbin Cpanel/ssl/ADD: exit 5","status":1,"version":"2.4","data":{"message":"Certificate verification failed!  The system did not find the root certificate that corresponds to the supplied Certificate Authority Bundle’s intermediate certificate. Please supply a full Certificate Authority Bundle with the root certificate included.","statusmsg":"Certificate verification failed!  The system did not find the root certificate that corresponds to the supplied Certificate Authority Bundle’s intermediate certificate. Please supply a full Certificate Authority Bundle with the root certificate included.","action":"install","status":0,"html":"Certificate verification failed!  The system did not find the root certificate that corresponds to the supplied Certificate Authority Bundle’s intermediate certificate. Please supply a full Certificate Authority Bundle with the root certificate included."},"exit_code":1280,"error":1,"timeout":0,"action":"fetch"}]
[2024-04-21 13:44:22 -0500] warn [uapi] Cpanel::Wrap::send_cpwrapd_request error: namespace=[Cpanel] module=[ssl] function=[ADD]: statusmsg=[adminbin Cpanel/ssl/ADD: exit 5]
--- 
apiversion: 3
func: install_ssl
module: SSL
result: 
  data: 
    cert_id: DOMAIN_com_9dfc5_73007_1721488995_2cd368d44c64a395a76757dbfdce85cc
    key_id: 9dfc5_73007_5e08944e645ddef9d75418b8f918c2bb
  errors: 
    - The certificate could not be installed on the domain “DOMAIN.com”.
    - Certificate verification failed!  The system did not find the root certificate that corresponds to the supplied Certificate Authority Bundle’s intermediate certificate. Please supply a full Certificate Authority Bundle with the root certificate included.
  messages: ~
  metadata: {}

  status: 0
  warnings: ~
DOMAIN.com: certificate is valid for more than 30 days (until Jul 20 15:23:15 2024 GMT)

[githubRover]

The error is coming from cPanel. I am not expert at cPanel but you could try copy/paste the cert, chain, and private key yourself into your cPanel screen. You may need to take that up with your hosting service if that fails.

The message is a little puzzling in that it suggests adding the Root certificate to the chain. I didn't think modern cPanel systems require the root cert in the chain. I might be wrong or yours might need it.

The other possibility is the script you use to update cPanel needs updating. Perhaps it is manipulating the chain.pem file wrongly now that it is shorter than before.

Maybe someone else here will be able to help. Or, try the Let's Encrypt community forum.

[Nepherim]

I tried copy/pasting the certs into cpanel, but it basically throws the same error.

The script being used to update cpanel is the one in the repo cpanel_cert_upload.

I'll try over in the LE forum also. Thanks again for you help here.

[Nepherim]

UPDATE: removing the existing chain, fullchain, and DOMAIN.com.crt files from .gettssl/DOMAIN.com resolved the issue. Not entirely sure why, but once I did that everything worked and updated cpanel.