Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL cert not updating #400

Open
tactmaster opened this issue Feb 8, 2023 · 46 comments
Open

SSL cert not updating #400

tactmaster opened this issue Feb 8, 2023 · 46 comments

Comments

@tactmaster
Copy link

Problem/Motivation

SSL Certs not getting update from lets encrypt

Expected behavior

Lets encrypt ssl cert being automatically updated

Actual behavior

ssl certs not being updated. Is you press renew get this error
image

Steps to reproduce

image

Proposed changes

@SoulOfNoob
Copy link

I had the same issue today.
It turned out, that my Pi got a new IP address and my port forwarding pointed to the wrong one.

I had to login to the Pi via SSH and open "/data/logs/letsencrypt/letsencrypt.log" inside the proxymanager docker container to find the exact issue was:

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

@tactmaster
Copy link
Author

This error I got

Some of certs are renewing some are not it is odd

  Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-2" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[08/Feb/2023:21:15:18 +0100] - 200 200 - GET https aaa.bbb.com "/api/config" [Client 10.0.0.153] [Length 6152] [Gzip -] [Sent-to 192.168.1.75] "-" "-"
[2/8/2023] [9:15:19 PM] [Express  ] › ⚠  warning   Command failed: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-2" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
Another instance of Certbot is already running.

@tactmaster
Copy link
Author

tactmaster commented Feb 15, 2023

Things I tried.
Resetting the database
Uninstalling and restoring from backup.

Deleting the certificate and asking for it again from the UI worked

@eric10k93
Copy link

I'm getting the same "Internal Error" simply when trying to create a brand new SSL cert. I was also getting the error "Another instance of Certbot is already running". I just rebooted my system to get it to shut down all processes... as obviously there was a stuck Certbot instance. After a restart, I wasn't getting the same error... Now I am just getting "Some challenges have failed" from Let's Encrypt and it's unable to generate a new certificate for me. I'm at a loss. If I use Let's Encrypt via the DuckDNS add on, it generates the SSL certs for me just fine and I am then able to reference them via NPM by adding them as "Custom" SSL certs. This works, but it's obviously not ideal because I then have to manually renew the certs anytime they come up for renewal.

@awdark
Copy link

awdark commented Mar 23, 2023

DuckDNS add on, it generates the SSL certs for me just fine and I am then able to reference them via NPM by adding them as "Custom" SSL certs. This works, but it's obviously not ideal because I then have to manually renew the certs anytime they come up for renewal.

Would you mind sharing your process? I have the DuckDNS add on but I don't know where it stores the certs so I can add them as custom. My cert expired and I want to keep it going until it hopefully gets patched.

I may be digging myself into a bigger hole tried to add homeassistant.something.duckdns.org
ended up with ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect TXT record "dns-01" found at _acme-challenge.homeassistant.something.duckdns.org","status":403}

I noticed in duckdns documentation the folder \ssl\ is supposed to contain the certificates but they are dated December and I tried to just install those anyways and got an error about using those certificates so I don't think those are the right ones.

EDIT
Problem in chair.
Still not 100% sure what fixed it, but I opened up my firewall and reverted the port number for port 80 to port 80.

@TerrorBite
Copy link

Also experiencing this issue.

Addon Version: 0.12.3

Upon attempting to create or renew a cert:

[4/10/2023] [3:23:18 AM] [SSL      ] › ℹ  info      Testing http challenge for DOMAIN.REDACTED
Uncaught SyntaxError: Unexpected end of JSON input
FROM
2023/04/10 03:23:19 [error] 298#298: *3 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.4, server: nginxproxymanager, request: "GET /api/nginx/certificates/test-http?domains=%5B%22DOMAIN.REDACTED%22%5D HTTP/1.1", upstream: "http://127.0.0.1:3000/nginx/certificates/test-http?domains=%5B%22DOMAIN.REDACTED%22%5D", host: "10.10.0.200:81", referrer: "http://10.10.0.200:81/nginx/certificates"
[03:23:20] INFO: Nginx Proxy Manager stopped, restarting...
[03:23:20] INFO: Starting the Manager...

Following this crash and restart…

[4/10/2023] [3:23:34 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[4/10/2023] [3:23:39 AM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #2: DOMAIN.REDACTED
[4/10/2023] [3:23:39 AM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-2" --agree-tos --authenticator webroot --email "EMAIL@REDACTED" --preferred-challenges "dns,http" --domains "DOMAIN.REDACTED" 
[4/10/2023] [3:23:44 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[4/10/2023] [3:23:44 AM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-2" --agree-tos --authenticator webroot --email "EMAIL@REDACTED" --preferred-challenges "dns,http" --domains "DOMAIN.REDACTED" 
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-j7u3fnai/log or re-run Certbot with -v for more details.
[4/10/2023] [3:24:05 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-1 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
    at ChildProcess.exithandler (node:child_process:400:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1093:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

After this point all attempted Let's Encrypt operations continue to fail.

@github-actions
Copy link

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label May 10, 2023
@edmundwatson
Copy link

It is still a problem.

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label May 17, 2023
@fgibaux
Copy link

fgibaux commented Jun 2, 2023

I believe that's more of an upstream issue, cf NginxProxyManager/nginx-proxy-manager#2881

@github-actions
Copy link

github-actions bot commented Jul 3, 2023

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Jul 3, 2023
@SiriosDev
Copy link

Absolutely not stale

@awdark
Copy link

awdark commented Jul 4, 2023

Do you run a firewall?
It seems like some zone protection or other filters may block these requests. Strangely they didn't show up on the logs as far as I could tell but running with everything on alert made it work smoothly.

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Jul 4, 2023
@Scags104
Copy link

Scags104 commented Jul 24, 2023

Same issue here but deleting, trying to reissue causes more problems. I should have let them expire on their own, lost a month now...

Error: Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-12" --agree-tos --authenticator webroot --email "[email protected]" --preferred-challenges "dns,http" --domains "rondemealie.duckdns.org" 
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-kuhouk5g/log or re-run Certbot with -v for more details.

    at ChildProcess.exithandler (node:child_process:400:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1093:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

@github-actions
Copy link

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Aug 24, 2023
@Scags104
Copy link

Still a problem

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Aug 25, 2023
@geigervlad
Copy link

just had the same issue. For me the issue was misconfiguration under router: port 80 was not forwarded/open.

Certmanager uses Port 80 to execute the HTTP check. Make sure it is open and forwarding to the correct machine

@SiriosDev
Copy link

just had the same issue. For me the issue was misconfiguration under router: port 80 was not forwarded/open.

Certmanager uses Port 80 to execute the HTTP check. Make sure it is open and forwarding to the correct machine

this issue talks about problems when enabling forcing on https, the problem is that http verification is also being forced on 443 sending the whole thing into error

@github-actions
Copy link

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Sep 27, 2023
@Scags104
Copy link

still an issue

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Sep 28, 2023
@sulazix
Copy link

sulazix commented Sep 29, 2023

I have the same issue

@eric10k93
Copy link

I feel kinda dumb coming back to this after all this time having issues... but I think I figured out my problem... I've been using this with a DuckDNS domain and it appears it needs a DNS challenge... the DuckDNS addon for HA handles the challenge with the supplied token, which is why I never saw any issues with that addon getting a valid cert. Basically, the problem boils down to there being poor documentation for both the DuckDNS addon and this NPM addon. It was never clear to me that I NEEDED to do the DNS challenge... but it seems obvious now. Basically, I stopped using the DuckDNS addon... I no longer use a custom cert in NPM. Just have NPM create and manage the certs and use the appropriate DNS challenge for your DDNS provider. This appears to be working fine for me this way. Time will tell if the certs update properly when needed, but they should now that they have the appropriate token.

@Scags104
Copy link

Scags104 commented Oct 5, 2023

I feel kinda dumb coming back to this after all this time having issues... but I think I figured out my problem... I've been using this with a DuckDNS domain and it appears it needs a DNS challenge... the DuckDNS addon for HA handles the challenge with the supplied token, which is why I never saw any issues with that addon getting a valid cert. Basically, the problem boils down to there being poor documentation for both the DuckDNS addon and this NPM addon. It was never clear to me that I NEEDED to do the DNS challenge... but it seems obvious now. Basically, I stopped using the DuckDNS addon... I no longer use a custom cert in NPM. Just have NPM create and manage the certs and use the appropriate DNS challenge for your DDNS provider. This appears to be working fine for me this way. Time will tell if the certs update properly when needed, but they should now that they have the appropriate token.

I just tried this. Disdabled DuckDNS, NPM began working when forcing a challenge and using my DuckDNS token
All seems to be ok for the moment!

Thanks for this

@dubhunter
Copy link

#462

For me, I can manually update fine. What does NOT work is auto-update.

Copy link

github-actions bot commented Dec 4, 2023

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Dec 4, 2023
@dubhunter
Copy link

Any help here?

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Dec 5, 2023
Copy link

github-actions bot commented Jan 5, 2024

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Jan 5, 2024
@tactmaster
Copy link
Author

tactmaster commented Jan 5, 2024 via email

@mathwizmf2009
Copy link

Yeah, my certs failed to renew this month. Not sure why, I'm hoping to dig into it more today. Hitting the button to manually renew worked on 2 of them, but not on my 3rd.

@kaffeepause07
Copy link

Do you use the "Force SSL" option in the ssl settings?
This was the cause in my homelab, i was able to fix it with a workaround. (custom force-ssl.conf)
For testing you can disable "Force SSL" temporary.

@akrigator
Copy link

I have faced with same issue: can create a certificate, but it fails to renew after 2 months, with error like: Failed to renew certificate npm-2 with error: The DNS response does not contain an answer to the question
In my case issue was in the local DNS setup. It has a special cname record for resolve services by NPM domain names in LAN, which uses the ^(.*\.)?mydomain\.duckdns\.org$ regex. The regex picks requests to mydomain.duckdns.org and *.mydomain.duckdns.org wildcard, but Let's Encrypt uses the _acme-challenge.mydomain.duckdns.org to verify domain owner by placing a token to TXT record. Let's Encrypt sends a request from LAN during certificate renewal process, the request goes to the local DNS where TXT record does not have the token to validate owner, since TXT is updated on DuckDNS.
So the fix for me was to exclude the _acme-challenge subdomain from resolving scope of local DNS:
^(?!_acme-challenge)(.*\.)?mydomain\.duckdns\.org$

@mathwizmf2009
Copy link

mathwizmf2009 commented Jan 5, 2024

I have faced with same issue: can create a certificate, but it fails to renew after 2 months, with error like: Failed to renew certificate npm-2 with error: The DNS response does not contain an answer to the question In my case issue was in the local DNS setup. It has a special cname record for resolve services by NPM domain names in LAN, which uses the ^(.*\.)?mydomain\.duckdns\.org$ regex. The regex picks requests to mydomain.duckdns.org and *.mydomain.duckdns.org wildcard, but Let's Encrypt uses the _acme-challenge.mydomain.duckdns.org to verify domain owner by placing a token to TXT record. Let's Encrypt sends a request from LAN during certificate renewal process, the request goes to the local DNS where TXT record does not have the token to validate owner, since TXT is updated on DuckDNS. So the fix for me was to exclude the _acme-challenge subdomain from resolving scope of local DNS: ^(?!_acme-challenge)(.*\.)?mydomain\.duckdns\.org$

This is interesting. This seems to have been related to my issue, but I'm not 100% sure how to fix it properly. My DNS rewrite only handles mysub.mydomain.duckdns.org (rather than *.mysub.mydomain.duckdns.org). I have a separate rewrite for mydomain.duckdns.org. My AdGuard Home addon is handling my DNS rewrites. I could see the DNS requests for mysub.mydomain.duckdns.org (no reference to _acme-challenge listed, but clearly labeled as a TXT request). I deleted the mysub.mydomain.duckdns.org rewrite just to test and manually refreshed the cert (which worked).

The more interesting thing is that my cert for mydomain.duckdns.org was able to manually refresh even with its rewrite still in place. All doing the DNS-01 challenge. But I don't even see the mydomain.duckdns.org TXT DNS request in the AdGuard logs.

When I test a ping to <something>.mysub.mydomain.duckdns.org, it doesn't route to my LAN, but rather to my public IP. I also see that specific request, including <something> within AdGuard. It isn't referencing an _acme-challenge sub-sub(?) domain within the AdGuard logs during renewal, and if it did then it shouldn't be routed to my LAN anyway based on my ping testing. Maybe I can change the Type of requests that are rewritten? I'm inexperienced with the custom filtering rules, so it looks like I'll need to do a bit more research into that side of things. Maybe I'm implementing my DNS rewrites poorly, or maybe I'm making totally incorrect assumptions based on my ping testing due to it being a different type of DNS request?

@akrigator
Copy link

This is interesting. This seems to have been related to my issue, but I'm not 100% sure how to fix it properly. My DNS rewrite only handles mysub.mydomain.duckdns.org (rather than *.mysub.mydomain.duckdns.org). I have a separate rewrite for mydomain.duckdns.org. My AdGuard Home addon is handling my DNS rewrites. I could see the DNS requests for mysub.mydomain.duckdns.org (no reference to _acme-challenge listed, but clearly labeled as a TXT request). I deleted the mysub.mydomain.duckdns.org rewrite just to test and manually refreshed the cert (which worked).

The more interesting thing is that my cert for mydomain.duckdns.org was able to manually refresh even with its rewrite still in place. All doing the DNS-01 challenge. But I don't even see the mydomain.duckdns.org TXT DNS request in the AdGuard logs.

Here a lot of things to check:

  1. Make sure that there is no another DNS in use. In my router got 2 additional DNS from ISP which were propagated to all DHCP clients. In result: requests were balanced between AdGuard and ISP DNSs
  2. Flush DNS cache in AdGuard after updating rules
  3. Flush DNS cache in router if it is used as proxy for AdGuard

When I test a ping to <something>.mysub.mydomain.duckdns.org, it doesn't route to my LAN, but rather to my public IP. I also see that specific request, including <something> within AdGuard. It isn't referencing an _acme-challenge sub-sub(?) domain within the AdGuard logs during renewal, and if it did then it shouldn't be routed to my LAN anyway based on my ping testing. Maybe I can change the Type of requests that are rewritten? I'm inexperienced with the custom filtering rules, so it looks like I'll need to do a bit more research into that side of things. Maybe I'm implementing my DNS rewrites poorly, or maybe I'm making totally incorrect assumptions based on my ping testing due to it being a different type of DNS request?

Try to monitor dig mydomain.duckdns.org TXT from lan and internet during renewal.
And as I know renewal in NPM will actually renew certificate only if it older then 2 month, unfortunately i don't know how to force it.

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Jan 6, 2024
Copy link

github-actions bot commented Feb 5, 2024

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Feb 5, 2024
@SiriosDev
Copy link

mope

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Feb 6, 2024
@mathwizmf2009
Copy link

For anyone else using AdGuard Home alongside their NPM and having issues with renewing their SSL certs, it might be that your AdGuard Home is doing a DNS rewrite of your TXT requests. You may be able to see this within your AdGuard logs. For me, I didn't see the _acme-challenge subdomain within the logs, but I DID see the Type: TXT. More on DNS-01 challenges

To solve this, I needed to remove the DNS rewrite from the Filters->DNS rewrites section and make a custom rule (Filters->Custom filtering rules). The Syntax for this is a bit annoying if you aren't used to it (like me), but after some trial and error plus reading the docs, I was able to come up with a rule to work for me that I couldn't find elsewhere:

|example.com^$dnsrewrite=1.2.3.4,dnstype=A|CNAME|AAAA

I didn't want to rewrite all subdomains, so the single pipe at the beginning | resolves this. If you want to include all subdomains, I believe you can use a double-pipe ||. I tried following the docs where it says |example.com^$dnsrewrite=NOERROR;A;1.2.3.4 adds an A record with the value 1.2.3.4, thinking it would only rewrite A records, but this wasn't the case and it rewrote all other types to NOERROR. The dnstype Rule Modifier appended on the end did work though!

Since I just upgraded my NPM add-on and applied these rules, I don't know yet if I'll have issues when my certs try to auto-renew in a couple months. Hopefully the combination of the add-on update and fixing my local DNS finally resolves this for me.

Copy link

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label Apr 14, 2024
@tactmaster
Copy link
Author

tactmaster commented Apr 14, 2024 via email

@Blogshot
Copy link

Blogshot commented Apr 14, 2024

I believe that's more of an upstream issue, cf NginxProxyManager/nginx-proxy-manager#2881

Yes, it is. The reason is that the "Force SSL" option will redirect the HTTP challenges to HTTPS, which the integrated certbot doesnt support apparently. A pull request is open at NginxProxyManager/nginx-proxy-manager#3121, but the fix hasn't been merged yet despite being available for ~8 months.

In the meantime, you can disable "Force SSL" for each proxy host you need to renew.

@tactmaster
Copy link
Author

tactmaster commented Apr 15, 2024 via email

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label Apr 15, 2024
Copy link

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

@github-actions github-actions bot added the stale There has not been activity on this issue or PR for quite some time. label May 15, 2024
@sulazix
Copy link

sulazix commented May 16, 2024

Unstale :-)
Still needed to disable "force SSL" to work with DNS challenge.

@dubhunter
Copy link

Still a problem. But for me, only auto-update. Manual works fine even with Force SSL.

@github-actions github-actions bot removed the stale There has not been activity on this issue or PR for quite some time. label May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests