Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suddenly getting 1034 errors from Cloudflare #985

Open
hernil opened this issue Nov 20, 2024 · 45 comments
Open

Suddenly getting 1034 errors from Cloudflare #985

hernil opened this issue Nov 20, 2024 · 45 comments
Labels
bug Something isn't working as expected

Comments

@hernil
Copy link

hernil commented Nov 20, 2024

This night I started getting 1034 responses from Cloudflare. It's not every request - I've had roughly 20 of them between two domains since about 02:00 CET (UTC+1). I have the two instances of cloudflare-ddns (one per domain) pinging every five minutes.

The full errors look like this:

😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
😞 Failed to detect the IPv4 address
🔔 Pinged the "/fail" endpoint of Healthchecks
⏰ Checking the IP addresses in about 5m0s . . .

The Cloudflare docs say Error 1034: Edge IP Restricted which suggest some rate limiting, but that's not really what the description says as far as I can understand - more in the way of configuration issues.

Common cause
Customers who previously pointed their domains to 1.1.1.1 will now encounter 1034 error. This is due to a new edge validation check in Cloudflare's systems to prevent misconfiguration and/or potential abuse.
Resolution
Ensure DNS records are pointed to IP addresses you control, and in the case a placeholder IP address is needed for “originless” setups, use the IPv6 reserved address 100:: or the IPv4 reserved address 192.0.2.0.

I have not started trying fixes yet (not sure if I can override the DNS endpoint by config) but I wanted to get this out here in case others are experiencing the same thing.

Additional info in case it's useful:

  • My IP hasn't changed in this time and resolution works as expected
  • A simple dig from the same source IP resolves just fine (even manually spammed a few dozen times)
  • My Cloudflare API token is active and fine. Hasn't changed at all
  • I've been running this since October 2023 and it's the first time I'm seeing this (and I have Healthcheck monitoring for failures so I would've been notified)

I'll use the opportunity to say that it's a great little project you've made @favonia. It's clear from the image trimming and change logs that you take great care in maintaining this. Thanks a lot! 😊

@favonia favonia added the bug Something isn't working as expected label Nov 20, 2024
@favonia

This comment was marked as resolved.

@hernil

This comment was marked as resolved.

@favonia
Copy link
Owner

favonia commented Nov 20, 2024

@hernil To be honest, I'm not sure what we can do to work around this. The tool has completed the HTTP connection to obtain the content, except that the content includes the string "error code: 1034." How frequently are you checking your IP address? It's really hard for me to imagine that you could trigger Cloudflare's rate limit. My hunch is that maybe there are some internal hiccups within Cloudflare. Are you still getting the error now?

@tsrats
Copy link

tsrats commented Nov 20, 2024

For another data point, I am getting this as well from my 2 instances, one to the Boston, US Cloudflare COLO and one to the Melbourne, AUS COLO. And they are not consistent or at the same time as each other. My thought too is there are issues inside Cloudflare causing this.

@bpbradley
Copy link

I also started getting these starting yesterday. It is about once per hour if I had to guess.

@fiedinnet
Copy link

fiedinnet commented Nov 21, 2024

getting the same error here today. suspect something changes in cloudflare structure. but i am not really sure.

@andrewkolda
Copy link

same rip

@hernil
Copy link
Author

hernil commented Nov 21, 2024

@hernil To be honest, I'm not sure what we can do to work around this. The tool has completed the HTTP connection to obtain the content, except that the content includes the string "error code: 1034." How frequently are you checking your IP address? It's really hard for me to imagine that you could trigger Cloudflare's rate limit. My hunch is that maybe there are some internal hiccups within Cloudflare. Are you still getting the error now?

I'm checking every 5 minutes for two different domains hosted on the same machine / IP. At the time of writing they've had 38 and 41 errors respectively in the last ~31,5 hours so if my morning coffee has kicked in that gives (38+41)/(31,5x12x2) ~= 10,5% error rate. The errors do not show a timing pattern as far as I can see and as of now one of my domains have been fine for the last three hours while the other has been up and down1 about 6 or 7 times in that period.

Here's the truncated logs with failure timestamps for both domains:

logs
2024-11-20T01:00:13.047672731Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T02:32:42.049942301Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T02:50:13.015443263Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T03:02:42.046080128Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T03:25:13.017760084Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T05:07:42.118805452Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T05:12:42.115540698Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T05:17:42.074177801Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T05:25:13.019028578Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T06:02:42.046846896Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T06:25:13.045881240Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T06:42:42.051805255Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T06:47:42.079008241Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T06:57:42.025879436Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T07:10:13.103825019Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T07:25:13.103656359Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T07:40:13.074855397Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T07:57:42.074850645Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T08:15:13.106862384Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T08:22:42.030097645Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T09:15:13.126301622Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T09:52:42.020477597Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T11:05:13.056574350Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T11:22:42.025654821Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T11:32:42.055093800Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T11:37:42.044299203Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T12:00:13.094666320Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T13:05:13.051875519Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T13:10:13.057388045Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T13:25:13.098482084Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T13:30:13.028165483Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T13:50:13.086594299Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T14:20:13.059156591Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T14:40:13.027029186Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T14:47:42.041231933Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T14:55:13.049177690Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T14:57:42.023806775Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T15:22:42.084356328Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T15:30:13.050251712Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T15:47:42.120993932Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T15:52:42.067369363Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T16:37:42.056780715Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T17:05:13.058223522Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T17:47:42.088514299Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T18:37:42.055864354Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T18:40:13.018071238Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T18:52:42.070072764Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T19:25:13.030396978Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T19:45:13.120157110Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T20:02:42.083515581Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T20:05:13.060764240Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T20:30:13.051784856Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T20:57:42.043929710Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T21:02:42.108672754Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T21:42:42.045651500Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T21:45:13.031520396Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T22:02:42.057693248Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T22:17:42.051279321Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T22:47:42.048349370Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T22:50:13.025125409Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T23:02:42.074088978Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-20T23:07:42.019996016Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T00:07:42.046933321Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T00:22:42.043014159Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T00:50:13.016290297Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T01:07:42.104314706Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T01:22:42.014707940Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T01:37:42.017465305Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T02:05:13.049701250Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T03:12:42.078018007Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T04:10:13.045910667Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T05:42:42.015193889Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T05:45:13.019317510Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T05:55:13.017517548Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T06:10:13.116739062Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T06:45:13.046705581Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T07:00:13.017919944Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034
2024-11-21T07:15:13.019974636Z 😞 Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034

Cloudflare is only providing DNS for this setup btw. There's no tunnelig, proxying or other things being served through Cloudflare.

Let me know if there's other bits of information that could be useful!

Footnotes

  1. I say up and down but it's just the check failing. No service is going due to this.

@favonia
Copy link
Owner

favonia commented Nov 21, 2024

FYI: I added a FAQ in README about this error (#987) while we are still trying to figure it out.

@hernil
Copy link
Author

hernil commented Nov 21, 2024

Don't know if really relevant but manually browsing to https://1.1.1.1/cdn-cgi/trace I'm able to replicate the error message (below) from time to time, but not when going to https://one.one.one.one/cdn-cgi/trace

screenshots

image
image

The host (1.1.1.1) resolved to an IP address that the owner of the website does not have access to.

Is a really weird error as it looks like Cloudflare is trying to resolve an IP?

@favonia
Copy link
Owner

favonia commented Nov 21, 2024

I would guess that someone screwed up the code to check whether the "owner" of the IP 1.1.1.1 (as the host) is Cloudflare itself. Maybe the code does not handle the case where the host is an IP. Anyway, it's relatively easy to change the updater to use one.one.one.one, but someone might have already set up their firewalls to allow specific IP addresses so I would like to be extra careful (one.one.one.one could be resolved to 1.0.0.1 instead of 1.1.1.1). However, if this is not resolved quickly, maybe a change is needed...

@hernil
Copy link
Author

hernil commented Nov 21, 2024

That's a fair theory and I agree on not doing anything drastic before knowing more about the issue.

No idea if this is related: https://www.cloudflarestatus.com/incidents/l0fmd6ccwqz2 it was not

@bpbradley
Copy link

I would guess that someone screwed up the code to check whether the "owner" of the IP 1.1.1.1 (as the host) is Cloudflare itself. Maybe the code does not handle the case where the host is an IP. Anyway, it's relatively easy to change the updater to use one.one.one.one, but someone might have already set up their firewalls to allow specific IP addresses so I would like to be extra careful (one.one.one.one could be resolved to 1.0.0.1 instead of 1.1.1.1). However, if this is not resolved quickly, maybe a change is needed...

Could we make the default behavior point to '1.1.1.1' for the updater, but configurable to use one.one.one.one instead? Or for it to try 1.1.1.1 and then fallback to one.one.one.one on error? Just a thought for a workaround.

@bpbradley
Copy link

Don't know if really relevant but manually browsing to https://1.1.1.1/cdn-cgi/trace I'm able to replicate the error message (below) from time to time, but not when going to https://one.one.one.one/cdn-cgi/trace

screenshots
image image

The host (1.1.1.1) resolved to an IP address that the owner of the website does not have access to.

Is a really weird error as it looks like Cloudflare is trying to resolve an IP?

Perhaps it has just been resolved? I just tested https://1.1.1.1/cdn-cgi/trace and I am not receiving the 1034 error. Refreshed about 100 times. I last had an issue about 20 minutes ago though. Time will tell.

@fiedinnet
Copy link

https://1.1.1.1/cdn-cgi/trace

from my side. issue still persist.

@bpbradley
Copy link

https://1.1.1.1/cdn-cgi/trace

from my side. issue still persist.

Weird, I am not getting it at all from that link. But I am still getting it from cloudflare-ddns. So yeah definitely not fixed.

@favonia
Copy link
Owner

favonia commented Nov 21, 2024

Inspired by @bpbradley's suggestion, I implemented a temporary feature (#988) to hopefully shut down the error messages:

  1. Use favonia/cloudflare-ddns:edge (development version) instead of favonia/cloudflare-ddns:latest. (In general, using the edge image is not recommended for production servers!)
  2. Use cloudflare.trace:URL, for example cloudflare.trace:https://one.one.one.one/cdn-cgi/trace, to replace the default URL.

I'm a bit reluctant to make this feature permanent yet. The code is good, but the design is not well-thought-out. Thanks to everyone who helped me diagnose the problem. Let's hope Cloudflare can just fix it.

@bpbradley
Copy link

bpbradley commented Nov 21, 2024

Inspired by @bpbradley's suggestion, I implemented a temporary feature (#988) to hopefully shut down the error messages:

  1. Use favonia/cloudflare-ddns:edge (development version) instead of favonia/cloudflare-ddns:latest. (In general, using the edge image is not recommended for production servers!)
  2. Use cloudflare.trace:URL, for example cloudflare.trace:https://one.one.one.one/cdn-cgi/trace, to replace the default URL.

I'm a bit reluctant to make this feature permanent yet. The code is good, but the design is not well-thought-out. Thanks to everyone who helped me diagnose the problem. Let's hope Cloudflare can just fix it.

Well done! I will give it a try now.

Edit: Updated. I will check back in a few hours to verify that it worked.

Edit: Haven't had an issue in 5 hours. I'm thinking this worked.

@hernil
Copy link
Author

hernil commented Nov 22, 2024

Applied the edge channel to one of my two instances about six hours ago and I've not seen that one error out since 😊

@fiedinnet
Copy link

Inspired by @bpbradley's suggestion, I implemented a temporary feature (#988) to hopefully shut down the error messages:

  1. Use favonia/cloudflare-ddns:edge (development version) instead of favonia/cloudflare-ddns:latest. (In general, using the edge image is not recommended for production servers!)
  2. Use cloudflare.trace:URL, for example cloudflare.trace:https://one.one.one.one/cdn-cgi/trace, to replace the default URL.

I'm a bit reluctant to make this feature permanent yet. The code is good, but the design is not well-thought-out. Thanks to everyone who helped me diagnose the problem. Let's hope Cloudflare can just fix it.

this solve my problem. thank you.

@Spyderkat
Copy link

where do i put this cloudflare.trace:https://one.one.one.one/cdn-cgi/trace, this error popped up for me also about 4 days ago

@bpbradley
Copy link

where do i put this cloudflare.trace:https://one.one.one.one/cdn-cgi/trace, this error popped up for me also about 4 days ago

- IP4_PROVIDER=cloudflare.trace:https://one.one.one.one/cdn-cgi/trace

@Spyderkat
Copy link

thanks @bpbradley

@tsrats
Copy link

tsrats commented Nov 27, 2024

Hopefully unrelated, I switched to the edge image and set my IP4_PROVIDER last night, and today for the first time got the following error:

😞 Failed to send HTTP(S) request to "https://one.one.one.one/cdn-cgi/trace": GET https://one.one.one.one/cdn-cgi/trace giving up after 1 attempt(s): context deadline exceeded
😞 Failed to detect the IPv4 address
💡 If your network is experiencing high latency, consider increasing DETECTION_TIMEOUT=5s

I generally do not have high latency, nothing else changed on my network, so not sure if one.one.one.one might respond differently (when working) than 1.1.1.1 ?

@hansenc0705
Copy link

Was hoping by now CF would have things fixed but maybe this is the new norm. Is there a config to change how often it polls in the off chance it is some sort of rate limiting though i cant see how for a service like this 5 minutes is too often.

@hansenc0705
Copy link

Maybe another option would be to use a different Cloudflare service for IP lookup?

Cloudflare owns icanhazip.com as well. Not the best or most professional URL but gets the job done.

https://major.io/icanhazip-com-faq/

Guergeiro added a commit to Guergeiro/iac that referenced this issue Nov 28, 2024
Change provider in an attempt to fix missing ipv4 domain records. This
is being tracked here: favonia/cloudflare-ddns#985
@apat183
Copy link

apat183 commented Nov 29, 2024

Was hoping by now CF would have things fixed but maybe this is the new norm. Is there a config to change how often it polls in the off chance it is some sort of rate limiting though i cant see how for a service like this 5 minutes is too often.

Set the env variable e.g - UPDATE_CRON=@hourly

I have done this, i still get the error but not as often, like once a day by reducing it to hourly. Not a fix but just helps alerts instead of when it was 5 mins and failing a lot.

It's documented nicely here: https://github.com/favonia/cloudflare-ddns?tab=readme-ov-file#%EF%B8%8F-all-settings under "Scheduling of IP detections and updates" for more options or explaining how to pick cron friendly names if you want to set something else.

@hansenc0705
Copy link

Thanks @apat183 went looking but completely missed it. While poking around I saw the option to change and use a different URL so I'm giving that a try now with icanhazip to see how that works. So far so good, it at least detected the IP.

@GIG0i
Copy link

GIG0i commented Nov 30, 2024

I had the same problem reported in this issue and have tried manually switching the url as suggested.
While this seamed to have worked initially, this morning I found the following error which also triggered a few times more (annoyingly 😞):

😞 Failed to send HTTP(S) request to "https://api4.ipify.org": GET https://api4.ipify.org giving up after 1 attempt(s): context deadline exceeded

Ideally I don't want to increase the timeout value, alternatively is it possible to configure retry attempts before giving up?

Apologies if I missed it in the documentation.

@hansenc0705
Copy link

hansenc0705 commented Nov 30, 2024

Had the same today

Failed to send HTTP(S) request to "https://icanhazip.com": GET https://icanhazip.com giving up after 1 attempt(s): context deadline exceeded
😞 Failed to detect the IPv4 address
💡 If your network is experiencing high latency, consider increasing DETECTION_TIMEOUT=5s

@favonia
Copy link
Owner

favonia commented Dec 1, 2024

@tsrats @GIG0i @hansenc0705 Thanks for the reporting. It's curious that multiple of you experienced high latency simultaneously with different servers. Was there a global network slowdown? How frequently the failures are?

@GIG0i The "retry" is for the HTTP server to instruct the updater to try again later. The updater will comply with such retry instructions. However, that doesn't seem to be the problem here. In this case, it seems the issue is that the network isn't working as expected, and thus retrying might not be helpful. Increasing the timeout value won't slow down the normal detection when everything works as expected, and the value is really for the worst-case scenario. Therefore, if you want the tool to retry, it indicates that you're actually okay with a large timeout value. It will make the updater wait longer before giving up.

An interesting idea that just came to my mind is whether a failed detection should trigger a shorter wait before the next detection. I suppose it's possible but requires careful design.

So far, it seems there have been many unfortunate events (Cloudflare's internal errors, network latency, etc.), and I feel sorry that you've had to endure them. On the other hand, I haven't seen anything that should be changed in the tool yet.

@favonia favonia pinned this issue Dec 1, 2024
@fiedinnet
Copy link

Just to let you know, i remove the old image and tried to reinstall the default latest image - and its working. I wonder whats Cloudflare doing in the background.

@favonia
Copy link
Owner

favonia commented Dec 3, 2024

@hernil @tsrats @bpbradley @andrewkolda @Spyderkat @GIG0i I can no longer trigger the error. It seems Cloudflare just fixed it. Could you possibly revert back to the latest image with your original setting, if you have tried the edge image? With a few more confirmations I will close this issue. 😃

@tsrats
Copy link

tsrats commented Dec 3, 2024

Hi @favonia, I reverted back to my prior set up, using latest (specifically 1.15.0) tag and 1.1.1.1 as the provider, and got a failure within 2 hours:

Failed to find the IP address in the response of "https://1.1.1.1/cdn-cgi/trace": error code: 1034

@hansenc0705
Copy link

So I switched for a few days to just URL and used icanhazip and that was fine, no errors. Today i switched to cloudflare.doh and its been OK so far.

@bpbradley
Copy link

@hernil @tsrats @bpbradley @andrewkolda @Spyderkat @GIG0i I can no longer trigger the error. It seems Cloudflare just fixed it. Could you possibly revert back to the latest image with your original setting, if you have tried the edge image? With a few more confirmations I will close this issue. 😃

I reverted and got an error within about 10 minutes :(

@favonia
Copy link
Owner

favonia commented Dec 3, 2024

(Just to clarify, cloudflare.doh uses a different (unofficial) protocol to get your IP from Cloudflare.)

@hansenc0705
Copy link

(Just to clarify, cloudflare.doh uses a different (unofficial) protocol to get your IP from Cloudflare.)

Ah good to know. Will switch back to the url method with icanhazip then.

@fiedinnet
Copy link

fiedinnet commented Dec 4, 2024

Okay, the issue is back.
works if using cloudflare.doh.
right now i think will use the IP url method

@favonia
Copy link
Owner

favonia commented Dec 9, 2024

Finally, I got my error 1034 as well. As this issue has remained open for three weeks, I am leaning towards changing the default cloudflare.trace URL to this one:

https://api.cloudflare.com/cdn-cgi/trace

The reason is that we will be contacting the API server anyways. As for cloudflare.doh, the default URL could switch to

https://cloudflare-dns.com/dns-query

After these changes, I believe we can drop all the complicated code about 1.1.1.1 and 1.0.0.1 🎉

Hope these changes will solve the problems with minimum friction.

@tsrats
Copy link

tsrats commented Dec 9, 2024

Finally, I got my error 1034 as well. As this issue has remained open for three weeks, I am leaning towards changing the default cloudflare.trace URL to this one:

https://api.cloudflare.com/cdn-cgi/trace

The reason is that we will be contacting the API server anyways. As for cloudflare.doh, the default URL could switch to

https://cloudflare-dns.com/dns-query

After these changes, I believe we can drop all the complicated code about 1.1.1.1 and 1.0.0.1 🎉

Hope these changes will solve the problems with minimum friction.

I will certainly be willing to test this for you when it's ready, I've been seeing errors almost hourly still (sometimes multiple an hour)

@favonia
Copy link
Owner

favonia commented Dec 10, 2024

I will certainly be willing to test this for you when it's ready, I've been seeing errors almost hourly still (sometimes multiple an hour)

I have changed the default URLs in the development version. The edge Docker tag (the development version) should run without any errors. If enough people can confirm this, I will make a minor release after cleaning up the code.

@tsrats
Copy link

tsrats commented Dec 10, 2024

Just updated both of my images to edge, will report back in a day or so if they are stable or not.

@favonia
Copy link
Owner

favonia commented Dec 11, 2024

Is there anyone testing cloudflare.doh with the edge tag?

@bpbradley
Copy link

The reason is that we will be contacting the API server anyways. As for cloudflare.doh, the default URL could switch to

I am on edge with default settings (no IP4_PROVIDER set) and have not had an issue since changing (over 24 hours).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working as expected
Projects
None yet
Development

No branches or pull requests

10 participants