How to check Reddit links?

[j-hui]

I set up a GitHub action to check links on my personal site, which happens to contain a Reddit link. Annoyingly, it fails with a 403, and I'm not sure what to do with it.

I've reproduced this issue in a test repo; the Lychee job generated this report: j-hui/lychee-test#2. Note that both valid and invalid links generate a 403, meaning I can't just filter out 403 responses or something like that.

Even worse, Lychee seems to report the invalid Reddit link as OK when I run lychee locally:

$ lychee --verbose README.md
✔ [200] file:///Users/j-hui/Documents/personal/lychee-test/files/file.txt
✔ [200] https://github.com/j-hui/lychee-test
✔ [200] https://www.reddit.com/r/pics/comments/deadbeef/this_post_does_not_exist/
✔ [200] https://lkml.org/lkml/2022/9/19/1105
✔ [200] https://www.reddit.com/r/pics/comments/92dd8/test_post_please_ignore/
  5/5 ETA 0s ████████████████████ Finished extracting links
🔍 5 Total ✅ 5 OK 🚫 0 Errors

Does anyone know how I can reliably check the validity of Reddit links? This is probably more specific to what Reddit's servers are doing, but I wasn't sure where to ask about that so I figured I'd start here. Thanks in advance!

[mre]

Sorry for the late response.
I don't know the answer.

Just double-checked in my browser.
Indeed, this link shows up as 404 in the network tab in Firefox:
https://www.reddit.com/r/pics/comments/deadbeef/this_post_does_not_exist/

However, it does print OK with lychee.
Curiously, it also shows a 200 OK with curl:

> curl -L -I https://www.reddit.com/r/pics/comments/deadbeef/this_post_does_not_exist/
curl -L -I https://www.reddit.com/r/pics/comments/deadbeef/this_post_does_not_existasdf/
HTTP/2 302
retry-after: 0
set-cookie: rdt=acfaa0a9f42ae897b82590c0ef59c353; path=/; domain=.reddit.com; samesite=none; secure;
location: /r/pics/comments/deadbeef/this_post_does_not_existasdf/?rdt=39868
content-type: text/html; charset=utf-8
accept-ranges: bytes
date: Fri, 05 Jan 2024 13:38:36 GMT
via: 1.1 varnish
strict-transport-security: max-age=31536000; includeSubdomains
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
set-cookie: edgebucket=rTJsKzExN6lGcFv5ow; Domain=reddit.com; Max-Age=63071999; Path=/;  secure
set-cookie: wwrbucket=f16e56c5cceff5e2d4a12eb7069bc358a70a7850f69ed2e2ebad3e924cd339c8; Max-Age=15
cache-control: private, max-age=3600
server: snooserv
report-to: {"group": "w3-reporting-nel", "max_age": 14400, "include_subdomains": true,  "endpoints": [{ "url": "https://w3-reporting-nel.reddit.com/reports" }]}, {"group": "w3-reporting", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting.reddit.com/reports" }]}, {"group": "w3-reporting-csp", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting-csp.reddit.com/reports" }]}
nel: {"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 1.0, "failure_fraction": 1.0}
content-length: 413

HTTP/2 200
content-security-policy: child-src 'self' blob: accounts.google.com;connect-src 'self' events.redditmedia.com o418887.ingest.sentry.io *.redd.it *.reddit.com www.redditstatic.com vimeo.com alb.reddit.com accounts.google.com/gsi/ w3-reporting.reddit.com reddit-uploaded-media.s3-accelerate.amazonaws.com reddit-uploaded-video.s3-accelerate.amazonaws.com reddit-subreddit-uploaded-media.s3-accelerate.amazonaws.com wss://*.wss.redditmedia.com wss://gql-realtime.reddit.com api.giphy.com js.stripe.com;default-src 'self';font-src 'self' data:;form-action 'none';frame-ancestors 'self' *.reddit.com *.snooguts.net;frame-src 'self' www.reddit.com www.youtube-nocookie.com player.vimeo.com *.redditmedia.com cdn.embedly.com redgifs.com www.redgifs.com accounts.google.com/gsi/ www.google.com/recaptcha/ recaptcha.google.com/recaptcha/ js.stripe.com hooks.stripe.com;img-src 'self' data: blob: https:;manifest-src 'self' www.redditstatic.com;media-src 'self' blob: data: *.redd.it www.redditstatic.com;object-src 'none';script-src 'self' 'unsafe-inline' 'unsafe-eval' www.redditstatic.com www.google.com/recaptcha/ js.stripe.com 'nonce-Z3Ma4JNfoKYYXvtW/DwSgA==';style-src 'self' 'unsafe-inline' www.redditstatic.com *.reddit.com accounts.google.com/gsi/style;style-src-attr 'unsafe-inline';worker-src blob:;report-to csp;report-uri https://w3-reporting-csp.reddit.com/reports
content-security-policy-report-only: object-src 'none';script-src 'nonce-Z3Ma4JNfoKYYXvtW/DwSgA==' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https:;frame-ancestors 'self' *.reddit.com *.snooguts.net;base-uri 'self';report-to csp;report-uri https://w3-reporting-csp.reddit.com/reports
x-is-wrs: false
cache-control: private, s-maxage=1, max-age=1, must-revalidate
content-type: text/html; charset=utf-8
accept-ranges: bytes
date: Fri, 05 Jan 2024 13:38:36 GMT
via: 1.1 varnish
vary: Accept-Encoding,accept-language
strict-transport-security: max-age=31536000; includeSubdomains
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
set-cookie: loid=000000000rfxw25vfv.2.1704461916696.Z0FBQUFBQmxtQVpjUllNSTl1MHdONUxNWG5rY0FsdVc4VlhmeHZIcDF4T0hsUkN5VHdSWjl1N1dmWXpmQkVsdHZBQndrMXpQck5KZ2swazJ2SURWc1AtaUlLd3E3WEhIaXFLcERvODVzcjUyV0c2QkZGZkJ0RHZ3SElNZTJtM2c2WXBoaGZxZjllTGQ; path=/; expires=Sun, 04 Jan 2026 13:38:36 GMT; domain=.reddit.com; samesite=none; secure
set-cookie: session_tracker=rdgoflmbiraifkhrpo.0.1704461916700.Z0FBQUFBQmxtQVpjNzQ5UER5U3lYVEZrQWJMQ0tXZGZFa3JvWHotaWxGMHZfbXRJOTZFajd4VWxhUWxScFA2cU94QXRRYWtiS010blhlNE1pcFpLREpzRkJlTU5DSXUtTGdxdE55MXRrOEFGQnhIN0ljR3BWN2Z0Z2EzX281b1laX1g3NFhGTTRrN1E; path=/; domain=.reddit.com; secure; SameSite=None; Secure
set-cookie: csrf_token=23c36bfc338d79cbd743949df9449121; path=/; domain=.reddit.com; samesite=strict; secure
set-cookie: token_v2=eyJhbGciOiJSUzI1NiIsImtpZCI6IlNIQTI1Njp2am9FL2w2SkFOeWxWK2VPcG9ySWRzcm5sVzRSTXgzMVVQaS8rQkNnUkdZIiwidHlwIjoiSldUIn0.eyJzdWIiOiJsb2lkIiwiZXhwIjoxNzA0NTQ4MzE2LjY5NjY0MywiaWF0IjoxNzA0NDYxOTE2LjY5NjY0MywianRpIjoiSnRIRExabVRTUk15cUpqeUc4M2hjRnNQSTR3cjZRIiwiY2lkIjoiMFItV0FNaHVvby1NeVEiLCJsaWQiOiJ0Ml9yZnh3MjV2ZnYiLCJsY2EiOjE3MDQ0NjE5MTY2OTYsInNjcCI6ImVKeGtrZEdPdERBSWhkLWwxejdCX3lwX05odHNjWWFzTFFhb2szbjdEVm9jazcwN2NMNGlIUDhuS0lxRkxFMnVCS0drS1dFRld0T1VOaUx2NTh5OU9aRUZTeUZUUjg0M3l3b2thVXBQVW1ONXB5bFJ3V1prTGxmYXNVS0RCNllwVlM2WjIwS1BTNXZRM0kxRnowNk1xbHhXSHRUWW8zSnBiR01LMnhQanpjWnFReXF1eTZsTVlGa29uOFdMZnZ5Ry10WS1mN2JmaEhZd3JLZ0tEX1RPdUZ4d1lfSERGSGJfbnByMGJGMndxTDNYZzlRLTEtTjI3Yk5tb2RtNV9WelB2emFTY1RtRzVpZll2N3QtQ1IxNDVIbVpVUWN3WWcwX3lyQWo2X0N2T29ES0JRV01KWWhQSTVBcmwyX19KZGl1VGY4YXR5ZC0tR2JFVFdfNHJSbW81eExFb1VfajZ6Y0FBUF9fWERfZTR3IiwiZmxvIjoxfQ.jSh7g1nVuSaTzhisEgQtOYLf_19xHfeKxGku7QrgQQ-iUbiOcNdnF_7KaRX2KKeSNFm1ohFu340YliO_HGYjJGdfDbJ6y1dP2Qjqx3FuXKPmwfgmJyCvL5FEFBIdYjgDvj2iEYkZVCE3j_lCLkgF8QXK7l0ES5HQr4cLVy9adGVezcCey6K_66EXXS2oKu9dGrH-oIr6hedwnPKWQpm9RJM3gvV8opjsTl0B151BCsTw0_Ms1ugly9x_UZnViV8_obgHrcglW6PbagBzYOUqbjr5B3ou4izUVCFDZrZDB2PMBGv6mRudu3BNOu_cuf1SriWb6zNk94nrHjC8gVxtvg; Path=/; Domain=reddit.com; Expires=Sat, 06 Jan 2024 13:38:36 GMT; HttpOnly; Secure
set-cookie: csv=2; Max-Age=63072000; Domain=.reddit.com; Path=/; Secure; SameSite=None
set-cookie: edgebucket=gIOfpNxA8OPsy0nOP3; Domain=reddit.com; Max-Age=63071999; Path=/;  secure
set-cookie: wwrbucket=e90014a9f63a830640fbad2a31837d9eab9b0901e9b79bb81382101d778c6b49; Max-Age=15
server: snooserv
report-to: {"group": "w3-reporting-nel", "max_age": 14400, "include_subdomains": true,  "endpoints": [{ "url": "https://w3-reporting-nel.reddit.com/reports" }]}, {"group": "w3-reporting", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting.reddit.com/reports" }]}, {"group": "w3-reporting-csp", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting-csp.reddit.com/reports" }]}
nel: {"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 1.0, "failure_fraction": 1.0}
>

I am not aware of any browser detection on Reddit, however that doesn't mean it could not exist.
If someone has any insights, that would be great.

[j-hui]

Yeah, I think this is something weird happening on Reddit's end. It only started happening about a month ago, and was working fine before then, so I'm guessing it's because Reddit changed something on their end.

[mre]

Okay, that's probably it. I just tested in lychee 0.14.0, and it's still the case.