Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containers getSchema throwing "hit infinite loop" #22243

Open
u12206050 opened this issue Apr 18, 2024 · 7 comments · May be fixed by #22245
Open

Containers getSchema throwing "hit infinite loop" #22243

u12206050 opened this issue Apr 18, 2024 · 7 comments · May be fixed by #22245

Comments

@u12206050
Copy link
Contributor

Describe the Bug

When our containers are scaling we found many of them crashing, looking at the logs we found this error.

This started happening after updating from 10.9.3 to 10.10.7

Screenshot 2024-04-18 at 13 23 25

To Reproduce

Scale up to multiple containers running directus 10.10.7

Directus Version

v10.0.7

Hosting Strategy

Self-Hosted (Docker Image)

@br41nslug
Copy link
Member

Can you try clearing Redis cache? this can be cause by lock in the cache that wasnt properly cleared

@u12206050
Copy link
Contributor Author

From the code I notice there is no flow path for what has to happen after the MAX_ATTEMPTS has been reached. I can make a PR that fixes that, since it should probably fall back to attempt fetching the schema itself. In a single container the retry logic doesn't make sense either, since there could be multiple requests waiting on the getSchema, causing all of them to be very close together when timing out.

@u12206050
Copy link
Contributor Author

I just realised that the schema isn't being cached to redis at all. Seems like this await sharedSchemaCache.set('schema', schema); is missing from the setSchemaCache function

@br41nslug
Copy link
Member

Before diving into the code have you tried clearing the redis cache as asked?

Seems like this await sharedSchemaCache.set('schema', schema); is missing from the setSchemaCache function

Thats not missing i think as it's likely not supposed to do that, the actual schema is stored with localSchemaCache where sharedSchemaCache stores a hash to sync multiple instances.

@u12206050
Copy link
Contributor Author

Yes I did clear the cache.

@that1matt
Copy link
Contributor

I had this exact same issue when upgrading to 10.10.5 for our development setup. I ended up removing the CACHE_SYSTEM_TTL env which was previously set to CACHE_SYSTEM_TTL=100 and restarted the container, and haven't seen this issue since.

@cdemi
Copy link

cdemi commented Apr 23, 2024

This is related to #22161

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🆕 Needs Triage
Development

Successfully merging a pull request may close this issue.

4 participants