Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): improve caching and place name matching #451

Merged
merged 10 commits into from
Dec 27, 2024
Merged

Conversation

steveoh
Copy link
Member

@steveoh steveoh commented Dec 24, 2024

This pr updates the caching layers, introduces a new cache, and adds levenshtein to place name matching.

The bigquery hosted service now runs on startup and every sunday (the scheduled query to load data from sheets runs on saturday) and hydrates an in memory cache. There was no reason to have more cache levels as it's static data. I also couldn't figure out firebase remote config :(

I introduced fusion cache since microsofts hybrid cache does not allow for named caches and we needed different settings for each cache. Fusion cache allows for a l1 in memory and l2 redis.

The firestore fusion cache will help reduce the amount of calls to read api keys. We sometime incur a read cost since there was a 1:1 between requests and firestore reads.

The place name fusion cache will allow us to correct minor place name mistakes. First I check the bigquery memory cache for our standard names and on miss check the fusion cache. On another miss, I get all our standard names and levenshtein to get the closest match. If one is found, I get the grids from the bq memory cache and update the fusion cache with the misspelling. Otherwise I set the misspelling to empty so it's not levenshteined again.

I chose not to add this feature to the zip codes but rather expect an exact match since zip codes are very similar and levenshtein could produce strange results. Zips use the bq memory cache.

@steveoh steveoh changed the base branch from main to dev December 27, 2024 19:41
@steveoh steveoh marked this pull request as ready for review December 27, 2024 19:41
@steveoh steveoh merged commit b85e7d7 into dev Dec 27, 2024
5 checks passed
@steveoh steveoh deleted the feat/fusion-cache branch December 27, 2024 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant