Google Floods Part 3: Adding Inundation Maps; COUNTRY=mozambique

[lowriech]

This adds inundation polygons from the Google Floods API.

https://www.loom.com/share/5b430b093ee546139f5ccbe3299c60c9?sid=049273de-18ff-4b6a-8718-b1a164c82b52

[lowriech]

@gislawill this is updated to call gauges, rather than rely on a previous call passing its attributes

[github-actions]

Build succeeded and deployed at https://prism-1332.surge.sh
(hash 3222328 deployed at 2024-11-23T14:41:42)

[gislawill]

These two files are each about 4 MB — which is unfortunately how large the response from Google is at this endpoint.

[gislawill]

Note: on 11/7/24, I received this error message intermittently but very frequently. Hoping the google floods has resolved it. Or will need to reach out to them

Request failed at url post https://floodforecasting.googleapis.com/v1/floodStatus:searchLatestFloodStatusByArea?key={GOOGLE_FLOODS_API_KEY}: {'regionCode': 'ML'}
Response status code: 502
Response content: (abbreviated)

    <title>Error 502 (Server Error)!!1</title>
    <p>The server encountered a temporary error and could not complete your request.
    <p>Please try again in 30 seconds.  <ins>That's all we know.</ins>

[wadhwamatic]

looks good from my side Will. some surprising discrepancies in Cambodia between NASA SERVIR flood extent and the Google model, but we've also known there to be overestimation by the SERVIR tool.

[gislawill]

The 502 errors from Google Floods have continued and seem to be a result of us hitting the searchLatestFloodStatusByArea endpoint too frequently. I've just emailed the google floods team about this issue.

In the meantime, I've added google floods response caching which gets around the issue for now. I've set the cache timeout to only 10 minutes as we know this data can change quickly.

@ericboucher, I'm curious for your thoughts on this caching (a45de24). Specifically,

• I'm not seeing any cache clearing so far. These new cached files could bloat the EC2 instance
• Would it be helpful to include the cache timeout logic in the caching utilities for easier use elsewhere?

[ericboucher]

Please explicit the inputs here, esp. the cache_key and how it should be used.

[gislawill]

Added some add comments and updated the params to be more explicit

[ericboucher]

@gislawill thanks for this. Yes I think it could be useful to add the cache timeout mechanism for other requests as well.

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

[wadhwamatic]

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

@ericboucher - yes, we should delete older files in this case and 30 days sounds like a good initial period to try

[gislawill]

@wadhwamatic and @ericboucher, we have many different options for clearing this cache. I think whichever option we go with, it should be easy to trigger the cache clearing automatically and manually. We'll also want to be able to easily adjust the "minimum age to clear" to allow for experimentation with this cache age and for clearing caches after a bug fix rollout.

The two good options would be a shell script and an endpoint:

• Script
  • A Python script (could be Bash too) that loops through the cache directory and clears files based on age
  • The age would default to 30 days but could be provided as an environment variable
    • A provided age of 0 would clear all cached files
  • This could be triggered manually by SSH-ing into the container and running it
• Endpoint
  • A FastAPI endpoint that loops through the cache directory and clears files based on age
  • The age would default to 30 days but could be provided in the request body
  • This could be triggered manually by any request client (i.e. curl)
    • Very easy, though not secure (but can't think of any real risks associated with it)

Two options for automated triggers:
- A cron daemon in the Docker image to keep simple
- An ECS Scheduled Task which is a slightly more complicated set up but easier to monitor and more resilient (retries, alerting)

I'm leaning towards a script that's triggered by an ECS Scheduled Task. Curious for your thoughts. Also curious if either of you would have a preference between adding this cache clearing in this PR vs a follow-up?

[ericboucher]

@gislawill the endpoint option seems to be convenient and allows any of us to trigger it manually when necessary. We could "secure" it with a token or something simple, that should be enough to test things out.

I'd prefer to have this added in a follow-up PR as it would make it a bit easier to discuss implementation and triggers. If we go with the endpoint option, we could use a simple GitHub action to run daily and call this endpoint.

[gislawill]

Sounds good to me @ericboucher. @wadhwamatic I've created issue #1371 for this work.

This PR should be ready for review. It's blocked on rollout by the copy + translations questions here: #1328 (comment)