Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fixture with sample analysis results #962

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

KlaasH
Copy link
Contributor

@KlaasH KlaasH commented Dec 10, 2024

Overview

This edits the setup script (technically the update script, which does most of the setup) to download and import a fixture containing a few neighborhoods and analysis results. Specifically neighborhoods and results for:

  • Grand Rapids, MI
  • Mackinac Island, MI
  • Pella, IA
  • Presque Isle, ME
  • Thunder Bay, ON, Canada
  • Groningen, Netherlands

These particular ones were chosen mostly-arbitrarily by me, with a goal of keeping them smallish but not all tiny, with some in the US and some not. (And Mackinac Island made the cut because it's the #1 low-stress connectivity champion, owing to the fact that cars are banned from the island.)

I also added one neighborhood, Montpelier, VT, for which I didn't import the analysis results. Because it seems like it might be useful to have a neighborhood with no current analysis ready to go so we can run an analysis and not have to check if what we're seeing afterwards is the result of that or an earlier one.

To load the fixture, run ./scripts/update --load-data. I made it optional because it really only needs to be done on initial setup, and it will waste time to import it again on subsequent updates (I also put the import_crash_data step in the same conditional block, for the same reason).

Resolves #959

Demo

image

Notes

I assembled the data by copying it from production, i.e.:

  • Downloading the boundary files from s3://production-pfb-storage-us-east-1/neighborhood_boundaries/root/
  • Finding the job UUIDs of the latest job for each neighborhood via the web interface, then downloading the analysis output files for that job from s3://production-pfb-storage-us-east-1/results/
  • Bundling up the files for each job into a .zip file
  • Adding the neighborhoods through the Neighborhoods admin page
  • Importing the analysis results through the Jobs admin page

Then I exported the fixture and uploaded it to S3:

./scripts/django-manage dumpdata --indent 2 -o sample_analysis_results.json.gz \
    pfb_analysis.AnalysisJob pfb_analysis.Neighborhood pfb_analysis.NeighborhoodWaysResults \
    pfb_analysis.CensusBlocksResults
aws --profile pfb s3 cp sample_analysis_results.json.gz "s3://test-pfb-inputs/"

Testing Instructions

  • Delete your existing dev instance with docker compose down -v (this is necessary because you need your organization_id to match what the fixture expects)
  • Run ./scripts/update --load-data to rebuild your instance and import the data
  • Run ./scripts/server and go to http://localhost:9301/#/. You should see the imported results in the "Leading Cities/Towns" panel on the landing page, and on the list if you click "All places"

Checklist

  • Add entry to CHANGELOG.md

Adds some lines to the 'update' script to download a data fixture containing
a few neighborhoods and analysis results and import it into the database.

Because the import is a bit slow and only needs to be done on initial setup,
this also modifies the 'update' script to take a '--load-data' argument, so
that it will only load the fixture (and load crash data, which also takes
some time and doesn't need to be repeated) on demand, not by default.

The fixture itself is on S3, in the 'test-pfb-inputs' bucket, and the script
automatically downloads it.
We already hard-coded a user UUID in this migration, possibly to facilitate
moving fixtures between development environments. This sets the organization
UUID to a fixed value as well, to facilitate transferring neighborhoods in
a fixture.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create fixture to populate local dev database
2 participants