Analyzing Model Outputs
There are a large number of model outputs produced by the ABM, described here. These include the following:
- Loaded auto networks: Volumes and travel times on auto network links by time of day.
- Transit boarding summaries:
- Auto Skims: Travel times, distances, and costs between each TAZ pair by auto mode and time of day.
- Transit Skims: Travel times and costs between each TAP pair by transit skim set and time of day.
- CT-RAMP outputs: The results of CT-RAMP models including household and person level choices, and lists of tours and trips.
- Trip tables: Trip tables containing aggregate (IE/EI,EE, and Commercial Vehicle) and CT-RAMP results.
- Log files: CT-RAMP report files that can be used for debugging in case of model crashes.
- Final assignment outputs summary file: A one-stop-shop Visum version file that contains all important assignment results. More details on the contents of this file can be found here. In addition to this summary version file, the CSV versions of the important highway and transit assignment outputs are also exported for easy reference and analysis.
In addition to these outputs, the ABM includes a Activity-based Model Visualization Tool that can be used to view CT-RAMP outputs and compare them to either base-year observed data or compare results to other scenarios. The two scenarios being compared are labelled as base and build within the tool system. The tool takes two sets of summary CSV files as inputs for the base and build runs. The directory structure of the Visualization Tool is presented below.
Each of the elements of the Visualization Tool setup are described in the table below:
| Directory/File | Description |
|---|---|
| Data directory | contains two sub-directories (base and build) which stores the CSV summaries files for the two scenarios |
| Outputs directory | the final HTML dashboard is created in this directory |
| Scripts directory | this directory contains all the R scripts – Master.R, _SYSTEMS_VARIABLES.R and SummarizeABM.R |
| Runtime directory | all the intermediate files are stored in this directory |
| Template directory | this directory contains the template R markdown file which contains the source code to generate the HTML dashboard. A CSV file containing the list of all required summary files is also stored in this directory. |
| generateDashboard.bat | the main batch file which orchestrates the creation of HTML dashboard |
The generateDashboard.bat DOS batch file is called by the RunModel.bat DOS batch file to run the Visualization Tool at the end of each ABM run. The Visualization Tool can be configured by setting the following properties in the RunModel.bat DOS batch file:
| Setting | Description | Default Value |
|---|---|---|
| BASE_SUMMARY_DIR | location of summary files from the base scenario | inputs\OHAS_Census_Summaries |
| BUILD_SUMMARY_DIR | location of summary files from the build scenario | outputs\other\ABM_Summaries |
| BASE_SCENARIO_NAME | base scenario name | OHAS |
| BUILD_SCENARIO_NAME | build scenario name | SOABM |
| IS_BASE_SURVEY | survey base indicator. Some labels are different for survey | Yes |
| BASE_SAMPLE_RATE | Sample rate of the base scenario. Build sample rate is same as the ABM run | 1.0 |
The generateDashboard.bat DOS batch file writes out all the properties into a parameters file. Next, the batch file calls the SummarizeABM.R script which processes the ABM outputs into the standard CSV summaries which are input to the Visualization Tool. Finally, the Master.R script is called which renders the HTML dashboard.
To view the results, open the output HTML file with Google Chrome or Internet Explorer. When the tool is open, the Welcome Page appears:
The menu at the top of the screen can be used to select different summaries from the model outputs. Click on "Overview" to see a summary of model results including total households, employment, population, jobs per worker, average household size, tours, trips, stops, tour rates, implied VMT, etc. For survey vs ABM comparison, same value is shown for total employment and jobs per worker statistics. Note: you may need to adjust the size of the window to fit your screen, by holding down the control key and using the mouse wheel.
Long-term model results including auto ownership model results, work from home percentages, and mandatory tour trip length frequencies can be viewed by clicking "Long Term" from the top menu. Other pages available from the Long-Term pull-down menu show district level flows, average trip lengths, MAZ-level employment vs workers and MAZ-level enrollment vs students comparison. Make sure to select purposes and markets of interest from the pull-down menus on each page before viewing results.
Tour level results are avaialble from the Tour-Level menu. These include daily activity pattern model results, number of mandatory and non-mandatory tours, escort tours, tours by time-of-day and tours by mode.
Trip level results include stop frequency and purpose distributions, stops by time-of-day, and trips by mode. Shown below are stop departure and arrival time distributions.
Assignment results include plots of loaded volumes versus either traffic counts (for comparisons to observed data) or to comparison scenario volumes. Daily and time-of-day summaries are available, as well as gap statistics and VMT comparisons.
The tool also shows comparisons of commercial vehicle and external aggregate model results (not shown here).
Some common checks to perform after a new model run include:
- If the MAZ data file was changed without changing the synthetic population, total resident model tours and trips will probably not change much. However, trip lengths, trips by time-of-day, trips by mode, VMT and assignment results may change. Are these changes consistent with the MAZ data changes?
- If the network was changed, are the changes in link volumes and/or transit boardings consistent with the network change?
- If the synthetic population changed, are the total households and persons on the overview page consistent with the intended synthetic population change? Are the changes in total tours and trips consistent with the change in households and persons?
CT-RAMP writes a series of log files to the logs folder during a model run. These log files are extremely useful for understanding the model as well as for debugging a model run. The key log files in the logs folder are:
- event.log - the main overall process log file
- event_hh.log - the household data manager
- event_mtx.log - the matrix data manager
- event-nodeX.log - each remote node if applicable
- There are also model component specific log files such as event-ao.log for the auto ownership model.
In addition to writing log files, CT-RAMP can trace model calculations for a user specified household. To trace results for household 2949465 for example, set the following in the orramp.properties file:
Debug.Trace.HouseholdIdList = 2949465
This tells CT-RAMP to write out all calculations for every person in household 2949465, including the results of the UEC calculations for each model.
Household Trace Results below contains a sample of the household trace results. The first screenshot shows a sample of the trace results for the relevant chooser - person number 1 in the traced household. As is shown below in the first image, key attributes of the household and person are traced such as:
- hhIncomeInDollars - household income
- hhSize – household size
- hhWorkers - number of workers
- homeTaz - home TAZ
- persAge - person age
As is shown in the second image, which is the auto ownership model trace for household 2949465, the value of the coefficient for each alternative times the value of each expression is logged. The dimensions of this output are:
- The header contains the household ID traced - HHID_2949465
- The 111 expression rows by five alternatives (0, 1, 2, 3, 4+ autos) correspond with the expressions and alternatives in the auto ownership UEC file
- Each row is an expression and each column is the coefficient times the value for the alternative
