A Rust testing framework loosely based on Cockroach Labs'
datadriven
framework for Go. It
combines several testing techniques that make it easy and efficient to write and
update test cases:
- Golden master testing (aka characterization testing or historical oracle)
- Data-driven testing (aka table-driven testing or parameterized testing)
- Keyword-driven testing
A goldenscript is a plain text file that contains a set of arbitrary input
commands and their expected text output, separated by ---
:
command
---
output
command argument key=value
---
output
The commands are executed by a provided Runner
. The expected output is usually
not written by hand, but instead generated by running tests with the environment
variable UPDATE_GOLDENFILES=1
and then verified by inspection before it is
checked in to version control. Tests will fail with a diff if they don't match
the expected output.
This approach is particularly useful when testing complex stateful systems, such as computer language parsing, operations on a key/value store, concurrent transactions in a SQL database, or communication between a cluster of Raft nodes. It can be very tedious and labor-intensive to write and assert such cases by hand, so scripting and recording these interactions often yields much better test coverage at a fraction of the cost.
Internally, the goldenfile
crate is used to manage golden files.
See the crate documentation for more information.
Below is an example goldenscript for the
dateparser
timestamp parsing
crate. It supports a single parse
command taking a timestamp argument, and
outputs the parsed timestamp in RFC 3339 format.
Running cargo test
asserts that the runner's output matches the file, failing
with a diff otherwise. Running UPDATE_GOLDENFILES=1 cargo test
(re)populates
the file with the runner’s output, verified by inspection.
parse 2024-04-30
---
2024-04-30T00:00:00+00:00
# Test various other date formats.
parse 2024-Apr-30
parse 2024.04.30
parse 04/30/2024
---
2024-04-30T00:00:00+00:00
2024-04-30T00:00:00+00:00
2024-04-30T00:00:00+00:00
# Test some error cases.
parse 30.04.2024
parse 30/04/2024
parse 30/04/24
---
Error: 30.04.2024 did not match any formats.
Error: 30/04/2024 did not match any formats.
Error: 30/04/24 did not match any formats.
# Strings containing special characters must be quoted using " or '.
parse "2024-04-30 11:55:32"
parse '2024年04月30日11时55分32秒'
---
2024-04-30T11:55:32+00:00
2024-04-30T11:55:32+00:00
The corresponding runner for this script:
struct DateParserRunner;
impl goldenscript::Runner for DateParserRunner {
fn run(&mut self, command: &goldenscript::Command) -> Result<String, String> {
// Only accept a parse command with a single argument.
if command.name != "parse" {
return Err(format!("invalid command {}", command.name))
}
if command.args.len() != 1 {
return Err("parse takes 1 argument".to_string())
}
// Parse the timestamp, and output the RFC 3339 timestamp or error string.
let input = &command.args[0].value;
match dateparser::parse_with(input, &chrono::offset::Utc, chrono::NaiveTime::MIN) {
Ok(datetime) => Ok(datetime.to_rfc3339()),
Err(error) => Ok(format!("Error: {error}")),
}
}
}
#[test]
fn dateparser() -> std::io::Result<()> {
goldenscript::run(&mut DateParserRunner, "tests/scripts/dateparser")
}
For more information on goldenscript syntax and features, see the crate documentation.