Suggestion: Mutate Result<Success, Error> values

[kevinmatthes]

It would be interesting if values of type Result<Success, Error> could be mutated, as well. Here are some ideas:

• Result<Success, Error> → Result<Error, Success>, i.e. also inverting all Oks to Errs and vice versa
• Result<Success, Error> → Result<Success, Success>, i.e. replacing all Error values with Success::default(), for example
• Result<Success, Error> → Result<Error, Error>, i.e. replacing all Success values with Error::default(), for example

The implementation might be more challenging than implementing changes like + → - as the types are altered but this way, incomplete tests only checking for success and / or error states might be identified more easily.

[sourcefrog]

As you've probably seen, we already mutate Result<S, E> into, recursively, generated values of Ok(S) and Err(E) (the latter only if an error value is configured.)

But, this seems to be about something else.

Could you give some more concrete examples of where the patterns you're suggesting would help?

I would tend to assume that, generally, the S and E types are disjoint -- error values are not acceptable obj values or vice versa.

For example I don't see why we would expect Err(S::default()) or Ok(E::default()) would even compile, in general. Or, why this would be any better than just saying Default::default() inside either of them? (Also, it seems rare that error values impl Default, but some might.)

It is a rare but valid case to have maybe Result<&str, &str> or Result<(), ()>. But, then, I think it's not really that swapping them is useful, but rather that it tells us something about probably-valid error values?

This does suggest a different idea though, which is that if we see both type parameters to the Result type, then we could recurse into the second: at the moment we only use the configured error value. For example if it's Error<usize, &str> we could generate Err("") etc.

[kevinmatthes]

As an example, please consider this function; https://github.com/kevinmatthes/aeruginous-io/blob/main/src/reading.rs#L124-L129.

    fn read_silently(&self) -> Result<String> {
        match std::fs::read_to_string(PathBuf::from(self.clone())) {
            Ok(s) => Ok(s),
            Err(e) => Err(e.into()),
        }
    }

It uses a custom Result type; https://github.com/sorairolake/sysexits-rs/blob/develop/src/exit_code.rs#L18. When testing this function, it suffices to test for the error state in general (https://github.com/kevinmatthes/aeruginous-io/blob/main/tests/reading.rs#L46-L56) although there are specific conversions of the default Error into this return type; https://github.com/sorairolake/sysexits-rs/blob/develop/src/exit_code.rs#L404-L443.

The example shows the following: testing just for error / success states in general suffices in certain situations to catch the generated mutants, if any, and cover the statements although the tests themselves still remain incomplete.

[sourcefrog]

Right, that looks like a pretty typical custom error enum.

Could you tell me what mutants you're suggesting we should generate in that example? The original post made it look like you wanted to return something like Err(s) but that won't compile...

[kevinmatthes]

My suggestion would be to mutate Err(e.into()) to Err(std::io::Error::new(std::io::ErrorKind::Other, "generated by cargo-mutants").into()) because the code structure of the example lines out that a standard library function (std::fs::read_to_string) is called which has a known return type (std::result::Result<std::string::String, std::io::Error>) and cannot produce the suggested replacement.

The original post was primarily intended for brainstorming and motivated by the fact that I was confused that there was no mutation for the Result I tried to handle in the discussed example. Due to this discussion, I found out that there was no mutation since my code contained a custom Result. I apologise for having caused even more confusion.

[sourcefrog]

We do have the error_values config and command line option.

So I think in this tree you might use something like --error ::sysexits::ExitCode::Software and it might exercise the error handling case. I'll have a look. I'm a little surprised that it's not already generating Ok values...

[sourcefrog]

This seems to work...

; cargo mutants --error ::sysexits::ExitCode::Software -vV -F Err
Found 4 mutants to test
ok       Unmutated baseline in 0.5s build + 0.1s test
Auto-set test timeout to 20s
caught   src/reading.rs:125:9: replace <impl PathBufLikeReader for T>::read_silently -> Result<String> with Err(::sysexits::ExitCode::Software) in 0.3s build + 0.1s test
caught   src/reading.rs:132:9: replace <impl PathBufLikeReader for T>::read_loudly -> Result<String> with Err(::sysexits::ExitCode::Software) in 0.3s build + 0.1s test
caught   src/reading.rs:78:9: replace <impl BufReadReader for T>::read_silently -> Result<String> with Err(::sysexits::ExitCode::Software) in 0.3s build + 0.1s test
caught   src/reading.rs:58:9: replace <impl BufReadReader for T>::read_loudly -> Result<String> with Err(::sysexits::ExitCode::Software) in 0.3s build + 0.1s test
4 mutants tested in 1s: 4 caught

and the mutants.out/log/ file shows a failure that looks like the error really was exercised: it fails on a line that's doing

assert_eq!({ &(b"test"[..]) }.read_silently().unwrap(), "test\n");

*** src/reading.rs:58:9: replace <impl BufReadReader for T>::read_loudly -> Result<String> with Err(::sysexits::ExitCode::Software)

*** mutation diff:
--- src/reading.rs
+++ replace <impl BufReadReader for T>::read_loudly -> Result<String> with Err(::sysexits::ExitCode::Software)
@@ -50,33 +50,17 @@
     /// # Errors
     ///
     /// See [`sysexits::ExitCode`].
     fn read_silently(self) -> Result<String>;
 }
 
 impl<T: std::io::BufRead> BufReadReader for T {
     fn read_loudly(self) -> Result<String> {
-        let mut result = String::new();
-
-        for line in self.lines() {
-            match line {
-                Ok(l) => {
-                    result.push_str(&l);
-                    result.push('\n');
-                    Ok(())
-                }
-                Err(e) => {
-                    eprintln!("{e}");
-                    Err(e)
-                }
-            }?;
-        }
-
-        Ok(result)
+        Err(::sysexits::ExitCode::Software) /* ~ changed by cargo-mutants ~ */
     }
 
     fn read_silently(self) -> Result<String> {
         let mut result = String::new();
 
         for line in self.lines() {
             result.push_str(&line?);
             result.push('\n');


*** run /Users/mbp/.rustup/toolchains/beta-aarch64-apple-darwin/bin/cargo build --tests --manifest-path /var/folders/kw/xnsl15v52_99rb2841_254_h0000gn/T/cargo-mutants-aeruginous-io-OeU8o7.tmp/Cargo.toml
   Compiling aeruginous-io v0.2.0 (/var/folders/kw/xnsl15v52_99rb2841_254_h0000gn/T/cargo-mutants-aeruginous-io-OeU8o7.tmp)
    Finished dev [unoptimized + debuginfo] target(s) in 0.24s

*** result: Success

*** run /Users/mbp/.rustup/toolchains/beta-aarch64-apple-darwin/bin/cargo test --manifest-path /var/folders/kw/xnsl15v52_99rb2841_254_h0000gn/T/cargo-mutants-aeruginous-io-OeU8o7.tmp/Cargo.toml
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests src/lib.rs (/var/folders/kw/xnsl15v52_99rb2841_254_h0000gn/T/cargo-mutants-aeruginous-io-OeU8o7.tmp/target/debug/deps/aeruginous_io-b9fa96522f67bb86)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/reading.rs (/var/folders/kw/xnsl15v52_99rb2841_254_h0000gn/T/cargo-mutants-aeruginous-io-OeU8o7.tmp/target/debug/deps/reading-b566abeff2fd73cf)

running 5 tests
test read_buf_reader_failure ... ok
test read_path_buf_like_failure_does_not_exist ... ok
test read_path_buf_like_failure_directory ... ok
test read_path_buf_like_success ... ok
test read_buf_read_success ... FAILED

failures:

---- read_buf_read_success stdout ----
thread 'read_buf_read_success' panicked at tests/reading.rs:42:49:
called `Result::unwrap()` on an `Err` value: Software
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    read_buf_read_success

test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test reading`

*** result: Failure

[kevinmatthes]

Thank you for your help! This solves the question for me. I will convert this discussion to a question and mark it as resolved.