misunderstanding the tool - too much work defining properties

[tomdavidson]

I maybe misunderstanding how to use fast-check because I'm finding it rather time consuming define properties such as a malformed zip code or po box lines to test a normalize function. Perhaps a property runner is not right for string transformers?

What is a right way to define properties the produces strings such as:

"p.o box"
"pobox#34S5341"
"p.o.box 31o943"
"p.o.b 325"
"pbox"
"po b x 34S5341"
"pobox 31O943apt#5"
"p o b 325"
"box #325"
"po box 34 S5341"
"pob 325, Apt 5"
"box 325"
"p.o. box 34/S5341"
"pobox 31o943 suite#5"
"pob 325, Suite 5"

The assertions also seem a bit tricky and the way I have been thinking about it I need to filter out runs and use non-trivial regex that might be flawed. Here's an example that might how I am doing multiple things wrong:

 it("should remove punctuation except the zip code's hyphen", () => {
      const digitArbitrary = fc.integer({ min: 0, max: 9 }).map(String);

      const zipCodeArbitrary = fc
        .stringOf(digitArbitrary, { minLength: 5, maxLength: 5 })
        .map((digits) => `${digits.slice(0, 5)}-${digits.slice(5)}`);

      const sentenceArbitrary = fc.array(fc.string(), { minLength: 1, maxLength: 10 }).map((words) => words.join(" "));

      fc.assert(
        fc.property(zipCodeArbitrary, sentenceArbitrary, (zip, sentence) => {
          const insertPos = Math.floor(Math.random() * sentence.length);
          const input = sentence.slice(0, insertPos) + zip + sentence.slice(insertPos);
          const output = removePunctuationWithZipcode(input);
          output.includes("-")
            ? expect(output).toMatch(/^(?:\d{5}-\d{4}|(?!\d{5}-\d{4})\d*-\d*)*$/)
            : input.includes("-")
            ? expect(input).not.toMatch(/\d{5}-\d{4}/)
            : expect(output).toBe(input);
        })
      );
    });

Its not "bad" and I am exposing some edges I had missed with example based testing, but it seems to take quite a bit of effort on this cases that I thought would be simple so I think I'm using it wrong.

Thank you.

[dubzzz]

Generating strings close to a given pattern

After re-reading through your message I'm not sure it's what you are looking for but as I already had the message ready let's keep it!

This feature is coming soon: https://twitter.com/ndubien/status/1673573855342755841?s=20

For now, maybe something like fc.stringMatching might help you but you need to somehow be able to express what string you are waiting for in terms of a regex.

But defining string not-matching something is not trivial as most of the time it should not match but should barely match and even with the helper in this tweet nothing prevent the generated string from matching except you.

I think these examples might help you to get some examples of proper usages of property based testing:

• https://dev.to/dubzzz/advent-of-pbt-2021-13ee (or the repo https://github.com/dubzzz/advent-of-pbt-2021)
• https://github.com/dubzzz/cracking-pbt/tree/main

Asserting the right things

When building properties for the first time, the most challenging part is to find what to assert. Generally, as we all think in an example-based testing manner, we want to assess the full input but that's often not what we want to do. In property based, I think the key is to know that we will rarely be able to write expect(out).toBe(value) but only check some sub-characteristics of the output.

As an example always express itself better than anything else, let's take one:

We have a shortestPath algorithm. It takes and array of oriented edges as inputs (aka {from: locationNameString, to: locationNameString, distance: positiveInteger}) and two positions (start and end). It returns the shortest distance to go from start to end and the full path.

If we wanted to check that algorithm here how you could think in a property based manner:

• Asserting that we have the shortest path is hard
• But knowing that the path being returned is valid is simple: we need to check that the proposed stops really exists, that they sum to the distance returned by the algorithm
• And that it's not too long is feasible: we need to know one path from start to end, if the returned one is shorter or equal then we are good

Our property would work as follow:

fc.assert(fc.property(
  fc.string(), // the name for the start point
  fc.array(fc.record({to: fc.string(), distance: fc.nat())), // the next stops in our known path
  fc.array(fc.record({from: fc.string(), to: fc.string(), distance: fc.nat())), // extra links
  (start, pathStops, extraLinks) => {
    const links = [...extraLinks];
    let lastStop = start;
    let knownDistance = 0;
    for (const stop of pathStops) {
       links.push({from: lastStop, ...stop});
       lastStop = stop.to;
       knownDistance += stop.distance;
    }
    // we know for sure a path from start to lastStop with distance being knownDistance
    // we expect to find it or better!
    expect(shortest(links, start, lastStop)).toLowerThanOrEqualTo(knownDistance);
  }
))

More at https://github.com/dubzzz/advent-of-pbt-2021

[tomdavidson]

Thank you, this is the type of commentary I needed! Didn't see it before, but stringMatching is supper slick and I think it could address several of my cases that I am going to rework to use. Here's one that seems a bit questionable. A simple function that is suppose to get all the po box variations looking the same is all regex:

const standardizePOBox = (str: string) =>
  str
    .replace(/(?:p [o0] box|p[o0]box|p[o0]b|(?<!p[o0]\s*)box)(?=\s*\d)/gi, "po box")
    .replace(/(po box)(\d+)/gi, "$1 $2");

I dont to use the same regex in the stringMatch() , or Im testing the .replace() works rather than my code? The test could look something like:

      fc.assert(
        fc.property(fc.stringMatching(/.*(pobox|po|p o |pob |p.o.box|p.o.b|p o b|box) +\d+.*/g), (poBox) => {
          const standardized = standardizePOBox(poBox);
          expect(standardized).toMatch(/po box \d+/i);
        })
      );

But then Im still coming up with the variation of po box? and if I do something clever in stringMatching then I may not be sure if my test's truth is doing what I want the function to do to or a defect in the stringMatching pattern.

I like the focus on the asserting the right things. If Im wishing I could use toBe(vaule) then I am probably trying to define the property too exact. My use of toMatch is probably just one step passed example based testing and I need to think more of what is the right truth for the case.

Do you typically end up with property and example tests so your not getting everything from the property test? It seems as if I was using zod rather than typia, the property generation could be quite streamlined.

[dubzzz]

Do you typically end up with property and example tests so your not getting everything from the property test?

I do keep the two layers. Examples make me able to check closely all the fields in details. Properties make me able to cover a wide range of inputs while possibly going in less details on the checks.

I believe both are suitable.