[Feature Request] Provide ScenarioOutline Metadata

[OwenK2]

My team is looking to implement the following logic in the afterScenario hook to update our system of record as the test suite runs:

if (scenario is last example in outline) {
    if (any example in this outline failed) {
        Send API call to update system of record to show test FAILED
    }
    else {
        Send API call to update system of record to show test PASSED
    }
}

Currently I am unaware of any way to get this information or perform this logic in Karate (if I am wrong please let me know and I can re-post on stack overflow and close this issue). Possible ideas here are to provide a karate.scenarioOutline object with the desired metadata. This could perhaps be in conjunction with an afterScenarioOutline hook although that may be excessive.

Possible karate.scenarioOutline

{
    totalExamples: 4
    exampleResults: [true, true, false, true],
    examples: [{col1: "value", col2: "value2"}, {col1: "value", col2: "value2"}, {col1: "value", col2: "value2"}, {col1: "value", col2: "value2"}]
    // ...
}

I started looking into the code to try and contribute but saw in the guidelines that I should run my idea by project leads before working on a PR.

[ptrthomas]

@OwenK2 thanks for checking, yes I think the best approach is for you to submit a PR

if you call karate.scenario it returns a JSON: https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/ScenarioBridge.java#L550

that JSON already has metadata about the index of the outline "example": https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/ScenarioResult.java#L197

I'm not sure if that is sufficient for you to get if it is the last example. I propose we add a new property called "exampleCount" which may be more "future proof".

[OwenK2]

Yeah I can't quite figure out if it is the last example just based on the exampleIndex. The other thing I would need is the results of the other examples that ran. Does it make sense to include exampleCount and exampleResults fields in karate.scenario? Or is that too odd since karate.scenario really should only have data about 1 scenario? I was thinking a karate.scenarioOutline would make sense but that's definitely harder to do.

[OwenK2]

@ptrthomas Working on the PR now, hope this is an ok place to ask questions like this. I have a test scenario like so:

Scenario Outline: Multi Outline
    * print <pass>
    * if(!<pass>) karate.fail()

    Examples:
    | pass   |
    | false  | 
    | false  | 

    Examples:
    | pass  |
    | true  | 
    | true  | 

I noticed that the example index resets per "example table". Meaning here the scenario example indexes go 1, 2 then 1,2 again. How does karate distinguish between the scenarios in the first table and the second? Based on the code karate queries scenarios via section ID + example ID but how does this work if 2 scenarios could have the same section and example index? Does there need to be an exampleTableIndex?

[ptrthomas]

@OwenK2 yeah, no one has ever needed this before, and we may indeed need examplesTableIndex - code changes would be needed here as well: https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/FeatureParser.java#L280

[OwenK2]

@ptrthomas PR created here: #2636. This is my first attempt at contributing to a public repo so please let me know if I need to fix anything. Please let me know what you think when you get a chance!

[ptrthomas]

@OwenK2 PR merged with thanks. I'll be trying to release 1.5.1 next week.

[OwenK2]

@ptrthomas Awesome, glad I could contribute!

[ptrthomas]

1.5.1 released