Add afterScenarioOutline & karate.scenarioOutline #2636
Conversation
| * match karate.scenario.exampleIndex == <num> | ||
|
|
||
| # Confirm scenarioOutline result | ||
| * match karate.scenarioOutline == |
There was a problem hiding this comment.
the whole PR looks good and I will definitely merge.
one question - is it really necessary to have a new scenarioOutline API, won't the existing scenario suffice ?
There was a problem hiding this comment.
Could put it in with karate.scenario but I feel like it's weird to store information about other scenarios in there. I think I prefer karate.scenarioOutline but I can change it to whatever you think is best.
There was a problem hiding this comment.
the way I look at it is at run time, only scenarios exist. some scenarios happen to originate from scenario outlines. does that make sense ?
There was a problem hiding this comment.
Yeah that makes sense, I definitely learned that the code is structured that way as well. Maybe the karate.scenario object can have an outline section when it originates from a Scenario Outline and it can include some/all of the same data I currently put into karate.scenarioOutline?
There was a problem hiding this comment.
since exampleTableIndex is added to ScenarioResult.java won't that be sufficient ? if exampleIndex is not -1 it means it originated from an outline. and you can check exampleTableIndex for your requirement. I'm still not sure how you can identify if something is the last example
There was a problem hiding this comment.
No worries totally understand the context switching. I don't think there is any way to iterate over the scenario results in karate.feature. I think karate.feature just contains high level metadata like description, title, etc...
Yeah there is no absolute need to update the system of record immediately. But it is kind of fun to watch it update realtime :) Our old method was to write files with the results and combine them at the very end of the suite. This is fine but it leads to lots of files and if we want to cancel the suite the system of record is not updated. So this really is just an enhancement. I can see value in providing the karate.scnarioOutline metadata but ultimately this is your project so I will respect whatever your final decision is.
Also one thing I forgot to mention is that in order to update the correct test in the system of record we have to keep track of an API ID. Currently this API ID is just defined as a karate variable in each Scenario, so we would lose these IDs in the afterFeature if there were multiple scenarios. We have a pretty large test suite so it would be infeasible for us make any large scale change to switch to afterFeature
There was a problem hiding this comment.
acknowledged. let me think over this a bit
There was a problem hiding this comment.
@OwenK2 I'm thinking that we should introduce a new afterScenarioOutline hook that may be much cleaner. I can see this also being useful for the @setup tag that has proven very useful - and being adopted more by teams. I would consider adding this to the Java hook - RuntimeHook
that said - I'm not yet able to figure when to call this hook and how to detect that the last scenario in an outline-batch has completed - and have this work even under parallel execution.
my current guess is here: https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/FeatureRuntime.java#L197 - where you can see a synchronize on the feature result. so perhaps we can add intelligence within that synchronize to a) see if we are part of an outline and b) see if all other scenarios in that outline are already in the feature-result.
let me know what you think. if we go down this path - I agree that we need a new data-structure like you did already that returns an array of results for a given scenario-outline.
There was a problem hiding this comment.
@ptrthomas This all sounds great to me, I will take a stab at it.
There was a problem hiding this comment.
I followed your recommendations and got something working. Did some basic testing. Please let me know what you think when you get a chance. Thanks!
| @@ -101,13 +104,13 @@ public List<Scenario> getScenarios(FeatureRuntime fr) { | |||
| if (selectedForExecution) { | |||
| Table table = examples.getTable(); | |||
| if (table.isDynamic()) { | |||
| Scenario scenario = toScenario(table.getDynamicExpression(), -1, table.getLineNumberForRow(0), examples.getTags()); | |||
| Scenario scenario = toScenario(table.getDynamicExpression(), examples.getIndex(), -1, table.getLineNumberForRow(0), examples.getTags()); | |||
There was a problem hiding this comment.
so following up from my previous comment - suggest since in this loop we know how many examples and example tables are there - we will be able to emit an extra boolean if this is the last scenario selected for execution within an outline
if that solves your current ask - I would just do that, as I really don't see a need for tracking counts of examples and tables :|
There was a problem hiding this comment.
Sounds good I can definitely make that simplification. What about the scenarioResults though? What do you feel would be the best way to get that information from an API perspective.
There was a problem hiding this comment.
For now I just removed the table index code in 6e84b5d. Didn't want to push too far ahead before we finalized something.
OwenK2
changed the title
Description
This PR adds a
karate.scenarioOutlinevariable that is accessable from within allScenario Outlines and after hooks. It contains metadata about the outline which may be useful. This PR also adds a new after hook that can be define via:* configure afterScenarioOutline = function() {...}This after hook runs at the end of a
Scenario Outlineafter the finalafterScenariohook has run.