.Net Processes - Map Step Feature

[crickman]

DRAFT

Motivation and Context

Fixes: #9193

Description

Map each value from a set to a map-operation and present the results as a set for potential reduction.

Includes:

• ProcessMapBuilder (Core)
• KernelProcessMap / KernelProcessMapState (Abstractions)
• LocalMap (LocalRuntime)

Features:

• Handles when output type has been transformed from input type
• Accepts either step or subprocess for map operation

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[esttenorio]

thinking about state propagation at this stage there may need a state validation to either pass the state or pass null (like no state was given) before letting the build logic take care of it.

Also thinking about if it's needed to have a stepId here or just keep treating the map steps as an array

[esttenorio]

General thoughts about State + Map:

Since the map is a subprocess with N steps (to be defined at runtime), I'm thinking of 2 scenarios:

• Map steps are volatile and no state is saved, and if the root process pauses while running the map, the last event received by the map gets re-propagated and the map steps get re-spawned -> only works if map steps steps are not too complex
• Assuming the Map steps are complex enough (another subprocess as step + state) different strategies could be taken, since at this point the map steps act as workers -> trying to get multiple jobs (same jobs) done, and return a result:
  • all Map steps steps are singletons and share same resources, so on pause/resume only 1 state per map step step is saved -> using as example the foodprep -> if 5 fish orders get triggered the ingredient stock should be globally -5
    • also the singleton step may be an interesting feature to explore when reusing steps so they all use same resources just used multiple times by different processes, emitting result to different parent processes
  • on pause, the most completed process state gets saved, and on resume this one gets passed as initial state to all map steps:
    • with this consideration, when loading a processes with state at most for map steps we offer to pass along the same initial state to all map children

[esttenorio]

Additional considerations - not for this PR but for the feature in general:

• timeout management: if spawning N events, have timeout logic and if some map steps are not done, null is returned
• limiting number of N elements: depending on how complex the inner steps are (nested subprocesses)
• volatile state: since map steps are spawned on runtime have checkpoints before and after the map step

[crickman]

Additional considerations - not for this PR but for the feature in general:

• timeout management: if spawning N events, have timeout logic and if some map steps are not done, null is returned
• limiting number of N elements: depending on how complex the inner steps are (nested subprocesses)
• volatile state: since map steps are spawned on runtime have checkpoints before and after the map step

I like these considerations. I feel that timeout and depth considerations apply generally to the entire framework, not just the map-step.

I very much like the volatile concept. I have to admit, I'm not entirely clear on the pause or cancel mechanics in the runtime. I don't see the cancel-token being propagated much, but I need to take a closer look

[crickman]

• all Map steps steps are singletons and share same resources, so on pause/resume only 1 state per map step step is saved -> using as example the foodprep -> if 5 fish orders get triggered the ingredient stock should be globally -5

  • also the singleton step may be an interesting feature to explore when reusing steps so they all use same resources just used multiple times by different processes, emitting result to different parent processes

• on pause, the most completed process state gets saved, and on resume this one gets passed as initial state to all map steps:

Ben and I did discuss this "functoid" constraint for the map operation and felt it might be too limiting. At this point, the feature is functional without resorting to constraining the map operation. Also if we apply the "volatile" concept, I wonder if the need exists to further constrain the map operation.

Another question that comes to mind is how to enforce such a constraint (singleton / functoid)...i suspect such enforcement may not be trivial.

What do you think?