version 7 - Design

[chris-pardy]

Starting a thread here to put notes on a version 7 design.
Version 7 is an opportunity to make a number of changes that have been suggested in issues filed here that would make the JSON rules engine more extensible at the cost of breaking backwards compatibility with the current 6.x versions.

Extensibility vs. Structure

With version 7 we want to consider the tradeoff between extensibility and having a known structure. For instance introducing the ability to add custom Condition classes or custom Rule subclasses allows for the system to be highly extensible but makes it much harder to reason about the input and output of the rules engine without knowing about all these extensions.

Ultimately we need to draw this line somewhere and my initial inclination is to draw it in favor of more extensibility in order to allow the broadest possible use of the rules engine.

Concepts

Starting from a clean stale the following basic concepts will be part of the rules engine

Facts
Facts are things that are known by the rules engine during execution.

Almanac
The Almanac is the system for storing and looking up Facts.

Conditions
Conditions are executable, contain a priority, and return a result containing a boolean value. Generally there are 2 types of conditions, comparison conditions which check the value of a fact against another value, and composite conditions like all, any, and not which use 1 or more nested conditions to produce a result. By being fully extensible it's possible to introduce other types of conditions.

Rules
Rules are also executable and contain a priority. They generally are comprised of a set of conditions which are executed. Rules produce events as a result of execution.

Engine
The engine provides the mechanism to evaluate rules, and conditions. It includes mechanisms to resolve the values of facts or other special objects. Changing the engine can significantly change the behavior of the system.

[ejarnutowski]

A nice feature would be a synchronous API. To be honest, it seems unusual to provide only an async API when there's no I/O. If there was a synchronous API and the consumer REALLY wanted to use it in an async way, they could wrap it in a promise.

[chris-pardy]

A nice feature would be a synchronous API. To be honest, it seems unusual to provide only an async API when there's no I/O. If there was a synchronous API and the consumer REALLY wanted to use it in an async way, they could wrap it in a promise.

So the asynchronous nature of the engine comes from the fact that Facts can be asynchronous and can do I/O in order to allow a synchronous mode of execution we'd have to ensure there were no async facts but the coding gymnastics for that are pretty hard so it's easier for the whole thing to be async. But this did open up a new line of thought around the role of all these things.

Currently I've been working under the assumption this this is the interface for a Condition

interface Condition {
  priority: number;
  evaluate(almanac: Alamanc): Promise<ConditionResult>;
}

Moving to have such a minimal interface for conditions it allows the introduction of new condition types which means that the engine is easier to extend, but the downside(?) is that there's very little for the Engine to do here. Effectively the conditions take care of everything.

An alternative approach makes the conditions and other data structures purely just data and moves the execution / evaluation of the rules entirely into the engine. This would allow for us to take the same rules and run them in an sync or async engine. For something like that you'd usually use a visitor pattern:

interface Condition {
    visit(visitor: Visitor): void
 }
 
 class ComparisonConditions {

    visit(visitor: Visitor): void {
        visitor.visitComparison(this.fact, this.operator, this.value);
     }
 }

The tradeoff between these two patterns is really about if we want to lock down the features that can be described by conditions or if we want to lock down the more general behavior of the engine.

[chris-pardy]

Parameters

The rules engine current supports what we've called condition references, which are useful constructs for doing rule inheritance. What would immensely up the power of these would be to add support for parameters that could be passed to the reference so in your rule you'd have something like:

{
    "condition": "sharedCondition",
   "parameter": { "minAge": 21 }
}

Then in your shared condition you could do something like:

{
    "fact": "age",
    "operator": "greaterThanInclusive",
    "value": { "parameter": "minAge" }
 }

This seems great but it opens up a few rabbit holes worth going down

Passing Facts to Parameters

This is one is no brainer but you should be able to do this:

{
     "condition": "sharedCondition",
     "paramters": {
        "minAge": {
           "fact": "legalDrinkingAge",
           "parameters": { "country": "US" }
         }
       }
     }

But if you can pass facts to the parameters of a condition reference then why can't you pass them to the parameters of a fact, and then while we're at it why no allow passing a parameter to the parameters of a fact, so something like:

{
     "fact": "age",
     "operator": "greaterThanInclusive",
     "value": {
         "fact": "legalDrinkingAge",
          "parameters": {
                "country": { "parameter": "country" }
          }
       }
}

This is great and we should support it but limit it to one level deep, effectively this is already done in events.

Using Parameters on the Left hand side.

Let's assume we want to re-use something like this but check different facts:

{
      "fact": "word",
       "operator": "endsWith",
       "value": "y"
}

In the current state of the engine we have a strict rule which is only facts on the left hand side, not a big deal since we mostly have symmetric operators, still if we wanted to change what facts we were checking for we'd have to do something like:

{
    "fact": "letterY",
    "operator": "endOf",
    "value": { "parameter": "text" }
 }

This is annoying but it also means that we can't write this rule without having this highly specific fact in place, wouldn't it be better if we could do this?:

{
   "parameter" : "text",
   "operator": "endsWith",
   "value": "y"
}

Ok, makes sense but if we've done that why bother limiting what can be on the left hand side of the operator at-all?

{
    "value": "y",
    "operator": "endOf",
   "value": { "parameter": "text" }
}

Admittedly we'll need a better way to specify the fact that the left-hand-side will be a value but that's solvable.

What else can we do with parameters?

Once we have parameters the question becomes, are they useful outside of condition references and the answer is without a doubt YES and they specifically enable us to cross another functional barrier which is dealing with iteration.

Currently if you have a fact with the value ["game", "trying", "salad"] and you wanted to ask if any of the words end in "ing" how would you go about this.

1. Use a mix of the "any" condition and the the "path" attribute - this is fine but you'll need to know the number of items which may change between runs.
2. Write a custom operator eg. someEndWith. This would work but it would require the custom operator which would take the ability to express this out of the condition author's hand.

What I'd propose is to add the for condition so:

{
   "for": ["game", "trying", "salad"],
   "as": "word",
   "every": {
      "parameter": "word",
      "operator": "endsWith",
      "value": "ing"
  }
}

we could now put that list of words behind a Fact and reference each word in the condition check.

Recap

So what have we added?

• Support for a new thing called Parameters - they behave like facts but have very specific scoping to empower isolation.
• Conditions should support a LHS of Facts, Parameters, and Values
• The first level of parameters should be resolved to actual values if thye're references to functions.
• We should add support for some kind of iteration construct.

[chris-pardy]

Functions

With parameters and iterations there's really only one last bridge to cross and that's functions. In short a function call should let you modify values before they are passed to an operator. Something like

{
   "fact": "age",
   "operator:" "greaterThenInclusve",
  "value": {
       "fn:coallesce": [
            { "parameter": "minAge" },
           21
        ]
   }
}

[chris-pardy]

JSON Structure

The current JSON structure is ok at being something that can be statically checked but not the best. For instance this object:

{
    "fact": "test",
    "operator": "equal",
    "value": 2,
    "all": []
}

This will be treated like an empty all condition because that is checked first a generally better approach would be to use discriminated unions:

{
    "type": "comparison",
    "fact": "test",
    "operator": "equal",
    "value": 2,
    "all": []
}

This type is unambiguous because of the type field. However it signals a move away from a format that tries to be more human readable and towards a format that is more machine readable

Assume we were adding support for functions you could have something like:

{ "fn:max": [10, { "fact": "count" }] }

or you could have something like

{
   "type": "function",
   "function": "max",
   "args": [10, { "fact": "count" }]
}

This also informs when we need to know about things like operators, in the current version we know about operators at execution when we resolve the string name to an actual Operator instance. However if we wanted to add an onlyOne composite that was like all or any that would require us to know about that support when we parse the json into conditions. Fully going down the path of dynamic you'd want to have all / any look something like:

{
     "type": "aggregate",
     "operator": "all",
     "conditions": []
}

If we're comfortable knowing about operators a head of time then we could represent comparisons like:

{ "equal": [{ "fact": "test" }, 2] }

[chris-pardy]

Iterators

One thing you can't do in version 6 of the JSON rules engine is run the same set of conditions across multiple items in a list. For our example let's assume you have a list of items and you want to know if every item in the list is greater than 2.

Option 1 - Dedicated Iterator, pluggable aggregate opreations

{
    "type": "foreach",
    "foreach": { "fact": "items" },
    "as": "item",
    "operation": "all",
    "condition": {
          "type": "comparsion",
          "operator": "greaterThan",
          "operands": [{ "parameter": "item"}, 2]
    }
}

In this case the "operation": "all" specifies that this should use the same logic as an all aggregation that had a list of different conditions to validate.

Option 2 - Dedicated Aggregator, pluggable lists

{
   "type": "aggregate",
    "operator": "all",
    "conditions": {
        "type": "foreach",
        "foreach": { "fact": "items" },
        "as": "item",
        "condition": {
            "type": "comparison",
            "operator": "greaterThan",
            "operands": [{ "parameter": "item" }, 2]
         }
     }
}

In this case the foreach becomes a special iterator that is fed into the same all aggregation that would be used across a list of conditions.

[srcmayte]

Parameters

The rules engine current supports what we've called condition references, which are useful constructs for doing rule inheritance. What would immensely up the power of these would be to add support for parameters that could be passed to the reference so in your rule you'd have something like:

{

    "condition": "sharedCondition",

   "parameter": { "minAge": 21 }

}

Then in your shared condition you could do something like:

{

    "fact": "age",

    "operator": "greaterThanInclusive",

    "value": { "parameter": "minAge" }

 }

This seems great but it opens up a few rabbit holes worth going down

Passing Facts to Parameters

This is one is no brainer but you should be able to do this:

{

     "condition": "sharedCondition",

     "paramters": {

        "minAge": {

           "fact": "legalDrinkingAge",

           "parameters": { "country": "US" }

         }

       }

     }

But if you can pass facts to the parameters of a condition reference then why can't you pass them to the parameters of a fact, and then while we're at it why no allow passing a parameter to the parameters of a fact, so something like:

{

     "fact": "age",

     "operator": "greaterThanInclusive",

     "value": {

         "fact": "legalDrinkingAge",

          "parameters": {

                "country": { "parameter": "country" }

          }

       }

}

This is great and we should support it but limit it to one level deep, effectively this is already done in events.

Using Parameters on the Left hand side.

Let's assume we want to re-use something like this but check different facts:

{

      "fact": "word",

       "operator": "endsWith",

       "value": "y"

}

In the current state of the engine we have a strict rule which is only facts on the left hand side, not a big deal since we mostly have symmetric operators, still if we wanted to change what facts we were checking for we'd have to do something like:

{

    "fact": "letterY",

    "operator": "endOf",

    "value": { "parameter": "text" }

 }

This is annoying but it also means that we can't write this rule without having this highly specific fact in place, wouldn't it be better if we could do this?:

{

   "parameter" : "text",

   "operator": "endsWith",

   "value": "y"

}

Ok, makes sense but if we've done that why bother limiting what can be on the left hand side of the operator at-all?

{

    "value": "y",

    "operator": "endOf",

   "value": { "parameter": "text" }

}

Admittedly we'll need a better way to specify the fact that the left-hand-side will be a value but that's solvable.

What else can we do with parameters?

Once we have parameters the question becomes, are they useful outside of condition references and the answer is without a doubt YES and they specifically enable us to cross another functional barrier which is dealing with iteration.

Currently if you have a fact with the value ["game", "trying", "salad"] and you wanted to ask if any of the words end in "ing" how would you go about this.

1. Use a mix of the "any" condition and the the "path" attribute - this is fine but you'll need to know the number of items which may change between runs.

2. Write a custom operator eg. someEndWith. This would work but it would require the custom operator which would take the ability to express this out of the condition author's hand.

What I'd propose is to add the for condition so:

{

   "for": ["game", "trying", "salad"],

   "as": "word",

   "every": {

      "parameter": "word",

      "operator": "endsWith",

      "value": "ing"

  }

}

we could now put that list of words behind a Fact and reference each word in the condition check.

Recap

So what have we added?

• Support for a new thing called Parameters - they behave like facts but have very specific scoping to empower isolation.

• Conditions should support a LHS of Facts, Parameters, and Values

• The first level of parameters should be resolved to actual values if thye're references to functions.

• We should add support for some kind of iteration construct.

I cannot stress enough how powerful this would be. However, I would say there is no need for "parameters" per se, just pass in params that match the params of the predefined conditionals.