feat: add URIs to schema

[fzipi]

what

• add URIs as a new field

why

• supporting using multiple URIs for testing multiple targets with the same parameters
• Make it possible to run the same test against multiple URIs go-ftw#256

[RedXanadu]

Does this PR break the axiom that 1 test = 1 request? Or has that already been broken?

What is the goal with this change? To reduce duplication?

How useful would this be in practice? E.g. do we have lots of tests that would then become 1 single test?

I may be misunderstanding this change, but it looks like a very significant change.

[fzipi]

Sorry, forgot to add the reference to coreruleset/go-ftw#256. This makes sense mostly for exclusion rules in plugins, where we need to test many URLs are not matching certain rules.

[theseion]

Suggested change

	// value: '/get?hello=world"
	// value: '/get?hello=world'

[theseion]

Suggested change

	// If URIs is set, URI will be ignored. URIs is useful for tests that need to send multiple requests.
	// If URIs is set, URI will be ignored. URIs is useful for running the same test against multiple URIs, e.g., when
	// matching on URI paths.

[theseion]

Does this PR break the axiom that 1 test = 1 request? Or has that already been broken?

That was never the case, technically, since even the old framework supported "stages".

What is the goal with this change? To reduce duplication?

Yes, mainly. For example, when testing rule exclusions, you often need to test multiple URI paths, while all other test data remains the same.

How useful would this be in practice? E.g. do we have lots of tests that would then become 1 single test?

Yes, most notably in plugins.

I may be misunderstanding this change, but it looks like a very significant change.

No, it should be straight forward and is backwards compatible.

[theseion]

While poring over the unix RCE FPs, I had a thought. Instead of making URIs an additional field, we could make "dynamic tests" a feature of go-ftw. There might be other use cases for "generating" tests, e.g., for word lists, and with the approach here we would need to modify the schema for every single option. However, if we introduce a templating mechanism (like you had already suggested once), we could generate tests from arbitrary data. We could instead extend the schema with something like this:

# ...
templates:
  - key: uri
     values: 
       - /get
       - /get?foo=bar
   - key: command
      values:
       - php
       - php%20foo

key would designate the template key and values a list of values. The templates could be Go template patterns, e.g.,

# ...
input:
  uri: "{{ uri }}"
  method: POST
  data: "cmd={{ command }}"

The set of tests would be the cross product of all combinations. We could maybe distinguish these "template tests" from existing tests by introducing a new "template test" type.

What do you think?

[fzipi]

You are getting nearer a vision I had this retreat about how the future will look like for writing a rule.

It definitely implies adding templating. Also, 99% of our tests only need one template. We tend to repeat 90% of the text we do in tests.

I plan to write a blog post about this, but will take me some time to process when heading back tomorrow.

Another good source of inspiration could be... nuclei templates.