Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: RuleTester supports processor #31

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: RuleTester supports processor #31

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4af9338
New: RuleTester supports processor
mysticatea Jul 15, 2019
07fd2eb
Update RFC
btmills Nov 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
70 changes: 70 additions & 0 deletions designs/2019-rule-tester-processors/README.md
View file
Open in desktop
@@ -0,0 +1,70 @@
- Start Date: 2019-07-16
- RFC PR: (leave this empty, to be filled in later)
- Authors: Toru Nagashima ([@mysticatea](https://github.com/mysticatea))

# RuleTester supports processor

## Summary

This RFC makes `RuleTester` class supporting `processor` option.

## Motivation

Currently, we cannot test rules with processors. This is inconvenient for plugin rules which depend on processors. For example, [vue/comment-directive](https://github.com/vuejs/eslint-plugin-vue/blob/6751ff47b4ecd722bc2e2436ce6b34a510f92b07/tests/lib/rules/comment-directive.js) rule could not use `RuleTester` to test.

## Detailed Design

To add `processor` option and `processorOptions` (see #29) to test cases.

```js
const { RuleTester } = require("eslint")
const rule = require("../../../lib/rules/example-rule")
const exampleProcessor = require("../../../lib/processors/example-processor")

const tester = new RuleTester()

tester.run("example-rule", rule, {
valid: [
{
code: `
<script>
console.log("Hello")
</script>
`,
processor: exampleProcessor,
processorOptions: {},
},
],
invalid: [],
})
```

### § `processor` option

This is a definition object of a processor that the "[Processors in Plugins](https://eslint.org/docs/developer-guide/working-with-plugins#processors-in-plugins)" section describes.

If this option was given, the tester applies the given processor to test code.

If the given processor didn't has `supportsAutofix:true`, the tester doesn't do autofix. Then if the test case had `output` option (except `null`) then the test case will fail.

### § `processorOptions` option

RFC #29 defines the `processorOptions`.

If this option was given along with `processor` option, it will be given to the processor.

## Documentation

The [RuleTester](https://eslint.org/docs/developer-guide/nodejs-api#ruletester) section should describe the new `processor` and `processorOptions` properties.

## Drawbacks

It's a pretty rare case of a rule depends on the processor's behavior. I'm not sure if this feature is needed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that this is needed either. Unit tests are supposed to test smallest unit of code (unit). So I think it's perfectly fine to feed some pre-processed code to the tests to check if the rule is working fine, and then create separate unit tests to verify that processor is working correctly as well. I think adding processor to the ruleTester will encourage bad practice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of discussion, can you elaborate on what bad practices this might encourage?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also be curious about the bad practices you are referring to and in general I'm not sure I follow this logic - integration tests are far more valuable than unit tests, you could definitely have processor unit tests and rule unit tests individually passing but not have a working solution for your users.

Additionally, you could argue that RuleTester is already not a true unit test for rules, given they operate on ASTs, not on source code which is what you provide to RuleTester.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaicataldo I'm talking about bad practice of increasing the scope of a "unit".
@JamesHenry I don't want to argue about which types of tests are more valuable. They are supposed to test different things, and each one is valuable in their own right. You are correct about RuleTester not operating on AST, which makes it none pure unit testing library. But in my opinion it's a convenience part of the library. It would be very inconvenient to test directly on AST, so library is providing convenience feature to make it easier. But the fact of the matter is that the library is called "RuleTester", and processors are not related to rules at all. It's also quite easy to test rules and processors separately. It's also reasonable easy to create integration tests outside of RuleTester to test the whole flow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I have written vue/comment-directive rule in the motivation section, I have a rule that requires a specific processor. That rule is to implement <!-- eslint-disable --> directive comments in HTML templates of Vue.js. That rule makes messages at every directive comment and vue.postprocess() function filters those messages and the messages which are disabled by the directive comments. That rule requires processor option for their unittest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mysticatea I don't want to derail this RFC, but it almost seems to me that what you need in your case is ESLint to support AST extensions to mark certain tokens as disable comments, which ESLint would then handle as if they were disable comments in JavaScript. Then you would be able to decorate your comment tokens in the Vue parser and ESLint would just apply its own disable logic the way it normally would. But maybe I'm missing something.


## Backwards Compatibility Analysis

There are no concerns about breaking changes.

## Related Discussions

- https://github.com/eslint/rfcs/pull/25#issuecomment-499877621