Bug: Error: EMFILE: too many open files

[last-Programmer]

Environment

Environment Info:

Node version: v21.7.2
npm version: v10.5.0
Local ESLint version: v9.0.0 (Currently used)
Global ESLint version: Not found
Operating System: win32 10.0.23606

What parser are you using?

@typescript-eslint/parser

What did you do?

I have migrated my web project to eslint 9 & flat configuration file.

What did you expect to happen?

eslint should complete running without any error

What actually happened?

eslint terminated with reporting the following error.

C:\S\eslint-emfile-repro git:[main]
npm run lint

[email protected] lint
eslint ./

Oops! Something went wrong! :(

ESLint: 9.0.0

Error: EMFILE: too many open files, open 'C:\S\eslint-emfile-repro\src\file_8367.tsx'
at async open (node:internal/fs/promises:637:25)
at async Object.readFile (node:internal/fs/promises:1246:14)
at async Promise.all (index 8189)
at async ESLint.lintFiles (C:\S\eslint-emfile-repro\node_modules\eslint\lib\eslint\eslint.js:863:25)
at async Object.execute (C:\S\eslint-emfile-repro\node_modules\eslint\lib\cli.js:461:23)
at async main (C:\S\eslint-emfile-repro\node_modules\eslint\bin\eslint.js:165:22)

Link to Minimal Reproducible Example

https://github.com/last-Programmer/eslint-emfile-repro.git

Participation

• I am willing to submit a pull request for this issue.

Additional comments

It is already reported here but was closed #17249

[nzakas]

Thanks for the report. Just to verify, are you saying that you didn't get this error with ESLint v8.x but did after upgrading to ESLint v9.0.0?

[last-Programmer]

@nzakas thank you for the reply. You are right i am not getting the error in ESLint v8.x but getting the error after upgrading to ESLint v9.0.0.

[nzakas]

And just to double-check, you are using the same eslint.config.js file with both v8.x and v9.0.0?

EMFILE errors are notoriously difficult to reproduce because they are just the operating system saying "I can't assign any more file descriptors." That means it could be something else you're running that's working on files at the same time.

Fundamentally, there is no difference between the files that ESLint reads and writes between v8.x and v9.0.0, so this may not be an actual problem with the code.

[last-Programmer]

@nzakas with v8.x i am using the old legacy config file and with v9.0.0 i am using the new flat config file.

EMFILE errors are notoriously difficult to reproduce

Can you please try this https://github.com/last-Programmer/eslint-emfile-repro.git repro and you can see that we get the EMFILE error every time when we lint. I have tried only in Windows 11. not quite sure how it behaves in mac os and linux.

[nzakas]

I can get the EMFILE error with your repro on Windows 10.

I have an idea of how to address this, will need some time to play around.

[aladdin-add]

This seems to be related to OS settings and is a very extreme case. Can we add it to the trouble-shooting page to help users to change the OS settings?

[nzakas]

@aladdin-add I don't think changing the OS settings is the correct approach here. I've submitted a PR.

[last-Programmer]

@nzakas thanks for the quick commit. I have tried the patch mentioned in #17249 and it works too.

[last-Programmer]

@nzakas FYI This does not happen in Sonoma Mac OS.

[aladdin-add]

On my macos (Sonoma v14.4), the limit is 10496. the provided example is 10000, so it won't happen.

Default settings may vary per os. If we want to fix the "very edge" (IMHO) case, the most reliable way is to use something like p-limit (as suggested in #17249)

[nzakas]

@aladdin-add Yeah, I don't like p-limit as a solution to this because, as you mentioned, we don't actually know what the limit should be on any given OS. The PR doesn't require any knowledge of that limit and just transparently handles retrying calls that fail with EMFILE errors.

We could up the number of files to 10500 in the test to account for the default MacOS limit. I'll also add a test that intentionally fails to verify.

[last-Programmer]

@nzakas @aladdin-add in my mac i have a project with more than 25000 typescript files and i am not facing any EMFILE errors with flat config. so the limit for mac os is not 10500. i did not remember changing any default settings

[nzakas]

Yes, the actual limit is much higher on Linux systems. It looks like Ubuntu has 65,000+. I've updated the PR to account for that by using the actual descriptor limit on the system.

[nzakas]

For posterity, here's what happened:

With eslintrc and the old API, we were reading files in sequentially, one at a time. That meant that large amounts of files could be read easily without running into EMFILE errors because there were never two files being read at the same time.

With flat config and the new API, reading is done asynchronously in order not to block on a single file read, and that means we can end up initiating multiple file reads at the same time, leading to the EMFILE error on the same amount of files that the eslintrc API could handle easily.

[last-Programmer]

@nzakas @mdjermanovic Thanks for the fix. When this will be released to public npm?.

[aladdin-add]

We generally publish once every 2 weeks, and the next release can be seen here: #18293

[last-Programmer]

This is happening with eslint v8.xx with flat config file also.