graphemer: 'अ', 'नु', 'च्', 'छे', 'द'
Intl.Segmenter: 'अ', 'नु', 'च्छे', 'द'

Probably related to orling/grapheme-splitter#26

graphemer seems to be unicode v15.0 compliant and uses GraphemeBreakTest v15.0 for testing.
However, if I replace this with GraphemeBreakTest v15.1, graphemer will fail the test. The test also passes in GraphemeBreakTest v15.1 when using Intl.Segmenter.

https://github.com/flmnt/graphemer/blob/master/tests/GraphemeBreakTest.txt
https://www.unicode.org/Public/15.0.0/ucd/auxiliary/GraphemeBreakTest.txt
https://www.unicode.org/Public/15.1.0/ucd/auxiliary/GraphemeBreakTest.txt

 if (!segmenter) {
        segmenter = new Intl.Segmenter();
    }

Per MDN - Intl.Segmenter() constructor, it expects a locales argument. Per locales argument docs, if it's omitted, the implementation's default locale will be used.

Does that mean that we could have different behavior in different environments? For example, linting passes locally but then fails on CI because the default locale is different?

Hmm. There may be differences depending on locale.
I've read the following links but still can't find a concrete example.

https://github.com/tc39/proposal-intl-segmenter/blob/master/README.md#why-should-we-pass-a-locale-and-options-bag-for-grapheme-boundaries-isnt-there-just-one-way-to-do-it
https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries:~:text=tailored%20grapheme%20clusters%20for%20specific%20locales

@ota-meshi are you still interesting in working on this? Are you still doing research?

I haven't looked into it much since then. Perhaps the language in which the problem seems to be a language that is unfamiliar to me so I am having trouble understanding it 😓

If that's the case, shall we close this?

What do you think is the best way to resolve the issue of incompatibility with Unicode v15.1?
Would you like to close this issue by reporting it on graphemer repository? In my opinion it is better to use Intl.Segmenter.

Can we specify en-US as the default language?

That idea sounds good to me! At least I don't think the behavior will change depending on the environment.

I like the idea of using Intl.Segmenter for simplicity but the results it produces will inevitably change when a new version of Node.js switches to a newer version of Unicode. The effect would be that the same rule on the same code could give different results on different versions of Node.js. I don't think this is terribly bad, I think it's actually better than depending on a huge semi-automatically generated program that eventually gets outdated (lib/rules/utils/patterns/letters.js is another one), but it's a complication to keep in mind. Regular expressions with character properties like \p{Mc} can also change their behavior with the version of Unicode, and we are already using them.

I think we're at the point where we either need someone to prototype this or we should close this issue. Any volunteers?

I went ahead and prepared a draft: #18110.

One problem that hasn't been discussed yet is that Intl.Segmenter is not implemented in Firefox (though it could be soon, the corresponding issue is marked as fixed: https://bugzilla.mozilla.org/show_bug.cgi?id=1423593). This means that if a rule using Intl.Segmenter were opened in the Playground in Firefox, it would crash. So if we decide to go for this change, we should at least wait until Intl.Segmenter is supported in all major browsers.

One problem that hasn't been discussed yet is that Intl.Segmenter is not implemented in Firefox

Ooh, great catch! I agree, we should definitely hold off until Firefox supports this.

Oops! It looks like we lost track of this issue. What do we want to do here? This issue will auto-close in 7 days without an update.

Blocked by Firefox implementing Intl.Segmenter, so not stale.

Intl.Segmenter is unflagged in Firefox 125, so we could consider using it instead of graphemer now. Anyway, there is also an alternative proposal to replace graphemer with a different package: #18359.

I'd definitely prefer using Intl.Segmenter.