kqueue: use EVFILT_USER as wakeup event if available

[panjf2000]

Eliminate the kernel overhead of pipe if we can.

[panjf2000]

Huh, a few confusing test failures surface.

[bnoordhuis]

I experimented with EVFILT_USER around 2013 and I remember it was broken on macos. Looks like it still is?

I'm afraid I don't really remember the details. Possibly it only worked as a oneshot event, although it seems libevent uses it in multishot mode.

[panjf2000]

I experimented with EVFILT_USER around 2013 and I remember it was broken on macos. Looks like it still is?

I'm afraid I don't really remember the details. Possibly it only worked as a oneshot event, although it seems libevent uses it in multishot mode.

EVFILT_USER doesn't seem to exist in the macOS man pages anymore. Maybe there is indeed something wrong with EVFILT_USER.

[panjf2000]

Besides, it looks like EVFILT_USER is available on NetBSD since this commit.

Unfortunately, FreeBSD and NetBSD are not available with GitHub Actions, but something like cross-platform-actions may come to help.

[vtjnash]

Would it be sensible to try creating a zero-second one shot timer timeout instead?

[panjf2000]

Would it be sensible to try creating a zero-second one shot timer timeout instead?

I don't think this is the right path. In that way, we're unable to ensure a real-time interruption to kqueue/epoll/... cuz it needs to wait for the in-flight uv__io_poll() to finish or for the next round to begin.

[vtjnash]

Isn't it waiting for those anyways?

[panjf2000]

Isn't it waiting for those anyways?

No, it isn't. And I think that's why async.c was created, to allow other threads to emit real-time interruptions to the main thread, waking it up from the polling immediately. Please correct me if I'm missing something.

[vtjnash]

A zero second timer should wake it immediately too, though I won't claim to know if that is a useful alternative to the existing pipe

[panjf2000]

Kindly ping @bnoordhuis

[bnoordhuis]

After thinking it over, I'm somewhat disinclined to merge this. It's for niche platforms that I personally don't care about, the changes get intermingled with platform code that I do care about, and the real-world performance improvements aren't clear.

I could maybe be persuaded if it gave a really good bump on the async handles benchmarks (grep test/benchmark-list.h for "async") but no promises.

[panjf2000]

After thinking it over, I'm somewhat disinclined to merge this. It's for niche platforms that I personally don't care about, the changes get intermingled with platform code that I do care about, and the real-world performance improvements aren't clear.

I could maybe be persuaded if it gave a really good bump on the async handles benchmarks (grep test/benchmark-list.h for "async") but no promises.

From where I stand, well-testing can justify merging code of improvement. Besides, NetBSD may not catch your attention, but FreeBSD is the Tier-2 support of libuv (also the most popular *BSD system). So if we can run the test suite of libuv on FreeBSD and no errors occur, I'd say it is worth going along.

[bnoordhuis]

Let's see where other maintainers stand on this.

[panjf2000]

I've finally discovered the root cause that had been failing the tests on macOS before, uv__async_send() might get called before uv__io_poll() with multi-threads, which resulted in an ENOENT returned from kevent() and the process got aborted. I think we can resume this PR now! @bnoordhuis @vtjnash

[panjf2000]

There might be some code conflicts with #4400.

[panjf2000]

Some references of EVFILT_USER on various platforms:

• FreeBSD kqueue
• NetBSD kqueue
• DragonFlyBSD annotated tag v4.0.0rc created
• macOS swift-nio
• XNU kernel

[vtjnash]

Okay, it wasn't clear that EVFILT_USER was available on XNU. Makes more sense now that is clear it is available in the XNU sources, just not documented. Could you run the benchmark on this?

[panjf2000]

Okay, it wasn't clear that EVFILT_USER was available on XNU. Makes more sense now that is clear it is available in the XNU sources, just not documented. Could you run the benchmark on this?

Environment

 Model : Mac Studio (2022)
    OS : macOS 14.1.2
   CPU : Apple M1 Max, 10-core CPU with 8 performance cores and 2 efficiency cores
Memory : 32GB unified memory

Benchmark command

build/uv_run_benchmarks_a million_async

Benchmark results

Branch: libuv:v1.x

ok 1 - million_async
# 10,348,986 async events in 5.0 seconds (2,069,797/s, 1,048,576 unique handles seen)

Branch: panjf2000:kqueue-wake

ok 1 - million_async
# 12,977,383 async events in 5.0 seconds (2,595,476/s, 1,048,576 unique handles seen)

@vtjnash

[saghul]

Nice!