This is actually used in libdispatch for pretty much all non-wall-clock
timers, so as you might imagine, it's used quite frequently and we were
processing it incorrectly, leading to stalls in some programs that use
such timers for short intervals/deadlines.
As SO_ACCEPTCONN does not work properly on these systems,
we have to rely on Darling to report listening sockets in
libsystem_kernel and keep a registry of listening sockets
in libkqueue.
The two most notable fixes are:
1) libsystem_kernel now delegates closing kqueue descriptor to us.
Rather than close the descriptor and inform us later, it allows us to
handle kqueue closing ourselves.
This avoids a double-close that was causing issues (most notably
with multi-threaded programs).
2) We now `map_delete` all references to kqueues being freed. We were
previously only deleting the original FD reference.
In addition, wherever `kqueue_delref` is called, the `kq_mtx` must be held.
This is required because that function can call `kqueue_free`, which
requires that lock to be held. While inefficient, this should help avoid
races between threads concurrently modifying the kqueue map. A better
solution should be devised later to make this more efficient.
libsystem_kernel's `sys_dup` and `sys_dup2` call `kqueue_dup`, so if we
use the normal `dup` call in libkqueue, we end up recursing back into
libkqueue (unecessarily).
Using this flag tells kevent to keep processing even when an error is
encountered adding an event. This means that it will actually check both
EVFILT_READ and EVFILT_WRITE now.
With the old code, we were leaking descriptors created for EVFILT_WRITE.
I added that race check back when I was trying to use libkqueue with update-sources because launchd was segfaulting and libkqueue seemed to be the cause, but I can't reproduce this anymore. The way it was, it was actually causing issues with EPOLLET events.
We were dropping events before because libdispatch calls kevent with this flag to update a knote without checking for new events; when it did that and we handed it some new events, it would ignore them since they didn't contain `EV_ERROR`, which meant those events were lost forever.
If the copyout callback for a filter indicates that the event should be ignored/dropped, we should not process EV_DISPATCH or EV_ONESHOT, as this would lead to missing actual events in the future.
Some filters (e.g. EVFILT_PROC) do specifically want to process EV_ONESHOT even when dropping an event, so this also introduces a special filter ID (EVFILT_DROP_POSTPROCESS) that enables the old behavior for individual events. Additionally, rather than just using `0` as a special filter ID, this commit introduces EVFILT_DROP (to be clear about its meaning).
See the darlingserver repo for the majority of the logic behind the new implementation.
TODO: libkqueue needs proper kevent_qos support in order to be able to use the data_out parameter passed in to kevent_qos calls (which is what *should* be passed as the default buffer to use when ext[0] is not available in EVFILT_MACHPORT).
If multiple threads are waiting on the same kqueue, they'll all be woken up at the same time, however only one can acquire the lock at a time. The one that acquires it first gets to consume some events from the kqueue while everyone else waits. But, after they're done and they release the lock, the next person will have *less events available*. We were previously neglecting to check for this and trying to consume more events than were available (leading to some nasty deadlocks with certain events like EVFILT_USER).
