Thread pool

[kvark]

Possible interface could be:

enum SpawnMode {
   Unlimited, // currently implemented mode, one thread per task
   ThreadPool(u8), //count
   AffinityMask(u64), //mask
}

[ghost]

Doing some research the normal strategy is a work-stealing thread pool that uses random selection of the victims. The random selection use a back off algorithm.

I think this a good strategy for small short lived tasks. Tasks can be started at any time, by any thread. If the thread is a scheduler thread the task will be enqueued into its work queue. If the thread is outside of the threadpool, it will use a channel of some sort to inject the tasks into the pool from outside. This could just look like another stealer queue to the other threads.

Tasks should get flagged if they want to run in the pool, or run on a dedicated thread. There is actually no real difference for the worker. The worker will run a thread stealing task, the difference would be that the scheduler would see that stealer task and request a new pool thread be created to take its place.

I think it makes sense to have two watermarks, a low watermark and a high watermark for worker thread count. Iff starting a thread stealing task would drop the thread count below the high watermark the new thread spawn logic is called. The high watermark would be use to limit the number of pool threads, Since stealing tasks would eventually return the thread, the high watermark can be used to determine if a thread needs to be killed when it goes idle.