-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Google Summer of Code 2018
This page contains project ideas for Google Summer of Code 2018!
Seastar is an open-source C++ framework for writing high-performance server applications. This framework allows you to write non-blocking, asynchronous code in a relatively straightforward manner (once understood). The high-performance is achieved by bringing together a set of architectural innovations, which includes shared-nothing design, high-performance networking, futures and promises, and message passing.
Examples of seastar applications include ScyllaDB, a high-performance NoSQL database compatible with Apache Cassandra, smf, set of libraries and utilities designed to be the building blocks for distributed systems, and Pedis, a faster drop-in replacement for Redis.
- Seastar's API can be referenced at: http://docs.seastar-project.org
- This tutorial contains very useful information for understanding better the primitives provided by the framework, its programming model, how to efficiently debug a Seastar program, and so on.
- If you have any question, feel free to send an e-mail to seastar's mailing list or contact the mentors directly.
- Please also refer to FAQ available in the project's official website.
If you're interested in knowing more about Seastar, gather details about other potential ideas or make a proposal based on these ideas, please send an e-mail to [email protected]. We would be happy to help.
Seastar provides its own I/O scheduler to prevent one component of the system from starving another. Glauber Costa, the creator of this feature, wrote two articles about it, which can be found here (part 1) and here (part 2).
The goal is to allow that a Seastar application can work with multiples disks, each of them having its own filesystem mounted at different directories. I/O scheduler was designed to work on behalf of a single disk, so support for multiple disks can be introduced by creating one instance of I/O scheduler for each disk available. If you read the article part 2, you'll know that I/O scheduler has parameters, e.g. -max-io-requests, to teach it about the limits of the disk (for example, how many requests can it handle in parallel?), so if there's more than one I/O scheduler, user should be able to specify the parameters for each existing one. It's also essential that I/O priority class system is extended to allow a specific priority class to be assigned to a particular I/O scheduler.
The tool iotune, also explained in article part 2, should be extended to work with multiple disks (accept multiple directories), which is about generating the I/O scheduler parameter values for each disk.
Relevant sources:
- iotune: https://github.com/scylladb/seastar/blob/master/apps/iotune/iotune.cc
- i/o scheduler relevant code: https://github.com/scylladb/seastar/blob/master/core/reactor.hh#L520, https://github.com/scylladb/seastar/blob/master/core/reactor.cc#L698
Expected results: Native userspace I/O scheduler being capable of working with multiple disks.
Difficulty: Medium
Required skills: C++
Mentor: Glauber Costa
HTTP/2 protocol will go very nicely with Seastar because it supports multiple asynchronous requests over a single TCP connection, unlike HTTP/1.1 which requires the server to send its responses in the same order that requests were received (a problem also known as HOL blocking). So HTTP/2 support will allow Seastar applications relying on HTTP protocol for its services to deliver a much better performance.
Seastar's HTTP module can be found here: https://github.com/scylladb/seastar/tree/master/http, and the HTTP server can be found here: https://github.com/scylladb/seastar/tree/master/apps/httpd.
HTTP/2 specification: https://tools.ietf.org/html/rfc7540
Expected results: HTTP module extended to provide applications with HTTP/2 support, and consequently better communication performance due to asynchronous nature.
Difficulty: Medium
Required skills: C++, HTTP/2 protocol
Mentor: Piotr Jastrzebski, Amnon Heiman
Improve behavior in congested environments (the Internet).
BBR: Congestion-Based Congestion Control [1][2] is a new TCP congestion control created by Google. Currently, congestion control in Seastar follows the standard but pretty old RFC5681[3] and RFC6582[4]. BBR is merged into Linux kernel since 4.9. Implementing BBR in Seastar will bring the modern congestion control algorithms to Seastar.
[1] http://queue.acm.org/detail.cfm?id=3022184
[2] https://github.com/google/bbr
[3] https://tools.ietf.org/html/rfc5681
[4] https://tools.ietf.org/html/rfc6582
Difficulty: Hard
Required skills: C++, TCP/IP protocol
Mentor: Asias He
Not all Linux filesystems seriously support asynchronous I/O (learn more about it). Under certain scenarios, they can cause the thread to block waiting for I/O, and that's not acceptable for an application that uses a thread-per-core design. XFS has proven to provide the best support for AIO so far.
So Seastar needs a filesystem that is fully asynchronous, and that's also aware of its shared-nothing design. That means, the on-disk format needs to take into account that each shard (Seastar terminology for core acting like an isolated machine) will have data that only belongs to itself, but very occasionally, two shards may share the same data. The good news is that there's no need for POSIX compliance. Seastar filesystem only needs to provide a notion of files (includes the ability to retrieve file info), and provide an asynchronous API for its users.
During the development:
- loopback device can be used as the block device for development purposes, because the student isn't expected to have an available block device around. In the future, we expect to bypass Linux by using SPDK, in which we'll have direct access to a NVMe drive.
- Take a look at file API as it provides the asynchronous API for communicating with the block device. Linux page cache is bypassed by using O_DIRECT flag.
Getting started: Please take a look at https://github.com/scylladb/seastar/blob/master/core/file-impl.hh to know how you could possibly implement the new file system by extending file_impl.
The new filesystem would start something like:
class native_file_impl : public file_impl {
// FIXME: implement
};Expected results: Native filesystem that Seastar applications can alternatively use on their applications, and that will potentially provide more performance because of its simplicity and friendliness to Seastar than its counterparts.
Difficulty: Hard
Required skills: C++, file system design and implementation, AIO, modern storage devices (like SSD and NVMe)
Mentor: Raphael S. Carvalho
Modern storage devices make it interesting for Seastar's memcached to persist data. Basically, we want to start persisting least-frequently-used data to disk when memory starts filling up. Important to mention that persistence should be made optional (disabled by default) to not affect users that aren't interested in using it.
Currently, a LRU list is used to keep track of recency of items (https://github.com/scylladb/seastar/blob/master/apps/memcached/memcache.cc#L373), and slab allocator is used to actually store items (https://github.com/scylladb/seastar/blob/master/core/slab.hh). So when there's memory pressure, slab will call back into memcached to evict items.
So instead of evicting items altogether, we should persist most of item's data and metadata to disk and only keeps in memory metadata to locate an item in disk and bring it back into memory. It's important that the on-disk format is fragmentation-proof, so a log-structured format (https://en.wikipedia.org/wiki/Log-structured_file_system) seems to be the most interesting approach. It's basically about sequentially writing data into a fixed-length log, and persisting that same log to disk when it fills up. Eventually, on-disk logs can be compacted together so as to reduce fragmentation.
https://groups.google.com/d/msg/seastar-dev/CR0zMBQMjIA/0CyOxVNwK2gJ can be used as a reference. It contains work from Paweł Dziepak towards this goal but it still doesn't add support to disk storage.
Expected results: Seastar memcached will optionally be able to persist least-frequently-used data using a log-structured on-disk format.
Difficulty: Medium
Required skills: C++, asynchronous I/O
Mentor: Raphael S. Carvalho