-
Notifications
You must be signed in to change notification settings - Fork 6
life of a service
THIS DOCUMENT NEEDS REVISION. IT IS SLIGHTLY OF SYNC
We borrow the notion of a "cell" from Google. In Google a cell is, roughly, a collection of machines that share a Chubby-installation -- in our world that would be a Zookeeper installation. One Zookeeper cluster should be within one datacenter and very likely inside one availability Zone. (Zookeeper clusters for managing inter-cell activity is beyond the scope of these notes).
Each machine in a cell knows where Zookeeper runs. For instance by being able to resolve a set of hostnames that should always resolve to the Zookeeper machines. We assume that the hostnames "zk1".."zkN" will resolve to the nodes in the Zookeeper cluser and, in the presence of a DNS server, the hostname "zk" points to A records for all the Zookeeper nodes. (If none of this makes any sense, ask someone who knows how DNS work to explain it to you).
Software artifacts are distributed in packages using the "copkg" format. The copkg format is a simple ZIP file containing the following directories:
- bin - binaries and jar files
- lib - extra libraries and anything that doesn't fit in bin or etc
- etc - global config (templates and configs that are identical for all instances)
- script.d - lifecycle management scripts in Python
A copkg package is identified by a package coordinate.
Essentially a package coordinate is not unlike a Maven coordinate. It
has three components: group, artifactId and is expressed as a ":"
delimited string. For example: com.comoyo:james:1.2.3 and
com.comoyo:idee:3.1.
The copkg format is also intended to support third party applications such as MongoDB, Apache httpd etc.
Given a Package Coordinate, it must be possible to resolve the download URL for the artifact given a base-URL for the software distribution web server. Software is distributed from an ordinary HTTP server that serves a directory tree. The precise mapping function from coordinate to URL is defined in the copkg library (work in progress).
Installing a copkg package consists of downloading it to the target machine, unpacking it in a staging area, verifying the contents and then perform an atomic move into the software directory structure. A software artifact is installed according to its package coordinate. Eg:
com.comoyo:idee:2.3 => /prefix/com.comoyo/idee/2.3
When a software artifact is started a runtime directory that is completely separate from the above install directory must exist. Once installed, nothing is allowed to be changed under the software directory.
We assume that the reader is familiar with Cloudname. In particular the notion of service coordinates. While Cloudname isn't directly used for deployment, it is used to uniquely identify services.
In this section we will provision and start the imaginary service FooService. We will provision 3 instances and these instances will connect to BarService. It is assumed that BarService runs already.
First we need to create the coordinates for FooService using the cn
command line program. (The cn program does not exist yet. Also,
don't get too hung up on verbosity at this point. We can automate
things later):
cn create 1.fooservice.test.ie
cn create 2.fooservice.test.ie
cn create 3.fooservice.test.ie
The above will create 3 coordinates for the FooService service. The creation of a coordinate does not imply that a service is running. It only means that the above 3 coordinates are defined for the cell we are currently in.
Starting a service is done by instructing Nodee to start the service. This is done by POSTing a JSON document to Nodee containing the following:
{
coordinate : "1.fooservice.test.ie",
package : "com.comoyo:fooservice:2.4",
params : {
"bar-service" : "rest.any.barservice.test.ie",
"foo-dir" : "some/local/dir/relative/to/runtime/directory",
"some-param" : "somevalue"
}
}
To start a service, Nodee only needs to know the service coordinate, the package coordinate and the list of parameters the service requires. At this point we will not address how to convey more bulky configuration.
When Nodee gets the above POST it will
- ensure that the package is installed in
/prefix/com.comoyo/fooservice/2.4 - ensure that the runtime directory exists at
/runtime/ie/test/fooservice/1 - change current directory to the runtime directory
- execute
/prefix/com.comoyo/fooservice/2.4/script.d/start.pywith the appropriate command line options derived from the params section plus a mandatory--coordinatecommand line option to determine the runtime directory
Note that the start script should return so that the running process will be detached from the shell or process that started it. This is to prevent Nodee developers from trying to implement process tracking directly in Nodee -- which will be fiddly and error prone and which leads to complicated scenarios should Nodee crash. In short: don't go there.
The start.py script will typically want to record a process ID or
similar when starting. It will also record the start.json document
POSTed to Nodee. This should always be done in the runtime directory.
It would be helpful if we define conventions for the runtime
directory. For instance PID files might belong in run: so for the
service instance 1.fooservice.test.ie the PID file would end up in:
/runtime/ie/test/fooservice/1/run/process.pid and the start.json
file would end up un /runtime/ie/test/fooservice/1/run/start.json.
(If a PID file exists and the process ID in that PID file exists, the start.py script should terminate with a nonzero return value indicating that the service appears to be running already or that cleanup/intervention is required).
This is where Nodee's responsibility for starting a service ends. From here on other parts of the system take over.
When a service starts the current directory will be the root of the runtime directory. The first thing a service needs to do is to claim its coordinate. Until a service has been able to claim its coordinate it should not alter any of the files in the runtime directory; it should assume that another process might exist.
When a node boots it will start up Nodee. When Nodee starts it will alert the world of its presence by registering with Zookeeper. An API will be implemented to abstract this process. This will be separate from Cloudname since we do not want to tie Nodee tightly to Cloudname.
Rather than being told its identity, Nodee will assume its identity
from the primary hostname of the host (whatever the hostname(1)
command on the machine will return -- minus domain name). This means
that we can provision new Nodee nodes without having to explicitly
configure each instance. We can just fire up a number of Amazon AMIs
in with one command.
In practice the registration process will be implemented in Zookeeper
as an ephemeral node. So a machine with hostname abc123 will turn
up in Zookeeper as the ephemeral node /nodes/abc123.
This node will serve a dual purpose. First it will make it possible
to see which Nodee instances are running by listing the /nodes
directory in Zookeeper. Second, it can be used by Nodee to convey
what services are running by making the JSON document from the POST to
start services available.
The only information a node needs to have is where the Zookeeper servers are -- and that should be available in the instance when it starts (through DNS or otherwise).
If Nodee crashes, any service still running on the machine will keep running while Nodee is restarted. Since any service started on Nodee is detached from its parent process a Nodee crash will not take down the services it has started.
If Nodee crashes the ephemeral node representing Nodee will disappear
thus providing us with a mechanism for monitoring this situation.
When Nodee starts up again, it will be able to figure out which
services are running by iterating over the runtime directories and
looking in run/start.json for each runtime directory.
...
An unresolved question so far is whether Nodee should automatically restart services that crash. This would necessitate Nodee being able to sense when a service crashes. It is tempting to do this by having Nodee be the parent process, but this complicates failures in Nodee itself so we are not going to do that. It is good enough to have some mechanism to periodically enquire what processes are running versus what processes should be running.
…or "things Bjørn can't be arsed to discuss anymore and which have just been decided".
We wish to implement a small number of programs that we can place a lot of trust in. This means that we have to step carefully and not try to do things that are hard to do well. We will not concern ourselves with automation of these programs initially -- initially we only care about implementing a minimal feature set in which we can place a lot of trust.
Automation and higher level functionality will come later and will be built on top of the first steps if, if only if, the basic components are implemented and found trustworthy.
There will be a low tolerance for bugs and breakages in the project.
We're not going to spend time building things we are going to throw away. We are going to make things that we can evolve and build on top of. This means that core tools get written in Java. We are not going to do things like package management over and over again.
We are not going to rush these things out. They are going to work and if they keep breaking we are going to step back and not implement a single feature more until the code is back to being rock solid. If we have recurring problems with quality we will stop the project until the problems are addressed.
Resolving package coordinates into URLs for download from the
distribution server, resolving package coordinates into filesystem
paths, installing and verifying packages are all responsibilities of
the copkg package library. Nodee will use the copkg library for
dealing with packages. copkg will initially only support install,
removal and listing of packages.
Nodee will be responsible for installing software artifacts, creating
runtime directories and starting/stopping/inspecting services through
the scripts contained in the script.d directory of the packages. It
will contain no "brains" capable of making independent decisions in
version 1. The initial version of Nodee will only support starting,
stopping and listing services that are running on the node.
Nodee does not interact with Cloudname -- however Nodee will use the Zookeeper instance in the cell to tell the world about its presence in a namespace that is separate from Cloudname.
Packages are intended to be usable both under Nodee in a datacenter and on a user's workstation. This is important in order to achieve higher iteration speed for developers. When installed on a workstation the packages will be installed in a user-configurable directory.
Nodee must be able to run in stand-alone mode on a workstation without Zookeeper available. It is acceptable to have a special command line option to tell Nodee that it shouldn't try to contact Zookeeper.