Astor, an alternative to StatsD+Graphite

This is: astor-collector

Metric collector and REST API. Based on EventMachine, Goliath and LevelDB.

You'll also need: astor-dashboard

Real-time metric visualization. Based on SocketStream (Node) and Backbone. Find it here.

Collector features

• API compatible with StatsD, push data with any StatsD client
• Second-by-second data resolution for active metrics
• Minute-by-minute data resolution for archived metrics
• Unlimited retention (well, as much as you can store)
• Metric trend monitoring and alerting
• Event streaming through ZeroMQ pub/sub
• REST API

Pre-requisites

• Ruby 1.9+
• ZeroMQ dev (>= 2.1, < 3.0)

Getting started

If you already have a working Ruby environment, should be as easy as:

bundle install
./astor

Deployment

Deploying Astor to EC2 is pretty straight forward. On AWS Console:

• Create a new Amazon EC2 instance using the Ubuntu 12.04 AMI.

• We'll use capify-ec2 so you need to tag your instance properly

• Customize your Security Groups and open the right ports

• Once the instance is up and running, ssh into it and run:

  sudo mkdir /var/astor
  sudo chown -R ubuntu:ubuntu /var/astor
  sudo apt-get update
  sudo apt-get install git build-essential libxslt-dev libxml2-dev ruby1.9.3 libzmq-dev
  sudo gem install bundler
  

Then, on your local clone:

• Copy config/astor.yml.sample to config/astor.yml and customize it if needed

• Copy config/deploy.rb.sample to config/deploy.rb and customize it with your repository

• Copy config/ec2.yml.sample to config/ec2.yml and complete it with your EC2 keys

  cap deploy:setup
  cap deploy
  

About Heroku: Is not possible to route UDP traffic at the moment but this may change in the future (https://groups.google.com/forum/?fromgroups=#!topic/heroku/z6haBrCQ9ZA)

REST API

id format is: type-metric-key, for example:

• product.clicks counter => c-product-clicks)
• api.request timer => ms-api-request
• current.users gauge => g-current-users

Availabe methods:

• GET http://127.0.0.1:8889/v1/metrics

  • List of metrics

• GET http://127.0.0.1:8889/v1/metrics/:id

  • Get datapoints (minute resolution) for the given metric id
  • Optional parameteres:
    • offset: -1day, -7days, -1month

• POST http://127.0.0.1:8889/v1/metrics/:id

  • Post a datapoint for the given metric id
  • Mandatory parameters:
    • value: metric value

• DELETE http://127.0.0.1:8889/v1/metrics/:id

  • Delete all datapoints for the given metric id

Event Stream

Check example

TODO

• Alerts:
  • Configurable thresholds (per metric maybe?)
  • Storage
  • Expose as REST resource (POST deploy notifications, etc)
• Configurable data retention