Web based metrics, monitoring, and observer.
We are talking about :wobserver
at ElixirConf 2017. Check out the presentation and samples and our other talk about Task Bunny.
Functionality:
- Drop-in monitoring though web interface.
- Metrics endpoint (
/metrics
) for system monitoring. (Default: Prometheus) - Monitoring automation through JSON API.
- Node management and discovery behind firewalls and load balancers.
- Easy to extend:
- Add custom metrics and pages for your project, just by adding them in the config.
- Just 3 lines of code to add pages/metrics for your library, when users have
:wobserver
installed. (See how.)
- wobserver
- Table of contents
- Installation
- Usage
- Configuration
- Library Integration
- Improvements
- Contributors
- license
Add Wobserver as a dependency to your mix.exs
file:
def deps do
[{:wobserver, "~> 0.1"}]
end
and add it to your list of applications:
def application do
[applications: [:wobserver]]
end
Then run mix deps.get
in your shell to fetch the dependencies.
Note: Check out plug mode to integrate with a Phoenix or other web application. (Prevents startup of separate web server.)
Run the following commands to build the project:
$ npm install
$ mix deps.get
$ mix build
Note: Use the package generated by mix build
if you want to include the local wobserver in your application.
(Unpack in your deps.)
Wobserver does not support being included directly from github. The required assets are not included in the repo in build form and can therefore not be used. It is possible to build locally and use the generated package. (See Build manually for more information.)
To view the web interface just enter http://<host>[:<port>]/
in the browser and it should show the :wobserver
interface.
The default port is 4001, but the port can be changed in the configuration.
A sample interface can be viewed here.
The API can be accessed by calling http://<host>[:<port>]/api/
.
The index will return 404
, but specific endpoints should return results.
The API provides a list of remote nodes by calling http://<host>[:<port>]/api/nodes
.
The API of remote nodes can be accessed by calling the API endpoint and prefixing the node name, host, or host:port.
For example considering the following node list:
[
{
"port": 4001,
"name": "node_prime",
"local?": true,
"host": "192.168.5.55"
},
{
"port": 80,
"name": "remote",
"local?": false,
"host": "80.23.1.165"
}
]
The following calls would all work for the first node:
(local
is a reserved name that always points to the local node.)
http://<host>[:<port>]/api/local/system
http://<host>[:<port>]/api/node_prime/system
http://<host>[:<port>]/api/192.168.5.55/system
http://<host>[:<port>]/api/192.168.5.55:4001/system
And these calls would work for the second node:
http://<host>[:<port>]/api/remote/system
http://<host>[:<port>]/api/80.23.1.165/system
http://<host>[:<port>]/api/80.23.1.165:80/system
The API provides a list of system information by calling http://<host>[:<port>]/api/system
.
The scheduler
is a list of load values (0-1) for each scheduler.
Example:
{
"statistics": {
"uptime": 459876,
"process_total": 122,
"process_running": 0,
"process_max": 262144,
"output": 1259201,
"input": 12945380
},
"scheduler": [
0.0037370416873916392,
0.0003088661849770247,
0.0003072993680801981,
0.00030274231847091137,
0.0004706952361156354,
0.00028556537348788645,
0.00025471141618606366,
0.0002522242536713918
],
"memory": {
"total": 30275576,
"process": 5242800,
"ets": 886544,
"code": 13635797,
"binary": 288744,
"atom": 594561
},
"cpu": {
"schedulers_online": 8,
"schedulers_available": 8,
"schedulers": 8,
"logical_processors_online": 8,
"logical_processors_available": "unknown",
"logical_processors": 8
},
"architecture": {
"wordsize_internal": 8,
"wordsize_external": 8,
"threads": true,
"thread_pool_size": 10,
"system_architecture": "x86_64-apple-darwin15.6.0",
"smp_support": true,
"otp_release": "19",
"kernel_poll": false,
"erts_version": "8.2",
"elixir_version": "1.4.0"
}
}
The API provides a list of allocators and their size by calling http://<host>[:<port>]/api/allocators
.
Example:
[
{
"type": "sl_alloc",
"carrier": 294912,
"block": 664
},
{
"type": "std_alloc",
"carrier": 1081344,
"block": 498184
},
{
"type": "ll_alloc",
"carrier": 35913728,
"block": 26080144
},
{
"type": "eheap_alloc",
"carrier": 9830400,
"block": 2634720
},
{
"type": "ets_alloc",
"carrier": 3178496,
"block": 890880
},
...
]
The API provides a list of applications and their descriptions by calling http://<host>[:<port>]/api/application
.
The information for a specific application, including the process hierarchy can be found by calling http://<host>[:<port>]/api/application/<application-name>
.
Example:
http://localhost:4001/api/application
[
{
"version": "0.1.0",
"name": "wobserver",
"description": "Web based metrics, monitoring, and observer."
},
{
"version": "1.3.0",
"name": "plug",
"description": "A specification and conveniences for composable modules between web applications"
},
{
"version": "1.1.0",
"name": "cowboy",
"description": "Small, fast, modular HTTP server."
},
{
"version": "1.2.1",
"name": "ranch",
"description": "Socket acceptor pool for TCP protocols."
},
{
"version": "0.6.1",
"name": "credo",
"description": "A static code analysis tool for the Elixir language with a focus on code consistency and teaching."
},
{
"version": "0.2.0",
"name": "bunt",
"description": "256 color ANSI coloring in the terminal"
},
{
"version": "1.6.5",
"name": "hackney",
"description": "simple HTTP client"
},
{
"version": "1.4.0",
"name": "logger",
"description": "logger"
},
...
]
http://localhost:4001/api/application/elixir
{
"pid": "#PID<0.59.0>",
"name": "<0.59.0>",
"meta": {
"status": "waiting",
"init": "proc_lib.init_p/5",
"current": "application_master.main_loop/2",
"class": "application"
},
"children": [
{
"pid": "#PID<0.60.0>",
"name": "<0.60.0>",
"meta": {
"status": "waiting",
"init": "application_master.start_it/4",
"current": "application_master.loop_it/4",
"class": "unknown"
},
"children": [
{
"pid": "#PID<0.61.0>",
"name": "elixir_sup",
"meta": {
"status": "waiting",
"init": "proc_lib.init_p/5",
"current": "gen_server.loop/6",
"class": "supervisor"
},
"children": [
{
"pid": "#PID<0.62.0>",
"name": "elixir_config",
"meta": {
"status": "waiting",
"init": "proc_lib.init_p/5",
"current": "gen_server.loop/6",
"class": "gen_server"
},
"children": []
},
{
"pid": "#PID<0.63.0>",
"name": "elixir_code_server",
"meta": {
"status": "waiting",
"init": "proc_lib.init_p/5",
"current": "gen_server.loop/6",
"class": "gen_server"
},
"children": []
}
]
}
]
}
]
}
The API provides a list of processes and their basic information by calling http://<host>[:<port>]/api/process
.
The information for a specific process, including a links, memory usage, and state can be found by calling http://<host>[:<port>]/api/application/<process-name>
.
The process name can be given as pid, name, or short pid.
So all the following are valid:
http://localhost:4001/api/process/<0.247.0>
http://localhost:4001/api/process/#PID<0.247.0> # Rememeber to url encode # -> %23
http://localhost:4001/api/process/Wobserver.Supervisor
Example:
http://localhost:4001/api/process
{
"processes": [
{
"reductions": 162714,
"pid": "#PID<0.247.0>",
"nr1": "0",
"message_queue_length": 0,
"memory": 11888,
"init": "timer_server",
"current": "gen_server.loop/6"
},
{
"reductions": 95,
"pid": "#PID<0.243.0>",
"nr1": "0",
"message_queue_length": 0,
"memory": 2792,
"init": "erlang.apply/2",
"current": "io.execute_request/2"
},
{
"reductions": 954,
"pid": "#PID<0.242.0>",
"nr1": "0",
"message_queue_length": 0,
"memory": 16808,
"init": "Elixir.IEx.Evaluator.init/4",
"current": "Elixir.IEx.Evaluator.loop/3"
},
...
]
}
http://localhost:4001/api/process/<0.247.0>
{
"trap_exit": true,
"state": "[]",
"relations": {
"links": [
"#PID<0.53.0>"
],
"group_leader": "#PID<0.33.0>",
"ancestors": [
"kernel_safe_sup",
"kernel_sup",
"#PID<0.34.0>"
]
},
"registered_name": "timer_server",
"priority": "normal",
"pid": "#PID<0.247.0>",
"meta": {
"status": "waiting",
"init": "proc_lib.init_p/5",
"current": "gen_server.loop/6",
"class": "gen_server"
},
"message_queue_len": 0,
"memory": {
"total": 0,
"stack_size": 9,
"stack_and_heap": 1974,
"heap_size": 1598,
"gc_min_heap_size": 233,
"gc_full_sweep_after": 65535
},
"error_handler": "error_handler"
}
The API provides a list of ports and their owners by calling http://<host>[:<port>]/api/ports
.
Example:
http://localhost:4001/api/ports
[
{
"output": 0,
"os_pid": "undefined",
"name": "forker",
"links": [],
"input": 0,
"id": 0,
"connected": "#PID<0.0.0>"
},
{
"output": 3,
"os_pid": "undefined",
"name": "efile",
"links": [
"#PID<0.4.0>"
],
"input": 46,
"id": 8,
"connected": "#PID<0.4.0>"
},
{
"output": 18810,
"os_pid": "undefined",
"name": "efile",
"links": [
"#PID<0.44.0>"
],
"input": 23874,
"id": 4680,
"connected": "#PID<0.44.0>"
},
...
]
The API provides a list of tables and their details by calling http://<host>[:<port>]/api/table
.
The information for a specific details, including a the actual data can be found by calling http://<host>[:<port>]/api/table/<table-name>
.
Example:
http://localhost:4001/api/table
Example:
[
{
"type": "set",
"size": 0,
"protection": "protected",
"owner": "#PID<0.247.0>",
"name": "timer_interval_tab",
"meta": {
"write_concurrency": false,
"read_concurrency": false,
"compressed": false
},
"memory": 304,
"id": "timer_interval_tab"
},
{
"type": "ordered_set",
"size": 7,
"protection": "protected",
"owner": "#PID<0.247.0>",
"name": "timer_tab",
"meta": {
"write_concurrency": false,
"read_concurrency": false,
"compressed": false
},
"memory": 304,
"id": "timer_tab"
},
{
"type": "set",
"size": 7,
"protection": "public",
"owner": "#PID<0.228.0>",
"name": "workstore",
"meta": {
"write_concurrency": false,
"read_concurrency": false,
"compressed": false
},
"memory": 5138,
"id": 417840
},
...
]
http://localhost:4001/api/table/timer_interval_tab
{
"type": "set",
"size": 0,
"protection": "protected",
"owner": "#PID<0.247.0>",
"name": "timer_interval_tab",
"meta": {
"write_concurrency": false,
"read_concurrency": false,
"compressed": false
},
"memory": 304,
"id": "timer_interval_tab",
"data": []
}
Metrics are available by calling http://<host>[:<port>]/metrics
.
The metrics are by default formatted for Prometheus, but can be configured to work with any system. An explanation of how to configure the metrics format and how to add metrics to the output will be added later.
http://localhost:4001/metrics
# HELP erlang_vm_used_memory_bytes Memory usage of the Erlang VM.
# TYPE erlang_vm_used_memory_bytes gauge
erlang_vm_used_memory_bytes{node="10.74.181.35",type="atom"} 553593
erlang_vm_used_memory_bytes{node="10.74.181.35",type="binary"} 359552
erlang_vm_used_memory_bytes{node="10.74.181.35",type="code"} 13533686
erlang_vm_used_memory_bytes{node="10.74.181.35",type="ets"} 1899472
erlang_vm_used_memory_bytes{node="10.74.181.35",type="process"} 6048552
# HELP erlang_vm_used_io_bytes IO counter for the Erlang VM.
# TYPE erlang_vm_used_io_bytes counter
erlang_vm_used_io_bytes{node="10.74.181.35",type="input"} 11301316
erlang_vm_used_io_bytes{node="10.74.181.35",type="output"} 618157
The port can be set in the config by setting :port
for :wobserver
to a valid number.
config :wobserver,
port: 80
Wobserver runs by default in :standalone
mode.
This means that :wobserver
will start its own :cowboy
listeners on a separate port.
Standalone mode is ideal for drop-in web viewing, but might not be ideal if another part of the application is already running an web server.
It is possible to enable :plug
mode to prevent :wobserver
from starting :cowboy
to handle web requests.
Standalone mode is the default operating mode.
A :cowboy
(ranch) listener will be started with 10 accepters and a websocket.
Set mode
to :standalone
in the :wobserver
configuration to force standalone mode.
Example:
config :wobserver,
mode: :standalone
Plug mode prevents :wobserver
from starting :cowboy
(ranch).
Set mode
to :plug
in the :wobserver
configuration to use plug mode.
Set remote_url_prefix
to the url prefix you put :wobserver
behind to make sure dns node discovery still functions.
Plug mode prevents :wobserver
from starting :cowboy
(ranch). Set mode
to :plug
in the :wobserver
configuration to use plug mode. Set remote_url_prefix
to the url prefix you put :wobserver
behind to make
sure dns node discovery still functions.
Add the following line of code to the application's router to forward requests to :wobserver
:
forward "/wobserver", to: Wobserver.Web.Router
Add the following option to the :cowboy
child_spec to enable use of the :wobserver
websocket:
dispatch: [
{:_, [
{"/ws", Wobserver.Web.Client, []},
{:_, Cowboy.Handler, {<your own router>, []}}
]}
],
Add the following line of code to the Phoenix router to forward requests to :wobserver
:
forward "/wobserver", Wobserver.Web.Router
Add the following option to your Phoenix applications Endpoint to enable use of the :wobserver
websocket (the
path should match what is in the 'forward' in your router):
socket "/wobserver", Wobserver.Web.PhoenixSocket
config.exs
config :wobserver,
mode: :plug,
remote_url_prefix: "/wobserver"
my_router.ex
defmodule MyApp.MyRouter do
use Plug.Router
plug :match
plug :dispatch
forward "/wobserver", to: Wobserver.Web.Router
end
application.ex
defmodule MyApp.Application do
use Application
alias Plug.Adapters.Cowboy
def start(_type, _args) do
import Supervisor.Spec, warn: false
options = [
dispatch: [
{:_, [
{"/wobserver/ws", Wobserver.Web.Client, []},
{:_, Cowboy.Handler, {MyApp.MyRouter, []}}
]}
],
]
children = [
Cowboy.child_spec(:http, MyApp.MyRouter, [], options)
]
opts = [strategy: :one_for_one, name: MyApp.Supervisor]
Supervisor.start_link(children, opts)
end
end
The method used can be set in the config file by setting:
config :wobserver,
discovery: :none
The following methods can be used: (default: :none
)
:none
, just returns the local node.:dns
, use DNS to search for other nodes. The optiondiscovery_search
needs to be set to filter entries.:custom
, a function as String.
No discovery: (default)
config :wobserver,
port: 80,
discovery: :none
Using dns as discovery service:
config :wobserver,
port: 80,
discovery: :dns,
discovery_search: "google.nl"
Using a custom function:
config :wobserver,
port: 80,
discovery: :custom,
discovery_search: &MyApp.CustomDiscovery.discover/0
Using an anonymous function:
config :wobserver,
port: 80,
discovery: :custom,
discovery_search: fn -> [] end
Both the custom and anonymous functions can be given as a String, which will get evaluated.
Metrics and metric generators can be added by setting them in the configuration.
To add custom metrics set the :metrics
option.
The :metrics
option must be a keyword list with the following keys:
additional
, for a keyword list with additional metrics.generators
, for a list of metric generators.
The following settings are accepted for additional
:
keyword
list, the key is the name of the metric and the value is the metric data.
The following inputs are accepted for metric generators:
list
of callable functions. Every function should return a keyword list with as key the name of the metric and as value the metric data.
For more information about how to format metric data see: Wobserver.Util.Metrics.Formatter.format_all/1
.
For example this configuration:
config :wobserver,
metrics: [
additional: [
example: {fn -> [red: 5] end, :gauge, "Description"},
],
generators: [
"&MyApp.generator/0",
fn -> [bottles: {fn -> [wall: 8, floor: 10] end, :gauge, "Description"}] end
fn -> [server: {"MyApp.Server.metrics/0", :gauge, "Description"}] end
]
]
Metrics and metric generators can also be added dynamically at runtime.
To register a metric you need to pass a keyword list to Wobserver.register
with the same data as you would set in the configuration file.
For example:
Wobserver.register :metric, [example: {fn -> [red: 5] end, :gauge, "Description"}]
To register a metric generator you need to pass a list of functions to Wobserver.register
.
For example:
Wobserver.register :metric, [&MyLibrary.Metrics.generate/0]
A custom formatter can be created for output of metrics by implementing the Wobserver.Util.Metrics.Formatter
behavior.
This custom formatter can be enabled in the configuration file by setting metric_format
.
For example this configuration:
config :wobserver,
metric_format: JsonFormatter
And this simple JSON formatter:
defmodule SimpleJsonFormatter do
@behaviour Wobserver.Util.Metrics.Formatter
def format_data(name, data, type, help) do
formatted_data =
data
|> Enum.map(fn {value, labels} ->
%{value: value, labels: Enum.into(labels, %{})}
end)
%{
name: name,
type: type,
description: help,
data: formatted_data
}
|> Poison.encode!
end
def combine_metrics(metrics) do
"[" <> Enum.join(metrics,",") <> "]"
end
def merge_metrics(metrics) do
"[" <> Enum.join(metrics,",") <> "]"
end
end
Produce the following output:
[
[
{
"type": "gauge",
"name": "erlang_vm_used_memory_bytes",
"description": "Memory usage of the Erlang VM.",
"data": [
{
"value": 654241,
"labels": {
"type": "atom",
"node": "192.168.1.88"
}
},
{
"value": 503464,
"labels": {
"type": "binary",
"node": "192.168.1.88"
}
},
{
"value": 14459399,
"labels": {
"type": "code",
"node": "192.168.1.88"
}
},
{
"value": 2073072,
"labels": {
"type": "ets",
"node": "192.168.1.88"
}
},
{
"value": 6008488,
"labels": {
"type": "process",
"node": "192.168.1.88"
}
}
]
},
{
"type": "counter",
"name": "erlang_vm_used_io_bytes",
"description": "IO counter for the Erlang VM.",
"data": [
{
"value": 29523254,
"labels": {
"type": "input",
"node": "192.168.1.88"
}
},
{
"value": 9960593,
"labels": {
"type": "output",
"node": "192.168.1.88"
}
}
]
}
]
]
Pages are custom views in the web interface and endpoints in the JSON API for an application or library.
There are two ways to add a custom page:
- config, set a list of custom pages in the mix config.
- registration, call
Wobserver.register/2
and dynamically add pages.
Adding more pages to :wobserver
can be done by setting the :pages
option.
The :pages
option must be a list of page data.
The page data can be formatted as:
{title, command, callback}
{title, command, callback, options}
- a
map
with the following fields:title
command
callback
options
(optional)
For more information and types see: Wobserver.Page.register/1
.
Example:
config :wobserver,
pages: [
{"Example", :example, fn -> %{x: 9} end}
]
Dynamically register a page with :wobserver
by calling Wobserver.register/2
.
The following inputs are accepted:
{title, command, callback}
{title, command, callback, options}
- a
map
with the following fields:title
command
callback
options
(optional)
The fields are used as followed:
title
, the name of the page. Is used for the web interface menu.command
, single atom to associate the page with.callback
, function to be evaluated, when the a api is called or page is viewd. The result is converted to JSON and displayed.options
, options for the page.
The following options can be set:
api_only
(boolean
), if set to true the page won't show up in the web interface, but will only be available as API.refresh
(float
, 0-1), sets the refresh time factor. Used in the web interface to refresh the data on the page. Set to0
for no refresh.
Example:
Wobserver.register(:page, {"My App", :my_app, fn -> %{data: 123} end})
Integrating a library with :wobserver
is done by calling Wobserver.register/2
, when the library loads, and dynamically adding pages and metrics.
To safely integrate with :wobserver
use the following code:
if Code.ensure_loaded(Wobserver) == {:module, Wobserver} do
Wobserver.register :page, {"My Library", :my_library, fn -> %{data: 123} end}
Wobserver.register :metric, [&MyLibrary.Metrics.generate/0]
end
The above code will make sure that the library only calls register, when :wobserver
is loaded.
This will prevent the library from trying to register, when :wobserver
is not installed.
For an implementation see the :task_bunny
library: TaskBunny, lib/task_bunny.ex
.
The above code will generate warnings while compiling the library.
warning: function Wobserver.register/2 is undefined (module Wobserver is not available)
There are two options to remove those warnings.
The mix.exs
file can be edited to excluded Wobserver
from reference checks.
To do this add the following line to project/0
in your mix file:
xref: [exclude: [Wobserver]]
The code can be rewritten to use Kernel.apply/3
.
The following code will be less readable and slightly slower, but will not generate warnings.
if Code.ensure_loaded(Wobserver) == {:module, Wobserver} do
apply Wobserver, :register, [:page, {"My Library", :my_library, fn -> %{data: 123} end}]
apply Wobserver, :register, [:metric, [&MyLibrary.Metrics.generate/0]]
end
- Cleanup namespaces.
- Cleanup readme, condense sample output.
- Overhaul web interface (make fancier/pleasant)
- OvermindDL1 - Phoenix Socket support and lots of issue reports.
Wobserver source code is released under the MIT License. Check LICENSE file for more information.