Live profiling with Grafana Pyroscope

Overview

This repository provides several services to demonstrate the features of the Grafana Pyroscope live profiling service.

Demo applications

• The cook service, written in Go, that:
  • interacts with a PostgreSQL database to retrieve restaurant data: menu, dishes, ingredients
  • serves this data over an HTTP API
• The waiter service, written in Python, that:
  • exposes an HTTP API to query data for a given restaurant
  • relays HTTP requests to the cook service

Storage

• A PostgreSQL 15 database to store and retrieve data
• A Redis key-value store to cache request responses

Load testing

• Locust scenarii to run load testing sessions on the cook and waiter services
• Toxiproxy network proxy to simulate network latency/jitter between the cook and postgres services

Observability

• Prometheus times-series database to store application metrics
• Pyroscope to store and explore live profiling data
• Grafana to explore live profiling data stored by Pyroscope

TL;DR - Service Addresses

Service	Address	Role
Cook API	http://localhost:8080	Go HTTP API backed by a PostgreSQL database
Cook Locust	http://localhost:8090	Load tests for the Cook API
Waiter API	http://localhost:8081	Python HTTP API backed by the Cook HTTP API
Waiter Locust	http://localhost:8089	Load tests for the Waiter API
Grafana	http://localhost:3000	Visualization & dashboards
PostgreSQL	localhost:5432	Relational database for restaurant data
Prometheus	http://localhost:9090	Time-series database
Redis	localhost:6379	Key-value store for API caches
Pyroscope	http://localhost:4040	Live profiling
Toxiproxy API	http://localhost:8474	Simulate network latency/jitter

Demo

Overview

Services

Demo stack

Preparation

1. Prepare the dataset for the demo
2. Build the cook application and populate the database

Run services with Docker

Pull the Docker images for the service containers:

$ docker compose pull

Build the application containers:

$ docker compose build cook waiter

Start the Docker services:

$ docker compose up -d

Run load test scenarii with Locust

Create and activate a Python virtualenv for Locust:

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -r loadtest/requirements.txt

Example - Run load tests on the waiter service's v1 API:

$ make locust-waiter-v1