Skip to content

Commit

Permalink
PMM-1011: Merge pull request #2 from percona/prometheus-master
Browse files Browse the repository at this point in the history
PMM-1011: Merge upstream changes
  • Loading branch information
arvenil authored Jun 21, 2017
2 parents 05a347d + 34fc5bb commit 7b07260
Show file tree
Hide file tree
Showing 272 changed files with 32,987 additions and 6,484 deletions.
2 changes: 1 addition & 1 deletion .promu.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
repository:
path: github.com/prometheus/mysqld_exporter
path: github.com/percona/mysqld_exporter
build:
flags: -a -tags netgo
ldflags: |
Expand Down
13 changes: 6 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
language: go
sudo: false

language: go
go:
- 1.7.1
- 1.8.x

script:
- make

notifications:
email: false

script:
- go get github.com/kardianos/govendor
- govendor sync
- go test -v
12 changes: 0 additions & 12 deletions AUTHORS.md

This file was deleted.

12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
## v0.10.0 / 2017-04-25

BREAKING CHANGES:
* `mysql_slave_...` metrics now include an additional `connection_name` label to support mariadb multi-source replication. (#178)

* [FEATURE] Add read/write query response time #166
* [FEATURE] Add Galera gcache size metric #169
* [FEATURE] Add MariaDB multi source replication support #178
* [FEATURE] Implement heartbeat metrics #183
* [FEATURE] Add basic file_summary_by_instance metrics #189
* [BUGFIX] Workaround MySQL bug 79533 #173

## 0.9.0 / 2016-09-26

BREAKING CHANGES:
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Prometheus uses GitHub to manage reviews of pull requests.

* If you have a trivial fix or improvement, go ahead and create a pull
request, addressing (with `@...`) one or more of the maintainers
(see [AUTHORS.md](AUTHORS.md)) in the description of the pull request.
* If you have a trivial fix or improvement, go ahead and create a pull request,
addressing (with `@...`) the maintainer of this repository (see
[MAINTAINERS.md](MAINTAINERS.md)) in the description of the pull request.

* If you plan to do something more involved, first discuss your ideas
on our [mailing list](https://groups.google.com/forum/?fromgroups#!forum/prometheus-developers).
Expand Down
1 change: 1 addition & 0 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Ben Kochie <[email protected]>
7 changes: 4 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.

GO := GO15VENDOREXPERIMENT=1 go
PROMU := $(GOPATH)/bin/promu
pkgs = $(shell $(GO) list ./... | grep -v /vendor/)
GO := GO15VENDOREXPERIMENT=1 go
FIRST_GOPATH := $(firstword $(subst :, ,$(shell $(GO) env GOPATH)))
PROMU := $(FIRST_GOPATH)/bin/promu
pkgs = $(shell $(GO) list ./... | grep -v /vendor/)

PREFIX ?= $(shell pwd)
BIN_DIR ?= $(shell pwd)
Expand Down
26 changes: 19 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Percona MySQL Server Exporter
# Percona MySQL Server Exporter [![Build Status](https://travis-ci.org/percona/mysqld_exporter.svg)][travis]

[![Build Status](https://travis-ci.org/percona/mysqld_exporter.svg?branch=master)](https://travis-ci.org/percona/mysqld_exporter)
[![Go Report Card](https://goreportcard.com/badge/github.com/percona/mysqld_exporter)](https://goreportcard.com/report/github.com/percona/mysqld_exporter)

Prometheus exporter for MySQL server metrics.
Expand All @@ -11,9 +10,10 @@ NOTE: Not all collection methods are supported on MySQL < 5.6

### Required Grants

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'XXXXXXXX';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost'
WITH MAX_USER_CONNECTIONS 3;
```sql
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'XXXXXXXX' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
```

NOTE: It is recommended to set a max connection limit for the user to avoid overloading the server with monitoring scrapes under heavy load.

Expand Down Expand Up @@ -58,10 +58,14 @@ collect.perf_schema.eventsstatements.limit | 5.6 | Limit t
collect.perf_schema.eventsstatements.timelimit | 5.6 | Limit how old the 'last_seen' events statements can be, in seconds. (default: 86400)
collect.perf_schema.eventswaits | 5.5 | Collect metrics from performance_schema.events_waits_summary_global_by_event_name.
collect.perf_schema.file_events | 5.6 | Collect metrics from performance_schema.file_summary_by_event_name.
collect.perf_schema.file_instances | 5.5 | Collect metrics from performance_schema.file_summary_by_instance.
collect.perf_schema.indexiowaits | 5.6 | Collect metrics from performance_schema.table_io_waits_summary_by_index_usage.
collect.perf_schema.tableiowaits | 5.6 | Collect metrics from performance_schema.table_io_waits_summary_by_table.
collect.perf_schema.tablelocks | 5.6 | Collect metrics from performance_schema.table_lock_waits_summary_by_table.
collect.slave_status | 5.1 | Collect from SHOW SLAVE STATUS (Enabled by default)
collect.heartbeat | 5.1 | Collect from [heartbeat](#heartbeat).
collect.heartbeat.database | 5.1 | Database from where to collect heartbeat data. (default: heartbeat)
collect.heartbeat.table | 5.1 | Table from where to collect heartbeat data. (default: heartbeat)


### General Flags
Expand Down Expand Up @@ -93,13 +97,21 @@ docker run -d -p 9104:9104 --link=my_mysql_container:bdd \
-e DATA_SOURCE_NAME="user:password@(bdd:3306)/database" prom/mysqld-exporter
```

## heartbeat

With `collect.heartbeat` enabled, mysqld_exporter will scrape replication delay
measured by heartbeat mechanisms. [Pt-heartbeat][pth] is the
reference heartbeat implementation supported.

[pth]:https://www.percona.com/doc/percona-toolkit/2.2/pt-heartbeat.html

## Example Rules

There are some sample rules available in [example.rules](example.rules)

[circleci]: https://circleci.com/gh/prometheus/mysqld_exporter
[circleci]: https://circleci.com/gh/percona/mysqld_exporter
[hub]: https://hub.docker.com/r/prom/mysqld-exporter/
[travis]: https://travis-ci.org/prometheus/mysqld_exporter
[travis]: https://travis-ci.org/percona/mysqld_exporter
[quay]: https://quay.io/repository/prometheus/mysqld-exporter

## Visualize
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.9.0
0.10.0
5 changes: 3 additions & 2 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ machine:
environment:
DOCKER_IMAGE_NAME: prom/mysqld-exporter
QUAY_IMAGE_NAME: quay.io/prometheus/mysqld-exporter
DOCKER_TEST_IMAGE_NAME: quay.io/prometheus/golang-builder:1.6-base
REPO_PATH: github.com/prometheus/mysqld_exporter
DOCKER_TEST_IMAGE_NAME: quay.io/prometheus/golang-builder:1.8-base
REPO_PATH: github.com/percona/mysqld_exporter
pre:
- sudo curl -L -o /usr/bin/docker 'https://s3-external-1.amazonaws.com/circle-downloads/docker-1.9.1-circleci'
- sudo chmod 0755 /usr/bin/docker
Expand Down Expand Up @@ -48,6 +48,7 @@ deployment:
owner: prometheus
commands:
- promu crossbuild tarballs
- promu checksum .tarballs
- promu release .tarballs
- mkdir $CIRCLE_ARTIFACTS/releases/ && cp -a .tarballs/* $CIRCLE_ARTIFACTS/releases/
- docker login -e $DOCKER_EMAIL -u $DOCKER_LOGIN -p $DOCKER_PASSWORD
Expand Down
89 changes: 89 additions & 0 deletions collector/heartbeat.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
// Scrape heartbeat data.

package collector

import (
"database/sql"
"fmt"
"strconv"

"github.com/prometheus/client_golang/prometheus"
)

const (
// heartbeat is the Metric subsystem we use.
heartbeat = "heartbeat"
// heartbeatQuery is the query used to fetch the stored and current
// timestamps. %s will be replaced by the database and table name.
// The second column allows gets the server timestamp at the exact same
// time the query is run.
heartbeatQuery = "SELECT UNIX_TIMESTAMP(ts), UNIX_TIMESTAMP(NOW(6)), server_id from `%s`.`%s`"
)

// Metric descriptors.
var (
HeartbeatStoredDesc = prometheus.NewDesc(
prometheus.BuildFQName(namespace, heartbeat, "stored_timestamp_seconds"),
"Timestamp stored in the heartbeat table.",
[]string{"server_id"}, nil,
)
HeartbeatNowDesc = prometheus.NewDesc(
prometheus.BuildFQName(namespace, heartbeat, "now_timestamp_seconds"),
"Timestamp of the current server.",
[]string{"server_id"}, nil,
)
)

// ScrapeHeartbeat scrapes from the heartbeat table.
// This is mainly targeting pt-heartbeat, but will work with any heartbeat
// implementation that writes to a table with two columns:
// CREATE TABLE heartbeat (
// ts varchar(26) NOT NULL,
// server_id int unsigned NOT NULL PRIMARY KEY,
// );
func ScrapeHeartbeat(db *sql.DB, ch chan<- prometheus.Metric, collectDatabase, collectTable *string) error {
query := fmt.Sprintf(heartbeatQuery, *collectDatabase, *collectTable)
heartbeatRows, err := db.Query(query)
if err != nil {
return err
}
defer heartbeatRows.Close()

var (
now, ts sql.RawBytes
serverId int
)

for heartbeatRows.Next() {
if err := heartbeatRows.Scan(&ts, &now, &serverId); err != nil {
return err
}

tsFloatVal, err := strconv.ParseFloat(string(ts), 64)
if err != nil {
return err
}

nowFloatVal, err := strconv.ParseFloat(string(now), 64)
if err != nil {
return err
}

serverId := strconv.Itoa(serverId)

ch <- prometheus.MustNewConstMetric(
HeartbeatNowDesc,
prometheus.GaugeValue,
nowFloatVal,
serverId,
)
ch <- prometheus.MustNewConstMetric(
HeartbeatStoredDesc,
prometheus.GaugeValue,
tsFloatVal,
serverId,
)
}

return nil
}
49 changes: 49 additions & 0 deletions collector/heartbeat_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package collector

import (
"testing"

"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
"github.com/smartystreets/goconvey/convey"
"gopkg.in/DATA-DOG/go-sqlmock.v1"
)

func TestScrapeHeartbeat(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("error opening a stub database connection: %s", err)
}
defer db.Close()

columns := []string{"UNIX_TIMESTAMP(ts)", "UNIX_TIMESTAMP(NOW(6))", "server_id"}
rows := sqlmock.NewRows(columns).
AddRow("1487597613.001320", "1487598113.448042", 1)
mock.ExpectQuery(sanitizeQuery("SELECT UNIX_TIMESTAMP(ts), UNIX_TIMESTAMP(NOW(6)), server_id from `heartbeat`.`heartbeat`")).WillReturnRows(rows)

ch := make(chan prometheus.Metric)
go func() {
database := "heartbeat"
table := "heartbeat"
if err = ScrapeHeartbeat(db, ch, &database, &table); err != nil {
t.Errorf("error calling function on test: %s", err)
}
close(ch)
}()

counterExpected := []MetricResult{
{labels: labelMap{"server_id": "1"}, value: 1487598113.448042, metricType: dto.MetricType_GAUGE},
{labels: labelMap{"server_id": "1"}, value: 1487597613.00132, metricType: dto.MetricType_GAUGE},
}
convey.Convey("Metrics comparison", t, func() {
for _, expect := range counterExpected {
got := readMetric(<-ch)
convey.So(got, convey.ShouldResemble, expect)
}
})

// Ensure all SQL queries were executed
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expections: %s", err)
}
}
6 changes: 3 additions & 3 deletions collector/info_schema_clientstats_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func TestScrapeClientStat(t *testing.T) {

columns := []string{"CLIENT", "TOTAL_CONNECTIONS", "CONCURRENT_CONNECTIONS", "CONNECTED_TIME", "BUSY_TIME", "CPU_TIME", "BYTES_RECEIVED", "BYTES_SENT", "BINLOG_BYTES_WRITTEN", "ROWS_READ", "ROWS_SENT", "ROWS_DELETED", "ROWS_INSERTED", "ROWS_UPDATED", "SELECT_COMMANDS", "UPDATE_COMMANDS", "OTHER_COMMANDS", "COMMIT_TRANSACTIONS", "ROLLBACK_TRANSACTIONS", "DENIED_CONNECTIONS", "LOST_CONNECTIONS", "ACCESS_DENIED", "EMPTY_QUERIES"}
rows := sqlmock.NewRows(columns).
AddRow("localhost", 1002, 0, 127027, 286, 245, 2565104853, 21090856, 2380108042, 767691, 1764, 8778, 1210741, 0, 1764, 1214416, 293, 2430888, 0, 0, 0, 0, 0)
AddRow("localhost", 1002, 0, 127027, 286, 245, float64(2565104853), 21090856, float64(2380108042), 767691, 1764, 8778, 1210741, 0, 1764, 1214416, 293, 2430888, 0, 0, 0, 0, 0)
mock.ExpectQuery(sanitizeQuery(clientStatQuery)).WillReturnRows(rows)

ch := make(chan prometheus.Metric)
Expand All @@ -38,9 +38,9 @@ func TestScrapeClientStat(t *testing.T) {
{labels: labelMap{"client": "localhost"}, value: 127027, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 286, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 245, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 2565104853, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: float64(2565104853), metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 21090856, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 2380108042, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: float64(2380108042), metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 767691, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 1764, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"client": "localhost"}, value: 8778, metricType: dto.MetricType_COUNTER},
Expand Down
6 changes: 3 additions & 3 deletions collector/info_schema_userstats_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func TestScrapeUserStat(t *testing.T) {

columns := []string{"USER", "TOTAL_CONNECTIONS", "CONCURRENT_CONNECTIONS", "CONNECTED_TIME", "BUSY_TIME", "CPU_TIME", "BYTES_RECEIVED", "BYTES_SENT", "BINLOG_BYTES_WRITTEN", "ROWS_READ", "ROWS_SENT", "ROWS_DELETED", "ROWS_INSERTED", "ROWS_UPDATED", "SELECT_COMMANDS", "UPDATE_COMMANDS", "OTHER_COMMANDS", "COMMIT_TRANSACTIONS", "ROLLBACK_TRANSACTIONS", "DENIED_CONNECTIONS", "LOST_CONNECTIONS", "ACCESS_DENIED", "EMPTY_QUERIES"}
rows := sqlmock.NewRows(columns).
AddRow("user_test", 1002, 0, 127027, 286, 245, 2565104853, 21090856, 2380108042, 767691, 1764, 8778, 1210741, 0, 1764, 1214416, 293, 2430888, 0, 0, 0, 0, 0)
AddRow("user_test", 1002, 0, 127027, 286, 245, float64(2565104853), 21090856, float64(2380108042), 767691, 1764, 8778, 1210741, 0, 1764, 1214416, 293, 2430888, 0, 0, 0, 0, 0)
mock.ExpectQuery(sanitizeQuery(userStatQuery)).WillReturnRows(rows)

ch := make(chan prometheus.Metric)
Expand All @@ -38,9 +38,9 @@ func TestScrapeUserStat(t *testing.T) {
{labels: labelMap{"user": "user_test"}, value: 127027, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 286, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 245, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 2565104853, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: float64(2565104853), metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 21090856, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 2380108042, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: float64(2380108042), metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 767691, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 1764, metricType: dto.MetricType_COUNTER},
{labels: labelMap{"user": "user_test"}, value: 8778, metricType: dto.MetricType_COUNTER},
Expand Down
11 changes: 8 additions & 3 deletions collector/perf_schema_events_statements.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,14 @@ const perfEventsStatementsQuery = `
SUM_SORT_MERGE_PASSES,
SUM_SORT_ROWS,
SUM_NO_INDEX_USED
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME NOT IN ('mysql', 'performance_schema', 'information_schema')
AND last_seen > DATE_SUB(NOW(), INTERVAL %d SECOND)
FROM (
SELECT *
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME NOT IN ('mysql', 'performance_schema', 'information_schema')
AND LAST_SEEN > DATE_SUB(NOW(), INTERVAL %d SECOND)
ORDER BY LAST_SEEN DESC
)Q
GROUP BY SCHEMA_NAME, DIGEST
ORDER BY SUM_TIMER_WAIT DESC
LIMIT %d
`
Expand Down
Loading

0 comments on commit 7b07260

Please sign in to comment.