Ideas: visualize HPI data in a dashboard

[karlicoss]

For quantified self stuff some heavy manipulation with pandas is usually necessary (e.g. I'm doing this in my dashboard), so I doubt a general purpose web dashboard would cover it (unless it support JS snippets, which would be very helpful!). But it would be cool to have a quick way of overviewing/visualizing/aggregating data in the browser even if it's not perfect.

Quick googling results in:

• metabase: https://github.com/metabase/metabase#metabase
• redash: https://github.com/getredash/redash#readme

Both of them rely on some database (e.g. sqlite). While it's a bit inconvenient, probably good enough as the first order approximation. Since I'm extensively using NamedTuples/dataclasses, it's possible to adapt the data automatically without any boilerplate. In addition, cachew already dumps sqlite databases, which can be used as input data.

It would be also cool to have a native app (less hassle + better performance), but I'm not sure how to even start googling for that.

Related: KrauseFx/FxLifeSheet#34 (I think we have similar goals!)

[karlicoss]

https://gridstudio.io/ looks promising

[karlicoss]

Started integrating with influxdb & Grafana, works pretty well for time series-like stuff

HPI/my/bluemaestro.py

Lines 194 to 220 in 6d9bc29

	def fill_influxdb() -> None:
	from itertools import islice
	from .core.common import asdict
	from influxdb import InfluxDBClient # type: ignore
	client = InfluxDBClient()
	db = 'db'
	mname = __name__.replace('.', '_')
	client.delete_series(database=db, measurement=mname)
	def dissoc(d, k):
	del d[k]
	return d # meh
	jsons = ({
	'measurement': mname,
	# todo maybe good idea to tags with database file/name? to inspect inconsistencies etc..
	# 'tags': {'activity': e.activity},
	'time': e.dt.isoformat(),
	'fields': dissoc(asdict(e), 'dt'),
	} for e in measurements())
	from more_itertools import chunked
	# "The optimal batch size is 5000 lines of line protocol."
	# some chunking is def necessary, otherwise it fails
	for chunk in chunked(jsons, n=5000):
	cl = list(chunk)
	logger.debug('writing next chunk %s', cl[-1])
	client.write_points(cl, database=db)
	# todo "Specify timestamp precision when writing to InfluxDB."?

https://twitter.com/karlicoss/status/1361100437332590593

HPI/my/rescuetime.py

Lines 85 to 109 in 6239879

	def fill_influxdb() -> None:
	from .core.common import asdict
	from influxdb import InfluxDBClient # type: ignore
	client = InfluxDBClient()
	db = 'db'
	measurement = __name__.replace('.', '_')
	client.delete_series(database=db, measurement=measurement)
	# client.drop_database(db)
	# todo create if not exists?
	# client.create_database(db)
	# todo handle errors.. not sure how? maybe add tag for 'error' and fill with emtpy data?
	vit = (e for e in entries() if isinstance(e, Entry))
	jsons = ({
	'measurement': measurement, # hmm, influx doesn't like dots?
	# hmm, so tags are autoindexed and might be faster?
	# not sure what's the big difference though
	# "fields are data and tags are metadata"
	'tags': {'activity': e.activity},
	'time': e.dt.isoformat(),
	'fields': {'duration_s': e.duration_s},
	# todo asdict(e),
	} for e in vit)
	# todo do we need to batch?
	client.write_points(jsons, database=db)

https://twitter.com/karlicoss/status/1360369025122000898