Skip to content

🦄🌟 Graph Database layer on top of Google Bigtable

License

Notifications You must be signed in to change notification settings

yildizdb/yildiz

Repository files navigation

Yildiz Graph Database

yildiz

Thin graph database layer on top of Google Bigtable.

Build Status Version MIT License node Swagger Mentioned in Awesome Bigtable

Intro

YildizDB acts as a highly scaleable HTTP layer in front of Google Bigtable. It helps you store billions of relations between nodes in edges and resolve them in milliseconds. A small access layer helps you manage multiple namespaces with ease. YildizDB scales to hundrets of Terabytes. YildizDB excells in $N:N$ non nested relationships.

Features

  • simple namespacing (table separation) with prefixes
  • simple access management through tokens
  • multi-tenancy through table prefixes (with access management)
  • high read and write performance
  • fast read access under heavy write load
  • able to handle billions of edges and nodes
  • scales beyond Terabytes
  • lightweight deployments (small Node.js footprint)
  • ttl feature for all resources
  • HTTP Open API
  • Kubernetes HELM charts

Build for high throughput $N:N$

  • Highly async API based on fastify
  • Thin layer on top of Google's GRPC Bigtable API
  • Hashing and translating all string identifiers into integer representations via murmurhash3 automatically

Fast random access to a node's edge data $1:N$

  • Multiple complex caching layers
  • Custom Bigtable cache table speed up by Google Memorystore (Redis)
  • All Memorystore hits also speed up by In-Memory store
  • Fetch job that keeps active nodes refreshed in cache

Available clients

  • Node.js Client
  • Any Http Client can be used to access the HTTP-Interface

Usage

  • You will need a Google Cloud Project with a running Bigtable cluster
  • Additionally YildizDB requires a Memorystore (or Redis) instance
  • Configure ./config/bigtable.json accordingly
  • Install and start: npm install -g yildiz and yildizdb -p 3058 -l ./config/bigtable.json
  • A word on configuration can be found here
  • Use right alongside your code
  • Spawn server via http interface

Deployment

  • YildizDB is designed to be deployed as simple platform service
  • it requires Node.js > 9.x.x and Redis > 3.x
  • It simply requires its npm module as well as a config file that describes the connections to Bigtable and Memorystore (Redis)
  • It scales best with an HTTP load balancer in front of it e.g. NGINX
  • We run and scale it very successfully in Google's Kubernetes Engine
  • We also offer HELM charts
  • Read some more about the config file here

Metrics & Monitoring

  • Yildiz exposes Prometheus Metrics @ /admin/metrics
  • Read more about it here

Developing YildizDB

Developing YildizDB with Google Bigtable backend

  • Configure ./config/bigtable.json accordingly
  • Run tests via yarn test
  • Start via yarn http

Documentation

Disclaimer

  • This project is not affiliated with Google
  • License is MIT see