Block Storage Example

A custom block storage example to manage (store/index etc.) block (aka file/document content) operations in a p2p manner.

Introduction

blockstorage is a p2p service that I wrote to understand the internal workings of block management. Important topics that I focusing on:

1. Libp2p
   • Kademlia DHT
     • Content Routing usage
     • Peer Routing usage
     • P2P Protocol usage
   • Peer Discovery (Boostrap Peer, Rendezvous, etc.)
2. IPLD
   • Concepts & Spec
   • Proof of Concept custom implementation (via protobuf)
3. Formats
   • Multi Address
   • CID (aka content identifier)

Libraries/Implementations

• go-objectstore-lib : Contains base abstraction of storing encoded/serialized objects

• go-objectstore-fs : Contains file system based implementation/functionality to store encoded/serialized objects

Layout

• api/proto : Contains protobuf definitions
• blockpb : Contains protobuf and grpc related objects according to store.proto
• blockpb/store_aux.go : Contains auxiliary functions/definitions to extends proto objects
• util/ctx.go : Contains context cheking helper function and error definitions
• peer : Contains p2p functions and definitions.
• errors.go : Contains blockstorage error definitions and error checking functions
• grpc : Contains blockstorage GRPC endpoint definition and RPC function implementations
• impl.go : Contains BlockStorage interface implementation and helper functions
• options.go : Contains BlockStorage construction option definitions
• peer.go : Contains p2p related protocol definition and functions
• storage.go : Contains blockstorage construction and BlockStorage interface definition

Status

blockstorage is still in progress.

TODOs

• Prepare Step
  • Project Structure
  • Makefile tasks
• Grpc Step
  • proto file
  • generate objects
  • add makefile task
  • implement service endpoint
• Peer Step
  • protocol definition
  • store definition
• Future
  • add block indexing mechanism
  • add garbage collection trigger mechanism (to temporaryStore)
  • add long term storage trigger