Block Storage Specification (draft)

Owned by Dominik Zajkowski
Last updated: Jul 23, 2018


Introduction

Purpose:

This document aims at providing a specification for a persistent storage of blocks for the BlockDAG use case.

References:

Jira Issue:  CORE-754 - Getting issue details... STATUS

Definitions:


Scope

Objectives & goals:

Non-goals:

  • refactor RSpace so that the projects share code.


To Be Determined

  • should the lmdb instance be shared with RSpace? (one file) ANSWER: NO
  • should the library support multiple block stores? ANSWER: NOT FOR THIS VERSION
  • should the Store depend on BlockHash and BlockMessage or should it be abstract K, V? ANSWER: PROBABLY NOT
  • what is the level of transactions?


Use Cases

  1. storing blocks (put)
  2. retrieving blocks (get)
  3. lookup of blocks (lookup). seems to be same as 2.
  4. view the whole db asMap

Design Considerations

Interfaces  

System Interface


Hardware Interface

The LMDB instance needs to be able to read and write to a file.

Software Interface

BlockStore v1 
trait BlockStore[F[_]] {
  def put(blockHash: BlockHash, blockMessage: BlockMessage): F[Unit]

  def get(blockHash: BlockHash): F[Option[BlockMessage]]

  def put(f: => (BlockHash, BlockMessage)): F[Unit]

  def asMap(): F[Map[BlockHash, BlockMessage]]
}


And instance:

BlockStore v1 object 
class InMemBlockStore[F[_], E] private ()(implicit
                                          monadF: Monad[F],
                                          refF: Ref[F, Map[BlockHash, BlockMessage]],
                                          metricsF: Metrics[F])
    extends BlockStore[F] {

  def get(blockHash: BlockHash): F[Option[BlockMessage]] =
    for {
      _     <- metricsF.incrementCounter("block-store-get")
      state <- refF.get
    } yield state.get(blockHash)

  @deprecated(message = "to be removed when casper code no longer needs the whole DB in memmory",
              since = "0.5")
  def asMap(): F[Map[BlockHash, BlockMessage]] =
    for {
      _     <- metricsF.incrementCounter("block-store-as-map")
      state <- refF.get
    } yield state

  def put(f: => (BlockHash, BlockMessage)): F[Unit] =
    for {
      _ <- metricsF.incrementCounter("block-store-put")
      _ <- refF.update { state =>
            val (hash, message) = f
            state.updated(hash, message)
          }
    } yield ()
}

Usage example:

BlockStore usage 
        BlockStore[F].put {
          awaitingJustificationToChild -= block.blockHash
          _blockDag.update(bd => {
            val hash = block.blockHash
            val newChildMap = parents(block).foldLeft(bd.childMap) {
              case (acc, p) =>
                val currChildren = acc.getOrElse(p, HashSet.empty[BlockHash])
                acc.updated(p, currChildren + hash)
            }

            val newSeqNum = bd.currentSeqNum.updated(block.sender, block.seqNum)
            bd.copy(
              childMap = newChildMap,
              currentSeqNum = newSeqNum
            )
          })
          (block.blockHash, block)
        }


User Interface

None. We are backend developers. We hail the matrix, and the matrix speaks to us.

Communications Interface

System Overview

Provide a description of the software system, including its functionality and matters relating to the overall system and design.  Feel free to split this up into subsections, or use data flow diagrams or process flow diagrams.  

Limitations

Assumptions and Dependencies

Architectural Strategies

Describe any design decisions and/or strategies that affect the overall organization of the system and its higher-level structures. These strategies should provide insight into the key abstractions and mechanisms used in the system architecture. Describe the reasoning employed for each decision and/or strategy (possibly referring to previously stated design goals and principles) and how any design goals or priorities were balanced or traded-off. Such decisions might concern (but are not limited to) things like the following:

  • Use of a particular type of product (programming language, database, library, etc. ...)
  • Reuse of existing software components to implement various parts/features of the system
  • Future plans for extending or enhancing the software
  • User interface paradigms (or system input and output models)
  • Hardware and/or software interface paradigms
  • Error detection and recovery
  • Memory management policies
  • External databases and/or data storage management and persistence
  • Distributed data or control over a network
  • Generalized approaches to control
  • Concurrency and synchronization
  • Communication mechanisms
  • Management of other resources


System Architecture