The Storage Layer will be used as library code in an existing Scala codebase.
The Storage.01 release of the Storage Layer is envisioned to cover the following use cases:
- It should allow the Rholang interpreter to persist and retrieve objects related to it's execution state. It WILL NOT execute those objects.
- It should be general enough to for use as a storage layer outside of the context of the Rholang interpreter. This will be delivered as a self-contained Scala library, targeting Scala 2.11.x, 2.12.x, and eventually 2.13.x
Here are a couple of issues which need to be addressed or resolved before attempting to devise a complete design:
How do we support keys longer than the
By default, LMDB is built with a maximum key size of 511 bytes.
We should generate the cryptographic hash (with a suitably fast hashing algorithm like BLAKE2b) of the serialized bytes of a Normalized Rholang Term. This hash is stored as the key in two (or more) key-value pairs in separate LMDB sub-databases.
One pair is (key: hash, value: actual key).
The other pair is (key: hash, value: actual value).
An important thing to note is that solution relies on the assumption that the serialization process is deterministic. That is, it always return the same bytes, regardless of the version of serialization library.
Early (somewhat outdated) sketch contained here: https://github.com/henrytill/rchain/commit/d662006cae5ce458cffd0be622a28971aec51366
This commit sketches out an approach to handling keys that are longer
than MDB_MAXKEYSIZE (511 bytes).
It creates two separate sub-databases, one for keys and one for values.
When a key-value pair is to be added, we generate a cryptographic hash
of the key using the BLAKE2b algorithm. The hash becomes the pair's key
in both the keys database and the values database. We store the
original key as the value in the keys database. We store the original
value as the value in the values database.
In discussions about the functionality of produce and consume, an important piece is the matching algorithm used to check if a channel matches against a certain pattern.
How is this check done?
Former user (Deleted) said "Run it through a matching algorithm - this needs to be written explicitly."
Kyle Butt said the following:
We can gradually improve the matching algorithm. It's the same algorithm used for
match in the interpreter.
Let's start with lists ground types and variables. That's what the RBL backend currently supports, and it's easy to do that much.
The matching algorithm is very similar to the spatial type checking algorithm found here. The basic idea is that
true in the model checker corresponds to a free-variable in the pattern.
It should be easier to do some of the matching because we have things binned. Not a lot easier, but let's take what we can get.
Matching may be expensive. We should account for it in the cost modeling.
Michael Stay (Unlicensed) has suggested that we use Protocol Buffers to define the data types common to Rholang and the Storage Layer. Protocol Buffers offer a straightforward means of binary serialization. The serialized Rholang objects will be stored in LMDB.
See the following tickets for a description of the planned work:
See System Dependencies for more information about Protocol Buffers.
Storage.01 will be programmed in Scala as a Scala library.
For the Rholang Interpreter, the class files will be linked transparently as part of the sbt build process, and included in a packaged Node.
For users outside of the RChain project, this will be delivered as a self-contained JAR, targeting Scala 2.11.x, 2.12.x, and eventually 2.13.x
(From the standpoint of a user writing Scala code)
coop.rchain.storage package, a user will instantiate a
Storage object with the desired configuration parameters like the location of the database file and the size of the underlying key-value map.
The Storage.01 is designed to be agnostic to the underlying data types that it persists. It does this by providing a generic, type class-based API. For more information about type classes, see above.
In order to use the library, a user will need to provide instances for two type classes:
Instances of the
Seralize type class are used to serialize and deserialize the data types that user wishes to persist in the storage layer.
Essentially, this means providing definitions for two functions:
encode, which converts a given object to
decode, which converts a given
Array[Byte] to that object
Instances of the
Match type class are used for matching a user-determined pattern type against the keys in the storage layer.
Essentially this means providing a definition for one function:
isMatch which takes a value of the some type
A representing patterns and a value of some type
B representing data and determines if they are a match, returning a
In the case of the Rholang interpreter, these are the the types that form Rholang's core Abstract Data Types (ADTs). We will provide instances of
Match for these data types.
The user will then use two functions to interact with the
These functions are presented below in the (deprecated) RSpace 0.1 Specification#API Section.
When a user has completed their interactions with the
Storage object, they will call a
close function to clean up its resources.