...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
1.1 Introduction
Commmunication module exposes three main abstraction:
- Transport Layer - implemented using gRPC
- Node discovery - implemented using specic version of Kademlia.
- RChain protocol - implemented in Connect.scala
1.2 Ideal world
In ideal world: Transport Layer has zero dependecies. It provides abstracton to send any (serializable) messages between two nodes.
Node Discovery is depended on Transport Layer. It will use Transport Layer to send messages, providing constant discovery of nodes in P2P network. It will expose two functionalities: addMe (to add given node to the network) and listPeers (to list peers [neighbours] that Node Discovery knows at the given moment). This abstraction does not know anything about RChain yet. It just gives an abstraction of a P2P network - allowing external entities (like for example RChain protocol) learn about neighbouring nodes.
RChain Protocol depends on both Transport Layer and Node Discovery. It uses NodeDiscovery.listPeers in order to learn about neighbouring nodes. It uses Transport Layer to send messages to those nodes. It exposes concept of a "connection" (exchange of ProtocolHandshake-ProtocolHandshake response) and ability to send messages of type Packet between nodes. RChain Protocol allows to send Packet messages only to nodes who previously went through the protocol handshake.
Diagram below shows dependency usage between all three abstractions in the ideal world
+-------------------------+ | | | RChain Protocol | | | +-------------------------+ | | uses |uses v | | +-----------+ | | | | | Node | | | Discovery | | | | | +-----------+ | | v v uses +--------------------+ | Transport Layer | +--------------------+
That's the dream. Reality is...
1.3 Reality
Reality is that Transport Layer, Node Discovery and RChain Protocol have circular dependencies. Some of those dependencies we could live with for now (just add overall complexity to the design), some are potential cause of weird bugs and security issues (and should be considered as high priority issues).
Diagram below illustrates those dependencies
uses messages +-----------+ +----------------------+ Transport | <---------------+ | | Layer | | | +-----------+ | use to send | +----------------------^ | messages | | uses to send messages | v | | | | +-----------------+ uses findPeers | | | <----------------------------------------------------+ | Node Discovery | | Rchain | | | | Protocol | +-----------------+ +--------------+ +------------------------------------------------^ peers only added by rchan protocol
Cycle: Node Discovery ← → Transport Layer
This cycle has lower priority. It is not crucial to fix it right away. Details in CORE-836 and CORE-839.
Cycle: Node Discovery ← → RChain protocol
This cycle has high priority and should be addressed ASAP. Details in CORE-770.
1.4 Detailed description of abstractions
Below detailed description of how the abstractions are currently implemented and what are the issues that have to be addressed.
1.4.1 Transport Layer
Transport Layer abstracts the communication pipeline, allowing two nodes sending messages between each other. Transport Layer is neither aware of Node Discovery nor RChain protocol. Current implementation TcpTransportLayer is missing integration tests, minimum set of missing tests are defined under CORE-607.
Transport Layer exposes following functions:
trait TransportLayer[F[_]] { def local: F[PeerNode] def roundTrip(peer: PeerNode, msg: Protocol, timeout: FiniteDuration): F[CommErr[Protocol]] def send(peer: PeerNode, msg: Protocol): F[Unit] def broadcast(peers: Seq[PeerNode], msg: Protocol): F[Unit] def receive(dispatch: Protocol => F[CommunicationResponse]): F[Unit] def disconnect(peer: PeerNode): F[Unit] def shutdown(msg: Protocol): F[Unit] }
...
Method local returns PeerNode representing given running node. This method should be removed and PeerNode information should be provided via `ApplicativeAsk` abstraction (see CORE-738 and CORE739 for detail).
...
Method disconnect should never be part of TransportLayer abstraction. It is TcpTransportLayer implementation detail. It will be removed from the abstraction (see CORE-835 for details).
...
Shutdown method gracefully shuts down the Transport Layer. After being called, no message can be send (via roundTrip, send, broadcast) nor received.
...
Methods roundTrip and send allow sending and receiving messages of type Protocol - object defined in routing.proto. Protocol message can be one of the following:
...
Commmunication module exposes three main abstraction:
- Transport Layer - implemented using gRPC
- Node discovery - implemented using specic version of Kademlia.
- RChain protocol - implemented in Connect.scala
Transport Layer
Transport Layer abstracts the communication pipeline, allowing two nodes sending messages between each other.
The specification below is governed by TransportLayerSpec.scala, see code for details.
TcpTransportLayer
when doing a round trip to remote peer
when everything is fine
- should send and receive the message
when response takes to long
- should fail with a timeout
when there is no response body
- should fail with a communication error
when peer is not listening
- should fail with peer unavailable error
when there was a peer-side error
- should fail with an internal communication error
when sending a message
- should deliver the message
- should not wait for a response
- should wait for message being delivered (pending)
when brodacasting a message
- should send the message to all peers
when shutting down
when doing a round trip
- should not send the message
when sending a message
- should not send the message
when broadcasting a message
- should not send any messages
Transport Layer exposes following functions:
trait TransportLayer[F[_]] { def roundTrip(peer: PeerNode, msg: Protocol, timeout: FiniteDuration): F[CommErr[Protocol]] def send(peer: PeerNode, msg: Protocol): F[Unit] def broadcast(peers: Seq[PeerNode], msg: Protocol): F[Unit] def receive(dispatch: Protocol => F[CommunicationResponse]): F[Unit] def disconnect(peer: PeerNode): F[Unit] def shutdown(msg: Protocol): F[Unit] }
- disconnect
Method disconnect should never be part of TransportLayer abstraction. It is TcpTransportLayer implementation detail. It will be removed from the abstraction (see CORE-835 for details).
- shutdown
Shutdown method gracefully shuts down the Transport Layer. After being called, no message can be send (via roundTrip, send, broadcast) nor received.
- roundTrip, send, broadcast
Methods roundTrip and send allow sending and receiving messages of type Protocol - object defined in routing.proto. Protocol message can be one of the following:
message Protocol { Header header = 1; oneof message { google.protobuf.Any upstream = 2; Ping ping = 3; Pong pong = 4; Lookup lookup = 5; LookupResponse lookup_response = 6; Disconnect disconnect = 7; } }
- Ping, Pong, Loookup, LookupResponse
Ping+Pong and Lookup+LookupResponse are part of the Node Discovery abstraction, so there is clear coupeling between those two abstraction. More details in CORE-836.
- Disconnect
Disconnect is currently send by TcpTransportLayer when it shuts down (Disconnect is being broadcast). This method is NOT needed as nodes should realize that other peers disconnect regardlessly if they managed to shut down gracefully or not. This message will be removed from the Protocol object (see CORE-838 for details).
- Any upstream
This is a placeholder for ANY possible message that is build on top Protocol. Effectively this is where messages from RChain protocol will be placed. Originally was designed to decouple TransportLayer, NodeDiscovery and RChain protocol (not successfully). This coupeling should be removed, see CORE-839 for details.
- Ping, Pong, Loookup, LookupResponse
Node Discovery
Node Discovery is depended on Transport Layer. It will use Transport Layer to send messages, providing constant discovery of nodes in P2P network. Node Discovery is implemented using Kademlia protocol.
A PeertTable with 1 byte addresses and k = 3
when adding a peer to an empty table
- should add it to a bucket according to its distance
- should not ping the peer
when adding a peer when that peer already exists but with different IP
- should replace peer with new entry (the one with new IP)
- should move peer to the end of the bucket (meaning it's been seen lately)
when adding a peer to a table, where corresponding bucket is filled but not full
- should add peer to the end of the bucket (meaning it's been seen lately)
- no peers should be pinged
when adding a peer to a table, where corresponding bucket is full
- should ping the oldest peer to check if it responds
and oldest peer IS responding to ping
- should drop the new peer
and oldest peer is NOT responding to ping
- should add the new peer and drop the oldest one
RChain Protocol
RChain Protocol depends on both Transport Layer and Node Discovery. It uses NodeDiscovery.listPeers in order to learn about neighbouring nodes. It uses Transport Layer to send messages to those nodes. It exposes concept of a "connection" (exchange of ProtocolHandshake-ProtocolHandshake response) and ability to send messages of type Packet between nodes. RChain Protocol allows to send Packet messages only to nodes who previously went through the protocol handshake.
Diagram below shows dependency usage between all three abstractions in the ideal world
+-------------------------+ | | | RChain Protocol | | | +-------------------------+ |
|
uses
|uses v
|
|
+-----------+ | |
|
| | Node
| |
| Discovery | |
|
| |
+-----------+
|
|
v
v uses
+--------------------+
| Transport Layer
Ping+Pong and Lookup+LookupResponse are part of the Node Discovery abstraction, so there is clear coupeling between those two abstraction. More details in CORE-836.
Disconnect is currently send by TcpTransportLayer when it shuts down (Disconnect is being broadcast). This method is NOT needed as nodes should realize that other peers disconnect regardlessly if they managed to shut down gracefully or not. This message will be removed from the Protocol object (see CORE-838 for details).
This is a placeholder for ANY possible message that is build on top Protocol. Effectively this is where messages from RChain protocol will be placed. Originally was designed to decouple TransportLayer, NodeDiscovery and RChain protocol (not successfully). This coupeling should be removed, see CORE-839 for details.
| +--------------------+
That's the dream. Reality is...
1.4 Detailed description of abstractions
Below detailed description of how the abstractions are currently implemented and what are the issues that have to be addressed.
1.4.1 Transport Layer
1.4.2 Node Discovery
Below we describe how Node Discovery works currently
- Relation to Transport Layer
Node discovery is using Tranport Layer to fullfil its work. It uses
+----------------+ | | | NODE DISCOVERY | | | +----------------+ | v uses +----------------+ | TRANSPORT | | LAYER | +----------------+
This is almost true, as mentioned in previous section, TransportLayer sends messages of type Protocol (routing.proto) that are very Kademlia specific. See CORE-838 and CORE-839 to see the issue how to remove that cyclic dependency between Node Discovery and Transport Layer.
2. How current Kademlia implementation is different from the original paper
...
3. Kademlia in etherum
Etherum also uses its own Kademlia-like protocol. Description under https://github.com/ethereum/wiki/wiki/Kademlia-Peer-Selection
1.4.3 RChain protocol
RChain protocol messages are defined in p2p.proto and are the following:
message Protocol {
oneof message {
Disconnect diconnect = 1;
Hello hello = 2;
ProtocolHandshake protocol_handshake = 3;
ProtocolHandshakeResponse protocol_handshake_response = 4;
Packet packet = 5;
}
}
Messages Hello and Disconnect are not used in the RChain protocol (even though defined) and will be removed from it (see CORE-840 for details).
RChain Protocol exposes few functionalities:
- Connect to bootstrap - feature that allows given node to connect to given bootstrap
- findAndConnect - feature that constantly triggers NodeDiscovery to look for new nodes and runs for them connection protocol
- Ability to send messegae of type Packet to nodes that exchanged the protocol handshake (went through the connection)
List of issues mentioned in this document
...