Developer testnet shows a deadlock in RSpace

Description

Observed in Rundeck shared by in discord...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 validator-01.rchain-dev.tk validator-02.rchain-dev.tk Deploying contract examples/tut-maps-methods.rho Response: Success! Proposing validator-03.rchain-dev.tk validator-04.rchain-dev.tk Response: Success! Proposing Response: Success! Block af10ce9222... created and added. Deploying contract examples/tut-hello-again.rho Response: Success! Proposing Response: Success! Block 96aea2a178... created and added. Deploying contract examples/tut-tuples-methods.rho Response: Success! Proposing validator-05.rchain-dev.tk Response: Success! Proposing Response: Success! Block 54d176b69a... created and added. Deploying contract examples/tut-registry.rho Response: Success! Proposing Response: Success! Block 32eeaec14c... created and added. Deploying contract examples/longfast.rho Response: Success! Proposing

There's no exception logged in any validator, but all proposes got stuck and all nodes stopped logging. I gathered that autopropose is connected to gRPC port, is waiting on something, and there's no traffic in the connection. The host is mostly idle, rnode is waiting on something as well. Looks like deadlock. That is on validator 5.

Stack traces https://drive.google.com/open?id=1WL73UuvLbzJSB6X6wze_7VdGg7tFLPNN

Environment

None

Status

Assignee

Dominik Zajkowski

Reporter

Kelly Foster

Priority

High

Affects versions

None

Components

Sprint

None

Epic Link

None

Labels

None

Fix versions

Configure