Genesis ceremony: Received invalid ApprovedBlock message not containing enough valid signatures

Description

I'm testing genesis ceremony with various wallets.txt files and 1 required signature. The network consists of 5 nodes, 1 bootstrap node (ceremony master, initial genesis block creator, started with --standalone) and 4 genesis validator nodes (started with --genesis-validator). All nodes are also started with --required-sigs 1.

The problem is that, with certain wallets.txt files, some validators fail to validate ApprovedBlock with the following messages in console:

1 2 3 4 5 6 7 2019-08-13 14:45:11,576 [tl-dispatcher-server-blob-60] INFO coop.rchain.comm.transport.StreamObservable - Streaming packet (type = BlockApproval) to peers rnode://5da6b63b6f0b67fb6737cf1d46369115ab36e1d3@node0.devnet.rchain-dev.tk?protocol=40400&discovery=40404 2019-08-13 14:45:11,621 [tl-dispatcher-server-blob-60] INFO coop.rchain.casper.engine.BlockApproverProtocol - Received expected candidate from rnode://5da6b63b6f0b67fb6737cf1d46369115ab36e1d3@node0.devnet.rchain-dev.tk?protocol=40400&discovery=40404. Approval sent in response. 2019-08-13 14:45:14,145 [tl-client-stream-queue-69] INFO coop.rchain.comm.transport.GrpcTransportClient - Streamed packet /var/lib/rnode/tmp/comm/20190813144511_45019171_packet.bts to rnode://5da6b63b6f0b67fb6737cf1d46369115ab36e1d3@node0.devnet.rchain-dev.tk?protocol=40400&discovery=40404 2019-08-13 14:45:17,590 [grpc-io-54] INFO coop.rchain.casper.engine.Initializing - Received ApprovedBlock message. 2019-08-13 14:45:17,802 [grpc-io-54] WARN coop.rchain.casper.Validate$ - Received invalid ApprovedBlock message not containing enough valid signatures. 2019-08-13 14:45:17,807 [grpc-io-54] INFO coop.rchain.casper.engine.Initializing - Invalid ApprovedBlock received; refusing to add. 2019-08-13 14:45:17,816 [grpc-io-54] WARN coop.rchain.casper.engine.Initializing - MultiParentCasper instance not created.

Following are results of testing the ceremony with various wallets.txt files so far. Number of entries in tested wallets file is number after test in wallets file name (I.e. wallets_block7613403_invalid_test804.txt contains 804 entries). In parentheses at the end is number if validator node, i.e. excluding bootstrap, that successfully processed ApprovedBlock.

For command lines see the Rundeck job linked for each test.

It's possible that the issue is non-deterministic and bisecting the original wallets file (testnet2-wallets.txt) to final boundary may not be successful. Here's list of recent successes compared to list above (and failures):

Logs:
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T10-48-38.devnet.node0.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T10-48-38.devnet.node1.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T10-48-38.devnet.node2.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T10-48-38.devnet.node3.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T10-48-38.devnet.node4.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T12-49-23.devnet.node0.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T12-49-23.devnet.node1.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T12-49-23.devnet.node2.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T12-49-23.devnet.node3.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T12-49-23.devnet.node4.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T14-33-06.devnet.node0.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T14-33-06.devnet.node1.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T14-33-06.devnet.node2.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T14-33-06.devnet.node3.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T14-33-06.devnet.node4.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T15-19-22.devnet.node0.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T15-19-22.devnet.node1.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T15-19-22.devnet.node2.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T15-19-22.devnet.node3.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T15-19-22.devnet.node4.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T16-19-05.devnet.node0.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T16-19-05.devnet.node1.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T16-19-05.devnet.node2.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T16-19-05.devnet.node3.genesis-sigs
https://files.rchain-dev.tk/heapdumps/devnet.rchain-dev.tk/2019-08-13T16-19-05.devnet.node4.genesis-sigs

Environment

None

Status

Assignee

Ovidiu Deac

Reporter

Tomáš Virtus

Priority

Medium

Affects versions

None

Sprint

None

Epic Link

None

Labels

None

Fix versions