Community Update 173

Owned by Rao Bhamidipati
Last updated: Jun 02, 2021

General

  • Status of the networks and Releases
  • We continue to work on improvements to the block merge version. Details are at Core team · GitHub
  • Note: Main difference between testing on Observer nodes vs. Validator nodes: When we test on the observer node, a large part of the code base is executed. The Block creation code runs only on the validator nodes. As a part of that, the portion of the Runtime Manager that plays the deploy is run on the validator node but not on the observer node.

Sprint 84 in progress

    • Main Focus is to continue Block merge work and improvements, resolve any identified bugs, prepare for hard fork 1 to eliminate the slashed validator, harden the main net, improve performance.  Current PR list is at https://github.com/rchain/rchain/pulls .  
    • Significant ongoing work in the following streams simultaneously
      (1) Leaderful block merge targeted for the current main net
      (2) Leaderless block merge changes that will get on the main net, even though may not become immediately operational
      (3) Block merge tasks that can be on the main net only after Hard fork 2
      (4) Hard fork 1 for balances
      (5) Planning for Hard fork 2 - identifying all items/tasks necessary, analyzing impact and interactions, specifying and reviewing the designs etc.
      (6) Implementing Hard fork 2 changes for items that are clear how to do
      (7) Performance improvements
    • Currently block merge is being tested on both Testnet 1 and Testnet 2. There's a 'hard-forked block merge' version on Testnet 2 that is being run as needed to apply patches, run tests, measure performance etc. This will be the eventual long-term version of block merge. 
    • The team is currently focused on creating new tests and/or refactoring the existing tests to accommodate block merge changes. They are also focused on understanding and improving the tests for safety oracle, fault tolerance etc. and making the necessary fixes in these areas. Optimizing cache clearing behavior is another area being worked on.
    • Separately, Gurinder and the team are working on performance tests on various configurations of the network, to both measure performance as well as identify any undesirable trends. 
    • The team is close to completing the necessary feature branches and rebasing code on these.
    • Some of the performance tests need high number of cpus and nodes to be run for a brief period of time and stopped (because this is expensive).  This was sufficient for diagnostic purposes and to get the performance numbers. We will now be running these tests for longer periods of time to detect any performance degradation as blocks accumulate.
    • Once we are done with tests and have a stable version, we should be able to make this Testnet 2 accessible to the community. 
    • Nutzipper is continuing to work on the remaining block merge changes, including enhancements. Tomislav and Will will help him with writing tests on all parts of the block merge functionality.
    • Hardfork 1 for balances: Will has completed the PRs for the first hard fork (carry forward balances without prior state). The initial report of balances at block height 804000 is now available for community review. Please check your rev addresses to make sure your balances are correct. Next week, the devs will have a presentation on the work and the code that generated these balances, so that interested community members can review their work and generate any questions/comments. Also, any one in the community can repeat their work and validate the results.
    • Balance Adjustments: Separately, there was work going on in the Governance committee (under Jimscarver's leadership) to collect forensic accounting evidence from folks claiming REV stuck in EtherDelta contracts etc. If the committee is convinced of the completeness and accuracy of the forensic evidence, they're making recommendations for adjustments. You can verify their work in the https://github.com/rchain/Governance-Committee/issues repository. The elected Board will take into account these recommendations, to make any necessary adjustments.
    • Performance Improvement efforts:   One of the community members has done some very nice work on a Rust implementation of parts of the Rholang interpreter, demonstrating the potential for 100x performance improvement. A lot of this is future potential, not immediately leverageable by the current code base. We will start exploring with him next week to see what parts of his work may be applicable and/or what other activities can be targeted for the near-term improvement of RNode performance, even as we plan for a future Rust client if that makes sense.  One of the candidate near term activities for performance is to replace protobuf with CapnProto. We anticipate that this would increase performance, as well as ensure (currently being verified) data structure compatibility between Scala and Rust clients.  As we work through all of this, we will update the roadmap.  I will also be adding a 'TPS/Performance Improvement' milestone to the roadmap, as requested by the community.  Tomislav has created an issue with critical paths for performance improvements https://github.com/rchain/rchain/issues/3413 so that we can report metrics in a coherent fashion and identify which paths offer best return on effort.
    • As we work through block merge and the PoS changes, the team is discovering more and more items/changes that need to be included in the second hard fork. We are being mindful to not expand the scope too much because that would increase risk of introducing too many bugs and interactions. One approach being considered is to create a safe process for soft forks, such that all changes need not go in as part of the hard fork.
    • VERY Preliminary performance numbers on the leaderful, 0.99 synchrony constraint (unoptimized) block merge version that will be eventually promoted to the main net:
      • 7x to 10x improvement (currently 41 sec per block on the main net vs 4 to 6 secs per block with block merge) over what we see on the main net currently, with 5 nodes on testnet 2, with the 0.11.0.alpha-2 version.  
      • Numbers likely to change as we move to leaderless 0.67 synchrony constraint, remove channels map, vary the number of nodes, vary the number of blocks for tests, vary machine resources, vary network distribution etc.. 
    • Block merge scope
      • The main net version of block merge works with synchrony constraint 0.99. It was easier to do it this way to identify and analyze bugs and take corrective action. The focus here is to just make sure the algorithm is correct and works well. For now, we will simply kick out the unmerged deploys back to the user. In future, we will see if there is a better way to handle this.
      • As we have been emphasizing in the last several community debriefs, we discovered an unanticipated need to remove the channel mappings to get the full performance on block merge. This removal is a data structure change and will need a hard fork. For this reason, the full performance of block merge will not be visible on the main net until after we perform the data hard fork. There are also a few optimizations that need to be completed. Status of all block merge tasks can be seen at <https://github.com/rchain/rchain/projects/4> . We are approximately 2 to 3 weeks away from completing the remaining block merge enhancements. 
      • The block merge code that does not need the hard fork will continue to be tested on the current testnet and move on to the main net. The current testnet remains available for regular use by dApp developers and others, as normal.

TEST NET 2:  To test the version of block merge with data changes that need a hard fork, we are creating a separate feature branch and a TEST NET 2 as the testbed for this feature branch.

  1. We will generate from TEST NET 2 what the eventual block merge performance numbers may look like. To enable this, Will and Gurinder will be working on setting up a transaction server on Testnet 2, so that revdefine.io can report the statistics on this net also.
  2. Once block merge tasks are complete, we will continue to work on other hard fork changes and test them on TEST NET 2 as well.
  3. Additionally, we will start experimenting with decentralized nodes on TEST NET 2 when we are ready for that.
  4. Through this whole duration, there will be no guarantees of data storage format compatibility on Test Net 2, as the data storage format changes are incrementally implemented. However, we encourage the community to start using Test Net 2 in addition to the current test net, so that we can quickly identify and resolve any issues with the upcoming hard forks. All new development should be targeted to TEST NET 2 and all current code MUST BE tested against TEST NET 2 to ensure future compatibility. 

The anticipated changes are mostly storage level format changes visible only to node operators. When we hard fork on the main net, we will be starting from an empty state, with REV balances only. On Test Net 2, this will be repeated multiple times.

    • There will be further testing, optimization, fixing bugs and creating necessary enhancements after that. Note that block merge speeds up certain areas that may reveal bottlenecks in other areas (disk i/o, disk storage capacity, memory etc.). So we need to get the right monitoring instrumentation in place and test this for a while before it's ready for the main net. Also the branches need to be merged, so we can make sure everything is working well with block merge and also do future branches off this new version.
    • Exchanges: Will has been working on clarifying the processes and creating the documentation to smooth bringing new exchanges online. 
    • Will has completed the code to get balances https://github.com/rchain/rchain/pull/3331 in preparation for the hard fork. He has been running test migration. This is taking long at the moment, we will try with more resources (cpu and memory) to see if we can make this go faster. The improved configuration and process is what we will follow when ready to do the actual hard fork with balances only (no state carried over).
    • We want the community to be able to inspect the code and also the results a few times, so that the hard fork is as smooth as possible. We also want to practice the hard fork a coupe of times. The first hard fork is to remove the slashed validator from the bonds map, which is stored inside the POS contract and possibly, reduce the validator funds quarantine period (currently too long).
    • SRE :
      Getting ready to set up additional nodes and new networks.
    • Planning for a couple of hard forks after block-merge is on the main net: One fork is to clear the slashed validator node from the main net, update bonds file and carry over the balances from the current main net to the new fork. This will be looked at as a practice/trial run of a hard fork. Second fork is to correct some bugs and other configuration items. An initial partial list of issues to be addressed in this fork is at https://github.com/rchain/rchain/issues/3315 . This list is likely to grow. Together, these should prepare us for the POS update and external validators. The rholang v1.1 update of system contracts and enabling rholang v1.1 broadly will likely be separate events, even though it is currently on that list. Rholang v1.1 should be a soft fork however. 
    • Tobi has created a nice Rholang focused documentation page <https://rchain-community.github.io/>
    • Rholang v1.1 (complete for now) : Joe has completed the implementation of the full source to source translation of the Rholang v1.1 syntax into current Rholang. He's resolving the last few review comments. He added tests and they're passing. The feature branch is at https://github.com/rchain/rchain/tree/feature/rholang-1.1  I encourage the community to test these releases early and often so that Joe gets the necessary testing feedback and we can move it towards deployment sooner than later. You can run a version of the node locally with this.  dckc pointed out that this can be a neat developer tool meanwhile - i.e. write your contracts in rholang v1.1 to generate the rholang v1.0 code.  rholang v1.1 is much more compact and is the future. So this should be practiced by the community and developers as much as possible. 
    • Quite a lot of work needs to be done before we get Rholang v1.1 into production on the main net. We are currently planning the steps and will have some milestone updates after this planning process is complete. A fuzzer to substantially increase the testing and identify performance optimization areas is one of the immediate steps. 
    • Rholang v1.1 spec is at https://drive.google.com/file/d/1FvPuiPm7ytGyklex_PEmVyJ8XgbTdIEg/view   Most of these changes will need extensive testing when the system contracts etc. are re-written using the Rholang1.1 syntax. These changes will be in a separate feature branch and we will likely run a separate network eventually, to thoroughly test this implementation. We also need to create documentation to help end users verify their existing code and/or how to modify it to use the new syntax. The expectation is that the new syntax will make the learning curve a lot easier for people familiar with other languages. Separately, we need to update the IDE level tools to work with the new syntax.  We will run much of this through the RCHIP and Tech Governance process.
    • Ongoing - Fix additional identified bugs
    • API and usability improvements focused on Developer adoption (no change this week) - in conjunction with the community
      • Tomislav is beginning to review the best practices, frameworks, widgets and code snippets being made available by other projects to make things easy for developers, so we can adopt those best practices for RChain. The platform-adoption channel is now open in pub-RChain Discord, to enable announcements and community feedback in this area.
      • Tobi has made a Rholang template available on codesandbox.io that allows one to execute rholang code on the RChain network in real time. codesandbox.io claim some 2 M monthly developer users and that 10 M apps have been developed on the platform. Tomislav had talked about enabling something similar on repl.it, which is a similar 'code and collaborate in the browser' platform.
      • We continue to consider partnerships for enterprise developers as well as engage with small developer teams that are already engaging in blockchain proposals and pilots. 
      • In Process - Will has mostly completed the OpenAPI/Swagger documentation of the RNode API  https://github.com/rchain/rchain/pull/3128  He's will try to resolve the reviewer comments after he's done with block merge work or as time permits. Familiarity with Swagger and the available swagger tools should make it easy for developers to use our API.
      • Future - Also in the thinking stage is an improvement to revdefine.io or elsewhere, a graph representation of payments / transfers on the network, with perhaps date+time and amount on the arcs.
    • Release 0.11.0 - Block merge
    • Release 0.12.0 and beyond
      • Tentative Near term road map is
        • 2021Q1 - block merge, (2 to 3 weeks more to complete the block merge enhancements)
        • 2021Q2 - updating PoS contract to enable external validators,
        • 2021Q3 - Rholang v1.1 operationalization,
        • 2021Q4 - sharding and name spaces, in that order.
      • Milestone dates have been revised based on our current estimates. There will be further updates as we revise the plans.
      • Creating a design spec and the code for how to update contracts on the main net.  This will enable addressing some of the pressing issues such as faster epoch times, and other issues listed below
      • Ongoing: Finish dependency updates and plugin updates
      • To Do: Improve error logging, debugging and monitoring infrastructure. This will help improve developer productivity, both on the core team as well as dApp developers
      • Completed for now - Rholang v1.1, Refactor Run Time Manager. We may need more refactoring in future, but we want to do this incrementally, as needed. 
      • Protection from 'too much free work' attack, policy to reject deploys when user doesn't provide sufficient phlo or does not have sufficient REV in the wallet to cover the phlo etc - https://github.com/rchain/rchain/issues/2997  and https://github.com/rchain/rchain/issues/2976 
      • Implement On-chain configuration and Shard configuration  https://github.com/rchain/rchain/pull/2986
    • Explanation for last finalized state (released in 0.10.0): The scope of this work enables (a) faster catchup by new nodes - you can start from the last finalized state - this is a differentiator for RChain (b) offloading older data, differentiated storage and retrieval strategies for the same (c) allows for a leaner / less bloated node. This change touches most parts of the code base. Having to pick between refactoring and work-arounds in various parts. Tomislav implemented a mostly modular and future-beneficial approach.
    • Developer + Researcher engagement and Hackathon 
      • Completed - Greg, Steve Ross-Talbot and the Dev team will be participating in half-day workshops with some Process Algebra folks in UK on Feb 4 2021. Follow up session was requested by some participants but not set up yet. 
      • Next hackathon tentatively scheduled for some time in March. 
      • We need to make building the developer ecosystem an ongoing task, just like development.  Ian, Steve Henley and team are actively reaching out to as many universities and groups as possible. Please promote the much improved hackathon page at https://rchain.coop/hackathon.html to all the groups you are part of. 
        • Ongoing - We need to further increase and properly size the Problem Statements, brief sessions on tips and tricks during the hackathon, and office hours during the hackathon. We will keep adding to the problem statements.
        • Ongoing - Learning examples,  links  to and improvement of documentation.  Dan (dckc) has substantially improved the Rholang learning resources. Current plan is to make https://rholang.github.io/ the hub of all learning materials. dckc has been making substantial improvements to the learning materials as well as the API tools.  The community is requested to contribute to any or all of these.
    • CI/CD:
      • Ongoing - We are slowly moving from Jira to Github for the development team, started publishing release notes in Github with release 0.9.25.  New issues are entered only in github. For a while, we will maintain in both Jira and Github. Looking to open up Github discussions and project management in the next few months.
    • TechGovernance:
      • Thursday 10 AM Eastern. Meets alternate weeks.  Last meeting focused on how to implement and manage shard configuration. Meeting notes are at https://docs.google.com/document/d/1fvxMC6Bt5XwbVaLzYPy6ZPB8KzJvASO2sKRC6ZCPwpI  Discussions and work on a model of economics to ensure network security is proceeding. Most recent discussion was about ensuring some amount of stake beyond a 4 month window, spreading out stake over a longer time and how to incentivize that, while ensuring that everyone pays for network security.
      • ToDo - There were questions from the community regarding enabling privacy and zero knowledge proofs on the chain - this is in active discussion. Greg will create a position statement on this and likely discuss the current state of the art, use cases and potential approaches in an RCast session.  Same with sharding and cross-chain operation. 
    • In progress - Excel Model for 'economics of sustainable network security and decentralization' to enable community understanding of the inter-relationships and community conversation around these issues.  Will be releasing the initial version soon. May have multiple releases.
    • dApp Development:
      • Looking for developers to help make progress on the RChat effort using zulip.  SRT and possibly Will to add 'Rchain to zulip' functionality. Invite any community members with the expertise to help in this.
    • Current Backlog (partial)
      • Improve merging in system deploys
      • Improve Casper by enabling more tests and resolving identified code issues
      • Improve BlockMerge 
      • Improve multi-parent Casper enablement
      • Resolve slashing bugs, including additional refactoring of RunTimeManager
      • Implement sharding capabilities
      • Improve logging to be able to learn what API calls are being used, so they can be related to resource use and performance etc
      • Operationalize Rholang 1.1 on the main net to improve syntax and user experience / learning curve
    • Tuesday TeachOuts by Tomislav (Tuesday 10 AM Eastern, 7 Am pacific) in Jimscarver's zoom room https://zoom.us/j/6853551826 

Tech-Governance meetings on Thursdays 10 AM Eastern, 7 Am Pacific 

Mercury requirements and acceptance criteria

Details on the acceptance criteria: Mercury acceptance criteria

Please see the documentation at https://github.com/rchain/rchain/blob/dev/docs/features.md. 

Testnet status

Please see RChain public testnet information to learn more about public testnet as well as a FAQ.

Tech Governance + Community testing

Thursdays at 14:00 UTC. Please see RChain community RNode testing for more information.

Blockers to Mainnet

NA

Risks to code completion for Mercury


Developer website

https://developer.rchain.coop 

Date