RChain RNode Metrics - Node Operator

Owned by Keith McQueen
Last updated: Oct 25, 2018

Introduction

The RChain RNode software is a complex distributed system. It is desirable that node operators manage, maintain, and tune their individual nodes for best reliability and efficiency. To do this, data is required on how various aspects of the node are functioning. Only by analyzing and visualizing this data can you as a node operator make educated decisions on how to deploy your software and hardware resources.

RNode Metrics Configuration

Prometheus

During the course of execution, RNode generates a variety of metrics and exposes these on request using a software interface known as Prometheus. These can be queried on demand by accessing the metrics port. By default this is port 40403, but may be changed when the RNode software is launched via the Rnode configuration file or on the RNode command line. See the User guide for running RNode documentation for more details.

For test purposes you can see these metrics using wget or curl by specifying the RNode IP address and the metrics port.

wget
wget <hostname>:40403/metrics


or

curl
curl -s <hostname>:40403/metrics


To configure Prometheus, modify the node/prometheus/prometheus.yml file.

Grafana

The data output by either of the above commands is in a raw format, and is not very human readable. For this purpose, sample configuration files are provided for the Grafana visualization tool. Grafana is able to poll the Prometheus metrics port and display the desired metrics in a human readable and understandable format. By default, Grafana uses port 3000. Bring it up using http://<hostname>:3000/

.

Default Grafana configuration files are:

node/grafana/grafana.conf

node/grafana/provisioning/dashboards/dashboard.yml

node/grafana/provisioning/dashboards/genesis-metrics.json

node/grafana/provisioning/dashboards/rnode-metric-counters.json

node/grafana/provisioning/dashboards/rnode-metrics.json


RNode Prometheus names

Counters

rspace_comm_produce_total
block_store_put_total
tracer_spans_created_total
rspace_comm_consume_total
replayrspace_comm_consume_total
replayrspace_comm_produce_total

Gauges

NOTE: Many of these will migrate to other metric types.

mempool_ps_eden_space_peak_commited
mempool_ps_old_gen_commited
memory_non_heap_used
memory_heap_max
mempool_compressed_class_space_commited
size rnode/rspace/casper
rnode/rspace
thread_total_started
mempool_code_cache_peak_max
mempool_ps_old_gen_peak_commited
mempool_code_cache_peak_commited
mempool_metaspace_init
mempool_metaspace_commited
mempool_code_cache_commited
mempool_compressed_class_space_peak_used
mempool_compressed_class_space_init
thread_count_daemon
gc_ps_scavenge_total_collections
thread_count_peak
mempool_ps_eden_space_commited
mempool_ps_old_gen_used
gc_ps_scavenge_total_collection_time
gc_ps_marksweep_duration
mempool_metaspace_peak_init
mempool_ps_survivor_space_used
memory_heap_commited
memory_heap_used
mempool_ps_survivor_space_peak_max
gc_ps_scavenge_end_time
mempool_code_cache_init
process_cpu_load
mempool_compressed_class_space_max
memory_non_heap_init
mempool_code_cache_peak_init
mempool_code_cache_max
entries rnode/rspace
entries rnode/rspace/casper
mempool_metaspace_peak_max
mempool_ps_old_gen_init
mempool_ps_survivor_space_peak_used
gc_ps_marksweep_total_collections
gc_ps_scavenge_duration
mempool_ps_old_gen_peak_used
mempool_metaspace_used
mempool_ps_eden_space_peak_used
mempool_ps_old_gen_max
thread_count
mempool_metaspace_peak_commited
mempool_ps_survivor_space_commited
memory_non_heap_max
mempool_compressed_class_space_peak_commited
mempool_metaspace_peak_used
mempool_ps_survivor_space_peak_commited
mempool_ps_eden_space_max
gc_ps_scavenge_start_time
mempool_ps_survivor_space_max
gc_ps_marksweep_start_time
mempool_compressed_class_space_peak_init
mempool_ps_survivor_space_peak_init
mempool_metaspace_max
mempool_ps_old_gen_peak_max
memory_non_heap_commited
mempool_ps_old_gen_peak_init
mempool_ps_eden_space_peak_max
mempool_code_cache_peak_used
mempool_compressed_class_space_peak_max
gc_ps_marksweep_end_time
mempool_compressed_class_space_used
process_cpu_time
mempool_ps_eden_space_init
memory_heap_init
mempool_ps_survivor_space_init
mempool_ps_eden_space_peak_init
gc_ps_marksweep_total_collection_time
mempool_code_cache_used
mempool_ps_eden_space_used

Histograms

span_processing_time_seconds

rspace.install

rspace.produce

rspace.consume


replayrspace.install

replayrspace.produce

replayrspace.consume


Metrics names within the RNode software

lmdb-block-store-put

Count of blocks entered into the LMDB block store

lmdb-block-store-get

Count of blocks retrieved from the LMDB block store

lmdb-block-store-find

Count of find operations issued to the LMDB block store

lmdb-block-store-as-map

Deprecated pending casper no longer needing whole DB in memory.

block-store-get

Count of blocks retrieved from the In Memory block store

block-store-as-map

Deprecated pending casper no longer needing whole DB in memory

block-store-find

Count of find operations issues to the In Memory block store

genesis-block-approvals

Only used during testing

protocol-ping-sends

protocol-lookup-send

ping-recv-count

lookup-recv-count

connects

disconnect-recv-count


process-cpu-time

process-cpu-load

memory-heap-commited

memory-heap-init

memory-heap-max

memory-heap-used

memory-non-heap-commited

memory-non-heap-init

memory-non-heap-max

memory-non-heap-used



rnode diagnostics:


List of connected peers:


List of discovered peers:


Node core metrics:
- Ping receivers:
- Lookup receivers:
- Disconnect receivers:
- Connects:
- P2P encryption handshake receivers:
- P2P protocol handshake receivers:
- Peers:
- From:
- To:

Process CPU:
- CPU load: %
- CPU time: ms

Memory usage:
+ Heap memory
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Non-Heap memory
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

Memory pools:
+ Name: Code Cache
+ Type: NON_HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

+ Name: Metaspace
+ Type: NON_HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

+ Name: Compressed Class Space
+ Type: NON_HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

+ Name: PS Eden Space
+ Type: HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

+ Name: PS Survivor Space
+ Type: HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

+ Name: PS Old Gen
+ Type: HEAP
+ Usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes
+ Peak usage:
- committed: bytes
- init: bytes
- max: bytes
- used: bytes

Garbage collectors:
+ Name: PS Scavenge
- Total collections:
- Total collection time: ms
- Current GC start time: ms
- Current GC end time: ms
- Current GC duration: ms

+ Name: PS MarkSweep
- Total collections:
- Total collection time: ms
- Current GC start time: ms
- Current GC end time: ms
- Current GC duration: ms

Threads:
- Threads:
- Daemon threads:
- Peak threads:
- Total started threads: