Introduction

This document describes how to configure network monitoring within the node provided by aemon application.

Configuration

By default monitoring is disabled. To turn it on monitoring.publisher.pubkey and monitoring.active = true need to be setup. To start posting transaction setup monitoring.publisher.autostart to true.

To post transactions account's public and private key is needed. For online processing only public key needs to be configured. Offline post-processing can be done by filtering transaction by publisher public key.

Metrics configuration

Monitoring uses statsd backend provided by apps/aecore/src/aec_metrics.erl. See metrics metrics.* configuration keys in apps/aeutils/priv/aeternity_config_schema.json

Metrics

Each metric uses ae.epoch.aemon. prefix.

Name Type Description
confirmation.delay histogram Number of keyblock created before signing transaction
forks.micro.count counter Count of observed micro-forks
forks.micro.height histogram Height difference of observed micro-forks
gen_stats.microblocks.total histogram Number of microblocks in a generation
gen_stats.tx.monitoring histogram Number of monitoring transactions in a generation
gen_stats.tx.total histogram Number of transactions in a generation
publisher.balance gauge Publisher balance
publisher.post_tx.max_adjustment counter Transaction posting error:
publisher.post_tx.nonce_too_high counter Transaction posting error:
publisher.post_tx.nonce_too_low counter Transaction posting error:
publisher.post_tx.success counter Successful transaction posts
publisher.queue.size histogram Number of transactions posted but not signed on chain
publisher.queue.ttl_expired histogram Number of transactions with expired ttl
block.propagation_time.key histogram Time until key-blocks reached this node in milliseconds
block.propagation_time.micro histogram Time until micro-blocks reached this node in milliseconds
block.time_since_prev.key histogram Time between key-blocks in milliseconds
block.time_since_prev.micro histogram Time between micro-blocks in milliseconds
block.tx.total.micro histogram Number of transactions in a microblock
block.gas.total.micro histogram Gas used per microblock
block.gas.per_tx.micro histogram Gas used per transaction in a microblock
block.size.per_tx.micro histogram Size of transactions in a microblock in bytes
chain.top.difficulty gauge Difficulty of the top block

How to read metrics

confirmation.delay

represents network latency. A high number might imply a busy network or unfair leaders.

forks.micro & gen_stats.microblocks.total

forks.micro represents the length of microfork. It shows a minimum number observed by monitoring, not an exact one. Based on behaviour observed in mainnet in the first half of 2019, roughly 33% of microblocks transactions are rewritten to the next generation. Use gen_stats.microblocks.total as a reference.

gen_stats.microblocks.total and gen_stats.tx.{monitoring,total}

Statistic metrics can be used to measure network saturation

publisher.post_tx.*

Metrics can be used to monitor mempool's transaction propagation. When publisher.post_tx.nonce_too_high is preset you might want to check mempool.nonce_offset configuration/

publisher.queue.*

For further network transaction propagation investigation. All transactions accepted by mempool are tracked by *.size. Over time it should correlate with gen_stats.tx.monitoring.

*.ttl_expired might imply low transaction fee or busy network.