KStats

KStats is meant to collect and deliver various Kaspa node statistics to a StatsD frontend.

KStats has ability to sample various JSON RPC response values at different frequencies and publish the sampled values to StatsD.

Config file

Example config file is available here: https://github.com/kaspanet/kstats/blob/master/config

Due to the fact that this config file is included in the git repository, you should not edit it. Instead, you can create a file names config.local which will be read and it's contents merged with the default config. You can also use config.<hostname> file name, to ensure that only kstats running on the specified hostname will read it.

Query and StatsD paths

Samples are grouped by intervals measured in seconds. A sampler entry is described by a query and a target StatsD sub-path as follows:

"10" : {
	'getInfo.blocks' : 'blocks',
},
"25" : {
	'getNetTotals.totalBytesRecv' : 'bytes_recv',
},

The query is comprised from the RPC method name getInfo followed by the property of the object returned by the RPC call blocks.

The statsd path is comprised of either the gauge name or the gauge name prefixed by sample attributes (right now T: is the only prefix that results in the value to become a 'difference per-second' measurement).

StatsD paths are formed as follows:

[Metrics].stats.gauges.hosts.<hostname>.nodes.<type>.<identifier>.<property>

Were:

• <hostname> is the hostname of the computer running KStats (or the hostname override set via --hostname= argument).
• <type> is the type of node as defined by a Supervisor-generated topology workspace. In KStats, the type defaults to kaspa
• <identifier> user-given name of the node or an auto-generated name based on the ip and port available in the node's peer information.
• <property> the sampler property (in the above example, this would be blocks)

Command line arguments

• --rpc=<address> provides ability to specify rpc endpoints to kstats via command line (also helpful when starting a docker container, as this allows you to specify target nodes(s) via docker run command line arguments)

• --rpc-cert=<filename> allows to manually specify rpc certificate file (or a certificate authority chain) in PEM format.

• --tls-reject-unauthorized=[true|false] enable/disable SSL connections to self-signed certificates (deault false).

• --supervisor=<address> address of the supervisor.

• --statsd=<address> address of statsd server. You can supply multiple endpoints by specifying multiple --statsd arguments.

• --dogstatsd=<address> address of DogStatsD server. Data delivered to this server will not contain full datastore path but instead a custom root (node info will be published under kaspa root and system metrics under kstats root).

• --dogstatsd-root=<path> root object name (or a dot delimited path) for DogStatsD submissions (default: kaspa).

• --hostname=<hostname> overrides default OS hostname (useful when running under docker and you want to identify the container in a specific manner)

• --system as of v0.5.0 kstats local system metrics are not collected by default as it is being frequently ran in a docker. Use this flag to enable. (please note that in KHost system metric collection is enabled by default).

• --docker runs KStats in docker mode (used by Dockerfile). When in docker mode, KStats uses /etc/kstats/config as the primary config file and disables CPU, Process, Storage and Network bandwidth monitoring.

Other flags used in the underlying frameworks and modules:

• --verbose dumps misc. information such as connection attempts and statsd variable dispatch.
• --test dumps StatsD updates in console but does not dispatch them to the StatsD server.
• --mute (khost only - mutes daemon console output)

Host and application monitoring

NOTE: the following functionality is currently available only via KHost applications.

System specific monitoring is capable of providing the following metrics:

• Overall host CPU usage
• Main process CPU usage
• Main process threads
• Main process memory usage
• Main process I/O
• Overall host storage usage (bytes)
• datadir usage (bytes)
• Network tx/rx (bytes)

Main process refers to the main executable, such as kaspad or mysql.

Running in Docker

IMPORTANT:

• Make sure that RPC addresses contain the username and password for the RPC connection.
• By default there are no RPC endpoints configured. You must either run the docker image once and then edit the created config file in /etc/kstats or use --rpc command line arguments.

# build/rebuild
sudo docker build -t kstats

# build/clean && build
sudo docker build --no-cache -t kstats

# run and connect rpc to x.x.x.x:port
sudo docker run -d kstats --rpc=user:pass@x.x.x.x:port

# monitor multiple nodes
sudo docker run -d kstats --rpc=user:pass@x.x.x.x:port --rpc=user:pass@y.y.y.y:port

# mount config folder to /etc/kstats
sudo docker run -d -v /etc/kstats:/etc/kstats kstats --rpc=user:pass@x.x.x.x:port

Docker-specific flags:

• -v maps /etc/kstats in the container to /etc/kstats in your host (if you run multiple kstats instances, they can share the same config file. While sharing the same folder, you can create config.<hostname> file for per-host configuration).