ADR-006: Metrics And Logs

• ADR
• ADR-006: Metrics And Logs

​

Status​

​

Context​

​

Implementation​

1. Implement log aggregation.
2. Add metrics for the number of incoming and outgoing transactions.
3. Add metrics for total vault supply.
4. Add metrics for total Cosmos-represented asset supply.
5. Calculate and monitor the ratio between vault and Cosmos asset supplies.
6. Monitor incoming and outgoing transaction volume.
7. Monitor node status, including health status and current height.
8. Implement TSS metrics.

​

Steps to implement​

1. Determine how to scrape metrics from nodes: 71.
2. Set up metrics storage: 72.
3. Research methods to expose and modify Cosmos-based metrics: 73.
4. Add business and infrastructure metrics for the Int3face node: 74.
5. Add business and infrastructure metrics for the Observer node: 75.
6. Deploy the metrics infrastructure to Hetzner: 76.
7. Set up a metrics dashboard using the reference template: 77.

​

Solution​

1. Implemented monitoring based on Prometheus and Grafana.
2. Prometheus and Grafana are deployed by docker-compose.
3. Also running as a demon:
   1. node-exporter to collect node metrics.
   2. cosmos-exporter to collect Cosmos metrics.
4. Monitoring repository: int3face-monitoring contains:
   1. Prometheus & Grafana configs
   2. Deployment scripts
   3. Docker-compose file
   4. Dashboards
5. Grafana dashboard (Hetzner): int3face-monitoring
   1. Login: admin
   2. Password: {please ask from a team member}
6. Prometheus (Hetzner): prometheus-ui

Architecture:

​

Metrics overview​

​

Observer Node Metrics​

[monitoring]  # The flag to enable the metrics server.  enabled = true  # The port to expose the metrics on.  port = 27727

​

Int3face Node Metrics​

##########################################################       Instrumentation Configuration Options     ##########################################################[instrumentation]# When true, Prometheus metrics are served under /metrics on# PrometheusListenAddr.# Check out the documentation for the list of available metrics.prometheus = true# Address to listen for Prometheus collector(s) connectionsprometheus_listen_addr = ":26660"# Maximum number of simultaneous connections.# If you want to accept a larger number than the default, make sure# you increase your OS limits.# 0 - unlimited.max_open_connections = 3# Instrumentation namespacenamespace = "cometbft"

• Consensus metrics: link to metrics
• Cosmos-exporter metrics (all the metrics provided by cosmos-exporter have the following prefixes):
  • cosmosvalidator* - metrics related to a single validator
  • cosmosvalidators* - metrics related to a validator set
  • cosmoswallet* - metrics related to a single wallet

​

Node Exporter Metrics​

• Node exporter metrics: link to metrics