Host network observability
This page documents the host-level network pressure signals and the committee connectivity gauges that issue #1688 asks for. It covers collection, scope, thresholds, alerts, operator actions and the label policy.
Scope
The node exports consensus and networking metrics on --metrics. It does not see what the kernel or the firewall drops before a packet reaches the QUIC listener. This page adds three host signals and one node signal:
- bounded firewall rule counters from named nft counters;
- UDP receive buffer errors from the kernel SNMP counters;
- conntrack occupancy against the configured maximum, with an explicit mode;
- unique reachable current-committee peers against the committee size, per network.
Every query on the dashboard and in the rules is scoped by job and instance. Node series add network. Interface series add device. No label carries an address, a peer identity, a URL or an error string.
Collection
node_exporter
Run one node_exporter per host in the host network namespace. The compose profile runs prom/node-exporter:v1.12.1 with these flags:
--path.rootfs=/host
--collector.textfile.directory=/textfile
--web.listen-address=127.0.0.1:9100
Keep the exporter on loopback or on the management network. Do not expose port 9100 to the public internet.
Textfile collector
etc/host-metrics/tn-host-network.sh writes the host series the exporter cannot produce itself. Install the script and its units:
sudo install -m 0755 etc/host-metrics/tn-host-network.sh /usr/local/bin/tn-host-network.sh
sudo install -m 0644 etc/host-metrics/tn-host-network.service etc/host-metrics/tn-host-network.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now tn-host-network.timer
The timer runs the collector 30 seconds after boot and every 15 seconds after that. The service writes tn_host_network.prom.<pid> under /var/lib/node_exporter/textfile and renames it into place, so the exporter never reads a partial file. The script needs root for nft. It needs nft and jq for the firewall series. Without them it reports tn_host_firewall_collector_available 0 and no firewall series.
Name the nft counters you want to observe with the tn_ prefix. Use tn_drop_ for drop rules and tn_accept_ for accept rules:
nft add counter inet filter tn_drop_public_udp
nft add rule inet filter input udp dport 49590 ip saddr != @validators counter name tn_drop_public_udp drop
The validators set
The drop rule above admits only the sources in the nft set validators.
Fill it from the node’s own view: tn_admissionSnapshot on the node’s loopback RPC lists, for every swarm, the committee window, the committee two epochs ahead, the eligible validator pool and the configured trusted and bootstrap peers, with the addresses the node would use for each.
A set built from it opens to a validator once it is eligible and its record resolves, and closes to a departed one once the node stops listing it.
The set leads the node’s own admission on purpose. Under Closed the node refuses an inbound peer it does not admit only after the QUIC/TLS handshake, because the PeerId is known only then, and the peer’s own dial can complete before the connection closes. Its own outbound dials to such a peer are refused before any transport work. The firewall is therefore what spares the host the handshake of a source outside the node’s view, and since it drops a peer before the node can see it, it must already pass every scheduled and eligible validator by the time the node admits one.
Build the set from three sources:
- the IP literal of every
/ip4/and/ip6/address inmultiaddrsandrecord_multiaddrsof every entry of every swarm, whatever its slots,sourceoradmitted; skipdnsaddresses rather than resolving them; - the operator’s static allowlist;
- the configured bootstrap peers.
The static list and the bootstrap peers are always in the set, so a wrong snapshot can never cut the launch set and the set is never empty.
IPv4 and IPv6 addresses need separate sets, one of type ipv4_addr matched by ip saddr and one of type ipv6_addr matched by ip6 saddr.
Replace the elements in one nft -f transaction, so the kernel applies the new set whole or not at all:
flush set inet filter validators
add element inet filter validators { 198.51.100.7, 203.0.113.10 }
Only an authoritative snapshot, as defined below, may run that transaction.
Fail-open rule
The set never shrinks on bad input. A snapshot is authoritative for membership only when every one of these holds:
- The RPC answered with a result, and its
versionis one the reader knows (1). - The primary swarm (
"network": "primary") reportsavailable: trueand a non-nulladmission. - The primary’s
tracked_epochis non-null and not lower than itsadmission.epoch. A lower value means the node has not published this epoch’s validator pool, and the swarm still lists the previous epoch’s keys. - The primary’s
tracked_epochis not lower than that of the last applied snapshot. - The primary’s
admission.fallbackisnullor"Unresolved".
Missing and Stale mean that no versioned committee window has arrived, for example right after a restart, when the node runs Open with the current committee only.
Contradictory means that membership itself is in doubt.
The effective admission mode does not matter: the firewall is a cost filter, independent of the node’s own gate.
When the snapshot is authoritative, the addresses of a key it no longer lists leave the set at that apply, unless the static list or the bootstrap peers still hold them.
A worker swarm that reports available: false, a null admission, or a tracked_epoch behind its own admission.epoch is not authoritative for its own entries: keep the addresses last applied for it and add any new ones it lists.
When the snapshot is not authoritative, change nothing.
That covers an unreachable RPC, a node without the method (JSON-RPC error -32601), the error -32001 (Not Found.) of a node whose networks have not started, an unknown version, and any failed check above.
Keep the loaded set, and record since when and why the feed is stale.
Never flush the set on an error path.
A key listed with "source": "unresolved" has no addresses.
Keep the addresses it last resolved to for 3 epochs after its last resolution, then let it contribute nothing until it resolves again.
That absorbs a transient kad miss without pinning a departed peer.
Age policy
- Poll the snapshot every 5 minutes and after every change to the static list. The node serves a cached snapshot for up to 1 second, so callers cannot load its swarms.
- The snapshot carries no generation time. The feed’s age is the time since the last authoritative snapshot was applied.
- Warn when the feed has been stale for more than 2 epochs, and keep the set: the last good view is still a superset of what the node admitted then.
- Alert at once on a missing method or an unknown
version, because neither clears by itself. - Do not expire an entry on its
record_timestamp. The node lists a record until a newer one supersedes it or the key leaves its sets, and it looks up again every tracked record signed more than two record publication intervals ago (24 hours by default).
Compose profile
etc/compose.yaml has an observability profile with node_exporter, Prometheus (prom/prometheus:v3.15.0) and Grafana (grafana/grafana:13.2.4). The profile is off by default, so make up is unchanged. Start it with:
GF_SECURITY_ADMIN_PASSWORD=change-me docker compose -f etc/compose.yaml --profile observability up -d
All three services bind to loopback: Prometheus on 9090, Grafana on 3000 and node_exporter on 9100.
Grafana has anonymous access off.
The admin password comes from GF_SECURITY_ADMIN_PASSWORD; the default telcoin-local is for a local stack only.
Grafana provisions two dashboards: “Telcoin Node Metrics” (etc/grafana/telcoin-node-metrics.json) and “Telcoin Worker Gateway” (bin/worker-gateway/grafana-worker-gateway.json), whose job variable scopes every panel to the jobs that scrape gateways.
Each validator, the observer, each worker gateway and node-record-api binds its metrics listener to its fixed bridge address (10.10.0.x:9101), so the host Prometheus can scrape it on a Linux host.
The ports are not published.
With every service healthy, Prometheus reports up at 1 for all eleven targets in the table below.
Prometheus jobs
| Job | Targets | Instance label | Series |
|---|---|---|---|
telcoin-node | 10.10.0.21 to 10.10.0.24 and 10.10.0.31, port 9101 | validator1 to validator4, observer | tn_* node series |
worker-gateway | 10.10.0.41 to 10.10.0.44, port 9101 | worker-gateway1 to worker-gateway4 | tn_worker_gateway_*, tn_info and reth_process_* |
node-record-api | 10.10.0.51:9101 | node-record-api | tn_node_record_api_* |
node | 127.0.0.1:9100 | the host | node_*, up and the tn_host_* textfile series |
Rules load from etc/prometheus/rules/*.rules.yaml. The scrape and evaluation interval is 15 seconds.
The observer is not a committee member, so it reports tn_network_committee_peers_expected 0 and the committee alerts skip it; TnNodeDown and TnNodeCommitteeSeriesAbsent cover it like a validator.
No rule matches the worker-gateway or node-record-api jobs.
The gateway’s own series are described in bin/worker-gateway/README.md (“Observability”), and those of node-record-api in bin/node-record-api/README.md.
Series reference
| Series | Source | Scope | Reset behaviour |
|---|---|---|---|
tn_host_firewall_rule_packets_total{rule} | nft named counter | host, per rule | resets on reboot or nft reset counters; use rate or increase |
tn_host_firewall_rule_bytes_total{rule} | nft named counter | host, per rule | same as packets |
tn_host_firewall_rules_truncated | collector | host | gauge; 1 when more than 32 tn_ counters exist and the collector keeps only the first 32 in name order |
tn_host_firewall_collector_available | collector | host | gauge; 0 when nft or jq is missing or nft -j list counters fails |
tn_host_conntrack_mode{mode} | collector | host | one-hot gauge: tracked, notrack or unsupported |
tn_host_udp_rmem_max_bytes | /proc/sys/net/core/rmem_max | host | gauge |
tn_host_network_collector_info{version} | collector | host | gauge; collector version |
node_netstat_Udp_RcvbufErrors, node_netstat_Udp6_RcvbufErrors | kernel SNMP | host | resets on reboot; use rate |
node_nf_conntrack_entries, node_nf_conntrack_entries_limit | node_exporter conntrack collector | host | gauges; absent when the module is not loaded |
node_network_receive_drop_total{device}, node_network_receive_errs_total{device} | node_exporter netdev collector | host, per interface | resets on reboot; use rate |
node_textfile_mtime_seconds{file} | node_exporter textfile collector | host | gauge; collector freshness. file is the path node_exporter reads, for example /textfile/tn_host_network.prom, so the rules match file=~"(.*/)?tn_host_network[.]prom" |
up{job="node"} | Prometheus | host | 0 when the exporter does not answer |
up{job="telcoin-node"} | Prometheus | node | 0 when the node metrics endpoint does not answer |
tn_network_committee_size{network} | node | node, per network | gauge; 0 until the first committee is known |
tn_network_committee_peers_expected{network} | node | node, per network | gauge; 0 until the first committee is known, and 0 on a node outside the current committee |
tn_network_committee_peers_connected{network} | node | node, per network | gauge; 0 until the first committee is known |
Recording rules keep the baselines operators use for threshold selection:
| Recording rule | Meaning |
|---|---|
tn_host:firewall_drop_packets:rate5m | packets per second across all drop_* counters, per host |
tn_host:udp_rcvbuf_errors:rate5m | UDP receive buffer errors per second, IPv4 and IPv6, per host |
tn_host:conntrack_occupancy:ratio | entries divided by the limit, per host, in every conntrack mode |
tn_network:committee_coverage:ratio | connected divided by expected committee peers, only while a committee is known |
Scope caveats
Counter reset
nft counters and the kernel SNMP counters reset on reboot. nft counters also reset on nft reset counters. All rules and panels use rate or increase. Both treat a drop in a counter as a reset and do not alert on the reset alone. The unit test file telcoin-host-network.test.yaml proves it with a counter that resets in the middle of the window.
Network namespace
The collector and node_exporter read the host network namespace. A container that runs them in its own namespace reports the container, not the host. The compose profile uses network_mode: host and pid: host for this reason. A node in a container still shares the host firewall and conntrack table, so the host series describe the pressure the node sees.
Interface
Interface drops and errors are per device. The device variable of the dashboard selects the interface. Pick the interface that carries consensus traffic. The firewall, UDP and conntrack series are host wide and have no interface label.
Exporter scope
One exporter observes one host. Several validators on one host share the same host series under one instance of the node job. The node series keep their own instance under the telcoin-node job. Nothing merges across instances: every rule keeps instance in its result and every panel legend shows it.
Explicit states
A missing signal is never a healthy zero. The state panel of the dashboard shows one of exporter-down, collector-absent, stale, firewall-unavailable or ok for the collector, and one of unsupported, collector-failed, notrack or tracked for conntrack. A host without the conntrack module shows unsupported while the textfile is present and collector-failed without it, because the node_exporter conntrack collector also fails there. An empty drop panel next to a collector-absent state means no data, not no drops.
notrack: at least one nft rule has anotrackstatement. Flows that match it do not enter the conntrack table; every other flow is still tracked.TnHostConntrackHighstays active in this mode, because the tracked flows can still fill the table.unsupported:/proc/sys/net/netfilter/nf_conntrack_maxis absent. The module is not loaded or the kernel has no conntrack.TnHostConntrackUnsupportedfires after 15 minutes so the state is reviewed, not ignored.- The collector reads the ruleset with
nft -j list rulesetandjqand counts only anotrackstatement inside a rule. A counter, chain or comment that is only namednotrackdoes not count. Withoutnftorjqthe mode istracked. A host that uses iptablesNOTRACKtargets showstracked. Document the exception for such hosts.
A notrack rule works only before conntrack sees the packet. Put it in a chain on the prerouting hook, and on the output hook for traffic the host sends, at priority raw (-300). Conntrack runs at priority -200, so a notrack rule in an input chain at priority 0 is too late and exempts nothing. Example for the primary and worker UDP ports:
table inet raw {
chain prerouting {
type filter hook prerouting priority raw;
udp dport { <primary>, <worker> } notrack
}
chain output {
type filter hook output priority raw;
udp sport { <primary>, <worker> } notrack
}
}
Committee connectivity semantics
The node sets three gauges per network (primary and worker-{id}):
tn_network_committee_size: the number of members of the current committee.tn_network_committee_peers_expected:size - 1when this node is a current member, else0. Every network is a full mesh between committee members, so a member expects every other member. A node outside the current committee (an observer, or a validator outside the committee of this epoch) exports 0, so both committee alerts skip it;tn_network_committee_sizeandtn_network_committee_peers_connectedstay exported for the dashboard. The node recomputes membership from its own BLS key on every committee update.tn_network_committee_peers_connected: the number of unique current-committee members with at least one established connection on this network.
All three are 0 until the node learns its first committee. The rules use expected > 0 as the restart guard, so a node that boots fires nothing until the committee is known. At an epoch change the previous committee is not counted: a member that leaves the committee drops out of connected without lowering coverage. A node outside the current committee also exports expected 0, so the guard skips it too.
The generic tn_network_connected_peers and tn_network_known_peers gauges count every peer. The dashboard overlays connected_peers on the committee panel only to show that committee coverage moves on its own. The unit test has connected_peers rising while committee coverage falls, and the alerts still fire.
Alerts
Every alert has a severity label (info, warning or critical), a summary, a description, an action and a runbook_url that points at a section of this page. Every expression carries a job matcher.
The thresholds below are launch defaults. To re-derive one, record the matching recording rule for a quiet launch window of at least one week, take the 99th percentile of that baseline, and add a margin of at least 50 percent. Keep the for duration at or above two scrape intervals so a single scrape cannot fire an alert.
Exporter and collector state
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnHostExporterDown | up{job="node"} == 0 for 2 minutes | critical | Restart node_exporter. Check the loopback bind and the scrape target. |
TnHostCollectorAbsent | exporter up without tn_host_network.prom for 10 minutes | warning | Install the collector script and enable the timer. Check the textfile directory flag. |
TnHostCollectorStale | file older than 120 seconds for 5 minutes | warning | Read journalctl -u tn-host-network.service. Fix nft, jq or permissions. |
Node state
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnNodeDown | up{job="telcoin-node"} == 0 for 2 minutes | critical | Check the node process and its logs. Check the metrics port and the scrape target. |
TnNodeCommitteeSeriesAbsent | node up without tn_network_committee_peers_expected for 15 minutes | warning | Confirm that the node release exports the committee gauges and that the scrape target is the node metrics endpoint. |
The committee alerts read the node gauges, so they cannot fire for a node that is down or does not export them. These two alerts cover that gap.
They cover every target of the telcoin-node job, the compose profile’s observer included.
Firewall drops
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnHostFirewallDropsRising | increase of a drop_* counter over 5 minutes above 100, for 5 minutes | warning | Compare with the QUIC Retry and inbound request panels. A rise that matches committee loss means a blocked peer. A rise with stable committee coverage is a scan or flood the firewall absorbs. |
TnHostFirewallCollectorUnavailable | tn_host_firewall_collector_available is 0 for 10 minutes | warning | Install nftables and jq. Run nft -j list counters as root to see the error. The drop panel is empty, not zero, while this fires. |
TnHostFirewallRulesTruncated | more than 32 tn_ counters for 10 minutes | warning | Remove or merge tn_ counters until at most 32 remain. The collector exports only the first 32 in name order. |
Launch default: 100 packets per 5 minutes. Re-derive it from tn_host:firewall_drop_packets:rate5m multiplied by 300.
UDP receive buffer errors
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnHostUdpRcvbufErrors | increase of RcvbufErrors over 5 minutes above 0, for 10 minutes | warning | Compare tn_host_udp_rmem_max_bytes with the buffer the node requests. Raise net.core.rmem_max and net.core.rmem_default. Check CPU starvation of the node. |
Launch default: any sustained error. A host that shows a constant low rate at launch gets its own threshold from tn_host:udp_rcvbuf_errors:rate5m.
Conntrack occupancy
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnHostConntrackHigh | entries divided by limit above 0.8 for 5 minutes, in every conntrack mode | warning | Raise net.netfilter.nf_conntrack_max, block the flow churn at the edge, or add notrack rules for the consensus UDP ports at priority raw (see Explicit states). The alert stays active with notrack, because other flows are still tracked. |
TnHostConntrackUnsupported | mode unsupported, or the exporter’s conntrack collector fails, for 15 minutes | info | Load nf_conntrack or enable the collector. If the host runs without conntrack on purpose, document it and silence the alert for that instance. |
Launch default: 0.8. Re-derive it from tn_host:conntrack_occupancy:ratio. The ratio and the alert ignore the conntrack mode, because notrack exempts only the flows it matches.
Committee connectivity
| Alert | Condition | Severity | Action |
|---|---|---|---|
TnCommitteeConnectivityDegraded | expected above 0 and connected below expected, for 5 minutes | warning | Find the missing members from their own gauges. Check firewall drops and QUIC Retry on both sides. Confirm the address manifest. |
TnCommitteeConnectivityLost | expected above 0 and connected below two thirds of expected, for 2 minutes | critical | Treat as an incident. Check host drops, conntrack and UDP errors on this host. Confirm the advertised addresses. Fail over to a sentry path if the private network is impaired. |
Launch default: two thirds. Re-derive it from tn_network:committee_coverage:ratio. A committee of four tolerates one missing member before quorum is at risk.
Controlled scenarios
Run these on a staging host to confirm the signals before launch. The host-scenarios job of .github/workflows/host-network-observability.yaml runs the first four on every pull request that touches the artifacts, checks that node_exporter labels the textfile with its full path, then checks the collector output with promtool check metrics.
| Scenario | Procedure | Expected observation | CI |
|---|---|---|---|
| Firewall drops | Add a tn_drop_* counter and a drop rule for a probe port. Send 50 datagrams to it. Run the collector. | tn_host_firewall_rule_packets_total{rule="drop_<name>"} is at least 50. TnHostFirewallDropsRising fires above the threshold. | yes |
| notrack | Add a notrack rule for the probe port in a prerouting chain at priority raw. Run the collector. | tn_host_conntrack_mode{mode="notrack"} is 1. A chain or a comment that is only named notrack leaves the mode tracked. TnHostConntrackHigh still fires if the tracked flows fill the table. | yes |
| UDP receive buffer pressure | Set net.core.rmem_max and net.core.rmem_default to 2304. Start a UDP listener that does not read. Send 300 datagrams of 1000 bytes. | The RcvbufErrors column of /proc/net/snmp increases. TnHostUdpRcvbufErrors fires after 10 minutes. | yes |
| Conntrack pressure | Set net.netfilter.nf_conntrack_max to 64. Open more than 64 UDP flows. When the table is full, the kernel drops each new flow and the sender gets EPERM. This is the expected result, not an error. | nf_conntrack_count reaches 80 percent of the limit. TnHostConntrackHigh fires after 5 minutes above the threshold. | yes |
| Committee loss and recovery | Block one committee member’s address on the primary port. Wait. Unblock it. | tn_network_committee_peers_connected falls by one while tn_network_connected_peers is unchanged or rising. TnCommitteeConnectivityDegraded fires after 5 minutes and clears after the unblock. | no (needs a committee) |
The unit test file etc/prometheus/rules/telcoin-host-network.test.yaml covers the rule side of every scenario: a counter reset, an absent exporter, several instances without merging, a notrack host, an unsupported host, a restart with zero gauges, and committee loss with recovery.
Label vocabulary policy
- Host series carry only
job,instance,rule,mode,device,file,collectorandversion. Node committee series carry onlynetwork. rulevalues come from nft counter names with thetn_prefix. The collector sorts them and keeps the first 32.tn_host_firewall_rules_truncatedis 1 when it cut the list. An attacker cannot create nft counters, so the operator bounds the vocabulary.modehas three values.networkhasprimaryandworker-{id}.- Committee connectivity is a count. There is no per-peer label, not even for committee members. A per-peer breakdown needs a bounded lifecycle for the label values and is out of scope here.
- No host series or node committee series carries a source address, a peer identity, a URL or an error text. The node’s only per-peer label is on series this page does not cover; Metric label policy describes it. The artifact test in
crates/network-libp2p/src/tests/ops_artifacts_tests.rsrejects any dashboard query that matches onaddr,address,ip,src,peer_idorurl, and any node or host query withoutjobandinstancematchers.
Related
- Issue #930: the scoping and label policy on this page applies to the work tracked there.
bin/worker-gateway/README.md(“Observability”): the gateway’s series and the “Telcoin Worker Gateway” dashboard, which the compose profile provisions.etc/README.md(“Docker Compose”): the compose testnet’s services, endpoints and the checks of its gateway path.- Validator production operations: firewall configuration and capacity monitoring.
- tn_admissionSnapshot: the admission view that fills the
validatorsset.