A C daemon and a Go API walk into a cluster: building Symonds

Symonds is my distributed systems monitoring suite: a C daemon that collects real-time metrics on each host, shipping them to a Go REST API backed by SQLite. It currently handles thousands of metric records per minute across 8+ cluster-style groups. These are the design decisions I'd defend — and a couple I'd revisit.

Why C for the collector

A metrics daemon runs on the machines you're worried about. If the host is under memory pressure, the last thing it needs is a collector with a runtime, a garbage collector, and a 50MB resident set.

The C daemon reads /proc directly, keeps a fixed-size buffer, and each 5-second collection cycle costs a few milliseconds of CPU. When the monitored host is unhealthy, the monitor stays cheap — which is exactly when that matters.

Why Go for the API

The API server has the opposite constraints: it handles concurrent ingestion from many daemons, serves dashboard queries, and needs to be easy to extend. That's Go's home turf — one goroutine per ingest connection, net/http, and a static binary to deploy.

The split gives each side the language its job actually wants, at the cost of maintaining a small wire protocol between them. Worth it.

SQLite now, PostgreSQL later — by design

Using SQLite for a metrics store raises eyebrows, but for a single-node API ingesting a few thousand rows a minute, it's honestly great: zero ops, one file, fast writes with WAL mode.

The important decision was writing the schema so PostgreSQL is a migration, not a rewrite:

CREATE TABLE metrics (
    id          INTEGER PRIMARY KEY,   -- BIGSERIAL in PostgreSQL
    host_id     INTEGER NOT NULL REFERENCES hosts(id),
    metric_name TEXT    NOT NULL,
    value       REAL    NOT NULL,
    recorded_at TEXT    NOT NULL       -- ISO 8601; TIMESTAMPTZ later
);
CREATE INDEX idx_metrics_host_time ON metrics (host_id, recorded_at);

The rules: no SQLite-only types, no implicit rowid tricks, timestamps as ISO 8601 strings, and every query goes through a thin storage interface in Go. Swap the driver, run the DDL, done.

The diagnostic endpoint that earned its keep

The feature I use most is a single REST endpoint for cluster-style health inspection: it returns per-host freshness (when did we last hear from this daemon?), ingest lag, and storage stats in one JSON blob.

GET /api/v1/health/cluster

Monitoring systems have a failure mode where they fail silently and you keep believing the last green dashboard. An endpoint whose whole job is "tell me whether the monitoring itself is healthy" turns that from a post-mortem discovery into a curl command.

What I'd revisit

• Batching at the daemon. Right now each cycle ships immediately; buffering a few cycles per request would cut API load further.
• Retention policy in the schema from day one. Time-series data grows forever unless you decide otherwise — deciding late means migrating a table that got big.

Building the whole path — from /proc reads in C to JSON over HTTP in Go — taught me more about observability than any dashboard tool has. If a system is worth running, it's worth knowing when it's lying to you.