Foreman: why I built a distributed job queue from scratch
The motivation and stack choices behind building a self-scaling distributed job queue from scratch.
Drawing lines: how I carved Foreman into services
Why three binaries instead of one, Postgres as source of truth, and the transactional outbox seam between Postgres and Kafka.
Exactly-once is a lie, but the outbox pattern gets close
Why you can’t atomically write to Postgres and Kafka, and how the outbox pattern with idempotent consumers gets you exactly-once semantics.
Goroutines, channels, and graceful death
Building a bounded worker pool with priority Kafka dispatch and clean SIGTERM shutdown — no work abandoned, no offsets lost.
Making a distributed system legible: OTel, Prometheus, and the dashboards I actually look at
Structured logs, OpenTelemetry traces across Kafka, Prometheus metrics, and the four Grafana panels that actually matter.
What I'd do differently: a retrospective on Foreman
What’s over-engineered, what I cut corners on, and the one thing I’d change if I started over.