Shohan Dutta Roy
Writing

Foreman

Building a distributed job queue from scratch in Go.

6 parts – Source on GitHub

Go · Kafka · Postgres · Redis · OpenTelemetry

  1. Part 1 Foreman: why I built a distributed job queue from scratch The motivation and stack choices behind building a self-scaling distributed job queue from scratch.
  2. Part 2 Drawing lines: how I carved Foreman into services Why three binaries instead of one, Postgres as source of truth, and the transactional outbox seam between Postgres and Kafka.
  3. Part 3 Exactly-once is a lie, but the outbox pattern gets close Why you can’t atomically write to Postgres and Kafka, and how the outbox pattern with idempotent consumers gets you exactly-once semantics.
  4. Part 4 Goroutines, channels, and graceful death Building a bounded worker pool with priority Kafka dispatch and clean SIGTERM shutdown — no work abandoned, no offsets lost.
  5. Part 5 Making a distributed system legible: OTel, Prometheus, and the dashboards I actually look at Structured logs, OpenTelemetry traces across Kafka, Prometheus metrics, and the four Grafana panels that actually matter.
  6. Part 6 What I'd do differently: a retrospective on Foreman What’s over-engineered, what I cut corners on, and the one thing I’d change if I started over.