Infrastructure · Open source
Watchrail
Watchrail is an open-source monitoring platform for teams that need more than a basic uptime check. It combines multi-location monitoring, incident management, maintenance scheduling, secure credentials, notifications, and public status pages in one operational system.
TypeScript / NestJS / Next.js / PostgreSQL / Redis / BullMQ / Docker
Watchrail is an open-source monitoring platform for teams that need more than a basic uptime check. It combines multi-location monitoring, incident management, maintenance scheduling, secure credentials, notifications, and public status pages in one operational system.
Built around real failure modes
A service can respond from one region and fail from another. DNS records can change unexpectedly. Certificates and domains can expire. A background job can stop reporting without exposing an HTTP endpoint.
Watchrail supports HTTP, TCP, DNS, API workflow, heartbeat, and domain-expiry monitors so these different failure modes can be observed through one consistent system.
Monitoring from multiple locations
Checks can run from independently deployed probe locations, including Lagos and Frankfurt. Watchrail combines their results using explicit confirmation rules, helping distinguish an isolated network problem from a service-wide incident.
The execution model is designed to remain deterministic even when regional results arrive late, out of order, or more than once.
From checks to incident response
Watchrail turns monitoring results into an operational workflow rather than a collection of disconnected alerts.
Confirmed failures open incidents with severity and grouping information. Operators can acknowledge, silence, resume, or manually resolve them without changing the service’s measured health. Escalation policies can send reminders through selected notification channels while an incident remains open.
Maintenance windows suppress the appropriate incident and notification behavior without stopping checks or rewriting historical uptime.
Reliable work between systems
Checks, incident transitions, and notifications cross several process boundaries. Watchrail uses transactional outboxes, durable queues, delivery leases, retry policies, and stale-claim fencing to coordinate that work safely.
These mechanisms protect the system from common distributed failure cases: duplicated jobs, delayed results, crashed workers, expired leases, and partially completed operations.
Secure by design
Monitoring credentials are part of the system’s security boundary.
Watchrail includes an encrypted workspace secrets vault with authenticated encryption, tenant-bound cryptographic context, versioned key rotation, and permission-controlled secret use. Sensitive values are resolved only when an authorized execution requires them and are never returned through normal API responses.
Outbound requests also apply network protections at connection time and across redirects.
Notifications without losing delivery history
Incidents can be routed to email, Slack, or signed webhooks. Notification delivery is durable and auditable, with retries, maintenance suppression, channel-specific routing, and immutable configuration versions.
Archiving a channel stops future delivery while preserving its history—an important distinction in an operational system.
Historical evidence, not just current state
Watchrail retains the context behind what changed. This includes check history, incident actions, uptime accounting, notification attempts, maintenance versions, DNS record changes, and domain-expiry observations.
For DNS monitoring, a new record set is recorded only after it is confirmed by separate authoritative rounds. Regional observations are compared deterministically, preventing result ordering from manufacturing false changes.
What I focused on
Building Watchrail has involved designing the boundaries between monitoring, security, persistence, and distributed execution—not only building the interface around them.
The project has given me space to work deeply on:
- Transactional and concurrency-safe data flows
- Multi-tenant authorization and isolation
- Distributed job execution and delivery guarantees
- Versioned configuration and historical correctness
- Secure secret storage and controlled credential resolution
- Incident response and notification workflows
- Responsive operational interfaces
Still being built
Watchrail is under active development. The goal is to create a monitoring system whose behavior remains understandable when failures overlap, results arrive out of order, and several workers act concurrently.
It is as much an exploration of dependable system design as it is a monitoring product.