Jenkins Sentinel monitors Jenkins build durations and pipeline health by treating each build as a discrete event stored in PostgreSQL. The system provides granular observability into pipeline performance and enables anomaly detection for build duration patterns.
For a detailed technical analysis and architectural decisions, refer to the design document. A demonstration of the system is available in this video.
Traditional Jenkins monitoring focuses on system-level metrics and distributed tracing. Jenkins Sentinel takes a different approach by capturing per-build duration data as time-series events. Each build becomes a database record with complete metadata, enabling complex queries and analysis across multiple dimensions.
The system consists of a Go daemon that polls Jenkins APIs, collects build metrics, and stores them in PostgreSQL. Python-based anomaly detection scripts analyze the collected data to identify unusual patterns in build durations. Grafana dashboards provide visualization of the metrics stored in the database.
The daemon operates in cycles, starting with configuration validation and database health checks. For each configured Jenkins pipeline, it determines the last processed build number and identifies any missing builds. The system fetches details for missing builds from the Jenkins API, handles rate limiting and failures through retry queues, and updates the database with new build records.
Build data includes execution times, queue times, blocked times, and buildable times. The daemon maintains state across restarts, preventing data duplication and enabling recovery from failures. When Jenkins is unavailable, the system retries with exponential backoff until connectivity is restored.
Anomaly detection runs as batch operations where the Go daemon queries recent build metrics and invokes Python detector scripts via subprocess. The daemon constructs JSON input containing time-series data for configured pipelines and passes it to detection algorithms through standard input.
Detector scripts apply statistical algorithms and return anomaly scores through standard output. Results are persisted back to the database with timestamps, scores, and threshold information. This design separates orchestration concerns from numerical computation while maintaining type-safe contracts between components.
Configuration allows per-pipeline customization of detection algorithms, parameters, time windows, and target metrics. Empty configuration disables detection for specific pipelines.
The repository includes Docker Compose configuration for local development with Jenkins, PostgreSQL, and Grafana. Jenkins is pre-configured with jobs defined in the jobs directory. Grafana provisioning automatically configures the PostgreSQL datasource and imports dashboard definitions.
Build the daemon using the provided Makefile. The config.json file defines Jenkins connection details, polling intervals, and pipeline configurations. Database migrations run automatically on daemon startup, creating necessary tables and indexes.
TimescaleDB migration will provide automatic partitioning and improved time-series query performance. Step-level data collection will capture individual stage metrics for deeper pipeline analysis. Multi-Jenkins support will enable unified dashboards across different environments.
LLM-powered failure analysis could offer triggered root cause analysis with natural language query capabilities. Data lifecycle management will automate retention policies and archival to object storage. Performance improvements include parallel build data collection using worker threads.