Design a CI/CD Pipeline System
Problem
Design GitLab's core domain: a CI/CD pipeline system that runs jobs on commits across many projects.
Requirements
Functional:
- Define pipelines (stages, jobs, dependencies) as config
- Schedule jobs on a pool of runners
- Stream logs; report status back to the MR
- Cache artifacts between stages
Non-functional:
- Millions of jobs/day, many self-hosted runners
- Fair scheduling and isolation
- Reliable, resumable
Discussion points
- Job DAG modeling and scheduling
- Runner pool, autoscaling, and isolation
- Artifact/cache storage
- Log streaming and status reporting
- Retries, timeouts, and fairness
added …