This document is a work-in-progress and represents a very early state of the Pods design. Significant aspects are not documented, though we expect to add them in the future. This is one possible architecture for Pods, and we intend to contrast this with alternatives before deciding which approach to implement. This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach.

Pods: CI Runners

GitLab in order to execute CI jobs GitLab Runner, very often managed by customer in their infrastructure.

All CI jobs created as part of CI pipeline are run in a context of project it poses a challenge how to manage GitLab Runners.

1. Definition

There are 3 different types of runners:

instance-wide: runners that are registered globally with specific tags (selection criteria)
group runners: runners that execute jobs from a given top-level group or subprojects of that group
project runners: runners that execute jobs from projects or many projects: some runners might have projects assigned from projects in different top-level groups.

This alongside with existing data structure where ci_runners is a table describing all types of runners poses a challenge how the ci_runners should be managed in a Pods environment.

2. Data flow

GitLab Runners use a set of globally scoped endpoints to:

registration of a new runner via registration token https://gitlab.com/api/v4/runners (subject for removal) (registration token)
requests jobs via an authenticated https://gitlab.com/api/v4/jobs/request endpoint (runner token)
upload job status via https://gitlab.com/api/v4/jobs/:job_id (build token)
upload trace via https://gitlab.com/api/v4/jobs/:job_id/trace (build token)
download and upload artifacts via https://gitlab.com/api/v4/jobs/:job_id/artifacts (build token)

Currently three types of authentication tokens are used:

runner registration token (subject for removal)
runner token representing an registered runner in a system with specific configuration (tags, locked, etc.)
build token representing an ephemeral token giving a limited access to updating a specific job, uploading artifacts, downloading dependent artifacts, downloading and uploading container registry images

Each of those endpoints do receive an authentication token via header (JOB-TOKEN for /trace) or body parameter (token all other endpoints).

Since the CI pipeline would be created in a context of a specific Pod it would be required that pick of a build would have to be processed by that particular Pod. This requires that build picking depending on a solution would have to be either:

routed to correct Pod for a first time
be made to be two phase: request build from global pool, claim build on a specific Pod using a Pod specific URL

3. Proposal

This section describes various proposals. Reader should consider that those proposals do describe solutions for different problems. Many or some aspects of those proposals might be the solution to the stated problem.

3.1. Authentication tokens

Even though the paths for CI Runners are not routable they can be made routable with those two possible solutions:

The https://gitlab.com/api/v4/jobs/request uses a long polling mechanism with a ticketing mechanism (based on X-GitLab-Last-Update header). Runner when first starts sends a request to GitLab to which GitLab responds with either a build to pick by runner. This value is completely controlled by GitLab. This allows GitLab to use JWT or any other means to encode pod identifier that could be easily decodable by Router.
The majority of communication (in terms of volume) is using build token making it the easiest target to change since GitLab is sole owner of the token that Runner later uses for specific job. There were prior discussions about not storing build token but rather using JWT token with defined scopes. Such token could encode the pod to which router could easily route all requests.

3.2. Request body

The most of used endpoints pass authentication token in request body. It might be desired to use HTTP Headers as an easier way to access this information by Router without a need to proxy requests.

3.3. Instance-wide are Pod local

We can pick a design where all runners are always registered and local to a given Pod:

Each Pod has it’s own set of instance-wide runners that are updated at it’s own pace
The project runners can only be linked to projects from the same organization creating strong isolation.
In this model the ci_runners table is local to the Pod.
In this model we would require the above endpoints to be scoped to a Pod in some way or made routable. It might be via prefixing them, adding additional Pod parameter, or providing much more robust way to decode runner token and match it to Pod.
If routable token is used, we could move away from cryptographic random stored in database to rather prefer to use JWT tokens that would encode
The Admin Area showing registered Runners would have to be scoped to a Pod

This model might be desired since it provides strong isolation guarantees. This model does significantly increase maintenance overhead since each Pod is managed separately.

This model may require adjustments to runner tags feature so that projects have consistent runner experience across pods.

3.4. Instance-wide are cluster-wide

Contrary to proposal where all runners are Pod local, we can consider that runners are global, or just instance-wide runners are global.

However, this requires significant overhaul of system and to change the following aspects:

ci_runners table would likely have to be split decomposed into ci_instance_runners, …
all interfaces would have to be adopted to use correct table
build queuing would have to be reworked to be two phase where each Pod would know of all pending and running builds, but the actual claim of a build would happen against a Pod containing data
likely ci_pending_builds and ci_running_builds would have to be made cluster-wide tables increasing likelihood of creating hotspots in a system related to CI queueing

This model makes it complex to implement from engineering side. Does make some data being shared between Pods. Creates hotspots / scalability issues in a system (ex. during abuse) that might impact experience of organizations on other Pods.

3.5. GitLab CI Daemon

Another potential solution to explore is to have a dedicated service responsible for builds queueing owning it’s database and working in a model of either sharded or podded service. There were prior discussions about CI/CD Daemon.

If the service would be sharded:

depending on a model if runners are cluster-wide or pod-local this service would have to fetch data from all Pods
if the sharded service would be used we could adapt a model of either sharing database containing ci_pending_builds/ci_running_builds with the service
if the sharded service would be used we could consider a push model where each Pod pushes to CI/CD Daemon builds that should be picked by Runner
the sharded service would be aware which Pod is responsible for processing the given build and could route processing requests to designated Pod

If the service would be podded:

all expectations of routable endpoints are still valid

In general usage of CI Daemon does not help significantly with the stated problem. However, this offers a few upsides related to more efficient processing and decoupling model: push model and it opens a way to offer stateful communication with GitLab Runners (ex. gRPC or Websockets).

4. Evaluation

Considering all solutions it appears that solution giving the most promise is:

use “instance-wide are Pod local”
refine endpoints to have routable identities (either via specific paths, or better tokens)

Other potential upsides is to get rid of ci_builds.token and rather use a JWT token that can much better and easier encode wider set of scopes allowed by CI runner.