• Workflow rules
• Default image
• Default variables
• Stages
• Dependency Proxy
  • Work around for when a pipeline is started by a Project access token user
• Common job definitions
• rules, if: conditions and changes: patterns
  • if: conditions
  • changes: patterns
• Best Practices
  • When to use extends:, <<: *xyz (YAML anchors), or !reference
    • Key takeaways
    • What can extends and YAML anchors do?
      • extends
      • YAML anchors
    • A great example

CI configuration internals

Workflow rules

Pipelines for the GitLab project are created using the workflow:rules keyword feature of the GitLab CI/CD.

Pipelines are always created for the following scenarios:

• main branch, including on schedules, pushes, merges, and so on.
• Merge requests.
• Tags.
• Stable, auto-deploy, and security branches.

Pipeline creation is also affected by the following CI/CD variables:

• If $FORCE_GITLAB_CI is set, pipelines are created.
• If $GITLAB_INTERNAL is not set, pipelines are not created.

No pipeline is created in any other cases (for example, when pushing a branch with no MR for it).

The source of truth for these workflow rules is defined in .gitlab-ci.yml.

Default image

The default image is defined in .gitlab-ci.yml.

It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick.

The images used in our pipelines are configured in the gitlab-org/gitlab-build-images project, which is push-mirrored to gitlab/gitlab-build-images for redundancy.

The current version of the build images can be found in the “Used by GitLab section”.

Default variables

In addition to the predefined CI/CD variables, each pipeline includes default variables defined in .gitlab-ci.yml.

Stages

The current stages are:

• sync: This stage is used to synchronize changes from gitlab-org/gitlab to gitlab-org/gitlab-foss.
• prepare: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages.
• build-images: This stage includes jobs that prepare Docker images that are needed by jobs in subsequent stages or downstream pipelines.
• fixtures: This stage includes jobs that prepare fixtures needed by frontend tests.
• lint: This stage includes linting and static analysis jobs.
• test: This stage includes most of the tests, and DB/migration jobs.
• post-test: This stage includes jobs that build reports or gather data from the test stage’s jobs (for example, coverage, Knapsack metadata, and so on).
• review: This stage includes jobs that build the CNG images, deploy them, and run end-to-end tests against Review Apps (see Review Apps for details). It also includes Docs Review App jobs.
• qa: This stage includes jobs that perform QA tasks against the Review App that is deployed in stage review.
• post-qa: This stage includes jobs that build reports or gather data from the qa stage’s jobs (for example, Review App performance report).
• pages: This stage includes a job that deploys the various reports as GitLab Pages (for example, coverage-ruby, and webpack-report (found at https://gitlab-org.gitlab.io/gitlab/webpack-report/, but there is an issue with the deployment).
• notify: This stage includes jobs that notify various failures to Slack.

Dependency Proxy

Some of the jobs are using images from Docker Hub, where we also use ${GITLAB_DEPENDENCY_PROXY_ADDRESS} as a prefix to the image path, so that we pull images from our Dependency Proxy. By default, this variable is set from the value of ${GITLAB_DEPENDENCY_PROXY}.

${GITLAB_DEPENDENCY_PROXY} is a group CI/CD variable defined in gitlab-org as ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/. This means when we use an image defined as:

image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge

Projects in the gitlab-org group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless ${GITLAB_DEPENDENCY_PROXY} is also defined there.

Work around for when a pipeline is started by a Project access token user

When a pipeline is started by a Project access token user (e.g. the release-tools approver bot user which automatically updates the Gitaly version used in the main project), the Dependency proxy isn’t accessible and the job fails at the Preparing the "docker+machine" executor step. To work around that, we have a special workflow rule, that overrides the ${GITLAB_DEPENDENCY_PROXY_ADDRESS} variable so that Dependency proxy isn’t used in that case:

- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/'
  variables:
    GITLAB_DEPENDENCY_PROXY_ADDRESS: ""

Common job definitions

Most of the jobs extend from a few CI definitions defined in .gitlab/ci/global.gitlab-ci.yml that are scoped to a single configuration keyword.

rules, if: conditions and changes: patterns

We’re using the rules keyword extensively.

All rules definitions are defined in rules.gitlab-ci.yml, then included in individual jobs via extends.

The rules definitions are composed of if: conditions and changes: patterns, which are also defined in rules.gitlab-ci.yml and included in rules definitions via YAML anchors

if: conditions

changes: patterns

Best Practices

When to use extends:, <<: *xyz (YAML anchors), or !reference

Reference

Key takeaways

• If you need to extend a hash, you should use extends
• If you need to extend an array, you’ll need to use !reference, or YAML anchors as last resort
• For more complex cases (e.g. extend hash inside array, extend array inside hash, …), you’ll have to use !reference or YAML anchors

What can extends and YAML anchors do?

extends

• Deep merge for hashes
• NO merge for arrays. It overwrites (source)

YAML anchors

• NO deep merge for hashes, BUT it can be used to extend a hash (see the example below)
• NO merge for arrays, BUT it can be used to extend an array (see the example below)

A great example

This example shows how to extend complex YAML data structures with !reference and YAML anchors:

.strict-ee-only-rules:
  # `rules` is an array of hashes
  rules:
    - if: '$CI_PROJECT_NAME !~ /^gitlab(-ee)?$/ '
      when: never

# `if-security-merge-request` is a hash
.if-security-merge-request: &if-security-merge-request
  if: '$CI_PROJECT_NAMESPACE == "gitlab-org/security"'

# `code-qa-patterns` is an array
.code-qa-patterns: &code-qa-patterns
  - "{package.json,yarn.lock}"
  - ".browserslistrc"
  - "babel.config.js"
  - "jest.config.{base,integration,unit}.js"

.qa:rules:as-if-foss:
  rules:
    # We extend the `rules` array with an array of hashes directly
    - !reference [".strict-ee-only-rules", rules]
    # We extend a single array entry with a hash
    - <<: *if-security-merge-request
      # `changes` is an array, so we pass it an entire array
      changes: *code-qa-patterns

qa:selectors-as-if-foss:
  # We include the rules from .qa:rules:as-if-foss in this job
  extends:
    - .qa:rules:as-if-foss