• Prerequisites
  • kubectl
  • Helm
  • PostgreSQL
  • Redis
• Decide on other options
  • Secrets
  • Networking and DNS
    • Dynamic IP addresses with external-dns
    • Static IP addresses
    • Compatibility with Istio protocol selection
  • Persistence
  • TLS certificates
  • Prometheus
    • Configure Prometheus to scrape TLS-enabled endpoints
    • Exporter endpoints
  • Outgoing email
  • Incoming email
  • Service Desk email
  • RBAC
• Next steps

GitLab chart prerequisites

Before you deploy GitLab in a Kubernetes cluster, install the following prerequisites and decide on the options you’ll use when you install.

Prerequisites

kubectl

Install kubectl 1.16 or later by following the Kubernetes documentation. The version you install must be within one minor release of the version running in your cluster.

Helm

Install Helm v3.3.1 or later by following the Helm documentation.

PostgreSQL

By default, the GitLab chart includes an in-cluster PostgreSQL deployment that is provided by bitnami/PostgreSQL. This deployment is for trial purposes only and not recommended for use in production.

You should set up an external, production-ready PostgreSQL instance. PostgreSQL 13 is the recommended default version since GitLab chart 6.0.

As of GitLab chart 4.0.0, replication is available internally, but not enabled by default. Such functionality has not been load tested by GitLab.

Redis

By default, the GitLab chart includes an in-cluster Redis deployment that is provided by bitnami/Redis. This deployment is for trial purposes only and not recommended for use in production.

You should set up an external, production-ready Redis instance. For all the available configuration settings, see the Redis globals documentation.

As of GitLab chart 4.0.0, replication is available internally, but not enabled by default. Such functionality has not been load tested by GitLab.

Decide on other options

You will use the following options with helm install when you deploy GitLab.

Secrets

You must create some secrets, like SSH keys. By default, these secrets are generated automatically during the deployment, but if you want to specify them, you can follow the secrets documentation.

Networking and DNS

By default, to expose services, GitLab uses name-based virtual servers that are configured with Ingress objects. These objects are Kubernetes Service objects of type: LoadBalancer.

You must specify a domain that contains records to resolve gitlab, registry, and minio (if enabled) to the appropriate IP address for your chart.

For example, use the following with helm install:

--set global.hosts.domain=example.com

With custom domain support enabled, a *.<pages domain> sub-domain, which by default is <pages domain>, becomes pages.<global.hosts.domain>. The domain resolves to the external IP assigned to Pages by --set global.pages.externalHttp or --set global.pages.externalHttps.

To use custom domains, GitLab Pages can use a CNAME record that points the custom domain to a corresponding <namespace>.<pages domain> domain.

Dynamic IP addresses with external-dns

If you plan to use an automatic DNS registration service like external-dns, you don’t need any additional DNS configuration for GitLab. However, you must deploy external-dns to your cluster. The project page has a comprehensive guide for each supported provider.

If you provision a GKE cluster by using the provided script, external-dns is automatically installed in your cluster.

Static IP addresses

If you plan to manually configure your DNS records, they should all point to a static IP address. For example, if you choose example.com and you have a static IP address of 10.10.10.10, then gitlab.example.com, registry.example.com and minio.example.com (if using MinIO) should all resolve to 10.10.10.10.

If you are using GKE, read more on creating the external IP and DNS entry. Consult your cloud or DNS provider’s documentation for more help on this process.

For example, use the following with helm install:

--set global.hosts.externalIP=10.10.10.10

Compatibility with Istio protocol selection

Service port names follow the convention that is compatible with Istio’s explicit port selection. They look like <protocol>-<suffix>, for example tls-gitaly or https-metrics.

Note, that Gitaly uses gRPC, but does not have this suffix due to findings in Issue #3822.

Persistence

By default the GitLab chart creates volume claims with the expectation that a dynamic provisioner creates the underlying persistent volumes. If you would like to customize the storageClass or manually create and assign volumes, review the storage documentation.

TLS certificates

You should be running GitLab with HTTPS, which requires TLS certificates. By default, the GitLab chart installs and configures cert-manager to obtain free TLS certificates.

If you have your own wildcard certificate, or you already have cert-manager installed, or you have some other way of obtaining TLS certificates, read more about TLS options.

For the default configuration, you must specify an email address to register your TLS certificates. For example, use the following with helm install:

--set certmanager-issuer.email=me@example.com

Prometheus

We use the upstream Prometheus chart, and do not override values from our own defaults other than a customized prometheus.yml file to limit collection of metrics to the Kubernetes API and the objects created by the GitLab chart. We do, however, by default disable alertmanager, nodeExporter, and pushgateway.

The prometheus.yml file instructs Prometheus to collect metrics from resources that have the gitlab.com/prometheus_scrape annotation. In addition, the gitlab.com/prometheus_path and gitlab.com/prometheus_port annotations may be used to configure how metrics are discovered. Each of these annotations are comparable to the prometheus.io/{scrape,path,port} annotations.

If you are monitoring or want to monitor the GitLab application with your installation of Prometheus, the original prometheus.io/* annotations are still added to the appropriate Pods and Services. This allows continuity of metrics collection for existing users and provides the ability to use the default Prometheus configuration to capture both the GitLab application metrics and other applications running in a Kubernetes cluster.

Refer to the upstream Prometheus chart documentation for the exhaustive list of configuration options and ensure they are sub-keys to prometheus, as we use this as requirement chart.

For instance, the requests for persistent storage can be controlled with:

prometheus:
  alertmanager:
    enabled: false
    persistentVolume:
      enabled: false
      size: 2Gi
  pushgateway:
    enabled: false
    persistentVolume:
      enabled: false
      size: 2Gi
  server:
    persistentVolume:
      enabled: true
      size: 8Gi

Configure Prometheus to scrape TLS-enabled endpoints

Prometheus can be configured to scrape metrics from TLS-enabled endpoints if the given exporter allows for TLS and the chart configuration exposes a TLS configuration for the exporter’s endpoint.

There a few caveats when using TLS and Kubernetes Service Discovery for the Prometheus scrape configurations:

• For the pod and service endpoints discovery roles, Prometheus uses the internal IP address of the Pod to set the address of the scrape target. To verify the TLS certificate, Prometheus must be configured with either the Common Name (CN) set in the certificate created for the metrics endpoint, or configured with a name included in the Subject Alternative Name (SAN) extension. The name does not have to resolve, and can be any arbitrary string that is a valid DNS name.
• If the certificate used for the exporter’s endpoint is self-signed or otherwise not present in the Prometheus base image, the Prometheus pod must mount a certificate for the Certificate Authority (CA) that signed the certificate used for the exporter’s endpoint. Prometheus uses a ca-bundle from Debian in its base image.
• Prometheus supports setting both of these items using a tls_config which is applied to each of the scrape configurations. While Prometheus has a robust relabel_config mechanism for setting Prometheus target labels based on Pod annotations and other discovered attributes, setting the tls_config.server_name and tls_config.ca_file is not possible using the relabel_config. See this Prometheus project issue for more details.

Given these caveats, the simplest configuration is to share a “name” and CA across all certificates used for the exporter endpoints:

1. Choose a single arbitrary name to use for the tls_config.server_name (for example, metrics.gitlab).
2. Add that name to the SAN list for each certificate used to TLS encrypt the exporter endpoints.
3. Issue all certificates from the same CA:
   • Add the CA certificate as a cluster secret.
   • Mount that secret into the Prometheus server container using the Prometheus chart’s extraSecretMounts: configuration.
   • Set that as the tls_config.ca_file for the Prometheus scrape_config.

The Prometheus TLS values example provides an example for this shared configuration by:

1. Setting tls_config.server_name to metrics.gitlab for the pod/endpoint scrape_config roles.
2. Assuming that metrics.gitlab has been added to the SAN list for every certificate used for the exporter endpoint.
3. Assuming that the CA certificate has been added to a secret named metrics.gitlab.tls-ca with a secret key also named metrics.gitlab.tls-ca created in the same namespace that the Prometheus chart has been deployed to (for example, kubectl create secret generic --namespace=gitlab metrics.gitlab.tls-ca --from-file=metrics.gitlab.tls-ca=./ca.pem).
4. Mounting that metrics.gitlab.tls-ca secret to /etc/ssl/certs/metrics.gitlab.tls-ca using an extraSecretMounts: entry.
5. Setting tls_config.ca_file to /etc/ssl/certs/metrics.gitlab.tls-ca.

Exporter endpoints

Not all of the metrics endpoints included in the GitLab chart support TLS. If the endpoint can be and is TLS-enabled they will also set the gitlab.com/prometheus_scheme: "https" annotation, as well as the prometheus.io/scheme: "https" annotation, either of which can be used with a relabel_config to set the Prometheus __scheme__ target label. The Prometheus TLS values example includes a relabel_config that targets __scheme__ using the gitlab.com/prometheus_scheme: "https" annotation.

The following table lists the Deployments (or when using either or both of Gitaly and Praefect: StatefulSets) and Service endpoints that have the gitlab.com/prometheus_scrape: true annotation applied.

In the documentation links below, if the component mentions adding SAN entries, make sure that you also add the SAN that you decided on using for the Prometheus tls_config.server_name.

For the webservice pod, the exposed port is the standalone webrick exporter in the webservice container. The workhorse container port is not scraped. See the Webservice Metrics documentation for additional details.

Outgoing email

By default, outgoing email is disabled. To enable it, provide details for your SMTP server using the global.smtp and global.email settings. You can find details for these settings in the command line options.

If your SMTP server requires authentication, make sure to read the section on providing your password in the secrets documentation. You can disable authentication settings with --set global.smtp.authentication="".

If your Kubernetes cluster is on GKE, be aware that SMTP port 25 is blocked.

Incoming email

The configuration of incoming email is documented in the mailroom chart.

Service Desk email

The configuration of incoming email is documented in the mailroom chart.

RBAC

The GitLab chart defaults to creating and using RBAC. If your cluster does not have RBAC enabled, you must disable these settings:

--set certmanager.rbac.create=false
--set nginx-ingress.rbac.createRole=false
--set prometheus.rbac.create=false
--set gitlab-runner.rbac.create=false

Next steps

Set up your cloud provider and create your cluster.