Troubleshooting Omnibus GitLab installation issues

Use this page to learn about common issues users can encounter when installing Omnibus GitLab packages.

Hash Sum mismatch when downloading packages

apt-get install outputs something like: 

E: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/trusty/main/g/gitlab-ce/gitlab-ce_8.1.0-ce.0_amd64.deb  Hash Sum mismatch

Run the following to fix this: 

sudo rm -rf /var/lib/apt/lists/partial/*
sudo apt-get update
sudo apt-get clean

See Joe Damato’s from Packagecloud comment and his blog article for more context.

Another workaround is to download the package manually by selecting the correct package from the CE packages or EE packages repository: 

curl -LJO "https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_8.1.0-ce.0_amd64.deb/download"
dpkg -i gitlab-ce_8.1.0-ce.0_amd64.deb

Installation on openSUSE and SLES platforms warns about unknown key signature

Omnibus GitLab packages are signed with GPG keys in addition to the package repositories providing signed metadata. This ensures authenticity and integrity of the packages that are distributed to the users. However, the package manager used in openSUSE and SLES operating systems may sometime raise false warnings with these signatures, similar to 

File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no):
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no): yes

This is a known bug with zypper where zypper ignores the gpgkey keyword in the repository configuration file. With later versions of Packagecloud, there may be improvements regarding this, but currently users have to manually agree to package installation.

So, in openSUSE or SLES systems, if such a warning is displayed, it is safe to continue installation.

apt/yum complains about GPG signatures

You already have GitLab repositories configured, and ran apt-get update, apt-get install or yum install, and saw errors like the following: 

The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F

or 

https://packages.gitlab.com/gitlab/gitlab-ee/el/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for gitlab-ee

This is because on April 2020, GitLab changed the GPG keys used to sign metadata of the apt and yum repositories available through the Packagecloud instance. If you see this error, it generally means you do not have the public keys currently used to sign repository metadata in your keyring. To fix this error, follow the steps to fetch the new key.

Reconfigure shows an error: NoMethodError - undefined method '[]=' for nil:NilClass

You ran sudo gitlab-ctl reconfigure or package upgrade triggered the reconfigure which produced error similar to: 

 ================================================================================
 Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
 ================================================================================

NoMethodError
-------------
undefined method '[]=' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:21:in 'from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in 'from_file'

Relevant File Content:

This error is thrown when /etc/gitlab/gitlab.rb configuration file contains configuration that is invalid or unsupported. Double check that there are no typos or that the configuration file does not contain obsolete configuration.

You can check the latest available configuration by using sudo gitlab-ctl diff-config (Command available starting with GitLab 8.17) or check the latest gitlab.rb.template.

GitLab is unreachable in my browser

Try specifying an external_url in /etc/gitlab/gitlab.rb. Also check your firewall settings; port 80 (HTTP) or 443 (HTTPS) might be closed on your GitLab server.

Note that specifying the external_url for GitLab, or any other bundled service (Registry and Mattermost) doesn’t follow the key=value format that other parts of gitlab.rb follow. Make sure that you have them set in the following format: 

external_url "https://gitlab.example.com"
registry_external_url "https://registry.example.com"
mattermost_external_url "https://mattermost.example.com"
note
Don’t add the equal sign (=) between external_url and the value.

Emails are not being delivered

To test email delivery you can create a new GitLab account for an email that is not used in your GitLab instance yet.

If necessary, you can modify the ‘From’ field of the emails sent by GitLab with the following setting in /etc/gitlab/gitlab.rb: 

gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'

Run sudo gitlab-ctl reconfigure for the change to take effect.

TCP ports for GitLab services are already taken

By default, Puma listens at TCP address 127.0.0.1:8080. NGINX listens on port 80 (HTTP) and/or 443 (HTTPS) on all interfaces.

The ports for Redis, PostgreSQL and Puma can be overridden in /etc/gitlab/gitlab.rb as follows: 

redis['port'] = 1234
postgresql['port'] = 2345
puma['port'] = 3456

For NGINX port changes please see Setting the NGINX listen port.

Git user does not have SSH access

SELinux-enabled systems

On SELinux-enabled systems the Git user’s .ssh directory or its contents can get their security context messed up. You can fix this by running sudo gitlab-ctl reconfigure, which sets the gitlab_shell_t security context on /var/opt/gitlab/.ssh.

In GitLab 10.0 this behavior was improved by setting the context permanently using semanage. The runtime dependency policycoreutils-python has been added to the RPM package for RHEL based operating systems in order to ensure the semanage command is available.

Diagnose and resolve SELinux issues

Omnibus GitLab detects default path changes in /etc/gitlab/gitlab.rb and should apply the correct file contexts. For installations using custom data path configuration, the administrator may have to manually resolve SELinux issues.

Data paths may be altered via gitlab.rb, however, a common scenario forces the use of symlink paths. Administrators should be cautious, because symlink paths are not supported for all scenarios, such as Gitaly data paths.

For example, if /data/gitlab replaced /var/opt/gitlab as the base data directory, the following fixes the security context: 

sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails/
sudo semanage fcontext --list | grep /data/gitlab/

After the policies are applied, you can verify the SSH access is working by getting the welcome message: 

ssh -T git@gitlab-hostname

All systems

The Git user is created, by default, with a locked password, shown by '!' in /etc/shadow. Unless “UsePam yes” is enabled, the OpenSSH daemon prevents the Git user from authenticating even with SSH keys. An alternative secure solution is to unlock the password by replacing '!' with '*' in /etc/shadow. The Git user is still unable to change the password because it runs in a restricted shell and the passwd command for non-superusers requires entering the current password prior to a new password. The user cannot enter a password that matches '*', which means the account continues to not have a password.

Keep in mind that the Git user must have access to the system so please review your security settings at /etc/security/access.conf and make sure the Git user is not blocked.

PostgreSQL error FATAL: could not create shared memory segment: Cannot allocate memory

The packaged PostgreSQL instance tries to allocate 25% of total memory as shared memory. On some Linux (virtual) servers, there is less shared memory available, which prevents PostgreSQL from starting. In /var/log/gitlab/postgresql/current: 

  1885  2014-08-08_16:28:43.71000 FATAL:  could not create shared memory segment: Cannot allocate memory
  1886  2014-08-08_16:28:43.71002 DETAIL:  Failed system call was shmget(key=5432001, size=1126563840, 03600).
  1887  2014-08-08_16:28:43.71003 HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request size (currently 1126563840 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
  1888  2014-08-08_16:28:43.71004       The PostgreSQL documentation contains more information about shared memory configuration.

You can manually lower the amount of shared memory PostgreSQL tries to allocate in /etc/gitlab/gitlab.rb: 

postgresql['shared_buffers'] = "100MB"

Run sudo gitlab-ctl reconfigure for the change to take effect.

PostgreSQL error FATAL: could not open shared memory segment "/PostgreSQL.XXXXXXXXXX": Permission denied

By default, PostgreSQL tries to detect the shared memory type to use. If you don’t have shared memory enabled, you might see this error in /var/log/gitlab/postgresql/current. To fix this, you can disable PostgreSQL’s shared memory detection. Set the following value in /etc/gitlab/gitlab.rb: 

postgresql['dynamic_shared_memory_type'] = 'none'

Run sudo gitlab-ctl reconfigure for the change to take effect.

Reconfigure complains about the GLIBC version 

$ gitlab-ctl reconfigure

/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)

This can happen if the omnibus package you installed was built for a different OS release than the one on your server. Double-check that you downloaded and installed the correct Omnibus GitLab package for your operating system.

Reconfigure fails to create the Git user

This can happen if you run sudo gitlab-ctl reconfigure as the Git user. Switch to another user.

More importantly: do not give sudo rights to the Git user or to any of the other users used by Omnibus GitLab. Bestowing unnecessary privileges on a system user weakens the security of your system.

Failed to modify kernel parameters with sysctl

If sysctl cannot modify the kernel parameters you could possibly get an error with the following stack trace: 

 * execute[sysctl] action run
================================================================================
Error executing action `run` on resource 'execute[sysctl]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----

This is unlikely to happen with non virtualized machines but on a VPS with virtualization like openVZ, container might not have the required module enabled or container doesn’t have access to kernel parameters.

Try enabling the module on which sysctl errored out.

There is a reported workaround described in this issue which requires editing the GitLab’ internal recipe by supplying the switch which ignores failures. Ignoring errors can have unexpected side effects on the performance of your GitLab server, so it isn’t recommended to do so.

Another variation of this error reports the file system is read-only and shows following stack trace: 

 * execute[load sysctl conf] action run
    [execute] sysctl: setting key "kernel.shmall": Read-only file system
              sysctl: setting key "kernel.shmmax": Read-only file system

    ================================================================================
    Error executing action `run` on resource 'execute[load sysctl conf]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '255'
    ---- Begin output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - ----
    STDOUT:
    STDERR: sysctl: setting key "kernel.shmall": Read-only file system
    sysctl: setting key "kernel.shmmax": Read-only file system
    ---- End output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - ----
    Ran cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - returned 255

This error is also reported to occur in virtual machines only, and the recommended workaround is to set the values in the host. The values needed for GitLab can be found inside the file /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf in the virtual machine. After setting these values in /etc/sysctl.conf file in the host OS, run cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - on the host. Then try running gitlab-ctl reconfigure inside the virtual machine. It should detect that the kernel is already running with the necessary settings, and not raise any errors.

You may have to repeat this process for other lines. For example, reconfigure fails three times, after having added something like this to /etc/sysctl.conf: 

kernel.shmall = 4194304
kernel.sem = 250 32000 32 262
net.core.somaxconn = 1024
kernel.shmmax = 17179869184

You may find it easier to look at the line in the Chef output than to find the file (since the file is different for each error). See the last line of this snippet. 

* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create
  - create new file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf
  - update content in file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf from none to 6d765d
  --- /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf 2017-11-28 19:09:46.864364952 +0000
  +++ /opt/gitlab/embedded/etc/.chef-90-omnibus-gitlab-kernel.shmall.conf kernel.shmall20171128-13622-sduqoj 2017-11-28 19:09:46.864364952 +0000
  @@ -1 +1,2 @@
  +kernel.shmall = 4194304

I am unable to install Omnibus GitLab without root access

Occasionally people ask if they can install GitLab without root access. This is problematic for several reasons.

Installing the .deb or .rpm

To our knowledge there is no clean way to install Debian or RPM packages as a non-privileged user. You cannot install Omnibus GitLab RPM’s because the Omnibus build process does not create source RPM’s.

Hassle-free hosting on port 80 and 443

The most common way to deploy GitLab is to have a web server (NGINX/Apache) running on the same server as GitLab, with the web server listening on a privileged (below-1024) TCP port. In Omnibus GitLab we provide this convenience by bundling an automatically configured NGINX service that needs to run its master process as root to open ports 80 and 443.

If this is problematic, administrators installing GitLab can disable the bundled NGINX service, but this puts the burden on them to keep the NGINX configuration in tune with GitLab during application updates.

Isolation between Omnibus services

Bundled services in Omnibus GitLab (GitLab itself, NGINX, PostgreSQL, Redis, Mattermost) are isolated from each other using Unix user accounts. Creating and managing these user accounts requires root access. By default, Omnibus GitLab will create the required Unix accounts during gitlab-ctl reconfigure but that behavior can be disabled.

In principle Omnibus GitLab could do with only 2 user accounts (one for GitLab and one for Mattermost) if we give each application its own runit (runsvdir), PostgreSQL and Redis process. But this would be a major change in the gitlab-ctl reconfigure Chef code and it would probably create major upgrade pain for all existing Omnibus GitLab installations. (We would probably have to rearrange the directory structure under /var/opt/gitlab.)

Tweaking the operating system for better performance

During gitlab-ctl reconfigure we set and install several sysctl tweaks to improve PostgreSQL performance and increase connection limits. This can only be done with root access.

gitlab-rake assets:precompile fails with ‘Permission denied’

Some users report that running gitlab-rake assets:precompile does not work with the omnibus packages. The short answer to this is: do not run that command, it is only for GitLab installations from source.

The GitLab web interface uses CSS and JavaScript files, called ‘assets’ in Ruby on Rails-speak. In the upstream GitLab repository these files are stored in a developer-friendly way: easy to read and edit. When you are a normal user of GitLab, you do not want these files to be in the developer friendly format however because that makes GitLab slow. This is why part of the GitLab setup process is to convert the assets from a developer-friendly format to an end-user friendly (compact, fast) format; that is what the rake assets:precompile script is for.

When you install GitLab from source (which was the only way to do it before we had omnibus packages) you need to convert the assets on your GitLab server every time you update GitLab. People used to overlook this step and there are still posts, comments and mails out there on the internet where users recommend each other to run rake assets:precompile (which has now been renamed gitlab:assets:compile). With the omnibus packages things are different: when we build the package we compile the assets for you. When you install GitLab with an omnibus package, the converted assets are already there! That is why you do not need to run rake assets:precompile when you install GitLab from a package.

When gitlab-rake assets:precompile fails with a permission error it fails for a good reason from a security standpoint: the fact that the assets cannot easily be rewritten makes it harder for an attacker to use your GitLab server to serve evil JavaScript code to the visitors of your GitLab server.

If you want to run GitLab with custom JavaScript or CSS code you are probably better off running GitLab from source, or building your own packages.

If you really know what you are doing, you can execute gitlab-rake gitlab:assets:compile like this: 

sudo NO_PRIVILEGE_DROP=true USE_DB=false gitlab-rake gitlab:assets:clean gitlab:assets:compile
# user and path might be different if you changed the defaults of
# user['username'], user['group'] and gitlab_rails['dir'] in gitlab.rb
sudo chown -R git:git /var/opt/gitlab/gitlab-rails/tmp/cache

‘Short read or OOM loading DB’ error

Try cleaning the old Redis session.

Apt error ‘The requested URL returned error: 403’

When trying to install GitLab using the apt repo if you receive an error similar to: 

W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/DISTRO/dists/CODENAME/main/source/Sources  The requested URL returned error: 403

check if there is a repository cacher in front of your server, like for example apt-cacher-ng.

Add the following line to apt-cacher-ng config(eg. in /etc/apt-cacher-ng/acng.conf): 

PassThroughPattern: (packages\.gitlab\.com|packages-gitlab-com\.s3\.amazonaws\.com|*\.cloudfront\.net)

Read more about apt-cacher-ng and the reasons why this change is needed on the packagecloud blog.

Using self signed certificate or custom certificate authorities

If you are installing GitLab in an isolated network with custom certificate authorities or using self-signed certificate make sure that the certificate can be reached by GitLab. Not doing so will cause errors like: 

Faraday::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed)

when GitLab tries to connect with the internal services like GitLab Shell.

To fix these errors, see the Install Custom Public Certificates section.

error: proxyRoundTripper: XXX failed with: “net/http: timeout awaiting response headers”

If GitLab Workhorse doesn’t receive an answer from GitLab within 1 minute (default), it will serve a 502 page.

There are various reasons why the request might timeout, perhaps user was loading a very large diff or similar.

You can increase the default timeout value by setting the value in /etc/gitlab/gitlab.rb: 

gitlab_workhorse['proxy_headers_timeout'] = "2m0s"

Save the file and reconfigure GitLab for the changes to take effect.

The change you wanted was rejected

Most likely you have GitLab setup in an environment that has proxy in front of GitLab and the proxy headers set in package by default are incorrect for your environment.

See Change the default proxy headers section of NGINX doc for details on how to override the default headers.

Can’t verify CSRF token authenticity Completed 422 Unprocessable

Most likely you have GitLab setup in an environment that has proxy in front of GitLab and the proxy headers set in package by default are incorrect for your environment.

See Change the default proxy headers section of NGINX doc for details on how to override the default headers.

Extension missing pg_trgm

Starting from GitLab 8.6, GitLab requires the PostgreSQL extension pg_trgm. If you are using Omnibus GitLab package with the bundled database, the extension should be automatically enabled when you upgrade.

If you however, are using an external (non-packaged) database, you will need to enable the extension manually. The reason for this is that Omnibus GitLab package with external database has no way of confirming if the extension exists, and it also doesn’t have a way of enabling the extension.

To fix this issue, you’ll need to first install the pg_trgm extension. The extension is located in the postgresql-contrib package. For Debian: 

sudo apt-get install postgresql-contrib

Once the extension is installed, access the psql as superuser and enable the extension.

  1. Access psql as superuser: 

    sudo gitlab-psql -d gitlabhq_production

  2. Enable the extension: 

    CREATE EXTENSION pg_trgm;
\q

  3. Now run migrations again: 

    sudo gitlab-rake db:migrate

If using Docker, you first need to access your container, then run the commands above, and finally restart the container.

  1. Access the container: 

    docker exec -it gitlab bash

  2. Run the commands above

  3. Restart the container: 

    docker restart gitlab

Errno::ENOMEM: Cannot allocate memory during backup or upgrade

GitLab requires 2GB of available memory to run without errors. Having 2GB of memory installed may not be enough depending on the resource usage of other processes on your server. If GitLab runs fine when not upgrading or running a backup, then adding more swap should solve your problem. If you see the server using swap during normal usage, you can add more RAM to improve performance.

NGINX error: ‘could not build server_names_hash, you should increase server_names_hash_bucket_size’

If your external URL for GitLab is longer than the default bucket size (64 bytes), NGINX may stop working and show this error in the logs. To allow larger server names, double the bucket size in /etc/gitlab/gitlab.rb: 

nginx['server_names_hash_bucket_size'] = 128

Run sudo gitlab-ctl reconfigure for the change to take effect.

Reconfigure fails due to “‘root’ cannot chown” with NFS root_squash 

$ gitlab-ctl reconfigure

================================================================================
Error executing action `run` on resource 'ruby_block[directory resource: /gitlab-data/git-data]'
================================================================================

Errno::EPERM
------------
'root' cannot chown /gitlab-data/git-data. If using NFS mounts you will need to re-export them in 'no_root_squash' mode and try again.
Operation not permitted @ chown_internal - /gitlab-data/git-data

This can happen if you have directories mounted using NFS and configured in root_squash mode. Reconfigure is not able to properly set the ownership of your directories. You will need to switch to using no_root_squash in your NFS exports on the NFS server, or disable storage directory management and manage the permissions yourself.

gitlab-runsvdir not starting

This applies to operating systems using systemd (e.g. Ubuntu 18.04+, CentOS, etc.).

Since GitLab 11.2, the gitlab-runsvdir starts during the multi-user.target instead of basic.target. If you are having trouble starting this service after upgrading GitLab, you may need to check that your system has properly booted all the required services for multi-user.target via the command: 

systemctl -t target

If everything is working properly, the output should show look something like this: 

UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
basic.target           loaded active active Basic System
cloud-config.target    loaded active active Cloud-config availability
cloud-init.target      loaded active active Cloud-init target
cryptsetup.target      loaded active active Encrypted Volumes
getty.target           loaded active active Login Prompts
graphical.target       loaded active active Graphical Interface
local-fs-pre.target    loaded active active Local File Systems (Pre)
local-fs.target        loaded active active Local File Systems
multi-user.target      loaded active active Multi-User System
network-online.target  loaded active active Network is Online
network-pre.target     loaded active active Network (Pre)
network.target         loaded active active Network
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target           loaded active active Paths
remote-fs-pre.target   loaded active active Remote File Systems (Pre)
remote-fs.target       loaded active active Remote File Systems
slices.target          loaded active active Slices
sockets.target         loaded active active Sockets
swap.target            loaded active active Swap
sysinit.target         loaded active active System Initialization
time-sync.target       loaded active active System Time Synchronized
timers.target          loaded active active Timers

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

22 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Every line should show loaded active active. As seen in the line below, if you see inactive dead, this means there may be something wrong: 

multi-user.target      loaded inactive dead   start Multi-User System

To examine which jobs may be queued by systemd, run: 

systemctl list-jobs

If you see a running job, a service may be stuck and thus blocking GitLab from starting. For example, some users have had trouble with Plymouth not starting: 

  1 graphical.target                     start waiting
107 plymouth-quit-wait.service           start running
  2 multi-user.target                    start waiting
169 ureadahead-stop.timer                start waiting
121 gitlab-runsvdir.service              start waiting
151 system-getty.slice                   start waiting
 31 setvtrgb.service                     start waiting
122 systemd-update-utmp-runlevel.service start waiting

In this case, consider uninstalling Plymouth.

Init daemon detection in non-Docker container

In Docker containers, GitLab package detects existence of /.dockerenv file and skips automatic detection of an init system. However, in non-Docker containers (like containerd, cri-o, etc.), that file does not exist and package falls back to sysvinit, and this can cause issues with installation. To prevent this, users can explicitly disable init daemon detection by adding the following setting in gitlab.rb file: 

package['detect_init'] = false

If using this configuration, runit service must be started before running gitlab-ctl reconfigure, using the runsvdir-start command: 

/opt/gitlab/embedded/bin/runsvdir-start &

gitlab-ctl reconfigure hangs while using AWS Cloudformation

The GitLab systemd unit file by default uses multi-user.target for both After and WantedBy fields. This is done to ensure service runs after remote-fs and network targets, and thus GitLab will function properly.

However, this interacts poorly with cloud-init’s own unit ordering, which is used by AWS Cloudformation.

To fix this, users can make use of package['systemd_wanted_by'] and package['systemd_after'] settings in gitlab.rb to specify values needed for proper ordering and run sudo gitlab-ctl reconfigure. After reconfigure has completed, restart gitlab-runsvdir service for changes to take effect. 

sudo systemctl restart gitlab-runsvdir

Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)

When starting up GitLab, if an error similar to the following is observed: 

FATAL: Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)

Check if the hostnames in use are resolvable and IPv4 addresses are returned: 

getent hosts gitlab.example.com
# Example IPv4 output: 192.168.1.1 gitlab.example.com
# Example IPv6 output: 2002:c0a8:0101::c0a8:0101 gitlab.example.com

getent hosts localhost
# Example IPv4 output: 127.0.0.1 localhost
# Example IPv6 output: ::1 localhost

If an IPv6 address format is returned, further check if IPv6 protocol support (keyword ipv6) is enabled on the network interface: 

ip addr # or 'ifconfig' on older operating systems

When IPv6 network protocol support is absent or disabled, but the DNS configuration resolves the hostnames as IPv6 addresses, GitLab services will be unable to establish network connections.

This can be resolved by fixing the DNS configurations (or /etc/hosts) to resolve the hosts to an IPv4 address instead of IPv6.

URI::InvalidComponentError (bad component(expected host component: my_url.tld) when external_url contains underscores

If you have set external_url with underscores (for example https://my_company.example.com), you may face the following issues with CI/CD:

  • It will not be possible to open project’s Settings > CI/CD page.
  • Runners will not pick up jobs and will fail with an error 500.

If that’s the case, production.log will contain the following error: 

Completed 500 Internal Server Error in 50ms (ActiveRecord: 4.9ms | Elasticsearch: 0.0ms | Allocations: 17672)

URI::InvalidComponentError (bad component(expected host component): my_url.tld):

lib/api/helpers/related_resources_helpers.rb:29:in `expose_url'
ee/app/controllers/ee/projects/settings/ci_cd_controller.rb:19:in `show'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:486:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:477:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'

As a workaround, avoid using underscores in external_url. There is an open issue about it: Setting external_url with underscore results in a broken GitLab CI/CD functionality.

Upgrade fails with timeout: run: /opt/gitlab/service/gitaly error

If the package upgrade fails when running reconfigure with the following error, check that all Gitaly processes are stopped and then rerun sudo gitlab-ctl reconfigure. 

---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 4886) 15030s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1

Refer to issue 341573 for more details.

Mirroring the GitLab yum repository with Pulp or Red Hat Satellite fails

Direct mirroring of the Omnibus GitLab yum repositories located at https://packages.gitlab.com/gitlab/ with Pulp or Red Hat Satellite fails when syncing. Different errors are caused by different software:

  • Pulp 2 or Satellite < 6.10 fails with "Malformed repository: metadata is specified for different set of packages in filelists.xml and in other.xml" error.
  • Satellite 6.10 fails with "pkgid" error.
  • Pulp 3 or Satellite > 6.10 seems to succeed, but only the repository metadata is synced.

These sync failures are caused by issues with the metadata in the GitLab yum mirror repository. This metadata includes a filelists.xml.gz file that normally includes a list of files for every RPM in the repository. The GitLab yum repository leaves this file mostly empty to work around a size issue that would be caused if the file was fully populated.

Each GitLab RPM contains an enormous number of files, which when multiplied by the large number of RPMs in the repository, would result in a huge filelists.xml.gz file if it was fully populated. Because of storage and build constraints, we create the file but do not populate it. The empty file causes Pulp and RedHat Satellite (which uses Pulp) repository mirroring of the file to fail.

Refer to issue 2766 for details.

Work around the issue

To work around the issue:

  1. Use an alternative RPM repository mirroring tool like reposync or createrepo to make a local copy of the official GitLab yum repository. These tools recreate the repository metadata in the local data, which includes creating a fully-populated filelists.xml.gz file.
  2. Point Pulp or Satellite at the local mirror.

Local mirror example

The following is an example of how to do local mirroring. The example uses:

  • Apache as the web server for the repository.
  • reposync and createrepo to sync the GitLab repository to the local mirror. This local mirror can then be used as a source for Pulp or RedHat Satellite. You can use other tools like Cobbler as well.

In this example:

  • The local mirror is running on a RHEL 8, Rocky 8, or AlmaLinux 8 system.
  • The host name used for the web-server is mirror.example.com.
  • Pulp 3 syncs from the local mirror.
  • Mirroring is of the GitLab Enterprise Edition repository.

Create and configure an Apache server

The following example shows how to install and configure a basic Apache 2 server to host one or more Yum repository mirrors. Consult the Apache documentation for details on configuring and securing your web server.

  1. Install httpd: 

    sudo dnf install httpd

  2. Add a Directory stanza to /etc/httpd/conf/httpd.conf: 

    <Directory “/var/www/html/repos“>
Options All Indexes FollowSymLinks
Require all granted
</Directory>

  3. Complete the httpd configuration: 

    sudo rm -f /etc/httpd/conf.d/welcome.conf
sudo mkdir /var/www/html/repos
sudo systemctl enable httpd --now

Get the mirrored Yum repository URL

  1. Install the GitLab repository yum configuration file: 

    curl "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh" | sudo bash
sudo dnf config-manager --disable gitlab_gitlab-ee gitlab_gitlab-ee-source

  2. Get the repository URL: 

    sudo dnf config-manager --dump gitlab_gitlab-ee | grep baseurl
baseurl = https://packages.gitlab.com/gitlab/gitlab-ee/el/8/x86_64

    You use the contents of baseurl as the source of the local mirror. For example, https://packages.gitlab.com/gitlab/gitlab-ee/el/8/x86_64.

Create the local mirror

  1. Install the createrepo package: 

    sudo dnf install createrepo

  2. Run reposync to copy RPMs to the local mirror: 

    sudo dnf reposync --arch x86_64 --repoid=gitlab_gitlab-ee --download-path=/var/www/html/repos --newest-only

    The --newest-only option only downloads the latest RPM. If you omit this option, all RPMs in the repo (approximately 1 GB each) are downloaded.

  3. Run createrepo to recreate the repository metadata: 

    sudo createrepo -o /var/www/html/repos/gitlab_gitlab-ee /var/www/html/repos/gitlab_gitlab-ee

The local mirror repository should now be available at http://mirror.example.com/repos/gitlab_gitlab-ee/.

Update the local mirror

Your local mirror should be updated periodically to get new RPMs as new GitLab versions are released. One way of doing this is using cron.

Create /etc/cron.daily/sync-gitlab-mirror with the following contents: 

#!/bin/sh

dnf reposync --arch x86_64 --repoid=gitlab_gitlab-ee --download-path=/var/www/html/repos --newest-only --delete
createrepo -o /var/www/html/repos/gitlab_gitlab-ee /var/www/html/repos/gitlab_gitlab-ee

The --delete option used in the dnf reposync command deletes RPMs in the local mirror that are no longer present in the corresponding GitLab repository.

Using the local mirror

  1. Create the Pulp repository and remote: 

    pulp rpm repository create --retain-package-versions=1 --name "gitlab-ee"
pulp rpm remote create --name gitlab-ee --url http://mirror.example.com/repos/gitlab_gitlab-ee/ --policy immediate
pulp rpm repository update --name gitlab-ee --remote gitlab-ee

  2. Sync the repository: 

    pulp rpm repository sync --name gitlab-ee

    This command must be run periodically to update the local mirror with changes to the GitLab repository.

After the repository is synced, you can create a publication and distribution to make it available. See https://docs.pulpproject.org/pulp_rpm/ for details.