configuring reverse proxies on nginx for gitlab-ce and jenkins
For the projects we are working on at the Software Technology program (TU Eindhoven), we are using different tools and methodologies in our daily workflow, which enable us to achieve our goals. Such tools include version control and continuous integration systems, as well as methodologies like Agile/Scrum and Kanban. For several years the program had a subscription to a third party service that offered this functionality out of the box. Recently, due to several reasons, we had to stop this subscription. One important question we had to answer at that point was: How to proceed?
The members of the software and hardware infrastructure team (Colin and me) had a meeting to explore the available options. Long story short, we had several desktop PCs sitting around, a variety of open source tools that provide (independently) parts of the functionality we wanted, and a strong will to configure an on-premises solution. This was not envisioned as a temporary solution for five, six, or nine months, but our goal was to create something that will be used both from our and future generations of trainees.
During our exploration of the available options, we were amazed by how many open-source tools are out there, but, in the end, we had to narrow down our options. Our final choice was to proceed with the installation of the following tools:
GitLab is used both for version control and project management. The issue board it offers has a lot of nice features (such as time tracking) that cover entirely our needs in terms of PM. We could use GitLab for continuous integration as well, as GitLab Runners, would probably be sufficient for our needs. However, Jenkins offers much more functionality and options for extensibility via its plugins. Therefore, we decided to proceed with Jenkins instead of GitLab’s Runners.
As mentioned before, we had several desktop PCs, which were not used. Hence, we took one of them, we installed Ubuntu Server 16.04 LTS as OS, and we continued with the installation of GitLab and Jenkins.
For this installation we decided to get the required software from the package manager of Ubuntu. The main reason behind this was the maintainability of the system. If, in the future, less experienced users try to update these packages, it will be an easy task for them. That would not be the case if we did the installation either from source or from a Docker container, as in both cases the future administrators should have more experience to proceed.
Both GitLab and Jenkins come bundled with their own web servers. In most cases this is super-convenient, as you don’t have to worry for the configuration of these servers. You just start the services of GitLab and Jenkins and everything works nicely. However, if you want to place both services on the same machine, both of their web servers will try to occupy the port 80 of the system. This means that if GitLab occupies port 80, Jenkins will have to get another port.
Let’s see the disadvantage of this solution using a small example: Let’s suppose that our machine has the URL http://mymachine.tue.nl, GitLab is listening on port 80, and Jenkins is listening on port 8085. In this case you can directly access GitLab using the URL of the machine, but in order to access Jenkins you will have to use the URL in the following format: http://mymachine.tue.nl:8085. That’s not super-convenient, right? That’s what we thought as well.
Wouldn’t it be much more convenient to access GitLab by following http://mymachine.tue.nl/gitlab and Jenkins at http://mymachine.tue.nl/jenkins? Sure, but how do you do that? Spoiler alert: using reverse proxies.
We will need the following packages from Ubuntu’s package repositories: gitlab-ce, jenkins, nginx. We will start with the installation and configuration of nginx and then we will proceed with the installation of GitLab and Jenkins.
The idea here is the following:
Every service (GitLab and Jenkins) will run using their own web server at different ports. We will use a separate nginx web server (on the same machine) to provide easier access to these services using reverse proxies.
Let’s obtain nginx:
sudo apt-get update
sudo apt-get install nginx
Are we missing something here? What about security? Take a nice warning message:
First things first, I strongly suggest that you follow this awesome tutorial of DigitalOcean on how to get a free SSL certificate using letsencrypt. From this tutorial you will only need steps 1, 2, 3, 6. The remaining configuration will be explained here.
Congratulations! Now let’s proceed with this post.
Since you got your shiny new SSL certificate it’s time to start configuring your nginx server to use that.
Step 1: We said before that we want to access GitLab from the location /gitlab and Jenkins from /jenkins. Let’s see how we can achieve this.
In the path /etc/nginx you can find the configuration file of nginx, called nginx.conf. Make sure that this file contains the following lines:
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
Step 2: Now, there are two ways to create your own configurations on nginx:
nginx.conf file to include your configuration. This, however, might lead to a messed up configuration file, which will (eventually) be difficult to maintain.sites-available. Such configurations can be used to manage your settings on HTTP (port 80) and/or HTTPS (port 443).We will proceed with the second option.
In nginx there are two folders for our configuration files: sites-available and sites-enabled. Long story short, in sites-available you can put all your configurations and in sites-enabled you can create symbolic links to the configuration files of sites-available that you want to enable. As it becomes clear, you can have configuration files in sites-available that are not enabled.
Let’s have a look at the default file inside sites-available. This file contains the default configuration of your server and looks like:
# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
# listen 80;
# listen [::]:80;
#
# server_name example.com;
#
# root /var/www/example.com;
# index index.html;
#
# location / {
# try_files $uri $uri/ =404;
# }
#}
From this structure we see that in the beginning we need to provide the server configuration (listening port, server address and name), then we specify the root of our server’s web directory, and at the end we configure the locations.
The locations is the part where we will mostly focus on. In order to configure our nginx to serve us Jenkins and GitLab under /jenkins and /gitlab, we need to create our own locations in the default configuration file.
For this post we will use the default file to store the HTTP configuration of our server. Afterwards, we will create a new file to store the HTTPS configuration.
The default file is not going to be very interesting. There we will configure nginx to redirect all our guests to the https address scheme.
Let’s start by providing the necessary info (server name, address, port):
server {
listen 80;
root /var/www/nginx;
server_name 131.155.XXX.XXX mymachine.tue.nl;
index index.html index.htm index.nginx-debian.html;
}
Do not forget to change the IP address and hostname of the server_name field to your server’s IP address and hostname.
Next, we will add two more locations (the / location should be already present).
server {
listen 80;
root /var/www/nginx;
server_name 131.155.XXX.XXX mymachine.tue.nl;
index index.html index.htm index.nginx-debian.html;
# Redirecting to the https location
location /gitlab {
return 301 https://$host$request_uri;
}
location /jenkins {
return 301 https://$host$request_uri;
}
# Needed for letsencrypt
location ~ /.well-known {
allow all;
}
# nginx's default location
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}
Now it’s time to test if the configuration we wrote is valid. This is particularly useful, as if there are mistakes nginx will let us know. If everything went well, by executing sudo nginx -t you should see the following:
~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Step 3: We can proceed with the creation of the HTTPS configuration file. In the sites-available folder, create a new file called ssl-enabled and give it the following configuration:
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
include snippets/sssl-mymachine.tue.nl.conf;
include snippets/ssl-params.conf;
server_name mymachine.tue.nl;
access_log /var/log/nginx/ssl.access.log;
root /var/www/nginx;
index index.php index.html index.htm index.nginx-debian.html;
# Redirecting to the https location
location /gitlab {
}
location /jenkins {
}
# Needed for letsencrypt
location ~ /.well-known {
allow all;
}
# nginx's default location
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}
Step 4: It’s finally time to create the reverse proxies. In the previous step we left the contents of location /gitlab and location /jenkins empty.
Let’s assume that our Jenkins will run on port 8085 and GitLab will run on 8086. When our nginx detects a request for the /jenkins location it should proxy this request to localhost:8085 and give us back the response of this proxied request. Since we’re encrypting our traffic over https, we can stop doing that as soon as the traffic reaches our nginx. Therefore, the traffic from nginx to the local web server of Jenkins can be proxied over http.
The configuration of location /jenkins looks like:
location /jenkins {
proxy_pass http://127.0.0.1:8085;
proxy_redirect http:// https://;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
In a similar sense, GitLab’s location configuration looks like:
location /gitlab {
client_max_body_size 0;
gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://localhost:8086;
}
The configuration file is now complete. Let’s create the symbolic link of our new configuration file to the folder sites-enabled, in order to enable it:
sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/ssl-enabled /etc/nginx/sites-enabled/
Step 5: Don’t forget to check if your nginx configuration is valid, like we did at the end of Step 2. If you get no errors you can reload your nginx service, in order to get the latest configuration. This can be done by executing sudo service nginx reload.
From the perspective of nginx’s reverse proxies, yes. You still have to install and configure your Jenkins and GitLab systems though. The installation can be done by executing sudo apt-get install gitlab-ce jenkins. After the installation finishes you can modify the configuration files of GitLab and Jenkins to make their web servers listen to the ports we used in this post.
The configuration file of GitLab (/etc/gitlab/gitlab.rb) must have (among others) the following attributes:
external_url 'https://mymachine.tue.nl/gitlab'
nginx['listen_port'] = 8086 # override only if you use a reverse proxy: https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-port
nginx['listen_https'] = false # override only if your reverse proxy internally communicates over HTTP: https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
After modifying GitLab’s configuration file, you have to run sudo gitlab-ctl reconfigure to enable the new settings.
Your Jenkins’ configuration file (/etc/default/jenkins) must have (among others) the following attributes:
NAME=jenkins
HTTP_PORT=8085
PREFIX=/$NAME
JENKINS_ARGS="--prefix=$PREFIX --httpListenAddress=127.0.0.1 --webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"
Afterwards you have to reload Jenkins to enable the new configuration by executing sudo service jenkins reload.