In this article, I’m gonna walk you through configuring on-premise GitLab Server, GitLab Runner, and CI/CD for containerized microservices with NGINX configuration as a load balancer. We’ll deploy and run all the containerized microservices on the same machine where GitLab and Runner are set up.
GitLab Server
GitLab allows you to host an on-premise GitLab server(or Git repository) that can be accessed from LAN(or WAN if you have a public IP address).
We can install the GitLab server either on a container environment like docker or on the host machine. In this, I’ll install it on the host machine (i.e. Ubuntu 20.04 machine).
Prerequisites
- Machine with at least 2 cores and 4GB of memory.
On-Premise GitLab Server
On-Premise Installation
Step-1: Update System and Install Required Dependencies
Open the terminal on the server(machine) and run the following commands:
$ sudo apt update
$ sudo apt upgrade -yAfter running these, let’s install some dependencies by running:
$ sudo apt install -y ca-certificates curl openssh-server tzdata perlStep-2: Install Postfix for Email Notifications (Optional)
Optionally, if you want to use the same system or server to send email notifications to users, then you can install postfix, an open-source mail transfer agent.
$ sudo apt install postfix -yWhile installing Postfix you’ll be asked to set it up, select ‘Internet Site’ option and add your DNS for mail name and configure other required things.
Further, you’ll need to install the mailutils package:
$ sudo apt install mailutilsOR, you can set up SMTP server to send email notifications instead of Postfix. To do this, you need to first install GitLab and edit /etc/gitlab/gitlab.rb.
Step-3: Add GitLab Package Repository and GPG Key
As the GitLab is not available on the base repository of Ubuntu. You need to add the GitLab package repository and GPG key by running the following command:
$ curl -sS <https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh> | sudo bashHere, I’ve added a package repository for GitLab Enterprise Edition, you can also add for GitLab Community Edition.
After adding the repository package, you can see the repository contents in:
$ cat /etc/apt/sources.list.d/gitlab_gitlab-ce.listStep-4: Install GitLab EE on Ubuntu (22.04 | 20.04 | 18.04)
Now update the system again and install GitLab EE on the machine:
$ sudp apt update
$ sudp apt install gitlab-eeYou’ll see output similar:
It looks like GitLab has not been configured yet; skipping the upgrade script.
*. *.
*** ***
***** *****
.****** *******
******** ********
,,,,,,,,,***********,,,,,,,,,
,,,,,,,,,,,*********,,,,,,,,,,,
.,,,,,,,,,,,*******,,,,,,,,,,,,
,,,,,,,,,*****,,,,,,,,,.
,,,,,,,****,,,,,,
.,,,***,,,,
,*,.
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \\
/ /_/ / / /_/ /___/ /_/ / /_/ /
\\____/_/\\__/_____/\\__,_/_.___/
Thank you for installing GitLab!Now edit external_url in /etc/gitlab/gitlab.rb to set hostname. You can also configure other parameters. Replace gitlab.example.com with valid domain name.
$ sudo nano /etc/gitlab/gitlab.rb
external_url "<http://gitlab.example.com>"OR, If you’re not going to use DNS. You can simply use your server’s IP address like in my case my local machine’s IP as:
external_url "<http://10.10.5.55>"When done start GitLab by running the following command:
$ sudo gitlab-ctl reconfigureAfter successful reconfiguration, check the status of GitLab:
$ sudo gitlab-ctl statusThe output will be similar:
run: alertmanager: (pid 92581) 18s; run: log: (pid 92343) 80s
run: gitaly: (pid 92590) 18s; run: log: (pid 91561) 189s
run: gitlab-exporter: (pid 92551) 20s; run: log: (pid 92078) 98s
run: gitlab-kas: (pid 92520) 22s; run: log: (pid 91845) 175s
run: gitlab-workhorse: (pid 92531) 21s; run: log: (pid 91985) 117s
run: grafana: (pid 92610) 17s; run: log: (pid 92471) 38s
run: logrotate: (pid 91486) 202s; run: log: (pid 91494) 201s
run: nginx: (pid 91993) 114s; run: log: (pid 92013) 110s
run: node-exporter: (pid 92540) 21s; run: log: (pid 92049) 104s
run: postgres-exporter: (pid 92601) 18s; run: log: (pid 92367) 76s
run: postgresql: (pid 91693) 184s; run: log: (pid 91704) 183s
run: prometheus: (pid 92560) 20s; run: log: (pid 92297) 88s
run: puma: (pid 91904) 132s; run: log: (pid 91917) 129s
run: redis: (pid 91521) 196s; run: log: (pid 91538) 193s
run: redis-exporter: (pid 92553) 20s; run: log: (pid 92217) 94s
run: sidekiq: (pid 91922) 126s; run: log: (pid 91934) 122sStep-5: GitLab Web Interface
Once the installation completes, open your URL set up as external_url http://gitlab.example.com on your browser. In my case, I have set it up locally using my server’s IP address. So, I can access it through ***http://10.10.5.55/***
Now, first, you can log in as a root user using username as root and password from /etc/gitlab/intial_root_password. (The password for root user is randomly generated and stored for 24 hours only.)
To check the password run the following:
$ cat /etc/gitlab/initial_root_password
# WARNING: This value is valid only in the following conditions
# 1. If provided manually (either via `GITLAB_ROOT_PASSWORD` environment variable or via `gitlab_rails['initial_root_password']` setting in `gitlab.rb`, it was provided before database was seeded for the first time (usually, the first reconfigure run).
# 2. Password hasn't been changed manually, either via UI or via command line.
#
# If the password shown here doesn't work, you must reset the admin password following <https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password>.
Password: dfdfkOtOjWp7v70OjkjtadsdfsafsadnSJAhcDbCNo9nTNGVC5UoSCyE=
# NOTE: This file will be automatically deleted in the first reconfigure run after 24 hours.Copy the password and login:
After logging in, first reset the root user password. To do this, go to root user profile > Preferences > Password. Change.
You can start, stop or restart all the GitLab components by using the following commands:
$ sudo gitlab-ctl start
$ sudo gitlab-ctl restart
$ sudo gitlab-ctl stopAlso, you can start individual components also:
$ sudo gitlab-ctl restart prometheus
ok: run: prometheus: (pid 2673) 8sThis is all you need to set up GitLab Server on Ubuntu Server.
GitLab Runner
GitLab Runner can be installed:
- In a container(Docker, K8s, or OpenShift)
- By downloading binary manually
- By using repository packages
In this, we’ll be installing Runner on the local machine as I have to deploy microservices on the same machine where GitLab and Runner will be running.
GitLab Runner – Installation
If you’re using a different server for a runner than that GitLab is configured, you need to first ssh into the server and follow the following steps. Otherwise, you can directly follow from step-1.
Step-1: Add Official Repository
First, add the official GitLab Runner Repository using the below command. You can find the Official Repository here.
$ curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bashStep-2: Install Runner on Ubuntu (22.04 | 20.04 | 18.04)
Run the following command to install GitLab Runner:
$ sudo apt update
$ sudo apt install gitlab-runner(Optional) To install a specific version:
$ sudo apt install gitlab-runner=10.0.0After Installation, check the GitLab Runner version:
$ sudo gitlab-runner --versionOutput:
Version: 15.0.0
Git revision: febb2a09
Git branch: 15-0-stable
GO version: go1.17.7
Built: 2022-05-19T20:03:43+0000
OS/Arch: linux/amd64To check the status:
$ sudo gitlab-runner statusOutput:
Runtime platform arch=amd64 os=linux pid=29368 revision=febb2a09 version=15.0.0
gitlab-runner: Service is running!You can Start, Stop, and Restart GitLab Runner by running the following:
$ sudo gitlab-runner start
$ sudo gitlab-runner stop
$ sudo gitlab-runner restartStep-3: Grant Permission to GitLab Runner User
After successful installation, you’ll see a gitlab-runner user in /home directory. And you need to grant the sudo permission to the gitlab-runner user. For this, open visudo file:
$ sudo visudoAdd the following in sudoers group and set NOPASSWD also as shown below:
gitlab-runner ALL=(ALL:ALL) ALL
gitlab-runner ALL=(ALL) NOPASSWD: ALLOutput:
GNU nano 4.8 /etc/sudoers.tmp
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
# Host alias specification
# User alias specification
# Cmnd alias specification
# User privilege specification
root ALL=(ALL:ALL) ALL
gitlab-runner ALL=(ALL:ALL) ALL
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL
# See sudoers(5) for more information on "#include" directives:
#includedir /etc/sudoers.d
gitlab-runner ALL=(ALL) NOPASSWD: ALLStep-4: Register GitLab Runner
Now, it’s time to register GitLab Runner to GitLab Server which was set up before.
- Log in to GitLab Server with username and password(create a user if you haven’t already and create a project).
- Navigate to Settings and click on CI/CD and click on Expand of Runners section.
- Copy the URL and registration token. And register the runner.
$ gitlab-runner register --name project-name-runner --url http://10.10.5.55/ --registration-token GR1348941Ah6PxTBi5S5asJ7fzYSaOR,
$ sudo gitlab-runner registerNote: During registration, you’ll be asked some questions such as tags, name, executor etc., Select executor as shell (If you’re using docker you can select docker or docker+machine as an executor). Since I’m deploying on the same machine I will select executor as shell. Or you can select whatever as per your requirements.
- Now you have successfully performed GitLab Runner Registration.
Check on the GitLab CI/CD > Runners section, you’ll see the newly registered runner.
Error: This job is stuck because the project doesn’t have any runners online assigned to it. Go to Runners page.
It’s due to tags you’ve added during configuration, but not added to your CI/CD job. To solve this, you can add tags to your CICD jobs Or do the following:
- Go to Settings > CI/CD > Runners (Click on Expand)
- Click on the edit icon of the newly created runner. And check Run untagged jobs box.
And here it is, you have successfully installed and registered gitlab-runner for your projects.
Uninstalling GitLab Runner
To completely remove gitlab-runner run the following commands:
$ sudo apt purge --autoremove -y gitlab-runner
$ sudo apt-key del 51312f3f
$ sudo rm -rf /etc/apt/sources.list.d/runner_gitlab-runner.list
$ sudo deluser --remove-home gitlab-runner
$ sudo rm -rf /etc/gitlab-runnerCI/CD for Containerized Applications
My project structure:
.
├── Dockerfile
├── .env
├── .git
├── .gitignore
├── .gitlab-ci.yml
├── package.json
├── README.md
├── .sample.env
└── srcStep-1: Configure .gitlab-ci.yml File
Now it’s time to configure CI/CD for containerized applications to deploy on the same machine where the GitLab Runner is set up. For this, let’s configure .gitlab-ci.yml file as:
stages:
- build
- deploy
before_script:
- IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"
dev-build-job:
stage: build
script:
# - export DOCKER_HOST=tcp://0.0.0.0:2375/
- cp $ENV_FILE .env
- docker build -t $APP_NAME:latest .
only:
- dev
tags:
- cloudyfox
dev-deploy-job:
stage: deploy
script:
- docker container rm -f $APP_NAME || true
- docker run -d -p $PORT:$PORT --name $APP_NAME $APP_NAME:latest
only:
- dev
tags:
- cloudyfoxStep-2: Add Required Variables and Deploy
Here, the pipeline has two stages: build and deploy. IMAGE_TAG variable is used to tag docker images later.
In the build stage, $ENV_FILE variable is copied from the GitLab variables to the project root directory as it is in the .gitignore file. So, create ENV_FILE File variable with the value of .env file. To do so, Go to GitLab Server and navigate to Settings > CI/CD > Variables (Click on Expand). And add a new File variable.
Also, I have added APP_NAME and PORT variables as they are required for me.
In the deploy stage, the docker built in the previous build stage is deployed by deleting a container of the same name if running which is deployed then deploys our docker container.
In this way, you can deploy all the containerized microservices to the machine. Whenever you push the changes to the remote repository, a new docker version will be built and run on the machine(where all GitLab, and Runner are configured.).
Configuring Nginx As a Load Balancer
Let’s say you have successfully run all the micro applications using different ports on the server. Then you have to set up the Nginx load balancer to handle the requests. So that you can handle it in accordance with the users’ requests.
Since my GitLab server is running in the same local host, I need to use a different port to serve microservices. So, let’s start by configuring NGINX on the server.
Step-1: Install NGINX
$ sudo apt update
$ sudo -H apt install nginx-common nginx-fullStart Nginx:
$ sudo nginxAnd start Nginx then check the status:
$ sudo systemctl start nginx
$ sudo systemctl status nginxOutput:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-06-15 09:05:01 +0545; 5h 20min ago
Docs: man:nginx(8)
Process: 875 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, stat>
Process: 1065 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUC>
Main PID: 1066 (nginx)
Tasks: 5 (limit: 9408)
Memory: 4.7M
CGroup: /system.slice/nginx.service
├─1066 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─1067 nginx: worker process
├─1068 nginx: worker process
├─1069 nginx: worker process
└─1070 nginx: worker processYou can Start, Stop, and Restart Nginx as:
$ sudo systemctl start nginx
$ sudo systemctl stop nginx
$ sudo systemctl restart nginxStep-2: Configure Nginx Load Balancer for Dockerized Apps
To configure Nginx as Loadbalancer, first, remove default conf file, /etc/nginx/sites-enabled/default and create /etc/nginx/conf.d/lb.conf with following content(change as your requirements)
I’ll be routing all microservices which are running in different ports to http://10.10.5.55:9999. (as http://10.10.5.55 (port 80) is already used by the GitLab server)
The requests for different services will be load balanced as:
upstream user {
server 127.0.0.1:5001;
}
upstream content {
server 127.0.0.1:5002;
}
upstream discussion {
server 127.0.0.1:5003;
}
upstream payment {
server 127.0.0.1:5004;
}
upstream read {
server 127.0.0.1:5007;
}
upstream subscription {
server 127.0.0.1:5005;
}
upstream card {
server 127.0.0.1:5006;
}
server {
listen 9999;
server_name localhost;
location /api/user {
proxy_pass http://user/api/user;
}
location /api/content {
proxy_pass http://content/api/content;
}
location /api/discussion {
proxy_pass http://discussion/api/discussion;
}
location /api/payment {
proxy_pass http://payment/api/payment;
}
location /api/read {
proxy_pass http://read/api/read;
}
location /api/subscription {
proxy_pass http://subscription/api/subscription;
}
location /api/card {
proxy_pass http://card/api/card;
}
}Here, every service has a different request URI, and all will be routed accordingly.
server_name is your domain name(if you have configured it previously), for me it’s localhost.
And test the syntax:
$ sudo nginx -t Output:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successfulNow, restart the Nginx service:
$ sudo service nginx restartGo to the browser and run http://10.10.5.55:9999/<microservice_endpoint>. You’ll get successful output. (Change 10.10.5.55 with your server’s IP or domain name)
In my case:
And that’s all for Nginx Loadbalancer(reverse-proxy) configuration.
Conclusion
Here you go, you have successfully configured GitLab server, GitLab Runner, CI/CD for dockerized microservices, and NGINX as a load balancer(reverse-proxy) at once.
Thank you!