Configure Autoscaling GitLab Runner On AWS EC2 Using Bastion Host

Table of Contents

Autoscaling GitLab runner means who has the ability to spin up and down VMs for your builds. If you want extremely useful, cost-effective, and scalable runners as per your needs, this is a great feature of GitLab Runner.

In this article, we’re going to configure autoscaling gitlab runner on AWS EC2 using Bastion and Manager instance concept.

Prerequisites:

Familiarity with AWS, AWS security groups, VPC, etc.
Quick read through Docker Machine (amazonec2) driver parameters
AWS credentials

Bastion Host and Manager Host

Bastion Instance is used as a best practice for accessing privately accessible hosts within a system environment. In our case, Manager Instance is not intended to be publicly accessible. So to access it for updates, and configuration changes, you log in to the bastion host and then access the manager host.

Create an EC2 instance in the public subnet so that we can SSH into it using keypair. This instance will be used only when we need to SSH manager instance. For this, I have used t2.mirco instance.

Security Groups Configuration:

The bastion host security group needs to allow inbound SSH from our client machine and outbound to the private EC2 instance security group.
The Manager Instance (private instance) security group must allow inbound SSH from the Bastion Instance security group.

Also Read: Configure OpenVPN Access Server on EC2 with SSL

CONFIGURE RUNNER ON MANAGER INSTANCE (EC2)

1. Configure SSH-Agent forwarding:

First, we need to choose a distribution that both docker and GitLab Runner support, like Ubuntu, Debian, CentOS, and RHEL. So that, the manager EC2 instance spawns new machines as required.

Note: As we have configured Bastion host, we can create Manager Instance in the private subnet.

Since the manager instance, itself will not run any jobs, so for our initial configuration, we can start with a smaller instance such as t2.medium. This instance will be the dedicated host and will always be up and running, thus it will be the only standard cost. Also, make sure you have an ssh keypair for your bastion instance.

Now, configure the SSH agent forwarding on our system:

Linux:

$ ssh-add -L keypair.pem

Mac:

$ ssh-add -K keypair.pem

In addition, we can verify the keys are available to ssh-agent as:

$ ssh-add -L

Now connect to bastion host from Linux/Mac:

$ ssh -v –A ubuntu@<bastion-IP-address or DNS-entry>

Note: ubuntu is used for the ubuntu ec2 instance. If you’re using another AMI this could be different. Also, -A option enables ssh-agent forwarding. If any issue connecting instance: click here

2. SSH the manager instance (private instance) from the bastion instance or host:

Since we have used agent forwarding we can simply connect using the same keypair as:

ssh -v ubuntu@<instance-IP-address or DNS-entry>

3. Install GitLab Runner, Docker, and Docker Machine on manager instance:

Install followings:

GitLab Runner
Docker
Docker Machine (ubuntu/Debian) or Docker Machine

4. Register the GitLab Runner:

Before configuring the runner, register the runner to connect with GitLab.

Get Runner token
Register the runner
When asked the executor type, enter docker+machine

5. Configure the GitLab Runner:

Edit config.toml file by running the following command:

$ sudo vim /etc/gitlab-runner/config.toml

Configure the following in config.toml:

concurrent = 6
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "GitLab Autoscaling Runner"
  url = "https://gitlab.com/"
  token = "FrxAsc-GWrRdfm_bVnpg"
  executor = "docker+machine"
  limit = 6
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker"
    privileged = true
    disable_cache = false
    oom_kill_disable = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "your_access_key"
      SecretKey = "your_secret_key"
      BucketName = "example-gitlab-runner-cache"
      BucketLocation = "eu-central-1"
[runners.machine]
    IdleCount = 0        # There must be 0 machines in Idle state - when Off-Peak time mode is off
    IdleTime = 600      # Each machine Idle state up to 600 seconds (then-will be removed) - when Off-Peak time mode is off
    MaxBuilds = 6       # Each machine can handle up to 6 jobs in a row (after this it will be removed)
    MachineDriver = "amazonec2"    # Docker Machine is using 'amazonec2' driver
    MachineName = "gitlab-docker-machine-%s"   # Each machine will have a unique name ('%s' is required)
    MachineOptions = [
      "amazonec2-access-key=your_access_key",
      "amazonec2-secret-key=your_secret_key",
      "amazonec2-region=eu-central-1",
      "amazonec2-vpc-id=vpc-0b69daxxxxxxx",
      "amazonec2-subnet-id=subnet-0a7207axxxxxxxx",
      "amazonec2-zone=b",
      "amazonec2-ami=ami-0f234es34232bf",
      "amazonec2-use-private-address=true",
      "amazonec2-root-size=32",
      "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-security-group=docker-machine",
      "amazonec2-instance-type=t2.medium",
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * 2-14 * * mon-fri *"]
      IdleCount = 0
      IdleTime = 600
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      IdleCount = 0
      IdleTime = 60
      Timezone = "UTC"

Breakdown of different sections in config.toml file:

Global section:

In this section, you can define the limit of the jobs that can be run concurrently across all runners. check_interval defines how often should check GitLab for new jobs, in seconds.

concurrent = 6
check_interval = 0

[session_server]
  session_timeout = 1800

runners section:

In this, the most important part is the executor which must be docker+machine. For more options: other options

[[runners]]
  name = "GitLab Autoscaling Runner"
  url = "https://gitlab.com/"  # your GitLab instance
  token = "FrxAsc-GWrRdfm_bVnpg" # Runner's token
  executor = "docker+machine"
  limit = 6

runners.docker section:

This defines the docker image to be used by child runners if it’s not defined in .gitlab-ci.yml.

Also, For more options: other options

[runners.docker]
    tls_verify = false
    image = "docker"
    privileged = true
    disable_cache = false
    oom_kill_disable = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

runners.cache section:

This section will speed up your jobs. Furthermore: reference(runners.cache), reference2(runners.cache.s3)

[runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "your_access_key"
      SecretKey = "your_secret_key"
      BucketName = "example-gitlab-runner-cache"
      BucketLocation = "eu-central-1"

runners.machine section:

This section tells GitLab Runner how and when to spawn new or remove old Docker Machine instances. Also, You can change as your need.

[runners.machine]
    IdleCount = 0        # There must be 0 machines in Idle state - when Off-Peak time mode is off
    IdleTime = 600       # Each machine Idle state up to 600 seconds (then-will be removed) - when Off-Peak time mode is off
    MaxBuilds = 6        # Each machine can handle up to 6 jobs in a row (after this it will be removed)
    MachineDriver = "amazonec2"    # Docker Machine is using 'amazonec2' driver
    MachineName = "gitlab-docker-machine-%s"   # Each machine will have a unique name ('%s' is required)
    MachineOptions = [
      "amazonec2-access-key=your_access_key",
      "amazonec2-secret-key=your_secret_key",
      "amazonec2-region=eu-central-1",
      "amazonec2-vpc-id=vpc-0b69daxxxxxxx",
      "amazonec2-subnet-id=subnet-0a7207axxxxxxxx",
      "amazonec2-zone=b",
      "amazonec2-use-private-address=true",
      "amazonec2-ami=ami-0f234es34232bf",
      "amazonec2-root-size=32",
      "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-security-group=docker-machine",
      "amazonec2-instance-type=t2.medium",
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * 2-14 * * mon-fri *"]
      IdleCount = 0
      IdleTime = 600
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      IdleCount = 0
      IdleTime = 60
      Timezone = "UTC"

Note: Here the “amazonec2-ami=ami-0f234es34232bf”, should be defined with the instance AMI id you want to use and available in your region. The default AMIs are Ubuntu 16 and no longer work as runners as mentioned here.

That’s it!

If you use the runner tags in ci file, configured during runner registration, it will work like charm.

Conclusion

In this article, we learned how to install and configure Autoscaling GitLab Runner on AWS EC2 instance using bastion host and manager host.

Also, read important article on GitLab Runner Fails to Run Jobs Using Docker Machine and AWS.

Thank you!