[Solved] GitLab Runner Fails to Run Jobs Using Docker Machine and AWS Autoscaling

Hello Guys, recently I faced an issue running GitLab runner which was set up on AWS EC2 instance. If you’re also using GitLab runner and AWS autoscaling then I’m sure you’re facings issues all of the sudden. Don’t worry you’ve come to the perfect place.

Issue: Suddenly GitLab Runner Fails to Run Jobs

If issues are similar like:

In GitLab:

In GitLab runner System:

ERROR: Error creating machine: Error running provisioning: error installing docker:   driver=amazonec2

Solution:

Now, default AWS instances which were ubuntu 16 are no longer supported as runners.

Thus, the installation of Docker fails on the EC2 instance spawned by Docker Machine and the job cannot run.

Also, Azure and GCP suffer from similar problems.

So, to solve this issue add something like “amazonec2-ami=ami-0f234es34232bf” to your MachineOptions[].

Adding “amazonec2-ami=ami-0f234es34232bf” in MachineOptions=[] with the instance AMI id you want to use and available in your region should fix the issue. The default AMIs are Ubuntu 16 and no longer work as runners as mentioned here.

MachineOptions = [
      "amazonec2-access-key=your_access_key",
      "amazonec2-secret-key=your_secret_key",
      "amazonec2-region=eu-central-1",
      "amazonec2-vpc-id=vpc-0b69daxxxxxxx",
      "amazonec2-subnet-id=subnet-0a7207axxxxxxxx",
      "amazonec2-zone=b",
      "amazonec2-use-private-address=true",
      "amazonec2-ami=ami-0f234es34232bf",
      "amazonec2-root-size=32",
      "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-security-group=docker-machine",
      "amazonec2-instance-type=t2.medium",
    ]

You can get a list of AMIs here. And It should be available in your region. Or you can search from the console EC2 instance AMIs

I hope this solved your problem.
Also checkout the articles on Setup Local GitLab Server, Runner, CI/CD, and Nginx Configuration and Configure Autoscaling GitLab Runner On AWS EC2 Using Bastion Host.

Thank you!