My GitLab Server Has Been Acting Strange / Was Ticketed for Outbound Denial of Service — What Should I Do?

We have recently observed an increase in ToS Violation tickets opened on Linodes with GitLab installed, specifically regarding GitLab versions that were affected by a critical vulnerability. We wanted to make our Community Questions members aware, and provide guidance for next steps.

GitLab Vulnerability (CVE-2021-22205)

On April 14, 2021, GitLab published a Security Release to address CVE-2021-22205, indicating a critical remote code execution (RCE) vulnerability within the service’s web interface. CVE-2021-22205 affects both GitLab Enterprise Edition (EE) and GitLab Community Edition (CE), starting from version 11.9. The vulnerability was patched in the following versions:

• 13.10.3
• 13.9.6
• 13.8.8

For some background, remote code execution consists of exploiting a vulnerability within a service installed on a server to remotely execute code without being properly authenticated, typically under the service’s specific user (in this case, the user being git).

Next Steps

According to AttackerKB, when the attack was simulated, you can determine if you’ve been affected by this vulnerability by examining the gitlab-workhorse log. On Ubuntu, that’s located in /var/log/syslog/gitlab/gitlab-workhorse/. Here is example output that their reverse shell generated:

{"command":["exiftool","-all=","--IPTC:all","--XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],"correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","error":"signal: killed","level":"info","msg":"exiftool command failed","stderr":"sh: 1: Trying: not found\nsh: 2: Connected: not found\nsh: 3: Escape: not found\nConnection closed by foreign host.\n","time":"2021-10-31T11:07:18-07:00"}
{"correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","error":"error while removing EXIF","level":"error","method":"POST","msg":"","time":"2021-10-31T11:07:18-07:00","uri":"/e7c6305189bc5bd5"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FKBH8HB3A5YR8S7PYYB5A8SN","duration_ms":7636442,"host":"10.0.0.7","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":422,"system":"http","time":"2021-10-31T11:07:18-07:00","ttfb_ms":7636436,"uri":"/e7c6305189bc5bd5","user_agent":"curl/7.47.0","written_bytes":2936}

While the general recommendation would be to simply update your GitLab version to patched versions, it wouldn’t mitigate further exploitation of your Linode, apart from the initial access. The best thing to do is to clone your Linode to transfer your known critical data over to a fresh new Linode. The reason for this is that with an RCE exploit, there could be malicious services and files that were remotely installed on your Linode, allowing for further exploitation. As for your Linode account, please monitor your Outbound Network Transfer to prepare for any overage fees (please see Resources for more information).

As always, please be sure to keep your applications and distributions updated and secured.

Resources:

• https://www.rapid7.com/blog/post/2021/11/01/gitlab-unauthenticated-remote-code-execution-cve-2021-22205-exploited-in-the-wild/
• https://about.gitlab.com/releases/2021/04/14/security-release-gitlab-13-10-3-released/#Remote-code-execution-when-uploading-specially-crafted-image-files
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-22205
• https://csirt.divd.nl/cases/DIVD-2021-00030/
• https://attackerkb.com/topics/D41jRUXCiJ/cve-2021-22205/rapid7-analysis?referrer=blog
• https://therecord.media/gitlab-servers-are-being-exploited-in-ddos-attacks-in-excess-of-1-tbps/
• https://www.linode.com/docs/guides/backing-up-your-data/
• https://www.linode.com/community/questions/18918/how-does-the-monthly-network-transfer-pool-work
• https://www.linode.com/docs/guides/securing-your-server/