Authored by Yun Zheng Hu
Recently, two critical vulnerabilities were reported in Citrix ADC and Citrix Gateway; where one of them was being exploited in the wild by a threat actor. Due to these vulnerabilities being exploitable remotely and given the situation of past Citrix vulnerabilities, RIFT started to research on how to identify the exact version of Citrix ADC and Gateway servers on the internet so that we could inform customers if they hadn’t patched yet.
The exact version information is helpful to determine whether a server is still vulnerable to particular vulnerabilities. We also used version information to derive version statistics and version adoption over time.
In the first part of this blog, we go into how we acquired and analysed Citrix ADC disk images for version identification. Then we go into how we found a way to determine the version build date and how we used the build dates to download missing Citrix ADC images to aid our version identification research. The last part goes into the version statistics of Citrix ADC servers on the internet and how we used this to measure the version adoption on the internet.
Skip to the end if you are interested in the statistics rather than the technical details of Citrix ADC version identification.
CVE-2022-27510 – Unauthorized access to Gateway user capabilities
On November 8th 2022, Citrix published a security bulletin for CVE-2022-27510, a critical authentication bypass vulnerability affecting Citrix ADC (formerly known as NetScaler) and Citrix Gateway. For this to be exploitable, the server must be configured as a Gateway (SSL VPN, ICA Proxy, CVPN, RDP Proxy).
CVE-2022-27518 – Unauthenticated remote arbitrary code execution
Less than a month later, on December 13th 2022, the National Security Agency (NSA) released a Cybersecurity Advisory that APT5 is actively exploiting Citrix ADC servers. However this advisory does not mention a specific CVE that is being abused but most likely a new vulnerability as on the same day Citrix published a blog with guidance and a new security bulletin detailing CVE-2022-27518, which is a new vulnerability and not to be confused with CVE-2022-27510. For this to be exploitable, the Citrix ADC or Gateway server must be configured as a SAML Service Provider or SAML Identity Provider.
Finding Citrix ADC & Gateway Servers on the Internet
Citrix ADC and Gateway servers are commonly internet-facing due to the nature of the appliance. For example, services like Shodan and Censys regularly scan the internet and identify these servers. Using this information, we can build a list of Citrix ADC & Citrix Gateway servers with the SSL VPN / Gateway service exposed to the internet. We used this to create an initial list of servers, and we found around 28.000 servers on the internet as of November 11th 2022.
Sadly, the exact version information is not available in the HTTP response of a Citrix ADC or Gateway server. However, we noticed that there is an MD5 hash-like value in the HTTP body when requesting the
Here we see the parameter
?v=6e7b2de88609868eeda0b1baf1d34a7e appended to several resource URLs. We extracted these hashes from Censys scan data to create a list of the most common version hashes. We found around 100 unique version hashes.
To see if we can map the version hash to exact versions, we begin by first spinning up our own Citrix ADC server to start exploring wether this version hash in this HTML page is static or generated.
More server appliances become available in the cloud, and Citrix ADC is no exception. You can easily find it in your favourite cloud marketplace. So gone are the days of the time-consuming process of downloading images and spinning up a VM to install the application; we can do this directly in the cloud! We used the Google Cloud Marketplace, but it’s also available on AWS and Azure.
After deploying Citrix ADC with a single click from the Cloud Marketplace, we log in to the shell and explore the file system to see if we can find the
index.html page and if it contains a hash value.
SSH into the VM and show the version, looks like we are on version
13.1 build 33.47:
yun@cloudshell:~ (rift-citrix-362712)$ gcloud ssh citrix-adc-vpx-instance ############################################################################### # # # WARNING: Access to this system is for authorized users only # # Disconnect IMMEDIATELY if you are not an authorized user! # # # ############################################################################### Done > show version NetScaler NS13.1: Build 33.47.nc, Date: Sep 23 2022, 13:12:49 (64-bit) Done
shell to enter the shell, and let’s find all
> shell root@ns# uname -a FreeBSD ns 11.4-NETSCALER-13.1 FreeBSD 11.4-NETSCALER-13.1 #0 e5f9d90507ab(heads/artesa_33_47)-dirty: Fri Sep 23 13:13:05 PDT 2022 root@sjc-bld-bsd114-228:/usr/obj/usr/home/build/adc/usr.src/sys/NS64 amd64 root@ns# find / -name 'index.html' /netscaler/ns_gui/admin_ui/gui_v2/swagger_ui/index.html /netscaler/ns_gui/vpn/index.html /var/netscaler/gui/vpn/index.html /var/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html /var/netscaler/logon/LogonPoint/index.html /var/python/lib/python3.7/site-packages/djangorestframework-3.11.0-py3.7.egg/rest_framework/templates/rest_framework/docs/index.html /var/python/lib/python3.7/site-packages/Django-3.0.5-py3.7.egg/django/contrib/admin/templates/admin/index.html /var/python/lib/python3.7/site-packages/Django-3.0.5-py3.7.egg/django/contrib/admindocs/templates/admin_doc/index.html
Looks like there are several directories with
index.html, let’s check
root@ns# head /netscaler/ns_gui/vpn/index.html <!DOCTYPE html PUBLIC "-//W3C//DTD XDEV_HTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title>Citrix Gateway</title> <link rel="SHORTCUT ICON" href="/vpn/images/AccessGateway.ico" type="image/vnd.microsoft.icon"> <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> <META content=noindex,nofollow,noarchive name=robots> <link href="/vpn/js/rdx/core/css/rdx.css?v=7afe87a42140b566a2115d1e232fdc07" rel="stylesheet" type="text/css"/> <link href="/logon/themes/Default/css/base.css?v=7afe87a42140b566a2115d1e232fdc07" rel="stylesheet" type="text/css" media="screen" />
index.html file contains the hash
7afe87a42140b566a2115d1e232fdc07, and if we search for this value on Censys, we get multiple results which is promising. So we can assume that version
13.1-33.47 maps to this hash value.
Let’s check which other files contain this hash value for good measure:
root@ns# grep -r 7afe87a42140b566a2115d1e232fdc07 /var/netscaler | cut -d: -f1 | sort -u /var/netscaler/gui/epa/epa.html /var/netscaler/gui/epa/errorpage.html /var/netscaler/gui/epa/posterrorpage.html /var/netscaler/gui/vpn/index.html /var/netscaler/gui/vpn/loading.html /var/netscaler/gui/vpn/logout.html /var/netscaler/gui/vpn/tmindex.html /var/netscaler/gui/vpn/tmlogout.html /var/netscaler/gui/vpns/choices.html /var/netscaler/gui/vpns/f_ndisagent.html /var/netscaler/gui/vpns/f_services1.html /var/netscaler/gui/vpns/f_services_linux.html /var/netscaler/gui/vpns/j_services.html /var/netscaler/gui/vpns/m_services.html /var/netscaler/gui/vpns/navui/refresh.html /var/netscaler/gui/vpns/nohomepage.html /var/netscaler/gui/vpns/postepa.html
When installing the application from the Google Cloud Marketplace, it does not give the option to choose a version to install. But of course, we want to install other versions to start building a list of known version hashes.
We noticed that a
deployment.zip file can be downloaded from the Google Cloud Marketplace:
deployment.zip we see that this contains Terraform scripts to deploy the cloud application and references the disk image it uses for installation. Luckily, Citrix also left a list in a Jinja template of other disk image names referencing different Citrix ADC versions.
Can these disk images be downloaded? Yes, it turns out you can!
Acquiring Cloud images
We used the following
gcloud commands to download Citrix ADC disk images:
gcloud compute images create, to download the image in our own Google Cloud project. This allows you to choose this image when you create a new VM. However, this does not make the image directly accessible for reading using other tools, for that we need to export it first.
gcloud compute images export, this exports the given image to a Google Cloud Storage (GCS) bucket using a different format such as
The raw disk image is 20 GB but exporting it as
qcow2 format we can reduce this to only 2 GB.
The following shell script does both steps in one script:
#!/bin/sh # # Export a Citrix ADC image to a Google Cloud Storage bucket # # Usage: # ./export-citrix-image.sh <image name> <bucket_path> # Example: # ./export-citrix-image.sh citrix-adc-vpx-10-standard-13-0-85-19 gs://my-bucket image="$1" gcs_bucket="$2" gcloud compute images create "$image" --source-image https://www.googleapis.com/compute/v1/projects/citrix-master-project/global/images/$image --verbosity debug gcloud compute images export --destination-uri "$gcs_bucket/$image.qcow2" --image "$image" --export-format qcow2
Now that we exported some known images from
deployment.zip to a GCS bucket, we can process them in bulk. Of course, having them archived and preserved can also be helpful for future research, especially for vulnerable versions, as they tend to get deleted.
This technique can also be used for other appliances in the Google Cloud Marketplace.
Dissect to the rescue!
What better way to process disk images in bulk than dogfooding our own opensource
dissect framework. While Citrix ADC runs on FreeBSD,
dissect can read its
UFS/FFS file systems just fine. After we fixed a few bugs that is.
We can also do everything in a Cloud Shell, without spinning up an extra VM.
Let’s mount our GCS bucket containing the
qcow2 images first using
yun@cloudshell:~ (rift-citrix-362712)$ mkdir bucket yun@cloudshell:~ (rift-citrix-362712)$ gcsfuse my-citrix-adc-bucket bucket 2022/12/11 15:48:25.442402 Start gcsfuse/0.41.9 (Go version go1.18.4) for app "" using mount point: /home/yun/bucket 2022/12/11 15:48:25.462744 Opening GCS connection... 2022/12/11 15:48:25.557883 Mounting file system "my-citrix-adc-bucket"... 2022/12/11 15:48:25.563799 File system has been successfully mounted. yun@cloudshell:~ (rift-citrix-362712)$ ls -1 bucket/citrix*.qcow2 citrix-adc-vpx-10-standard-13-0-83-27.qcow2 citrix-adc-vpx-10-standard-13-0-87-9.qcow2 citrix-adc-vpx-10-standard-13-0-88-14.qcow2 citrix-adc-vpx-10-standard-13-1-21-50.qcow2 citrix-adc-vpx-10-standard-13-1-33-52.qcow2 citrix-adc-vpx-byol-13-0-76-31.qcow2 citrix-adc-vpx-byol-13-0-79-64.qcow2 citrix-adc-vpx-byol-13-0-82-45.qcow2 citrix-adc-vpx-byol-13-0-88-16.qcow2 citrix-adc-vpx-byol-13-1-33-49.qcow2 citrix-adc-vpx-byol-13-1-33-52.qcow2 citrix-adc-vpx-byol-13-1-33-54.qcow2 citrix-adc-vpx-byol-13-1-37-38.qcow2 citrix-adc-vpx-express-13-0-83-29.qcow2 ...
Now we install the latest version of
pip3 in a virtualenv:
yun@cloudshell:~ (rift-citrix-362712)$ python3 -mvenv dissect yun@cloudshell:~ (rift-citrix-362712)$ source dissect/bin/activate (dissect) yun@cloudshell:~ (rift-citrix-362712)$ pip3 install --pre dissect
Dissect will install different command line tools. One of these tools is called
target-shell, which can read disk images and file systems in various formats and gives you a shell-like interface to explore the file system.
Now let’s open a disk image using
target-shell, we specify the
-q flag to hide some warnings:
Dissect does not (yet!) support Citrix ADC, so instead it gives us access to the discovered filesystems. We see two partitions, and we found the first one is the
/boot partition and second one is the
target-shell has a basic
find command, but the output can be piped to an external tool such as
citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /> cd fs1 citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> find . | grep index.html /fs1/netscaler/gui/vpn/index.html /fs1/netscaler/gui/vpn/tmindex.html /fs1/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html ^C
Let’s cat the first
index.html file (
grep is used to limit the output for this example):
citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> cat /fs1/netscaler/gui/vpn/index.html | grep ?v= | head -n1 <link href="/vpn/js/rdx/core/css/rdx.css?v=c9e95a96410b8f8d4bde6fa31278900f" rel="stylesheet" type="text/css"/> citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1>
Nice, we find that for version
13.0-83-27 the version hash is
Not everyone realizes this, but with just this one command, we loaded a FreeBSD image in
qcow2 disk format containing multiple
UFS/FFS2 partitions by using Python and
dissect in a Cloud Shell. No hassle of installing additional tools and performing tedious steps to mount images. The future is now!
To even further automate this we can use the
dissect Python API or we can make a simple shell oneliner using
target-fs, which can execute some basic commands against a disk image:
It’s good to mention that at this point, we also started to investigate if the version hash can be calculated by MD5 summing variants of the version string, but without any luck.
Identifying the build date
After processing our acquired cloud images, we found that we still had version hashes from the internet without a known version, so it seemed that not all versions were listed or available as a cloud image. We did manage to acquire some cloud images by guessing the image name, but this was not enough to fill in the gaps.
We finally resorted to the Citrix downloads page to look for other versions we didn’t have yet. A Citrix account is required but it’s open for registration. However, it seemed that not every version that was released is listed on the Citrix downloads page, especially older builds that were replaced by a new build. We found a GitHub project that scraped Citrix download links that were useful for our research, and we found that these links are still valid (after logging in).
While our known version list kept growing, we still missed certain hashes that were quite common in the dataset. We naturally wanted to know which version that was, and at this point, we found an interesting way to determine the approximate build date of the Citrix ADC server version.
In the disk image we found the gzip-compressed file named
rdx_en.json.gz under the
citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> find . | grep rdx_en.json.gz /fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz /fs1/netscaler/gui/admin_ui/rdx/core/lang/rdx_en.json.gz ^C citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /> ls -la /fs1/netscaler/gui/vpn/js/rdx/core/lang | grep rdx_en -rw-r--r-- 1001 513 35 2021-09-27T14:01:20 rdx_en.json.gz
Let’s run the
file command on this gzip file:
citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> file /fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz /fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz: gzip compressed data, was "rdx_en.json", last modified: Mon Sep 27 14:01:20 2021, from Unix
Mon Sep 27 14:01:20 2021. Cool, we retrieve the timestamp of when this gzip file was created and we found that this timestamp is an accurate depiction of when the version was created/released, pretty neat! This JSON file seems to be used for translations purposes, but in newer versions this file is just an empty gzipped JSON dictionary.
rdx_en.json.gz file resides in the
vpn sub-directory, it can also be downloaded remotely by accessing the following URI
We now have a list of version hashes, known versions, and approximate build dates. Using the build dates we can now deduce the approximate version of a version hash that we don’t know the version of yet.
For example, the version hash
4434db1ec24dd90750ea176f8eab213c was still missing its version number, and we already processed all the cloud images and available download links from the Citrix download page. But armed with the knowledge of the build date
2022-06-29 13:46:08 of this version hash, we can deduce that this build date falls between the build dates of the known versions
12.1-65.25. Table for clarity:
|build date||version hash||version|
Ok, the possible versions can be:
These versions are not listed on the download page but does a version within this range exist and can it still be downloaded? By looking at the Citrix download links of known versions we see that the Download ID looks to be incremental, and that the files have a specific format. For example:
URL format example: https://downloads.citrix.com/[DOWNLOAD_ID]/build-[VERSION]_nc_64.tgz https://downloads.citrix.com/20651/build-12.1-65.15_nc_64.tgz <-- known url https://downloads.citrix.com/20???/build-12.1-65.??_nc_64.tgz <-- enumerate this url https://downloads.citrix.com/21408/build-12.1-65.25_nc_64.tgz <-- known url
Can we enumerate the URL and then download the file? The answer is yes, by using a small Python script to enumerate the download link and version, we find that the following URL returns a
build-12.1-65.17_nc_64.tgz, we confirmed that version
12.1-65.17 maps to the version hash
4434db1ec24dd90750ea176f8eab213c. This technique proved to be very useful for filling in most of the gaps in versions in our dataset, with a few missing.
Our compiled list of version hashes, approximate build dates, and versions can be found in this gist: https://gist.github.com/fox-srt/c7eb3cbc6b4bf9bb5a874fa208277e86
Now that we have mapped most of the known version hashes to a version, we can measure how many versions there are active on the internet, and if they are still vulnerable to
The following graph shows the Top 20 active versions on the internet, and also shows if that version is vulnerable to any of the two recent CVEs:
We see that the majority is on version
13.0-88.14, which is not vulnerable to either of the two CVEs. The runner up is version
12.1-65.21 which is not vulnerable to
CVE-2022-27510, but it is to
CVE-2022-27518. There are also many servers that do not return a version hash at all so for those servers we cannot identify the exact version.
Note that for
CVE-2022-27518 a pre-condition of SAML is required for it to be exploitable, so knowing only the version does not fully indicate if the server can be exploited but it’s still a good indicator that it should be updated.
The following graph shows the Top 9 countries using Citrix ADC / Gateway and how many servers are still vulnerable to the two critical CVEs. For most countries we see an obvious drop of servers vulnerable to
CVE-2022-27518 after the NSA and new Citrix advisory publication.
This graph shows the Top 20 countries using Citrix ADC / Gateway and how many servers are properly updated so they are protected against both CVEs.
In this blog, we’ve shown how we performed the version identification of Citrix ADC and Citrix Gateway servers by analysing disk images exported from Google Cloud Marketplace using
dissect. We also demonstrated that gzip files can be helpful for timestamp information and how we utilised this to find and download missing Citrix ADC builds.
Finally, we used the version identification data to measure the versions of internet-facing Citrix ADC and Gateway servers over time and see that the NSA and Citrix advisory really helped with updates. However, some servers remain vulnerable to CVE-2022-27510 or CVE-2022-27518.
We hope this blog creates extra awareness for these two Citrix CVEs and that our research on version identification contributes to future studies.