Lessons learned from a Man-in-the-Middle attack

It’s become a widely accepted mantra that experiencing a cyber breach is a question of ‘when’ and not ‘if’. For Fox-IT ‘if’ became ‘when’ on Tuesday, September 19 2017, when we fell victim to a “Man-in-the-Middle” attack. As a result of the multi-layered security protection, detection and response mechanisms we had in place, the incident was both small and contained, but as a cyber security specialist it has made us look long and hard at ourselves. While the police investigation is still on-going, we are sharing details of this incident with the public now that we feel confident that most details are sufficiently clear. This is about who we are and what we do. We believe that ultimately in these cases, transparency builds more trust than secrecy and there are lessons to be learned, both good and bad, that we want to share.

So what happened?

In the early morning of September 19 2017, an attacker accessed the DNS records for the Fox-IT.com domain at our third party domain registrar. The attacker initially modified a DNS record for one particular server to point to a server in their possession and to intercept and forward the traffic to the original server that belongs to Fox-IT. This type of attack is called a Man-in-the-Middle (MitM) attack. The attack was specifically aimed at ClientPortal, Fox-IT’s document exchange web application, which we use for secure exchange of files with customers, suppliers and other organizations. We believe that the attacker’s goal was to carry out a sustained MitM attack.

Because we detected and addressed the breach quickly we limited the total effective MitM time to 10 hours and 24 minutes. In the scheme of the industry average time of detection of weeks this was a short exposure, but we couldn’t prevent the attacker from intercepting a small number of files and information that they should not have had access to. An important first step in our response was to contact Law Enforcement and share the necessary information with them to enable them to start a criminal investigation.

The incident unfolded as follows (all times are CEST):

Sept 16 2017 First reconnaissance activities against our infrastructure that we believe are attributable to the attacker. These included regular port scans, vulnerability scans and other scanning activities.
Sept 19 2017, 00:38 The attacker changed DNS records for fox-it.com domain at a third party provider.
Sept 19 2017, 02:02 Latest moment in time that we have been able to determine that clientportal.fox-it.com still pointed to our legitimate ClientPortal server. This means that traffic destined for the ClientPortal was not being intercepted yet.
Sept 19 2017, 02:05-02:15 Maximum 10-minute time window during which the attacker temporarily rerouted and intercepted Fox-IT email for the specific purpose of proving that they owned our domain in the process of fraudulently registering an SSL certificate for our ClientPortal.
Sept 19 2017, 02:21 The actual MitM against our ClientPortal starts. At this point, the fraudulent SSL certificate for ClientPortal was in place and the IP DNS record for clientportal.fox-it.com was changed to point to a VPS provider abroad.
Sept 19 2017, 07:25 We determined that our name servers for the fox-it.com domain had been redirected and that this change was not authorized. We changed the DNS settings back to our own name servers and changed the password to the account at our domain registrar. This change will have taken time to have full effect, due to caching and the distributed nature of the domain name system.
Sept 19 2017, 12:45 We disabled the second factor authentication for our ClientPortal login authentication system (text messages), effectively preventing users of ClientPortal from successfully logging in and having their traffic intercepted. Other than that, we kept ClientPortal functional in order not to disclose to the attacker that we knew what they were doing, and to give ourselves more time to investigate. At this point, the MitM against ClientPortal was still active technically, but would no longer receive traffic to intercept as users would not be able to perform two factor authentication and log in.
Sept 19 – Sept 20 2017 A full investigation into the incident was undertaken, along with notification of all clients that had files intercepted and the relevant authorities, including the Dutch Data Protection Authority. A police investigation was launched and is still ongoing. Based on the outcome of our investigation, we understood the scope of the incident, we knew that the attack was fully countered and we were prepared to re-enable two factor authentication on ClientPortal in order to make it fully functional again.
Sept 20, 15:38 ClientPortal fully functional again. Our internal investigation into the incident continued.

What kind of information did the attacker intercept?

As mentioned, the attacker was able to redirect inbound traffic to ClientPortal and emails going to the fox-it.com domain for a short period of time. At no stage did they have access to any external or internal Fox-IT system, or indeed system level access to our ClientPortal.

When investigating the attack, we made heavy use of our own technology. For all Fox-IT traffic, we employ CTMp network sensors to detect anomalies and alert us of attacks. All our sensors do full packet capture and this is what allowed us to determine precisely, beyond any doubt, which information was intercepted by the attacker, the timeline of the event and who was affected by the attack.

Because of this we know the following about the information that the attacker was able to intercept:

  • We know the following facts about the information that the attacker intercepted:
    1. Nine individual users logged in and their credentials were intercepted and all users were contacted immediately. Because we use two factor authentication, these credentials alone were not enough for the attacker to log in to ClientPortal.
    2. Twelve files (of which ten were unique) were transferred and intercepted.
      • Of these, three files were client confidential in nature, none were classified as state secret. Files classified as state secret are never transferred through our ClientPortal.
      • Other files were in the public domain and not confidential.
      • All affected clients in respect of these files were contacted immediately.
    3. One mobile phone number was intercepted and this person was informed.
    4. A subset of names and email addresses of ClientPortal users were intercepted.
      • We determined that this information alone would not be considered sensitive, but we have nevertheless informed those people affected.
    5. The names of accounts in ClientPortal were intercepted.
      • This is a list of the names of organizations that we have exchanged files with in the past. This includes organizations such as customers, partners and suppliers.
      • After review, we determined that the names alone would not be considered sensitive.
  • Email: we know that the attacker intercepted Fox-IT email for a maximum of 10 minutes, and that during those 10 minutes, emails destined for Fox-IT were redirected to an external email provider. Since email distribution is decentralized on the internet, we have no way of determining which emails were intercepted by the attacker during that time.

The start of the incident response process

We have determined that the attacker successfully logged in to the DNS control panel of our third party domain registrar provider using valid credentials. This raises two key questions:

  • How were valid credentials obtained?
  • Why was access to the domain registrar possible with single factor rather than multi factor authentication?

On the question of how the credentials were obtained we have conducted extensive internal investigations alongside those conducted by the police investigating the incident. We interviewed people that had access to the vault, carried out extensive forensic investigation of all the systems that the password could been used on, and performed additional network forensics. We currently have no evidence that the password was stolen from any of our own internal systems. Right now, based on our own and the police investigation, we have strong evidence that supports our hypothesis that the adversary gained access to our credentials through the compromise of a third party provider. Investigations are ongoing.

A factor which possibly helped the attacker was that the password had not been changed since 2013. While our password is considered strong and can withstand brute force attacks, it was not changed because it was hardly ever used: DNS settings in general change very rarely. Nevertheless, this was something we could clearly have improved in our process.

The infrequency of our engagement with our domain registrar is also behind the answer to the second question, as to how the attacker was able to gain access to the DNS records solely using a username/password combination. Two factor authentication (2FA) – whereby the initial attempt to log-on can only be completed once a code sent to an authorized device is entered as second factor authentication – should be standard practice here. We chose our domain name registrar provider 18 years ago when 2FA was neither a consideration nor a possibility. We were surprised to find that the registrar still does not support 2FA. It is always worth asking: does your DNS registrar support 2FA? The answer may surprise you.

Reasonably quick detection but room for improvement

We hold our hands up to a number of errors on prevention, but we did decidedly better in the areas of detection and response. Once the attack was ongoing, we detected it reasonably quickly: within a few hours compared to the typical industry average of weeks before an attack is detected.

Our Security Operations Center (SOC) had noticed that a number of scans for weaknesses on our infrastructure were made in the days leading up to the attack. These were treated as regular “background noise on the internet” and not followed up on. While paying attention to those scans would probably not have helped us predict the following attack, it would certainly have raised our attention and we have now established mechanisms to improve early warning of suspicious security probing of this nature.

Effective response, containment and scope determination

In general, once an incident is detected, the most important goal becomes to limit the incident and determine the impact of the attack. Once we knew about the incident, we acted quickly and we were able to limit the impact of the incident quite effectively and in a timely manner.

The most important action that we took in order to limit the impact of the attack was to disable 2FA, in lieu of disabling login functionality altogether. We did this specifically to prevent people from successfully logging in but so as not to alert the attacker to the fact that we knew of the attack. This allowed us to better understand the modus operandi and scope of the attack before taking specific actions to mitigate it, an approach which is standard operating procedure for our CERT team. From that moment on, nobody could log in, effectively preventing traffic to our ClientPortal from being intercepted. Note that this did not directly stop the attack, but it stopped its effectiveness.

The use of full packet capture and CTMp network sensors was crucial in determining the scope of the attack. We could, within a few hours of finding out about the attack, determine exactly who was affected and what the scope of the attacker was. This helped us to understand the incident with confidence and to quickly notify those directly affected and the Dutch Data Protection Authority.

Lessons learned and recommendations

Looking back on this incident, we learned the following lessons and have the following recommendations:

  • Choose a DNS provider that doesn’t allow changes through a control panel but requires a more manual process, considering that name servers are very stable and hardly every change. If you do require more frequent changes, use 2FA.
  • Ensure that all system access passwords are reviewed regularly and changed, even those which are used rarely.
  • Deploy certificate transparency monitoring in order to detect, track and respond to fraudulent certificates.
  • Deploy full packet capture capabilities with good retention in crucial points of your infrastructure, such as the DMZ and border gateways.
  • Always inform Law Enforcement at an early stage, so that they can help you with your investigations, as we did in our case.
  • Make it a management decision to first understand an attack before taking specific actions to mitigate it. This may include letting an attack continue for a short period of time. We consciously made that decision.

Looking back in conclusion

While we deeply regret the incident and the shortcomings on our part which contributed to it, we also acknowledge that a number of the measures we had in place enabled us to detect the attack, respond quickly and confidently and thereby limited the scale and length of the incident. And that’s why the twin mantras in security should always be followed:

  • layered security; and
  • prevention, detection and response.

At the time that ‘if’ becomes ‘when’, it is the combination of these that ultimately determines your overall resilience and cyber security stance.

Criminals in a festive mood

This morning the Fox-IT Security Operations Center observed a large number of phishing e-mails that contained a link to a downloadable zip file. Anyone downloading and opening that zip file would infect themselves with banking malware, that would subsequently try to lure the victim into divulging their credit card information.

So far nothing new: e-mail as attack vector, distribution of the Zeus Panda banking trojan, targeting the same institutions.

Except that this time, it appears the criminals are preparing for the festive season. What stood out to us is the inclusion of a number of local retailers that are now targeted by this banking trojan. Some targeted websites which were extracted from the configuration are:

  • Coolblue
  • Booking.com
  • Otto
  • Amazon
  • De Volksbank (SNS, ASN & Regiobank)
  • ING
  • ABN Amro
  • Knab
  • Triodos

So now, when someone has infected themselves with this malware by opening the malicious zip file, not only will the malware ask for their credit card details when they visit their bank’s website, but also when they visit an online retailer for (Christmas) shopping.

Read on for recommendations, notable observations, stats and screenshots and technical details including indicators of compromise.

Recommendations
The usual recommendations for end users apply: be alert for criminals attacking you by sending legitimate looking e-mails with links and attachments. And be alert for websites behaving differently and asking for credit card details or other personal data where they normally don’t. If you suspect an infection, you may check out a website from a different device to see if it behaves the same. If it doesn’t, you may be infected.

The e-mail itself is nothing out of the ordinary. It appears to be targeting the Netherlands and Germany, using Dutch text and faking the Dutch DHL Group. This is what it looks like:

Beste heer/mevrouw,
UW ZENDING IS ONDERWEG ,Informatie Over Uw Zending is in dokument.

Controleer hieronder uw zending- en contactgegevens. Klik op om te bevestigen.
Bedankt dat u heeft gekozen voor On Demand Delivery.
DHL Express – Excellence. Simply delivered.
Nederlandse Post DHL Group

For organisations, the recommendations are also familiar: isolate any infected systems prior to cleaning them, change any password that was used after infection and consider client certificates on that system compromised. You may refer to the indicators of compromise later on in this post.

Additional interesting observations
The malware that is being distributed is called Zeus Panda, which we’ve followed for almost two years now. This is a variant of the Zeus family of malware that Fox-IT has observed since around 2006, for the purpose of protecting its own customers. The name Zeus Panda comes from the web panel used by the malware operators.

At the time of writing, the two malicious zip file referred in the emails received a little over 48 thousand clicks, mostly in the Netherlands, but also in other parts of Western Europe and some in North America. Out of those 48 thousand clicks, only 11 thousand came from a Window system, which is the only platform that the malware runs on. The other 37 thousands people were safe! A clear example and proof of the shotgun approach that criminals still successfully use.

Also interesting is the clunky nature of the injects. As shown in the screenshots below, the code that the criminals inject into the website on the infected system looks, well, unfinished.

Full statistics
The link in the email is a Google Shortened URL, which downloads the zip-file from
hxxp://partytimeevents.nl/contactgegevens%2012_2017_10_00_.zip
hxxp://stegengaweb.nl/files/contactgegevens%2012_2017_10_00_.zip

By default Google shortened URL’s keeps track of the following statistics:
– Amount of clicks
– Used Browsers
– Referrers
– Countries
– Platforms

Requesting the statistics of the shortened URL results in the following statistics for:

 

Screenshot of the inject asking for credit card details
From an infected system, Zeus Panda will inject extra code into a website. Once the code is injected into one of the targeted web pages, an extra form is added for creditcard information. For example, Coolblue’s webshop page would look like this, clunky and unfinished. Please note that Coolblue has no control over the fact that criminals attempt to inject code into their website from infected machines.

coolblue_main

Zeus Panda Banker web inject

EDIT: the total click count for both domains has increased to a total of 66 thousand, even though both ZIP-files are not available anymore.

Indicators of Compromise
—Dropper—
hxxp://partytimeevents.nl/contactgegevens%2012_2017_10_00_.zip (compromised website)
hxxp://stegengaweb.nl/files/contactgegevens%2012_2017_10_00_.zip (compromised website)
hxxp://axprofessional.it/onenl.exe

—Command-and-Control—
hxxps://avimart.ru/3inexowtoqiyzlonyunku.dat
hxxps://astronatal.ru/2odirnaogfaugdoxiwoex.dat
hxxps://abci.ru/1yhubydnopyakleqinyyx.dat
185.224.133.57 (SSL connection)

—External panel for injects—
hxxps://adsfun.club/

—Hashes—
contactgegevens 2012_2017_10_00.zip
MD5: aefc0fe15836165291cb66eac5ffd177
SHA256: 588e31ac96bd6318f787602e87f86b75d4b5537679e11ba5a509589148033275

contactgegevens 12_2017_10_00_.js
MD5: deb9a0aa69270a0b263b80ed13880b24
SHA256: eb65b1d5f5b3ccc263a4984275c084b63b0a262a87d55887d6a4d744a75e4112

onenl.exe
MD5: 4ac38a4efa276f8d64c1ed39a53e7ab8
SHA256: e556273db50d4588d7e4b5183d06d39b0ebedbb094fc2a39b59416212c829324

Detection and recovery of NSA’s covered up tracks

Part of the NSA cyber weapon framework DanderSpritz is eventlogedit, a piece of software capable of removing individual lines from Windows Event Log files. Now that this tool is leaked and public, any criminal willing to remove its traces on a hacked computer can use it. Fox-IT has looked at the software and found a unique way to detect the use of it and to recover the removed event log entries.

Introduction

A group known as The Shadow Brokers published a collection of software, which allegedly was part of the cyber weapon arsenal of the NSA. Part of the published software was the exploitation framework FuzzBunch and post-exploitation framework DanderSpritz. DanderSpritz is a full-blown command and control server, or listening post in NSA terms. It can be used to stealthy perform various actions on hacked computers, like finding and exfiltrating data or move laterally through the target network. Its GUI is built on Java and contains plugins written in Python. The plugins contain functionality in the framework to perform specific actions on the target machine. One specific plugin in DanderSpritz caught the eye of the Forensics & Incident Response team at Fox-IT: eventlogedit.

DanderSpritz with eventlogedit in action

Figure 1: DanderSpritz with eventlogedit in action

eventlogedit

Normally, the content of Windows Event Log files is useful for system administrators troubleshooting system performance, security teams monitoring for incidents, and forensic and incident response teams investigating a breach or fraud case. A single event record can alert the security team or be the smoking gun during an investigation. Various other artefacts found on the target system usually corroborate findings in Windows Event Log files during an investigation, but a missing event record could reduce the chances of detection of an attack, or impede investigation.

Fox-IT has encountered event log editing by attackers before, but eventlogedit appeared to be more sophisticated. Investigative methods able to spot other methods of event log manipulation were not able to show indicators of edited log files after the use of eventlogedit. Using eventlogedit, an attacker is able to remove individual event log entries from the Security, Application and System log on a target Windows system. After forensic analysis of systems where eventlogedit was used, the Forensics & Incident Response team of Fox-IT was able to create a Python script to detect the use of eventlogedit and fully recover the removed event log entries by the attacker.

Analysing recovered event records, deleted by an attacker, gives great insight into what an attacker wanted to hide and ultimately wanted to achieve. This provides security and response teams with more prevention and detection possibilities, and investigative leads during an investigation.

Before (back) and after (front) eventlogedit

Figure 2: Before (back) and after (front) eventlogedit

eventlogedit in use

Starting with Windows Vista, Windows Event Log files are stored in the Windows XML Eventlog format. The files on the disk have the file extension .evtx and are stored in the folder \Windows\System32\winevt\Logs\ on the system disk, by default. The file structure consists of a file header followed by one or more chunks. A chunk itself starts with a header followed by one or more individual event records. The event record starts with a signature, followed by record size, record number, timestamp, the actual event message, and the record size once again. The event message is encoded in a proprietary binary XML format, binXml. BinXml is a token representation of text XML.

Fox-IT discovered that when eventlogedit is used, the to-be-removed event record itself isn’t edited or removed at all: the record is only unreferenced. This is achieved by manipulation of the record header of the preceding record. Eventlogedit adds the size of the to-be-removed-record to the size of the previous record, thereby merging the two records. The removed record including its record header is now simply seen as excess data of the preceding record. In Figure 3 this is illustrated. You might think that an event viewer would show this excess or garbage data, but no. Apparently, all tested viewers parse the record binXml message data until the first end-tag and then move on to the next record. Tested viewers include Windows Event Viewer as well as various other forensic event log viewers and parsers. None of them was able to show a removed record.

Untouched event records (left) and deleted event record (right)

Figure 3: Untouched event records (left) and deleted event record (right). Note: not all field are displayed here.

Merely changing the record size would not be enough to prevent detection: various fields in the file and chunk header need to be changed. Eventlogedit makes sure that all following event records numbers are renumbered and that checksums are recalculated in both the file and chunk header. Doing so, it makes sure that obvious anomalies like missing record numbers or checksum errors are prevented and will not raise an alarm at the system user, or the security department.

Organizations which send event log records on the fly to a central log server (e.g. a SIEM), should be able to see the removed record on their server. However, an advanced attacker will most likely compromise the log server before continuing the operation on the target computer.

Recovering removed records

As eventlogedit leaves the removed record and record header in its original state, their content can be recovered. This allows the full recovery of all the data that was originally in the record, including record number, event id, timestamps, and event message.

Fox-IT’s Forensics & Incident Response department has created a Python script that finds and exports any removed event log records from an event log file. This script also works in the scenario when consecutive event records have been removed, when the first record of the file is removed, or the first record of a chunk is removed. In Figure 4 an example is shown.

Recovering removed event records

Figure 4: Recovering removed event records

We have decided to open source the script. It can be found on our GitHub and works like this:

$ python danderspritz_evtx.py -h
usage: danderspritz_evtx.py [-h] -i INPUT_PATH [-o OUTPUT_PATH]
 [-e EXPORT_PATH]

danderspritz_evtx.py - Parse evtx files and detect the use of the danderspritz
module that deletes evtx entries

optional arguments:
 -h, --help show this help message and exit
 -i INPUT_PATH, --input INPUT_PATH
 Path to evtx file
 -o OUTPUT_PATH, --output OUTPUT_PATH
 Path to corrected evtx file
 -e EXPORT_PATH, --export EXPORT_PATH
 Path to location to store exported xml records

The script requires the python-evtx library from Willi Ballenthin.

Additionally, we have created an easy to use standalone executable for Windows systems which can be found on our GitHub as well.

Danderspritz_evtx.exe

Hashes:
md5    c07f6a5b27e6db7b43a84c724a2f61be 
sha1   6d10d80cb8643d780d0f1fa84891a2447f34627c
sha256 6c0f3cd832871ba4eb0ac93e241811fd982f1804d8009d1e50af948858d75f6b

 

Recommendations

To detect if the NSA or someone else has used this to cover up his tracks using the eventlogedit tool on your systems, it is recommended to use the script on event log files from your Windows servers and computers. As the NSA very likely changed their tools after the leaks, it might be farfetched to detect their current operations with this script. But you might find traces of it in older event log files. It is recommended to run the script on archived event log files from back-ups or central log servers.

If you find traces of eventlogedit, and would like assistance in analysis and remediation for a possible breach, feel free to contact us. Our Forensics & Incident Response department during business hours or FoxCERT 24/7.

Wouter Jansen
Fox-IT Forensics & Incident Response