Post mortem report on the sinowal/nu.nl incident


March 14th 2012, another normal day in the office, I looked at my massive to-do list and had planned quite some work for the day. And like every normal day my planning of the day is rudely interrupted by another incident. Our network analysts at the Security Operations Center looked very busy, and they were calling in for help from the Intel department. The SOC was very busy handling many incidents which started exactly from 11:30 CET. We analyzed the network traffic, the exploit kit and the installed Trojans and found it to be common Eastern European/Russian cybercrime components known as Nuclear Pack exploit kit, SmokeLoader and the Sinowal/Torpig/Mebroot Trojan. We contacted SurfRight to inform them this was happening, but fortunately they were already working on it. We had met with SurfRight a week earlier and discussed the problems with the Sinowal MBR infection and the lack of detection and removal in most current cleaning/removal products, they also knew about it and were still working on it. So this was a good time for them to invest some extra time into it, and with success as within a short time they were able to add detection and next day they had the bootkit cleaning functionality ready.

The second step was to investigate which site was responsible for the infection, typically on large websites which have many visitors a day, the infection comes from an advertisement. Possibly through a compromised advertising service redirecting many visitors to an exploit kit. This happened, for example, in August 2011. Interestingly that attack on OpenX ad server software was also used to distribute the Sinowal malware. Another method used is the purchase of legitimate advertisements for a certain country, and after the advertiser has checked the ads to be legitimate, modify the page to forward visitors to an exploit kit, this technique is known as malvertising. Malvertising actually requires money, so often stolen creditcards are used to buff the advertising account credits to allow many clicks before the credit runs out.

So, we started to look at the data we had available, when looking at network traffic it does not add any hints on where the compromise is, as the redirect is added to the DOM of the main page, so the referrer to the exploit kit is actually http://www.nu.nl. Due to the earlier incident in August 2011 I started looking at the downloaded content from the advertisement servers. Some time passed, it is painful work manually puzzling through all the data to find a needle in a haystack. Well, I could not find the source of infection, while our browser sandbox system indicated every hit for nu.nl was causing a redirect to the exploit kit. It started to puzzle me and only one thing remained, looking at the files of the nu.nl website itself, there was nothing which stood out on the main page such as a remote script, iframe or obfuscated javascript. I started to download the various javascript files where one stood out because it had a modification date of exactly 11:30:00 CET.

  HTTP/1.1 200 OK
  Server: Apache
  Vary: Host
  Last-Modified: Wed, 14 Mar 2012 10:30:00 GMT
  Cache-Control: max-age=86400
  Expires: Thu, 15 Mar 2012 10:31:54 GMT
  Content-Type: application/x-javascript
  P3P: policyref="http://www.nu.nl/w3c/p3p.xml", CP="NON DSP COR NID CURa ADMa DEVa PSAa PSDa IVAa IVDa OUR IND UNI COM NAV INT STA"
  Content-Length: 2290
...

Response headers of http://www.nu.nl/files/g.js

The javascript is obfuscated with a simple obfuscator, but when you would look at this manually it would be easy to skip it as javascript files are often packed or compressed to save bandwidth. The file was simple and for us only interesting to see where it would redirect clients, which was interesting as it redirected systems specifically with Windows Vista/7/8 to a separate location in  the Nuclear Pack exploit kit. We were already in contact with nu.nl and forwarded the malicious script location to them and it was deleted soon after. Both the mainpage modification by including g.js and the script were added recently.

<script src=”hxxp://www.nu.nl/files/g.js” type=”text/javascript”></script>

After the file was overwritten with ‘OK’ at 13:23 CET by the nu.nl staff, the site still caused infections in our sandbox, we rechecked the main site and instead of g.js it now contained a link to gs.js.

<script src=”hxxp://www.nu.nl/files/gs.js” type=”text/javascript”></script>

This file was created before at 13:09:10 CET:

HTTP/1.1 200 OK
Server: Apache
Vary: Host
Last-Modified: Wed, 14 Mar 2012 12:09:10 GMT
Cache-Control: max-age=86400
Expires: Thu, 15 Mar 2012 12:27:41 GMT
Content-Type: application/x-javascript
P3P: policyref="http://www.nu.nl/w3c/p3p.xml", CP="NON DSP COR NID CURa ADMa DEVa PSAa PSDa IVAa IVDa OUR IND UNI COM NAV INT STA"
Content-Length: 2127
...

Response headers of http://www.nu.nl/files/gs.js

We informed the nu.nl staff of this change and at 13:42 CET the gs.js file was overwritten with ‘OK’ by the nu.nl staff. At this time we experienced no infections in our browser sandbox.

The amount of visitors to nu.nl around lunch time on a work day are of course huge, especially from corporate networks. The amount of alerts our Security Operations Center staff was getting combined with how effective the exploit for java typically is, with around one in eight visitors being infected, a little over 12%, led us to believe this was a huge incident and we scaled up our research efforts. At some networks there were hundreds of successful infections which, comparing to the previously referenced incident in August 2011, was of a completely different scale. Also the relative low profile presence of the exploit kit named Nuclear Pack, the use of newly setup hosts and URLs and the use of blocking of visitors outside of NL, which all results in less effective blocking by security products, such as blacklisting. Our educated estimation based on the amount of infections of previous incidents of similar scales makes it very plausible that the amount of infections is 100.000 or more, we found no hints that the related server hosting the exploit kit and C&C for SmokeLoader was overloaded, so that was not a factor that played any role during this infection campaign.

The exact time of publishing the initial g.js might indicate that there might have been a task for publishing introduced in the CMS system, which further increases the likelihood that this was a well planned attack. We suspect this because the exploit kit, the executables and the smokeloader C&C server were all created for the purpose of infecting visitors of nu.nl. The domains, urls and server were not blacklisted during the attack and the 2 smokeloader executables were virgin crypted and were not detected by any anti-virus product that we used to analyze the binary. Even 9 hours after the smokeloader Trojan executables were in the wild, the executables were not recognized by the majority of anti-virus products as can be seen at:

https://www.virustotal.com/file/e625fdb49695886ee2781d6e2c3755f7fc8c9c56ca208bdd14e51f11466abe10/analysis/1331756363/

https://www.virustotal.com/file/c667f39573b4bbd10aa26e841a9dda3ab0407de12606e9a58e16fc9427ce7dc1/analysis/1331756265/

The exploit kits were located at the following URLs:

Windows Vista/7/8 systems were redirected to:

hxxp://svitart .in/index.php?adaebb3f91ea88ba4bc624b6625e5d52
hxxp://svitchudes .in/index.php?9a41e20f564247648add4e6f1ec18e65

Windows XP and other systems were redirected to:

hxxp://accbud .in/index.php?4cfcb816db07831247fe5e2db2878634
hxxp://accent6 .in/index.php?0acb5dbb36f93141b027d416261af42e

A successfully exploited system through one of the three vulnerabilities will download and execute the following url:

/server_privileges.php?<random>=<digit>

An example:

/server_privileges.php?36d3677b09fc971f6d37ac96b1d2cf56=3

This will load files from the system which are located on:

hxxp://accent6 .in/files/load1.exe (for Windows XP and earlier systems)
hxxp://accent6 .in/files/load2.exe (for Windows Vista/7/8 systems)

Both files, load1.exe and load2.exe were uploaded on the exploit kit server at 09:26 CET on the 14th of March 2012.

The IP addresses used for these exploit kits were 188.95.50.55, 188.95.50.56, 188.95.50.57 and 188.95.50.58 which were hosted in The Netherlands at serverboost.nl, and not in India as indicated by other publications. We provided the domains and the IP addresses to Spamhaus and also contacted serverboost.nl directly. Thanks to the quick work of Spamhaus, within a short time the exploit kit domains and the active and backup domain for the smokeloader bot were taken offline, and soon after the server itself was also taken offline which made the SmokeLoader botnet defunct, even if there were cached DNS entries. While the systems were still infected, the botnet was no longer under the control of the attacker.

The SmokeLoader Trojan is a resident downloader Trojan which has become more and more popular amongst cybercriminals since the end of 2011. Its basic purpose is to make sure the system remains infected and can be used to infect the system with other malware, other malware in the market are for example DLoader2010, MyLoader and Bredolab. This version was slightly different than previous versions we analyzed.

The Trojan main body is executed in a different way than most similar trojans, it creates a suspended svchost.exe using CreateProcessInternalA and not CreateProcessA, this is used to bypass userland hooking code used in some sandboxes, it creates a section in the current process, copies the Trojan body to it, and maps this section from the remote process. The Trojan body is a position independent block of code and data and not a full MZ executable. So it does not inject a complete decrypted Trojan executable in the remote process, but only the Trojan executable body which was also done similarly in the DLoader2010 malware family. It does not overwrite the existing svchost.exe binary completely in memory, but maps only the Trojan body in the suspended svchost.exe process and overwrites the entry point of the svchost.exe binary in memory with a push ret combination that pushes the location of the mapped Trojan body to the stack which causes the next return instruction to divert to the Trojan code block. The Trojan body decrypts itself and resolves its own imports.

The Trojan copies itself to the Application Data folder (Windows XP/2000) or AppData\Roaming folder (Windows Vista/7), and uses a filename based on the pVolumeSerialNumber, which it will xor with 0x6454263F and the last 6 characters of the result are used as the filename with appended .exe. To make sure the Trojan starts on boot It will create a registry entry pointing to the copied executable in:

HKEY CURRENT USER
Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
Stardock

The Trojan monitors this startup key and when it is deleted, it will add it again, using a watchdog thread. The Trojan code also keeps a file handle open to the executable to make it more difficult to remove.

This variant of SmokeLoader connected to the following urls for command and control traffic:

hxxp://prolesok .in /sml/index.php
hxxp://accedoproductmd .in /sml/index.php

The sent traffic is encrypted with xor and encoded by base64, but the downloaded binary is sent in plain over the wire. The decrypted traffic looks like the following:

cmd=getload&login=<botid>&sel=PiuPi&ver=5.1&bits=0&file=0&run=ok

The botid is calculated from the md5 of the computername with concatenated the pVolumeSerialNumber xor 0x6454263F, resulting in a 40byte string, which is used on the serverside to distinguish bots, which is stored in the cname column in the database, which is the unique key for the bot.

Extract from SmokeLoader database schema:

CREATE TABLE `bots` (
 `id` int(11) NOT NULL auto_increment,
 ...
`cname` varchar(80) NOT NULL,
...
`seller` varchar(5) NOT NULL,
 ...
PRIMARY KEY  (`id`),
 UNIQUE KEY `cname` (`cname`)
 ) ENGINE=MyISAM  DEFAULT CHARSET=cp1251 AUTO_INCREMENT=1;"

Extract from SmokeLoader PHP code:

mysql_query("INSERT INTO `bots` (`id`, `ip`, `os`, `country`, `time`, `cname`, `work`, `seller`, `bits`)
VALUES ('', '".$ip."', '".$os."', '".$country."', '".time()."', '".$login."', '1', '".$sel."', '".$bits."')
ON DUPLICATE KEY UPDATE time=".time().";");

Interesting is that the sel= parameter is the seller id which comes from the common use of having varying infection sources, in case you distribute your SmokeLoader binary via different methods or pay per install providers you can distinguish the number of loads of the SmokeLoader binary. In this case the seller nickname is ‘PiuPi’, which is embedded in the SmokeLoader binary, as you can see in the above database scheme the seller id is only five characters long. Interestingly the five characters match a user on AntiChat, a Russian language underground forum, which was compromised and its database leaked a few years ago. The full nickname is `piupiupo’ using a Yandex email address, a Russian free mail provider, and from the database leak we can see that in 2010 he logged in from a Russian IP address located in Moscow, which belongs to a consumer ISP.

Now comes the interesting part, the SmokeLoader C&C server distributed Sinowal, which has been around for a good 6 years now and has been active in The Netherlands from time to time since 2007. The SmokeLoader distributed a component known as ‘miniloader’, which downloads the installer component from the Sinowal installer server. On Windows 2000 and Windows XP it will install the MBR bootkit, which is used since the end of 2007 as the method of startup, which is also commonly referred to as Mebroot. Only a few security products are able to detect this modified MBR from a running system and even less are able to actually clean it. HitmanPro from SurfRight and Tdsskiller from Kaspersky are two products that were confirmed to remove the bootkit component on Windows 2000 and Windows XP systems. On Windows Vista and Seven systems the threat would install a userland component but we have not verified this during the nu.nl compromise.

We have investigated a couple hundred infections on corporate networks, but one odd thing appeared, the Trojan did appear to be malfunctioning, from all the infections we investigated, only one infection actually connected to the Sinowal command and control infrastructure that would indicate a successful infection. We are not sure why this happens and are unable to verify the reason for this. We have heard the same story from other researchers in the field and are not able to verify why this happens, but time will tell…

Michael Sandee et al

Principal Security Expert

23 thoughts on “Post mortem report on the sinowal/nu.nl incident

  1. What browser vulnerability caused the downloaded exploit kit to be executed? Or did the user have to click OK?

    • Hi Simon,

      There were a number of exploits used, the first one that was executed was the Java Rhino exploit, which caused the largest number of infections. There were already several writeups which detail the javascript and exploits used, so I did not reiterate this topic. Also I did not read those other writeups to verify their correctness and hence I did not link them from this blogpost. Unfortunately those writeups seem to be in Dutch mostly.

  2. Pingback: ‘Russische criminelen achter cyberaanval op NU.nl’ – GOnieuws

  3. Pingback: ‘Russische criminelen achter cyberaanval op NU.nl’ | NuNieuws.eu

    • The drive-by would not have succeeded, as the g.js or gs.js javascript on nu.nl might have worked, because users of nu.nl might have whitelisted the site, but the exploit kit also requires javascript and the loading of a java applet or PDF file, which all requires interaction when using NoScript. But, I have not tested it personally.

  4. Pingback: Russische crimineel faalt in opzet botnet via Nu.nl | Tech-nieuws

  5. Hi, thanks for the informative article. To be honest it’s the first time I have heard about this blog, but I’ll surely come back more often. :-)

    I would like to know, are only windows PCs at risk of being infected? And, is it enough to reinstall the boot loader on Windows PCs?

    Finally, a more subjective question — do you think nu.nl (and possibly others) reacted to the public in a good way? Is it easy for average Joe or Jane to check for infection and clean if infection has been found?

    Thank you

    • While it is impossible to verify everything, it is my expectation and observation that the exploit kit in question only affects windows based systems. The used trojans, SmokeLoader and Sinowal, are both suitable only for the Microsoft Windows platform, specifically Windows 2000 and later.

      Using a bootable media and reinstalling the MBR should work, however the proposed solutions by using tdsskiller from Kaspersky or HitmanPro from SurfRight would be better in a way. Especially HitmanPro, if you were infected at the nu.nl incident, because it means you have a piece of outdated software on your system which is vulnerable, and you might have been infected with a range of other malware during the past month(s).

      From my point of view, nu.nl did a reasonable job, well lets say they are not experiencing this every day, so given the circumstances I think they did not do a bad job. A few minor issues, such as the time the site was fixed was wrongly posted on the site which caused administrators at corporate networks to narrow down their searches while this was not actually correct information. As you can see from the above blog post the actual time frame where infections took place was 11:30 till 13:42 and not 11:30 till 12:30.

      The most important thing to note is, there are only few people which specialize in this line of work, especially within a small country like The Netherlands, and with these kind of cases the devil is often in the details. For us it is daily business, but for most of the people out there it is a very unusual and uncommon problem they are facing. But this incident shows again, everyone can be affected by this and it is difficult to handle these types of incidents and provide concrete advice without having all the background details.

  6. Pingback: Example of how malware/virus spreads | TeknoNerd

  7. Pingback: Russische crimineel faalt in opzet botnet via Nu.nl | SFIX – Advanced Security Services & ICT Solutions

  8. Pingback: NU.nl gehackt: Malware analyse | Weblog | Sijmen Ruwhof

  9. You end with your report with the text “Time will tell”.

    Do you have any idee how much time it will take te be sure this is harmless? We really would like to reassure our customers en move on to the ‘business as usual’

    • It is not a matter of time per se, there are many variables to verify, the reason we mentioned time will tell is because we cannot know everything but we do know that something was wrong in the cases we investigated. In time we will see the reports from other affected parties who might have additional information on this. A large number of systems was infected but did not get the real intended payload running for a number of reasons, some of those reasons we know and some of those we do not know. This makes it hard to apply this information to the total amount of infected systems, so while we can make a good estimate on how many systems are affected by the initial infection, we cannot say anything about the amount of real infections that is now active, but we are sure at least a percentage of the systems are active.

      Everybody who worries, do not read our blogpost in a way that it is an excuse to not do anything (you might conclude otherwise from some information in the press), if you were affected your systems are using outdated software, additionally they might have been infected with one or more pieces of malware causing a possible information leak. See it as a wake-up call. A number of URLs have been mentioned, if your proxy logs contain the the url which fetches the payload (executable) from the exploit kit, or if your logs contain references to the two urls we mentioned of the SmokeLoader command and control server, you should know that the system has been compromised. Verifying the additional Sinowal infection requires some more information that we cannot explain in a few words.

      A significant amount of Anti-Virus product vendors now recognizes it, but some remain that do not detect the SmokeLoader infections:

      https://www.virustotal.com/file/e625fdb49695886ee2781d6e2c3755f7fc8c9c56ca208bdd14e51f11466abe10/analysis/1332150024/

      https://www.virustotal.com/file/c667f39573b4bbd10aa26e841a9dda3ab0407de12606e9a58e16fc9427ce7dc1/analysis/1332150155/

  10. Pingback: Dutch Users Served SINOWAL for Lunch | Virus / malware / hacking / security news

  11. Pingback: Ceptera Security Newswire » Dutch Users Served SINOWAL for Lunch:

  12. Pingback: 《漏洞攻擊》荷蘭新聞網 nu.nl 遭入侵,專挑午餐播報時段啟動病毒,蒐集系統資訊 | 雲端防毒是趨勢

  13. Pingback: Exploit Kit plays with smart redirection (amended) | Security Antivirus Virus

  14. Pingback: IT Secure Site » Blog Archive » Exploit Kit plays with smart redirection (amended)

  15. Pingback: Умные редиректы на Nuclear Pack | | CopyBase.RU - Интересное из сетиCopyBase.RU — Интересное из сети

  16. Pingback: Nuclear Pack Exploit Kit plays with smart redirection | ESET ThreatBlog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s