Building Bowser – A password cracking story

At Fox-IT we perform a lot of penetration tests. Invariably we encounter hashed versions of passwords that need to be tested for strength. We suspected that with a relatively small investment most passwords could be cracked, regardless of their complexity. It turns out this is true for any password of 8 characters or less. 

This is a story about passwords. It’s intended for hardware lovers and IT security managers alike. The purpose is to change the views on what requirements a password has to fulfill to be considered “safe”. Basically, this story should convince you that conforming to the default Microsoft Password Complexity Requirements (http://technet.microsoft.com/en-us/library/cc264456.aspx) barely bothers hackers trying to crack your password, and explains why everyone should be switching to passphrases. Don’t bother with taking the first letter of all words in a sentence, just use the whole sentence. The longer the better. Cracking passwords in a short timeframe is something we were able to do on a fairly limited budget, but will scale up very nicely with a larger one. Imagine what someone with virtually unlimited funds (intelligence agencies, countries) is able to do to your precious password hashes.

Over the years we’ve done our fair share of penetration tests, both on external services and internal networks. This includes web applications, Windows- and Unix networks, mobile applications, etc. During these tests it’s not uncommon to come across hashed passwords in one form or another. While some protocols (like NTLM) are vulnerable to a technique called “passing-the-hash” (essentially using the hash itself as a password), sometimes you just want its plaintext counterpart. For the sake of consistency, this story will focus on NTLM (Windows) passwords.

One of the ways of obtaining the plaintext password is by “cracking” the hash. You can go about this using different techniques. The most basic of which being a “brute-force” attack, simply trying all combinations of letters, numbers, special characters, etc. and see if the resulting hash matches the hash you are trying to crack.

Another technique is using wordlists. These are huge lists containing hundreds, thousands and even millions of words depending on the size of the list. By simply hashing each word on the list one by one, and seeing if the resulting hash matches your target hash you can try and “crack” it this way.

Modern tools allow you to use these methods in smarter ways, by combining wordlists with brute-force techniques, or applying rules to words on the list (adding capitals, appending numbers or special characters, etc.). Still, this can be a time consuming project when using regular desktop hardware.

Enter the video card or GPU (Graphics Processing Unit). While designed for running games, these cards have power in all the right places for use in password cracking. I could go on explaining about the how, why and other technical mumbo-jumbo, but sometimes pictures indeed are better than words. This screenshot was taken from the password cracking tool hashcat (http://www.hashcat.net) and shows the speed of cracking an NTLM hash while running on two cores of an Intel i7 3840QM CPU:

Almost 34 million words per second (34MH/s). Sounds like a lot right? Well, it’s not, really. At this rate, trying all 8 character passwords (uppercase letters, lowercase letters, decimals and special characters) takes approximately four to five months.

Let’s try cracking that same password using oclHashcat, hashcat’s big brother, designed for use with GPU’s. This screenshot was taken while running oclHashcat on our old cracking machine with a single AMD Radeon 7970 GPU:

Wow, a single GPU can do almost 18000 MH/s. Compared to the 34MH/s the CPU can do, that’s around 53000% more per second. At this rate, trying all 8 character passwords will take somewhere around five days. We are impatient people though, so we wanted more speed.
This is when we decided to build a new dedicated server, designed for one thing, and one thing only: cracking password hashes. This new server, codename “Bowser”, would be a 4U server chassis from Tyan (FT72B7015), have two Intel Xeon CPU’s, two 1TB SSD’s, 4x8GB of RAM and most important, eight AMD Radeon R9 290X GPU’s. Time to place some orders!
The first things to arrive were the GPU’s. Just look at these beauties:

The next day, the CPU’s came in:

The rest however, took a while. The Tyan chassis wasn’t in stock and had to be back-ordered from Taiwan. Almost two weeks went by, until:

Finally! Time to start building. Instead of boring you with a wall of text, I’ll just let the pictures do the talking:

Quite something right? Now time for the hard part. Making the system secure enough for customer data. The OS we’ve chosen to install was Debian 7 at first, but it looked like the 3.2 kernel shipped with Debian 7 did not very much like our hardware. Random kernel panics and freezes everywhere. So, on to Ubuntu Server 12.04.3.

The first 1024MB of the first SSD was partitioned as unencrypted /boot partition. The first 1024MB of the second SSD was partitioned as unused space. This was done because the rest of both SSD’s were configured to run as a RAID0 volume. Why? It’s not business critical data, and large wordlists on slow disks really impact the cracking performance.

On the newly created RAID0 volume, a LUKS encrypted partition was made. Using LVM, a volume group was created on that partition. This volume group contained two logical volumes. The root file system (/) and some swap space.

After OS installation was done, it was time for some benchmarks! Firstly, we made sure that the BIOS switch on all the cards was set to “Über mode” (http://www.hardocp.com/article/2013/10/23/amd_radeon_r9_290x_video_card_review/2#.Uukqp3VdXZg). Then we installed a tool called od6config (http://epixoip.github.io/od6config/) to fix the fan speed of the cards at 100%, the GPU clock at 1000MHz and the Memory clock at 1250Mhz. Because a running X session is necessary to use the GPU’s, we also installed Ubuntu Desktop and the AMD 13.12 driver package.

The firewall was configured to only allow communication needed for remote management. All other incoming and outbound traffic was disallowed.
Finally, benchmark time! Feast your eyes on this:

145GH/s per second! That is a performance increase of around 800% compared to the single card. This means that we are able to retrieve all 8 character NTLM hashed passwords (uppercase, lowercase, decimal and special characters) within 24 hours! All the while, due to the insane cooling capabilities of the Tyan chassis, temperatures keep well within acceptable ranges:

Note: adapter 1 has a higher temperature than the rest, because this adapter is responsible for showing the desktop.
After some tuning, we managed to get a 100 MHz core overclock on all the cards. This resulted in a 20Gh/s increase in performance with a total of almost 190GH/s (while cracking a single NTLM hash):

That’s the equivalent of 9 of these cards running at stock speeds. The temperatures after about 19 hours seem almost unaffected. The card responsible for showing the desktop peaked at 76C, while the other cards average at about 65C.

It does take a considerable amount of power to run this machine though:

2500 Watts, 12 Amps. Nice.

I can hear you thinking “Why not just use rainbow tables and get it over with”. While it’s true that rainbow tables can be an effective way of cracking passwords, they are not a magical solution. First off, the tables themselves require a lot of storage space (for example: around 1TB for alpha-numeric passwords, length one to eight). Secondly they take a very long time to generate and every type of hash needs a new table. Lastly, with a large list of hashes to crack (say, a dump of all active directory passwords) the pre-calculation when using rainbow tables can take much longer than just using wordlists or brute-force attacks. In most cases, it is faster to just brute-force a hash then to use rainbow tables on it. Besides, the combination of wordlists, rules and brute-force can usually crack more than 80% of the hashes already. If you really need a specific hash cracked, you can always resort to rainbow tables after brute forcing the bulk of the hashes.

Well, that’s it. Feel free to leave any questions you might have in the comments, and happy cracking!

Donny Maasland, Pentester at Fox-IT