Chapter 10: The Art of Tracking – Stealing The Network

Chapter 10 The Art of Tracking

by Mark Burnett

Tuesday

Its 2:00 a.m., and I can barely keep my eyes in focus, much less keep my brain clear. It’s Tuesday now, but to me, it’s just a really long Monday. I stare at the painting on the wall across from my desk.

It’s strange that I’ve been sitting at this desk for a week now, staring at this painting, but I never actually looked at what I was seeing. In the painting, a middle-aged man stands with his back to me, looking out his front door. He is wearing a loose, light-blue shirt and pants that could be either his pajamas or some strange, oriental-looking outfit. Outside his door is a vast ocean—no land, just endless ocean. And it’s hard to judge exactly how far down the water is. Perhaps he could step out his doorway right into the water, or perhaps it’s 20 feet below the house. Either way, the painting makes me feel unsettled. I wonder why the painter gave the man those blue pajamas, why he painted the man’s house that ugly, pastel yellow, and why this man has ocean outside his front door. It almost looks as if the man himself is wondering these same things; as if he just barely discovered that out the front door of his pastel-yellow house is endless ocean.

I look back at my laptop’s screen and see that my query has completed. I searched for all IP addresses that hit the Web site more than 500 times in the last four months. The query returned 3,412 IP addresses, which sounds like a lot, but is much less than the total of 28,366 unique IP addresses that visited the site in the last four months.

I need to be more specific. I adjust my query to include only those IP addresses that sent requests resulting in some error code. I type in the new query and hit Enter.

Another thing that bothers me about the painting is that the angles are such that they make it look as if it hangs crooked, although it doesn’t (I know because I measured it). I wonder what’s better—having the painting technically balanced but looking crooked or tilting off-balance so that visually it looks straight. It bothers me that it looks tilted, but it may just bother me more knowing that it actually isn’t hanging correctly.

Time zones, IP addresses, and HTTP result codes—these are the leads I have as a forensics expert. I track down hackers and re-create what they’ve done. This particular contract is for my primary client, an insurance company. When a customer they insure is hacked, they call me in to investigate. My job is to figure out what the hacker did, and just as important, what the hacker didn’t do. My report determines if the company gets $10,000 or a $100,000 for their claim. Before the insurance company cuts a check, the managers want to know exactly who did what, how they did it, whose fault it was, and how they can prevent the problem from happening again. Adjusters, auditors, regulators, law enforcement, lawyers, and judges will regularly review and scrutinize my reports. I need to be accurate, meticulous, credible, and objective. But ultimately, it’s my client I need to gratify.

In this case, a hacker broke into a large software company, stole some source code, and posted it on the Internet. The company was able to get the source code removed from the site within a few hours, but the damage had been done. They paid a large consulting firm to get them secure, but the insurance company flew me in to do the investigation. For a week now, I have gathered every log file I could find, sanitized and normalized the data, and loaded it into my analysis database. I have very little to go by, and the log files are not as complete as I had hoped.

This was all triggered because, last month, a programmer had checked out his code for the weekend from the source control system. On Monday, when he went to return the code, he got a warning that his module was currently checked out by another user. Since he was the only one who worked on that module, he got suspicious and reviewed the source control log files. An administrator connecting from their SQL Server had checked out the module. In fact, this administrator checked out all the modules. The next day, a customer tipped them off that their source code appeared on a public Web site. They brought in this consulting firm to get them secure. These security consultants completely rebuilt much of their network and made some changes in their Web application. Unfortunately, in doing so these consultants also destroyed much of the evidence, and I don’t know exactly how the network looked at the time of the intrusion.

I do have four months’ worth of IIS log files, which are better than nothing. I actually suspected the Web server as the point of intrusion. The Web server was the most direct route to the SQL Server. Otherwise, they would have to penetrate the DMZ firewall, then the internal firewall, and then try to break into the SQL Server. Ironically, this company thought they were being more secure by placing the SQL Server on their internal network. While this did make the SQL Server slightly more difficult to attack directly, it also allowed the SQL Server to see the internal network. If you can get to the SQL Server, you can get to the whole network. This mistake cost the company a vital piece of intellectual property: their source code.

I figured this would be a quick job—an obvious SQL injection exploit. SQL injection is the manipulation of HTML form input in such a manner that it allows an attacker to submit any SQL command, including stored procedures that allow the execution of operating system commands. The hacker likely found some vulnerable Web form, figured out how to manipulate the SQL statement, and then used the SQL Server to attack the source control system on the internal network. With limited log files, it would be extremely difficult to circumstantiate this theory.

The first problem was that the server no longer exists. This company had three data centers, each in a different time zone, and the administrators often transferred servers between these data centers. Consequently, I had no way of knowing what time zone was set on the server, or if the server’s clock was even accurate. My evidence would be completely useless if I did not have any proof of timing.

To determine the server’s time settings, I needed to correlate any event in the IIS logs with an external event. I did have a single log file from an intrusion detection system (IDS) that an administrator ran for one day after discovering the intrusion. The system used to run the IDS was still around, so I could verify its time zone setting: GMT-05:00. This would become my baseline system. By comparing that log with the IIS logs, I was able to find two separate events that appeared in both. The recorded time difference between matching events was five hours and eight minutes. The eight minutes I could attribute to inaccurate server clocks, and the rest I could calculate.

IIS always logs events using Greenwich Mean Time (GMT). However, IIS determines GMT by taking the server’s time zone and adding the appropriate number of hours. For instance, if the server is set at GMT-05:00, then IIS will add five hours to the system time to determine GMT. IIS also has an option to cycle log files at midnight local time rather than midnight GMT. If a log file cycles at GMT, the first entries in each log file will be recorded around midnight (0:00). If the log files are cycled using local time, the first entries in each log file will be some offset of midnight, GMT. Opening a random IIS log file, I saw logs entries such as these:

Since the logged events were starting each day at around 05:00, I could calculate that midnight local time was equal to 05:00 GMT. Therefore, the time zone was GMT-05:00. But GMT doesn’t follow daylight saving time (DST), which occurs between the first Sunday in April and the last Sunday in October. Since this was October 14, the local clock was one hour ahead. Hence, the local server must have been configured for GMT-06:00.

Using a copy of the IIS logs I loaded into a database, I adjusted each time field to exactly correspond to the IDS logs. But this was the easy part.

To save disk space, the IIS logs were configured to log minimal information, which does not include the query string. Not having this information would make it very difficult to prove SQL injection. I would have to dig deeper. To start, I would need to figure out which log entries were suspicious.

As a forensics expert, I find my self viewing the world not as continents, countries, and cities, but as class A, B, and C networks in the IPv4 address space. An IP address like 194.95.176.5 feels much different to me than an IP address like 217.22.166.29; the first is definitely German, while the latter feels more Russian. IP addresses that start with numbers like 24, 65, 66, 209, and 216 are most likely from the U.S.; 202 and 203 from Asia; and so on. Much of my time is spent looking at huge lists of IP addresses and identifying which are friendly and which are hostile. Among the hostile IP addresses are two classes: a hacker’s real IP address and the addresses of innocent—if you can call a lame administrator innocent—systems taken over by hackers.

I classify IP addresses by the traffic they produce. If they always produce legitimate traffic, they are friendly. If they always produce malicious traffic, they are hostile. The trick is classifying all the IP addresses that fall somewhere between friendly and hostile. To give me a head start, I have a collection of IP addresses that, at least at one time, were known to be hostile. I also collect underground lists of public proxies, SMTP relay servers, and IP addresses of people hanging out in hacking-related chat rooms. My database will flag any log entry that matches any of these lists. The system does have its flaws. I can’t make conclusions from these lists, but they do help to narrow my research.

I keep a separate list of IP addresses used by one particular hacker I have tracked for some time now. I don’t know his name or his real IP address, but I know his work. At first, I thought he might be involved with this hack, because it just sounded like something he would do. But I haven’t been able to find anything that correlates to any other IP addresses he has used. In fact, I really haven’t found much of anything pointing to anyone.

Looking back at my laptop, I see that my last query has finished, but the results still tell me nothing. I want to quit for the day, but my client is expecting a report tomorrow. I have nothing to report—no suspicious log entries, no hacker’s IP addresses, and no evidence of SQL injection. I doubt they will be too impressed with how I figured out the server’s time zone. I have queried everything I can think of—most active IP addresses, IP addresses grouped by class B and C networks, unusual spikes in traffic, large numbers of 404 or 403 errors, and large numbers of hits in a short period of time. I have stared at raw log files for so many hours that all the dates, IP addresses, and URLs are beginning to blend. It’s like staring at static on a TV screen.

But if you stare at static long enough, you might begin to see patterns emerge. And that’s what forensics is all about: finding the patterns that lead you to the hacker. Every bit of information in a log file, as meaningless as it looks, is valuable. Each millisecond of time, each result code, and every variation from the norm can be the piece of information that leads me to the hacker.

I read the numbers and words aloud over and over, waiting for something—anything—that stands out. Thirty minutes pass as I go through page after page, reading aloud the random bits of data scattered in hundreds of megabytes of logs.

“15:49:05…97.201.18.5…GET…109.12.98.82…POST…login.asp…200…checklogin.asp,” I whisper to myself. The numbers sweep through my mind, spinning me around in my chair, lifting me up from the floor.

“302,” I say out loud, “redirect.”

The pen in my hand drops to the floor, and my head falls back in my chair. I know I’m fading to sleep. Despite all the caffeine and sugar I have consumed, I cannot muster the energy to stop myself. I fall into a dream world where log entries, dates, and IP addresses seem so much more clear and concrete, yet with a strange abstract importance, as if each one were some kind of living being. I ponder the peculiarity of it all.

“Excuse me…” a female voice suddenly breaks in.

I know I’ve heard something outside my dream world, and it takes me a moment to realize I need to wake up. Confused, I open my eyes and see a woman standing at my office door.

“Can I get your trash can?” she asks in a slight accent, probably somewhere in the 200.0.0.0/8 range.

“Oh, okay,” I try to respond, but the words never make it out of my mouth.

I clear my throat and fumble for the trash can, gathering up some papers on the floor. I am suddenly struck, as if someone grabbed me from behind and violently shook me out of my daze. One of the papers I am holding has nothing but five log entries:

“Oh, duh!” I exclaim, staring at this paper and forgetting momentarily that the cleaning lady is waiting for the trash can.

“Oh, sorry,” I say as I hand it over to her, placing the extra papers in the can, but keeping hold of the one that caught my attention.

“GET…200,” I whisper, “and it’s him.

In the three years I have worked in Internet security, I have learned a lot about hackers. Hackers go through stages as they develop their skills. At first, they want to impress others and be accepted. Consequently, they do lame stuff like defacing Web sites and boasting of their hacks in public chat rooms. This is the stage where many hackers get caught, although they are usually scared enough to take some measures to conceal their real IP address. As their skills increase, they move onto more sophisticated hacks and become a little more subdued—bragging only to their close circle of friends. Yet, something strange happens at this point. They gain this superhuman ego and begin to think they’ll never get caught, so they attempt bold attacks from their own IP address. Eventually, if they still haven’t been arrested, they become master hackers and confide in maybe only one other person. Oddly enough, master hackers once again take care to conceal their identity, but now they do it because they’re wiser, not because they fear.

You can tell how skilled hackers are by what tools they use. When they start, they use some publicly available tool. As time goes on, they begin to customize the tool to make it stealthier or more effective. Eventually, they develop their own set of custom tools. The funny thing is that they probably don’t realize that the more custom their tools and the more refined their techniques, the easier it is for me to profile them.

This particular hacker I have been pursuing is beginning to make the transition to master hacker, but I know he is still arrogant enough to use his real IP address. I just haven’t found it yet. My hunt for him began 18 months ago, when I was called in to investigate an intrusion at a large university. Someone discovered a password cracker running on one of their servers, which resulted in a major security audit. The insurance company flew me in to do my own investigation. The university’s network was such a mess, I couldn’t imagine how anyone—whether hacker or administrator—could ever find anything. There were plenty of holes, and the hacker apparently saw the university’s disorganized but high-bandwidth network as a good launching point for other attacks. Through my investigation, I gathered mounds of evidence but could never produce anything conclusive enough to pass onto authorities. Still, this was only the first of several encounters I would have with this hacker.

During my investigation, I found a suspicious file in one of the Web server’s content directories. It was a custom script that allowed an attacker to upload files to the Web server. When the investigation ended, I continued my research. Using search engines, I found another Web server that had the same file. I contacted this company, and the managers let me take a look around their server.

A month later, I read about an e-commerce company that was hacked. The method described sounded similar to the work of my hacker. I called them and offered my services. They weren’t interested in hiring me, but they did share some information they had gathered. By studying these intrusions, I learned that this hacker often took over the systems of insecure cable-modem users. Doing my own probing, I found that these systems were usually Windows boxes with blank administrator passwords. I even broke into some of these systems myself, hoping to gather more evidence. All I needed was his real IP address. I knew it was recorded somewhere. The trick was correlating it to the attacks. I gathered the IP addresses of systems he had hijacked, along with proxy servers he had used. With each intrusion, my ability to spot his work improved—the better he got, the better I got.

What grabbed my attention in these particular log entries was the IP address. I recognized it as one of the many my hacker had commandeered. What struck me next was the 200 HTTP result code.

HTTP result codes record how the server handled the request. A 404 code means a file wasn’t found. A 302 code means a request was redirected. A 200 code means the request was handled successfully. The interesting thing here is that the previous request to checklogin.asp had a 302 result, but this request returned a 200 code. Looking at the source code for checklogin.asp, I saw the following:

There were some obvious problems here. First, it doesn’t filter form input and is vulnerable to SQL injection. Second, it uses the generic Request object instead of specifically requesting the Request.Form object. What this means is that anyone can send the user and pwd parameters either through a form or as part of the query string, like this:

This is significant, because such a request will show up in the IIS logs as a get request rather than a post, as my log entry showed:

But, the question remained: Why was I seeing a 200 result code? Following the logic of checklogin.asp, a username and password could either match or not match. If the username and password matched, the user would be redirected to menu.asp, resulting in a 302 code. If either the user-name or the password were incorrect, the client would be redirected to login.asp, also resulting in a 302 code. The only other possibility I could think of was an ASP error, but that would show up as a 500 error in the logs. At least, I assumed it would show up that way.

Assumption—it’s one of the worst things when investigating an intrusion. I have been burned by assumptions—mine or those of others—so many times that the word itself sends up a red flag whenever I say it. I have learned that I need to double-check everything.

So, I browse to the company’s test Web server and force an error by entering invalid data in the login form. The response is exactly what I would expect:

I open the IIS log files, and there it is: 200. Even though the ASP page returned an error, it wasn’t an ASP error. I try the same thing on my own Web server, and I don’t get the same results. But on this server (perhaps it’s the ODBC driver), I get a 200 result code. And that’s all I need. The only way to get a 200 code on this page is if an ODBC error occurs. All I need to do now is find all requests that match those criteria. I construct a new query in my database and hit Enter.

And there it is: a complete list of IP addresses that tried this. The reason I couldn’t find this stuff before is because the 200 made the traffic look legitimate. I cross-reference the IP addresses, and sure enough, it’s definitely him.

Now that I have all the IP addresses he used, I take each and build another query to see what else he did. An hour ago, I had nothing to go on. Now, I have hundreds, possibly thousands, of log entries. I print them (10 pages’ worth), lean back in my chair, and stare at them to see what patterns emerge. Immediately, these entries catch my attention:

Why was he suddenly getting 500 errors? Perhaps it’s a CGI script timeout. Each entry is about five minutes apart, and the default CGI script timeout in IIS is 300 seconds. Suddenly, I realize that this checklogon.asp script doesn’t return anything, so he won’t be able to see the results of any commands he sends. Somehow, he will need to send the results back to his PC. Once, I saw a hacker who actually had SQL Server e-mail him the results. I do have the company’s SMTP logs, but I see nothing suspicious occurring during that time period. And no e-mails have ever originated from the SQL Server box. I’ve heard it suggested that data could be returned as part of an ICMP echo request, but I know this guy, and he’s too lazy to bother with something like that.

Then I realize that no matter what method was used, it would involve establishing some kind of TCP/IP connection. But there’s nothing that would have recorded outgoing connections. It’s likely that the SQL Server has made few outgoing TCP connections, so on a long shot, I type the following:

DNS caching is a Windows 2000 client service that caches the most recent DNS queries for a period of time so it doesn’t need to perform another lookup to resolve the same hostname. The cool thing about this service is that it also keeps a handy record of what names have been recently resolved on the system. For the most part, the results are what I would have expected:

But there was one entry (not shown here) that seemed quite suspicious: the DNS name of an ISP in Brazil. Is it possible that I’ve finally discovered his IP address? Not just some box he had seized, but his real IP address? The first thing I do is perform some searches on the IP address, just to see what turns up. I perform a WHOIS query at www.arin.net, to see who actually owns the IP address. It refers me to www.lacnic.net, and I check http://www.geobytes.com/IpLocator.htm to see if I can determine his physical location. I also run some searches on Google (both Web and Usenet searches). It turns out the IP address is an ISP’s Web server. Another false alarm—it’s just an open proxy server.

Still, I search for that IP address in the IIS logs, and I find a single log entry coming from it. Even more interesting are some log entries immediately following:

This is a classic “check-this-out” event. What happens is that someone does some cool hack, and a couple minutes later, he tells some buddies in a chat room to check out what he just did. Next, you see several distinct IP addresses hitting the same URL within a very short time. These events are extremely important in a forensics investigation, because they allow me to make a relationship connection. Not only does it associate an IRC nick with an IP address, but it also tells me who else this hacker associates with.

IRC monitoring is particularly fun. I have spent hundreds of hours developing a custom IRC monitoring tool. This tool connects to IRC networks all around the world and searches for lists of IP addresses I provide. And it does it over and over, for as long as I keep the program running. After a few days, I can usually find at least some of the IP addresses I’m looking for. For now, I enter the four IP addresses I found in the logs and click the Connect button.

The program spawns several application windows, each with raw IRC traffic scrolling so fast that it’s hardly useful (but looks extremely cool). In the main results window, I already have two matches. Each time it gets an IP address match, it performs a WHOIS lookup for that nick. The program does generate many false matches, but the two users it found are sitting in the same chat room, #haxordobrazil.

Of all the skills required of a forensics expert, few are as important as the ability to speak (or at least read) as many foreign languages as possible. I speak Italian and Spanish fluently enough to convince a native speaker that I, too, am a native speaker. I can sufficiently communicate in Portuguese, and somewhat less French. I can’t speak German, but I can understand about 50 percent of what I read in German. The next language I would learn is Russian, but for some reason, it intimidates me. For other languages, I have enough friends in enough countries for most of what I encounter. For what’s left, there’s http://babelfish.altavista.com.

#haxordobrazil, hackers from Brazil—Brazilian hackers. I’m getting closer.

I seriously consider joining the IRC channel, but realize that I could completely spoil my investigation if they realize someone is on to them. For now, I keep my IRC logger running.

At least, now I have something to report to my client. And just in time, because it’s almost 9:00 a.m., and people are beginning to arrive for a new day. Here I am, my eyes so red I need to wear sunglasses to bear the brightness of my monitor, wearing the same clothes and sitting in the same seat as I was yesterday when everyone left for the day.

“I can’t believe I actually found him,” I tell myself. I get up to close my office door, then settle in to my chair and close my eyes for a short nap. Finally, I can sleep.

But not for long. An hour has passed, but it was hardly satisfying. I hear two quick knocks at my office door.

“So what have you got? Didn’t you go back to your hotel last night?” he asked. He was the CIO for the software company, my boss for the couple weeks of this investigation.

“What, and miss out on all the fun here?” I respond, “I do have some good news. I found the hole, but I still need to gather some notes. I’ll go into more detail at our meeting.”

My voice must have an obvious slur, because he gives me a questioning look. Just then, one of his employees approaches him with an apparent emergency. He looks back at me, gives me an “okay, let’s talk later” wave, and walks away.

That day went by fast. We had a meeting and talked about what to do next. I was informed that they suspected the hackers still had access, which was probably the emergency earlier. We reviewed some strategies, I talked about the SQL injection bugs I saw in the source code, and I wrote some reports. Later, we had some more meetings, and I wrote more reports. That day, at 5:00 p.m., I rushed out with everyone else.

Wednesday

I don’t remember actually falling asleep, or even laying down on my bed. I just wake up the next morning, still wearing the same clothes I’ve had on for the past 48 hours. But I feel great.

In the shower, I think about my strategy for the day. I need to find some solid, credible evidence I can hand over to authorities.

Evidence is tricky. I’m in a strange position, because I’m not law enforcement, but I’m also not a normal part of this company’s business. If I want to start logging more information or install an IDS, I write up a policy and have the company establish it as a regular business process. If I just go in there and use all my tools to gather evidence, especially doing it in anticipation of legal action, the evidence I produce loses credibility and could potentially be deemed inadmissible in court. But to collect information I can use to gather clues, I do whatever I want. Today, I’m going to put a Snort box on the network and watch for those IP addresses. I’m also going to add some rules to record all the X-FORWARDED-FOR HTTP headers that proxy servers sometimes add. Unfortunately, IIS doesn’t log custom HTTP headers, but a simple Snort rule gives me a wealth of information.

Back at the office, I settle in and glance through my e-mail. I am shocked when I read my first message:

My stomach sinks, as a million questions race through my mind. How could he possibly have known? Where did he get my e-mail address? Is he an insider? Does he have an accomplice on the inside? What else does he know about me?

Just then, I hear two quick knocks on my office door, followed by, “Hey!”

It’s the CIO. My face must show my distress, because he quickly asks me, “Dude, what’s wrong?”

“How many people know I’m doing this investigation?” I ask him.

“I don’t know, maybe five,” he answers.

“Do you trust those five?” I inquire.

He is about to answer, but pauses, as if he just remembered something that would cause him to question how much he trusted everyone.

Before arriving at an investigation, I always make sure the client is careful to not tell everyone what I’m doing there. I never know if I’m investigating an insider job, and I certainly don’t want an insider to be warned of my investigation. Once I was hired to investigate an employee for corporate espionage. One of the managers sent an e-mail to the other managers, making them aware of my investigation and asking for their full cooperation while I was there. Unfortunately, the guy I was investigating was one of the managers who received this e-mail. When I got there, his laptop had been securely erased, reformatted, and reinstalled.

“Well,” I tell the CIO, “we have a problem here. This hacker has my e-mail address. Any ideas how he got it?”

I explain the situation, and he leaves to go talk with the company VP. The first thing I do is check out my own Web and mail servers to make sure nothing there has been compromised. There is no sign of any intrusion.

Then I realize that I have communicated with various employees via e-mail, and perhaps he has somehow intercepted someone’s e-mail. I wonder if all the company passwords were changed after the break-in. One of the first things people do after an intrusion is change passwords, but usually they change only a few key passwords, failing to realize that the intruder could very well have acquired hundreds of other logins. In fact, it doesn’t really help much to change only selected passwords after an intrusion, because if the intruder has just one way back into the network, he can easily discover all the other passwords again.

I talk with the CIO, and we decide to do a password sweep of the entire company. It takes the rest of the day and well into the night. We change every domain account, every local administrator account on every PC, and every router and switch account. We change hundreds of external accounts, including those for domain registrars, payment processing services, online banking, and so on. We even have all the employees change their personal Hotmail and instant messenger passwords. I’m actually quite surprised how eager all the employees are to participate in this, and many of them bring often-overlooked accounts to our attention.

I also change all my own passwords.

When we’re finished and most people have left, I sit down at my laptop to write this guy the response I’ve been composing in my head all day. Being so upset earlier, I failed to realize how useful it was to have some kind of communication with him. At least now I have a name for him, Daddo. It’s kind of a lame name. I guess I had hoped for better. I write up my response:

It was hardly five minutes before I got the response:

He’s trying to sound tough, but he must be scared. How could you not be scared knowing that someone is getting paid just to find you? Nevertheless, I, too, am a bit scared. I know the skill level of the hacks he has already done, but I also know he’s lazy. How much better would he be if he were motivated enough? Just to be sure, I add a couple more rules to the IDS sensors on my own servers.

I save the two e-mail messages. They may serve as evidence later, although by looking at the headers, I see that he apparently used a proxy server to send them. I pack up my laptop and head back to the hotel. On the way out, I notice sticky notes on nearly everyone’s desk—all the new passwords. I hope we trust the cleaning lady.

Thursday

The next morning, I get to my office and see a brown package on my desk. For a moment, I wonder if this guy would actually try sending me a mail bomb. But it’s not a bomb. It’s a hard drive from the company’s West Coast colocation center, where the main Web site used to operate. Over the past year, they’ve been moving their data operations from a colocated facility to their own in-house data center. They made the final transition just a month before the break-in occurred. However, they never took down the old servers; instead, they just updated the DNS entries to point to their new data center. This is the hard drive from the old Web server.

I unpack my drive imager and try to find a place to plug it in. The five outlets on my power strip are filled with two laptops, a scanner/fax/printer device, a hub, and a paper shredder—all essential equipment for a computer investigator. After hesitating for a moment, I decide to pull the plug on the paper shredder. I set the drive on the drive imager and wait for it to do its job. I am told this server was shut down immediately after the break-in and never used since.

One of the biggest problems I face in my investigations is the corruption of evidence. Few administrators know what to do when they get hacked, but most administrators feel compelled to do something. Usually what they do is wrong. Even many security experts unwittingly corrupt evidence.

Once I was called to investigate an intrusion where a bank’s Web server was used as a warez dump. A system administrator, trying to act prudently, immediately deleted the entire warez directory. He then notified the Chief Information Security Officer (CISO) of the intrusion. Eventually, I was called in. When I arrived, the CISO informed me that he had immediately taken the server offline and did some investigation of his own. He had also moved the log files to his own PC. There, he went through and put asterisks before any log entries that he thought looked suspicious.

“I burned this all to a CD,” he said as he handed me gold CD in a clear, plastic case.

“Oh, and I ran a backup right after the intrusion to preserve any evidence,” he explained.

“Great,” I said, but my heart sank. I didn’t want to get too angry with him, because I’m sure he meant well, but most of our evidence was now spoiled.

“You documented all this, right?” I asked.

“No, but if you need that, I can,” he responded.

“Why did you move the log files from the server?” I questioned.

“Well, we didn’t want to lose them when we reformatted,” he told me.

“Great,” I said again.

What frustrated me is that this guy really had no clue how much damage he and the other administrator had done. By removing the warez directory, they wiped out any evidence that a crime was committed. Perhaps I could have recovered that data, but they reformatted the drive and reinstalled the server, which was then actively used. I wasn’t likely to be able to find anything on the disk after that. The log files were largely useless as evidence, because there was hardly any proof that they were authentic. Besides, he had already gone through and modified the data by adding his asterisks. Of course, this changed the last-accessed and last-modified dates of the files. But that didn’t matter, because the backup process changed the last-accessed dates for every file on the system. And I guess none of this really mattered, because the system no longer existed anyway.

My advice to all administrators is this: If you don’t know how to handle evidence, then don’t handle evidence. A hacked server is a crime scene. If you encountered a dead body, you wouldn’t break out a kitchen knife and start your own autopsy. You would call the police. If you are an administrator and you get hacked, pull the plug on the server, remove the hard drives, and place them in a physically secure location. If you need to use the server, buy some more hard drives, and you can put it back into service.

Some forensics experts don’t agree with the advice to pull the plug on a victim machine. They argue that this could potentially cause loss of data. While this may be true, I personally prefer to pull the plug, at least with Windows servers. Keep in mind that many Windows servers are configured to wipe the swap file or possibly run scripts when they shut down. Furthermore, the shutdown process inevitably creates event log entries that could potentially overwrite older event log entries. If you just pull the plug, the server is exactly how it was at the time the intrusion was discovered. Keep in mind that I’m talking about only when a server you own has been hacked. There are many other situations, such as when law enforcement performs a raid, that require other techniques.

Once the server is secured, don’t make backups, don’t boot it up again, and don’t mount the drive in another PC to make copies of data. Speaking of backups, if you already do have backups for the server, pull those tapes from your backup rotation and secure them along with the server. Don’t just pull the most current backup, but also get all backups you have for that server. These backups can provide a vital history of file activity on a server.

I look at the drive imager and see that it’s only about a third of the way completed.

“Brazilian hackers,” I say to myself.

I still want to join that IRC channel, but I don’t have enough evidence to do something that risky.

Eventually, the drive finishes imaging. I mount the imaged copy in an external USB drive bay and plug it into my laptop. First, I want to see the IIS log files.

In the log files directory, the first thing that catches my eye is the number of log files—almost a thousand. I also notice that the logs continue almost until the server was shut down, about a month after the DNS was changed to point to the new data center. I open the last log file, and I’m very surprised at what I see: They logged the query strings on this server.

This particular log file is mostly filled with Nimda and script kiddy scans. I close this file and look for the largest file in the last month the server was up. There are several that are significantly larger than the rest. I open the largest and see before me a log entry that I’ve seen all too often:

Directory traversal—this is bad. Apparently, the server was not patched. I can tell from the 200 result code. Once a server is patched, a 404 is returned. What’s interesting here is that they used the _vti_bin directory instead of the more commonly seen scripts directory, which was smart.

This Web server was configured with separate partitions, a common security practice. Doing this is supposed to prevent directory traversal vulnerabilities. And normally, it will. Anyone trying a directory traversal exploit on this server using the scripts directory would get a 404 error, making them think the server is not vulnerable. However, the server is vulnerable. Because the Web root is on a separate partition, you can’t traverse up to the c:\winnt directory. So, it returns a 404: File not found. This actually throws off many hackers. But not this guy. When the FrontPage server extensions are installed, they are mapped to a directory on the system partition, and there is no way to change that directory. If the server extensions are installed and a server is not patched, then you have problems.

I browse through the logs with amazement. I now know exactly what he did. The funny thing is that after the DNS switch, most of the log entries are his. Apparently, he was attacking the server using its IP address rather than the hostname. When the host record changed, he was the only one still using the old IP address. This certainly saves me much time sifting through log files.

If I cut off everything but the query string, I get a complete shell history of every command he entered and, if I look closely. I can even see some that he typed wrong:

It looks like he had trouble using TFTP to get his files, because that port was specifically blocked at the firewall. You can see the different commands trying to diagnose the problem. I have a couple more IP addresses to add to my list.

I also notice that some log entries contain ODBC errors:

The list goes on with hundreds of ODBC errors, again documenting nearly everything he did. And he did a lot. Based on this new evidence, I know that he saw directory listings, viewed ASP source code, accessed the database, learned database connection passwords, mapped the network, and so on. At this point, all he could really do was access the IIS and SQL Servers at the colocation center. But with the information he gathered, it probably didn’t take him long to penetrate the server at the new data center. And by doing that, he gained access to the corporate network.

I really have all the evidence I need concerning the intrusion itself. Now, I just need to figure out who this guy is. I start up my IRC monitoring tool and enter the two new IP addresses. It spawns a few new windows, scrolling IRC traffic faster than I can read. Then one entry appears in the results window:

I can’t do anything but stare at my monitor. I found him. I actually found him! I knew he was still arrogant enough to use his real IP address. Now it’s time to join IRC. Before I do that, I send him an e-mail:

I send the e-mail and wait for his reply. I know he’s online, so it shouldn’t take long. After about 10 minutes, I get his response.

That’s my cue. I connect to IRC and join the channel:

Then there is a pause, as if I just walked up to a group of people gossiping about me. I can almost sense everyone in the channel sitting there looking at my nick.

Yes! That felt good.

I spend a few minutes to type up another taunting e-mail and click Send. My mail client hangs for a minute, and then returns an error: Connection refused. I try to ping my mail server:

So, I try pinging Yahoo, which works fine:

Suddenly, my mail client plays the sound it does when I have new mail. Okay, I guess it works again. There’s a message from Daddo:

He’s afraid. And I’m afraid. I guess I need to be ready for a defense if I’m going to go on the offense. I spend the rest of the day hardening my mail server. I block ICMP at my firewall as well as TCP connections with source ports of common services. I also block all unassigned IP addresses, based on the Bogon list at http://www.cymru.com/Documents/bogon-list.html. Just in case this isn’t enough, I configure my Snort sensor to log all incoming traffic for the next few days. It will take gigabytes of disk space, but it’s a good precaution.

Before I quit for the day, I send an e-mail to a good friend in Brazil:

I try not to get too shady with my investigations. I hire other people to do that. Basilio is an excellent hacker, and an IP address is all he needs. I pack up my laptop and head back to the hotel. My head hurts, and I’m exhausted. But I can’t sleep. Soon, I’ll get him.

Friday

The next morning, the CIO catches me in the hall.

“Hey, your friend sent some of us a threatening e-mail,” he tells me, “Daddo, or whatever his name is.”

“Yeah, he’s been sending them to me, too,” I respond.

“Are you close to finding him?” he asks.

“Yes,” I answer confidently, “very close.” “Send me a copy of the e-mail. Be sure to send the raw message so I get the headers, too.”

I’ve been collecting the headers from each e-mail Daddo has sent me. He uses a free Web-based e-mail service, but is always careful to use a proxy server. He must keep a list of proxies, because each e-mail has a totally different IP address. These IP addresses are all important, so I always save them.

By the time I sit down at my desk and boot up my laptop, I’ve already received the CIO’s e-mail. I open the attached message to check the headers, but one header in particular looks strange:

I’ve never seen two IP addresses in the header before. Normally, that field contains the IP address of the person sending the e-mail. How can someone possibly send an e-mail message from two addresses? Unless maybe one is a proxy. It looks like Daddo made another mistake. Just to be sure, I create an account with this Web-based service and find myself a proxy that uses the x-forwarded-for http header. Sure enough, I get the same results in my own headers: my real IP address followed by the proxy server’s IP address.

I do a DNS lookup on the IP address and discover something that completely changes the course of my investigation: Daddo works for a well-known Internet security consulting firm in Brazil. He’s both a black hat and a white hat—a gray hat.

It’s strange how hackers’ minds work. You might think that white hat hackers would be on one end of the spectrum and black hat hackers on the other. On the contrary, they are both at the same end of the spectrum, with the rest of the world on the other end. There really is no difference between responsible hacking and evil hacking. Either way, it’s hacking. The only difference is the content. Perhaps that’s why it’s so natural for a black hat to go white, and why it’s so easy for a white hat to go black. The line between the two is fine, mostly defined by ethics and law. To the hacker, ethics and laws have holes, just like anything else.

Many security companies like to hire reformed hackers. The truth is that there is no such thing as a reformed hacker. These hackers may have their focus redirected and their rewards changed, but they are never reformed. Getting paid to hack doesn’t make them any less of a hacker.

Hackers are kind of like artists. Artists will learn to paint by painting whatever they want. They could paint mountains, animals, or nudes. They can use any medium, any canvas, and any colors they wish. If the artist someday gets a job producing art, she becomes a commercial artist. The only difference is that now she paints what other people want.

With commercial hackers, it’s almost like they think the definition of a white hat is never having been caught. I’m not saying that all white hat hackers are bad. I’m just saying you should know whom you’re dealing with.

Okay, so now I know his ISP, where he works, and where he hangs out on IRC. My next step is to find a name. The security company has four consultants. He could be any of them. Maybe I could figure out the ISP each of them uses. Gathering their company e-mail addresses is pretty simple. A quick search of their Web site turns up two, and a Bugtraq search turns up two more, plus the personal e-mail address of one of them.

Finding e-mail addresses is surprisingly easy, as long as you know where to look. I now know the company e-mail addresses of the four consultants, and I know which ISP owns Daddo’s IP address. I can use that information to find any correlation between the two. The most obvious place to find e-mail addresses is a regular search engine. Next, there are plenty of Web sites for finding people: people.yahoo.com, bigfoot.com, anywho.com, infospace.com, whowhere.com, and more. If I still don’t find anything, I can check sites like classmates.com, reunion.com, and alumni.net.

MIT has a database of everyone who has posted a message to a Usenet newsgroup. To find out how to query their database, send an e-mail to mail-server@rtfm.mit.edu, with the following in the message body:

You can also search Usenet posts at groups.google.com. The nice thing about Google’s Usenet search is that you can view the raw headers of any posts, potentially revealing their IP addresses.

Unfortunately, people don’t always use the free e-mail accounts that their ISP provides. This way, they avoid having to change e-mail addresses every time they change their ISP. Using all these techniques, I find only one personal e-mail address, and it’s the wrong ISP. At this point, I need to step back and look at my options.

One thing I could do is take the evidence I have so far and turn it over to the FBI. Often, I do just that when I’m at this point in the investigation. But doing that also cuts me out of the loop. Since the FBI can’t share information about an ongoing investigation, those investigators won’t tell me anything they discover. Another problem is that once I get them involved, I have more limitations on what I can do. For example, if an FBI agent asks me to do something, I’m acting as their agent, and I’m now subject to their rules of investigation. Another reason I don’t want to pass this onto the FBI is because I doubt that they will do anything with it. This guy is outside the country, and that makes it more difficult for them to subpoena ISP records. Besides, I have tracked this guy too long to let someone else get the credit for finding him. This is personal.

I decide to give Basilio some time to do his exploring. He responded to my e-mail and said he would take the job (although he upped the price to $2,500). I send him an e-mail telling him what I know, including where Daddo works. This was his response:

I write reports the rest of the day and decide to take the weekend off. I really need some sleep. Before I leave, I get one last e-mail from Daddo:

An offer of truce? He must be getting really scared. I shut down my laptop and head back to my hotel. Although I do have a restful weekend, my mind doesn’t leave the investigation for too long. I have probably well exceeded the scope of this investigation, and my client is paying for it. I decide to wrap it up by Tuesday, even if I don’t know his name by then. He’s sure to turn up again. Besides, it was much more exciting when I didn’t know anything about him. I wonder to myself what Basilio will find.

Monday

First thing Monday morning I get this e-mail:

I print the e-mail and head down the hall to the CIO’s office.

“What’s up?” he asks, as I enter his office.

“He drives a blue Honda Civic,” I tell him.

He glances down at the paper in my hand, then back up at my face. “So you know who he is?”

“And I know where he works.”

“So now what?” “I’ll write up a final report, gather my evidence, and send a report on to the FBI. They’ll take it from here. I’ll also be sending my final report to the insurance company.”

“Ouch, be gentle,” he begs.

I smile, then head back to my office. I spend a couple hours writing reports, and we all meet with the FBI later that afternoon. I detail the evidence I’ve gathered and hand them a report, along with a box of evidence, complete with a chain of custody and detailed notes of everything I did in my investigation. One of the agents is intelligent and pretty cool; the other one is a condescending ass. They ask me a few questions, and one of them (the ass, not the intelligent one) brags that they have a bust coming up at DEF CON and maybe this guy will make the list if he attends.

What an idiot to blurt something out like that, I think to myself. I wonder how many surprise busts he has blown because of his big mouth.

After the meeting, I return to my office and see two e-mails in my inbox, one from Basilio and one from Daddo. I read Basilio’s e-mail first:

Damn, how does he keep doing this? At least I have that Snort sensor logging everything. After I wrap this up, I need to do an investigation on my own box.

Then the e-mail from Daddo:

I can’t resist the opportunity to chat with him, so I fire up my IRC client.

After typing that, I feel bad. He doesn’t type anything for a moment.

I lean my head back and stare at the ceiling. I actually do feel bad for this guy. I mean, he has a wife and a kid. And the potential for a good career (if he would just stop hacking). Do I really want to send him to prison? I guess it’s out of my hands now anyway.

People don’t understand hackers. They don’t understand what motivates them or what deters them. Few people know how to catch them, and even fewer know what to do once they have them. They are a menace to society, yet so many people revere them, even hire them. They steal, but what they steal isn’t something tangible like a wallet or a car—it’s just a network. They steal the network.

We say goodbye, and I shut down my laptop. I pack up everything, preparing to go home. I sling one bag over my shoulder and hold the other two by their handles. I reach over to shut off the office light, and once again notice the painting. I see a man in his pajamas looking out his front door at endless ocean. Maybe the ocean had been there all along. Maybe he isn’t staring at what’s outside his door—this vast ocean—but what isn’t outside his door. I tilt the painting slightly so that it looks balanced, although technically now it isn’t. I flip the light switch and walk out.

Daddo—kind of a lame nick.