Incident Response: Breaking the Cycle

An alert comes into your email inbox, notifying of a possible intrusion noticed by one of your sensors. What do you do next? Unfortunately, a common answer to this question, is the analyst rushes to logon to the machine that is potentially breached - which is exactly the WRONG thing to do!

I guess it is human nature to readily go to an ailing object, in our case the victim system. But, when doing so in responding to possible intrusions, you potentially contaminate evidence, or possibly alert the intruder that you know of their existence - losing the upper hand you currently possess in organizing your plan. Even if you do go to that victim machine and start running commands such as 'w' or 'ps -ef', how can you trust their output - trojaning system utilities is a common reinforcement procedure after a successful compromise.

As I have discussed in a previous post, having a robust security monitoring infrastructure is imperative to your noticing and responding to incidents. With centrally located data, you can form a much clearer picture as to what could have happened, or set your mind at ease that nothing has happened - all without touching the victim machine.

So what are some ways to use our security monitoring infrastructure for IR:

Flow data: If you collect flow data, from say Argus, SANCP, or even Cisco Netflow - you could have all the connections that both the source machine and victim machine have made. Querying for what other hosts, ports, and how much data has been transferred are all helpful clues. You may notice strange connections to IRC servers, or large amounts of data transferred over FTP. Flow data can prove invaluable when you don't have access to collected full content data, due to technical or political limitations.

Network IDS Alerts: They aren't just for real-time alerting! Why not search your recorded IDS alerts for previous hits on your victim's IP address or maybe the attacking source's network address. Any otherwise insignificant alert like a portscan, may provide better context to the investigation at hand.

Logs: Both system and application logs can play a huge part here. The key point that needs to be made, is these logs must be collected centrally. They are too easily removed from the "hacked" machine if so desired by an attacker. Logs can show you who logged in recently or what account may have been compromised for instance.

HIDS: If you run a HIDS, you may have file modification alerts - noticing an addition to /etc/passwd or a change to your DNS resolver. These alerts and your system logs may be your only concrete evidence if an encrypted channel such as SSL or SSH is being used.

Having some sort of SIM/SEIM or Event Management system, allows you to review these alerts from one interface, cross-reference, or correlate. Sometimes, logging onto the victim machine will unfortunately be your only option - for instance, if an important clue is only available in running memory on the system. When you get to this stage, remember to always use trusted read-only media with the tools you require. The most important point to take away from this post, is to exhaust your options BEFORE touching the victim machine.