We spend a lot of time trying to prevent security incidents from occurring. What sometimes gets lost in all of this preparation is a plan for dealing with an incident should it actually occur. This document is a synopsis of several incident handling guides that provides a high-level framework for dealing with either the realization that a system has been compromised or the recognition that a system is under active attack. The SANS Reading Room has a large set of papers about incident response at http://www.sans.org/infosecFAQ/incident/incident_list.htm.
- Remain calm
Or, in the words of the Hitchhiker, "Don't panic!" (with apologies to Douglas Adams, author of The Hitchhiker's Guide to the Galaxy). To successfully handle any perceived emergency situation, you must remain calm so that you can assess what's going on around you and react in a methodical manner. A compromised system/network or an attacker on the loose demands well-thought out action; and frankly, the bad guys have probably been in your computer for days or maybe even longer, and another few minutes won't make much difference. And you're probably going to have to rebuild your compromised servers anyway...
- Notify your organization's management and activate your response plan to get help
Your security policies should identify the pecking order of who gets called when if there's a security event. Individuals with particular responsibility for the affected server(s) and/or network(s) should be notified, as well as any information security personnel. The severity of the incident (and your own policies) will dictate who else is brought in: your ISP, department head, corporate officers, the press, law enforcement, consultants, response centers, etc. Notify whoever is necessary to assess the situation and get it under control — but it is generally best to maintain a "need to know" stance and communicate, at least initially, with only the necessary parties.
Whenever possible, use telephones and faxes during a computer security incident. If the attackers have full access to your computer, then they can possibly (probably?) read your mail. If you use your computer, this allows them to know when you report the incident and what response you got. There is a real possibility that other systems at your site have also been compromised and one or more packet sniffers are running on your network. So, if you absolutely must use a computer to communicate and you are fairly certain it can't be intercepted, then use a different system and/or dial-up ISP access if possible.
- Take good notes
This cannot be stressed enough — document, document, document!! Maintain a log of everything you see and do, everyone you speak with, and the team working with you. This will not only help you in criminal cases (and in remembering the events at a trial that might take place a year or two down the road), but also helps in the investigation/forensics process, post-event analysis, and as an educational/intelligence gathering vehicle for others in the infosec community. Notes should be detailed, organized, and complete, and should reflect the basic "who, what, where, when, and how" ("why" might be left for later on). Keep copies of any altered files before restoring your system!!
- Contain the problem
Take any necessary steps to keep the problem contained and prevent it from spreading to other systems and/or networks. This may well involve disconnecting the compromised system from your network and/or disconnecting your network from the Internet. Containment may require a physical disconnect or might be accomplished while you clean up and recover; circumstances, including whether you are dealing with an active attack or the aftermath of an attack, will dictate what is a prudent action. Note that the latter approach (containing the problem while still on-line) might well leave you vulnerable to additional attacks.
- Gather evidence and make backups
For purposes of learning what happened and to have evidence for future analysis (and possible prosecution), make backups of operating system and file system information, as well as any state and network information (e.g., output from netstat or route). Keep a detailed history of this activity if you have even the least suspicion that this information will be used in a criminal or civil trial; digital signatures and file timestamps are part of the procedures you should follow to maintain the custody chain. If possible., coordinate your evidence gathering with that of a second source, such as an ISP or another network (if you detect another network that is involved in this incident). Finally, as you make the backups, consider where they are going and who will be using them; if possible, make multiple copies and secure one for historical purposes while analyzing/sharing the other(s).
- Get rid of the problem and maintain your business
This step might be easier said than done. If your server has been compromised, you should totally rebuild it from scratch unless you are 100% certain what the entire problem is. Before you can eliminate the problem, you need to be sure that you understand the cause of the incident. What vulnerability did the intruder use to gain access and what have you done to prevent another attack? You should rebuild the server and applications from original media.
The next issue, of course, is that of re-installing content. Note that some files that were exploited might have been on your system for some time already and, therefore, might have been backed up as part of your regular operations. So, although you've rebuilt the operating system and applications software, you might very well reinstall files that can be exploited over and over. Again, it is imperative that understand how this incident happened to avoid this eventuality.
Finally, business continuity is a major issue. Get rid of the problem and get your server back on line as soon as possible.
- Do a post mortem
Once the situation is resolved and you're back in operation, get all relevant parties together to review the incident and the response. Review your security policies and operational procedures to see what changes, if any, are required. To the extent possible, contact appropriate incident response agencies, such as CERT/CC or incidents.org, and share your knowledge with them.
Hold this meeting a day or two after the incident is deemed "over," when everyone is rested and has had time to reflect on what happened and why, and what went well and what didn't go so well. Don't do it immediately while people are still tired and don't wait weeks when people will forget — and will have moved onto other things.
Return to Vermont InfraGard Home Page.