That's what should have happened, but even if it wasn't there are certain steps that anyone running an incident response team should follow in order to accomplish the main goal of any such cleanup: getting the network back to supporting business as usual as quickly as possible.
There are seven key things breach-repair leaders should do, according to Wade Woolwine, the manager of strategic services for Rapid7, who outlined the steps last week at his company's United customer conference. "It's all about recovering the business back to normal operations," Woolwine says.
Here are the seven steps:
Each incident may call for a different set of players. For example, if the first notification of a breach comes from the FBI calling to say it's found out the corporate network was breached, one of the first people to call is the company legal officer.
Or if the breach involves loss of critical corporate data the trade secrets that represent the value of the company the executive board has to be called in. In the case of personally identifiable customer information being compromised, compliance teams need to be tapped to coordinate notification of those affected in accordance with laws and regulations.
This is a changing cast of characters based on the circumstances of each breach.
The critical time in any breach investigation is the first 24 to 48 hours, and gathering hard data about what machines were breached, how they were hacked and what information might have been stolen is essential. It will determine what specific actions to take to secure the network from the immediate threat. "You will make decisions as evidence materializes," Woolwine says.
This is the time to rely on the response team, which is a different group from "the right people" mentioned above. This is the team that will respond to any incident in order to determine what happened, when and how.
During this phase it is important to communicate effectively with the team to hear what they've found out and to hear from the response leader what others know so far so they can better determine what they should do next. "Rely on your team," he says, but challenge their assumptions about what happened in order to ensure that ongoing decisions and analysis are based on fact not speculation.
This is different from the overall incident response plan referred to above that was made before the breach. This plan is specific to the current incident and lays out the details of how to respond to its particular damage.
It should include communications what to tell employees, the board of directors, the public, law enforcement and regulators. The message for all of these parties needs to be formulated and transmitted effectively and in a timely manner, he says.
A plan must include technical analysis of the breach, who will participate, what their roles will be and what each person will do to determine the broad scope of the attack and to zero in on the details of those machines most affected.
The plan must use the analysis to figure out where an active threat still resides and to box off that part of the network so it is rendered ineffective until it can be cleaned up.
Most importantly the plan must include how to restore normal business processes, whether by calling on backups, adjusting firewalls, blocking IP addresses or reimaging corrupted machines.
This step has three parts. First the team needs to quickly triage affected hosts to find indicators of compromise. This must happen quickly typically within two to four hours tapping event logs, file systems and the like to create a distilled timeline of what happened to corrupt the machines.
Once a set of IoCs has been found, they should be put into other security systems that can spot them elsewhere on the network. If that finds more compromised hosts, they need to be triaged and if more IoCs are found, they need to be fed into the security systems.
The most compromised hosts undergo a deep investigation for a full understanding of what the attacker did on that system and to use that analysis to create a remediation plan. That effort should have as its goal making sure the attacker has been purged from the environment.
The leader needs to clear roadblocks so team members can dedicate themselves to remediating the problem. In many organizations the incident response team isn't a group dedicated full-time to incident response. Rather they are individuals with other job responsibilities, so it's important to make it clear to their managers that they are needed to deal with this top priority.
The leader also needs to ensure the effective flow of information within the team so members get the information most relevant to their part of the task quickly.
In incident response there's no room for speculation outside the response team. Speculation is necessary in order to weigh the possibilities of what has occurred as evidence starts to trickle in, but it shouldn't be spread around, Woolwine says.
Anything that is communicated outside the team should be supported 100% by evidence and for good reason. For instance, there's nothing worse than having to tell the board that initial reports were inaccurate, he says.
Also messages should be tailored for individual recipients. Information about the breach that is told to the board should be tailored to answering the question, "How and how soon can business get back to normal" he says.
Within a week of cleaning up a breach, the team should reassemble and discuss its actions, what went right, what went wrong and how to be better prepared the next time.
The positives should be highlighted and adopted as sound procedures for future use. Negatives should be noted and fixed. If they were part of the incident response plan, the plan should be updated.
These sessions should also include others who weren't involved directly in the response but who may offer informed perspectives that team members might miss. For example, those managers who supervise team members when they're not responding to an incident can be a good addition, Woolwine says. They have likely heard a version of the experience and may have helpful thoughts. Also including them may make them more receptive to freeing up team members the next time there's an incident.