Mistakes made in Incident Response

[ Updated : Added  "10 Common Mistakes of Incident Responders" at the bottom]

The following post will brake one major rule I adhere to  when blogging, a post shall have not more than 10% of content that is not authored by myself. The content of this post resonated so well with me however that I decided to make an exception.

The following is attributed to Alit-Reza Anghaie a.k.a Packetknife.com. For those of you in similar situations I can only warmly recommend to consider and follow the advice. The emphasis is mine.

[Start of Excerpt] 

Alit-Reza Anghaie

I've had a fairly long and quite unintentional career in InfoSec ranging from Academic to Entertainment to Defense. Along the way a lot of mistakes were made or observed. This post marks the first in many installments to share lack of foresight turned into a graying face ghillie.

I'm not quite sure of the right format but I'm going with a Top Twenty - so I'll keep on the biggest pain points as I see them.

• Becoming engrossed in "the" incident: If an incident is even remotely close to remarkable - it's not alone. It came with friends or like-minded enemies will join in shortly. No IR should start without immediate consideration of this aspect - wait and it'll be too late. You fall into either a public-facing incident category in which phishing and other fraud starts immediately as well as the normal poking and prodding. Or you fall into a "quiet" but targeted incident which means the adversary didn't just have one play to make. Identify the other impending incidents immediately.
• Not completing a IR cycle (or Going back to the way it was): I wrote a whole post about this. Basically revisit everything - something was wrong - don't just go back to the way things were before. That's a false promise.
• Current VALIDATED configurations and documentation: Self explanatory? Almost always a huge IR gap. If you're environment is "too agile" for documentation pack up and get into agriculture, you're kidding yourself. When did Agile mean unknown? And configuration management of standard images is so ridiculously easy to do it's inexcusable. And configuration and software awareness of your whole environment will be especially key through the IR cycle. If you don't have a good handle on configuration management and infrastructure documentation, you can't perform any reasonable level of IR.
• Believing you're too small: You're never too small to have a business breaking incident. I'd conservatively say at least thirteen of the listed pain points apply at SOHO levels.
• Out-of-band communications, Out-of-reach documentation, the War Room: You simply should not be doing your IR on the existing email system, calendar system, etc. Your IR plans and past experiences shouldn't be readily available in the usual places. And you need a War Room with whiteboards and quiet walls. If you don't have a ready out-of-band system then set your team up w/ Countermail accounts or a Google Apps account specifically for InfoSec use - something - just don't keep your IR in-band.
• Quick reference cards of personnel and vendor resources: This isn't hard - and almost entirely overlooked. Who and how to reach everyone and grease the joints with your vendors and know their security contacts as well. Be sure to include duties information as well in case someone else has to grab the stack and step-in on an interim IR basis.
• An employee and public relations plan: Keeping everybody in the dark is a bad idea - it develops immediate conspiracy and bad habits. At a minimum your employees should all know what to say to any visitor or outside caller, journalists, or your industry partners.
• Out of order response - patching - cleaning: Patching post-compromise just seals the bullet in. Cleaning without having a proper patch response is being a glutton for punishment. I haven't seen anybody not capable of determining the right order and procedures - generally I just see people a bit too panicked. Regardless of how it happens make sure you have a IR action/decision tree that accommodates for this variance.
• Not planning for conferences or vacations: You're going to get hit during Black Hat, DEFCON, and every time your staff Tweets they'll be away or Foursquare checks-in they are away. Especially if they say anything about drinking and dancing. Plan for it - even if all lips are sealed the "industry timing" will come into play at some point. As well as Holidays and the occasional Red Sox World Series.
• You can't train "up" when the time comes: Having tools or books or a training or two under your belt is nice but you simply can't train "up" for an IR. It doesn't matter how talented your resources are, they're going to hit a limit. They'll need help or rest. You need to train and practice regularly, I suggest at least quarterly or after each key staffing turnover, with a simulation of the biggest peer-group incident you know of. And don't underestimate the willingness of forensics and IR gear to fail - so train for IR "continuity" as it were in both people and gear.
• Validate audit controls: Especially any "watching the watcher" controls, log tampering controls, sudo, Administrator access, etc. Make sure they're tuned in, tested, and right at the onset of a IR not a liability. 
• Cutting out the other stakeholders: Security personnel who don't participate in devops and production are missing out. Likewise devops and other teams need in on Security and particularly IR regularly. They are better suited to spot changes, nuances, and odd behavior - and you'll want that as malware and adversaries increasingly cloak themselves within your daily operational norms.
• Failing to consider Legal concerns: Prosecution and chain of evidence is the immediate thought that should occur. Additionally you seriously need to consider liability to third-party data (including customer privacy), potential Export Control escapes, and danger to acquisitions and mergers. Consider this all ahead of time, in your quarterly dry runs.
• Letting Legal concerns dominate your IR: Conversely many organizations became paralyzed with fear over Legal repercussions or something of bloodlust to prosecute. Reality dictates you'll have limited ROI from any prosecution anyway - and liability won't make much of a difference if you can't get your business on rails again. So practice the right stuff but if you stumble or accidentally skip steps - don't stop your IR just to beg Legal for forgiveness. You can beg later. 
• Business Continuity might also mean Compromised Continuity: This one surprises the heck out of me but a lot of organizations fail over, do their IR, only to realize - yep - their continuity production was also part of the compromise. How this repeatedly escapes some pretty big enterprises is curious and unknown to me. Don't be one of them. Also, in one case the failover was intentionally prepped with more dirty bags of tricks so don't be afraid of the fine-tooth comb to check for embedded adversary forces.
• Not enough automation: So many IR tasks can be automated - do so - now. Automation also includes validation and failure resiliency, please don't forget. It's remarkable how much time can be saved when you have scripts setup to shutdown and isolate segments, start pulling logs, grab backups off of near-line storage, etc. So much of an IR can be at least partially automated.
• "We can't stop production": Sure you can and THEY will - this is the belief that prevents all InfoSec progress within an organization. The next closest preventative is InfoSec mistakes themselves (e.g. security usability) but that's another Top Twenty. Bottom line is find a way - segregation, better production continuity, whatever - IR can't wait for a production issue. I've been in the most sensitive lives-on-the-line situations and IR waited and it only got worse until all sorts of previously uncharted failure modes were "explored"..
• Not communicating with industry peers: Increasingly if you're experienced it, so is one of your peers at the same time. Have open lines of communication, know your peers, and be ready to leverage each other's realtime IR status. You'd be surprised how many Security officers want more collaboration with their competitive peers than not. 
• Not Red Teaming - HARD: If you can't think of a baker's dozen ways to compromise your own network - you're doomed. Your IR team should also have a thorough Red Team view and experience against your own network. In larger organizations it's worth bringing in an outside Red Team annually and try not to bring back the same team. And never - ever - use a Red Team your regular IT services supplier provides. It's not worth the savings when you inevitably find out there was some "fudging" of the efforts.
• Blame, Shadow IT, and Clouds: Blame is an IR killer and a huge waste of time - you can do RCAs and finger point after you're bloody settled (if you must). And trying to eliminate Shadow IT, BYOD, or unsanctioned Clouds during IR is a mistake - you don't address them anymore than is necessary to complete the IR. Oh - and that means you DO address them. As a matter of practice I seek them out almost immediately in any new position or engagement - and then they get rolled into the documentation. If at that point a C-level wants to seek-and-destroy, so be it.
• One last note - for the InfoSec practitioner - there will always be a fall guy. Insist and implement the suggestions above and it's not likely to be you.

[END of Excerpt]

Addendum: I'd like to add "10 Common Mistakes of Incident Responders" by McAfee