Hard target
18 03, 14 Filed in: Blog Posts
Businessweek is reporting that Target spent $1.6 million to install FireEye (a next-generation network monitoring solution), they had an operations center in Bangalore monitoring the FireEye solution, the FireEye solution alerted on the malware penetrating Target's network, and the operations center treated it as a false positive and ignored it. Also revealed in this article is that Target's CEO said they were certified PCI
compliant in September of 2013 (I'm assuming he means that this was when they completed their last Report on Compliance). For the icing on the cake Businessweek made this their cover story with a huge “Easy Target” headline (complete with a cute animated online version) which demonstrates the potential PR fallout from a breach like this.
The article is here.
Compliance, monitoring, and response
For quite a while now I’ve been beating the drum on the message that you can't rely on protection mechanisms alone (firewalls, patching, etc.) to secure a network and the data within it; given enough time a motivated attacker will find a way in. You have to be able to detect the intruder and be able to respond to him in order to limit the damage he can cause. This is why banks have cameras, alarms, guards, and a hotline to the police despite also having a vault to keep valuables in. I've raised this point in the context of the Target breach before as well: we already knew that the breach was based on malware that had been modified to evade antivirus detection and this illustrates the need for monitoring and response capability rather than relying on antivirus alone. Reports indicated that Target first found out about the breach when they were informed of it by Federal authorities, likely because the stolen cards had already turned up on underground markets and had been traced back to Target via Federal or bank fraud analysis units. This indicates that Target's detection and response capabilities were not effective but was not surprising: 69% of breaches are first detected by an external party according to the Verizon 2013 Data Breach Investigations Report. Now the FireEye revelation, indicating that Target had all the right pieces in place to detect and respond to this breach, changes the nature of the conversation a bit.
Based on what we now know about the FireEye deployment it appears that Target was in fact trying to do all the right things: they became PCI compliant, they had robust monitoring infrastructure (FireEye) in place as required by PCI-DSS, and they had actual human beings reviewing the alerts generated by those monitoring systems also as required by PCI-DSS. Regardless of how effective the offshore operations center was (which I'm sure will become a topic of much speculation) these 3 points alone demonstrate more security effort than is apparent at most companies that handle credit cards. We are doing assessment work for major companies that haven't even attempted to become PCI compliant yet (some in the retail sector), most of these companies (compliant or not) have not invested in monitoring infrastructure any more advanced than IDS/IPS and basic system log collection, and manually reviewing these logs is usually an often overlooked side-job assigned to an overworked IT guy.
So here is where I disagree with Businessweek's characterization of "Easy Target" (although I'll admit it does make a great headline): In light of this revelation I would say that Target is likely one of the harder targets. Despite the enormous impact of this breach it is still only a single breach and should be viewed in light of Target's overall security efforts. I would be very interested to see numbers around how many attacks Target successfully stopped with their monitoring capabilities before this attack slipped through. This breach did still happen though and companies will want to know why and what they can do to protect themselves; based on what we know now I would say that Target made 2 errors, both relatively minor when compared to how atrocious security is in most organizations. The 2 errors both have to do with how monitoring is conducted; specifically what behaviors generate alerts and how false positives are handled.
False positives
Any security monitoring system, whether it is a network intrusion detection system, a motion sensor in a bank, or a metal detector at an airport, can be tuned to be more or less sensitive and a FireEye deployment is no different. The tuning capability exists because there is unfortunately no such thing as a security sensor that only alerts on threats without ever generating false positive results: a metal detector that alerted on any metal at all would alarm every time a person with metal fillings in their teeth or metal rivets in their jeans walked through, a motion sensor that alerted on any motion at all would alarm every time a spider crawled across the floor, and network monitoring system that alerted on any activity would inundate its operators with alerts on normal activity. Tuning the system to be less sensitive in order to eliminate false positives is not as simple as it may seem: if a metal detector is tuned only to detect a lump of metal the size of a gun it will fail to alarm when a group of people each carries through a single component of a gun for reassembly on the other side. In order for security technology to be effective it must be tuned to be sensitive enough that it will detect most of the conceivable threats and an allowance must be made for humans to thoroughly investigate the potential false positives that will inevitably occur as a result.
Published information on Target's response indicated that the FireEye solution labelled the ignored threat as "malware.binary", a generic name for a piece of code that is suspected to be malicious even though it does not match any of the patterns for more widely spread malware that has been analyzed and given a name. So far this indicates that Target has likely tuned their monitoring solution well enough as it did detect the actual threat and generated an alert based on it (a system that had been tuned to be too permissive wouldn't have generated an alert at all). Where Target's system fails is the human response to that alert: It is likely that Target's monitoring center received many of these generic alerts on a regular basis, most of them either false positives or simple attacks that were automatically blocked by other security mechanisms; after too many of these false positive generic alerts the humans responsible for responding to them will learn to ignore them. This is like asking each person who sets off the metal detector if they have metal fillings and sending them on their way without further inspection if they respond in the affirmative; it wouldn't be a surprise at all if something slipped through at that point. The only way to make effective use of the security solution is to actually investigate each alert and resolve the cause; this is time consuming and expensive but not nearly so much as a breach. It appears that this is the key piece of Target's process that failed.
Behavior monitoring
The second error is something I am inferring from what was not mentioned: specifically any alerts based on activities on the network. Malware is a "known bad", a chunk of code that is suspected to be suspicious because it exhibits certain characteristics. The same could be said for most alerts generated by intrusion detection and prevention systems: they are based on network traffic exhibiting known suspicious characteristics such as a chunk of network traffic that would exploit a known vulnerability in an email server or a computer that quickly tries to connect to each of the protected systems in turn. Attempting to monitor a network by only watching for "known bad" traffic is akin to setting up a firewall to allow all network traffic except that which is explicitly denied (a practice that was mostly abandoned many years ago). The standard for configuring firewalls today is to deny all traffic by default and to only allow specific "known good" services to pass through when they are explicitly defined and this is the method we must look into for effective network monitoring as well: Define "known good" traffic and alert when anything else out-of-the-ordinary happens on the network.
The actual method used to penetrate and infect the network aside, reports indicate that credit card data was sent from Target's point-of-sale terminals to another compromised server on Target's network where it was then encrypted and sent out for the attackers to retrieve over the Internet. This represents the exfiltration of a huge amount of data and, had Target been looking for anything other than "known bad" traffic, provides 2 opportunities for detection: point-of-sale terminals would have suddenly started interacting with a system on Target's own internal network that they did not normally interact with and then that system suddenly started sending large amounts of encrypted traffic to a system on the Internet that it had never communicated with before. Neither of these communication vectors would have been flagged as "known good" and therefore should have triggered alerts for investigation. Unfortunately almost no-one monitors networks in this way and Target can't really be faulted for not being on the bleeding edge of security best-practices.
Law enforcement
There is a third failing that is worth mentioning here, one that is not at all under Target's control but that nevertheless contributed directly to this breach and many others around the world: the inability or unwillingness of law enforcement to stop criminals who operate primarily online. In the physical world we are used to the concept that when a bank gets robbed the police will respond, investigate, and at least attempt to identify and arrest the offender but in the online world this simply isn't happening all that often.
There are various published reports identifying the individuals behind the malware used at Target and the website used to sell the stolen credit card numbers. These reports weren't the results of Secret Service investigations or NSA metadata collection programs, rather they were identified, fairly easily, by private individuals piecing together information from social media sites and underground forums. Unsurprisingly to anyone in the security industry, the implicated individuals are all young, from Eastern Europe, and have been engaged in these activities for many years. The economic realities in much of Eastern Europe is such that there aren't many legitimate career opportunities for bright young computer enthusiasts. Given the sad state of information security in the rest of the world and the potential income it isn't surprising that many of these kids, who under different circumstances may have been the brains behind a multi-million dollar Silicon Valley startup, are turning to crime against corporations on the other side of the planet. With the recent events unfolding in Ukraine perhaps there is a glimmer of hope that these economic conditions will start changing in the near future.
One would assume, if these are just broke kids with a knack for computers and they are so sloppy about protecting their identities that someone with computer know-how (and some knowledge of the Russian language) can figure out who they are, that law enforcement must already be heading for their door but things are not so simple: a significant fraction of online crime crosses borders and while large breaches like Target attract law enforcement attention a small business owner would be hard-pressed to get any meaningful law enforcement response to a breach regardless of the consequences for his business. Local law enforcement agencies usually don't have the resources to conduct investigations across state lines, never mind national borders. In the post 9/11 world Federal law enforcement priorities are often focused elsewhere, often in the name of "national security"; the agencies that have historically focused on information security seem to be more concerned with threats posed by other governments than criminal enterprises and the FBI is now spinning itself as a counterterrorism and foreign intelligence agency. The political realities in Eastern Europe are also such that the cooperation between Western law enforcement agencies and their local counterparts that would be necessary to bring offenders to justice would be difficult or non-existent, the recent events unfolding in Crimea indicate that any change in this status-quo is unlikely. For the foreseeable future the attackers will be mostly left to their own devices, honing their skills across hundreds or thousands of attacks until they have the capability to penetrate even the most well defended network.
Where do we go from here?
Technology alone can't solve all our problems. Hopefully most of us know that already but there were quite a few vendors at the RSA conference this year proclaiming that their technology would have prevented the Target breach or, even more ludicrously, claiming that it would have prevented the Snowden breach at NSA. If technology could in fact solve all of our woes then, in light of Target's $1.6 million dollar investment in
FireEye's solution, any organization that hasn't spent that enormous amount on security technology should be very worried. This also demonstrates once again that compliance alone is not security either: we don't know who Target's PCI assessor was or if they took the compliance mandate seriously (versus taking the checkbox approach) but from what I've read so far it is entirely possible for this breach to occur in the manner that it did even if Target was serious about compliance. We need to treat compliance as a minimum standard, a guideline upon which we should build security appropriate to our own threat environment. And finally, it is becoming increasingly obvious that the next step in the cat-and-mouse game of security is to increase real-time monitoring and response capabilities to make more effective use of the technology that we have deployed and to make sure that the people tasked with that response must have the time and resources to conduct proper investigations (no more pretending that the overworked IT guy will have time to do it).
compliant in September of 2013 (I'm assuming he means that this was when they completed their last Report on Compliance). For the icing on the cake Businessweek made this their cover story with a huge “Easy Target” headline (complete with a cute animated online version) which demonstrates the potential PR fallout from a breach like this.
The article is here.
Compliance, monitoring, and response
For quite a while now I’ve been beating the drum on the message that you can't rely on protection mechanisms alone (firewalls, patching, etc.) to secure a network and the data within it; given enough time a motivated attacker will find a way in. You have to be able to detect the intruder and be able to respond to him in order to limit the damage he can cause. This is why banks have cameras, alarms, guards, and a hotline to the police despite also having a vault to keep valuables in. I've raised this point in the context of the Target breach before as well: we already knew that the breach was based on malware that had been modified to evade antivirus detection and this illustrates the need for monitoring and response capability rather than relying on antivirus alone. Reports indicated that Target first found out about the breach when they were informed of it by Federal authorities, likely because the stolen cards had already turned up on underground markets and had been traced back to Target via Federal or bank fraud analysis units. This indicates that Target's detection and response capabilities were not effective but was not surprising: 69% of breaches are first detected by an external party according to the Verizon 2013 Data Breach Investigations Report. Now the FireEye revelation, indicating that Target had all the right pieces in place to detect and respond to this breach, changes the nature of the conversation a bit.
Based on what we now know about the FireEye deployment it appears that Target was in fact trying to do all the right things: they became PCI compliant, they had robust monitoring infrastructure (FireEye) in place as required by PCI-DSS, and they had actual human beings reviewing the alerts generated by those monitoring systems also as required by PCI-DSS. Regardless of how effective the offshore operations center was (which I'm sure will become a topic of much speculation) these 3 points alone demonstrate more security effort than is apparent at most companies that handle credit cards. We are doing assessment work for major companies that haven't even attempted to become PCI compliant yet (some in the retail sector), most of these companies (compliant or not) have not invested in monitoring infrastructure any more advanced than IDS/IPS and basic system log collection, and manually reviewing these logs is usually an often overlooked side-job assigned to an overworked IT guy.
So here is where I disagree with Businessweek's characterization of "Easy Target" (although I'll admit it does make a great headline): In light of this revelation I would say that Target is likely one of the harder targets. Despite the enormous impact of this breach it is still only a single breach and should be viewed in light of Target's overall security efforts. I would be very interested to see numbers around how many attacks Target successfully stopped with their monitoring capabilities before this attack slipped through. This breach did still happen though and companies will want to know why and what they can do to protect themselves; based on what we know now I would say that Target made 2 errors, both relatively minor when compared to how atrocious security is in most organizations. The 2 errors both have to do with how monitoring is conducted; specifically what behaviors generate alerts and how false positives are handled.
False positives
Any security monitoring system, whether it is a network intrusion detection system, a motion sensor in a bank, or a metal detector at an airport, can be tuned to be more or less sensitive and a FireEye deployment is no different. The tuning capability exists because there is unfortunately no such thing as a security sensor that only alerts on threats without ever generating false positive results: a metal detector that alerted on any metal at all would alarm every time a person with metal fillings in their teeth or metal rivets in their jeans walked through, a motion sensor that alerted on any motion at all would alarm every time a spider crawled across the floor, and network monitoring system that alerted on any activity would inundate its operators with alerts on normal activity. Tuning the system to be less sensitive in order to eliminate false positives is not as simple as it may seem: if a metal detector is tuned only to detect a lump of metal the size of a gun it will fail to alarm when a group of people each carries through a single component of a gun for reassembly on the other side. In order for security technology to be effective it must be tuned to be sensitive enough that it will detect most of the conceivable threats and an allowance must be made for humans to thoroughly investigate the potential false positives that will inevitably occur as a result.
Published information on Target's response indicated that the FireEye solution labelled the ignored threat as "malware.binary", a generic name for a piece of code that is suspected to be malicious even though it does not match any of the patterns for more widely spread malware that has been analyzed and given a name. So far this indicates that Target has likely tuned their monitoring solution well enough as it did detect the actual threat and generated an alert based on it (a system that had been tuned to be too permissive wouldn't have generated an alert at all). Where Target's system fails is the human response to that alert: It is likely that Target's monitoring center received many of these generic alerts on a regular basis, most of them either false positives or simple attacks that were automatically blocked by other security mechanisms; after too many of these false positive generic alerts the humans responsible for responding to them will learn to ignore them. This is like asking each person who sets off the metal detector if they have metal fillings and sending them on their way without further inspection if they respond in the affirmative; it wouldn't be a surprise at all if something slipped through at that point. The only way to make effective use of the security solution is to actually investigate each alert and resolve the cause; this is time consuming and expensive but not nearly so much as a breach. It appears that this is the key piece of Target's process that failed.
Behavior monitoring
The second error is something I am inferring from what was not mentioned: specifically any alerts based on activities on the network. Malware is a "known bad", a chunk of code that is suspected to be suspicious because it exhibits certain characteristics. The same could be said for most alerts generated by intrusion detection and prevention systems: they are based on network traffic exhibiting known suspicious characteristics such as a chunk of network traffic that would exploit a known vulnerability in an email server or a computer that quickly tries to connect to each of the protected systems in turn. Attempting to monitor a network by only watching for "known bad" traffic is akin to setting up a firewall to allow all network traffic except that which is explicitly denied (a practice that was mostly abandoned many years ago). The standard for configuring firewalls today is to deny all traffic by default and to only allow specific "known good" services to pass through when they are explicitly defined and this is the method we must look into for effective network monitoring as well: Define "known good" traffic and alert when anything else out-of-the-ordinary happens on the network.
The actual method used to penetrate and infect the network aside, reports indicate that credit card data was sent from Target's point-of-sale terminals to another compromised server on Target's network where it was then encrypted and sent out for the attackers to retrieve over the Internet. This represents the exfiltration of a huge amount of data and, had Target been looking for anything other than "known bad" traffic, provides 2 opportunities for detection: point-of-sale terminals would have suddenly started interacting with a system on Target's own internal network that they did not normally interact with and then that system suddenly started sending large amounts of encrypted traffic to a system on the Internet that it had never communicated with before. Neither of these communication vectors would have been flagged as "known good" and therefore should have triggered alerts for investigation. Unfortunately almost no-one monitors networks in this way and Target can't really be faulted for not being on the bleeding edge of security best-practices.
Law enforcement
There is a third failing that is worth mentioning here, one that is not at all under Target's control but that nevertheless contributed directly to this breach and many others around the world: the inability or unwillingness of law enforcement to stop criminals who operate primarily online. In the physical world we are used to the concept that when a bank gets robbed the police will respond, investigate, and at least attempt to identify and arrest the offender but in the online world this simply isn't happening all that often.
There are various published reports identifying the individuals behind the malware used at Target and the website used to sell the stolen credit card numbers. These reports weren't the results of Secret Service investigations or NSA metadata collection programs, rather they were identified, fairly easily, by private individuals piecing together information from social media sites and underground forums. Unsurprisingly to anyone in the security industry, the implicated individuals are all young, from Eastern Europe, and have been engaged in these activities for many years. The economic realities in much of Eastern Europe is such that there aren't many legitimate career opportunities for bright young computer enthusiasts. Given the sad state of information security in the rest of the world and the potential income it isn't surprising that many of these kids, who under different circumstances may have been the brains behind a multi-million dollar Silicon Valley startup, are turning to crime against corporations on the other side of the planet. With the recent events unfolding in Ukraine perhaps there is a glimmer of hope that these economic conditions will start changing in the near future.
One would assume, if these are just broke kids with a knack for computers and they are so sloppy about protecting their identities that someone with computer know-how (and some knowledge of the Russian language) can figure out who they are, that law enforcement must already be heading for their door but things are not so simple: a significant fraction of online crime crosses borders and while large breaches like Target attract law enforcement attention a small business owner would be hard-pressed to get any meaningful law enforcement response to a breach regardless of the consequences for his business. Local law enforcement agencies usually don't have the resources to conduct investigations across state lines, never mind national borders. In the post 9/11 world Federal law enforcement priorities are often focused elsewhere, often in the name of "national security"; the agencies that have historically focused on information security seem to be more concerned with threats posed by other governments than criminal enterprises and the FBI is now spinning itself as a counterterrorism and foreign intelligence agency. The political realities in Eastern Europe are also such that the cooperation between Western law enforcement agencies and their local counterparts that would be necessary to bring offenders to justice would be difficult or non-existent, the recent events unfolding in Crimea indicate that any change in this status-quo is unlikely. For the foreseeable future the attackers will be mostly left to their own devices, honing their skills across hundreds or thousands of attacks until they have the capability to penetrate even the most well defended network.
Where do we go from here?
Technology alone can't solve all our problems. Hopefully most of us know that already but there were quite a few vendors at the RSA conference this year proclaiming that their technology would have prevented the Target breach or, even more ludicrously, claiming that it would have prevented the Snowden breach at NSA. If technology could in fact solve all of our woes then, in light of Target's $1.6 million dollar investment in
FireEye's solution, any organization that hasn't spent that enormous amount on security technology should be very worried. This also demonstrates once again that compliance alone is not security either: we don't know who Target's PCI assessor was or if they took the compliance mandate seriously (versus taking the checkbox approach) but from what I've read so far it is entirely possible for this breach to occur in the manner that it did even if Target was serious about compliance. We need to treat compliance as a minimum standard, a guideline upon which we should build security appropriate to our own threat environment. And finally, it is becoming increasingly obvious that the next step in the cat-and-mouse game of security is to increase real-time monitoring and response capabilities to make more effective use of the technology that we have deployed and to make sure that the people tasked with that response must have the time and resources to conduct proper investigations (no more pretending that the overworked IT guy will have time to do it).