Bad things seem to always happen on clear crisp Fridays. I was up early, ready to rock and roll. Like usual, I let the dogs outside around 5:30 AM. I then had a cup of coffee and checked the news on my Twitter feeds (yes, I know, but Twitter can give me the snapshot of the day before I head to the train). No Twitter feed. Hmm. Nothing. I fooled around on other websites and they all seemed to be having major issues too. Connectivity was really bad across the board. After about a half hour of sniffing around, I went to work rather annoyed at the lost productivity of my morning. At work, I found out it too had no Twitter feed either. Grrrrrr. I found out that the Internet was indeed “broken” again. In a bad way, and arguably for the second time in a matter of weeks.
What happened on Friday, October 21, 2016 to the East Coast portion of the US Internet backbone is pretty complex but it can be boiled down in English. But that is not the point of this article. The real point is given what we know, what can a US business (large or small) do to prevent their whole operation from falling apart under the weight of a distributed denial of service attack (or “DDoS” attack) that suddenly ceases Internet connectivity. Again, based on what we know today about the attack, there are no good answers. Just progressively “less good” answers that more than likely will cost money to operationalize. But Friday’s attack was a game-changer and we are now clearly on notice that it might not be a fluke.
What Happened?
Like a good newspaper article, it’s first important to start with what happened. At around 7:00AM or so ET, an unknown entity commenced a large-scale DDoS attack against Dyn, a company that provides domain name services (“DNS”) to large and small corporations and other entities. Very generally speaking, a domain name service is like the “interpreter of the Internet.” Rather than a customer having to connect with an entity via a group or pattern of numbers, like 1.1 2.2. 33, 44. 5.567 (an internet “address”), the customer only needs to enter the name XYZcompany.com on the toolbar—the XYZcompany.com being the domain name. This is generically called a “query.” The customer is then connected through an open internet connection to XYZcompany.com. In the good old days, like 10 years ago, these sorts of connections happened instantaneously. The companies had multiple open DNS connections (say e.g. 20 open connections), and the volume of traffic generally then generally allowed those 20 open connections to handle all the traffic it needed too.
Ok, this sounds normal. So why did the Internet go down? Rather than 20 customers pinging XYZcompany.com, the attackers commenced something called a “botnet attack.” A botnet is akin to a robot army of devices attacking the Internet through something called command and control services. The botnet “generals” first take control of a large group of devices by compromising them with malware. Then at an appropriate moment, the general will launch a DDoS attack against a target.
Now nothing about this is new. DDoS Attacks have been happening for the past couple of years. Mostly hit and run attacks—one or two hours at the most. Then operations restored. Not today. In the last few weeks, DDoS have been occurring more frequently, and with much more power than ever before. Why? Well, the Internet of Things (“IoT”), of course. What, how could the IoT do this. Well, because in dreaming that our whole world, home and office could be connected, we gave internet address to everything. Our home thermostat. Our toaster. Even our refrigerator. And we did this primarily without thinking about network security or design. We just got the IoT products out there, because they were cool. A few weeks ago, the proved not to be so cool, when a DDoS attack facilitated by a botnet of compromised IoT devices (compromised by malware called Mirai) took down the website of famous cybersecurity blogger Brian Krebs. This attack was very powerful (about double what was normally happening in the cyber ecosystem until that time). Ok, you say. So what? Then that same week another DDoS attack was commenced against OVH, a French media company. This attack was double the size of the Brian Krebs attack.
So the picture worsens till Friday, when Dyn suffered at least three massive attacks of epic size and scope. According to Dyn, 10’s of millions of IP addressed were used to commence the attacks. This attack took down Dyn for the good part of the day, and with it scores of other websites and cloud applications that depend upon network connectivity. We don’t know the size of the Dyn attack, but it appears to be multiples of the OVH attack. Another bad omen.
Who Did it? Why It Happened?
On the “who did it” part of the newspaper story, we are sure there are plenty of investigations commenced immediately after the 7 AM attack on Friday. There are obviously plenty of people to point the finger at, both nation-states and hacktivists. The one common thread of information is that to commence such huge attacks against DYN, the attackers need to have plenty of firepower, and plenty of money. There are plenty of other fingers that can be pointed as well, mainly to ourselves. Many look to the loose nature of many companies in provisioning domain name servers for the customers. If the domain name resolver can only respond to the query of XYZcompany.com customer that is a good thing. If the domain name is “open,” meaning it is not attached to any particular address or customer or authorized task, then a hacker seeking to start a botnet can use the open domain name server to commence a botnet against a third party. This is a basic cyber hygiene issue. Many are good at network maintenance and clean up. Others are not. But an open domain name server can be a mischief maker if left unattended.
Finally, over the last week you have heard about the “security of the Internet of things.” This appears to be a giant oxymoron. It has appeared to be true over the last few weeks that many of the devices used in the Murai botnet attacks and again in the Dyn Attacks were simple Internet-connected devices like DVRs and digital cameras, which has internet connectivity but very in the form of password or network security. Some had no passwords. Or default passwords. Others had simple passwords like 0123456 that can be easily figured out by an advanced attacker. Without security, the devices were compromised/taken over and used in the above attacks. More on this point later.
The Four Things you Can Do to defend against large scare DDoS attacks:
DDoS attacks like in the case of Brian Krebs, OVH and Dyn can and will happen. There is even a greater chance of them happening now because of copycats and because, for the price of a Manhattan dinner for two, you can hire a “DDoS Botnet for Sale.” Thus here are some of the finer points that any business should consider when coming up with an anti-DDoS plan of action;
- Make sure that you have hired an anti-DDoS cloud-based remediation service. These services, though not cheap, act by detecting and targeting bad network traffic away from your company’s network server, so that it does not become “choked” with bad traffic. There are many of these services out there like Verisign or Akamai. They definitely should be considered, especially if you are a larger company or financial institution.
- Make sure you inquire about Alternative DNS Services should your main DNS service go down. It is like having a spare tire. If one DNS service is overwhelmed, you can transition to your back up DNS server. Life goes on. No hiccups. Just relief that you still are in the game. Same for alternate ISP capacity in case your primary ISP goes down via an attack. This is called “overprovisioning” your network infrastructure. Again, this costs money, but it’s better than having your website go silent.
- Basic blocking and tackling always is a good idea. There are programs that can be run against the domain names servers of any particular company to check for open DNS resolvers (meaning those domain names not attached to a name, employee or task). In fact this website states, “Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s. We have collected a list of 32 million resolvers that respond to queries in some fashion. 28 million of these pose a significant threat (as of 27-OCT-2013).” After Friday’s Dyn attack, DNS resolver housekeeping should jump high on the network maintenance list. It’s not a cure-all, but any little bit helps when you are being attacked. There is other blocking and tackling here as well, including patching, and password maintenance that also might be able to help defend against a DDoS attack.
- Finally, make sure that you “know” when you are being attacked. In the trade, this is called “visibility.” Do you know the normal state of your network traffic? Meaning what is normal, abnormal, or, for lack of a better term, “messed up.” In order to invoke any of your incident response plan for DDoS attacks (like e.g. an anti-DDoS mediation service), it is important to know when to flip the switch on your DDoS mediation provider. Some providers can do this automatically for you. Other companies want their fingers on the trigger. But most of all, it’s important to know if and when you are being attacked.
Paul Ferrillo is counsel in Weil, Gotshal & Manges’ Litigation Department.
