Hacker News new | comments | show | ask | jobs | submit login
Massive Dyn DNS outage
157 points by jtmarmon 3 hours ago | hide | past | web | 98 comments | favorite
Sites down:

- DYN

- Twitter

- Etsy

- Github

- soundcloud

- spotify

- heroku

- pagerduty

- shopify

- intercom (app, not landing page)

Note that if these sites seem to be up to you, it's likely that your machine has cached the DNS response for these sites.

Some of these sites seem to work when using a UK VPN






All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?

reply


From where I'm looking at the internet (central Europe), I don't notice anything.

Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn

reply


Not really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.

reply


The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn.

We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.

reply


That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.

reply


N.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}

reply


Exactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down.

Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.

reply


How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.

reply


Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed.

Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.

reply


Why not? That's the whole idea behind having more than one NS server isn't it?

reply


they need a round robin dns that is geographically dispersed

reply


Important: PagerDuty.com seems affected by this outage. So keep a real good eye on your graphs today -- you might not receive the alert.

reply


DYN is experiencing a DDoS

https://www.dynstatus.com/incidents/nlr4yrr162t8

reply


Yep. Seems to be the root of the problem.

reply


GitHub employee here. We're monitoring an incident with our upstream DNS provider:

https://twitter.com/githubstatus/status/789433336083001344

reply


Can you CC on HN, twitter doesn't work for people as well. HN seems to be fine.

reply


> We're monitoring an incident with our upstream DNS provider.

[pic](https://mikevanrossum.nl/stuff/gh.png)

reply


not very useful to post a status update on a site that is experiencing the same issues with the same DNS provider.

maybe post a github gist? oh wait...

reply


Should there be a global internet status page at an easily memorizable "vanity" IP address?

reply


Hahahahaha you do realize Twitter is one of the affected sites, right?

reply


> our upstream DNS provider

Maybe you should have more than one. Then I could actually carry out my work...

reply


Can we add a line to our /etc/hosts file with a hard-coded DNS entry to fix it in the meantime? What's the IP address?

reply


For me (EU) github.com resolves to 192.30.253.112.

reply


I looked up an ip for twitter (via a google search) and still timeout on pings.

[edit: twitter and github are both accessible again.]

reply


github homepage is unicorn now

reply


I just realised how online adult entertainment has the most redundancy of any Internet service category, bar none.

reply


You’re right :)

  $ dig @8.8.8.8 ns +short pornhub.com
  ns2.p44.dynect.net.
  ns3.p44.dynect.net.
  ns4.p44.dynect.net.
  sdns3.ultradns.net.
  sdns3.ultradns.com.
  sdns3.ultradns.org.
  sdns3.ultradns.biz.
  ns1.p44.dynect.net.
PornHub has better DNS redundancy than Github and Twitter.

reply


This is going to be a fun day. This little DNS outage is likely to cause millions of lost revenue for many industries.

Im dead in the water and I cant complain on twitter :-(

reply


change your name server at your registrar to something else, add all necessary entries in the new DNS and be up before DDoS is stopped.

reply


Sadly we're too interconnected. Every company that relies on that DNS should do what you suggest, but the control is definitely not in our ( users ) hands.

reply


Yeah, users are screwed. Unless they have a little more experience with how unreliable cloud can be, and they made a local copy of everything* their work depends on, just in case.

*Everything that can be local.

reply


nah our corp dns is fine, its all the cloud services we and everyone else uses. Thank Sergey and Larry their stuff still works

reply


I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

[x-posted on https://news.ycombinator.com/item?id=12759697 as well]

reply


Reddit is out -- my productivity is up, and I'm not happy about it.

reply


Working here. Try a non-US VPN if you want to stop doing work again.

reply


Sendgrid support page is down as well https://support.sendgrid.com/hc/en-us

reply


http://downdetector.com/status/level3/map/

reply


My Uber partner app crashed at 8am I was trying to complete a trip and it frozen my phone. It took about 5min for me to be able to enter back but it asked me for my SSN and permission to do a background check which is standard by Uber but I had already done so. Should I be concern that my personal data has been compromised? I contacted Uber but their idiots support people don't seem to have a clue and third fix is super basic like restart your phone, turn airplane mode or data on and off

reply


GitHub pretty slow in Brazil

reply


A lot of the bigger sites are very slow from Brazil today :( And SP is also suffering from a lack of power in some areas. Happy Friday!

reply


yeah, i thought it was my internet but it seems like a ton of people are having slow internet issues.

reply


Brazucas unite :-)

Alguém teve algum outro problema fora os citados aqui?

reply


same here. getting the "page taking way too long to load" page every time.

reply


Linkedin doesn't seem to load page's contents as well.

reply


All sites look good for me in Europe using 8.8.8.8.

Edit: It actually looks like most of the sites are loading faster then normal.

Edit2: I have cleared the cache's on my machine and router. Still works.

reply


http://okta.com is not working

reply


Chiming in from sunny Los Angeles

Up as of 5:30AM PST: Twitter, Etsy, Github, Soundcloud, Spotify, Dyn DNS

Down: Heroku, Pagerduty

Might want to use a VPN to another area

reply


Xclinton hack. News held back the 20th through 12 n 13th dump. Was a diplomatic dinner last night in n y c I live on east coast n e c

reply


If you add these sites ip addresses to your /etc/hosts file locally then you don't have to worry about this for the most part.

reply


bigcommerce, volusion, new relic, optimizely, wistia, volusion, aweber, cnn, campaign monitor, all down for me. The biggest thing is seeing that ALL shopify stores are offline, so much $$$ being lost right now.

reply


CircleCI is also having issues viewing and running tests, viewing from Amsterdam: https://status.circleci.com/

reply


Seems to be fine from the UK, so those of you with suitable VPNs might like to try that.

reply


Five Thirty Eight!

Today was a very long train ride without Twitter or the poll tracker.

reply


Shopify too: https://status.shopify.com/incidents/z714f9fjg9q0

reply


All of the sites listed are back up for me, and still using Dyn DNS name servers.

reply


ditto for me

reply


From Ireland all of the above are resolving (with no cache).

reply


Is this another IP webcam etc. attack? Does anyone know of a write-up from a researcher in possession of one of these currently exploited bits of kits?

reply


8.8.8.8 seems to know about github.com and github.io at least -- so I can work. (Maybe 8.8.8.8 built to resist censorship and therefor also attack?)

reply


https://github.com/ is up for me.. NorthEast USA

maybe I missed something

reply


looks like they are all back online now.

reply


Pulling on AWS using github.com as origin failed had to add the ip to /etc/hosts.

reply


https://github.com/ is up for me. North East

reply


github.com itself isn't resolving in some parts of the world. status.github.com is a beautiful default nginx 404 page.

reply


Mongo Cloud is having trouble too!

reply


Wiki leaks news held back all day for the most part the 20th. Clinton mafia attack. Had diplo,Ativan dinner in n y. Last night

reply


LinkedIn?

reply


Dyn seems down

reply


Include all website that use Github to sign-in or sign-up

reply


gitter as well (https://twitter.com/gitchat)

reply


Sentry (at least web dashboard) is down.

reply


Braintree is also experiencing these problems.

reply


We switched from Dyn to Route53 a few weeks ago … lucky.

reply


No parts of Github are up for me (Ottawa). Over to you, Bitbucket.

reply


Need to setup a round robin dns - geographically dispersed

reply


Looks like unsplash.com is down too

reply


CircleCI is also down. As well as github hooks to slack.

reply


CNN and TheGuardian are down as well

reply


Yup, I'm seeing TheGuardian down, from Edinburgh/Scotland.

reply


Need round robin dns - geographically dispersed

reply


Okta seems to be down.

reply


python.org and cpan.org are down.

reply


i have highly available biscuits

reply


Must be the Russians.

reply


PubNub is down.

reply


launchpad.net seems to be down.

edit: Using Google public DNS fixes things.

reply


Heroku is down as well.

reply


a bunch of services using fastly are also impacted

reply


Airbn is also down :(

reply


Intercom as well :(

reply


box.com seems to be down too.

reply


npmjs.com seems to be down as well.

reply


Paypal

reply


thenextweb.com seems to be down as well (connecting from DC)

reply


just add the open DNS you are allset

reply


Works fine from Asia.

(When asking e.g. ns1.p34.dynect.net directly.)

reply


zendesk admin panel is down here in brazil too

reply


use open DNS

reply




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: