Update - We have rerouted Sao Paulo traffic to Rio de Janeiro while we investigate further, which only adds ~6ms of RTT to devices routing to Sao Paulo. There should not be any packet loss at this time.
We may have to temporarily re-enable Sao Paulo with the packet loss occurring to continue troubleshooting, but this will be performed during off-hours and in small windows.
Sep 15, 2023 - 19:32 UTC
Investigating - Our Sao Paulo location is currently experiencing heavy packet loss.
We are investigating and will provide updates as they come.
Sep 15, 2023 - 18:32 UTC
Recursive DNS Services
?
Degraded Performance
90 days ago
99.99
% uptime
Today
Current Status
?
Degraded Performance
90 days ago
99.99
% uptime
Today
Past incidents (last week)
Operational
90 days ago
100.0
% uptime
Today
Scheduled Maintenance
Operational
90 days ago
100.0
% uptime
Today
Threat Intelligence API
?
Operational
90 days ago
100.0
% uptime
Today
Website
?
Operational
90 days ago
100.0
% uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Related
No incidents or maintenance related to this downtime.
Resolved -
Service in our Seattle (SEA) location was severely degraded and may have been completely unusable.
This affected the following ISPs (among others): * Comcast * Charter/Spectrum * Shaw (CA) * T-Mobile
We have identified the root cause as a bug within our monitoring and automated remediation processes, which is the system that determines if servers are healthy and should announce (be in production). We are working on improving this processes immediately to prevent similar situations in the future.
If you are still experiencing issues in Seattle, please reach out to support@quad9.net
Issue Start: 3:00 UTC, September 14th Issue Stop: 6:52 UTC, September 14th
Sep 14, 07:17 UTC
Resolved -
Service in our Sydney (SYD) location was severely degraded and may have been completely unavailable from 1:16 UTC to 3:45 UTC.
Some ISPs in Australia route to our other Sydney PoP (qsyd2), so only some ISPs were affected.
We have identified the root cause as a bug within our monitoring and automated remediation processes, which is the system that determines if servers are healthy and should announce (be in production). We are working on improving this processes immediately to prevent similar situations in the future.
If you are still experiencing issues in Sydney, please reach out to support@quad9.net
Issue Start: 1:20 UTC, September 14th Issue Stop: 6:55 UTC, September 14th
Sep 14, 07:08 UTC