Update - We have rerouted Sao Paulo traffic to Rio de Janeiro while we investigate further, which only adds ~6ms of RTT to devices routing to Sao Paulo. There should not be any packet loss at this time.

We may have to temporarily re-enable Sao Paulo with the packet loss occurring to continue troubleshooting, but this will be performed during off-hours and in small windows.

Sep 15, 2023 - 19:32 UTC
Investigating - Our Sao Paulo location is currently experiencing heavy packet loss.

We are investigating and will provide updates as they come.

Sep 15, 2023 - 18:32 UTC
Recursive DNS Services ? Degraded Performance
90 days ago
99.99 % uptime
Today
Current Status ? Degraded Performance
90 days ago
99.99 % uptime
Today
Past incidents (last week) Operational
90 days ago
100.0 % uptime
Today
Scheduled Maintenance Operational
90 days ago
100.0 % uptime
Today
Threat Intelligence API ? Operational
90 days ago
100.0 % uptime
Today
Website ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Sep 16, 2023

No incidents reported today.

Sep 15, 2023

Unresolved incident: Sao Paulo (QGRU1) - Packet Loss.

Sep 14, 2023
Resolved - Service in our Seattle (SEA) location was severely degraded and may have been completely unusable.

This affected the following ISPs (among others):
* Comcast
* Charter/Spectrum
* Shaw (CA)
* T-Mobile

We have identified the root cause as a bug within our monitoring and automated remediation processes, which is the system that determines if servers are healthy and should announce (be in production). We are working on improving this processes immediately to prevent similar situations in the future.

If you are still experiencing issues in Seattle, please reach out to support@quad9.net

Issue Start: 3:00 UTC, September 14th
Issue Stop: 6:52 UTC, September 14th

Sep 14, 07:17 UTC
Resolved - Service in our Sydney (SYD) location was severely degraded and may have been completely unavailable from 1:16 UTC to 3:45 UTC.

Some ISPs in Australia route to our other Sydney PoP (qsyd2), so only some ISPs were affected.

We have identified the root cause as a bug within our monitoring and automated remediation processes, which is the system that determines if servers are healthy and should announce (be in production). We are working on improving this processes immediately to prevent similar situations in the future.

If you are still experiencing issues in Sydney, please reach out to support@quad9.net

Issue Start: 1:20 UTC, September 14th
Issue Stop: 6:55 UTC, September 14th

Sep 14, 07:08 UTC
Sep 13, 2023

No incidents reported.

Sep 12, 2023

No incidents reported.

Sep 11, 2023

No incidents reported.

Sep 10, 2023

No incidents reported.

Sep 9, 2023

No incidents reported.

Sep 8, 2023

No incidents reported.

Sep 7, 2023

No incidents reported.

Sep 6, 2023

No incidents reported.

Sep 5, 2023

No incidents reported.

Sep 4, 2023

No incidents reported.

Sep 3, 2023

No incidents reported.

Sep 2, 2023

No incidents reported.