Network Status News & Information

On 9 June 2026 the Stockholm, Sweden site experienced a service disruption affecting external connectivity. The incident's timestamps are given in UTC:

• 2026-06-09 06:51:45 - 2026-06-09 08:48:59 (1h 57m 14s) - ER-SE cluster;
• 2026-06-09 07:48:59 - 2026-06-09 08:48:59 (59m 57s) - XSIG-SE cluster.

According to our Service Level Agreement, all impacted customers have received a compensation of 3 days automatically for all impacted Stockholm, Sweden KVM VPS services.

The initial trigger was a diagnostic show bgp ipv4 regexp query whose pattern contained a nested quantifier (.+_)+ subject to catastrophic backtracking. Evaluating it saturated a CPU core inside the FRR BGP daemon (bgpd) for over four minutes, blocking its main thread. The watchdog deemed the daemon unresponsive and restarted it; the shutdown path then hit a segmentation fault and the process aborted. The cause of that segfault is still under investigation.

The trigger on its own would have been a brief disruption. Three separate conditions turned it into a two-hour outage:

• The daemon hit the same segmentation fault on every shutdown, so each restart aborted and bgpd entered a restart loop that never reached a clean, converged state.
• Graceful restart was configured with restart-time, select-defer-time and rib-stale-time all set to 3600 seconds. The daemon defers best-path selection until every peer has finished re-sending its routes or that one-hour deadline expires; every restart reset the wait, so selection never ran. The full IPv4 table was received and validated, routes were usable but no best paths selected on either the inter-cluster session or the AS208453 upstream. The condition cleared only once the restarts stopped and a single instance was left to run long enough to converge.
• Separately, a weird IPv6 link-local next-hop issue on the AS3399 session blocked IPv6. AS3399 advertised IPv6 routes with link-local next-hops that FRR could not resolve (sent a v6 LL next-hop and there's no peer interface), so no IPv6 routes were installed. This cleared once AS3399 was migrated and link-local next-hop capability was negotiated.

Forwarding continued on previously installed routes for roughly an hour under graceful restart, until those routes were withdrawn. More information about each event is provided below in a timeline.

2026-06-09 06:42-06:47 UTC. Diagnostic query saturates the BGP daemon

A show bgp ipv4 regexp query was issued with a pattern containing a nested quantifier (.+_)+ that is subject to backtracking. Evaluation consumed a CPU core for the entire duration (CPU HOG: command took 257905ms), blocking the daemon's main thread. The watchdog declared the daemon unresponsive (bgpd state -> unresponsive : no response yet to ping sent 90 seconds ago) and issued a restart. The shutdown triggered a segmentation fault in the daemon's thread-teardown path, and the process aborted. The cause of the segfault is under investigation.

2026-06-09 06:47-08:30 UTC. Restart loop and selection/install failures

The BGP daemon was restarted repeatedly. Each shutdown hit the same segmentation fault, so no restart reached a clean, converged state. On the sessions that did establish, the full IPv4 table was received, but graceful restart deferred best-path selection and the restart loop kept resetting that wait, so no best paths were selected on either the inter-cluster session or upstreams. IPv6 was blocked separately: the AS3399 upstream session advertised IPv6 routes with IPv6 link-local next-hops that FRR could not resolve (sent a v6 LL next-hop and there's no peer interface), so no IPv6 routes were installed either. Restart attempts also surfaced transient finite-state-machine errors such as [FSM] Failure handling event BGP_Start in state Idle ... last reset: No AFI/SAFI activated for peer. Customer connectivity was lost for both address families.

2026-06-09 06:52 UTC. ER-SE loses reachability

The ER-SE cluster lost external connectivity shortly after the FRR crash. Holding no independent upstream path other than the IXP, network availability degraded significantly.

2026-06-09 07:49 UTC. Graceful-restart grace expires

The XSIG-SE cluster had retained forwarding on previously installed routes thanks to graceful restart. When the grace period expired, the stale routes were withdrawn and the relevant services lost connectivity completely. Off-site monitoring from multiple locations confirmed the loss at this point.

2026-06-09 07:50-08:05 UTC. Unscheduled maintenance: upstream migration

AS3399 had historically been terminated on the XSIG-SE routing engine. The recently added AS208453 was deliberately brought up there as well, given the smaller capacity of the equipment there and because AS208453 provides volumetric filtering. Implementation of the multihomed design (migrating AS3399 to ER-SE and leaving AS208453 as XSIG-SE's primary upstream) had been intentionally deferred until AS208453's routing was proven stable, at which point the move was to be made without customer-facing impact. The outage forced it early. The AS3399 session established on ER-SE at approximately 08:05 UTC and traffic began recovering through it.

2026-06-09 08:30-08:49 UTC. Primary router recovery

A clean restart of the XSIG-SE routing stack, without the AS3399 session, brought BGP up and best-path selection finally converged, so route installation began to succeed. A final reset of the remaining transit session, with link-local next-hop capability negotiated, cleared the last stuck routing state.

The network was completely healthy by 08:48:59 UTC, as confirmed by external monitoring.

Corrective actions

The following actions address the conditions that prolonged the incident:

• Graceful-restart timers: we are reviewing the restart-time, select-defer-time and rib-stale-time values (currently 3600 seconds each) so that recovery is no longer gated on a one-hour convergence deadline and repeated restarts cannot indefinitely defer best-path selection.
• Multihomed upstream design: the migration of AS3399 to ER-SE, with AS208453 retained as XSIG-SE's primary upstream, was completed during the incident and remains in place, so neither cluster now depends on a single upstream beyond the IXP.

The following attack waves towards Stockholm, Sweden were observed:

• 2026/04/16 18:19:00 - 2026/04/16 18:23:00
• 2026/04/16 18:26:15 - 2026/04/16 18:29:00
• 2026/04/16 18:34:15 - 2026/04/16 18:36:45
• 2026/04/16 18:39:45 - 2026/04/16 18:41:30

The following attack waves towards Poland, Warsaw were observed:

• 2026/04/16 18:38:30 - 2026/04/16 18:41:45
• 2026/04/16 18:44:00 - 2026/04/16 18:46:00

After traffic analysis, we have came with a conclusion that attack was not aimed at any particular customer, rather it targeted our network in both Stockholm, Sweden and Warsaw, Poland locations.

Unfortunately, we do not have capacity to absorb this kind of traffic without impact for our customers, therefore urgent priority is given to seek upstream specialized in DDoS protection for networks (unlike fine-tuned filters for a particular type of service like web or gaming servers).

Between April 6 and April 8, 2026, a series of 8 related events and 1 unrelated DDoS event that together caused multiple periods of service disruption. The incidents' timestamps are given in UTC timezone:

• 2026-04-06 16:30:47 - 2026-04-06 16:47:49 (17m 2s);
• 2026-04-06 17:03:57 - 2026-04-06 17:10:32 (6m 35s);
• 2026-04-06 17:13:53 - 2026-04-06 17:15:12 (1m 19s);
• 2026-04-06 17:19:51 - 2026-04-06 17:23:14 (3m 23s);
• 2026-04-06 17:39:31 - 2026-04-06 18:43:48 (1h 4m 17s);
• 2026-04-06 18:49:33 - 2026-04-06 18:51:40 (2m 7s);
• 2026-04-07 18:50:38 - 2026-04-07 18:56:22 (5m 44s);
• 2026-04-08 09:31:29 - 2026-04-08 09:33:39 (2m 10s);
• 2026-04-08 13:59:09 - 2026-04-08 14:03:37 (4m 28s).

According to our Service Level Agreement, all impacted customers have received a compensation of 3 days automatically for all impacted Stockholm, Sweden KVM VPS services.

The initial trigger was the launch of a performance diagnostic tool that caused all QEMU virtual machines on the host to consume 100% of their allocated vCPU. Resetting the affected VMs led to a crash of the FRR routing daemon, which in turn exposed configuration management issues that prolonged recovery over the following days. A separate, unrelated DDoS event targeting an upstream provider caused a final disruption on April 8.

As a result of these events, the following corrective actions were taken:

• During one of the events, server reboot was performed (2026-04-06 17:39:31-18:43:48), which updated the host kernel to 5.14.0-570.49.1.el9_6.x86_64. It includes the required upstream fix;
• Deployment of a lab environment replicating the exact production network configuration is to be scheduled. All configuration changes must be validated in this environment in a scenario of a full daemon restart.

This incident demonstrated that reliance on live patching alone is insufficient - the relevant fix carried no CVE identifier and therefore was never delivered as a live patch. A transition toward scheduled full kernel restarts is required to ensure all upstream fixes, including those without CVE classification, are applied in a timely manner. The primary obstacle is the absence of a spare hypervisor node for each cluster type, which prevents live migration of guest virtual machines during maintenance. Deployment of additional hardware is to be scheduled to enable zero-downtime kernel updates going forward.

The case of configuration load failure is more straightforward and could've been easily avoided - root cause was a route-map entry combining a deny verdict with an on-match goto NUMBER statement. This combination was successfully applied at runtime via frr-reload.py --reload /etc/frr/frr.conf and therefore was overlooked. However, ingestion of the full configuration during FRR startup caused not only route-map to be interpreted incorrectly, but impacted some other parts of the configuration. This left the system in a broken state with established BGP sessions, but broken filtering - FRR indicated that all inbound prefixes were filtered out and nothing was announced, but this was not entirely accurate, as some prefixes were still being routed prior to the server reboot.

More information about each event is provided below in a timeline.

2026-04-06 16:30:47-16:47:49. System profiling tool causes CPU saturation

The perf top profiling tool was launched on a physical server running Linux kernel version 5.14.0-427.40.1.el9_4.x86_64 on an Intel Skylake-SP (Xeon Scalable 1st Generation) host. This caused all QEMU virtual machines to either hang or enter a fault/retry loop (page-faults on the PEBS DS buffer address and the kernel attempts to handle it repeatedly) causing complete consumption of allocated vCPU resources. Stopping perf top did not resolve the CPU saturation, therefore all affected virtual machines were individually reset to mitigate the issue. Unfortunately, reset process took a significant amount of time and ultimately caused the FRR Zebra daemon to crash, leading directly to the next incident.

The root cause is a kernel bug in KVM's PMU emulation for Precise Event Based Sampling (PEBS). Running perf against a QEMU/KVM process causes guests to suffer a page fault when the host attempts to store PEBS records into guest memory. When used system-wide, this affects all running guests on the host.

Critically, the el9_4 kernel had the PEBS emulation feature backported without the corresponding fix. The definitive fix was backported only in el9_6. Despite the use of live patching, it had not delivered this fix either, as the upstream commit carries no CVE identifier.

2026-04-06 17:03:57-17:10:32. FRR crash and route withdrawal

Following virtual machine resets, the Zebra daemon (component of the FRRouting suite) crashed. The BGP graceful-restart timer expired before sessions were re-established, causing routes to be withdrawn by our peers. Upon restart, FRR failed to load its saved configuration, preventing automatic recovery.

2026-04-06 17:13:53-17:15:12 and 2026-04-06 17:19:51-17:23:14. Failed FRR restart attempt

A further attempt was made to restart FRR. The same configuration that had been running successfully prior to the crash could not be properly loaded due to an obscure error message commit failed session-id 2 on Unknown-FD-16 req-id 3 source-ds: candidate target-ds: running validate-only: 0: reason: 'Failed to create cfgdata: config validation failed'.

2026-04-06 17:39:31-18:43:48. Server reboot

Due to concerns about possible undefined behaviour beyond the observed QEMU vCPU saturation, the physical server was rebooted. The concerns turned out to be unfounded.

On boot, FRR was intentionally asked to load a much older configuration backup, as the current configuration could not be read. This backup lacked route-map definitions for action communities (215467:1{0,1,2,3,9}:PEER-ASN). BGP sessions came up and FRR operated as expected. However, further investigation into configuration load failures were due.

2026-04-06 18:49:33-18:51:40. Route-map configuration attempt fails

An attempt to add the missing route-maps with the correct action-community definitions was unsuccessful. The changes were reverted to avoid further disruption.

2026-04-07 18:50:38-18:56:22. Corrected configuration deployment

A corrected FRR configuration was prepared and deployed to prevent recurrence of the configuration-load failures. An FRR restart was attempted to validate the fix, but the initial load failed due to a stale lock on the management datastore:

[Z0W46-Z41X9] mgmt_ds_lock: ERROR: LOCK already taken on DS:running by session-id 2 (requesting session-id 0)

A second restart was required to clear the leftover management datastore candidates. FRR loaded the corrected configuration successfully on the second attempt.

2026-04-08 09:31:29-09:33:39. BGP daemon crash and FSM failure

The BGP daemon crashed. An automated restart was attempted but failed due to a finite-state machine error:

[J9K4Q-T8STY][EC 33554466] 45.154.199.2 [FSM] Failure handling event BGP_Start in state Idle, prior events (null), (null), fd -1, last reset: No AFI/SAFI activated for peer

Manual intervention was required due to the failure to automatically re-establish some iBGP sessions in a timely manner.

2026-04-08 13:59:09-14:03:37. Upstream provider faced massive DDoS attack

An upstream provider was hit by a carpet-bombing DDoS attack. Our network was not the target, however, the upstream congestion caused severe collateral packet loss. Some traffic continued to be routed, but throughput dropped drastically from normal levels. Service recovered once the upstream provider mitigated the attack.

On 24th March there were two incidents impacting availability of customers' resources (UTC+0 timestamps):

1. from 2026-03-24 19:41:11 to 2026-03-24 20:05:03 (23m 57s);
2. from 2026-03-24 20:09:28 to 2026-03-24 20:21:29 (12m 1s).

According to our Service Level Agreement, you have received a compensation of 1 day automatically for all impacted Warsaw, Poland KVM VPS services.

The issue originated from a failure in the routine process of updating FRRouting software router. Unfortunately, there was an issue with zebra process causing daemons control plane to be stuck and preventing BGP from functioning properly. Therefore, manual intervention was required, however it took significant time to remediate the underlying issue.

Given the above, we are reviewing our software update procedures to implement safeguards that prevent recurrence.

We sincerely apologize for the disruption and appreciate your continued trust.

We are observing partial network degradation since 2026-03-17 19:34 UTC. As a temporary measure we have rerouted traffic to minimize impact while investigating the issue.

Seems to be resolved as of 2026-03-17 20:19 UTC.

We have observed our uplink (AS3399) to have network degradation causing partial unreachability and BGP session flap.

As of 2025/10/22 02:47, full connectivity restored.

2025/10/22 03:41 - AS3399 / Obenet has recurring network degradation.

2025/10/22 03:49 - connectivity restored.

Unfortunately, there was an unplanned network degradation since 2025-09-18 18:18:17 UTC until 2025-09-18 18:28:43 UTC (10 minutes 26 seconds).

We are sorry for caused inconveniences, please accept our sincere apologies.

Thanks for your patience and trust.

On 30th July there was unscheduled hypervisor node reboot due to kernel crash, impacting customers' servers from 2025-09-11 10:53:00 UTC+0 to 2025-07-30 11:01:12 UTC+0 (7m 48s).

Some virtual machines took longer to boot, however incident resolution does not account that.

We deeply regret the disruption caused and are sorry for any inconvenience. Thanks for your trust.

On 30th July there was unscheduled hypervisor node reboot due to kernel crash, impacting customers' servers from 2025-07-30 03:04:18 UTC+0 to 2025-07-30 03:17:20 UTC+0 (13m 2s).

Some virtual machines took longer to boot, however incident resolution does not account that.

We deeply regret the disruption caused and are sorry for any inconvenience. Thanks for your trust.

We have observed our uplink (AS3399) to have network degradation causing partial unreachability and BGP session flap.

2025/07/19 15:04 - IPv6 prefixes were temporarily rerouted to AS6939

2025/07/19 15:20 - Connectivity stabilized

2025/07/19 15:30 - BGP session with upstream flaps again

2025/07/19 16:55 - No connectivity problems since 15:55 UTC+0

We have observed our uplink (AS3399) to have network degradation causing partial unreachability and BGP session flap. We are looking into this situation.

Update 2025/07/18 09:55 - Upstream (AS3399) resolved the issue, no network problems detected since 09:20 UTC 0.

On 13th July there was unscheduled hypervisor node reboot, impacting customers' servers from 2025-07-13 03:16:15 UTC+0 to 2025-07-13 03:24:00 UTC+0 (7m 45s).

Some virtual machines took longer than usual to boot, incident resolution does not account that.

Handful of VPS had boot problems (less than 5), from our knowledge, all of them were detected and resolved manually by our staff later.

We deeply regret the disruption caused and are sorry for any inconvenience. Thanks for your trust.

We are informing you about planned network maintenance scheduled for tomorrow (Wednesday, 18 June 2025) from 13:00 to 16:00 UTC+2. Network unavailability of up to ten minutes is expected during this time frame.

This maintenance is necessary for an upcoming network upgrade.

Thank you for your understanding.

On 15th April there were two incidents impacting availability of customers' resources:

1. from 2025-04-15 06:29:05 UTC+0 to 2025-04-15 07:04:00 UTC+0 (34m 55s);
2. from 2025-04-15 17:21:57 UTC+0 to 2025-04-15 17:35:07 UTC+0 (14m).

First incident was the hypervisor host system experiencing a kernel heap corruption that ultimately triggered a panic in the nf_tables subsystem and caused virtual machines to reboot. Unfortunately, it is not possible to figure out exact cause of heap corruption. However, after a reboot we have ensured that kernel live patching is actually working and that patches are applied. We have no reasons to suspect hardware being faulty either, Linux EDAC subsystem haven't reported any issues yet.

Second incident was due to an error in the updates installation process that caused network misconfiguration, but was not promptly detected, thus it took a significant time to restore services availability.

According to our Service Level Agreement, you have received a compensation of 2 days automatically for all impacted Warsaw, Poland KVM VPS services.

We deeply regret the disruption caused and are sorry for any inconvenience. Thanks for your continued trust despite this month being challenging for our team.

Our monitoring detected network unavailability from 2025-04-08 07:20:40 UTC+0 to 2025-04-08 09:58:09 UTC+0.

According to our Service Level Agreement, you have received a compensation of 5 days automatically for all impacted Warsaw, Poland KVM VPS services.

The issue originated from a failure in the FRRouting software router, where the watchfrr daemon did not restart the unresponsive bgpd process as expected during a failure scenario like this one. Our investigation is ongoing, as we have not yet determined the precise trigger for the failure of bgpd process.

Contributing factors includes limited staff availability today  at the morning, which significantly delayed resolution. We are now working on the enhancement of protective measures for similar incidents.

We sincerely apologize for the disruption and appreciate your continued trust.

Our monitoring detected hypervisor unavailability since 2025-03-09 11:18:15 UTC+0 until 2025-03-09 11:31:15 UTC+0 that impacted customers' virtual machines.

According to our Service Level Agreement, you have received a compensation of 2 days automatically for all impacted Warsaw, Poland KVM VPS services.

After a thorough investigation, we have determined that this incident was caused by a memory leak resulting from auxiliary software used on our hypervisor nodes. We have implemented corrective measures and are actively monitoring the situation to ensure stability.

Thanks for your patience and trust.

There was a severe packet loss since 2024-07-08 23:12 UTC+0 until 2024-07-09 16:24 UTC+0 on egress at one of our indirect upstream providers, namely Lumen/Level3 (AS3356). Unfortunately, due to a single-homed nature of our network, we were unable to re-route impacted traffic promptly ourselves. Please accept our deepest apologies for caused inconveniences.

Given another recent downtime, we feel sorry for breaking your expectations. According to our Service Level Agreement, you will receive a compensation of 14 days automatically for all KVM VPS services.

Nevertheless, we have a good news for you - as of yesterday (July 12, 2024), our Warsaw site's network is multi-homed and redundant physically thanks to new dedicated connection to Orange S.A. network (AS5511)! This allows us to prevent similar incidents in the future.

If you are concerned regarding our network changes (for example, worse routing than it was before), feel free to reach our technical support department to discuss it further with us.

Thanks for your patience and trust.

Unfortunately, there was an unplanned network downtime since 2024-07-01 13:33:15 UTC until 2024-07-01 13:54:15 UTC (21 minutes). This is due to an issue caused by a system update.

Due to configuration files and interface settings were reset, therefore manual intervention was required.

According to our Service Level Agreement, you will receive a compensation of 1 day automatically.

We are sorry for caused inconveniences, please accept our sincere apologies.

Thanks for your patience and trust.

You could have observe virtual servers' performance degradation in the period since 2024-03-26 16:10:30 until 2024-03-26 16:20:00 and since 2024-03-26 16:21:15 until 2024-03-26 16:27:00 (UTC+0).

Our team has investigated this incident and prepared all necessary fixes to prevent similar issues in the future, however their appropriate deployment requires physical server reboot. Unfortunately, the virtual machine management platform we use (VirtFusion) does not implement managedsave feature, which would allow us to save the state of virtual machines despite hypervisor node reboot.

This letter is to inform you about planned technical maintenance since 2024-04-03 09:00:00 until 2024-04-03 10:00:00 (UTC+0). The expected virtual machine downtime is up to 5 minutes. After the maintenance is completed, you may need to restore the software if it does not support autorun.

Please accept our sincere apologies for caused inconveniences.

We are informing you about urgent planned network maintenance tomorrow (Thursday, 07 March 2024) since 10:00 until 16:00 UTC+0. Brief network unavailability is expected within this time frame.

The maintenance is necessary to improve our network. This will provide us with a flexibility for managing network routing policy ourselves and is a preparation stage for further redundant networking setup.

Thanks for understanding.

We are informing you about two network incidents during the night of February 13 to 14, timezone is UTC+0:

1. since 2024-02-14 01:06:15 until 2024-02-14 01:11:30 (5 minutes 15 seconds),
2. since 2024-02-14 01:17:30 until 2024-02-14 01:42:00 (24 minutes 30 seconds).

These events are related to the technical maintenance on the upstream side. We were assured that proactive measures to minimize any possible impact and careful monitoring will be undertaken, which prevented us from responding in a timely and appropriate manner.

According to our Service Level Agreement, network uptime is 99.95% monthly. The total downtime was 29 minutes 45 seconds, thus all affected services will be extended by 1 day in addition to the period paid at the time of receipt of this communication.

We would like to sincerely apologize for Warsaw VPS unavailability between 2023-11-12 21:04:30 UTC - 2023-11-12 22:55:00 UTC. The cause of this incident is a DDoS attack impacting our entire network.

All the required measures were taken to mitigate similar incidents in the future.

Also, as per our Service Level Agreement policy, we have renewed all impacted services for 3 days.

Thanks for understanding and patience.