SWITCHengines

The Swiss Academic Cloud Service

[SWITCHengines @SWITCHengines]

All systems are working as expected. We have formally closed the incident.

Sorry for the problems this has caused. We are investigating why the switch started to act up.

^jcf

[SWITCHengines @SWITCHengines]

We have been able to fix the root cause (a switch was blackholing internal traffic in our compute and storage clusters). This has been fixed for an hour now and we have seen most services recover.

Our engineers are monitoring the situation.

^jcf

[SWITCHengines @SWITCHengines]

We are experiencing problems with various parts of SWITCHengines that lead to various problems with services running on it. We are assembling a team of engineers now to diagnose and work on it.

^jcf

[SWITCHengines @SWITCHengines]

Data, data everywhere...come help us keep it safe https://t.co/oZCUomkBAw

[SWITCHengines @SWITCHengines]

The SWITCHengines dashboard is currently unavailable. VMs are running normally but it is not possible to make changes to VMs or to provision new projects or VMs via the web. API/command line access is still available. We are investigating.
^AHa

[SWITCHengines @SWITCHengines]

Communication via the message queuing system has been restored. All functions are reporting normal operation in Lausanne again.
^AHa

[SWITCHengines @SWITCHengines]

Control plane operations such as starting/stopping/resizing VMs are currently unavailable in Lausanne. Running VMs are not impacted. Underlying connectivity is verified, and we continue to investigate.
^AHa

[SWITCHengines @SWITCHengines]

Performance on control plane in ZH degraded again this afternoon. We have increased the resources for key components which has relieved the performance issues. We are continuing to monitor closely while debugging root cause.
^AHa

[SWITCHengines @SWITCHengines]

Performance stabilised in ZH from 13:00 following troubleshooting. It is now possible to provision or deprovision VMs with reasonable response times. Running VMs were not impacted.
^AHa

[SWITCHengines @SWITCHengines]

Since c 11:30 see degraded performance on SWITCHengines control plane Zürich. Running VMs are not impacted but creating, deleting or changing VMs is impacted and some APIs are not responding. We are investigating and may need to restart some systems to restore service.
^AHa

[SWITCHengines @SWITCHengines]

Supporting new services in 2021 - after a successful pilot we're looking for expressions of interest in Office365 backup with your Data@SWITCH @SWITCH_ch ICT Focus.

[SWITCHengines @SWITCHengines]

Operational improvements 2021 - more OpenStack upgrades, new S3 cluster, ISMS maturity @SWITCH_ch ICT Focus

[SWITCHengines @SWITCHengines]

Our strategic focus areas 2021 - Data@ SWITCH, supporting data management and data science use cases for higher education & research in CH @SWITCH_ch ICT Focus

[SWITCHengines @SWITCHengines]

We're part of the basic infrastructure roots in the swissuniversities Open Science vision @SWITCH_ch ICT Focus

[SWITCHengines @SWITCHengines]

BlueBrain Nexus on k8s running on SWITCHengines presented at @SWITCH_ch ICT Focus 2020

[SWITCHengines @SWITCHengines]

@widodh @larsmb @danvanderster @Ceph Thanks to @simonl for pointing out that we have an older cluster:

$ ceph osd dump | head -n 3
epoch 2228763
fsid 2a246621-98fc-40dd-829d-8e885452627e
created 2014-06-26 19:26:34.724491

[SWITCHengines @SWITCHengines]

@widodh @larsmb @danvanderster @Ceph Oh damn - we are the youngsters on the block:

[SWITCHengines @SWITCHengines]

Connectivity in Lausanne has been restored. Cause was a subset of IPv4 routes being blackholed unexpectedly.
^AHa

[SWITCHengines @SWITCHengines]

We are experiencing network problems in our LS region. This causes VM IPv4 connections interrupted. We are investigating the problem.

^HR

[SWITCHengines @SWITCHengines]

We are experiencing network problems in our LS region. This causes VM IPv4 connections interrupted. We are investigating the problem.

^HR

[SWITCHengines @SWITCHengines]

All assignments of IPv4 addresses have been restored. If you still experience problems, please file a ticket with [email protected]