The incident has been resolved following the completion of the rollback deployment across the Gitaly nodes. All nodes have returned to a stable state, repository access has been restored. Incident team is currently investigating the root cause. https://t.co/tv9buxOgRs
Gitaly nodes have stabilized following the completed rollback deployment. The incident has been mitigated and we remain focused on continued monitoring. https://t.co/tv9buxOgRs
A rollback release has been deployed to all Gitaly nodes. They are stable at present. However, we are continuing to monitor the situation. https://t.co/tv9buxOgRs
We are currently rolling back the deployment of Gitaly. We're waiting for the deploy to land, and we are continuing the investigation. https://t.co/tv9buxOgRs
Further mitigation change are currently being applied, and we are continuing to investigate the issue. HTTP 500 errors may continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow. https://t.co/tv9buxOgRs
A further mitigation change has been applied, and we are assessing its impact. HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow. https://t.co/tv9buxOgRs
Further mitigation is being rolled out while we continue to investigate HTTP 500 errors affecting the GitLab UI and Git operations. We will provide further updates as the mitigation progresses. https://t.co/tv9buxOgRs
The restart of the affected Gitaly node did not improve the issue; HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. We are investigating and will provide further updates as the situation changes. https://t.co/tv9buxOgRs
Investigation continues. We will shortly be restarting the affected Gitaly node after reconfiguration. Further updates will be provided accordingly. https://t.co/tv9buxOgRs
Mitigation has not yet fully restored the GitLab UI and Git operations in certain projects. We are still investigating HTTP 500 errors and will provide a further update as more information becomes available. https://t.co/tv9buxOgRs
We are continuing mitigation and monitoring recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations for certain projects. We will provide a further update as mitigation progresses. https://t.co/tv9buxOgRs
Weโre continuing mitigation efforts and have observed some recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations when accessing certain projects. https://t.co/tv9buxOgRs
Weโve identified an abuse vector as the root cause and begun mitigation efforts. Early signs indicate that the service is recovering. https://t.co/tv9buxOgRs
We're investigating reports of customers unable to open projects, receiving error 500 instead. Details in https://t.co/jgDMJ5bXKY. https://t.co/tv9buxOgRs
We are no longer observing elevated error rates and have identified the root cause. We are marking this incident as resolved while we implement additional fixes behind the scenes. https://t.co/tv9buxOgRs
We are still seeing cases but tat a much less frequent rate. We will continue looking for the root cause and monitor the error rate. https://t.co/tv9buxOgRs
Weโve detected issues affecting artifact uploads. The team is actively investigating, and weโll provide another update as soon as we have findings to share. Thank you for your patience. https://t.co/tv9buxOgRs
Policy evaluations and MR approvals are executing normally. We're continuing to monitor the situation while working on improvements to prevent this in the future. More: https://t.co/HbEHEwJn8x https://t.co/tv9buxOgRs