NTP Pool Project

Helping everyone know what time it is.
@[email protected]

[NTP Pool Project @NTPPool]

[status] Monitoring: ok, we are stable again -- hopefully. https://t.co/SS7O5eSklI

[NTP Pool Project @NTPPool]

[status] Investigating: The newly reset database cluster (from last night) decided to blow itself up again. I think the extra query load and data from the new monitoring system is making the setup fragile.

Investigating. https://t.co/RXH9yTJSbH

[NTP Pool Project @NTPPool]

[status] Identified: The database croaked again! I'm restoring from a ~20 minute old backup. https://t.co/Nf35PSdK5V

[NTP Pool Project @NTPPool]

[status] Resolved: Database cluster has been re-installed and stable for about an hour now. https://t.co/qrgF7Y247m

[NTP Pool Project @NTPPool]

[status] Investigating: The database is having an outage (again!). Investigating. https://t.co/mDiz7tWLTK

[NTP Pool Project @NTPPool]

[status] Monitoring: A fix has been implemented and we are monitoring the results. https://t.co/RWyXTJdwqq

[NTP Pool Project @NTPPool]

[status] Completed: The monitoring system was paused for about an hour, otherwise the upgrade was uneventful.

https://t.co/GDJEQP9j1u https://t.co/EfVCEnl4xT

[NTP Pool Project @NTPPool]

[status] Scheduled (Aug 3, 2025, 00:45 UTC): The system will be upgraded to support the new monitoring agent.

https://t.co/e6txcDsp93
https://t.co/MLlr9ORgDS https://t.co/YX3l5ye4YZ

[NTP Pool Project @NTPPool]

[status] In Progress: Scheduled maintenance is currently in progress. We will provide updates as necessary. https://t.co/prRkGBiJs7

[NTP Pool Project @NTPPool]

[status] Monitoring: It's been recovered (since about an hour ago). More information at https://t.co/ksitrK0szh https://t.co/jb5HPPO0pJ

[NTP Pool Project @NTPPool]

[status] Investigating: The upgraded database is having a hard time, again! https://t.co/n7aUVTNiA1

[NTP Pool Project @NTPPool]

We published a postmortem on our July 3rd incident: https://t.co/1cupYKLmXr

[NTP Pool Project @NTPPool]

[status] Investigating: A couple days ago I upgraded the older MySQL cluster to a newer one. It worked fine and then ... DIdn't.

The DNS and NTP services continue to operate, but the management website and monitoring is having a comp… https://t.co/RO4fKZcPZb

[NTP Pool Project @NTPPool]

[status] Resolved: This incident has been resolved. https://t.co/ZHYSFUkhok

[NTP Pool Project @NTPPool]

[status] Resolved: The system has been fully functional since the previous update; we've been monitoring since. https://t.co/McfroX1B2P

[NTP Pool Project @NTPPool]

[status] Identified: A fiber cut is affecting the management system kubernetes cluster. The NTP Pool DNS service is unaffected.

https://t.co/jVT66m1Ojd https://t.co/8LrGYNSDyB

[NTP Pool Project @NTPPool]

[status] Identified: Fiber cut has been fixed and kubernetes cluster restored (over night). https://t.co/YPnCDoXAQk

[NTP Pool Project @NTPPool]

[status] Monitoring: The immediate problem was fixed a couple hours ago. A complete fix will be implemented, likely with a brief outage, later today.

A few server operators got erroneously notified by the monitoring system when the s… https://t.co/reOcOCyHz5

[NTP Pool Project @NTPPool]

[status] Investigating: One of the database servers for the central system is malfunctioning.

Serving DNS queries is unaffected. https://t.co/gK8nRUkwj1