FSMG

The Free Software Mirror Group. We provide New Zealand mirrors for free and open source software projects.

[FSMG @fsmgnz]

Want to help us decide what to mirror? Open an issue for something new at https://t.co/n1jNmT2wUX or give an existing issue a thumbs-up

[FSMG @fsmgnz]

If you've noticed slower than normal download speeds to FSMG over the past few weeks, there was a fault on one of the paths used by our upstream provider that affected a small subset of our traffic

This path has been isolated while it is being repaired which has fixed the issue

[FSMG @fsmgnz]

@therussdotcom @CatalystNZ Fixed. Thanks for reporting the issue.

[FSMG @fsmgnz]

The proper fix has now been deployed and our monitoring looks healthy again.

[FSMG @fsmgnz]

You may have had issues reaching FSMG in the past hour.

We had some firewall rules unintentionally start dropping some internal traffic which confused our anycast BGP system.

This caused our service to be unavailable in small bursts between 9pm and 10pm

[FSMG @fsmgnz]

A workaround has now been implemented now which restored connectivity and we're busy implementing a proper fix.

[FSMG @fsmgnz]

A big thanks to @MetanameNZ for helping us quickly fix the issue by rolling back our DS records to use our original KSK. As well as helping us fix the root problem so this won't occur again.

[FSMG @fsmgnz]

We are aware of an issue with our DNS infrastructure causing invalid DNSSEC responses right now. The FSMG team is currently working on a fix.

[FSMG @fsmgnz]

We got in a state where we had both our registrar's DNSSEC infra as well as our own enabled

A KSK rollover occurred on one set of infra which got published as the DS record for https://t.co/OjyBaanSJV

Since our zone was signed with a different KSK, DNSSEC validation would fail

[FSMG @fsmgnz]

@_Drag0s_ Hi! We have an issue open for this at https://t.co/O0u87Aav0u however we unfortunately don’t have enough capacity at the moment.

[FSMG @fsmgnz]

This upgrade was completed successfully.

^BC

[FSMG @fsmgnz]

This afternoon at 2:00pm UTC+12 we will be taking our Wellington node out of service for 1.5 hours to upgrade it to the latest stable version of Debian.

We don't expect any outage as our Hamilton node will take over serving content during this window.

[FSMG @fsmgnz]

This work was completed successfully.

Initial speed testing looks good, we can now pull traffic between our two nodes at 828 megabytes per second!

[FSMG @fsmgnz]

In an hour from now we will be performing maintenance on our Wellington node to upgrade it to 10gig.

Traffic should be automatically rerouted to our Hamilton node so we don't expect an outage.

[FSMG @fsmgnz]

@__fincham @ss2342 I guess one caveat is our current hardware probably can't saturate full 10gig. Our next generation of hardware will.

[FSMG @fsmgnz]

@ss2342 It sure does, if you are on the REANNZ network you can access the node over their 100gig national backbone, otherwise you will be limited to the speed your provider peers with REANNZ at (normally 10gig at every IX)

[FSMG @fsmgnz]

This upgrade was completed successfully, let us know if you spot any issues.

^BC

[FSMG @fsmgnz]

This afternoon at 1:00pm UTC+12 we will be taking our Hamilton node out of service for an hour to upgrade it to the latest stable version of Debian.

We don't expect any outage as our Wellington node will take over serving content during this window.

[FSMG @fsmgnz]

We're investigating some brief outages on traffic to our Hamilton site. ^MF