RIT SE Systems

RIT Software Engineering Systems

[RIT SE Systems @RITSESystems]

SE email is flowing once again. The mail server has been moved to new hardware.

[RIT SE Systems @RITSESystems]

All SE systems should be back up after the electrical outage.

[RIT SE Systems @RITSESystems]

Apparently one of our UPSs had a power issue. That UPS contained one of our VM hosts so nitron, control, the website, and some other support machines were down. I'm checking on their recovery now.

[RIT SE Systems @RITSESystems]

All OpenBSD/UNIX servers are back up and going using the new file server.

[RIT SE Systems @RITSESystems]

OpenBSD (unix) servers will go down at 8:30pm for the cutover to the new file server.

[RIT SE Systems @RITSESystems]

Mightor went down with the power and three of our VM hosts did not deal well. Reboots are fixing the problem.

[RIT SE Systems @RITSESystems]

Power blinked in the building. One of our UPSs seems to be bad. Assessing the damage.

[RIT SE Systems @RITSESystems]

The network jack for one of our VM hosts went dead and disconnected all its VMs from the network.
The host has been moved to another network jack and all services are back online.

[RIT SE Systems @RITSESystems]

The mail submission server was upgraded earlier and there were a couple bumps. Mail is flowing normally now.

[RIT SE Systems @RITSESystems]

nitron didn't want to communicate with mightor after mightor's crash, so I restarted nitron. Service is restored.

[RIT SE Systems @RITSESystems]

All services are restored and seem stable. Have a good night.

[RIT SE Systems @RITSESystems]

Board is swapped. Verifying service restoration.

[RIT SE Systems @RITSESystems]

Systems downtime starting in 5 minutes.

[RIT SE Systems @RITSESystems]

Systems will be down starting at 8pm tonight in order to install a new control board for the file server. I expect the whole process to take less than 2 hours. Updates will be posted here.

[RIT SE Systems @RITSESystems]

Well, after being up for more than 48 hours and giving hope the problem might be solved, mightor has crashed again. This time the vendor and I should have more information to find the cause.

[RIT SE Systems @RITSESystems]

Windows shares have been re-enabled. System is staying up. I think we have a root cause. Just need to verify things.

[RIT SE Systems @RITSESystems]

Mightor has panicked three times so far today. I'm working on it.

[RIT SE Systems @RITSESystems]

Windows shares are temporarily disabled while the problem is worked on.

[RIT SE Systems @RITSESystems]

There seems to have been a brief network interruption. It is being investigated.

[RIT SE Systems @RITSESystems]

It's unclear what happened but nitron went unresponsive. Ultimately I had to hard reset the VM. Things seem back to normal now.