Systemwide outage

2022-12-01 20:00: We are operating at reduced capacity over a backup connection at this time and have reduced bandwidth. The internet is going to be on, but slow. Thank you for your understanding.

2022-12-01 16:51: All services restored as of 16:45. A huge thanks to support provided to Cogent by ABQ IX and the private members who did the heavy lifting to bring us back online.

Thank you all for your patience, please watch for a more detailed postmortem.

2022-12-01 14:08:Cogent has gained access as of 1:01 PM and is working with a local exchange provider to bring all equipment back online. They have not provided an ETR, but this is large progress. More to come as we know more.

EDIT 2022-12-01 9:30: We just received an update from Cogent saying they will have access to repair their failed equipment and restore our service at 14:00 this afternoon. As we receive more updates we will update our socials and messages.

EDIT 2022-12-01 3:00: As of 3:00 AM Dec 1st, Cogent crews have been unable to gain access to the damaged equipment responsible for this outage. They have rescinded their previous ETR and do not have an update at this time.

We sincerely apologize for this continued outage and are working to restore service across different avenues at this time. Thank you all for your ongoing patience.

EDIT 2022-11-30 21:40: We just received word that Cogent, the party responsible for the current outage, will gain access at 12:00 tonight and estimate a service restoration time of approximately 06:00 on the 1st. We have our fingers crossed. Thank you for your patience.

EDIT 2022-11-30 14:47: As of writing, our services remain down and we have no estimated time of resolution. Repair crews are still awaiting access to begin their repairs in a data center in Albuquerque, where the issue is centered. As soon as we have more information we will provide an update.

EDIT 2022-11-30 11:00: As of 11:00 AM we are still down and do not have a solid time of resolution. Repair crews are still awaiting secure access to complete their repairs. As soon as we have more information we will provide an update.

EDIT 2022-11-30 05:47: Update received at 5:23 this morning states crews are working to gain access to an area controlled by a non-responsive site manager in Albuquerque to complete the repair. No ETR at this time. Thank you all for your patience.

EDIT 21:36: We just received word that the outage is due to damage in the middle of the state caused by a pole failure that also had powerline involvement. Services and crews are working to repair damaged fiber now. We do not have an official ETR, but we believe services are likely to be restored tonight.

As of 16:10 today 29 Nov 2022 we are experiencing a systemwide outage from our upstream provider, Cogent. We are currently awaiting status updates at this time and do not have an estimated time of resolution. Thank you for your patience.