Phone Outages for February 19, 2021
-
I opened ticket 98714 a few minutes ago.
-
@JaredBusch said in Phone Outages for February 19, 2021:
@scottalanmiller said in Phone Outages for February 19, 2021:
Who all is experiencing inbound call issues? We have clients nationwide with no inbound calling. Says that number is not in service. Happened at 9:36am central time.
We are getting the same issue from multiple carriers.
@Skyetel is having some kind of upstream issue I assume.
From my T-Mobile cell phone, a call to a client # fails with a network error message.
A call to my company number succeeds.The failing number succeeds when called from a system using a Spectrum (Charter) trunk.
We are seeing voip.ms down as well.
-
Skyetel status was still green when I reported the issue.
Edit: imgur down? I cannot paste in a screenshot
-
That makes it sound like T-Mobile and AT&T have failed. Maybe they've hosed their dialing tables.
-
@scottalanmiller said in Phone Outages for February 19, 2021:
That makes it sound like T-Mobile and AT&T have failed. Maybe they've hosed their dialing tables.
Except, as I noted, I can call the Bundy number and another client number form my cell. those calls completed.
-
not pasting images sucks....
now says identified. upstream provider hosing things.
-
@JaredBusch said in Phone Outages for February 19, 2021:
@scottalanmiller said in Phone Outages for February 19, 2021:
That makes it sound like T-Mobile and AT&T have failed. Maybe they've hosed their dialing tables.
Except, as I noted, I can call the Bundy number and another client number form my cell. those calls completed.
Yes, we have some numbers working, too. Just not many.
-
-
No updates yet for a bit.
-
As I wrote that, we got a call!
-
-
No surprise, but Cloudflare is affected as well...
-
-
It is now working for me to the numbers that were not.
@skyetel is awesome. Cannot wait to see a postmortem.
-
We just published this on our status page, but here's what happened:
From approximately 7:45am to 8:45am PST, we were impacted by a routing issue with one of our carrier partners. This caused inbound phone calls to fail before hitting the Skyetel network, and resulted in callers greeted with error tones.
One of the pillars of the Skyetel architecture is our redundant routing, and many of our customer's have asked why it failed today. We do our best to be transparent, and so here’s what happened.
As part of our interconnection agreements with our carrier partners, we require multiple redundant connections from physically separate networks that automatically redirect call flows from an unhealthy region to a healthy one on a call-by-call basis. Ordinarily, this would mean that if a carrier partner was having an issue in one of their data centers, Skyetel's traffic would be automatically shifted into a different data center instantly.
Unfortunately, the routing table that we established with our partner that dictates how calls should be routed was mistakenly deleted by an authorized member of our carrier partner's team. This prevented our routing tables from being followed and resulted in the call failures observed this morning. As soon as this routing table was reestablished, we were able to reenable our call routing and service was immediately restored.
We have been, and will continue to be, in talks at the highest level with our carrier partner about how this happened, and what safeguards are available to prevent this from happening again. Skyetel’s President is addressing this personally, and all options are on the table. He’s, uh, pretty upset.
It’s is important to note that we do not use this carrier partner for a lot of Skyetel's transit. As such, only 7-8% of our DIDs were impacted. Furthermore, no outbound calling or other features were impacted by this.
We know that issues like this are frustrating and we sincerely apologize for any inconvenience this may have caused.
-
@Skyetel said in Phone Outages for February 19, 2021:
Skyetel’s President is addressing this personally, and all options are on the table. He’s, uh, pretty upset.
Hahhhaahahahahhah
