VoIP One-way Audio and Voice drops
-
We replaced our old legacy system with a FreePBX based system and moved away from a PRI onto SIP trunks at the end of year last year. Aside from some latency issues with our firewall that affected a few users at a time, we haven't had any issues and have been enjoying some of the more advanced features that going with a phone system newer then the one we had provided.
On Friday I came into the office and there were some serious issues with the phone system. All of our calls were experiencing intermittent one-way audio, the calls would be fine for 5-15 seconds then immediately drops out for 5-30 seconds and then come back up again. Needless to say this wasn't a great thing and has been the first major issue since introducing the system.
9:00AM I began running a ping test against the PBX, SIP Trunk, and Firewall. The PBX and SIP Trunk reported latency between <1 and 20 ms with an average around 4ms. Not as low as usual but not as high as I was thinking. Jitter was around 1% between the PBX and the SIP Trunk. The firewall on the other hand was another thing entirely, we were seeing between <1ms to 3000ms, with an average around 400-500ms (every trace was a bit different on the average). I noticed that when the latency to the firewall was spiking we would get the audio problems. I was assured that this was anticipated and that ICMP was pushed back in the queue in favor of higher priority packets.
I called Meraki Support at 10:00AM, I also reloaded and restarted the FreePBX server at this point thinking it may be related to some memory usage. We began troubleshooting the issue after looking at firewall utilization which was not outside of the norm. We went through a number of different scenarios and captured packet traces on both the LAN and WAN side of the connection while calls were going on and the issue was cropping up. This continued until ~3:30PM. At this point we were seeing ~50-60% packet loss. I had been emailing with our ISP (SIP Trunk Provider) throughout the day as well and they were seeing a high rate of SIP re-transmission packets but nothing was going on with the SIP trunk and the problem was isolated to us on their end.
At 4:00PM the problem resolved itself, calls could be made with no loss of packets or audio interruptions. I tested for an additional two hours and all call I sent and received were completed successfully.
Saturday morning at 8AM I get an email from someone who came in early who said calls would hangup after a few moments. I began testing with this user and calls would last for 5-6 seconds before hanging up due to a critical packet missing (according to the Asterisk CLI). This stopped happening around 10AM when I began to really test it and setup a remote extension to call back and forth from.
Today, I am in the office testing so that hopefully there will be no issues tomorrow. Thus far there doesn't appear to be anything wrong with the latency or drop outs. I've been on a call for 30 minutes now and haven't had any issues.
My question is, if this happens again tomorrow, is there anything I should look at/record so when I bring in the big guns (PBX support company) they have all the information they need to dig into the issue.
-
One way calls are normally the result of networking issues, not the PBX. The firewall is the most common place for this to happen. It can be the result of the NAT tables on the firewall having issues, of SIP-ALG being enabled or of STUN failing for some reason. So I would start there.
- Did anything change on the firewall?
- Is SIP-ALG (sometimes under a different name) getting enabled somehow?
- Did your STUN server (an external thing) go down during that time?
-
Things like latency pretty much cannot cause one way audio. Latency and packet drops do not manifest in this manner.
-
@scottalanmiller said:
- Did anything change on the firewall?
Nothing had changed on the firewall end that I know of. The last update was on Monday of the week before.
- Is SIP-ALG (sometimes under a different name) getting enabled somehow?
I don't see an option for SIP-ALG on the Meraki firewall? Is this where this would be setup or would it be on the PBX itself? I do have SIP (Voice) traffic and traffic destined for the SIP trunk setup with a higher priority then all other traffic in the network. This has been enabled since we started with the FreePBX server last November.
- Did your STUN server (an external thing) go down during that time?
Where would the STUN server be located? Is it generally something that is selected or does it go through the ISP?
-
@scottalanmiller said:
Things like latency pretty much cannot cause one way audio. Latency and packet drops do not manifest in this manner.
That is good to know. The odd relationship between latency and voices drop outs was purely coincidence. If the router's queue was overloaded would you see this behavior or would it be more robotic audio and lagged communications?
-
I tried a SIP-ALG detector from Nextiva (http://dc.nextiva.com/sipalg/). It says there is no SIP-ALG.
-
@coliver said:
Where would the STUN server be located? Is it generally something that is selected or does it go through the ISP?
Generally this is set on the phones and the PBX.
-
@coliver said:
@scottalanmiller said:
Things like latency pretty much cannot cause one way audio. Latency and packet drops do not manifest in this manner.
That is good to know. The odd relationship between latency and voices drop outs was purely coincidence. If the router's queue was overloaded would you see this behavior or would it be more robotic audio and lagged communications?
That does suggest that the router was the issue and having problems. When fully loaded it might have behaved badly.
-
Sounds like it would either be a network issue or the SIP Provider.
Meraki APs are great, there routers not so much but That may not be your issues unless it's undersized for your company.
Re-tramsmit suggests to me it could be a MTU issues or a QoS issue (mis configuration or under powered router)
-
@thecreativeone91 said:
Sounds like it would either be a network issue or the SIP Provider.
Meraki APs are great, there routers not so much but That may not be your issues unless it's undersized for your company.
Re-tramsmit suggests to me it could be a MTU issues or a QoS issue (mis configuration or under powered router)
I was on the phone with Meraki support for several hours on Friday. I've been having an on going issue with their malware protection option that has caused the router to lockup at 100% cpu usage (which you have no way of knowing because their GUI is way too simple), which I've disabled. The support rep was confident that it wasn't a capacity issue.
-
@coliver said:
@thecreativeone91 said:
Sounds like it would either be a network issue or the SIP Provider.
Meraki APs are great, there routers not so much but That may not be your issues unless it's undersized for your company.
Re-tramsmit suggests to me it could be a MTU issues or a QoS issue (mis configuration or under powered router)
I was on the phone with Meraki support for several hours on Friday. I've been having an on going issue with their malware protection option that has caused the router to lockup at 100% cpu usage (which you have no way of knowing because their GUI is way too simple), which I've disabled. The support rep was confident that it wasn't a capacity issue.
I wonder how their support rep was so confident in it not being a capacity issue? Seems rather unlikely that it was, but Meraki is not high end gear and it could easily be. That there was an issue like that around the same time is awfully suspicious.
-
@scottalanmiller said:
@coliver said:
Where would the STUN server be located? Is it generally something that is selected or does it go through the ISP?
Generally this is set on the phones and the PBX.
Does this still apply if the phones and PBX are on the same network? All phones and registered extensions are local to this site.
-
@coliver said:
@scottalanmiller said:
@coliver said:
Where would the STUN server be located? Is it generally something that is selected or does it go through the ISP?
Generally this is set on the phones and the PBX.
Does this still apply if the phones and PBX are on the same network? All phones and registered extensions are local to this site.
It applies to the PBX but not the phones, in that case.
Are you getting one way audio on extension to extension calls too? Or only when hitting the external trunk?
-
@scottalanmiller said:
@coliver said:
@scottalanmiller said:
@coliver said:
Where would the STUN server be located? Is it generally something that is selected or does it go through the ISP?
Generally this is set on the phones and the PBX.
Does this still apply if the phones and PBX are on the same network? All phones and registered extensions are local to this site.
It applies to the PBX but not the phones, in that case.
Are you getting one way audio on extension to extension calls too? Or only when hitting the external trunk?
Only when hitting the external trunk. The one way calling was happening on every external call either people calling in or extensions dialing out.
-
Yeah, that is almost certainly an issue with the Meraki and/or missing STUN settings. It is possible that under light load and no STUN, things "just worked" and only when things get heavy that STUN is needed. That's one option. Another is that the Meraki simply behaved badly under load.
-
Experiencing the same issues again this morning. I get some flapping from my SIP trunk provider but not on every call.
-
I hope @coliver doesn't mind me asking a few VOIP questions here considering his issues.
@scottalanmiller mentioned STUN servers. In the case of Coliver where he only has internal phones, is the STUN server provided by his SIP provider? or does Coliver need to set one up himself?
If Coliver wanted to have phones outside of the network, would Coliver have to set a STUN server up himself? Am I correct that is seems that a STUN server sorta acts like a reverse proxy for SIP?
-
@Dashrender said:
I hope @coliver doesn't mind me asking a few VOIP questions here considering his issues.
@scottalanmiller mentioned STUN servers. In the case of Coliver where he only has internal phones, is the STUN server provided by his SIP provider? or does Coliver need to set one up himself?
If Coliver wanted to have phones outside of the network, would Coliver have to set a STUN server up himself? Am I correct that is seems that a STUN server sorta acts like a reverse proxy for SIP?
Nope, no problems I was wondering the same thing myself.
-
Just got a report that users who use our Network Extender are also having the same issues that our "landline" users do.
The network extender, from my understanding, is basically a VoIP gateway. Which tells me I can rule out both the PBX and the SIP Trunk if a different version of VoIP is also having issues.
-
@Dashrender said:
I hope @coliver doesn't mind me asking a few VOIP questions here considering his issues.
@scottalanmiller mentioned STUN servers. In the case of Coliver where he only has internal phones, is the STUN server provided by his SIP provider? or does Coliver need to set one up himself?
There is no STUN server in this scenario. STUN is for devices to PBX, not SIP trunks.
If Coliver wanted to have phones outside of the network, would Coliver have to set a STUN server up himself? Am I correct that is seems that a STUN server sorta acts like a reverse proxy for SIP?
You are not required to setup a STUN server for outside phones, but it can smooth out problems when dealing with end users behind gear that may or may not play nice. For @Bundy I use a public STUN server simply because there are only 2-3 extensions that are not inside an OpenVPN tunnel in the first place.
