topic Re: Audio Issues with Asterisk via PA-2050 in General Topics

Audio Issues with Asterisk via PA-2050

Isosat — Thu, 05 Sep 2013 14:24:52 GMT

I am currently attempting to cut over our office's internal gateway from a BSD firewall to our PA-2050 (running PAN-OS 4.1.9). When attempting the cutover, I can get all services to work properly with the exception of our two VoIP servers (running Trixbox, which is Asterisk-based). I can get the servers to make a call, but once connected there is no audio on either end. Both servers are using a 1:1 NAT through the firewall and I am only allowing SIP and RTP ports to be allowed from the internet. The only changes that were made were to the PA-2050's Ethernet interfaces to replace the existing gateway for the VoIP servers.

I had previously worked with Palo Alto Support to work through this. They had me set up an Application Override across all ports for both servers, which worked while testing. Unfortunately, every attempt at going live results in a lack of audio when calls are connected. Viewing the traffic log, it appears as if the RTP traffic is flowing normally. I really need to get this cutover completed, but am unable to do so until I can get the audio working. Has anyone else run into a similar issue? If so, how were you able to get audio working properly.

Re: Audio Issues with Asterisk via PA-2050

Phoenix — Thu, 05 Sep 2013 14:46:29 GMT

Hello,

This is more of a troubleshooting scenario and may have to check different parameters as we narrow down the issue. Some points I think are:

> Is the app asterisk-iax allowed, based on the type of application may be some traffic is not being passed.

> Some times voip related apps may have variations on behavior with change in App content version as decoders would be getting updated, so if it was working at certain content version and not in another that may be the reason.

> If the voip servers are internal, did you check by disabling nat rules to see if natting has issue. Also it is good test mechanism to test with 1:1 nat but looks like this was tested already.

> Did you try and see the global counters at the time of passing traffic, this would give us an idea on the drops done by pan.

We will have to set up filters for specific source and destination and run the global counters command while traffic passing.

"show counter global filter packet-filter yes delta yes "

Run this back to back multiple times which would give some idea.

But I would suggest if a case is not opened yet pls open to isolate the issue.

Thanks

Re: Audio Issues with Asterisk via PA-2050

ericgearhart — Thu, 05 Sep 2013 14:50:37 GMT

FYI I just recently had to build App overrides for RTP and RTCP to get our host voice solution working in a remote office that is using a PA200. Leave the SIP App-ID alone! I noticed that doing an app override for SIP broke audio too. I have app overrides for just RTP and RTCP and it's working great.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Thu, 05 Sep 2013 15:41:07 GMT

I just set up our test PBX using this setup and am able to make calls and hear audio. I'm not completely convinced that this will work though, because I had audio working on it before using Application Override across all ports. Right now, I'm only using it for RTP and RTCP over UDP ports 9999 - 20001.

Re: Audio Issues with Asterisk via PA-2050

apasupulati — Thu, 05 Sep 2013 16:34:50 GMT

Hello,

If you configure filters for the VOIP traffic and check the global counters, do you see any drop counters for Packets matching rtp/rtcp predicts of sip? To check for counters use this command with filters turned on: show counter global filter packet-filter yes delta yes

I ran across a known issue on PANOS 4.1.5, where asterisk based VOIP packets matching rtp/rtcp predicts of sip are dropped due to waiting for predicts merging, causing audi issues. This was addressed in PANOS 5.0 release. Do you have any plans to upgrade?

Thanks,

Aditi

Re: Audio Issues with Asterisk via PA-2050

ericgearhart — Thu, 05 Sep 2013 17:02:05 GMT

Right I had to do the same thing essentially... I built my override for UDP from ports 1024-65535. Luckily there is a specific pair of Class Cs that provide our hosted VoIP, which I used in the 'Destination' column in my PA-200 so that I didn't just flat out override ALL UDP 1024-65535 traffic. I dunno if that matches up with your case here.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Thu, 05 Sep 2013 17:59:22 GMT

egearhart,

We have our own VoIP servers in the office, so I attached the APP Override to those source addresses. The servers limit RTP traffic to ports 10000-20000, which is why I only set those ports on the override.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Fri, 06 Sep 2013 11:46:48 GMT

Well, I just attempted another cutover this morning and the audio was still not there. I don't understand how this will work on a test PBX (which is cloned from one of the two existing PBX in the office) but not work when I attempt to go live. All I've done was disconnect the BSD firewall, replace the interface IPs in the Palo Alto firewall, and then clear the ARP tables in our edge router and in the PA-2050.

I had opened a ticket with Palo Alto Support on Wednesday and after working with them for several hours, I still do not have an answer as to what could be causing this, much less what a permanent solution would look like. I've been attempting this since last Saturday and really need to have this resolved ASAP.

apasupulati We had let the licenses expire on our PA-2050 when we purchased a new PA-3020, so upgrading the OS isn't an option at the moment.

Re: Audio Issues with Asterisk via PA-2050

ericgearhart — Fri, 06 Sep 2013 14:54:54 GMT

I'd say there's got to be some crucial difference between what you're testing in your lab versus what you've got in production... are you using a lab PA2050 but then when you try to go live you're using a PA3020?

Can you pull pcaps of a working lab session and a non-working production session, and maybe "diff" the two pcaps against each other?

Re: Audio Issues with Asterisk via PA-2050

Isosat — Fri, 06 Sep 2013 15:06:01 GMT

The PA-2050 is being used for testing while the two production VoIP servers are using the BSD firewall as their gateway. I pull the Ethernet cables from that firewall while the PA-2050 is committing changes (changing the addresses of interfaces to the addresses that were being used by the BSD firewall). I know that the NAT rules are working, because the calls are going out to our SIP provider and connecting to my cell phone. Each time I back out of the cutover, replacing the interfaces with their previous addresses, I then lose audio on my test VoIP server. No changes have been made on any PBX.

I do have some PCAPs for failed calls, but have not yet been able to do so for successful calls.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Fri, 06 Sep 2013 17:12:57 GMT

I just discovered something rather interesting:

I had moved my testing PBX from our old PA-2050 to our new PA-3020 without using any Application Override. I was able to make calls and hear audio going both ways. I then moved it back to the old PA-2050 (using APP Override) and was also able to hear audio both ways. I was hoping to be able to keep the Default Gateway on my internal network the same, but it appears the only way to make the audio work is to move the gateway of my VoIP servers.

Re: Audio Issues with Asterisk via PA-2050

ericgearhart — Fri, 06 Sep 2013 17:24:07 GMT

Is the VoIP servers' default gateway the BSD firewall? Why can't the PA3020 take the place of the BSD firewalls, and become the VoIP servers' gateway?

Sorry, if you were here we'd be whiteboarding this out and it'd be much easier to see what's going on

Re: Audio Issues with Asterisk via PA-2050

Isosat — Fri, 06 Sep 2013 17:50:16 GMT

The VoIP servers are currently using the BSD firewall as their default gateway. We are trying to move everything off of the BSD onto the PA-2050 for easier management. We are currently using the PA-3020 between our Internet fiber and our Edge Router in transparent vWire mode. We want the PA-2050 to handle all of our NATs as the owner of the company does not want to have a single point of failure. The PA-2050 has been handling the NAT for several other servers (two web servers and two DNS servers) in our DMZ without any issues. Also, during the cutover, every other service (internet, emails, etc.) works without any issues. It's only the audio over VoIP that doesn't work.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Mon, 09 Sep 2013 16:04:03 GMT

Just attempted another cutover this morning. I enabled the NAT and Security rules on my PA-2050. While that was committing, I commented out the same rules in our BSD firewall. I then changed the Default Gateway for one of my VoIP servers from the BSD firewall to the PA-2050. I then cleared the ARP tables in our Edge Router. When making a call, I could hear audio on the receiving end, but could hear nothing from the office phone behind the VoIP server.

I feel like I'm getting closer, but am still missing something.

Re: Audio Issues with Asterisk via PA-2050

Chatri — Mon, 09 Sep 2013 20:42:59 GMT

If you Look at the sessions that are getting created for SIP and RTP you will be able to see if the NAT is being properly applied or not.

Now, the SIP INVITE packet will have the internal address in the SDP payload, and PA will have to NAT the SDP payload using the SIP NAT 1:1 NAT rule ( If no App override is in place)

One way Audio, would usually mean, either the PA is not doing the correct NAT on the Payload which would mean the RTP session that are opened up will have the Internal Address, in which case the remote end will not be able to route the response correctly. Or If there is application Override in place, then the PA will not do a NAT at all on the INVITE payload, which would mean that the RTP session ( Audio) will get opened up with the Private address of the call initiator.

The best way to verify if the PA is doing the NAT correctly, is to capture This traffic at receive and transmit stage on the PA, then look at the INVITE packet ( layer 7 payload ) to see if the Proposed IP in the Transmit capture is NATted by comparing it to the receive capture INVITE payload.

Note: If the clients are NOT NAT AWARE ( if the clients are not configured to know their NATted IP), then application override will not solve the problem.

You can upload the captures here along with SIP and RTP session details and I can take a look.

Hope that helps.

Aditya

Re: Audio Issues with Asterisk via PA-2050

Isosat — Thu, 12 Sep 2013 12:35:06 GMT

I think it may be a NAT issue after my latest cutover attempt. I changed the default gateway of one of the VoIP servers to my PA-2050's interface again and continue to get one-way audio. This is also after commenting out the NAT rule in the BSD firewall as well as clearing the ARP table in the Edge Router. I decided to reboot the VoIP server and then was completely unable to reach the internet on the server, so all calls immediately got an "All circuits are busy now" message. I have the NAT working for several other servers, and the NAT rules match those exactly. This is incredibly frustrating and Palo Alto Support has largely been unhelpful.

Re: Audio Issues with Asterisk via PA-2050

Isosat — Thu, 12 Sep 2013 18:44:21 GMT

Found out some more information after spending most of my day with Support. Our Edge Router keeps showing an ARP entry for the address I'm attempting to NAT my VoIP server to. I've already commented out the NAT rule in that firewall, as well as deleted two static routes that had been entered into the firewall. I'm still unable to clear that ARP entry from my Edge Router. It is a Cisco router, and I can't seem to find any answers that permanently remove the ARP entry.

Re: Audio Issues with Asterisk via PA-2050

Chatri — Thu, 12 Sep 2013 20:10:20 GMT

Hello,

If you have a case with PA support open, could you mention the case number?

Re: Audio Issues with Asterisk via PA-2050

hshah — Thu, 12 Sep 2013 21:05:35 GMT

Please verify if SIP app override is done on Port UDP/5060 if SIP is using UDP. Otherwise it should be on TCP/5060

Re: Audio Issues with Asterisk via PA-2050

Isosat — Mon, 16 Sep 2013 15:30:49 GMT

Ok, it looks like the issue has finally been resolved. What finally solved the issue was setting up the addition of two Application Overrides for SIP: one for UDP and one for TCP. To be safe, we set the port numbers pretty wide open (1034-65535).