Strange Behavior with SIP traffic related to ALG

[rburgoyne_gci]

I'm running into an issue where specific NAT and Security policy names or numbers change then the SIP traffic stops working. I found that if I clear the sessions post change then everything starts working again. I believe this is related to ALG, like the SIP traffic is taking the parent session even though it is no longer valid. I see these sessions drop into a discard state.

I'm wondering if anyone else has experience the same behavior, we are on 6.0.8.

[pulukas]

It sounds like what you are running into is session rematch on commit.  I am pretty sure this is the default setting and it sounds like you would prefer to turn this off.

Device tab > Setup > Session sub tab

Uncheck "Rematch Sessions"

What this setting does is force all sessions to be evaluated again when the security policy is changed.  For complex sessions in progress (like sip) you can see the behavior you describe.

The advantage of having this session is slightly higher security.  Whenever policies are changes all sessions are immediately checked again so any in progress session that would no longer be permitted is stopped right away instead of allowing it to play out to the natural conclusion of the session.

[rburgoyne_gci]

The problem seems to be the opposite of what you're describing. The policies are changing but the traffic is not getting re-matched for SIP and the session gets stuck in a discard state because its trying to hit incorrect policies via a parent session. I have to manually clear the sessions after a change to the NAT or Security policies which is kind of annoying.

[pulukas]

Can you confirm that "rematch sessions" is checked and remove it if it is?

I do think this is the issue.  With rematch we clear all your current sessions and force the new policy set to apply immediately.  With sip we are then losing this connection to the original session that the ALG maintains.

Without rematch all your current SIP sessions are untouched and will end naturally with session completion.

[rburgoyne_gci]

Rematch is currently checked, and we are not seeing the expected rematch behavior. Also the SIP sessions dont seem to timeout on their own, I let one sit for 24+ hours and it still showed the session in a discard state. Even rebooting the phone didn't resolve the issues. The only fix is to manually clear these SIP sessions and then everything works again. I've had to position the NAT and ACL at the top of the list so that their line numbers won't ever change as a work around.

[pulukas]

Sorry not to be clear.  This is what I am trying to say.

Session rematch is the default and I expected it to be turned on.  (which you verify)

I suspect this is your problem.

Please turn off Session rematch

[rburgoyne_gci]

Right, but the session rematch is the behavior that I want is what I am saying. I want rematch after policy changes and it works for everything except SIP.

[fjwcash]

The naming is confusing, but I think I'm starting to understand this.

"Session rematch enabled" means that:

  - client connects to server using current policies, IPs, routing, etc, and a session is created in the firewall

  - you change the policies in such a way that the IP, routing, NAT changes

  - you commit the change, then the firewall goes through the existing sessions and applies the new policies to them.  Because the NAT or IPs or whatnot changed, the existing sessions no longer work, and are put into DISCARD state

  - BUT, and here's the kicker, your client doesn't know the rules changed, and continues to use the old IPs, so the packets it's sending match the existing sessions, and the packets are DISCARDed

  - AND, the server doesn't know the IPs changed, and continues to use the old IPs, so the packets it's sending also match the existing sessions, and the packets are DISCARDed

Since this is UDP, the client and server never really knows when to stop sending to the old IPs until some timeout or reset happens to force it to start over.

I'm a little unsure on exactly how "Session rematch disabled" works, but this is my best guess:

  - client connects to server using current policies, IPs, routing, etc, and a session is created in the firewall
  - you change the policies in such a way that the IP, routing, NAT changes
  - you commit the change.  The existing sessions at the time of commit are essentially ignored and allowed to timeout normally (basically going through the old rules?), and any new sessions created go through the new rules.

We're seeing this with our current SIP setup, where we started with a destination NAT on the server side, didn't know the SIP application set a custom UDP timout of 1 hour, and have "Session rematch enabled" as the naming makes it seem like it does the opposite of what it actually does.  We then decided to remove the DNAT, committed the change, and suddenly a bunch of phones that were working suddenly stopped working (sending data to the old server IP and matching a DISCARD session), phones that just booted were working (new session created using the new server IP), and manually clearing out the sessions caused things to start working (usually) for everyone.

Then we turned off Session Rematch, made another change to the firewall that actually broke routing (although we didn't notice right away) and phones that were working would stop working at different intervals (their existing sessions timed out normally), and phones that rebooted (to pick up the new server IP) would stop working (new rules were denying the traffic) as no new sessions were being created.   We reverted back to a known-good snapshot from the NAT setup, and everything started working again.

I think if we had working non-NAT security policies in place, with Session Rematch turned off, and a proper/low UDP timeout in the "sip" App-ID, then everything would have worked as expected (clients would pick up the new server IP, new sessions would be created, and traffic would pass; and the old sessions would just expire out of the session table).

Unfortunately, there's not a lot of information out there that puts all this information in one place.  🙂  You have to scrounge around and pick bits from here and there and put it all together offline.  😄

Cheers,

Freddie

[Roby_Sreejith]

Can you try app override policy for sip, this might solve your probs

[fjwcash]

That is actually the next thing we are going to try.  🙂  Just discovered the Application Override documentation.

[fjwcash]

We ended up disabling the ALG in the SIP application, removing all Application Override Policies, removing the sip application from the Security Policy, and just doing straight UDP/TCP port filtering in the Security Policy.  Our SIP/RTP traffic is passing through correctly.  And our QoS Policy is now matching the traffic correctly, where it was only matching SIP traffic before.