Long VoLTE call drop - OPTIONS outside ipsec tunnel about docker_open5gs

[RafalArciszewski]

Hi,
I did a long test and I achieved 10h !!!
Here is a screenshot from my two lab smartphones.

However, I used my previous git snapshot with some configuration tweaks you gave.
When I am using latest commits I am unable to have VoLTE call. UE sends SIP 508 Precondition Failure.
Something you changed is causing this. But I cannot figure out what is it.

20201027_precondition_change.zip

[RafalArciszewski]

Ok I understand. You have removed unnecessary Security-Server fields so now the messages are <1300. Is this MTU parameter configurable on PCSCF? I suspect a bug in the mobile, but this could be workaround if I lower MTU to i.e. 1000B.

The call last 15 minut just to send you the pcap. I will make longer calls today. I will give an info about the results.

[herlesupreeth]

I have a problem with long UE-UE VoLTE calls. After 1h the MT UE lost its connection to PCSCF and eventually call is dropped sometimes by MO UE sometimes by IMS.

I too observed the same. Since it was for 1hr long calls i didn't bother to fix it

Regarding the UDP packets inside the IPSec, its completely upto UE to send the packets over desired transport protocol (TCP/UDP)

Thanks a lot for the pcap, i may have found the cause, in the UPDATE SIP Request after 1 hr i see several Security-Server being added which is clearly a bug. Its needs fixing in the kamailio source code. I will work on it and let you know.

[herlesupreeth]

I have issued a fix for UPDATE SIP Request mentioned above, can you please remove the existing docker image of kamailio and recreate the docker image and recreate the image for PCSCF as well. Then give it a try and let me know if this issue is solved or not.

[RafalArciszewski]

Thanks. The call survived after 1h. I will test is with longer period.
However now few messages (PRACK,ACK,UPDATE,BYE) were sent during call to UE MT (100.65.0.8) as UDP inside ipsec. The UE response outside ipsec but it works somehow. The Security-Verify field has an information about port from UE MO. Is that ok?

20201008_volte_1_15h_call.zip

[herlesupreeth]

The UE response outside ipsec but it works somehow.

I too find it a bit strange, the 200 OK for PRACK and UPDATE are sent by UE outside of IPSec tunnel.

Thanks a lot for testing it. Please let me know how long the call can last.

The Security-Verify field has an information about port from UE MO. Is that ok?

Security-Verify only has information regarding the IPSec ports agreed between UE and PCSCF.

[herlesupreeth]

The UE response outside ipsec but it works somehow.

I have pushed another fix for this issue. Can you please remove the existing docker image of kamailio and recreate the docker image and recreate the image for PCSCF as well. Then give it a try and let me know if all the replies fro UE are going through the IPSec?

[RafalArciszewski]

Unfortunately nothing changed. PRACK and UPDATE sent towards UE as UDP inside ipsec. resposnse outside ipsec.

[herlesupreeth]

can you send me the pcap?

[herlesupreeth]

Unfortunately nothing changed. PRACK and UPDATE sent towards UE as UDP inside ipsec. resposnse outside ipsec.

It doesn't matter UDP or TCP is being sent by UE, UE uses TCP if MTU of the message is greater than some value most of the time and also for reliability. Important thing is that all the message go through IPSec tunnel with ESP header

[RafalArciszewski]

I attach pcap from today. Comparing to the first pcap there is the difference that PRACK and UPDATE are sent by PCSCF as UDPoverrIPSEC. In the pcap from to days ago PCSCF was sending those messages as TCPoverISPSEC.

20201009_volte_15min_call.zip

[herlesupreeth]

Similar to UE, if the mtu of final packet exceeds 1300 the PCSCF will change the protocol from UDP to TCP so its normal. But how long did the call last for?

[herlesupreeth]

On the PCSCF the MTU is configurable, you can find it in the top of kamailio_pcscf.cfg file

[RafalArciszewski]

1h long call and drop :(
OPTIONS send by PCSCF as TCP outside IPSEC.

20201009_volte_1h_drop.zip

[herlesupreeth]

Thanks a lot. I appreciate your effort :)

I now reworked on the commit which had IPSec related fixes. Can you please remove the existing docker image of kamailio and recreate the docker image and recreate the image for PCSCF as well. Please let me know if the call lasts for more than an hour or not.

[RafalArciszewski]

I will test it. In the meantime I've check that this issue occur also when no VoLTE call is established.

[herlesupreeth]

That is quite strange, it shouldn't happen for OPTIONS ping

[RafalArciszewski]

I've changed MTU to 500B and run your latest image. Unfortunately problem still exists.

20201009_1345_volte_1h_drop.zip

Maybe I should check without OPTIONS enabled?

[herlesupreeth]

A word of caution when lowering the MTU check to 500, it can have unpredictable behaviour. 1300 is the safest bet as per the spec as well

[RafalArciszewski]

Hi,
today I tested the issue with disabled OPTIONS to exclude its influence.

##!define WITH_NATPING in pcscf.cfg

Please see at UE_MO flows (ip.addr==100.65.0.16). It has two tcp sessions:

1. (tcp.stream eq 167) port 6100 for UE originating procedures
2. (tcp.stream eq 169) port 5100 for PCSCF originating procedures

In the second tcp session there is no message flowing after the call establishment. There is no need for that in that scenario. In the first tcp session UE_MO sends UPDATE message every 15 minute.

It seems that when second tcp session is released due to timeout, also the first tcp session is considered by PCSCF as closed. When next UPDATE message arrives (packet 3992) there is no response from PCSCF. UE_MO repeats several times, try to reeastablish tcp session to 6100 sending SYN but due to lack of PCSCF response UE_MO eventually closes everything and re-register to IMS again starting on port 5060.

Please see the pcap in the attachment. On PCSCF console when tcp session is terminated there is a log:

pcscf | 85(135) INFO: cdp [authstatemachine.c:270]: auth_client_statefull_sm_process(): after callback of event 1
pcscf | 85(135) INFO: cdp [authstatemachine.c:270]: auth_client_statefull_sm_process(): after callback of event 1
pcscf | 95(145) INFO: cdp [authstatemachine.c:270]: auth_client_statefull_sm_process(): after callback of event 17
pcscf | 95(145) INFO: cdp [authstatemachine.c:425]: auth_client_statefull_sm_process(): state machine: AUTH_EV_RECV_STA about to clean up
pcscf | 95(145) INFO: cdp [authstatemachine.c:270]: auth_client_statefull_sm_process(): after callback of event 17
pcscf | 95(145) INFO: cdp [authstatemachine.c:425]: auth_client_statefull_sm_process(): state machine: AUTH_EV_RECV_STA about to clean up

This console log does not have timestamps, is there any other logs on PCSCF I can check?
Perhaps there is a way to configure TCP keepalive?

Regards,
Rafał
20201012_volte_1h_no_options.zip

[RafalArciszewski]

I found a way to change TCP keepalive to 20minutes but it didn't help. After 1h PCSCF closes tcp session. There must be some kind of "session timeout" somewhere.

[RafalArciszewski]

There is another interesting thing. After closing TCP session on Rx interface PCSCF send to PCRF SessionTerminationRequest but PCRF response with DIAMETER_MISSING_AVP.

[herlesupreeth]

@RafalArciszewski

For the MISSING AVP, can you send me the pcap?

For the long VoLTE call drop, I had a deep inspection into IPSec issue and it will take some time to fix. I will let you know once the fix is out.

[RafalArciszewski]

@herlesupreeth

HI,
I will wait for IPSEC fix.
Regarding Missing AVP, this is also happening in the pcap from the first comment. 1h_volte_call.zip (attached below)

I have a few notices (filter (ip.addr==100.65.0.5 or diameter) && !(diameter.cmd.code == 280)):

1. First registration attemp occur on ports 40586<->5060. No ipsec.
2. PCSCF send AAR (packet 183)

• Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;3
• Ports in flow description are a little incorrect
  AVP: Flow-Description(507) l=67 f=VM- vnd=TGPP val=permit out ip from 172.30.0.114 5060 to 100.65.0.5 5060
  AVP: Flow-Description(507) l=66 f=VM- vnd=TGPP val=permit in ip from 100.65.0.5 5060 to 172.30.0.114 5060
  AVP: Flow-Description(507) l=67 f=VM- vnd=TGPP val=permit out ip from 172.30.0.114 5061 to 100.65.0.5 5061
  AVP: Flow-Description(507) l=66 f=VM- vnd=TGPP val=permit in ip from 100.65.0.5 5061 to 172.30.0.114 5061
  AVP: Flow-Usage(512) l=16 f=VM- vnd=TGPP val=AF_SIGNALLING (2)
• Subscription-Id-Data: sip:[email protected]

2. PCRF send AAA (packet 185) - Result ok
3. tcp session 40586<->5060 is closed
4. PCSCF in my opinion should send session termination request now but it didn't happen.
5. TCP session 43163<->6101 inside ipsec is established. Second register.
6. (packet 223) PCSCF send AAR

• Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;4
• Ports in flow description are a little incorrect
  AVP: Flow-Description(507) l=69 f=VM- vnd=TGPP val=permit out ip from 172.30.0.114 44908 to 100.65.0.5 44908
  AVP: Flow-Description(507) l=68 f=VM- vnd=TGPP val=permit in ip from 100.65.0.5 44908 to 172.30.0.114 44908
  AVP: Flow-Description(507) l=69 f=VM- vnd=TGPP val=permit out ip from 172.30.0.114 44909 to 100.65.0.5 44909
  AVP: Flow-Description(507) l=68 f=VM- vnd=TGPP val=permit in ip from 100.65.0.5 44909 to 172.30.0.114 44909
  AVP: Flow-Usage(512) l=16 f=VM- vnd=TGPP val=AF_SIGNALLING (2)
• Subscription-Id-Data: sip:[email protected]
  Why PCSCF send AAR in case of IPSEC scenario? The UE will comunicate with PCSCF over ESP so no port are available there. Maybe ports here should be like 'any' (I don't know if it is possible in standards)

7. (packet 224) PCRF send AAA - Result Ok
8. TCP 44908<->5101 is established but no AAR is send
9. Call is starting INVITE/SEssion PRogress etc.
10. (packet 315) PCSCF send AAR

• Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;5
• Flows are ok
  AVP: Flow-Description(507) l=69 f=VM- vnd=TGPP val=permit out 17 from 172.30.0.109 49000 to 100.65.0.5 50026
  AVP: Flow-Description(507) l=68 f=VM- vnd=TGPP val=permit in 17 from 100.65.0.5 50026 to 172.30.0.109 49000
  AVP: Flow-Description(507) l=69 f=VM- vnd=TGPP val=permit out 17 from 172.30.0.109 49001 to 100.65.0.5 50027
  AVP: Flow-Description(507) l=68 f=VM- vnd=TGPP val=permit in 17 from 100.65.0.5 50027 to 172.30.0.109 49001
• Subscription-Id-Data: sip:[email protected]:44908 ??

11. (packet 317) PCRF send AAA - Result OK
12. Call ongoing
13. After 1h TCP 43163<->6101 is closed
14. (packet 2349) PCSCF send STR for session Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;3
15. (packet 2351) PCRF send STA with Result DIAMTER_MISSING_AVP
16. (packet 2745) PCSCF send STR for session Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;5
17. (packet 2747) PCRF send STA - Result Ok.
18. What about pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;4 session? It never cleared so probably stay still in PCRF.

There is also AAR for second UE (Session-Id: pcscf.ims.mnc024.mcc260.3gppnetwork.org;1200527915;6) with Subscription-Id-Data: tel:3961 . If I am not mistaken PCRF is matching Rx to Gx based on Framed-IP-Address, right? Otherwise it wouldn't work.

20201007_volte_1h_call.pcapng.zip

Regards,
Rafał

[herlesupreeth]

If I am not mistaken PCRF is matching Rx to Gx based on Framed-IP-Address, right?

True, there is one to many relation between Gx and Rx sessions in PCRF

For the long VoLTE call disconnect after 1 hr, can you the following diff and give it a try?

--- a/pcscf/pcscf.xml
+++ b/pcscf/pcscf.xml
@@ -11,8 +11,8 @@
        QueueLength="8"
        TransactionTimeout="5"
        SessionsHashSize="128"
-       DefaultAuthSessionTimeout="3600"
-       MaxAuthSessionTimeout="3600"
+       DefaultAuthSessionTimeout="36000"
+       MaxAuthSessionTimeout="36000"
 >
        <Peer FQDN="pcrf.EPC_DOMAIN" Realm="EPC_DOMAIN" port="3868"/> 

For the IPSec, i am still working on the best possible solution and its taking some time

[RafalArciszewski]

Hi, unfortunately changing parameters in pcscf.xml did not help.

[herlesupreeth]

Thanks for testing it out. Are you using a commercial eNB or srslte?

[RafalArciszewski]

commercial eNB

[herlesupreeth]

Thanks a lot for pcaps. I found the culprit causing this issue. Please change the following and give it a try. This will definitely fix this issue

--- a/pcscf/kamailio_pcscf.cfg
+++ b/pcscf/kamailio_pcscf.cfg
@@ -99,7 +99,7 @@ enable_tls=yes
 #!ifdef WITH_TCP
 # life time of TCP connection when there is no traffic
 # - a bit higher than registration expires to cope with UA behind NAT
-tcp_connection_lifetime=3615
+tcp_connection_lifetime=36000
 # If a message received over a tcp connection has "alias" in its via a new tcp
 # alias port will be created for the connection the message came from (the
 # alias port will be set to the via one).

[herlesupreeth]

Sorry, i left out one more addition. Here is the complete diff

--- a/pcscf/kamailio_pcscf.cfg
+++ b/pcscf/kamailio_pcscf.cfg
@@ -99,7 +99,7 @@ enable_tls=yes
 #!ifdef WITH_TCP
 # life time of TCP connection when there is no traffic
 # - a bit higher than registration expires to cope with UA behind NAT
-tcp_connection_lifetime=3615
+tcp_connection_lifetime=36000
 # If a message received over a tcp connection has "alias" in its via a new tcp
 # alias port will be created for the connection the message came from (the
 # alias port will be set to the via one).
@@ -388,6 +388,7 @@ modparam("ims_registrar_pcscf", "is_registered_fallback2ip", 1)
 modparam("ims_registrar_pcscf", "ignore_reg_state", 1)
 modparam("ims_registrar_pcscf", "ignore_contact_rxport_check", 1)
 modparam("ims_registrar_pcscf", "pending_reg_expires", 30)
+modparam("ims_registrar_pcscf", "subscription_expires", 36000)
 
 #!ifdef WITH_REGINFO
 modparam("ims_registrar_pcscf", "subscribe_to_reginfo", 1)

[RafalArciszewski]

How to do it? Should I get latest commit from 'herlesupreeth/docker_open5gs' ? My kamailio_pcscf.cfg looks like this:
kamailio_pcscf.zip

Can you send me raw file, perhaps?

[herlesupreeth]

kamailio_pcscf.zip

please find the kamailio_pcscf.cfg attached

[RafalArciszewski]

Something wrong with IPSEC_MAX_CONN parameter.

pcscf        |  0(32) CRITICAL: <core> [core/cfg.y:3536]: yyerror_at(): parse error in config file /etc/kamailio_pcscf/kamailio_pcscf.cfg, line 408, column 54-67: Invalid arguments
pcscf        |  0(32) CRITICAL: <core> [core/cfg.y:3539]: yyerror_at(): parse error in config file /etc/kamailio_pcscf/kamailio_pcscf.cfg, line 408, column 68:
pcscf        | ERROR: bad config file (3 errors)
pcscf        |  0(32) WARNING: <core> [core/ppcfg.c:223]: pp_ifdef_level_check(): different number of preprocessor directives: 1 more #!if[n]def as #!endif
pcscf        |  0(32) WARNING: <core> [core/mem/q_malloc.c:487]: qm_free(): WARNING: free(0) called from cdp_avp: cdp_avp_mod.c: cdp_avp_destroy(226)
pcscf        |  0(32) INFO: cdp [cdp_mod.c:255]: cdp_exit(): CDiameterPeer child stopping ...
pcscf        |  0(32) INFO: cdp [cdp_mod.c:257]: cdp_exit(): ... CDiameterPeer child stopped
pcscf        |  0(32) ERROR: ims_ipsec_pcscf [ims_ipsec_pcscf_mod.c:309]: mod_destroy(): Error destroying spi generator
pcscf        |  0(32) ERROR: ims_ipsec_pcscf [ims_ipsec_pcscf_mod.c:313]: mod_destroy(): Error destroying port generator
pcscf        | loading modules under config path: /usr/lib64/kamailio/modules_k/:/usr/lib64/kamailio/modules/:/usr/lib/kamailio/modules_k/:/usr/lib/kamailio/modules/:/usr/lib/x86_64-linux-gnu/kamailio/modules/:/usr/local/lib64/kamailio/modules
pcscf exited with code 255

[herlesupreeth]

i guess you are using the latest commit from this branch. That variable was introduced in this commit 1c05520

[RafalArciszewski]

Its much better now. The call last 2h! :)

After 2h PCSCF send to UEs BYE message with "X-Reason: QoS failed". (again outside ipsec tunnel)
This was probably ignored by UEs (no answer), but since PCSCF closed RTP channels in RTPEngine, eventually UEs send BYEs with Reason "RTP timeout"

Analyzing the log from pcscf is hard as there are no timestamps but I think there is a problem with Rx. I will increase DefaultAuthSessionTimeout and MaxAuthSessionTimeout in pcscf.xml file and try again.

pcscf | 97(137) CRITICAL: cdp [session.c:480]: cdp_sessions_timer(): lifetime+grace TIMEOUT
pcscf | 97(137) INFO: cdp [authstatemachine.c:270]: auth_client_statefull_sm_process(): after callback of event 20
pcscf | 97(137) ERROR: cdp [routing.c:274]: get_routing_peer(): get_routing_peer(): No connected DefaultRoute peer found for app_id 16777236 and vendor id 0.
pcscf | 97(137) ERROR: cdp [authstatemachine.c:870]: Send_STR(): unable to get routing peer in Send_STR

I attach the pcap and log.

20201021_volte_2h_call.zip

[RafalArciszewski]

Sorry just noticed that in pcscf.xml there already was 36000.

    <Acceptor port="3871" bind="PCSCF_IP"/>

    <Auth id="16777236" vendor="10415"/> <!-- 3GPP Rx -->
    <Auth id="16777236" vendor="0"/> <!-- 3GPP Rx -->

    <DefaultRoute FQDN="pcrf.EPC_DOMAIN" metric="10"/>

But on Rx authorization period is 7200.

[herlesupreeth]

I have fixed it just now, please refer to this commit 121fef6 . Now the calls will last for 10hrs if everything works fine

[RafalArciszewski]

Ok, I achieved 2h 10minutes. I will plan longer testing and give you the outcome.

As I understand the maximum call time should be 10hours? Whats the main problem here, the timers only?

Thanks

[herlesupreeth]

As I understand the maximum call time should be 10hours? Whats the main problem here, the timers only?

Yes, it was purely about subscription expiring and Rx session timeout.

[RafalArciszewski]

is there any way to renew the subscription and Rx session?

[herlesupreeth]

i believe, phone should resend SIP register after a certain contact expire time

[herlesupreeth]

Let me know how long the call lasts. I would like to close this issue

[herlesupreeth]

I pushed one more change now e8b8771. Its unrelated to long VoLTE call issue, but it would be great if you could take this change and do a call and send me a pcap (this could be a fix for IN-Dialog IPSec issue I see in your pcap)

[RafalArciszewski]

I have downloaded latest git commit and rebuild all docker images.
I noticed that in pcscf.xml:
DefaultAuthSessionTimeout="3600"
MaxAuthSessionTimeout="3600"
Is that ok? On Rx however I can see 10h:

The bigger problem is now that VoLTE calls are not working. It seems that NOTIFY msg is send outiside IPSEC (new TCP session is established). There is no answer from UE.

20201022_volte_notify_problem.zip

[herlesupreeth]

Damn, thanks a lot for testing it out. I guess the latest commit is screwing things up even though the documentation in Kamailio says otherwise. I will revert it back.

[herlesupreeth]

I have downloaded latest git commit and rebuild all docker images.
I noticed that in pcscf.xml:
DefaultAuthSessionTimeout="3600"
MaxAuthSessionTimeout="3600"
Is that ok?

Thats fine. Its set in ims_qos settings in kamailio_pcscf.cfg and does not depend on settings in pcscf.xml

[herlesupreeth]

Nice, thank you for testing it out. I have reverted the change which was breaking the call (#7 (comment)). Can you please re-compile PCSCF by removing the cache and give it a try (you may want to force pull git repo changes - this e8b8771 commit should NOT be there )? For me the calling is working just fine

[herlesupreeth]

Closing due to inactivity. Please re-open the issue with pcaps and logs if you still face issue.