control ICMP rules created with LoadBalancer type service about cloud-provider-aws

[TBBle]

MSS isn't negotiated, it's simply announced by each end as to what their receive-size MSS, for segments going to that end. It defaults to 'MTU - 20 (IP Header minimum size) - 20 (TCP header minimum size)'.

However, MSS is only ideally set to be small-enough to fit in a single packet, and PMTUD has historically interefered with this as much as it's helped (See 2.3.).

So I would ignore the MSS here, it's not really relevant to the discussion, as a single TCP segment can be split across multiple IP datagrams without issue.

---

The necessity of "destination unreachable: needed fragmentation" here is that you immediately get back a response to know that the TCP layer will have to resend, and the IP layer now knows that this resend and and any future segments of sufficient size should be fragmented locally before transmission.

Without that message, a narrow-MTU link between your NLB will cause do-not-fragment packets to simply disappear when they are too large. TCP will resend, and those will disappear too. This is the stalling problem from the original bug report, and also a historically known issue (See 2.1.).

[TBBle]

Are you referring to the Path MTU Discovery (Type 3, Code 4) rule? That's pretty important for the good functioning of the network, and isn't really a security risk.

It was added deliberately in kubernetes/kubernetes#24254 because without it, transfers out from your Load Balancer can stall.

[infa-ddeore]

I am talking about this rule, is this not a security issue allowing for 0.0.0.0/0 ?

I will read through the links you provided.

[TBBle]

That's right, that's not a security issue, it's needed for your traffic to actually flow correctly and with good performance, as that message is sent to you by intermediate routers when you are sending too-large packets, that will otherwise be dropped.

There's a useful diagram of what happens if you remove that rule under "ICMP MESSAGES BEING BLOCKED". Removing that rule will turn the entire Internet into a PMTUD Black Hole from the perspective of your cluster.

It possibly should always be 0.0.0.0/0, even if you do set loadBalancerSourceRanges, because it's not your recipient who replies, but a router between you and the recipient. However, it's possible that the reply's source IP is given as the destination, not the sender, in order to get through well-meaning but overly-restrictive firewalls. But best to leave it as 0.0.0.0/0.

Note that with IPv6, PMTUD becomes mandatory.

[infa-ddeore]

I am trying to understand in what case the problem would occur.

If client is accessing a service over ELB and ELB is responding with higher MSS then intermediate routers will send ICMP response to ELB?

loadBalancerSourceRanges are applied to TCP as well as ICMP rule, so there needs to be separate annotation for ICMP since you recommend it to keep open regardless of other ports' restriction.

[TBBle]

You've identified the problem, yes. (With Network LoadBalancer, the response size will be per the EC2 instance, not the LB). Also, to be clear, overly-large MSS will just be discarded. However, MSS derives from MTU, so overly-large MSS implies overly-large MTU, and packets larger than a hop's MTU with Do Not Fragment set (generally the case, and effectively always set in IPv6), generate the ICMP response.

I'm not sure if there's some magic in the Security Group that recognises an ICMP type 3 packet as being sent on behalf of an allowed source address, i.e. parsing the type 3 header which includes the data about the packet being responded to. I could easily believe such magic exists, because this'd be pretty easy to get wrong if you're blindly applying IP restrictions.

Otherwise this is indeed a bug in the setup, that it will reject Fragmentation Required responses when loadBalancerSourceRanges is in use, as they will not come from an allowed IP address.

This probably needs to be tested by someone who is on an MTU-constrained connection, which many home Internet users are (due to PPPoE and related protocols) but equipment in a datacentre is generally not.

[infa-sathkuma]

I am also looking for the scenario where the PMTU to work with the ELB . AWS security group has a specific rule for ICMP type 3 DF set .
MSS is a negotiated value. We can see that exchange in wireshark capture. The upper bound of the advertised TCP MSS is most likely the MTU of the first hop route. Now what makes the MTU higher ? the biggest payload element is TCP MSS, which would have been sent over with SYN packet.
most of the security scans are flagging ICMP 0.0.0.0/0 not sure if it looks at type 3.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.