Follow

Dear and community, I have a conundrum. We have IPsec (transport mode) set-up between all our servers; we are also starting to use Rancher. Rancher sets up its own IPsec-protected network (tunnel mode).

So we end up with IPsec in IPsec, which sucks. Specifically, can't seem to get MTU right, there is *always* a window of packet sizes that simply will *not* get through.

Anybody knows of a way to tell the back-end IPsec "so, if it's a Rancher's IPsec packet, don't touch it"?

Fun fact, the window of packet sizes that will never go through is exactly 93 bytes. So, packets smaller than X go through, packets larger than X+92 go through, but packets with sizes of X to X-92 (inclusive) will not.

X depends on the MTU set on Rancher's virtual network interface. Specifically, it's exactly MTU - 121.

Show thread

@wxcafe you jest, but in my frustration and desperation, at 5AM, after 18 staight hours of debugging this bugger, I did set the MTU on the Rancher interface to 400.

Packets with sizes between 279 and 372 bytes (inclusive) never got through.

X = MTU - 121
279 = 400 - 121

:blobugh: 🤦

@rysiek solved? maybe you can use SPD entries to exclude traffic trom IPsec?

@saper solved-ish: github.com/rancher/rancher-cat
github.com/rancher/rancher/iss

It's a weird one. I explored the idea of excluding IPsec on SPD level but did not find a way. I did not invest a lot of time into this. If you have an idea how to do this, do let tell, it does sound like the right way to deal with it.

@saper they're generated by: git.occrp.org/libre/metro

To e specific: git.occrp.org/libre/metro/blob

So, something like:
spdadd "{LOCAL_IPV4}/32 ${REMOTE_IPV4}/32 any -P out ipsec
esp/transport//require ah/transport//require;

(and reverse, and same for IPv6)

@rysiek just to check, since I do not see it enabled in racoon.conf - you are not using UDP encapsulation for ESP?

@saper no; nat_traversal is not set and the default is, I believe, "yes" (which means "use when NAT is discovered during phase 1 negotiatnion"); there is no NAT between the peers though, and in tcpdump I only see ESP/AH.

@rysiek no ideas. what really wonders me how non-IP traffic (your already encrypted ESP) gets encapsulated by IPsec again. Maybe you can find out why.

@saper I am *GUESSING* it's IPsec transport mode on the outside, IPsec *tunnel* mode on the inside. Quite possible that the rancher IPsec (the one in tunnel mode, on the inside) uses NATT now that I think of it, but not sure.

@rysiek can you sniff stuff going out of rancher somehow?

@saper not really, no. I guess I would have to use Wireshark's IPsec decryptor.

@rysiek my tcpdump(1) has -E:

-E Use spi@ipaddr algo:secret for decrypting IPsec ESP packets that
are addressed to addr and contain Security Parameter Index value
spi. This combination may be repeated with comma or newline sep‐
aration.

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!