Somewhere in Australia, 2003

EDNS, pf, and IPv4+6 Packet fragmentation

Alternatively: what the fuck is this "No response was received until the UDP payload size was decreased" thing all about?!

(tl;dr: Disable pf’s packet normalization / scrubbing for IPv6 only)

If you’ve ever done anything with DNSSEC you’ve probably used DNSViz.net to see what’s going on. That and Verisign’s DNSSec Debugger.

Yesterday it was pointed out to me that DNSViz.net was throwing out warnings for a DNS slave zone that I have been hosting for a while. The warnings looked a bit like this:

No response was received until the UDP payload size was
decreased, indicating that the server might be attempting to
send a payload that exceeds the path maximum transmission unit 
(PMTU) (UDP_0_EDNS0_32768_4096)

This looks like I messed up somewhere and it should be a relatively easy fix, right? Except I checked … and it looked like, for once, I didn’t mess up.

So off to the troubleshooting mines we go. The first obvious place to look was bind. It has the option edns-udp-size, which I had not set (the 4096 byte default seems reasonable enough).

After fiddling with bind a bit and trying to reproduce the problem, I could not. No matter what requests I sent to bind I would get the correct result back each and every time.

So then remote troubleshooting. This sounded easy, but proved to be a bitch and a half in practice. The first problem I had was that the only the one test system I have with half-way decent IPv6, outside of the hosting environment, is in my home-network (which is all sorts of weird), and the other is going through Hurricane Electric, who are pretty good at dropping packets.

So that was a dead end. I’d probably have to understand what the problem was rather than reproduce it somehow. After doing some research, it looked like there is a pretty interesting difference in the way that IPv4 and IPv6 handle fragmented packets.

In IPv4 packet size is not static. Any router in between you and the host you’re trying to reach can disassemble, reassemble, and resize packets as they see fit. A pretty useful feature no doubt, because there are people like me who use PPPoE, and there are servers with decent network connections who use jumbo frames.

In IPv6, however, the use of PMTU (Path MTU) discovery appears to be required. Only the sending party can adjust the MTU and packets aren’t (supposed to be) reassembled until they reach the receiving host.

This might explain why I was seeing the problem only on IPv6 and not on IPv4, even though there are few differences in network settings (particularly MTU) between the two.

Now my DNS server is behind two firewalls that both do some (fairly straightforward) NAT stuff. One is iptables, the other is pfSense, or FreeBSD pf, depending on how you want to look at it.

Pinging back and forth between hosts (ping -s 2000 is my friend), it looked like the pfSense system was the culprit. Digging into the options I found one that looked suspicious:

[ ] Disable Firewall Scrub

Disables the PF scrubbing option which can sometimes interfere with NFS traffic.

NFS of course uses large UDP packets, just like EDNS.

I turned this option on (to disable scrubbing) and lo and behold the DNSViz.net warning disappeared. For IPv6. It was now showing up for IPv4. Ugh.

Digging into it a bit more, it turns out that when you have that option unticked, pfSense generates pf-rules like:

scrub on vtnet1 fragment reassemble

Sounds about right. A bit more fiddling later, and it looked like my pfSense was indeed reassembling the IPv6 packets (which is a bad idea to begin with), and then trying to shove them into its WAN interface in their reassembled state. The packets being >1500 bytes in size, that would trigger an ICMPv6 message that the packet was too big and so the packet would make like Jimmy Hoffa and disappear.

Disabling packet normalization, however, would trigger problems on IPv4. For some reason I can guess at but don’t fully understand, when you stop reassembling IPv4 packets, NAT rules are no longer applied to the fragments. This would cause pfSense to forward packets from my DNS server’s internal address (10.x.x.x) out onto its WAN port. The router on the other side would then yell something about martians and drop the packets harder than a Browns receiver drops a pass.

Because reassembly is allowed in IPv4, the obvious solution was to turn scrubbing back on for just IPv4. Unfortunately pfSense is a steaming pile of PHP, so that wasn’t all too easy. Apparently, according to the NetGate docs you’ll have to edit a PHP file that lives in /etc. Gross.

That file contains a function called filter_generate_scrubing() (Yeah, a single b), which has a bunch of if/else statements that generate part of your pf.conf. Still gross.

if (!isset($config['system']['disablescrub'])) {
  $scrubrules .= "scrub on \${$scrubcfg['descr']} all {$scrubnodf} {$scrubrnid} {$mssclamp} fragment reassemble\n"; // reassemble all directions
}

And this would produce the resulting scrub on vtnet1 all fragment reassemble rules.

Some fucking around with the, infuriating, pf-syntax later I decided the output should instead be: scrub on vtnet1 inet all fragment reassemble. So I replaced that line, and re-enabled packet normalization in the pfSense GUI:

$scrubrules .= "scrub on \${$scrubcfg['descr']} inet all {$scrubnodf} {$scrubrnid} {$mssclamp} fragment reassemble\n"; // reassemble all directions

Now my IPv6 packets were no longer being touched (so PMTU was working like it was supposed to), and my NAT rules were applied to my reassembled-and-then-disassembled IPv4 packets again.

Great. Success.

I’m not sure if this is a bug in pfSense. I’d say yes, because as far as I can tell you’re not supposed to disassemble IPv6. But then, maybe they have some good reason to do so anyway. There appears to be a lot of disagreement. ¯\_(ツ)_/¯. At least there is a work-around now.