I wonder how many bugs there are like this in low-level networking infrastructure that cause issues which go undiagnosed for years before someone with enough knowledge to diagnose them figures it out. I keep reading about weird networking glitches triggered by specific bit patterns, but certainly if I ever encountered such a think I don't think I'd be able to diagnose it.
One WTF I've seen: a consumer router which failed to NAT outgoing packets if their endian-swapped destination IP matched the LAN subnet, i.e. anything sent to X.1.168.192 would be forwarded to WAN with 192.168.1.Y source IP and get lost. Took me a few weeks to figure out why certain random websites don't work.
Reminds me of an issue [1] I ran into with Symantec Firewall looking at UDP fragments as if they were full packets, and blocking them if the value where the port number would be on a full packet matched a rule.
Not a network code issue but my favorite anecdote for this type of thing. There was a Xen hypervisor bug for YEARS that would cause the clock to jump 20(?) minutes occasionally. The root cause was found in inline assembly due to a compiler oddity. It was found during some bug day/week thing by somebody with the assembly knowledge who just happened to look at that code and see an issue. He had no idea of the significance of what he found :D
I remember it was patched and out by debian right around Christmas; quite the present. This bug made me feel very helpless. I recall there being more information around the circumstances of its discovery on the Xen mailing list.
At the university I used to work out, the swapped out the Cisco firewalls for Junipers. One of my applications immediately started having random errors. A week of troubleshooting with the network team, and Juniper support, concluded that the routers were silently dropping some UDP packets. No logs for it, and no rules to explain it, and no pattern to it either. It took days to just convince the network guys that it really might be the firewall, since it wasn't reporting any issues or any reason for the packet loss.
Also no fix for it. We had to switch the application to TCP in order to work around the problem.
There were issues with Cisco switches and G2/G3 HP Proliant servers which caused duplex flapping. We had to force it to 100/Full on both servers and switches to fix the issue.
I seem to recall that a history passed through HN a while back that was about someone diagnosing a fault in a router beyond their ISP. This because their VPN or some such would fail when it got routed through that specific path.
> I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee.
If I have a network scanning tool that tells me what's on my network, it's nice to know that an unknown prefix is likely an Apple product, or an HP printer, etc etc.
> "This 4/6 MAC issue was well documented in BCP128 back in 2007. The control-word drafts mentioned that there would be dragons related to 4 and 6 back in 2004."
This reminds me a little of what Mr. Prosser said to Arthur Dent in the Hitchhiker's Guide to the Galaxy about the demolition notice for Arthur's house: "It was on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying beware of the leopard."
You can't make the assumption that engineers will be able to keep all of this in their heads. The IEEE is not going to review every document and put together a compendium of bear traps awaiting them in the future, and they aren't going to review every historic document before every decision, either.
If the few that knew what would happen made an incorrect assumption of prior knowledge, then they dropped the ball in reminding those that needed to know.
Since nobody remembered, they should have just been informed of the mistake and a possible solution or assistance provided.
Ethernet frames start with the destination mac address, IP payloads start with the version. It appears that in some networking equipment, there are places where it may have a ethernet frame or an IP playload and uses the first four bits to determine: if it starts with 4, it treats it as IPv4, and 6, IPv6.
So if it gets an ethernet frame destined for your address starting with 4 or 6, it will think it's actually an IP payload, and discard it because it doesn't validate as a IP packet or otherwise misdirect it.
Edit to add: In MPLS networks, 'P routers' (core routers) by design don't know if the payload is Ethernet or IP; but may want to peak into the payload to do hashed traffic balancing or validating packets before forwarding. Hashing the packet as an IP packet when it's an ethernet packet is problematic because you want all packets on a given TCP connection to have the same hash value -- but since you misidentified the type of packet, the bytes used as input to the hash aren't static for a given connection so different packets in the connection may take different paths and may arrive out of order.
Seems like an attempt at being "clever". L2 equipment shouldn't care what kind of packet it is, but we're trying to subvert the original spec and come up with new hacks by "peeking" into the payload and make assumptions about what it is in order to rule or queue it differently.
Obviously in hindsight this is a bad idea and "idiotic", but I can't see the immediate fault of whomever first decided to try this a decade ago when MAC addresses were sequential, IPv4 was fairly abundant, and we didn't really even have smartphones and the explosion of connected devices.
Ethernet has always "cared" what type of upper payload its carrying. It has to, there are any other things besides IP that run directly on top of Ethernet frames. For instance 0x0800 means that the ethernet frame is carrying an IPv4 datagram.
What the GP means is that layer 2 switches should forward traffic agnostic of what the Ethertype field says. That field only exists so layer 3 knows what it's dealing with.
You might enjoy watching https://www.youtube.com/watch?v=cXSwoKu9zOg&t=5s - a video recording about "What's Up With Poor Performance Towards MAC Addresses Starting With 4 or 6?" presented at the North American Network Operator Group (NANOG).
An MPLS router would like to know if the encapsulated packet is IPv4 or IPv6. If it is, the packet can be safely put onto multiple links which have the same approximate costs. They look at the first byte because in IP that is where the version number is stored[1]. Because this assumption may not be true, MPLS can carry Ethernet traffic for example, some times non IP traffic can match. Additionally, until recently, there were no MAC addresses which started with 4 or 6.
[MPLS is] ... often referred to as a layer 2.5 protocol. It was designed to provide a unified data-carrying service for both circuit-based clients and packet-switching clients which provide a datagram service model. It can be used to carry many different kinds of traffic, including IP packets, as well as native ATM, SONET, and Ethernet frames.
"any application that defines an FEC
that does not take measures to prevent the values 0x4 and 0x6 from
occurring in the first nibble of the payload may be subject to IP
ECMP and thus having their flows take multiple paths and arriving
with considerable jitter and possibly out of order."
That's not enough context I feel for most readers. I'm quoting a bit more here which I think will better explain the issue:
"In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP. Providers of MPLS equipment sought to continue this IP ECMP behavior. As shown above, it is not possible to know whether the payload of an MPLS packet is IP at every place where IP ECMP needs to be performed. Thus vendors have taken the liberty of guessing the payload. By inspecting the first nibble beyond the label stack, existing equipment infers that a packet is not IPv4 or IPv6 if the value of the nibble (where the IP version number would be found) is not 0x4 or 0x6 respectively. Most deployed LSRs will treat a packet whose first nibble is equal to 0x4 as if the payload were IPv4 for purposes of IP ECMP.
A consequence of this is that any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."
> "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP."
You can pretty much carry "Any Transport over MPLS" (AToM). As mentioned, IP is the most common protocol, of course, but you can "tunnel" almost anything you can think of over an MPLS path: Ethernet, ATM, PPP, HDLC, SONET, etc. Basically, you dump a frame into one side and it comes out the other unmodified -- regardless of the network in between.
As noted in the linked e-mail, an ISP providing (layer 2) transport has (little to) no control over the payload so there's nothing they can do to prevent running into this issue.
Wow it blows my mind. Both that a switch might have issues forwarding packets with a perfectly valid MAC, and that there are so many MAC squatters out there that the IEEE felt this was a good strategy. I guess its going to get harder to use MACs like CA:FE:00:C0:FF:EE to easily spot your packets coming from your home grown FPGA ethernet MAC implementation :-)
That's already a locally-administered address as the first byte ends in the hex nibble 'A'.
Aside from that, the IEEE's actions are a little odd here, but when you read BCP128[1]; you can see, it's really not their fault. This is due to vendors choosing a wacky and poorly thought out mechanism for identifying packet types on MPLS networks.
Certain routers are processing both IP packets and ethernet frames through the same logic. They want to know which it is. Ethernet frames start with the destination mac address, which used to be sequential and reaching higher numbers seemed far away so most started with 0 or 1. IP packets start with the IP version, so they're 4 or 6.
As a result, some vendors basically implement the following:
def is_ethernet_frame(data):
return not (data.startswith(4) or data.startswith(6))
And similar to the problems with
os.name.startswith('Windows 9')
type issues, now we reach the higher numbers, these systems have issues.
To save space in the MPLS header, it doesn't specify what comes next. So inside an MPLS packet may be Ethernet, IP, or something else and in many cases the switch/router has no way of knowing what is inside the packet, but it still wants to find the 5-tuple to generate ECMP entropy.
Until fairly recently (in IEEE/OUI terms, ~2012), the distribution has been entirely sequential. FTA:
"I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee"
Nonetheless it was a broken parse: they needed to expend the memory to figure out what the actual protocol is and not jump up from the datalink layer if it's not called for. I guess I'm just saying I really can't blame IEEE here.
Oh, I don't blame the IEEE wholly - but I think that they probably changed tack after realizing that a number of manufacturers were simply not paying the fees to actually be assigned a block of OUIs. I view it as kind of a cudgel to be used against vendors, and end-users (read: netops/sysops) are caught in the middle.
Wow that's some serious persistence to work a Cisco TAC case all the way up to it becoming a documented bug. I feel for those network engineers. I can only imagine how many hours they spent on the phone with TAC. I've been there, its not fun.
It's wonderful to see how bugs in network hardware are suddenly the problem of the registration authority and they need to delay a few years more. How about the vendors start fixing their bugs instead? And if you have hardware and there will be no fix, consider that the next time your going to spend your money on hardware.
