There are a few absolutes in this world. Death. Taxes. And MAC addresses are unique.
Or so I thought.
We were doing a big maritime installation for a customer, somewhere in Europe, the kind of project the company had done probably a hundred times. I’d personally done 20 or 30 of them by that point. They’re all hard, but they’re understood. That difficulty is the reason we get hired.
We were about a month into a 3-month project. A decent amount of equipment was installed and we’d started our QA process, going back through, double-checking that everything was registered and working correctly.
Then a trend started showing up.
We’d visit a location and find that the settings we’d configured a few days earlier were missing. We’d re-enter them, move on, come back later, gone again. Over the course of about a week, the pattern got harder to ignore. We kept going back to the same spots, re-entering the same information.
Then we narrowed it further. When we’d update Device A, Device B would go offline. Every time.
We got the home office involved. “Hey, development, we have a strange problem.” They started looking through the logs. We were looking through ours. And it was one of my guys on-site who said it first:
“I think these two devices have the same MAC address.”
No. That can’t happen. MAC addresses are unique. That’s the whole point. Death, taxes, and MAC addresses.
We ran the check. And sure enough, duplicate MAC addresses. On custom devices we’d purchased from a third-party vendor.
It got worse. After going through all our logs, we found that we were about halfway through the installation and had up to 5 duplicate MACs in some areas. We started losing our minds, because this is not supposed to be possible.
The customer’s IT team honestly almost giggled. They understood the magnitude of the problem, they just seemed relieved that they’d hired a vendor to deal with it.
We went back to the hardware manufacturer. This can’t happen. But here’s what we’re seeing.
Their answer: “We inadvertently used the same production template multiple times in our manufacturing run. Yes, we shipped duplicate MAC addresses.”
OK. Fix it.
“Sure. It’ll take 3 to 6 months to build another custom run.”
The project was due in two months.
The 48-hour fix
Working with our development team, we built a software workaround and pushed it to every device on the project in about 48 hours.
Here’s how it worked: when a device connected to the database for the first time and the database had never seen that MAC address before, it assigned a serialized sequence number. That became the device’s new identity. The next time a device connected with a MAC the database had already seen, it asked for the serial number. If the device didn’t have one, it got assigned one. Now it was a unique, known device, regardless of what the hardware said.
Was it a perfect fix? No. The duplicate MACs were still there at the network layer. I’m sure the customer’s network team is still scratching their heads to this day. But we now had serialized devices attached to serialized locations with known IP addresses, and we could unicast and multicast to them reliably.
We delivered the project, to the best of my knowledge,on budget and, more importantly, on time.
Death, taxes, and MAC addresses. Two out of three ain’t bad.