Quit Thinking and Look

It all started while configuring a DHCP scope.

The environment in question has a server for booting machines from the network to be imaged and for the first time, we were expanding this to other subnets. That meant we needed to configure options in DHCP so that PXE clients can find the imaging server when booting. Not a major issue, right? Just configure options 66 and 67 and watch the machines boot.

DHCP on this network is handled on the same switch where the SVI for the subnet terminates. It's an Aruba switch, which means there's a frustrating need to disable DHCP before making configuration changes. Oh, so that subnet just won't have DHCP for a few minutes?. Nope, none of the subnets that terminate on this device will have DHCP while making configuration changes.

Accepting this requirement, I started configuring the scope:

EvilSwitch(config)# dhcp-server pool vlan1-pool
EvilSwitch(vlan1-pool)# option 66 ip 192.0.2.10
DHCP server does not allow raw option 66.

Unsurprisingly, I fared no better with option 67. It turns out that doing this requires using named commands instead;

tftp-server server-ip 192.0.2.10
bootfile-name "boot\x86\wdsnbp.com"

Slightly odd, but those are two very common options to set, so I understand the desire to make the configuration simpler (though I'd prefer to at least be allowed to use the raw options if I'd like). For anyone who isn't familiar with WDS, the x86 version of wdsnbp.com will start the boot process. These machines aren't doing UEFI imaging, so the x86 version works perfectly. Future project: UEFI imaging.

So with the proper options set, I connected a machine, button-mashed F12 and...nothing. Well, not quite nothing. I got a helpful "PXE Error" message. Can it not reach the PXE server? Can it find it, but not download the file? Did I forget to re-enable the scope because of that silly requirement to disable it before making config changes?

I quickly ruled out the obvious problems by booting normally with a machine on that network and confirming that I received an address, that I can reach the WDS server and that a Windows-based TFTP client was able to download the file. The latter after going down a long rabbit hole suspecting a problem with TFTP block size.

As it turns out, using a machine running Insider Preview wasn't a good idea due to an apparent bug in the TFTP client. I put that laptop aside and (after confirming my initial problem with PXE booting was still present) downloading the file from within Windows worked perfectly. Apparently, the optional TFTP client doesn't get much testing attention.

The obvious ruled out, I started thinking about what's different on the PXE boot attempts from downloading the image via command line. I checked the filename and IP address in the scope options for the 50th time and ruled them out again. With the obvious out of the way, it was time to put my number one troubleshooting rule to work, courtesy of David J. Agans in his book Debugging Rules

Quit thinking and look!

Once the most likely "obvious" issues are out of the way, stop guessing at the problem and look at what is happening. In this case, the "obvious" issues are things like double checking the filename, IP address, that ping works, etc. There's a good argument that trying to download the file from Windows defeats this paradigm. There's a great argument that even thinking about things like MTU violates this (probably because it does).

Regardless, the way to see what is happening on this is to start looking at packets. And since TFTP isn't encrypted, this is an especially good time to stop making up hypotheses and dive in. I started Wireshark on the server-side and filtered for TFTP packets only:

There's the problem! The PXE client is trying to download the wrong file name, obviously I entered the wrong...oh. Upon seeing this, I hopped on the web to try and find any mention of this in the Aruba manuals, but haven't found anything about string encoding - if anyone has seen this before, let me know!

What's happening is the switch CLI took the string "boot\x86\wdsnbp.com" and treated the backslashes as an escape sequence. So, given that double quoting strings in this OS apparently interprets backslashes as an escape, the next logical step is to write the filename as:

bootfile-name "boot\\x86\\wdsnbp.com"

That did it. The next time I ran the test, the DHCP server passed on the filename with single backslashes, leaving literal backslashes instead of the escape sequence it previously sent. The test machine downloaded the boot file correctly and proceeded through the rest of the imaging process.

The lesson here? Remember to follow your own advice. I've probably shared that quit thinking and look quote with everyone I've ever worked with, and for good reason. Having real data about what's happening is better than even the best brainstorming session.