# Source: boot-net.adoc

*Note: This file could not be automatically converted from AsciiDoc.*

== Network booting

This section describes how network booting works on Raspberry Pi 3B, 3B+ and 2B v1.2.

On Pi 4 and Pi 5, network booting is implemented in the second stage bootloader in the EEPROM. For more information, see xref:raspberry-pi.adoc#raspberry-pi-bootloader-configuration[Raspberry Pi bootloader configuration].

We also have a xref:remote-access.adoc#network-boot-your-raspberry-pi[tutorial about setting up a network boot system].

Network booting works only for the wired adapter built into the above models of Raspberry Pi. Booting over wireless LAN is not supported, nor is booting from any other wired network device.

=== Network boot flow

To network boot, the boot ROM does the following:

* Initialise on-board Ethernet device (Microchip LAN9500 or LAN7500)
* Send DHCP request (with Vendor Class identifier DHCP option 60 set to `PXEClient:Arch:00000:UNDI:002001`)
* Receive DHCP reply
* (optional) Receive DHCP proxy reply
* ARP to tftpboot server
* ARP reply includes tftpboot server ethernet address
* TFTP RRQ `bootcode.bin`
 ** File not found: Server replies with TFTP error response with textual error message
 ** File exists: Server will reply with the first block (512 bytes) of data for the file with a block number in the header
  *** Raspberry Pi replies with TFTP ACK packet containing the block number, and repeats until the last block which is not 512 bytes
* TFTP RRQ `bootsig.bin`
 ** This will normally result in an error `file not found`. This is to be expected, and TFTP boot servers should be able to handle it.

From this point the `bootcode.bin` code continues to load the system. The first file it will try to access is `<serial_number>/start.elf`. If this does not result in an error then any other files to be read will be prepended with the `serial_number`. This is useful because it enables you to create separate directories with separate `start.elf` / kernels for your Raspberry Pis.

To get the serial number for the device you can either try this boot mode and see what file is accessed using tcpdump / wireshark, or you can run a standard Raspberry Pi OS SD card and `cat /proc/cpuinfo`.

If you put all your files into the root of your TFTP directory then all following files will be accessed from there.

=== Debugging network boot mode

The first thing to check is that the OTP bit is correctly programmed. To do this, you need to add `program_usb_boot_mode=1` to `config.txt` and reboot (with a standard SD card that boots correctly into Raspberry Pi OS). Once you've done this, you should be able to do:

[source,console]
----
$ vcgencmd otp_dump | grep 17:
----

If row 17 contains `3020000a` then the OTP is correctly programmed. You should now be able to remove the SD card, plug in Ethernet, and then the Ethernet LEDs should light up around 5 seconds after the Raspberry Pi powers up.

To capture the Ethernet packets on the server, use tcpdump on the tftpboot server (or DHCP server if they are different). You will need to capture the packets there otherwise you will not be able to see packets that get sent directly because network switches are not hubs!

[source,console]
----
$ sudo tcpdump -i eth0 -w dump.pcap
----

This will write everything from eth0 to a file named `dump.pcap`. You can then post-process or upload the packets to cloudshark for communication.

==== DHCP request / reply

As a minimum you should see a DHCP request and reply which looks like the following:

----
6:44:38.717115 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 348)
    0.0.0.0.68 > 255.255.255.255.67: [no cksum] BOOTP/DHCP, Request from b8:27:eb:28:f6:6d, length 320, xid 0x26f30339, Flags [none] (0x0000)
	  Client-Ethernet-Address b8:27:eb:28:f6:6d
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 12:
	      Vendor-Option, Vendor-Class, BF, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135, TFTP
	    ARCH Option 93, length 2: 0
	    NDI Option 94, length 3: 1.2.1
	    GUID Option 97, length 17: 0.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
	    END Option 255, length 0
16:44:41.224619 IP (tos 0x0, ttl 64, id 57713, offset 0, flags [none], proto UDP (17), length 372)
    192.168.1.1.67 > 192.168.1.139.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0x26f30339, Flags [none] (0x0000)
	  Your-IP 192.168.1.139
	  Server-IP 192.168.1.1
	  Client-Ethernet-Address b8:27:eb:28:f6:6d
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 192.168.1.1
	    Lease-Time Option 51, length 4: 43200
	    RN Option 58, length 4: 21600
	    RB Option 59, length 4: 37800
	    Subnet-Mask Option 1, length 4: 255.255.255.0
	    BR Option 28, length 4: 192.168.1.255
	    Vendor-Class Option 60, length 9: "PXEClient"
	    GUID Option 97, length 17: 0.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68
	    Vendor-Option Option 43, length 32: 6.1.3.10.4.0.80.88.69.9.20.0.0.17.82.97.115.112.98.101.114.114.121.32.80.105.32.66.111.111.116.255
	    END Option 255, length 0
----

`Vendor-Option Option 43` contains the important part of the reply. This must contain the string "Raspberry Pi Boot". Due to a bug in the boot ROM, you may need to add three spaces to the end of the string.

==== TFTP file read

When the Vendor Option is correctly specified, you'll see a subsequent TFTP RRQ packet being sent. RRQs can be replied to by either the first block of data or an error saying file not found. In a couple of cases they even receive the first packet and then the transmission is aborted by the Raspberry Pi (this happens when checking whether a file exists). The example below is just three packets: the original read request, the first data block (which is always 516 bytes containing a header and 512 bytes of data, although the last block is always less than 512 bytes and may be zero length), and the third packet (the ACK which contains a frame number to match the frame number in the data block).

----
16:44:41.224964 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 49)
    192.168.1.139.49152 > 192.168.1.1.69: [no cksum]  21 RRQ "bootcode.bin" octet
16:44:41.227223 IP (tos 0x0, ttl 64, id 57714, offset 0, flags [none], proto UDP (17), length 544)
    192.168.1.1.55985 > 192.168.1.139.49152: [udp sum ok] UDP, length 516
16:44:41.227418 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 32)
    192.168.1.139.49152 > 192.168.1.1.55985: [no cksum] UDP, length 4
----

=== Known problems

There are a number of known problems with the Ethernet boot mode. Since the implementation of the boot modes is in the chip itself, there are no workarounds other than to use an SD card with just the `bootcode.bin` file.

==== DHCP requests time out after five tries

The Raspberry Pi will attempt a DHCP request five times with five seconds in between, for a total period of 25 seconds.  If the server is not available to respond in this time, then the Raspberry Pi will drop into a low-power state. There is no workaround for this other than bootcode.bin on an SD card.

==== TFTP server on separate subnet not supported

Fixed in Raspberry Pi 3 Model B+ (BCM2837B0).

==== DHCP relay broken

The DHCP check also checked if the hops value was `1`, which it wouldn't be with DHCP relay.

Fixed in Raspberry Pi 3 Model B+.

==== Raspberry Pi boot string

The "Raspberry Pi Boot   " string in the DHCP reply requires the extra three spaces due to an error calculating the string length.

Fixed in Raspberry Pi 3 Model B+.

==== DHCP UUID constant

The DHCP UUID is set to be a constant value.

Fixed in Raspberry Pi 3 Model B+; the value is set to the 32-bit serial number.

==== ARP check can fail to respond in the middle of TFTP transaction

The Raspberry Pi will only respond to ARP requests when it is in the initialisation phase; once it has begun transferring data, it'll fail to continue responding.

Fixed in Raspberry Pi 3 Model B+.

==== DHCP request/reply/ack sequence not correctly implemented

At boot time, Raspberry Pi broadcasts a DHCPDISCOVER packet. The DHCP server replies with a DHCPOFFER packet. The Raspberry Pi then continues booting without doing a DHCPREQUEST or waiting for DHCPACK. This may result in two separate devices being offered the same IP address and using it without it being properly assigned to the client.

Different DHCP servers have different behaviours in this situation. dnsmasq (depending upon settings) will hash the MAC address to determine the IP address, and ping the IP address to make sure it isn't already in use. This reduces the chances of this happening because it requires a collision in the hash.