Extroot configuration

This guide describes how to configure OpenWrt to use a storage device (USB or SATA or SD card or whatever) to expand your root filesystem, to install freely all the packages you need.

In most supported devices OpenWrt splits the internal storage into rootfs and rootfs_data partitions which are merged together into a single writable overlay filesystem.

This way OpenWrt fits even in tiny amounts of internal storage (as low as 4 MiB), but still allows to write settings and install some packages in the writable partition without changing all linux programs used. Extroot works by setting another overlay partition in the external storage device, and during boot this new overlay partition will be mounted over the internal storage's overlay partition. This approach allows easy fallback in case the external storage device is removed, as your device will still have its own overlay partition and thus will load all configuration from there. Which means that it will behave exactly the same as just before you set up extroot.

Note

This configuration will not be able to be used on devices that do not have the /overlay partition on mtd or on ROMs that do not have /overlay partition at all. In the first case OpenWrt will not want to read the configuration of /etc/config/fstab (FS#2231); in the latter case you can work around it by mounting the external/additional disk directly to /.

Devices with 8 MiB flash or more should have enough space to install the required packages, otherwise create a custom image. Remove all packages you have installed to add secondary functionality, as they are only wasting space now. Leave only those needed to access the internet.
After you make the extroot you will have all space you need to install secondary packages.

Follow USB installation guide to set up USB storage in OpenWrt, then install packages needed for a partition with ext4 filesystem:

opkg update
opkg install block-mount kmod-fs-ext4 e2fsprogs fdisk

Connect with ssh to the device. Configure /etc/config/fstab to mount the rootfs_data in another directory in case you need to access the original root overlay to change your extroot settings:

DEVICE="$(sed -n -e "/\s\/overlay\s.*$/s///p" /etc/mtab)"
uci -q delete fstab.rwm
uci set fstab.rwm="mount"
uci set fstab.rwm.device="${DEVICE}"
uci set fstab.rwm.target="/rwm"
uci commit fstab

Or you can identify the rootfs_data partition manually:

grep -e rootfs_data /proc/mtd

The directory /rwm will contain the original root overlay, which is used as the main root overlay until the extroot is up and running. Later you can edit /rwm/upper/etc/config/fstab to change your extroot configuration (or temporarily disable it) should you ever need to.

See what partitions you have using the following command:

block info

You will see similar output:

/dev/mtdblock2: UUID="9fd43c61-c3f2c38f-13440ce7-53f0d42d" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock3: MOUNT="/overlay" TYPE="jffs2"
/dev/sda1: UUID="fdacc9f1-0e0e-45ab-acee-9cb9cc8d7d49" VERSION="1.4" TYPE="ext4"

Here mtdblock are the devices in internal flash memory, and /dev/sda1 is the partition on a USB flash drive that we format to ext4:

DEVICE="/dev/sda1"
mkfs.ext4 ${DEVICE}

Now we configure the selected partition as new overlay via fstab UCI subsystem:

eval $(block info ${DEVICE} | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci commit fstab

We now transfer the content of the current overlay inside the external drive and reboot the device to apply changes:

mkdir -p /tmp/cproot
mount --bind /overlay /tmp/cproot
mount ${DEVICE} /mnt
tar -C /tmp/cproot -cvf - . | tar -C /mnt -xf -	
umount /tmp/cproot /mnt
reboot

1. LuCI → System → Mount Points should show USB partition mounted as overlay.
2. LuCI → System → Software should show free space of overlay partition.

The USB partition should be mounted to /overlay. Free space for / should be the same as /overlay.

# grep -e /overlay /etc/mtab
/dev/sda1 /overlay ext4 rw,relatime,data=ordered
overlayfs:/overlay / overlay rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work
 
# df /overlay /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              7759872    477328   7221104   6% /overlay
overlayfs:/overlay     7759872    477328   7221104   6% /

• Analyze the preinit stage of the boot log:

block info; uci show fstab; logread | sed -n -e "/- preinit -/,/- init -/p"

• If you receive a “block: extroot: UUID mismatch” error in your logs after upgrading, remove .extroot-uuid from the volume:

mount /dev/sda1 /mnt
rm -f /mnt/.extroot-uuid /mnt/etc/.extroot-uuid
umount /mnt

• Do not use vfat (FAT/FAT32); it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using mkfs.ext4 /dev/sda1 as per the example).
• If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase config global / option delay_root. On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot.

uci set fstab.@global[0].delay_root="15"
uci commit fstab

• : might be outdated Add option force_space in /etc/opkg.conf to allow installation of packets bigger than your /rom partitions free space:

echo option force_space >> /etc/opkg.conf

• Another possibility to consider and try is to modify /etc/rc.local as described in 14946 ticket, which in the case of running Chaos Calmer r44266 in the Comtrend AR-5387un, has been the only thing that allowed me to achieve extroot:

export PREINIT=1
mount_root

• If you are putting the extroot on a non-USB device such as a mmc card all modules needed acccess the device should be in appropriate file in /etc/modules-boot.d. For example using a sdhci card on a mt7688/mt7628 device /etc/modules-boot.d/mmc needs have two lines added:

mmc_core
mmc_block
sdhci
mtk_sd

Save opkg lists to /usr/lib/opkg/lists stored on the extroot, instead of in RAM. This makes package lists survive reboot and saves some RAM.

1. Navigate to LuCI → System → Software → Configuration to change /var/opkg-lists to /usr/lib/opkg/lists.
2. Navigate to LuCI → System → Software → Actions → Update lists to do an initial build of the package list onto extroot.

sed -i -e "/^lists_dir\s/s:/var/opkg-lists$:/usr/lib/opkg/lists:" /etc/opkg.conf
opkg update

If your device fails to read the lists due to small RAM such as 32MB, enable swap.

# Create swap file
dd if=/dev/zero of=/overlay/swap bs=1M count=100
mkswap /overlay/swap
 
# Enable swap file
uci -q delete fstab.swap
uci set fstab.swap="swap"
uci set fstab.swap.device="/overlay/swap"
uci commit fstab
/etc/init.d/fstab boot
 
# Verify swap status
cat /proc/swaps

It's a good idea to include the usb-modeswitch tool in the image. There is a caveat: if the /overlay points to a memory card sitting in a slot of the dongle - the otherwise working pivot overlay set-up will break in the later stages of OS boot. This is because the usb-modeswitch (while disabling the CDROM and enabling the modem) would also intermittently affect the card-reader in the dongle thus hurting the file system. To avoid this you need a dongle that can be pre-configured to enable its modem or network adapter (and the card-reader as well) on the power-up, without the need to do it with the usb-modeswitch on the router.

Insert your dongle in a desktop and use a terminal to send the necessary AT-commands. Check your dongle's initial configuration:

at^setport?
^SETPORT:A1,A2;1,3,2,A1,A2
OK

The meaning of the above report can be understood with the following command:

at^setport=?
^SETPORT:A1: CDROM
^SETPORT:A2: SD
^SETPORT:A: BLUE TOOTH
^SETPORT:B: FINGER PRINT
^SETPORT:D: MMS
^SETPORT:E: PC VOICE
^SETPORT:1: MODEM
^SETPORT:2: PCUI
^SETPORT:3: DIAG
^SETPORT:4: PCSC
^SETPORT:5: GPS
^SETPORT:6: GPS CONTROL
^SETPORT:16: NCM
OK

So, in the example above we have a dongle with CDROM and card-reader available in the first configuration (to the left of the ; character), and with modem, control and diagnostic interfaces, and card-reader available in the other configuration. It is between these configurations the usb-modeswitch switches the dongle on the router.

Your goal is to disable the CDROM and enable the modem (the 1 above) or the network adapter (the 16 above) while leaving the card-reader enabled (the A2 above). NOTE: Never disable the PCUI (the 2 above) - this will lock you out from your dongle!

Some dongles accept a 'disable all' operand (the FF below). Place the list of all the functions you need on your dongle by default to the right of the ; character according to their codes from the dongle's answer above:

at^setport="ff;1,2,3,a2"
OK
 
at^reset
OK
 
at^setport?
^SETPORT:;1,2,3,A2
OK

This sequence has disabled the CDROM and made the modem, control and diagnostic interfaces and the card-reader available by default - without any usb-modeswitch interaction. Thus only one configuration exists now in the dongle - see the ; character, there is nothing to the left of it now.

Pre-configuration support: Huawei E3131s-2 f/w v21.158.47.00.1094

Follow: Fstab not mounting cifs at boot or through CLI

This section applies to OpenWrt snapshot, but not to OpenWrt releases, as the kernel-related packages (and the packages requiring them) in releases will only receive fixes and security patches.

DO NOT try to do upgrades using opkg upgrade. You will likely end up with an inconsistent state and soft-bricked router that way:

• The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
• Another problem that can arise is if you try to upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
• Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like iptables will be broken.

This method is useful for devices with 4 MiB flash or less. In the default OpenWrt firmware images there are no tools to make extroot, as the build system currently makes only barebone images. The only way to go for these devices is to rebuild a firmware image with the right packages using the Image Builder. The Image Builder can run only in a 64bit Linux operating system, so if you don't have a linux system on hand, look up a tutorial to install Ubuntu 64bit in VirtualBox. Then go in the same download page where you can download the firmware for your device and scroll down until you find a file starting with “OpenWrt-imagebuilder”. Download it and extract it in a folder in the Linux system.

Open a terminal in that folder, and write:

make info

This will write on screen all the possible profile names for the devices supported by that Image Builder, so we can build the image for the right device. Each entry will look like this:

tl-wr1043nd-v1:
    TP-LINK TL-WR1043N/ND v1
    Packages: kmod-usb-core kmod-usb2 kmod-ledtrig-usbdev

First line is the profile name, the second line is a full descriptive name of your device, third line is a list of default packages for that device, and should list some packages about USB or Sata or whatever other storage device.

In my case I have a TP-LINK TL-WR1043N/ND v1, so the profile name for my device is tl-wr1043nd-v1 Now you need to write the command to start building the image (note how the name after the PROFILE= is my device's profile name, please use the profile name for yours):

make image PROFILE=tl-wr1043nd-v1 PACKAGES="block-mount kmod-fs-ext4 kmod-usb-storage kmod-usb-ohci kmod-usb-uhci"

This will build a firmware image that is able to read a partition formatted with ext4 filesystem. Sadly the package e2fsprogs with the tools for ext4 filesystem is too large to fit in 4 MiB devices.

Afterwards, open the folder bin inside the Image Builder folder, then open the target folder, then the folder you find in it (it has a device-type-specific name), and then inside a folder called generic and you should reach the flashable images. Choose the right image (factory or sysupgrade) and install it.

Then you will have to format the USB drive with ext4 filesystem, and to do that you will need to use a Linux LiveCD or gparted disk. Sadly this is inconvenient but as said above we cannot fit formatting tools in devices with 4MB of flash.

You can use the openwrt-auto-extroot ImageBuilder frontend to build a custom firmware image that will automatically format and set up extroot on any plugged-in, but not yet setup storage device.

cat << "EOF" > /etc/uci-defaults/90-extroot-restore
if uci -q get fstab.overlay > /dev/null \
&& [ ! -e /etc/extroot-restore ] \
&& lock -n /var/lock/extroot-restore \
&& [ -e /etc/opkg-restore-init ]
then
UUID="$(uci -q get fstab.overlay.uuid)"
OVRL="$(block info | sed -n -e "/${UUID}/s/:.*$//p")"
mount "${OVRL}" /mnt
BAK="$(mktemp -d -p /mnt -t bak.XXXXXX)"
mv -f /mnt/etc /mnt/upper "${BAK}"
touch /etc/extroot-restore
if grep -q -e "\s/overlay\s" /etc/mtab
then cp -f -a /overlay/. /mnt
fi
umount "${OVRL}"
lock -u /var/lock/extroot-restore
reboot
fi
exit 1
EOF
cat << "EOF" >> /etc/sysupgrade.conf
/etc/uci-defaults
EOF