While the intention in the past hasn't been to make OpenWrt self hosting, with a few additions it can be. This is still experimental and requires workarounds for bugs in perl, llvm, and OpenWrt itself. This is a work in progress, but building OpenWrt inside of OpenWrt is possible.
Why Build OpenWrt on OpenWrt?
So far this has been tested on armv7l and aarch64 devices, but will likely work on most if not all OpenWrt supported architectures. However the device's hardware does need to meet some minimum requirements:
Note1: The initial toolchain build (especially of llvm-bpf) is quite RAM-hungry. The given memory requirements should be considered a minimum if you are building your own llvm-bpf. If you are able to use a pre-built llvm-bpf, then you can get away with less RAM-per-core. If you run this procedure on a QEMU virtual machine, dont use ballooning memory as it will result in oom-errors (during tools/cmake).
Note2: Your swap space (and to a lesser extent, the storage space for the build itself) should be on a hard drive or high quality solid state storage device. It should not under any circumstances be on the device's internal eMMC storage. This is to avoid NAND flash wear as this process is write heavy especially with swapping. Also, the larger your storage device is the more the wear will be spread.
Elements of the build system don't like to be built as root. OpenWrt is essentially single-user, so to get around this:
export FORCE_UNSAFE_CONFIGURE=1
You might want to add the above to your ~/.profile
First of all you need a lot of OpenWrt packages:
opkg install pkg-config make gcc diffutils autoconf automake check git git-http patch libtool-binopkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip bzip2 flock wget-sslopkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b perlbase-ipc perlbase-module perlbase-extutils perlbase-time perlbase-json-pp python3 opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq coreutils-joinopkg install libncurses-dev zlib-dev musl-fts libzstd ln -s libncursesw.a /usr/lib/libncurses.aopkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr xxdNote3: The test systems this was performed on had quite a bit of perl pre-installed, so there may be more perl packages required.
Before we can go further, we have our first workaround:
OpenWrt Bug: OpenWrt doesn't put execute permissions on the scripts for the automake, autoconf, and libtool packages. This prevents a lot of build systems (including OpenWrt's own) from working correctly. To fix:
chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL
chmod +x /usr/share/autoconf/Autom4te/*
chmod +x /usr/share/libtool/build-aux/*
A few dev libraries are needed. Two of them are are included in OpenWrt, but by default OpenWrt strips them (using sstrip) so that you can't actually compile against them. Switch to your ~/devel folder (or wherever you're going to use as your base build directory) and do the following:
git clone https://github.com/xhebox/libuargp.git cd libuargp make make prefix=/usr install ln -s libargp.so /usr/lib/libargp.so.0 cd ..
git clone https://github.com/void-linux/musl-fts.git cd musl-fts ./bootstrap.sh ./configure --prefix=/usr make make install cd ..
git clone https://github.com/void-linux/musl-obstack.git cd musl-obstack ./bootstrap.sh ./configure --prefix=/usr make make install cd ..
ar -rc /usr/lib/libdl.a ar -rc /usr/lib/librt.a ar -rc /usr/lib/libpthread.a ar -rc /usr/lib/libresolv.a
wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz tar -xvzf zstd-1.5.2.tar.gz cd zstd-1.5.2/lib make make prefix=/usr install cd ../..
ln -s bzip2 /usr/bin/bunzip2 ln -s bzip2 /usr/bin/bzcat
git clone https://github.com/akimd/bison.git git clone https://github.com/swig/swig.git cd swig sh autogen.sh ./configure --prefix=/usr --without-pcre opkg install bison m4 chmod +x ./swig/Tools/config/install-sh export BISON_PKGDATADIR=/home/buildbot/bison/data cp /usr/share/autoconf/m4sugar/* /home/buildbot/bison/data/m4sugar/ export M4=m4 make make install
The single tool needed in the build system that OpenWrt doesn't support is rev, a little-known and littler-used Unix tool. It's part of util-linux, which does provide a lot of packages for OpenWrt, but rev isn't one of them. It is, however, built as part of the process below, so it'll be be installed later.
This is based on the build system instructions, with a bunch of deviations to fix various bugs along the way:
Make sure you're in your devel directory from before. The following assume you are building SNAPSHOT, but can be (with appropriate changes) be used for release versions as well.
git clone https://git.openwrt.org/openwrt/openwrt.git cd openwrt git checkout master git pull ./scripts/feeds update -a ./scripts/feeds install -a
wget https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/config.buildinfo -O .config
make defconfig
wget https://va1der.ca/~public/openwrt/llvm/llvm-bpf-15.0.7.Linux-armv7l.tar.xz tar -xvaf llvm-bpf-15.0.7.Linux-armv7l.tar.xz
make menuconfig mkdir tools/llvm-bpf/patches wget https://va1der.ca/~public/openwrt/patches/llvm-bpf/901-fix-fuzzer-linking.patch -O tools/llvm-bpf/patches/901-fix-fuzzer-linking.patch wget https://va1der.ca/~public/openwrt/patches/llvm-bpf/902-use-libatomic-as-required.patch -O tools/llvm-bpf/patches/902-use-libatomic-as-required.patch
After musl-1.2.4 there were changes, such as lseek64 missing, and replaced with lseek, so llvm fails to build. To fix this, you need to make a minor change to llvm-bpf's Makefile, just under include $(INCLUDE_DIR)/cmake.mk add following “HOST_CFLAGS+= -D_LARGEFILE64_SOURCE”, so it will look a bit like this:
... CMAKE_SOURCE_SUBDIR := llvm include $(INCLUDE_DIR)/host-build.mk include $(INCLUDE_DIR)/cmake.mk HOST_CFLAGS+= -D_LARGEFILE64_SOURCE LLVM_BPF_PREFIX = llvm-bpf-$(PKG_VERSION).$(HOST_OS)-$(HOST_ARCH) ...
Once those patches are copied and Makefile is fixed, you can build llvm. Caution! This is memory intensive, and a multi-core build on a system with marginal memory can push the system so far into memory-debt swapping that it can damage solid-state storage. A single-core build will take longer, but is safer. This is going to take a long time. Note that a drawback of OpenWrt's build system is that any time the build of any one package is interrupted, it starts that package from the beginning the next time. You don't want compilation to stop on a random net blip after ten hours, so use nohup. And start getting into the habit of using V=sc and logging everything, because this whole process is still experimental:
nohup nice make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &
Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us.
NOTE: Once llvm-bpf is out of the way, if you have multiple cores and enough RAM per core, then you should be ok to use multi-core compiling. Always keep a periodic eye on your build, though and htop is your friend. For all examples throughout these instructions, -j 1 will be supplied to make for pasted commands for safety. But feel free to increase the core count at your discretion.
nohup nice make -j 1 V=sc toolchain/install > ~/toolchain01.out &
As with llvm, and all else, periodically check htop and the output log, debug, and repeat.
make download
rev. OpenWrt doesn't make a package out of it, even though it's built by default when OpenWrt builts packages out of its own util-linux. So we're going to build OpenWrt's util-linux then copy rev out. This is quick, probably no need to log it: make -j 1 package/util-linux/compile cp $(find . -wholename \*ipkg-install/usr/bin/rev) /usr/bin/rev
( cd feeds/packages/lang/perl && wget https://va1der.ca/~public/openwrt/patches/perl/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )
That fixes the first two bugs. This next fixes perl's configure bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it:
wget https://va1der.ca/~public/openwrt/patches/perl/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch
wget https://va1der.ca/~public/openwrt/patches/perl/test_index.perl -O ~/test_index.pl chmod +x ~/test_index.pl ~/test_index.pl
This will tell you if your perl is affected by the index() bug. If it is then build perl with the above bug patch above and copy libperl.so over top of of your system's buggy one:
make -j 1 package/perl/compile cp $(find staging_dir -name libperl.so) /usr/lib/perl5/5.28/CORE/ ~/test_index.pl
The test should now show the index() bug is fixed
nohup, tell make to give verbose output with V=sc, and continue to redirect that output to a log: nohup nice make -j 1 V=sc world > world01.out &
As noted above, it's beneficial to use a prebuilt llvm if possible. The following are available:
The above procedure has been tested, but this is still very much an experimental process and Your Mileage May Vary™. On the first successful attempt output was up to log file world13.out on two different platforms. The majority of errors were simple ones, caused by missing tools or inadequate ones where the basic busybox version wasn't good enough. There still may be missing packages in the above procedure, since it was tested on existing systems with preset installation packages that you may or may not have. There may be other subtle issues on other architectures. When htop tells you compilation has stopped, look at the log. Jump to the end, then (if you were doing multi-core builds) search backwards for “Error” (often with capital E) to find what exactly went wrong. Most of the time it's a missing tool. Sometimes the issue has been subtle problems with multi-core building. The timing of and staging of different parts of a build when doing a multi-core build changes on different architectures, and deficiencies in dependency settings may not show up in normal x86 builds that do on other archtectures. If you, for example, run into issues of symbols not being found on linking, sometimes simply redoing the build, or building that package in a single-thread build resolves the issue. Feel free to post your results in the forum thread for this.
There were quite a number of bugs that were exposed while developing this procedure. Bugs in OpenWrt, Perl, LLVM, and other places. However, one shouldn't consider the presence of bugs the deciding factor on whether a system is, in general, self-hosting. OpenWrt needs one small utility (rev), and three bandaid libs to cover for GNU extensions to GLIBC. That makes OpenWrt about 99.9% self-hosting ready. This is a great achievement, especially for an embedded OS where self-hosting wasn't even the goal, and more especially considering the enormous amount of complex code and a build system that uses so many different build tools. The OpenWrt devs are to be greatly commended for an excellent embedded OS!