Theory and Practice of crossbuilding on Debian and Ubuntu

This is a complicated subject with several aspects and different solution depending exactly what you are trying to do, and which distro/release you are trying to do it on. This page is the top-level resouce on this wiki, with links to other pages giving useful info or examples.

Crossbuilding support is still evolving in Debian/Ubuntu as we migrate to a fully multiarch world, but we do now have quite good tools which should let you get what you want done without too much pain. There has been a lot of development in the last couple of years, and the situation is OK in squeeze, quite good in Precise and we expect it to be reasonably complete in Wheezy/Quantal.

This page exists to gather information on both the theory and the practice of the current state of things, with a view to working out solutions for the remaining issues, and also providing a basic understanding for people new to the Debian cross-building world.

QuickStart - if you just want to build something and don't want a lot of whys and wherefores, start here.
CurrentPackageCrossBuildStatus - Status pages for various projects: Which packages currently build, and how the ones that don't, fail
MultiarchCrossbuilding - Howto for using the multiarch support to do cross-building. This will become more and more useful as more libraries and -dev packages are converted to be multiarch-compliant.

CrossBuildd - Build Daemon infrastructure giving visibility to which packages currently cross-build PackagingGuidelines - Info on how to make sure your package cross-builds.

UsingXdeb - Notes of setting up and using xdeb
UsingPdebuildCross - Notes on setting up and using pdebuild-cross
CrossBuildLampStack - Cross Bootstrapping A LAMP Stack Using xdeb
CrossBuilding - Spec of work in natty Linaro/Ubuntu development cycle

Terminology

Debian and GNU have names for things which have quite exact meanings, whilst people often use the same terms to mean rather different things. This can be thoroughly confusing when discussing this subject so lets start with a glossary, and try to clarify how terminology is used on this page:

Architecture
This is also commonly refered to as a 'port'. An architecture refers to a CPU instruction set (ISA) combined with a particular ABI and endianness. So amd64, sparc and armel are architectures. Often abbreviated to 'arch', and also often used to mean 'CPU architecture' independent of ABI. When discussing things, be careful of whether someone means 'CPU arch' or 'CPU+ABI+endianness' arch. It is used in the latter sense on this page.
ABI
Application Binary Interface. This is the calling convention used between one piece of code and another. One ISA (or CPU architecture) can have more than one ABI. It defines the sizes and order of arguments passed on the stack so that a C program can call a Java program and so on. Things with a matching ABI can be linked together. Some CPU architectures (like ARM) define a number of possible ABIs.
Flavour
This is the term used for the set of optimisations and/or hardware capabilites the code is built for. e.g using floating point or not (and which sort of FPU on arm), which variant of a the CPU ISA (i486, i586, i686, armv4, armv5, armv7), which affect both available instructions and suitable optimisations. For a given architecture (i.e. CPU type, endianness and ABI), there may well be a wide range of optional instructions and components (co-pros, multimedia units) and
Build/Host/Target

These are the GNU terms used by autotools and the toolchain. They are used consistently but don't sit well with normal english usage, even amongst computer scientists. Build is the machine we are building on. 'host' is the machine the code will run on. 'target' is the machine code will be generated for. This last is only used when referrig to code-gneration tools such as compilers. So a normal cross-build of a package in GNU-speak is done on the 'build' machines, and makes packages suitable for installation and running on the 'host' machine.

Unfortunately anyone not steeped in this terminolgy will tend to use 'host' to mean the machine they are doing the building on, and maybe 'target' to refer to the machine they are cross-building for. This document avoids using the term 'host' wherever possible as it just confuses. It uses 'build' or 'native' for the build machine and 'target' or 'cross-arch' for the machine we are building for (i.e 'targetting'). Clear as mud eh?

Native
This normally means 'the arch we are running on', which in GNU terminology is 'build' whilst building. xdeb rather unfortunately uses it to mean 'target' architecture throughout, which is just adds to the confusion. On this page 'native' is used to mean 'current arch', which for cross-building, is generally the build machine arch.
Foreign
This is the term for any architecture which is not the one the package was built for (i.e. targeted at). It is used instead of 'target' when we are not talking about any particular architecture, just 'another one'. Multiarch uses the term in dependency specifications.
Arch(itecture)-dependent/arch(itecture)-independent
This refers to dependencies. A dependency is arch-dependent if it must be satisfied by a package of the same architecture as the depending one (i.e. as is the case for libraries). A dependency is arch-independent if it doesn't matter which arch satisfies the dependency. This is generally true for tools, which just need to run on whichever arch is doing the building (like make, cat, flex, libtool, xsltproc). Some tools have output which varies according to architecture. These need to be dealt with carefully in order to have them both run, and produce correct output.
Suite
Suite is a name of a particular stability of release. Debian has 'unstable', 'testing' and 'stable' suites. A suite points to a particular codename and which codename is deemed 'stable' or 'testing' only changes when a codename is released and everything moves along one. Ubuntu has a different model where Codename and Suite always match up, so the distinction is largely arbitrary. You can usually refer to either Suite or Codename when referencing a release, but just occasionally it matters which you use.
Codename
A suite is the name for a particular release/distribution in a repository. So 'lenny', 'squeeze', 'sid' in Debian, and 'maverick', 'precise', qantal' in Ubuntu are all codenames. These names follow the particular release through its lifetime from initial development, through stable release to obsolesence and obscurity.

Overview

Debian cross-building is 'classic' cross-compiling, where the package build process needs to be aware that it is cross-building in order to avoid running binaries built for the target architecture, and tests on the compiled binaries. This is different from mechanisms which use emulation to run foreign binaries, such as Scratchbox and OBS. Autotools and CMAKE provide support for this, but a package's build rules need to be written correctly for this support to work properly. It is often not well-tested, or understood by upstream authors so many packages do not cross-build properly.

There are a number of issues that complicate cross-building. Binaries which are built and used during the build process need to be correctly treated so that they are built for the build machine and not the target machine. If they are also shipped in the final package then either that package must be installed as a build-arch dependency, or the binary must be built twice: once for the build architecture (for use during the build) and once for the target architecture (to go in the package). This is the area where much cross-building complexity lies. Qemu can be used to deal with such issues, but that doesn't help on architectures where it is not available. Tools which introspect objects that have just been built, for example can be an issue.

Build-dependencies need to be handled so that tools used at build-time are satisfied with build-architecture packages in the usual way whilst libraries (and arch-dependent headers) are satisfied by packages from the target architecture.

Aspects of the problem

Specifying a target architecture for a build

This aspect is easy and has been working for many years (since about 2001), originally with the dpkg-cross tool, but increasingly the important functionality has moved into dpkg itself.

You use the standard Debian build commands and supply a debian target architecture name. Because all the build tools (debuild, pbuilder, etc) all end up issuing a dpkg-buildpackage command this works everywhere in the form dpkg-buildpackage -a<archname>. That sets a dpkg status which can be interrogated by any tools that need to during the build using dpkg-architecture. Normally just the debian/rules file does this, and uses the result to set build-system (usually GNU configure) options. Non-auto* packages need to use corresponding mechanisms which are generally less well supported.

This system depends on the Debian packaging being cross-aware and knowing which things to do differently when cross-building. Many packages should at least build these days, but they may not be right. This is not a well-tested aspect of Debian/Ubuntu. Patches to fix breakage are found in the repositories of cross-built distros such as: emdebian crush, openembedded, buildroot, ptxdist, hackable etc. We are some years into the process of making Debian cross-buildable and still have some way to go, with more work needed to make the process work well for things that need multi-stage builds to bootstrap themselves (most languages), and some more complex builds where tools are built and then used in the build.

Crossing packages

You cannot install both the native and target libraries/headers/binaries in the same place in a filesystem. and prior to multiarch the package-management system could only install packages for one architecture: the native one (and arch-independent packages and source packages). Multiarch provides a proper, general, solution to this problem, by allowing you to install packages for more than one architecture at a time without file clashes. Arch -dependent libraries and headers are put in /usr/lib/<triplet> and /usr/include/<triplet> and the toolchain knows about that so stuff 'just works'. That is now deployed, but not all depended-upon packages have yet been 'multiarchified'. We are working up from the base packages in the system so the lower down the stack, the more likely things are to work. Check the links at the top of this page for info on the current state of play.

Architecture-dependent tools have <triplet> put on the front of the command-name, so we get <triplet>-gcc and <triplet-pkgconfig. Configure systems deal with most of this automatically.

If multiarch-based cross-building doesn't yet work for you, the older 'dpkg-cross' scheme can still be used. The mechanism here is converting packages to move files to new 'cross' paths, have a -cross on the end of the package name to indicate that they have been converted, and to have their architecture changed to be arch-independent. i.e we take the native libfoo1 package and make a libfoo1-cross package out of it with the files moved into /usr/<arch-triplet>/. Similarly for libfoo-dev -> libfoo-dev-cross. These packages are suitable for installation onto a foreign architecture (they are labelled arch 'all').

Nothing actually reserves this ('-cross') part of the package namespace so we have some -cross packages which are real, native, packages (such as dpkg-cross, apt-cross, pdebuild-cross), but most of them are packages that have had their paths munged in order to make them installable on a machine of a different architecture.

The tool that does this work is dpkg-cross, and it has been around for many years.

It moves headers, libraries, pkg-config and copyright files to new paths in the new package. All other files are thrown away. It also adjusts dependencies to match the new -cross namespace (e.g libxau-dev-armel-cross depends on libxau6-armel-crossa and x11proto-core-dev-armel-cross. This means that most packages, when run through dpkg-cross come out empty. Only library packages and -dev packages normally contain anything relevant. dpkg-cross does not do anything 'clever' - it does simple, predicitable, transformations on paths and dependencies. So an important thing to note is that it does not try to remove dependencies which are arch-independent - i.e. every dep gets converted to <dep>-cross. It is the responsibility of a higher-level calling tool such as apt-cross or xdeb to prune the list with the -X option as required.

Ensuring cross- and native build-dependencies are installed

This used to be a royal pain but multiarch and the MultiarchCross extensions solves it cleanly at long last.

Packages have both dependencies (things needed for the build architecture) and cross-dependencies (things needed for the host architecture), and do not specify which are which. For example, dhcp3 build-depends on: debhelper, dpkg-dev, groff, dpatch, po-debconf, libldap2-dev, libssl-dev, libcap-dev, hardening-wrapper.

Of these only libldap2-dev, libssl-dev and libcap-dev are cross-dependencies. The others are native dependencies which need to be supplied for the build architecture. Some mechanism is needed for tools to know which architecture of each dependency to install. A method to include appropiate metadata in packages for this was first proposed at Debconf7 (Edinburgh, 2007) by Simon Richter. Split the Build-deps into Build-deps: and Build-deps-tools:, where build-deps-tools are native dependencies. This has been implemented and used in Emdebian Crush and works fine. It could be done in Debian/Ubuntu itself, but Multiarch [same] [foreign] syntax provides essentially the same information so it made more sense to re-use that. The catch is that sometimes there are exceptions - for example something where the build depends on the build-arch version of a library. A syntax for these exceptions is specified in the MultiarchCross specification, and is being implemneted in apt and dpkg.

With multiarch support it should be a simple matter of configuring a foregn architecture in dpkg, then doing apt-get install -a <arch> build-dep <package> to get all the build-dependencies and cross-build dependencies installed. This only works once all the build-deps have appropriate multiarch meta-data to say that they are arch-independent or that they are co-installable (if they are) and exceptions to the default rules are marked in depending packages. Again this work is proceeding from the base system, upwards. In Wheezy and precise it works for a fairly small subset of stuff.

However you can still use the mechanism if you know what build-deps you need by doing apt-get install <package1> <package2> <package3>:<arch> <package4>:<arch>

If you are still doing dpkg-cross style building then getting cross-dependencies installed is much harder. You cannot use the multiarch mechanisms because the paths are different.

In the absence of proper metadata cross-building tools have tried various methods to gets the right packages installed. apt-cross and xdeb both have heuristics to guess from the package-name whether something is a cross-dep or not - they attempt to 'terminate' the cross-deps tree back into the native dependency tree by truncating the -cross dependencies listed in the -cross packages they create. Unfortunately this is hard to get right as there are a lot of special cases. xdeb has configuration black and whitelists to adjust this process. xapt avoids this problem by simply installing all the deps in both native and crossed form, which is very stupid but also effective. apt-ma-emu has its own set of heuristics.

All these tools work to a useful degree but there are issues and pros and cons with these approaches. These are explained below.

The cross-upgrade problem

The problem with the -cross package namespace mechanism is that every package thus created and installed is not a real package listed in a package file. libxau-dev-armel-cross is effectively a 'made-up' name. This means that every single -cross package is listed as 'locally generated/obsolete' by apt. This isn't a problem until you try and upgrade things:

1. We rename the packages using dpkg-cross.

1.1 Renamed packages appear in the /var/lib/xapt/status file and in the main system /var/lib/dpkg/status file.

1.2 Renamed packages do not appear in /var/cache/apt/pkgcache.bin

1.3 pkgcache.bin is under the sole control of apt and is rebuilt every time apt-get update is run.

1.3.1 pkgcache.bin only ever contains what apt is able to download from an apt source listed in or under /etc/apt/sources.list*

1.3.2 pkgcache.bin is what apt uses to resolve dependencies, falling back to /var/lib/dpkg/status only if something is mentioned which is not in pkgcache.bin

1.3.3 *IF* apt falls back to /var/lib/dpkg/status, apt and aptitude consider such listings as obsolete.

1.3.3.1 Obsolete packages are expendable, according to apt/aptitude, and are therefore removed if any non-obsolete package finds that an obsolete one is getting in the way.

1.3.3.2 Obsolete packages are not counted into the set of packages which apt or aptitude are capable of upgrading.

1.5 Net result is that either apt will refuse to upgrade -cross packages or will remove -cross packages (with all dependencies) whilst upgrading other packages.

1.5.1 Therefore, having -cross packages on a system causes apt to remove the -cross packages preferentially to any other package, causing toolchain breakage.

1.5.2 In addition, sometimes apt gets so confused that it either forces the removal of the entire cross-building toolchain or halts the normal system upgrade with a message that there is no solution and that the entire system is borked.

2.3 Therefore, renaming packages - as a method - is incompatible with the long term needs of a Debian system and can only be reliably managed inside a disposable (or frequently reset) chroot.

2.3.1 Any hope of a cross-building autobuilder must avoid putting any -cross packages onto the main system.

Use of chroots

Managing cross-built packages

Rebuilding everything for a new ABI/Flavour

To do this successfully you first need a suitable toolchain. In theory a cross-toolchain for an architecture can produce output for many different flavours (and even multiple ABIs), but there are problems in practice, to do with exactly how the toolchain was built and what code is in the libgcc that comes with it.

The most reliable way to target a particular flavour is to rebuild the cross-toolchain specfying the desired flavour as the default output. We should work on ways to make it possible to simply configure the desired defaults.

Cross-installing

Tools

xdeb

xdeb is a tool that started life as Chromium-os builder. It is a high-level tool that does source downloading, cross-building, cross-dependency satisfaction, package-crossing, build-recursion, bootstrapping, and caching already-built packages all itself. It is designed to take a set of sources and build them all to produce a distro. Using the --prefer-apt option you can make it satisfy dependencies from a repository instead of from local builds. Using --only-explicit --force-rebuild you can make it only build the packages specified on the command line so it can be used as a 'build package foo' tool.

It is written in python.

dpkg-cross

dpkg-cross is a venerable tool, and does three things.

1) Includes support for use of build-system (GNU autotools, cmake etc) cache files to set things which cannot be determined without the presence of the target platform, such as C type sizes and available hardware support, and package-specific tests which would normally be determined by running a build-time tool. All tests which go to the hardware to find something out need an alternative cache entry to allow cross-building in this way.

2) It also provides a mechanism to select a default target arch for cross-building, which is very convenient if you normally have one target as it saves a lot of typing -aarmel in various places. This is set either with dpkg-reconfigure dpkg-cross or setting default_arch in /etc/dpkg-cross/cross-compile

3) The other important aspect of dpkg-cross is its use for converting a native package 'foo' into a 'foo-cross' package installable on another architecture. It does this by moving arch-dependent files into new locations and throwing away arch-independent files, and making the resulting package arch_all. This is arguably an abuse of the package namespace and multiarch will remove the need to do things this way in the future. See 'using dpkg-cross' below for practical info.

It used to include a replacement dpkg-buildpackage script which understood the -a<arch> option, but that functionality now subsumed into dpkg itself, which is much neater.

The idea has been mooted to move all of the dpkg-cross functionality into dpkg, so that the package is no longer needed, but it may actually remain the best way to keep the cross-config/cache files.

apt-cross

Apt-cross is now obsolete. Its use is discouraged, and it has been removed from Debian Squeeze onwards. Use xapt instead. It is/was a tool to download a package from the archive of the target architecture, then run dpkg-cross on it to make a package suitable for cross-building against. It tried to determine which packages were 'cross-relevant' and which not, in order to prune unecessary dependencies. This proved very difficult to get right and maintain, leading to the tool's abandonment.

It is written in perl.

xapt

xapt is a comparitively simple tool which satisfies cross-build-dependencies by finding the build-deps of the given package, downloading the native packages, crossing them with dpkg-cross, and installing them, even though many of those packages will be empty. It is quick, simple and reliable, in contrast with all the other tools which try to decide what is a cross-dep and what isn't.

The downside of this approach is that a lot of stuff gets downloaded and installed, so it can be slow, and the system it is run on is rapidly polluted with hundreds of -cross packages. For this reason it is intended only to be run in a disposable cross-build chroot. This is done by pdebuild-cross, which runs xapt in its chroot(s).

It is written in perl and has minimal dependencies.

pdebuild-cross

pdebuild-cross is a tool that is used exactly like pdebuild, but will do cross-builds instead. It has the same advantages of doing everything in a clean chroot so that you know when dependencies have not been specified. And the same disadvantages of being one step removed so it's fiddly to deal with failed builds, and the time overhead of unpacking the chroot and installing all the cross-dependencies for every build. It comes with a helper script pdebuild-cross-create which calls multistrap (or potentially debootstrap) with a suitable configuration to create a chroot populated with cross-build tools. This makes it trivial to set up and use a cross-build environment.

It is written in perl.

apt-ma-emu

apt-ma-emu emulates the effect of having multiarch support on a system that hasn't got it yet. It works by converting the package files that are downloaded to include -cross packages and -cross dependencies. It uses a blacklist and some heuristics to try and get this uncertain process right.

The major advantage of this method is that the -cross packages actually exist in a packages file so are not all 'locally-generated/obsolete', and upgrades work. See 'The cross-upgrade problem' above for why this is an issue. Feedback on the use of this tool for practical cross-building would be welcome.

The tool is described in this thread: http://lists.debian.org/debian-embedded/2010/01/msg00059.html

Until very recently (maverick and squeeze) it was awkward to use because it needs either a multiarch-ready apt (v0.8) or a patched-and-rebuilt apt. Trying to install apt v0.8 from Debian experimental apt broke other libapt-using packages (aptitude, libept, synaptic etc), and tried to remove your kernel. It should now work fine in either distro, but is not actually uploaded.

Code is available here: http://mentors.debian.net/cgi-bin/sponsor-pkglist?action=details;package=apt-ma-emu

Practice

using dpkg-cross

dpkg-cross works on already-downloaded package files.

dpkg-cross -a armel <package.deb> Converts the armel <package.deb>

dpkg-cross -a armel -X pkg-config <package.deb>

From version 2.6.3 it can correctly deal with Multiarch packages. By default it will leave multiarch packages alone. the -M option will force it to convert them anyway, moving the files to the old /usr/<triplet> locations.

Using xdeb

Xdeb is a high-level tool for cross-building which combines the tasks of apt-cross, bootstrapping dependency-breaking, ordering builds or downloading dependencies, caching build packages, and running dpkg-cross and dpkg-buildpackage to generate cross-deps and run builds. It's major advantage is that it can build a whole set of packages from sources in order to target a new flavour.

See UsingXdeb page for details of usage

Using Pdebuild-cross

Pdebuild-cross is a tool to do pbuilder-style cross-builds in a clean chroot, satisfying dependencies from a repository. It builds one package at a time, like dpkg-buildpackage or pdebuild.

See UsingPdebuildCross for details of usage

Using a target host

See CrossBuildingPerl for an example

CategoryCrossCompile

Platform/DevPlatform/CrossCompile/CrossBuilding (last modified 2012-10-19 18:08:39)