Project

General

Profile

Toolchain

Development efforts to get rid of prebuilt binaries in the Replicant source code are tracked on this page. Prebuilts usually belong to the toolchain, but they can also be found in other parts of the source code. Most of them reside in the prebuilts directory.

Goals

These binaries are committed to AOSP git repos and in most cases, it is poorly documented, how they can be rebuilt, or the documentation is outdated. Compiler rebuild scripts rely on the prebuilt compilers to build the compilers. It is possible that non-free software was used to build the prebuilts (e.g. Google-internal Java compiler). To keep the build trustworthy, we need to make sure, that as little prebuilts as possible are used.

The focus should be on prebuilts that run on the build machine, so basically all prebuilts that belong to the toolchain. We first need to make sure that the toolchain can be trusted by developers and users that build Replicant from source. Then we can focus on binaries that only run on the target devices.

Only building an image for a device needs to work for now. Making e.g. the SDK build target work is a lot more challenging as it depends on a lot more prebuilts.

Strategy

Building as much from source as possible is one strategy to achieve this. Using packaged versions of the prebuilts that are provided by GNU/Linux distributions is an alternative way to ensure that the binaries can be trusted. If the prebuilts are available as packages, then these should be preferred before a clean build from source is investigated. A clean build means that no prebuilts from the AOSP or LineageOS source code are required.

Working on getting build tools packaged by distributions should also be part of the strategy.

Choosing a main target distribution for building Replicant

Debian currently provides by far the most packages that offer replacements for prebuilts. A lot of tools were only packaged for Debian Stretch, so they aren't yet available in most Debian-based distributions. The packages are DFSG-compliant and in accordance to the GNU FSDG. Debian only has free software in its main repository, so no non-free software is installed when setting up a Debian build environment for Replicant.

Debian has their own team that is dedicated to packaging Android tools, the Debian Android Tools team. We are cooperating with them.

For these reasons, it makes sense to primarily work with Debian Stretch as build system. Requiring certain packages, that are only available in Debian Stretch, breaks the build for other distributions for now. This is a setback we need to accept before packages become available in more distributions. #1787 tries to find a temporary solution to make Replicant buildable on other distributions without relying on untrustworthy prebuilts again.

Already done

Below is a (likely incomplete) overview of the work that has already been done.

Compilers

The gcc-arm-none-eabi toolchain from Debian is used to build kernel images. The Android ARM cross-compiler and the host toolchains (Clang and GCC) are built from source. The host toolchains are built natively and don't rely on Google's modified Ubuntu sysroot.

The Jack and Jill Java compilers are built from source while prebuilts in the source tree are either built from source or they were replaced with binaries from Debian packages.

Sources for the compilers are in toolchain/src and the compilers are built in other subdirectories of toolchain. Only for Clang and LLVM, sources from the external dir are used.

The rudimentary script build-toolchain in vendor/replicant takes care of building the compilers and setting up the toolchain.

The build of the relocation packer was integrated in the Android build system.

Individual tools and libraries

Quite a few prebuilt tools and libraries, especially Java libraries, could be replaced with binaries from Debian packages. These include the Manifest Merger, Bison, Yasm, Flex, Proguard, Findbugs, SWIG and EMMA.

For tools, the path is usually changed from the path to the prebuilt binary to the path of the tool in /usr. JAR archives are mostly replaced with symlinks to the files from /usr/share (if they are not built from source). The symlinks are necessary as Android Makefiles expect local paths to prebuilts.

The prebuilt Python interpreter is also not in use anymore.

Manifest

Repositories with prebuilts, that are not needed to build an image, are not fetched by default. Users need to specify the group the repo belongs to in order to fetch the repo. Groups can be specified by adding -g GROUP_NAME to the repo init command. Some repos were completely removed if they were deemed unnecessary.

$PATH

No prebuilt binary should be in $PATH, so one can be sure that no prebuilt binary is run when executing commands in a shell that has environment variables set from envsetup.sh or lunch. Only prebuilts/devtools/tools is still part of $PATH, but the repo is not fetched by default. However, it is not fully clear if prebuilts get copied to out/host/linux-x86/bin.

TODO

See open issues in the build system category.

More tasks are listed below.

NDK

The Android ARM cross-compiler build uses the sysroot from prebuilts/ndk/current/platforms/android-21/arch-arm. The compiler needs to be bootstrapped with proper 1st and 2nd stage compilers so it doesn't rely on the sysroot from the NDK.

Some Makefiles in build reference binaries from prebuilts/ndk. It needs to be checked if they are used during an image build and if so whether they can be replaced.

SDK

There are still prebuilt libraries and target APIs in prebuilts/sdk. It needs to be investigated which of the libraries are needed to build an image and if the needed ones are already packaged by Debian. It is probably too much work to replace the different target API jar files, so we have to wait until they are packaged by Debian.

Prebuilts in other places

Same as above. It needs to be checked if they are used during an image build and if they can be replaced.

repo

Get the repo tool back into the main repository of Debian. It is currently in the contrib repo due to bug #855846. A substantial patch is needed, which should remove the self-updating part of repo or make it optional.