Project

General

Profile

Toolchain » History » Version 12

Wolfgang Wiedmeyer, 04/27/2017 06:18 PM
add relocation packer

1 1 Wolfgang Wiedmeyer
h1. Toolchain
2
3 6 Wolfgang Wiedmeyer
{{>toc}}
4
5 2 Wolfgang Wiedmeyer
Development efforts to get rid of prebuilt binaries in the Replicant source code are tracked on this page. Prebuilts usually belong to the toolchain, but they can also be found in other parts of the source code. Most of them reside in the @prebuilts@ directory.
6 1 Wolfgang Wiedmeyer
7 4 Wolfgang Wiedmeyer
h2. Goals
8 1 Wolfgang Wiedmeyer
9 4 Wolfgang Wiedmeyer
These binaries are committed to AOSP git repos and in most cases, it is poorly documented, how they can be rebuilt, or the documentation is outdated. Compiler rebuild scripts rely on the prebuilt compilers to build the compilers. It is possible that non-free software was used to build the prebuilts (e.g. Google-internal Java compiler). To keep the build trustworthy, we need to make sure, that as little prebuilts as possible are used.
10 1 Wolfgang Wiedmeyer
11 4 Wolfgang Wiedmeyer
The focus should be on prebuilts that run on the build machine, so basically all prebuilts that belong to the toolchain. We first need to make sure that the toolchain can be trusted by developers and users that build Replicant from source. Then we can focus on binaries that only run on the target devices.
12 1 Wolfgang Wiedmeyer
13 8 Wolfgang Wiedmeyer
Only building an image for a device needs to work for now. Making e.g. the SDK build target work is a lot more challenging as it depends on a lot more prebuilts.
14 4 Wolfgang Wiedmeyer
15
h2. Strategy
16
17 5 Wolfgang Wiedmeyer
Building as much from source as possible is one strategy to achieve this. Using packaged versions of the prebuilts that are provided by GNU/Linux distributions is an alternative way to ensure that the binaries can be trusted. If the prebuilts are available as packages, then these should be preferred before a clean build from source is investigated. A clean build means that no prebuilts from the AOSP or LineageOS source code are required.
18
19 4 Wolfgang Wiedmeyer
Working on getting build tools packaged by distributions should also be a goal.
20
21
h3. Choosing a main target distribution for building Replicant
22 1 Wolfgang Wiedmeyer
23 9 Wolfgang Wiedmeyer
Debian currently provides by far the most packages that offer replacements for prebuilts. A lot of tools were only packaged for Debian Stretch, so they aren't yet available in most Debian-based distributions. The packages are DFSG-compliant and in accordance to the GNU FSDG. Debian only has free software in its main repository, so no non-free software is installed when setting up a Debian build environment for Replicant.
24 1 Wolfgang Wiedmeyer
25 5 Wolfgang Wiedmeyer
Debian has their own team that is dedicated to packaging Android tools, the "Debian Android Tools team":https://wiki.debian.org/AndroidTools. We are cooperating with them.
26 1 Wolfgang Wiedmeyer
27 5 Wolfgang Wiedmeyer
For these reasons, it makes sense to primarily work with Debian Stretch as build system. Requiring certain packages, that are only available in Debian Stretch, breaks the build for other distributions for now. This is a setback we need to accept before packages become available in more distributions. #1787 tries to find a temporary solution to make Replicant buildable on other distributions without relying on untrustworthy prebuilts again.
28
29
h2. Already done
30
31 8 Wolfgang Wiedmeyer
Below is a (likely incomplete) overview of the work that has already been done.
32 5 Wolfgang Wiedmeyer
33 1 Wolfgang Wiedmeyer
h3. Compilers
34
35 9 Wolfgang Wiedmeyer
The gcc-arm-none-eabi toolchain from Debian is used to build kernel images. The Android ARM cross-compiler and the host toolchains (Clang and GCC) are built from source. The host toolchains are built natively and don't rely on Google's modified Ubuntu sysroot.
36
37
The Jack and Jill Java compilers are built from source while prebuilts in the source tree are either built from source or they were replaced with binaries from Debian packages.
38
39
Sources for the compilers are in @toolchain/src@ and the compilers are built in other subdirectories of @toolchain@. Only for Clang and LLVM, sources from the @external@ dir are used.
40
41
The rudimentary script @build-toolchain@ in @vendor/replicant@ takes care of building the compilers and setting up the toolchain.
42
43 12 Wolfgang Wiedmeyer
The build of the relocation packer was integrated in the Android build system.
44
45 9 Wolfgang Wiedmeyer
h3. Individual tools and libraries
46
47 10 Wolfgang Wiedmeyer
Quite a few prebuilt tools and libraries, especially Java libraries, could be replaced with binaries from Debian packages. These include the Manifest Merger, Bison, Yasm, Flex, Proguard, Findbugs and EMMA.
48
49 11 Wolfgang Wiedmeyer
For tools, the path is usually changed from the path to the prebuilt binary to the path of the tool in @/usr@. JAR archives are mostly replaced with symlinks to the files from @/usr/share@ (if they are not built from source). The symlinks are necessary as Android Makefiles expect local paths to prebuilts.
50
51 10 Wolfgang Wiedmeyer
The prebuilt Python interpreter is also not in use anymore.
52 9 Wolfgang Wiedmeyer
53 7 Wolfgang Wiedmeyer
h3. Manifest
54 1 Wolfgang Wiedmeyer
55 9 Wolfgang Wiedmeyer
Repositories with prebuilts, that are not needed to build an image, are not fetched by default. Users need to specify the group the repo belongs to in order to fetch the repo. Groups can be specified by adding @-g GROUP_NAME@ to the @repo init@ command. Some repos were completely removed if they were deemed unnecessary.
56
57 1 Wolfgang Wiedmeyer
h3. $PATH
58
59 9 Wolfgang Wiedmeyer
No prebuilt binary should be in @$PATH@, so one can be sure that no prebuilt binary is run when executing commands in shell that has environment variables from  @envsetup.sh@ or @lunch@. Only @prebuilts/devtools/tools@ is still part of @$PATH@, but the repo is not fetched by default. However, it is not fully clear if prebuilts get copied to @out/host/linux-x86/bin@.
60 1 Wolfgang Wiedmeyer
61
h2. TODO
62 8 Wolfgang Wiedmeyer
63
See "open issues in the build system category":/projects/replicant/issues?set_filter=1&f[]=status_id&op[status_id]=o&f[]=category_id&op[category_id]=%3D&v[category_id][]=153&f[]=&c[]=tracker&c[]=status&c[]=priority&c[]=subject&c[]=assigned_to&c[]=updated_on&c[]=category&c[]=cf_21&group_by=.
64
65
More tasks are listed below.
66 4 Wolfgang Wiedmeyer
67
h3. NDK
68
69
The Android ARM cross-compiler build uses the sysroot from @prebuilts/ndk/current/platforms/android-21/arch-arm@. The compiler needs to be bootstrapped with proper 1st and 2nd stage compilers so it doesn't rely on the sysroot from the NDK.
70
71
Some Makefiles in @build@ reference binaries from @prebuilts/ndk@. It needs to be checked if they are used during an image build and if so whether they can be replaced.
72
73
h3. SDK
74
75 1 Wolfgang Wiedmeyer
There are still prebuilt libraries and target APIs in @prebuilts/sdk@. It needs to be investigated which of the libraries are needed to build an image and if the needed ones are already packaged by Debian. It is probably too much work to replace the different target API jar files, so we have to wait until they are packaged by Debian.
76 5 Wolfgang Wiedmeyer
77
h3. Prebuilts in other places
78
79
Same as above. It needs to be checked if they are used during an image build and if they can be replaced.
80 3 Wolfgang Wiedmeyer
81
h3. @repo@
82 1 Wolfgang Wiedmeyer
83 11 Wolfgang Wiedmeyer
Get the @repo@ tool back into the main repository of Debian. It is currently in the contrib repo due to "bug #855846":https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%23855846. A substantial patch is needed, which should remove the self-updating part of @repo@ or make it optional.