Replicant

h1. Porting Replicant to Android 11

{{>toc}}

Active development has moved towards AOSP 11.

The basic features working on Replicant 9 and 10 have been confirmed to work on Replicant 11, but not everything has been tested yet on either version. The source code and build instruction for both these previous versions have been kept to do regression tracking:

* [[DeprecatedPortingToAndroid9|Porting Replicant to Android 9]]

* [[DeprecatedPortingToAndroid10|Porting Replicant to Android 10]]

h2. Precautions

See [[RunningReplicant11]] before installing Replicant 11 on your device to not break it.

h2. Building Replicant 11

h3. Source code

<pre>

$ repo init -u https://git.replicant.us/replicant-next/manifest.git -b replicant-11-dev

$ repo sync

</pre>

Alternatively a shallow copy of the source tree can be fetched in order to save on disk space:

<pre>

$ repo init -u https://git.replicant.us/replicant-next/manifest.git -b replicant-11-dev --depth=1

$ repo sync -c

</pre>

To unshallow a specific module:

<pre>

$ cd path/to/module

$ git fetch --unshallow <remote>

</pre>

h3. Build dependencies

h4. For Trisquel 8

<pre>

sudo apt-get install bc bison build-essential bsdmainutils ccache curl flex g++-multilib gcc-multilib gettext git gnupg gperf imagemagick lib32ncurses5-dev lib32readline-dev lib32z1-dev liblz4-tool libncurses5-dev libsdl1.2-dev libssl-dev libwxgtk3.0-dev libxml2 libxml2-utils lzop python-mako python3-mako pngcrush rsync schedtool squashfs-tools xsltproc zip zlib1g-dev

sudo apt-get install gcc-5-arm-linux-gnueabi

</pre>

You will also need to install repo as described in the bug #2199.

h4. For Trisquel 9

This section is a work in progress.

<pre>

sudo apt-get install bc bison build-essential bsdmainutils ccache curl flex g++-multilib gcc-multilib gettext git gnupg gperf imagemagick lib32ncurses5-dev lib32readline-dev lib32z1-dev liblz4-tool libncurses5-dev libsdl1.2-dev libssl-dev libwxgtk3.0-dev libxml2 libxml2-utils lzop python-mako python3-mako pngcrush rsync schedtool squashfs-tools xsltproc zip zlib1g-dev

sudo apt-get install gcc-5-arm-linux-gnueabi

</pre>

You will also need to install repo as described in the bug #2199.

h3. Fixing the build environment

h4. Allow system binaries for building

By default, the Android 11 build system can only use the prebuilt binaries it ships.

While having binary toolchains is better for reproducible builds, and that the binaries are free software, this creates a number of issues:

* We need to be able to rebuild the binaries, and so far no one did it yet.

* Binaries are way harder to trust than source code and not everyone trust Google.

As GNU/Linux distribution's tools can be rebuilt and are easier to trust, we are using that for now.

h4. Java heap space

The Java heap size is automatically set according to the available system memory. On machines with 8 GB or less RAM, it is set to a value which is too low, and will result in the following error during the build:

<pre>

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

</pre>

The heap size can be "increased with an envirnoment variable":https://stackoverflow.com/a/60474592/1313087:

<pre>

$ export _JAVA_OPTIONS="-Xmx3g"

</pre>

h4. Reduce parallel jobs to avoid killed processes

Increasing the Java heap space is not enough to get a successful build on machines with 8 GB or less RAM. It is also necessary to reduce the number of parallel jobs, to avoid processes from being killed due to lack of memory. This typically happens during the build of @frameworks/base@ components.

For greater speed, you may let your build run with the defaults, wait for it to fail due to killed processes, and then relunch the build with:

<pre>

$ make -j1

</pre>

By default, "Ninja":https://ninja-build.org/manual.html, the underlaying build system for Android, used when you run @make bacon@, computes the number of parallel jobs according to the number of CPUs on your machine (typically #CPUs + 2 parallel jobs).

h3. Launching the build

You can either launch builds with our wrapper script like that:

<pre>

$ ./vendor/replicant/build.sh i9305

</pre>

It will try to use all the the CPU cores you have available by using one job per processor + one extra job.

You can also manually launch the build with the usual AOSP commands (so you could adjust the number of jobs with -j for instance):

<pre>

$ source build/envsetup.sh

$ lunch lineage_i9305-userdebug

$ make -j2

</pre>

h3. Install the images

h4. From scratch

Due to the bug #2235 we can't use the same commands than before with Replicant 9 and 10:

<pre>

$ cd out/target/product/i9305

$ sudo heimdall flash --BOOT boot.img --USERDATA userdata.img --SYSTEM system.img 

</pre>

Instead we have to install it in a different way:

* First we need to flash the images with @heimdall flash --BOOT boot.img  --RECOVERY recovery.img --SYSTEM system.img@

* Then we need to reboot to recovery during the boot by pressing the volume up, menu and power buttons

* Then we need to select @Wipe data/factory reset@ and then @Factory data reset@

* And finally we can boot to Replicant 11 by selecting @Reboot the system now@

h4. Update previous installation

<pre>

adb remount

adb sync

</pre>

h3. Get adb

As the device IDs are the ones given by the Linux kernel, they are not in the adb udev rules, so for now it requires to run adb as root:

<pre>

$ sudo adb shell

* daemon not running; starting now at tcp:5037

* daemon started successfully

i9305:/ #                                 

$ sudo adb kill-server

$ adb shell

* daemon not running; starting now at tcp:5037

* daemon started successfully

error: no devices/emulators found

</pre>

So make sure to kill the adb-server and run it as root:

<pre>

$ adb kill-server

$ sudo adb shell

* daemon not running; starting now at tcp:5037

* daemon started successfully

i9305:/ # 

</pre>

h3. Boot progress

You can also follow the boot progress with adb:

<pre>

adb logcat

adb logcat -b main

</pre>

Note that the device can go into suspend at any time, so adb might be interrupted. That looks like that:

First you get a shell

<pre>

$ sudo adb shell

i9305:/ #

</pre>

Then the connection is interrupted:

<pre>

$ adb shell

i9305:/ # [randomdev@fullyfreelaptop ]$                                                                                                     

</pre>

The effect with adb logcat is similar.

h3. Getting the latest changes

* The repositories are being constantly modified with @git push --force@ as we are trying things out, and don't want to make the commits history look too dirty, so be sure to backup your local changes.

* Sometimes the manifest repository is also modified with @git push --force@. In that case the following commands will loose all the work you did locally but will make the repository consistent with upstream repositories again:

<pre>

$ rm -rf .repo/manifests .repo/manifests.git .repo/manifest.xml

$ repo init -u https://git.replicant.us/replicant-next/manifest.git -b replicant-11-dev

$ repo sync --force-sync

</pre>

* The following command might also be necessary to make the state consistent with upstream repositories again, when the manifest history wasn't rewritten, but it will also loose all the work you did locally: 

<pre>

$ repo sync --force-sync

</pre>

h3. Known working distributions

|_. Distribution |_. Works |_. Comments |

| Trisquel 8 | Yes | Tested with an lxc chroot:

                     * Builds images fine       |

| Debian 10 | Yes | |

h3. Build VM

If you use Parabola, you may be interested in running Trisquel 8 in LXC.

To do that first debootstrap a Trisquel 8 rootfs. 

Parabola's debootstrap does support Trisquel 8 and its manual has an example on how to do that:

<pre>

$ man debootstrap

[...]

# debootstrap flidas flidas-root http://archive.trisquel.info/trisquel

</pre>

Then you can use virt-manager to setup the LXC instance.

The advantages of this solution are that:

* The LXC guest and host shares their resources (CPU, RAM) with almost no penalty

* Trisquel 8 is not a rolling release distribution

The disadvantage of this solution are that:

* you need to configure the Trisquel 8 LXC instance (vimrc, sshd_config, etc)

* It's more complicated to setup

* The Android build system outputs a warning message about not being able to use namespaces which may become mandatory in newer Android versions

h3. Known issues

h4. Trisquel 8

With Trisquel 8 we have several issues.

First we can't update the repo, else we have:

<pre>

$ repo --force-sync

repo: error: Python 3 version is too old; Please use Python 3.6 or newer.

</pre>

The repo from the package may not work if you have python2 in /usr/bin/python.

To workaround both issues you could do that:

<pre>

$ sudo apt install repo

$ sudo install -m 755 /usr/bin/repo /usr/local/bin/

$ sudo sed 's&^#!/usr/bin/python$&#!/usr/bin/python3&' /usr/local/bin/repo -i

</pre>

h4. Using latest repo (from .repo/repo/repo) ?

If you use your distribution's repo, you might have an old version of repo.

When that happens, repo notify you about it:

<pre>

... A new version of repo (2.8) is available.

... New version is available at: /home/replicant/replicant-11/.repo/repo/repo

... The launcher is run from: /usr/local/bin/repo

!!! The launcher is not writable.  Please talk to your sysadmin or distro

!!! to get an update installed.

</pre>

So in case you have a recent enough python3, you can upgrade.

Trisquel 8 doesn't have a recent enough python3 to do that.

h4. latest repo not working due to python2 in /usr/bin/python

If you have the latest repo and python2 is in /usr/bin/python, you might end with an issue like that:

<pre>

$ repo init [...]

Traceback (most recent call last):

  File "[...]/replicant-11/.repo/repo/main.py", line 56, in <module>

    from subcmds.version import Version

  File "[...]/replicant-11/.repo/repo/subcmds/__init__.py", line 38, in <module>

    ['%s' % name])

  File "[...]/replicant-11/.repo/repo/subcmds/upload.py", line 27, in <module>

    from hooks import RepoHook

  File "[...]/replicant-11/.repo/repo/hooks.py", line 472

    file=sys.stderr)

        ^

SyntaxError: invalid syntax

</pre>

This is because /usr/bin/python is python2 instead of python3.

In that case you can use @python3 $(which repo)@ instead of @repo@) like that:

<pre>

$ python3 $(which repo) init [...]

</pre>

Or you can install repo in /usr/local/bin and modify the @#!/usr/bin/env python@ inside it with the following commands:

<pre>

$ sudo install -m 755 .repo/repo/repo -t /usr/local/bin

$ sudo sed 's&^#!/usr/bin/env python$&#!/usr/bin/env python3&' /usr/local/bin/repo -i

</pre>

Note that the sed command doesn't work for older versions of repo. See the section above about making it work on trisquel8 for one that works on older versions of repo.

h2. Cleanups to be done

* Make adb work as user by using the right USB IDs, and make userspace do the USB setup.

* -Make the kernel not use hardcoded CMDLINE_FORCE- Not possible unless the bootloader is changed or Linux is very heavily patched.

* Make the kernel not use hardcoded partitions if possible (though we use system as root)

* Make a clean Gatekeeper HAL module implementation instead of using the same hack than goldfish

* Look at "init.rc":https://android.googlesource.com/platform/system/core/+/master/init/README.md documentation to see if init.rc can be overriden clearly with the override statement to see if it's possible to keep the serial console patch for -eng

h2. Upstreaming status

* The stock bootloader is incompatible with Linux, see [[BootloadersIncompatibleWithLinux#Devices-with-the-Exynos-4412]] for more information. So we maintain patches to enable the Galaxy SIII, and Galaxy Note II to boot with the stock bootloader. In the long run we need to look into using u-boot in the kernel partition as using u-boot instead of the stock bootloader currently require nonfree and non-redistributable software (BL1).

* For the patches that are not merged yet, see the "issues of the redmine upstreaming sub-project":https://redmine.replicant.us/projects/upstreaming/issues

h2. Graphics status

Progress of the graphics related tasks is tracked at [[GraphicsReplicant10]].

h2. Modem status

*libsamsung-ipc:* 

* Fully tested under GNU/Linux only

* Can initialize completely the modem and receive messages (see the #1954 bug report for the logs)

* Needs more cleanup but there is now a better abstraction

*libsamsung-ril:* 

* Ported to Replicant 9 using a wrapper for the API >=12 in libsamsung-ril source code that needs to be removed

* Tested under Replicant 9 without up to date libsamsung-ipc (no modem init)

* Tested and validated under Replicant 6 (doesn't break telephony)

h3. Modem status TODO

* Implement the missing part to shut down the modem, close the interfaces and such, in order to need to reboot after each test.

* Continue to clean up the libsamsung-ipc and libsamsung-ril patches, test the patches in Replicant 6 when applicable, and merge them in the upstream repositories.

* Convert the firmware loading driver to the upstream API and then adapt libsamsung-ipc for that. This should also benefit other devices like the Galaxy SIII 4G, and the Galaxy Note II 4G which probably don't need much more to get their modem supported by upstream Linux.

* Cleanup and convert the rest of the drivers to look like the ones for the Nokia N900 and adapt the userspace in libsamsung-ipc, and merge libsamsung-ipc support for that once the kernel API is stable.

* Test the code under Replicant 10 when the graphics status will enable to have decent enough speed to do some testing through human interaction.

h2. TODO

First month of full time equivalent work:

|_. Time estimation  |_.Task |_. Comments |

| DONE | -boot a device under AOSP9- | Only boots with graphics, not much more |

| -7h- DONE | -build it under a "FSDG compliant distribution":https://www.gnu.org/distros/free-distros.html like Trisquel8- | -WIP for AOSP, It's difficult to do precise time estimations as it could work out of the box or require one full time month of work depending on how much issues are encountered- 

Builds under Trisquel8|

| -21h- DONE | * -port the changes from AOSP9 to LineageOS 16-

* -cleanup the code-

* -build the kernel from the Android build system-

* -make sure it builds with an FSDG compliant distribution-

* -document the build procedure- |

Status:

* Boots with adb.

* Has ultra slow graphics |

| 14h | find, remove and document proprietary software in LineageOS 16 | |

| 21h | find, remove and document privacy issues in LineageOS 16 | |

| 7h | -Add support for the touch keys driver in the  galaxy-s3 dts- | "applied":https://lists.infradead.org/pipermail/linux-arm-kernel/2019-November/694100.html |

| 7h | -upstream the AAT1290 flash led Linux dts for the galaxy-s3 boards- | "Now in 5.3":https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.3&id=4e4dfcb2a425cccc6dd1fb7d46e060cd57999afc |

| 7h | rebrand LineageOS as Replicant | |

| 70h | -port and- cleanup the the Galaxy SIII (i9300) "modem":https://github.com/fourkbomb/linux/tree/modem Linux driver from 4.16 to 5.0| See the [[PortingToAndroid9/#Modem-status|modem status]] for more details |

|_\3. Total: 147h (~1 month) |

Second month of full time equivalent work:

| * port  libsamsung-ril and libsamsung-ipc to Android 9

* Make the modem driver and libsamsung-ipc work together | 157h | See the [[PortingToAndroid9/#Modem-status|modem status]] for more details |

|_. Total |_. 157h | ~1 month |

Third month of full time equivalent work:

|_. Task |_. Time estimation |_. Comments |

| -port the sensors libraries and other device specific libraries-

Look which sensor libraries can be used | 70h | Already done by the unofficial LineageOS port of the Galaxy SIII (i9300), needs testing |

| add support for Audio with the upstream kernel driver | 70h | Might be way faster, depending on what Android 9 uses

                                                               See also "this bugreport":https://github.com/CustomROMs/android_local_manifests_i9300/issues/1 |

| -add partial (no modem) support for the Galaxy SIII 4G (i9305) and- factorize the code with i9300 | 14h | * The source code on which the work was based changed from AOSP to an unofficial LineageOS port to a port of i9305 support for AOSP by Joonas to the official LineageOS so it's now supported by default

* The work to factorize the code between the i9300 and i9305 still need to be done |

 |

|_. Total |_. 154h | ~1 month |

|_. Task |_. Time estimation |_. Comments |

| create a recovery | 21h | |

| add internal WiFi support and validate the functionality | 6h | |

| add external WiFi dongles support | 20h | External dongles support might be tricky |

| -create new- update the install and upgrade instructions | -35h- | -Our current install instructions don't scale as we have one copy for each device.-

-We also created generic instructions but they tend to be harder to follow[1] than the device specific ones.-

-This will be made in a modular format (for instance in LaTeX) that enables to generate per device install instructions without requiring to have multiples copy of the same text.-

-The instructions will need to be able to be modified and compiled on an FSDG compliant distribution.-

Mostly done: 

* The installation instructions are now generic enough. 

* Some long standing TODO were also done along the way like adding backup instructions for the EFS.

* The current instructions are still for Replicant 6.0 and will need to be updated for Replicant 9.0 |

|_. Task |_. Time estimation |_. Comments |

| Estimate the amount of work to Reduce the attack surface | ? | |

| Estimate the amount of work to add in-system upgrades | ? | |

fn1. The generic instructions were tested at Install parties in Paris 

h3. Devices support:

h4. Easy, because it's similar enough to the Galaxy SIII (I9300)

|_\3. Galaxy Note II (N7100) |

|_. Task |_. Time estimation |_. Comments |

| port the EA8061 LCD Linux driver | 35h | |

| port the S6EVR02 LCD Linux driver | 35h | |

| port the MAX77693 flash led Linux driver | 7h | |

| android: add support for the Note II (N7100) and factorize the code with Galaxy SIII (i9300) and Galaxy SIII 4G (i9305) | 14h | Should be similar to the Galaxy SIII |

| port the sensors libraries and other device specific libraries | 70h | It's difficult to evaluate how much time it could take |

| add support for Audio with the upstream kernel driver | 14h | Should be similar to the Galaxy SIII |

|_\3. Galaxy Note 8.0 (N5100) and 8.0 WiFi (N5110) |

|_. Task |_. Time estimation |_. Comments |

| Evaluate the time required to do the port | 14h | TODO |

h4. Needs more work and unknown upstream Linux status

|_. Device |_. Time estimation |_. Comments |

| Galaxy S II (i9100) | | Linux: devboard dts upstream? unknown status |

| Galaxy Note (N7000) | | unknown Linux upstream status |

| Galaxy Nexus (I9250) | |/2. OMAP4, no dts upstream |

| Galaxy Tab 2 7.0 (P3100), 7.0 WiFi (P3110), 10.1 (P5100), 10.1 WiFi (P5110) | |

| GTA04 >= A4 | | TODO: a RIL needs to be written, userspace GPS support is missing, audio scenarios, etc |

h2. Documentation

h3. Replicant 6.0 changes

See [[Replican6Changes]].

h3. Other rebases

See the [[Samsung-ipc]] page.

h2. Other attempts

* It might be interesting to contact the people doing ports once we have something working well enough.

* It is also interesting to look at other attempts to understand if a given device is powerful enough to run Android 9 and what configuration was used to achieve it.

|_. Device(s) |_. Repository |_. status |_. Comments |

| i9300 | "CustomROMs":https://github.com/CustomROMs/android_local_manifests_i9300 | * "February 8 2020 Pie release":https://forum.xda-developers.com/galaxy-s3/development/rom-lineageos-16-0-t3875899 | |

| i9300 | "Team InFusion":https://github.com/team-infusion-developers/android_local_manifests_i9300 | * "August 20 2019 Pie release":https://forum.xda-developers.com/galaxy-s3/development/rom-optimized-lineageos-16-0-team-t3940142 | Issues: * Uses a Samsung kernel

* Uses too many nonfree libraries

=> Probably nothing we could reuse from its code |

| n7100 | "ComicoTeam":https://github.com/ComicoTeam/android_device_samsung_n7100 | * "January 4 2020 Pie release":https://forum.xda-developers.com/galaxy-note-2/development/rom-lineageos-16-0-20190112-comico-t3889281 | |

| i9100 | "rINanDO":https://github.com/rINanDO/android_device_samsung_galaxys2-common | * "March 20 2020 Pie release":https://forum.xda-developers.com/galaxy-s2/development-derivatives/pie-i9100-t3850588 

* "July 19 2020 Android 10 release":https://forum.xda-developers.com/galaxy-s2/development-derivatives/rom-lineageos-17-0-t4022733 | |

h3. Links for other attempts

* https://www.xda-developers.com/android-pie-android-9-port-custom-roms/

h3. CustomROMs i9300 components

|_. Repository |_. Tree path |_. Dependencies |_. Function |_. Comments |

|/8. https://github.com/CustomROMs/android_hardware_samsung

lineage-16.0 branch | hardware/samsung/macloader | | Loads the MAC Address of the WiFi network interface | Might be useful |

| hardware/samsung/wifiloader | | Loads the wifi kernel module (like modprobe) and setup firmware filesystems permissions | May be useful |

| hardware/samsung/audio | | seems to contains ril related stuff as well | Look if the ril stuff is required, go for standard audio |

| hardware/samsung/lineagehw/hidl/livedisplay | | livedisplay is a feature similar to what redshift does on GNU/Linux | Not sure if it works with mainline |

| hardware/samsung/exynos/multimedia/utils/ | seem meant for audio/video decoding offload | assembly obtimized color conversion and resize code | check assembly code license, not sure if useful |

| all other directories in hardware/samsung/exynos/ | nonfree firmwares, nonfree software?, smdk kernel? | audio/video decoding offload | Avoid using that |

| hardware/samsung/exynos3 |/2. nonfree firmwares?, nonfree software?, smdk kernel? |/2. some light libraries, display stuff (gralloc, etc), 2D acceleration (FIMG), camera (FIMC), 3D acceleration, etc |/2. Avoid using that for now |

| hardware/samsung/exynos4 |

h2. Known error messages that are safe to ignore

* TestHarnessModeService: Failed to start Test Harness Mode; no implementation of PersistentDataBlockManagerInternal was bound

* JniUtils: Could not load native library jni_latinimegoogle

h2. Links 

* "Android build requirements - hardware and software":https://source.android.com/setup/build/requirements

* "Why LineageOS Developers are building Android Go-optimized custom ROMs":https://www.xda-developers.com/android-go-old-android-8-1-oreo/

* "Android Go recommended default values for propreties for optimization":https://android.googlesource.com/platform/build/+/master/target/product/go_defaults_common.mk

* "Team InFusion i9300 optimized system.prop":https://github.com/team-infusion-developers/android_device_samsung_i9300/blob/lineage-16.0/system.prop#L23

* "Hack to fix high CPU usage caused by logd":https://github.com/team-infusion-developers/android_device_samsung_smdk4412-common/commit/c66015514fc9779edfed2665f67e841f3620c71e

* "Use low-end video codecs":https://github.com/team-infusion-developers/android_device_samsung_smdk4412-common/commit/adc919f04b981a4fe005fab2cf443596a22992b2

* "Optimize ActivityManager cached apps":https://github.com/team-infusion-developers/android_device_samsung_smdk4412-common/commit/24bc62970d35eb5a7a257463e7dbfe873ceb5779

* "Use 1Gb Dalvik config":https://github.com/team-infusion-developers/android_device_samsung_smdk4412-common/commit/8a7cebc5af772a2949420ac4a383f5f75236c65c