Upstream » History » Revision 420

Revision 419 (Denis 'GNUtoo' Carikli, 07/07/2022 09:51 PM) → Revision 420/424 (Denis 'GNUtoo' Carikli, 07/07/2022 09:52 PM)

h1. Upstream Linux 


 h2. Tracking upstream patches 

 We have a "new issue tracker": for tracking upstream status of various patches. 

 h2. Benefits of using Upstream Linux 

 Currently, Replicant uses device specific Hardware Abstraction Layers, because device manufacturers implemented non-standard kernel interfaces. However, Android works with upstream kernels and supports plug-n-play hardware nowadays, so it makes sense to have generic Hardware Abstraction Layers for the standard interfaces of the Linux kernel (ALSA, V4L2, etc). 

 * It would allow supporting external WiFi dongles such as the ones supported by the ath9k_htc driver and free firmwares without the need for a specific application or configuration. 
 * It would make devices last longer by alleviating the device specific maintenance burden: If LineageOS stops supporting a Replicant supported device, Replicant would need to maintain it by its own. This would require a lot of work, unless the device is already supported the upstream Linux kernel and generic hardware abstractions layers. This would also enable Replicant to support devices that are not currently supported by LineageOS with a lot less work. 
 * It would enable the support for devices that are or will be added to upstream Linux. 

 As GNU/Linux expects standard kernel interfaces, this would also enable to run GNU/Linux out of the box on such devices. 
 This has some interesting outcomes: 
 * The device specific work could be shared between GNU/Linux communities and Replicant communities. This could result in less work to do to support individual devices. Since Android libraries depends on Android's libc, non-standard proprietary libraries might be harder to reuse than the free software implementations, so we might get even more collaboration thanks to that. 
 * It would enable GNU/Linux distributions to more easily support smartphones and tablets, which would hopefully enable FSDG distributions to be able to focus on usability instead of hardware support. This way, if one day Android devices stop using the Linux kernel, stops being free software, or if the code takes directions that are too much problematic, already having GNU/Linux based Android alternatives would reduce the amount of work needed to be able to get again a fully free software distribution for smartphones and tablets. 
 * Older devices with less amount of RAM than Replicant current minimum requirements could be used with GNU/Linux and possibly repurposed for other usages, reducing the amount of electronic devices waste. 

 h2. Requirements 

 * For the other standard interfaces (like ALSA, etc) a device running a upstream Linux Kernel with as few patches as possible is required. 

 h2. Devices 

 It is best to use a device that requires the least amount of work to be functional under Replicant. 
 More precisely we want to minimize: 
 * The work needed to have the device usable with upstream Linux. 
 * The work porting or writing Android hardware abstractions layers. 

 To achieve that we can choose a device that: 
 * requires no or very minimal work to be fully supported by Linux. 
 * have less hardware features (so we don't need to support them in Linux and in the HALs). 
 * is easy to buy, so the work can be shared among multiple people. 
 * doesn't have more freedom flaws than the devices currently supported by Replicant 

 It is also a good idea to keep one image per device at first, as trying to make a single image that 
 would work on all ARM device supported by upstream Linux is complicated: Even GNU/Linux 
 distributions have a hard time doing that for ARM devices. 

 h2. Linux upstream status 

 In some cases, even if the upstream status looks good, nonfree bootloaders can get in the way. We have a list of stock bootloaders incompatible with upstream Linux in this page: [[BootloadersIncompatibleWithLinux]]. 

 See "LinuxSupportedDevices": in the upstreaming sub project for the upstream status for various devices. 

 h1. Bootloaders 

 h2. Bootloader status 

 See the "BootloaderStatus": page on the upstream subproject. 

 h2. See also 

 * [[Google Summer of Code 2018]] 

 h1. GNU/Linux components 

 h2. Modem support 

 |_. Protocols |_. Implmentation |_. comments | 
 | QMI | Android <-> RIL <-> libqmi-ril to be completed <-> libqmi | | 
 |/2. * QMI 
 * AT 
 * Other | Android <-> RIL <-> libraries to be written |  
 | "android_frameworks_opt_telephony_ril_ofono": + ofono + ofono backend (AT, QMI, etc) | * Using ofono would enable us to share more effort with upstream GNU/Linux and support many other protocol like AT for the GTA04 or qmi-ril for the Galaxy SIII 4G (I9305) or the Galaxy Note II 4G (N7105) 
 * According to the "README":, it has already been tested with most of Replicant 6 code but on a smartphones not yet supported by Replicant. Calls, Audio, SMS, etc are known to work. 
 * "": seems to use introspection to automatically generate the API between the Framework Java RIL and itself (which replaces rild) (See the "official documentation": for background information on the Android architecture) 
 * Replicant and oFono based Java RIL "video presentation": | 
 |/5. samsung-ipc | Ofono (rilmodem backend/driver) <-> rild <-> libsamsung-ril <-> libsamsung-ipc | * Might be usable for GNU/Linux distributions with "libhybris": 
 * Could be usable for testing Replicant as ofono could run on the host computer and the rild socket could be exported with adb 
 * Some forks exist: check if they still have interesting patches 
 | Android <-> rild <-> libsamsung-ril <-> libsamsung-ipc | * Currently in use in Replicant 
 * Well integrated with Android 
 * Potentially usable by other distributions 
 * No known way to support different modems protocols in the same Replicant image with that | 
 | Android <-> Ofono <-> libsamsung-ipc | * "An ofono fork with libsamsung-ipc support is available": 
 "Patches": to add that upstream were "refused": because upstream didn't want to make the project become GPLv3 (libsamsung-ipc was GPLv3 at the time) but now libsamsung-ipc has been relicensed to GPLv2+ 
 * Could be used to have generic a Replicant image supporting many devices with very different modems protocols (like libsamsung-ipc or QMI based ones) and have ofono do the modem detection | 
 | FSO <-> libsamsung-ipc | * Probably not easily usable in Replicant 
 * Is FSO still actively maintained? 
 * Was used by SHR and potentially other GNU/Linux distributions supporting the Openmoko GTA04 smartphones | 

 h2. Upstream userspace hardware support libraries 

 |_. Usage |_. Replicant |_. GNU/Linux |_. comments | 
 | Bluetooth stack | BlueDroid | Bluez | | 
 | GPS hardware support | ? | gpsd | | 

 h2. Upstream non-hardware specific userspace 

 |_. Usage |_. Replicant |_. GNU/Linux |_. comments | 
 | Unix command line tools | ? | * Busybox 
 * Coreutils | * Busybox already has Android specific code in it but no  
 * Busybox build is very similar to Linux, and Linux can be built by Android | 

 h2. MTP 

 * look at to see if it can be integrated in Android. It is known to work with the upstream kernel. 
 * Also compare with other implementation, including the Android one. 

 h2. libc 

 |_. Feature |_. Advantages |_. Disadvantages |_. sustainability | 
 | glibc + libhybris | * You just need an to compile GNU/Linux software | * You need to link the Android part that need bionic functions to libhybris | TODO: Evaluate how close to bionic is libhybris | 
 | bionic | Android default | * You spend lot of time trying to run GNU/Linux debug tools like evtest on Android: 
 * Parabola cannot be used on old kenrels (FATAL: kernel too old) 
 * GuiX and libreCMC might not have the package and might need tweaking to be recompiled with an old glibc and kernel headers 
 * TODO: try crosstool-ng 
 * Most of the other GNU/Linux distributions are not FSDG compliant or do not support ARM 
 * The software might not work on Android due to missing bionic functions like versionsort(...) | 

 h2. Other projects interested in using upstream Linux and/or contributing to it 

 h3. PostmarketOS 
 "pmOS wiki: upstream kernel": 
 "pmOS wiki: i9305 with upstream kernel": 

 h3. O-DROID 

 h3. osmocomBB 

 h3. Mali 

 h3. LineageOS 

 h3. oFono 

 h3. Parabola 

 h1. GNU/Linux distributions 

 h2. Halium 

 Some GNU/Linux distributions like Plasma Mobile, LuneOS, Ubuntu Touch use libhybris through the "Halium project": to reuse proprietary Android libraries 

 h2. Mer 

 * Mer uses ofono for handling the modem 
 * For OpenGL it's not clear if they use nonfree drivers. It would be worth looking into it. Their stack seem to be able to use free implementations as well as nonfree implementations. 

 See the "Mer architecture": for more details. 

 h2. PostmarketOS 

 "PostmaketOS": is probably not using libhybris at all and is instead working with upstream and probably doesn't depend on nonfree libraries 

 h2. Replicant strategies 

 Replicant is currently involved with upstream GNU/Linux, but it would probably be a good idea to collaborate more with    distributions that don't depend on libhybris. 

 h1. Android distributions 

 h2. AOSP 

 AOSP doesn't support many devices, but it seem to support some devboards.  

 The advantage is that in general devboards are well supported by upstream kernels. 

 Supported devboards[1]?: 
 | Devboard | Freedom issues | Comments | 
 | Beagleboard-X15 | No free GPU driver | Well supported by GNU/Linux, free bootloader | 
 | Cuttlefish emulator | ? | ? | 
 | Other | ? | ? | 

 The official documentation also "mentions some devboards": . 

 fn1. Some of the devboards in this page are probably supported: 

 h2. LineageOS 

 See "": for more information on LineageOS expectations. 

 Replicant has "a script": that is able to parse the "LineageOS wiki data": . That can be useful to find information on devices supported by LineageOS. 

 The "devices supported by LineageOS 16": have either: 
 * A Qualcomm SOC with an integrated modem inside (MSM*) for many devices 
 * A Qualcomm SOC without a modem inside (APQ*) for many devices 
 * A HiSilicon Kirin 970 SOCs for devices like: 
 ** The "Huawei Honor View 10": 
 ** The "Huawei P20 Pro": 
 * A Samsung Exynos 7580 SOC for the following device 
 ** "Galaxy S5 Neo": (Smartphone): It has shared memory between the modem and the SOC 

 It may be because they rely on nonfree software to support devices, which has not been ported to Android 9, or may be because they need more time to add devices with other SOCs like exynos. 

 h2. CustomROMs 

 CustomROMs seem to be a space for unofficial port of LineageOS. The idea is probably to develop the port until they manage to meet LineageOS's "": . When the requirements are met, they are probably included in LineageOS directly. 

 Interesting ports: 
 * The Galaxy SIII (GT-I9300) is being ported to more recent LineageOS versions. There is "a thread on the collaboration with Replicant": in github. The person/people doing the port seem to alternate between an upstream kernel and a older kernel based on the vendor code depending on the amount of work required to finish the port and/or the difficulties encountered. As we are also using a kernel based on upstream Linux we collaborate on that part by reusing each other work when applicable. 

 h2. DivestOS Mobile 

 This distribution is based on LineageOS. While they seem to reuse nonfree software to make the hardware work, they spent some time cleaning up LineageOS code itself. 

 For instance it contains patches to remove privacy issues and nonfree software included in CyanogenMod or LineageOS. 

 As we need to do that too in Replicant as long as we use LineageOS as base, it would be interesting to collaborate more with DivestOS on that part. 

 Web site: 
 *Onion*: http://6sdlxbqgcxdbkvysoir2qvqqs5ro3fxgyl3phvuphcdyklv7rg57jhid.onion 

 h2. Fairphone 

 As October 2020, they are porting the Fairphone 2 to Android 9. However to do that, they are using 3.4 kernel and not any upstream kernels. 

 In addition, they don't seem very interested in free software bootloaders, and the Fairphone 1 and 2 use Qualcomm System On a Chip. 

 However they are probably interested in free software libraries to make the port to newer Android versions easier. 

 h2. Android x86 

 At the time of writing, the most recent images are RC images for Android 9, but there are branches for Android 10. 

 As they use (mostly?) upstream kernels, they sometimes have interesting code. 

 *Source code*:;pf=android-x86 
 *Manifest*: git:// 


 | Branch | Android version | 
 | q-x86 | 10.0 | 
 | pie-x86 | 9.0 | 
 | oreo-x86 | 8.1 | 
 | nougat-x86 | 7.1 | 
 | marshmallow-x86 | 6.0 | 
 | lollipop-x86 | 5.1 | 
 | kitkat-x86 | 4.4 | 
 | jb-x86 | 4.3 | 
 | ics-x86 | 4.0 | 
 | honeycomb-x86 | 3.2 | 
 | gingerbread-x86 | 2.3 | 
 | froyo-x86 | 2.2 | 
 | eclair-x86 | 2.1 | 
 | donut-x86 | 1.6 | 
 | cupcake-x86 | 1.5 | 

 h2. AOSP and LineageOS 

 | Feature | AOSP | LineageOS | 
 | Code quality | Better than LineageOS | | 
 | Documentation | Better than LineageOS | | 
 | root | Not sure if supported or not | Supported | 
 | Minor release versioning | supported with Git tags | Not supported. 
 For instance frameworks/native has no LineageOS tags (only cm-1x tags) and has no other branch for lineage OS 16.0 than the lineage-16.0 branch, as the other branches name are for other things. This was confirmed after running 'git fetch github' in frameworks/native to fetch the other branches that are not fetched by repo by default. 
 Despite that, some code was even merged in after the lineage-16.0 release: 
 commit 22abc3cf4077644463a2dc1c59a5a74e9518ea16 
 Merge: 9e96d54 8a6b6c3 
 Date:     Sat Jul 13 18:57:45 2019 +0200 

     Merge remote-tracking branch 'aosp/pie-gsi' into lineage-16.0-pie-gsi 
 </pre> | 
 | GUI features for developers 
 * Advanced reboot | ? | Supported | 
 | Integrated kernel builds | Unsupported | Supported | 

 h2. Building a collaboration with other Android distributions 

 Given that: 
 * Replicant doesn't want to support devices that have more freedom issues than the ones currently supported. 
 * Many other Android distributions probably don't have the same goals with the freedom of the devices they support. 
 * Replicant 9 is based on upstream Linux and will have to maintain its userspace libraries. 
 * Some users may still want to support devices that have more freedom issues than the ones supported by Replicant. 

 It could be a good idea to share the maintenance of the code used to make Replicant 9 with other distributions. 

 h2. Design decisions 

 Some decisions have been taken by upstream projects, for instance the the Android Open Source Project (AOSP) is pushing device manufacturers to use "signed bootloaders": and not give the users the ability to replace those bootloaders. Therefor it is best to revisit such decisions and decide whether or not to implement a given feature. 

 h3. Various 

 | Feature | Advantages | Disadvantages | 
 | adb and root at boot | Easier to debug: 
 * We get the logs at boot 
 * May be able to diagnose non-booting devices (partition not mountable, etc) | Way less secure: 
 * Vulnerable to "Juice_Jacking": 
 * Vulnerable to an attacker that just connect an usb cable to a running phone 
 * Breaks the user's expectation of security (lock screen etc) | 

 We can get both: we can enable users to get logs at boot while avoiding any security issues.  

 To do that we can keep disable adb and root disabled at boot, and enable users to add it back by editing the boot.img or recovery images. We have a tutorial for that in the [[AddingADBRootToAnImage]] page and we are working on script to automate it even more. 

 h3. root filesystem related 

 |_. Feature |_. Advantages |_. Disadvantages |_. sustainability | 
 | System as root | * The kernel size can be bigger 
 * You have to hardcode the root partition in the cmdline or use PARTLABEL which might be a security issue: if a microSD has a partition named SYSTEM, the kernel may boot on it instead | having an initramfs adds some flexibility: 
 * Selecting partitions can potentially be more    flexible | | 
 | System as root + dm-verity + "dm-init": | * The kernel size can be bigger 
 * The partition selection is flexible and secure    | | | 
 | read-only /system | * more secure and more easily understandable by users and developers | * need to ship in replicant user-scripts or add them to vendor/ | | 

 h3. Gatekeeper HAL backend 

 *Background information*: Gatekeeper is a daemon that is used to store passwords and other secrets. 

 | Backend used | Advantages | Disadvantages | 
 | simple userspace implementation | * Fast to do 
 * Simple to understand  
 * Good enough for most use cases| | 
 | kernel keyring (man 7 keyrings) | * Secure 
 * The Linux kernel is well known and updated regularly 
 * Some users are already used to the userspace/kernel security model 
 * Probably fun to implement, can learn how to implement Android daemons and how to use the keyring along the way | 
 | Free software Trusted Execution Environment (TEE) | * Android does it | * Require access to TrustZone, which doesn't work for all SOCs 
 * Unfamiliar to users and developers (it's supposed to tick in suspend, knowledge about TrustZone is less spread than Linux, etc) 
 * Probably requires to port a TrustZone OS to every SOC or phone  
 * The Linux kernel is well known and updated regularly | 
 | Proprietary software Trusted Execution Environment (TEE) | None: we really want to get rid of it if possible |  
 Cannot be trusted: 
 * Not free software 
 * Not under the user control | 

 h3. Handling dynamic major/minor /dev/ nodes 

 *Background information*: We can manage to avoid this use case for now. 

 | Backend used | Advantages | Disadvantages | sustainability | 
 | Android default + Upstream Linux | | * does not work by default, probably impossible to make it work as-is | 
 | Android default + Upstream Linux + very dirty userspace scripts | * Minimal changes | * Not robust 
 * Might eat up resources 
 * Already implemented cleanly in mdev | | 
 | Android default + hacked drivers to use fixed major/minor | * Minimal changes in Linux 
 * No changes required on Android side | * Require to change userspace software (libsamsung-ipc) | * Not upstreamable in Linux | 
 | devtmpfs + hacked Android init | * Minimal changes if upstreamed 
 * Init is hard to debug | * Complex to do as debugging at this stage is complicated | * Need to be upstreamed in Android | 
 | Stock Andrdroid init + mdev (busybox) | * Bad integration in the Android build system | * Changes are minimal | * Android build system integration need to be upstreamed or maintained | 

 h3. Firmware loading 

 | Backend used | Advantages | Disadvantages | sustainability | 
 | system/core/init/firmware_handler.cpp | * Stock Android implementation | None | Already upstream in Android | 
 | Something else | None | The stock Android implementation is good enough and firmware loading is trivial anyway | Not upstream in Android | 

 h3. Embeddable web engine 

 The Android API for embedding a web engine is WebView. 

 | Project | Comments | 
 | "AOSP WebView API implemented with Chromium": | * Built from Chromium compiled for WebView 
 * Latest additions to the API seem more and more tied to Chromium (e.g. getWebViewRenderProcess, getWebChromeClient) |  
 | Old AOSP WebView API implemented with WebKit | * Not used anymore, since Android 4.4 
 * Does not implement the current WebView API | 
 | "GeckoView": | * Not the same API as WebView  
 * Could be used to implement WebView 
 * Would need to be modified to expose more features from Gecko (e.g. zoom) while others are straightforward | 
 | "Qt WebEngine": | * Not the same API (C++ instead of Java) 
 * Probably the smallest subset of Chromium available | 
 | "Wine Internet Explorer implementation": | * Uses Gecko under the hood 
 * Might be interesting to look at how they did it | 

 h1. Web engine backend 

 We have an issue with webview (bug #1780): 
 * Using a recent webview based on chromium would create freedom issues as we don't know the license of chromium 
 * Using and old webview based on webkit would bring back many security issues 

 To solve that we need to find a solution that depends on a good upstream as we are not going to write a web browser engine ourselves. 

 h2. Upstream web browser engine comparison 

 This lists upstream projects that are not forks tracking another upstream project. 

 | Project | Comments | 
 | Gecko | * Clear licensing 
 * Friendly upstream: The tor-browser project works with Mozilla to upstream privacy features in Firefox. So we could probably work with them as well too. | 
 | Chromium | * Unclear licensing  
 * Google has goals for Chromium that are directly opposed to our goals (tracking, linked to google) 
 * Probably unfriendly upstream because of the opposed goals | 
 | Webkit | ? | 

 h2. Forked web browser engine comparison 

 TODO: Add various chromium versions here. 

 h2. How and if to implement a webview compatible API 


 h1. Tools and build systems 

 Replicant has an issue with licensing (bug #1973) where we don't know under which license is Replicant. This due to the fact that the Android build system doesn't use a package manager during the build, and so it doesn't have license definition for each repositories. 

 A good way to fix that and also gain the ability to natively build GNU/Linux components like MESA or ofono would be to use a build system that use a package during at least during the build. 

 Some other communities have issues that do or could also benefit from that: 
 * GNU/Linux distributions need to package Android tools which are built with the custom Android build system 
 * Some distributions mixes GNU/Linux and Android 

 h2. Projects 

 |_. Project |_. use case |_. Comments | 
 | Android | - Build images 
 - Build the Android NDK and SDK 
 - Build the Android tools | - Wrapping build systems (like autotools, cmake, etc) is way too primitive: 
 -- In LineageOS (not AOSP) The kernel is wrapped with .mk files, but the downside is that it runs make inside Linux source each time it needs to compile something 
 -- In AOSP there is no infrastructure for building software with other build systems, still mesa is built in it, but not the kernel | 
 | Archlinux | - Build and package Android tools(Fastboot, adb, etc) | - relies on a "fragile script": 
 - The package for Android tools is self contained and doesn't have its dependencies (like liblog, libcutils, etc) splited in other packages |  
 | Debian | | relies on "custom Makefiles": | 
 | [[GuixBuildSystem|Guix]] | - Build android tools  
                              - Build the android ndk | Findings: 
                                                        - Guix uses "android-make-stub": to wrap  
                                                        - We have real packaging of dependencies, however not all dependencies are exported, most are though 
                                                        - The Android build system is wrapper in a ndk-android-build-system function 
                                                        - The package definition needs very light and straigtforward patching. See [[GuixBuildSystem|Guix]] for more details. | 
 | "Robotnix": |    - Build AOSP images with Nix with package definitions | This project is being funded by NLnet. 
                                                                                                                               Once completed it has the potential to replace the Android build system. 
                                                                                                                               Since it uses package definitions it could fix many issues we have in Replicant but we will need to understand how much maintenance from our side it would need (probably not a lot) 
 - It depends on NixOS: @inherit (inputs) nixpkgs nixpkgsUnstable androidPkgs;@ (from "pkgs/default.nix": => We need to also look at how much work it would be to cleanup the nixOS dependencies (if we just have a toolchain we might just need to replace linux-headers by linux-libre-headers and remove what is not used). Alternatively it might be possible to reuse NixOS packages definition and create guix packages from them through guix import. | 
 | "The GNU/Linux distribution of quectel-modems": | Mix an Android kernel with GNU/Linux userspace | | 
 | "AsteroidOS": | Mix an Android kernel with GNU/Linux userspace | | 
 | "openembedded-android": | Build the Android NDK? | strongly outdated version of openembedded | 

 h2. Approaches 

 | Keep the Android build system | + <pre> 
                                   (+) Reduced maintenance cost 
                                   - (-) Hard to integrate software with other build systems (linux, mesa)  
                                   - (-) Hard to audit the licenses                                          licenses</pre>                                    | 
 | Package everything in Guix      | (+) Fixes the licensing situation 
                                   (+) Fixes prebuilt situation 
                                   /!\ Replicant is dead if we can't maintain all the packages 
                                   /!\ Replicant is dead if we depend on non-bootstrapable software        | 

 h1. Coding standards and style: 

 | Language | Stadards               | Used in                    | Tools                                                              | Comments                         | 
 | C          | Kernel coding style    | * Linux kernel 
                                     * libsamsung-ipc           | * Has scripts/ to check that can easily be imported |                                  | 
 | C          | curl[3]                | * curl                     | * Has scripts/ in (lib)curl source code                 |                                  | 
 | C          | GNU coding style       | GNU projects (GRUB, etc) | ?                                                                  |                                  | 
 | C          | pep7[1]                | * Python C code            | ?                                                                  |                                  | 
 | C          | sysexits.h             | * libsamsung-ipc tools     | ?                                                                  | used to standardize exit codes | 
 | Guile      | ?                      | * Guix                     | * Has guix style (and guix lint)                                   |                                  | 

 Having a code style is good because:  
 * Having inconsistent style makes it harder to read the code, spot bugs, etc 
 * Fixing inconsistent style too late results in commits that are extremely hard to review[2]. 
 * Fixing inconsistent style in unrelated commits makes rebasing code painful 

 fn2. Example: