Graphics acceleration¶
Currently, all supported devices on Replicant lack a free software driver for their GPU. This means that OpenGL ES (GLES) rendering must be done on the CPU (software rendering). The current approach to software rendering on Replicant 6 is based on libAGL, an optimized GLES 1.x implementation that uses libpixelflinger software renderer. Development on both these libraries ceased in 2013 and no work was done to support newer GLES versions.The major consequences of this are that:
- Critical applications like web browsers crash due to lack of GLES 2.0 (#705). Replicant currently uses an out-dated browser that has many security flaws.
- Replicant relies on patches to the Android framework to make things like the camera application work.
- The rendering speed has degraded over the newer Android versions, like Android 6. Even applications that do not crash become difficult to use due to the huge rendering delays.
This task aims to fix all these severe issues by putting together a new graphics stack for the Android 9 port. This new graphics stack must be compatible with Android 9 Hardware Abstraction Layers (HAL) and provide at least GLES 2.0 rendering. It should be flexible enough to do software rendering with Mesa or SwiftShader (Google's current software renderer), and also GPU rendering on devices where a free GPU driver is available.
Joonas Kylmälä has done some introductory work on the Android 9 port for the i9305, and found a way to have it rendering to the screen. He put together a graphics stack composed of: gbm_gralloc (Gralloc HAL) + drm_hwcomposer (Hardware Composer HAL) + Mesa with kms_swrast driver with softpipe backend (CPU software renderer for GLES). This will be the starting point for this task.
Joonas' prototype shows that Replicant can use gbm_gralloc, an existing graphics memory allocator (gralloc) HAL maintained(ish) by Android-x86, in conjunction with drm-hwcomposer, a libre implementation of Android's Hardware Composer HAL based on Linux's DRM, to achieve software rendering with Mesa. This solution avoids the need to write a custom gralloc for Replicant, and takes advantage of the hardware acceleration for composition provided by drm-hwcomposer on devices with a free software DRM driver (e.g. exynos-based smartphones and tablets).
Much more testing is needed to confirm that gbm_gralloc can be the definitive gralloc HAL on Replicant 9. It still wasn't tried with SwiftShader, or even Mesa's llvmpipe backend, which is a must since the softpipe backend is too slow to be usable.
Joonas' tests showed that some other components will need our attention, mostly the drm/exynos driver. In order to make drm/exynos work with gbm_gralloc and drm-hwcomposer Joonas had to disable DRM-Auth and hack some missing pixel formats into it by using the default pixel format for everything. Besides drm/exynos, we will also have to make the graphics stack work with the virtual GEM (vGEM) driver, in order to support devices that lack a real drm driver.
The major and first goal of this task is thus to build upon Joonas's prototype and put together a stable and fully free graphics stack, compatible with GLES 2.0, that does software rendering through Mesa's llvmpipe with a decent performance.
Hardware requirements: A computer that is able to build Replicant. A Samsung Galaxy S3 or S3 4G to run the current Replicant 9 port.
Difficulty: Medium / Hard
Requirements/Prerequisites: Knowledge of C++, kernel interfaces knowledge or the ability to learn them
Expected outcomes:- Graphics stack with decent performance on software rendering
- Working GLES 2.0 implementation
- Fast enough graphics
- F-Droid applications not crashing anymore because of GLES.
Time estimation:
Step | man-hours |
---|---|
Set up the development environment, including the current Replicant 9 port on the test device. | 24 |
Read AOSP documentation and understand all details of the graphics stack. | 16 |
Adapt the build files to use Mesa's llvmpipe backend instead of softpipe. Fix potential LLVM version incompatibilities between Mesa and Android. | 40 |
Properly implement the missing pixel formats in drm/exynos and try to have it merged into upstream. | 72 |
Find a proper way to use DRM-Master and DRM-Auth with gbm_gralloc and drm-hwcomposer. | 40 |
Create test scenarios and check if the graphics stack works as expected. Consider alternative grallocs if necessary. | 40 |
Make the graphics stack work with vGEM driver besides drm/exynos. | 40 |
Document the design decisions. | 16 |
TOTAL | 288 |
Subtasks¶
The following sub-tasks could also be worked on after finishing writing the gralloc:SwiftShader¶
SwiftShader is Google's current software renderer that is capable of GLES 2.0 and is now under work to support Vulkan.
Mesa is the preferred renderer on Replicant for several reasons such as its support for both software and hardware (GPU) rendering, and its big community, with hundreds of active contributors. However, Mesa lacks a Vulkan software renderer. With Vulkan soon becoming a requirement for new Android versions, we must make sure that Replicant's graphics stack can use SwiftShader in order to become futureproof. Furthermore, SwiftShader was built with performance in mind, specially for ARM CPUs, and may bring speed improvements on some devices.
The goal of this sub-task is thus to create a compile-time or run-time option that allows using Replicant 9 with SwiftShader as it's software renderer instead of Mesa.
Hardware requirements: A computer that is able to build Replicant. A smartphone or tablet that is supported by Replicant to be able to test the result.
Difficulty: Medium
Requirements/Prerequisites: Knowledge of C++, Makefiles and git. Android's graphics stack knowledge or the ability to learn them.
Expected outcomes:- SwiftShader running on Replicant.
- Working Vulkan implementation.
Time estimation: 40 man-hours.
llvmpipe optimizations¶
Mesa is a highly versatile library that can be extended with device drivers to allow it to be used in different environments ranging from software emulation to complete hardware acceleration. One such driver is the Gallium llvmpipe driver, which is a software rasterizer that uses LLVM to do runtime code generation. It only needs a CPU to run graphics computations and thus brings full GLES support to all Replicant devices.
llvmpipe has been integrated in Replicant 6 but it's not activated by default yet as it is very slow. It is also not fully complete.
To fix that, llvmpipe and/or the integration of it in Replicant should be optimized. We should first start by configuring llvmpipe and/or Mesa to not implement very expensive OpenGL operations. If that's not sufficient, or if that breaks application compatibility, various software or hardware features (ARM NEON, hardware 2D acceleration, etc) could be used to improve the speed.
Considerable speed improvements may be achieved with a fine-tuned emulation for division instructions. The ARM cores on many Replicant devices do not have hardware support for the SDIV/UDIV instructions. We should profile some apps and check whether GLES functions requiring divisions are to blame for the poor performance.
Hardware requirements : A computer that is able to build Replicant. A smartphone or tablet that is supported by Replicant to be able to test the result.
Difficulty: Medium / Hard (depending on the amount of optimizations required)
Requirements/Prerequisites: See with Mesa project
Expected outcomes: faster llvmpipe on ARM devices, able to run apps such as Fennec F-Droid (Firefox).
Time estimation:
Step | man-hours |
---|---|
Setup a testing and benchmarking environment | 40 |
Disable expensive OpenGL operations. Check speedup and stability. | 24 |
Recap matrix operations (Linear Algebra) and study ARM NEON. | 48 |
Do a profiling of several apps to find the most used GLES operations. | 32 |
Use Ne10 library or Neon Intrinsics for the most used GLES operations. | 80 |
Fix bugs, re-write the code where needed, get it stable. | 80 |
TOTAL | 304 |
Lima driver¶
Lima is a free software Mesa driver for ARM Mali-4xx (Utgard) GPUs. These GPUs are present in several Replicant supported devices such as Galaxy S2, S3, S3 4G, Note and Note 2.
Lima aims to full GLES support but it is still in development. However the current implementation status already allows the hardware acceleration of several tasks. GPU-based hardware acceleration is faster and less power hungry than software rendering, both by several orders of magnitude. It would allow Replicant devices to run applications with a performance close to that of non-free devices.
Hardware requirements : A computer that is able to build Replicant. A Replicant device with a Mali-4xx GPU that can run mainline Linux (e.g. Galaxy S3 or Note 2).
Difficulty: Medium
Requirements/Prerequisites: See with Lima project
Expected outcomes: Lima driver being used for GLES rendering on a supported device.
Step | man-hours |
---|---|
Rebase Lima's Linux kernel DRM driver on top of forkbomb's Midas on Mainline kernel. | 80 |
Replace mainline Mesa for Lima's Mesa (with their driver). | 16 |
Build and test thoroughly with synthetic and real applications. | 40 |
Create a fallback mechanism that uses the software renderer for GLES functions not yet implemented in Lima. | 100 |
TOTAL | 236 |
Software Bill of Materials¶
- Replicant 9
- Mesa (in particular: Gallium llvmpipe driver)
- gbm_gralloc
- drm-hwcomposer
- Linux kernel (in particular: drm/exynos driver and vGEM driver)
- SwiftShader
- Ne10 library
- Neon Intrinsics
- Lima
"Graphics acceleration on Replicant" nlnet Grant application¶
Project name | Graphics acceleration on Replicant |
---|---|
Website / wiki | https://redmine.replicant.us/projects/replicant/wiki/Tasks_funding#Graphics-acceleration |
Abstract: Can you explain the whole project and its expected outcome(s) in 1200 characters
Replicant is a fully free software Android distribution which is approved by the FSF. All supported devices on Replicant currently lack a free software driver for their GPU. As such, OpenGL ES (GLES) rendering must be done on the CPU through software rendering (SR). Replicant's current renderer is both incomplete and slow. It causes essential apps like web browsers to crash due to lack of GLES 2.0, and many other apps run too slow to be usable. This project aims to fix this by complementing Android's 9 graphics stack. Adding a few missing components will created of a fully-free, fast and compliant graphics stack. First we will write a gralloc (graphics memory allocator) tailored for SR that is compatible with drm-hwcomposer (a libre implementation of Android's Hardware Composer HAL). This gralloc enables drm-hwcomposer to work with SurfaceFlinger and SwiftShader, creating a stack capable of GLES 2.0 on the CPU of all Replicant devices. Afterwards we will integrate and optimize Mesa's llvmpipe SR, which offers better community support than SwiftShader. As last step we will add support for the Lima driver, which will bring an even faster GPU-backed GLES to at least 5 devices.
Have you been involved with projects or organizations relevant to this project before? And if so, can you tell us a bit about your contributions? |
SEE TEMPLATE |
---|
Requested Amount (Between 5000 and 50000 Euros) | 50000 Euros |
---|---|
Does the project have other funding sources, both past and present? | SEE TEMPLATE |
Explain what the requested budget will be used for?
The budget will only be used to fund this project through contract work. We estimate that this project should take 868 man-hours to reach full completion, with 632 man-hours being enough to reach all software rendering goals, leaving only the GPU rendering to be done. A detailed run-down of this estimate is available at https://redmine.replicant.us/projects/replicant/wiki/Tasks_funding#Graphics-acceleration So far we have a team of two people interested on working on this project (the two authors and submitters of this application). Both can commit to the project on a part-time regime (17.5 hours per week), which means that the project should be fully completed in about 6 months. We will make sure that everybody has a chance to apply for doing contract work. If we take the cost of a freelance developer in the USA (75 to 150 USD per hour) as a basis, to enable people living in Europe and the USA to apply, we can fund between 380 and 760 man-hours with the 50000 EUR budget. This should be enough to cover all work on software rendering plus the initial work on GPU rendering. As happens on all software projects, getting a precise time/effort evaluation is a difficult endeavour, specially when dealing with a project that is heavy on research such as this one. If the software rendering goals are not reached when the 50000 EUR budget runs out, or if the Replicant project deems it necessary to have GPU rendering, it will use its existing funds to pay for contract work if no volunteers are found to finish the project. The Replicant project will also make sure that the people working on this project have the necessary hardware to do it, for instance by shipping or reimbursing the purchase of a compatible smartphone with the Replicant project funds.
Compare your own project with existing or historical efforts.
Past Replicant versions have relied on patches to the Android framework to make software rendering work. These patches were quite specific for Replicant and had no use elsewhere. This made them unfit for upstreaming or sharing with any other project. Android's Project Treble new graphics stack allows us to follow a different approach this time. Instead of patching the Android framework, we will implement one of the well defined Android HALs (Hardware Abstraction Layer): the gralloc HAL. The end result will be a software library that can prove to be useful on several projects besides Replicant (e.g. Android-x86 project) and thus fit for upstreaming. Furthermore, past Replicant versions relied on Google's software renderers (ligAGL and libpixelflinger) for OpenGL ES support. As quite a few other Google's open-source projects, these two had no community behind them and got stalled as soon as Google deprecated them. This time will we take a different approach. Although our first graphics stack will rely on Google's SwiftShader renderer, we will then move our efforts into Mesa. Mesa is a big community project, with hundreds of active contributors and great community support. It includes the llvmpipe software renderer along with new drivers in development for GPUs present on current and future Replicant devices. Mesa should provide a stable and maintained platform for years to come.
What are significant technical challenges you expect to solve during the project, if any?
We expect to solve significant technical challenges during this project: 1. Implementation of the first Android gralloc library compatible with software rendering. 2. Development of free-software benchmarks for OpenGL ES on Android, used to test our optimizations to llvmpipe. 3. Optimization of llvmpipe by at least one order of magnitude. 4. Running an exynos based smartphone with fully free-software GPU graphics acceleration.
Describe the ecosystem of the project, and how you will engage with relevant actors and promote the outcomes?
This project will re-use code from several projects such as Android, drm-hwcomposer, Mesa and Lima driver. Whenever possible we will foster collaboration with these projects and submit our changes upstream. The Replicant project contributors and the FSF will supervise contractors to do the work. A blog post will announce that the Replicant project has got some funding for this specific task, and that it is looking for a contractor to work on it. This is to make sure that everyone has equal chances in the application process. Then the most suited contractor will be selected. Only contractors that already have worked on similar tasks as part of free and open source software projects will be chosen. This way we can look at their existing contributions and make sure that they are able to do the task before engaging with them. The Replicant project will also make sure that the contractor has or gets the hardware required to work on the task, before starting to work on it.
Attachments | SEE TEMPLATE |
---|
Updated by Denis 'GNUtoo' Carikli about 2 years ago · 1 revisions