Project

General

Profile

Actions

Developer guide

Prerequisites

Developing on Replicant isn't much harder than developing on any other free software project as it doesn't require specific knowledge. In fact, you'll probably learn a lot along the way regarding how hardware works, how the Android system is composed, how the kernel works, etc, but you don't need to know all of this to start. However a basic set of skills is required, among which:
  • C language programming skills and the ability to understand other languages such as C++, Java and shell scripts.
  • Makefile skills (no need to know every detail about it, a general idea of how Makefiles work is enough). If you don't know how Makefiles work, Wikipedia has an good article on make that has some examples Makefiles.
  • Git skills (basically, how to commit changes, send them to our repos, dealing with branches without making a mess, etc). If you don't know git yet, you can find some documentation about Git at: https://git-scm.com/documentation to learn it.

If you think you can cope with the requirements, then developing on Replicant should cause you no particular issue.

The porting guides provide instructions for porting a new device to Replicant and also offer some tips for developing on Replicant.

Have a look at the Tasks page and feel free to ask around for help to get started.

Notes on writing free software replacements

Writing free software replacements for non-free components may require more skills depending on what you're trying to achieve, though there may be people with the adequate knowledge to help you and from whom you will likely learn a lot.

Code hosting

Replicant's source code is hosted at git.replicant.us. If you plan to contribute to Replicant, you are welcome to host your Replicant-related projects there under your own username. You only need to contact one of Replicant's developers and ask for an account. Please include the name, username and email address that should be used to create your account. Your repos will then show up on the contributor repos page.

Requirements for submitting patches

There are two ways to get your patches included:
  • You can either send them directly to the mailing list
  • You can also open an issue on the issue tracker and attach the patches to the issue. At some point the patch will need to be sent to the mailing list by someone.

Replicant developers will then review your changes on the mailing list.

There are many ways to send patches to the mailing list
  • You can simply attach the patch to a mail
  • If you like the command line, git has a tool called git send-email to send patches directly from it . If it's too much hassle for you to set up git send-email, sending the patches with your favorite mail client should be fine, too.
  • If you want to send the patch from your command line to your mail client, git's git imap-send can do that
As there are many git repositories in Replicant, it's also best if you indicate in one way or in another to which repository the patch applies, and for which Replicant version the patch is. This can also be done in many ways:
  • If you attach your patch in a mail you could simply mention it in the mail.
  • If you use git-format-patch and send your patch through the command line you could use the subject prefix in git-format-patch to add the required information. For instance when sending a third version of a patch for vendor_replicant only for Replicant 6, it could look like that:
    git format-patch -1 --subject-prefix="PATCH][v3][Replicant 6.0][vendor_replicant" 
    

    The mail would then have a subject that looks like that:
    Subject: [PATCH][v3][Replicant 6.0][vendor_replicant] Add build script to simplify the build procedure.
    

How to make patches

Creating and sending patches can be hard the first time.

While there is a recording of talk on the topic on how to do it right, for Replicant you don't need to do it right the first time.

When you have done some modification that you want to be integrated in Replicant, if you're not confident enough with git, make a copy of the repository with the changes in some safe place.

We will take a real example on how to do a modification. For that we will use the Replicant www.replicant.us git repository.

The replicant.us website source code and content is in git. It contains the following:

Let's suppose that a new Replicant 6.0 release is out but that everybody forgot to send a patch to the website and that the website has the following:

Latest images: Replicant 6.0 0003. Replicant supports up to 13 different devices!

and that instead we want to have the following:

Latest images: Replicant 6.0 0004. Replicant supports up to 13 different devices!

So the first thing to do would be to verify both on the bug tracker and on the mailing list that there isn't already a patch for that. If there isn't any we can safely go on.

To do the patch, we need to get the source code. This can be done with the following command:

git clone https://git.replicant.us/infrastructure/www.replicant.us

We then go into the website directory that has all the source code:

cd www.replicant.us

We then look where the text we want to modify is:

git grep "Replicant 6.0 0003." 

It can potentially return many lines, but it will have somewhere a line that looks like this one:

index.php:      <div class="alert alert-success" role="alert">Latest images: <strong>Replicant 6.0 0003</strong>. Replicant supports up to <strong>13</strong> different devices!</div>

We can open this index.php file with a text editor and change "Replicant 6.0 0003" to "Replicant 6.0 0004".

We can see that git know that we modified the index.php file:

git diff

It will then have something like that:

diff --git a/index.php b/index.php
index 080510a..fd6a8c9 100644
--- a/index.php
+++ b/index.php
@@ -2,7 +2,7 @@
 <?php include_once("include/autoloader.php"); ?>

 <div class="container" role="main">
-       <div class="alert alert-success" role="alert">Latest images: <strong>Replicant 6.0 0003</strong>. Replicant supports up to <strong>13</strong> different devices!</div>
+       <div class="alert alert-success" role="alert">Latest images: <strong>Replicant 6.0 0004</strong>. Replicant supports up to <strong>13</strong> different devices!</div>
        <div class="row">
                <div class="col-md-8">
                        <div class="panel panel-default">

However even if git know about it, we still need to make it record the modification in what is called a commit.

A commit contains a record of what has been changed, along with an explanation of why the change was necessary.

To create a commit, the 'git commit' command can be used, however since we're going to write an explanation of why the change was necessary we will first need to tell git which text editor to use for that.

To do that you can do something like that in the command line:

export GIT_EDITOR=your-favorite-editor

For instance if you like gedit, you can do something like that to make git use gedit in the current shell:

export GIT_EDITOR=gedit

We also need to give git a valid email and a name. Both will appear in the commit.

Else git will refuse to create the commit message and will output an error that looks like this one:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com" 
  git config --global user.name "Your Name" 

to set your account's default identity.
Omit --global to set the identity only in this repository.

So we then need to run something like that:

  git config --global user.email "you@example.com" 
  git config --global user.name "Your Name" 

Then in the same shell you can run the following command:

git commit -s

It will then open the text editor you choose and let you write text that describes why the change was necessary.

The convention is to have a first line that is not too long which summarize why that change is needed.

Then you can write longer text to describe the change details or have more in depth argumentation of why the change is needed.

Since here Replicant releases have already been made in the past, it's very likely that similar changes have already been made before.

We will take advantage of that and look what message the previous contributor wrote.

To do that, we can open a new shell, and go in the www.replicant.us directory and run the 'git log' command in there:

git log

This will show many commits. You can navigate with the keyboard arrows and quit by pressing the 'q' key.

Among the many commits you can see this one:

commit 526edccd8d688544602ae3da1c4d9c5ffdc058ca
Author: Denis 'GNUtoo' Carikli <GNUtoo@no log.org>
Date:   Thu Dec 14 12:37:21 2017 +0100

    index: Replicant 6.0 0003 images have been released

    Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@no log.org>

The sumarry is this line:

index: Replicant 6.0 0003 images have been released

And it doesn't contain a more in depth explanation because it was not deemed necessary.

The line with "Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@no log.org>" has been automatically generated by git when using '-s' in the 'git commit -s' command.

It means that you certify that you have the (legal) right to publish the patch.

For more details on it, you can see the "Developer's Certificate of Origin" in the Linux kernel documentation on submirting patches

So you can write something like "index: Replicant 6.0 0004 images have been released" to tell that the change is needed because new images have been released.

After saving and closing your text editor, git has now a new commit.

You can now generate a file from it with

git format-patch -1

The easiest way is then to write an email to the mailing list and attaching the patch in it.

As Replicant has many many git repository it's also good not to forget to mention for which repository is the patch. Here the patch is for the www.replicant.us git repository.

An email like that would be good enough for that:

Hi,

I've here's a patch that I attached for the www.replicant.us repository.

As there are sometimes different versions of Replicant being worked on in parallel it's also a good idea to also mention the version it is meant for, when it is relevant.

For instance here as there isn't one website per Replicant version it's not relevant here.

What happens after sending the patch?

Once the patch has been sent you will need to wait for people to review it. You can also try to find people on IRC that are willing to review your patch to speed things up.

As Replicant contributors are sometime very busy it can take some time.

If you get no response in one week, you can try to respond to your patch asking again people to review it.

If you get some response, you will typically get some comments on the patch.

This usually mean that your patch is good and that people are interested in it, but that you still need to fixes some things before it can be integrated in Replicant.

Once you fixed what needed to be fixed, you then need to send a second version of the patch with the fixes inside.

When doing that it's best to try to indicate in some way that the patch you send is the second version.

You could for instance mention it in the mail subject for instance.

The patch will then need to be reviewed again, and if everything is good it will be merged. If not you will probably get some more comments that you need to address in a third version of the patch.

Pushing patches

If you already have the ability to push patches into the main Replicant repositories, you still need to send your patches to the mailing list if they are to be applied on a Replicant version that is currently supported, or if they apply to the Replicant website.

You will then need to wait either for:
  • One other developer to review the patch and add per's Acknowledgement.
  • One week if no other developer did review your patches.

Writing free software replacements

Here are some tips that may help you achieving a free software replacement for a specific component (some may be more or less relevant regarding the nature of what the component does):
  • Look for interested people from other projects: LineageOS people are constantly fighting with non-free blobs and are sometimes happy to help replacing one.
  • Try to make the non-free binary as verbose as possible by enabling all the possible debug options on the config file or such.
  • You can use string to try to find out the command line debug options, list of commands, etc.
  • Run the program in strace and analyze the trace to understand what the program does.
  • Add verbose debug prints in the concerned kernel driver (with printk and show them via the dmesg tool).
  • Read the corresponding kernel driver: you can sometimes learn a lot by reading comments or headers.
  • Collect data out of the kernel driver (via debug prints) and out of the non-free binary (via debug prints on the upper-layer).
  • If there is a mathematical algorithm involved, force the values returned by the kernel to the non-free binary and analyze how it reacts, for instance with spreadsheet software.
  • If you're directly dealing with a hardware component, try to find a datasheet for the chip, it may hold precious details. In the best case, you may also be able to find a reference software implementation!
  • Make sure that the methods you use are legal where you live, and that the Replicant project is able to legally redistribute your work. In doubt, you could ask the Replicant developers in the mailing list and/or at the private contact address. The Replicant project has also access to legal advise through the FSF.

Upstreaming work

It is generally a good idea to send some changes back to upstream, assuming that they will benefit from it as well.

When it is about the replacement of a non-free component present in the upstream systems, make sure that your replacement is reliable and complete.
Contact the interested developers on the upstream projects before attempting to send your replacement.

LineageOS

The LineageOS project has a checklist of requirements for adding support for a device.

As the LineageOS developers have to make sure that the checklist requirements are met, make sure that your replacement is complete enough to either meet the checklist requirements, or that the LineageOS developers are interested in working together with you to make sure that the free software replacement meets such requirements.

The LineageOS team uses Gerrit to manage patch submissions. The process to get your patch included in LineageOS repos is explained on their wiki: Gerrit

You can also push directly using git using the following scheme (untested):

git push ssh://<sshusername>@review.lineageos.org:29418/LineageOS/<projectname> HEAD:refs/for/<branchname>

AOSP

The Android Open Source Project uses Gerrit to manage patch submissions. Some information about submitting patches to AOSP is available: https://source.android.com/source/submit-patches.html

You can push to AOSP's review using:

git push https://android-review.googlesource.com/platform/system/core HEAD:refs/for/master

Commonly-used terminology

In order to keep everything clear and consistent, we use the following terms with a precise meaning in mind:
  • Driver: Software that is part of the kernel (builtin or as a module) and deals with communicating with the hardware
  • Hardware Abstraction Layer (HAL): Software that runs in user-space and deals with communicating with the hardware (usually through a kernel driver)
  • module: Android HALs are also often called modules, so we may referrer to e.g. the audio HAL as the audio module
  • blob: Proprietary HAL
  • firmware: Software that does not run on the main processor (the CPU) but rather in a separate chip (e.g. the Wi-Fi firmwares runs on the Wi-Fi chip)

Wiki guidelines

In order the keep the wiki simple and consistent, a few guidelines must be followed when editing.

Regarding the content:
  • Only Replicant-specific topics should be covered by the wiki: there is no point in writing usage guides for generic Android aspects, such as the user interface.
  • The content on each page should only be relevant to the latest Replicant version: make sure to update the content with newer Replicant versions.
  • Substantial changes must be discussed before modifying the wiki.
  • A comment should be added in the comment field at the bottom that shortly describes the change.
  • Make use of the wiki Textile formatting possibilities: quick reference and detailed syntax overview
    By using [[Index]], it's possible to link to the start page of the wiki.
Regarding the writing style:
  • Every page in the wiki has to be written in correct English, we do not aim to provide information in any other language.
  • Addressing readers directly should be avoided when possible: Instead, what is described should always be the subject of sentences.
  • Links to pages should be incorporated in text (Instructions to install the tools shouldn't be: Instructions to install the tools: ToolsInstallation).
Regarding the naming of pages: Regarding the naming of devices:
  • Devices should be named after their model number and codename, without mention of the manufacturer.
  • Common device naming conventions should be followed consistently (the Galaxy S 3 (I9300) is not to be called Samsung S3 GT-I9300 or Galaxy S III).

Repositories

When working with Replicant repos, make sure to avoid breaking things. For instance, if you push a commit introducing a compilation error, it will break the whole build process.
It is better to create separate branches (that are not used by the official manifest branches) when your work is still in progress.
Creating branches that add debug infos on a particular topic is usually a good idea since it will save you time next time you want to debug the same component.

See SourceCodeRepositories for more details about how to create or mirror repositories on Replicant server.

When creating a repository

In order to keep repo naming consistent, please name repositories by their name on the tree, replacing the / by _.
For instance, when forking the LineageOS repo: android_device_samsung_crespo, rename it to device_samsung_crespo on the Replicant repos.
This creates a more consistent way of naming repositories and makes it easier when pushing: just look at the location in the source tree and replace / by _.

Renaming a repository

You need to ask someone with SSH access to git.replicant.us to do that.

Creating a symlink has several side effects:
  • It looks less nice with ls
  • It also look very confusing in git.replicant.us cgit web interface as the repositories are there twice

So instead it's better to edit the /etc/apache2/sites-enabled/git.replicant.us.conf configuration file.

For instance if you want to rename user-scripts.git in vendor_replicant-scripts.git (because some of the scripts will start being shipped on the devices), you can run mv user-scripts.git vendor_replicant-scripts.git and then add the following line in the git.replicant.us.conf apache configuration file:

Redirect /replicant/user-scripts.git /replicant/vendor_replicant-scripts.git

When creating a branch

Official Replicant branches are named the following way:
  • The replicant- prefix
  • The Replicant version

Such as: replicant-2.3 This should be used on the projects repositories as well as the manifest repository.
Any other branch should be considered as Work In Progress (WIP) and thus not be part of any official branch of the manifest.

There is although one exception, with the master branch, that can be used by any project and be in any manifest given that the code held in the master branch will work on any Replicant version.

New images release

  1. Modify the changelog in the vendor files:
    cd path/to/replicant-6.0/vendor/replicant
    edit CHANGELOG.mkdn
    git add CHANGELOG.mkdn
    git commit -sS -m "Replicant 6.0 0001 images release" 
    git push git@git.replicant.us:replicant/vendor_replicant.git replicant-6.0
    
  2. Increment the release in the release scripts:
    cd path/to/release-scripts
    edit releasevars.sh
    git add releasevars.sh
    git commit -sS -m "Replicant 6.0 0001 images release" 
    git push git@git.replicant.us:replicant/release-scripts.git replicant-6.0
    
  3. Tag all the repositories with the release tag script:
    path/to/release-scripts/releasetag.sh path/to/replicant-6.0
    
  4. In the manifest repo, merge the replicant-6.0-dev branch into the replicant-6.0 branch and increment the release in the manifest:
    cd path/to/manifest
    git checkout replicant-6.0
    git merge replicant-6.0-dev
    edit default.xml
    git add default.xml
    git commit -sS -m "Replicant 6.0 0001 images release" 
    git push git@git.replicant.us:replicant/manifest.git replicant-6.0
    
  5. Tag the manifest:
    git tag -u 5816A24C10757FC4 replicant-6.0-0001 -m "Replicant 6.0 0001 images release" 
    git push git@git.replicant.us:replicant/manifest.git replicant-6.0-0001
    
  6. Verify all tags:
    cd .repo/manifests
    git verify-tag $(git describe)
    cd ../..
    repo forall -ec ' { echo "Verifying $REPO_PROJECT" && git verify-tag $(git describe) 2>/dev/null; } || { echo "Error: verification failed!" && exit 1; } '
    
  7. Update prebuilts and start the build (in a newly opened shell with the Replicant keys and certificates installed):
    path/to/release-scripts/releasebuild.sh path/to/replicant-6.0
    
  8. Release the images with the release script:
    rm -rf path/to/images/replicant-6.0/0001
    mkdir -p path/to/images/replicant-6.0/0001
    path/to/release-scripts/release.sh path/to/replicant-6.0 path/to/images/replicant-6.0/0001
    
  9. Sign the binaries with the release script:
    path/to/release-scripts/release.sh path/to/replicant-6.0 path/to/images/replicant-6.0/0001 signatures
    
  10. Compress the release files
    cd path/to/images/replicant-6.0
    tar -cjf 0001.tar.bz2 0001
    
  11. Upload the release to OSUOSL:
    rsync -P -4 -ze ssh 0001.tar.bz2 replicant@ftp-osl.osuosl.org:/home/replicant/data/images/replicant-6.0/
    
  12. Unpack the release on OSUOSL, ensure permissions are correct and run the trigger-replicant script
  13. Update ReplicantImages with the release
  14. Update each device's page with the release
  15. Update ReplicantStatus with the latest status
  16. Verify if other wiki pages need to be updated due to changes introduced by the release (e.g. build pages or ToolsInstallation)
  17. Announce the release on the blog
  18. Update the release on the website and IRC topic

New device documentation

1. Create the device main page, following the naming guidelines applied to other devices (e.g. the Samsung Galaxy S II GT-I9100 is called Galaxy S 2 (I9100) and its page is GalaxyS2I9100)
2. Create all the related sub-pages (build guide, install guide and firmwares list at least), following the naming guidelines applied to other devices (e.g. GalaxyS2I9100Build, GalaxyS2I9100Installation and GalaxyS2I9100LoadedFirmwares)
3. Link the sub-pages to the main page in the index
4. Update the ReplicantStatus page of the wiki with the current status of the device
5. Modify the Index page of the wiki and add the new device in the following sections:

6. Add new issues categories to the Replicant project Redmine

7. Add the device to the Supported devices page on the website

Updated by Denis 'GNUtoo' Carikli almost 4 years ago ยท 104 revisions

Also available in: PDF HTML TXT