Project

General

Profile

Actions

Issue #1916

open

Write instructions to migrate from self-build images to releases (Was: Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition))

Added by Denis 'GNUtoo' Carikli almost 6 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
Website and wiki content
Target version:
-
Start date:
03/08/2019
Due date:
% Done:

0%

Estimated time:
10.00 h
Resolution:
Device:
Grant:
Type of work:
Wiki editions

Related issues

Has duplicate Replicant - Issue #1977: Test the installation instructions from scratchResolved11/29/2019

Actions
Actions #1

Updated by Fil Bergamo almost 6 years ago

  • Assignee set to Fil Bergamo
  • Estimated time set to 10.00 h

I can take care of testing the instructions on the following devices:

- i9100 (Samsung Galaxy S2)
- i9300 (Samsung Galaxy S3 GSM)
- N7100 (Samsung Galaxy Note2)
- N5110 (Samsung Galaxy Note 8.0 WiFi)
- P3100 (Samsung Galaxy Tab 2 7.0 GSM)

Actions #2

Updated by Denis 'GNUtoo' Carikli over 5 years ago

  • Category set to Installation instructions
Actions #3

Updated by Kurtis Hanna about 5 years ago

  • Blocked by Issue #1960: Build release candidate image for 6.0 0004 added
Actions #4

Updated by Kurtis Hanna about 5 years ago

  • Blocked by deleted (Issue #1960: Build release candidate image for 6.0 0004)
Actions #5

Updated by Kurtis Hanna almost 5 years ago

  • Has duplicate Issue #1977: Test the installation instructions from scratch added
Actions #6

Updated by Denis 'GNUtoo' Carikli almost 5 years ago

  • Subject changed from make sure that the upgrade instructions work for Replicant 6.0 0004 to Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition)

The upgrade instructions don't work.
The data/ partition is incompatible and the application data need to be backed up and restored.

Actions #7

Updated by Kurtis Hanna almost 5 years ago

Is there any reason why we shouldn't add installation instructions where TWRP is used? A user reported that it can be used to install Replicant without a PC. https://redmine.replicant.us/boards/3/topics/14715

It is my understanding that TWRP is free software, but they have an android app in Google Play that isn't free software.

Actions #9

Updated by Denis 'GNUtoo' Carikli over 4 years ago

Between Replicant 6.0 0003 to Replicant 6.0 0004 RC1 we may have had some changes affecting data/ like the fix of the bug #1827, however between the 0004 RC1 and 0004 RC2 we have no such changes.

However, since Replicant 0004 RC1, instead of having replicant-6.0 in Settings->About phone->Replicant version, we now have something like replicant-6.0-0004-rc1 or replicant-6.0-0004-rc2. So I wonder if it's related or not.

In any case that change is desirable as it helps users understand if they need to upgrade or not, so the best way is to fix it somehow.

Here are ways we could look into:
  • Debug this issue
  • Making a TWRP version that is fully free software (so with a deblobbed kernel). Thanks a lot for the pointer on that.
  • Making sure that the future Replicant 10 recovery doesn't do that check and enable to safely install older versions of Replicant as well (for instance by checking gpg signatures and displaying versions in a generic way).
Actions #10

Updated by Denis 'GNUtoo' Carikli over 4 years ago

$ repo grep "on top of incompatible data" returns only one entry.

In build/tools/releasetools/edify_generator.py we have:

  def ValidateSignatures(self, command):
    self.script.append('package_extract_file("META-INF/org/replicant/releasekey", "/tmp/releasekey");')
    # Exit code 124 == abort. run_program returns raw, so left-shift 8bit
    self.script.append('run_program("/tmp/install/bin/otasigcheck.sh") != "31744" || abort("Can\'t install this package on top of incompatible data. Please try another package or run a factory reset");')

In turn otasigcheck.sh seem to be in vendor/replicant/prebuilt/common/bin/otasigcheck.sh .

$ repo grep "incompatible data" returns more than one entry but only the entry above is code, the rest are only comments.

edit1: added $ repo grep "incompatible data"

Actions #11

Updated by Denis 'GNUtoo' Carikli over 4 years ago

Also, using

-CM_DISPLAY_VERSION := $(REPLICANT_VERSION)
+CM_DISPLAY_VERSION := replicant-6.0-0004-rc3

instead of:
-RELEASE=replicant-6.0
+RELEASE=replicant-6.0-0004-rc2

In vendor/replicant/config/common.mk didn't help.

Actions #12

Updated by Denis 'GNUtoo' Carikli over 4 years ago

I don't know why the return code is to be shifted left by 8, but 124 << 8 31744, and we have the following comment as well in the code described above: "Exit code 124 abort. run_program returns raw, so left-shift 8bit", so this is the issue in otasigcheck.sh:

  if [ "$pkgkey" != "$relkey" ]; then
     echo "You have an installed system that isn't signed with this build's key, aborting..." 
     exit 124
  fi

Now we need to understand more that check not to create any side issues if we remove it.

Actions #13

Updated by Kurtis Hanna over 4 years ago

Would following these directions help us fix this issue? https://gist.github.com/ttimasdf/0cdd4b958dd00ff2bcdb125415aa1269

Actions #14

Updated by Denis 'GNUtoo' Carikli over 4 years ago

Thanks, I've used that information to look for something similar in LineageOS but I then figured out that it was better to look for that in TWRP since you pointed to me that a user got it working with TWRP.

In fact they have scripts to handle that case, and this one might be something like the script we are looking for:
https://raw.githubusercontent.com/omnirom/android_bootable_recovery/android-9.0/prebuilt/fix_permissions.sh

There are also other scripts in prebuilt:
https://github.com/omnirom/android_bootable_recovery/blob/android-9.0/prebuilt
/fix_permissions.sh

Note that TWRP is GPLv3 as per their COPYING so we won't have a license issue here even if that file lacks licensing information. We'd just create the header with the proper info instead.

edit1: modified link from prebuilt to the actual file and re-added to prebuilt
edit2: fixed edit1 message, added COPYING

Actions #15

Updated by Kurtis Hanna over 4 years ago

This LineageOS patchset seems to have never been added to any of their branches: https://github.com/LineageOS/android_vendor_cm/commit/2f7c7decc4cd5b42f044a7841a74468e4cacd694#diff-a5ab82d48d569fdffe41cbf3b31a5767

Is it compatible with Replicant 6?

Actions #16

Updated by Denis 'GNUtoo' Carikli over 4 years ago

Kurtis Hanna wrote in #note-15:

This LineageOS patchset seems to have never been added to any of their branches: https://github.com/LineageOS/android_vendor_cm/commit/2f7c7decc4cd5b42f044a7841a74468e4cacd694#diff-a5ab82d48d569fdffe41cbf3b31a5767

Is it compatible with Replicant 6?

Thanks for the findings.

It's not compatible as-is but replacing the old and new keys and certificates inside result in something that works.

To do that it's possible to use the script that I made to automatize it a bit.

Here's how to do it with the current version of that script:
  • The python script I pushed I wrote needs to be checked out, and it's also a good idea to get all all the files inside that same directory (templates/, etc) to avoid any dependency issues.
  • A device that is not used and can completely wiped is required. It needs to be compatible with the old and new images.
  • Then a full factory reset needs to be done in the Replicant recovery
  • After that the old images needs to be installed on the device and the device has to complete its boot
  • Then /data/system/applications.xml needs to be extracted from it and renamed to something meaningful like old_applications.xml
  • Then a full factory reset needs to be done in the Replicant recovery again
  • Then the new images needs to be installed on the device and the device has to complete its boot
  • Then the new /data/system/applications.xml needs to be extracted from it and renamed to something meaningful like new_applications.xml

Then the script can be used like that:

$ ./generate_key_migration_script.py path-to-replicant-root-tree/vendor/replicant/prebuilt/common/bin/keys-migration.sh old_applications.xml new_applications.xml

Limitations

It's probably not a big issue to require to extract the applications.xml from the phone, however for the new one you need to build the target image first to then be able to run the python script to then build the final image.

So the next step would be to find a way to use the newly generated certificate and encode them in the format expected by applications.xml and re-use it in the generation of keys-migration.sh

Further developments

We could also add the old certificates in a repo and/or download them from the FTP to not have to boot a device to build an image.

Key policies

We also need to define policies as for what to do with the generated keys.

So far:
  • Paul Kocialkowski did all the Replicant 4.2 releases
  • Wolfgang Wiedmeyer did all the Replicant 6.0 releases until Replicant 6.0 0003

As users wiped their data between major Replicant version and that a change of maintainer didn't occur within a minor version they didn't have the issue.

Worlfgang most probably kept the generated keys between the Replicant 6.0 releases.

We probably need a different security model:
  • Having to keep these keys and/or passing them around is not ideal.
  • Having to make new migration scripts each time the developer making the release changes is not ideal either unless it's almost completely automated.

update1: fix grammar
update2: fix typo

Actions #17

Updated by Kurtis Hanna about 4 years ago

  • Blocks Issue #1960: Build release candidate image for 6.0 0004 added
Actions #18

Updated by Denis 'GNUtoo' Carikli about 4 years ago

$ openssl x509 -in vendor/replicant-security/platform.x509.pem -inform PEM -pubkey  | grep -B 100 "\-----END PUBLIC KEY-----" | grep -v "^--" | xargs echo -n  | sed 's# ##g' ;echo
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB
$ grep 'MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB' packages.xml 
            <public-key identifier="1" value="MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB" />
Next steps I:
  • Update the python script to use vendor/replicant-security certs
  • Add support for multiple old certificates/key sets
  • We can take the keys/certs from the releases on the wiki image page and add them to git in the data repository for instance
  • We add the data repository to the manifest
  • We then update the python script to sed all known keys/certs
    => We have something that always works for upgrades (assuming we add new certs in the data repo at each new release)
Next steps II:
  • We also add support for creating a script to downgrade the images
  • We add instructions to use that script in the recovery. Most of the instructions can be reused from backuping the EFS.

Update1: Added the next steps
Update2: Status update + change Steps I
Update3: Status update

Actions #19

Updated by Denis 'GNUtoo' Carikli about 4 years ago

  • Subject changed from Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition) to Write instructions to migrate from self-build images to releases (Was: Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition))
  • Category changed from Installation instructions to Website and wiki content
  • Target version deleted (Replicant 6.0 0004)

All that is now pushed, so we don't need specific instructions for users.

Instructions for people releasing images also have been pushed as part of the Replicant release scripts.

So remains instructions for people running their own builds to be written on the wiki.

Because of all that I've moved modified that bugreport to reflect that this is the only missing thing to do and that it doesn't affect the release anymore.

Actions #20

Updated by Kurtis Hanna about 4 years ago

  • Blocks deleted (Issue #1960: Build release candidate image for 6.0 0004)
Actions #21

Updated by _I3^ RELATIVISM over 3 years ago

  • Type of work Wiki editions added
Actions

Also available in: Atom PDF