Project

General

Profile

Issue #1916

Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition)

Added by Denis 'GNUtoo' Carikli over 1 year ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
Category:
Installation instructions
Target version:
Start date:
03/08/2019
Due date:
% Done:

0%

Estimated time:
10.00 h
Resolution:
Device:
Grant:

Related issues

Has duplicate Replicant - Issue #1977: Test the installation instructions from scratchNew11/29/2019

Actions
Blocks Replicant - Issue #1960: Build release candidate image for 6.0 0004In Progress10/09/2019

Actions
#1

Updated by Fil Bergamo over 1 year ago

  • Assignee set to Fil Bergamo
  • Estimated time set to 10.00 h

I can take care of testing the instructions on the following devices:

- i9100 (Samsung Galaxy S2)
- i9300 (Samsung Galaxy S3 GSM)
- N7100 (Samsung Galaxy Note2)
- N5110 (Samsung Galaxy Note 8.0 WiFi)
- P3100 (Samsung Galaxy Tab 2 7.0 GSM)

#2

Updated by Denis 'GNUtoo' Carikli over 1 year ago

  • Category set to Installation instructions
#3

Updated by Kurtis Hanna about 1 year ago

  • Blocked by Issue #1960: Build release candidate image for 6.0 0004 added
#4

Updated by Kurtis Hanna about 1 year ago

  • Blocked by deleted (Issue #1960: Build release candidate image for 6.0 0004)
#5

Updated by Kurtis Hanna 10 months ago

  • Has duplicate Issue #1977: Test the installation instructions from scratch added
#6

Updated by Denis 'GNUtoo' Carikli 10 months ago

  • Subject changed from make sure that the upgrade instructions work for Replicant 6.0 0004 to Write upgrade instructions from Replicant 6.0 0003 to Replicant 6.0 0004 (incompatible /data partition)

The upgrade instructions don't work.
The data/ partition is incompatible and the application data need to be backed up and restored.

#7

Updated by Kurtis Hanna 9 months ago

Is there any reason why we shouldn't add installation instructions where TWRP is used? A user reported that it can be used to install Replicant without a PC. https://redmine.replicant.us/boards/3/topics/14715

It is my understanding that TWRP is free software, but they have an android app in Google Play that isn't free software.

#9

Updated by Denis 'GNUtoo' Carikli 3 months ago

Between Replicant 6.0 0003 to Replicant 6.0 0004 RC1 we may have had some changes affecting data/ like the fix of the bug #1827, however between the 0004 RC1 and 0004 RC2 we have no such changes.

However, since Replicant 0004 RC1, instead of having replicant-6.0 in Settings->About phone->Replicant version, we now have something like replicant-6.0-0004-rc1 or replicant-6.0-0004-rc2. So I wonder if it's related or not.

In any case that change is desirable as it helps users understand if they need to upgrade or not, so the best way is to fix it somehow.

Here are ways we could look into:
  • Debug this issue
  • Making a TWRP version that is fully free software (so with a deblobbed kernel). Thanks a lot for the pointer on that.
  • Making sure that the future Replicant 10 recovery doesn't do that check and enable to safely install older versions of Replicant as well (for instance by checking gpg signatures and displaying versions in a generic way).
#10

Updated by Denis 'GNUtoo' Carikli 2 months ago

$ repo grep "on top of incompatible data" returns only one entry.

In build/tools/releasetools/edify_generator.py we have:

  def ValidateSignatures(self, command):
    self.script.append('package_extract_file("META-INF/org/replicant/releasekey", "/tmp/releasekey");')
    # Exit code 124 == abort. run_program returns raw, so left-shift 8bit
    self.script.append('run_program("/tmp/install/bin/otasigcheck.sh") != "31744" || abort("Can\'t install this package on top of incompatible data. Please try another package or run a factory reset");')

In turn otasigcheck.sh seem to be in vendor/replicant/prebuilt/common/bin/otasigcheck.sh .

$ repo grep "incompatible data" returns more than one entry but only the entry above is code, the rest are only comments.

edit1: added $ repo grep "incompatible data"

#11

Updated by Denis 'GNUtoo' Carikli 2 months ago

Also, using

-CM_DISPLAY_VERSION := $(REPLICANT_VERSION)
+CM_DISPLAY_VERSION := replicant-6.0-0004-rc3

instead of:
-RELEASE=replicant-6.0
+RELEASE=replicant-6.0-0004-rc2

In vendor/replicant/config/common.mk didn't help.

#12

Updated by Denis 'GNUtoo' Carikli 2 months ago

I don't know why the return code is to be shifted left by 8, but 124 << 8 31744, and we have the following comment as well in the code described above: "Exit code 124 abort. run_program returns raw, so left-shift 8bit", so this is the issue in otasigcheck.sh:

  if [ "$pkgkey" != "$relkey" ]; then
     echo "You have an installed system that isn't signed with this build's key, aborting..." 
     exit 124
  fi

Now we need to understand more that check not to create any side issues if we remove it.

#13

Updated by Kurtis Hanna about 2 months ago

Would following these directions help us fix this issue? https://gist.github.com/ttimasdf/0cdd4b958dd00ff2bcdb125415aa1269

#14

Updated by Denis 'GNUtoo' Carikli about 2 months ago

Thanks, I've used that information to look for something similar in LineageOS but I then figured out that it was better to look for that in TWRP since you pointed to me that a user got it working with TWRP.

In fact they have scripts to handle that case, and this one might be something like the script we are looking for:
https://raw.githubusercontent.com/omnirom/android_bootable_recovery/android-9.0/prebuilt/fix_permissions.sh

There are also other scripts in prebuilt:
https://github.com/omnirom/android_bootable_recovery/blob/android-9.0/prebuilt
/fix_permissions.sh

Note that TWRP is GPLv3 as per their COPYING so we won't have a license issue here even if that file lacks licensing information. We'd just create the header with the proper info instead.

edit1: modified link from prebuilt to the actual file and re-added to prebuilt
edit2: fixed edit1 message, added COPYING

#15

Updated by Kurtis Hanna about 2 months ago

This LineageOS patchset seems to have never been added to any of their branches: https://github.com/LineageOS/android_vendor_cm/commit/2f7c7decc4cd5b42f044a7841a74468e4cacd694#diff-a5ab82d48d569fdffe41cbf3b31a5767

Is it compatible with Replicant 6?

#16

Updated by Denis 'GNUtoo' Carikli about 2 months ago

Kurtis Hanna wrote in #note-15:

This LineageOS patchset seems to have never been added to any of their branches: https://github.com/LineageOS/android_vendor_cm/commit/2f7c7decc4cd5b42f044a7841a74468e4cacd694#diff-a5ab82d48d569fdffe41cbf3b31a5767

Is it compatible with Replicant 6?

Thanks for the findings.

It's not compatible as-is but replacing the old and new keys and certificates inside result in something that works.

To do that it's possible to use the script that I made to automatize it a bit.

Here's how to do it with the current version of that script:
  • The python script I pushed I wrote needs to be checked out, and it's also a good idea to get all all the files inside that same directory (templates/, etc) to avoid any dependency issues.
  • A device that is not used and can completely wiped is required. It needs to be compatible with the old and new images.
  • Then a full factory reset needs to be done in the Replicant recovery
  • After that the old images needs to be installed on the device and the device has to complete its boot
  • Then /data/system/applications.xml needs to be extracted from it and renamed to something meaningful like old_applications.xml
  • Then a full factory reset needs to be done in the Replicant recovery again
  • Then the new images needs to be installed on the device and the device has to complete its boot
  • Then the new /data/system/applications.xml needs to be extracted from it and renamed to something meaningful like new_applications.xml

Then the script can be used like that:

$ ./generate_key_migration_script.py path-to-replicant-root-tree/vendor/replicant/prebuilt/common/bin/keys-migration.sh old_applications.xml new_applications.xml

Limitations

It's probably not a big issue to require to extract the applications.xml from the phone, however for the new one you need to build the target image first to then be able to run the python script to then build the final image.

So the next step would be to find a way to use the newly generated certificate and encode them in the format expected by applications.xml and re-use it in the generation of keys-migration.sh

Further developments

We could also add the old certificates in a repo and/or download them from the FTP to not have to boot a device to build an image.

Key policies

We also need to define policies as for what to do with the generated keys.

So far:
  • Paul Kocialkowski did all the Replicant 4.2 releases
  • Wolfgang Wiedmeyer did all the Replicant 6.0 releases until Replicant 6.0 0003

As users wiped their data between major Replicant version and that a change of maintainer didn't occur within a minor version they didn't have the issue.

Worlfgang most probably kept the generated keys between the Replicant 6.0 releases.

We probably need a different security model:
  • Having to keep these keys and/or passing them around is not ideal.
  • Having to make new migration scripts each time the developer making the release changes is not ideal either unless it's almost completely automated.

update1: fix grammar
update2: fix typo

#17

Updated by Kurtis Hanna about 1 month ago

  • Blocks Issue #1960: Build release candidate image for 6.0 0004 added
#18

Updated by Denis 'GNUtoo' Carikli about 1 month ago

$ openssl x509 -in vendor/replicant-security/platform.x509.pem -inform PEM -pubkey  | grep -B 100 "\-----END PUBLIC KEY-----" | grep -v "^--" | xargs echo -n  | sed 's# ##g' ;echo
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB
$ grep 'MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB' packages.xml 
            <public-key identifier="1" value="MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1OieLpOFLahyvUGDoJhB0GnLNfJ7qf+VEg59vW50yhFZAj0HseFBzXSy+PVtkIAAuqIyXfPwgg0lAfnPh5oW2gvBqW6dTZC1DPH84Fg3FCqu2T+fEXpdRafpWjLG2oWfgupWWg9Yg/QH9Ft9LbTMLGiXa/OpmVfx3aJrW/EzPdUMR87xuZmJeSoTC1VCgv/lkqnft8xxJrt+43sXz5OKM56Ia3ardBIdBxrlwvDCP46Fp351rx1+RTpSBg7zDBiRaJLbRQJtcA6bEpZriylslndnACh45jXKDrYV55k0m5Dh+Zj+BRZK+/eyPk2JCZp/pgdWt6HaQvz9/wlWXhF9lQIDAQAB" />
Next steps I:
  • Update the python script to use vendor/replicant-security certs
  • Add support for multiple old certificates/key sets
  • We can take the keys/certs from the releases on the wiki image page and add them to git in the data repository for instance
  • We add the data repository to the manifest
  • We then update the python script to sed all known keys/certs
    => We have something that always works for upgrades (assuming we add new certs in the data repo at each new release)
Next steps II:
  • We also add support for creating a script to downgrade the images
  • We add instructions to use that script in the recovery. Most of the instructions can be reused from backuping the EFS.

Update1: Added the next steps
Update2: Status update + change Steps I
Update3: Status update

Also available in: Atom PDF