Replicant » Upstreaming patches

dl lud wrote:

PIT (Partition Information Table) files contain the partition layout for Samsung phones and tablets. They can be used by flashing tools such as Heimdall to repartition the device.

So far, Replicant did not require any device to be repartitioned, and rather sticked with the stock partition table. However, repartitioning may be needed in the near future, due to issues like #1873.

If we require repartitioning, we need to handle it in a very robust way, else we would have a huge number of issues to handle ("bricked" device, can't switch to another distribution, can't install old Replicant versions to do a test, I lost all my data, etc).

However, having tools to do that is highly desirable.

We could start by making something experimental/dangerous for more advanced users to try it on devices with no important data at all (they would have to backup the EFS before for instance). This could help us understand the consequences of repartitioning and users would be able to fix most issues encountered by themselves if needed.

We could probably manage to do that with:

• Software tools with a very basic interface
• Corresponding tutorial(s) and documentation

Unfortunately, there is no free software tool available to edit PIT files. There is only a proprietary tool dubbed PIT Magic.

Heimdall includes a sub-module, called libpit that can read a PIT file and print the contents in a human readable format. This can be used through Heimdall's print-pit command.
This existing feature could be leveraged to create the reverse command, say build-pit, that grabs the human readable text format and generates the corresponding binary PIT.

Due to modularity and separation of concerns, I would rather have both these commands (print-pit and build-pit) in a separate tool. Heimdall should only do flashing ("do one thing, do it right").

I guess that a tool with a robust backend (libpit) and a quick and dirty interface could work.

In the longer run, I wonder what the ideal interface should look like. We have some examples already:

• Something inspired from "mtdparts=NAND:1M(u-boot)ro,7M(kernel)ro,-(jffs2)"
• fdisk / gdisk / cfdisk / sfdisk / * parted
• U-boot's FIT format

So for:

--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: EFS
Flash Filename: efs.img
FOTA Filename: 

So with the mtdpart like notation we could have:

"pit=eMMC:40k@24k(EFS)rw,[...]" 

And with the FIT-like format:

/* pit_types.h */
#ifndef PIT_TYPES_H
#define PIT_TYPES_H

#define BINARY_TYPE_AP 0
#define DEVICE_TYPE_MMC 2
#endif PIT_TYPES_H

# include <pit_types.h>

// user defined name or no name
partition@6 {
    partition_name = "EFS";
    flash_filename = "efs.img";
    binary_type = <BINARY_TYPE_AP>;
    device_type = <MMC>;
    identifier = <3>; // I've no idea what that is
    attributes: = <ATTRIBUTES_READ|ATTRIBUTES_WRITE>;
    update_attributes = <UPDATE_ATTRIBUTE_FOTA>;
    partition_block_offset = <24576>
    partition_Block Count: 40960
    description = "Modem data partition";
};

What do you think of these interfaces?

What do you think of the following:

• The most easy interface to implement could be implement implemented at first, like the mtdpart one, in a way that leaves room for other implementations
• And if someone finds the time, a more advanced and full featured interface could be implement as well, for more advanced users and use cases.

The only advantage I see on adding this to Heimdall, is that Heimdall is already packaged in several GNU/Linux distros. Users could easily access this new feature once a new version is released.

In the long run, it would probably make sense to have the tool maintained by heimdall, so we could for instance add a new tool that would use that heimdall library to the heimdall project. For instance we'd have heimdall and mkpit for instance. This way they would ensure it works over a bigger variety of devices than the ones we care about, and continue to do so if we stop caring about devices using PIT partition tables.

However this isn't much relevant, as editing PIT files is not something end-users are going to do. The PIT file must be crafted according to needs of the Android distribution, and thus is a task for Replicant maintainers.

If we do the interface right, we won't need to have the crafting in the code of the tool, but instead through command line and/or configuration files.

If one day repartitioning becomes mandatory we could do that:

• Require users to send us the PIT to support a specific eMMC size for a given device. This way we could publish that pit publicly and make sure that everyone is able to restore the previous state of their device, even if they didn't backup the PIT.
• Make the recovery and/or rootfs do the repartitioning by shipping the tool inside Replicant and running it in the first installation. Here if mkpit is in all distributions, if anything goes wrong during that phase, users would be able to more easily recover. The Galaxy S already did something like that for the EFS and modem partitions as they were stored on top of a samsung specific block device layer that required a nonfree kernel module or a free userspace implementation. See the thread about BML for more details on that.

We might also need to write a wrapper around mkpit and other partitioning tool: If I understood well, if you want to repartition, modifying the PIT is most of the time not enough: as the interesting partitions are also covered by GPT, you also need to modify that for the kernel to take it into account. And both need to match. Generating a mtdpart-like or FIT-like description in C looks trivial enough to do. So it shouldn't be hard to wrap.

For testing we it would probably also be a good idea to be able to read a PIT and encode it to a FIT-like description: this way you can take a PIT, encode it in the FIT description, re-create a PIT with that and compare both with cmp, without the need of having any devices.

We should probably also create a repository of PITs, as it's just some configuration data, we can probably do that. As I've almost all the supported devices, I could at least add the devices I have there.

Denis.

Denis 'GNUtoo' Carikli wrote:

What do you think of these interfaces?

I would first stick with Heimdall's/libpit's text format. It has all that we need and makes things easier to test.
We should decide about other formats later, once we get this first one working, and have a good grasp of the PIT binary format.

If we do the interface right, we won't need to have the crafting in the code of the tool, but instead through command line and/or configuration files.

Yes. What I meant is that creating PIT files is not something that end-users will do. End-users will just flash with the PIT file that Replicant provides. Therefore this tool will mostly be used by Replicant (and other distros) developers, not end-users.

• Make the recovery and/or rootfs do the repartitioning by shipping the tool inside Replicant and running it in the first installation.

I would rather avoid this, at least at first. Adds complexity to the recovery for something that already works pretty well with Heimdall. End-users have to use Heimdall either way to flash the recovery, so for a start, adding in a few options like --repartition --PIT new-partitions.pit seems a better approach.

We should probably also create a repository of PITs, as it's just some configuration data, we can probably do that.

Yes, that will come handy.

dl lud wrote:

Denis 'GNUtoo' Carikli wrote:

What do you think of these interfaces?

I would first stick with Heimdall's/libpit's text format. It has all that we need and makes things easier to test.
We should decide about other formats later, once we get this first one working, and have a good grasp of the PIT binary format.

If that looks easier, it's probably best to do it this way. I was assuming that because the other format are meant for parsing they were easier to implement. Though for the FIT-like one you need to plug in dtc (the device tree compiler), so it might turn out to be harder, even if we have an usage example in u-boot, and probably coreboot as well.

If we do the interface right, we won't need to have the crafting in the code of the tool, but instead through command line and/or configuration files.

Yes. What I meant is that creating PIT files is not something that end-users will do. End-users will just flash with the PIT file that Replicant provides. Therefore this tool will mostly be used by Replicant (and other distros) developers, not end-users.

• Make the recovery and/or rootfs do the repartitioning by shipping the tool inside Replicant and running it in the first installation.

I would rather avoid this, at least at first. Adds complexity to the recovery for something that already works pretty well with Heimdall. End-users have to use Heimdall either way to flash the recovery, so for a start, adding in a few options like --repartition --PIT new-partitions.pit seems a better approach.

As I understand, using that command only changes the PIT and flash the partition files using the new PIT.
This has several drawbacks for a robust implementation (which can be done later):

• It cannot read the PIT and change it in one go. According to Wikipedia , the GT-I9300 has (at least) 3 possible eMMC sizes: 16, 32, and 64 GB. So we'd need to generate 3 PITs for 1 device, hoping that the users didn't already re-partition. Having it run on the device would enable to programmatically do extensive checking and only proceed if everything is fine. If not, users would need to send a PIT to make sure to have a backup somewhere for them and other users in the same configuration, unless they have some completely custom partition scheme that is not from another distribution. In that case they would be expected to backup their own PIT and do some action like provide the custom PIT somehow in a second location, just to tell the installer that everything is ok. The action could be something like 'adb shell "mkdir /some-dir/pit/" ; adb push custom_pit.bin /some-dir/pit/', with a good name for 'some-dir' that users wound't use when manually backuping the PIT or doing other things within the recovery. The 'some-dir' could be something like 'installer-automatic' 'automatic-installer' or the same kind of directory used on the Galaxy S (GT-I9000).
• We could also handle the GPT at the same time without needing extensive tricks to flash a different GPT.