CTS and CI for replicant

Added by Loic Dachary 7 months ago


I don't think there any kind of continuous integration for replicant at the moment. I suppose something like https://source.android.com/compatibility/cts/index.html could be used for that purpose. Even though it may not pass all tests, having it run would be helpful to validate a change anyway. I realize this requires an actual device plugged in a server: it's more difficult that the usual CI ;-)

What do you think ? Would it be useful or would it be a waste of time ?

Replies (5)

RE: CTS and CI for replicant - Added by Loic Dachary 7 months ago

A page was added explaining how to setup and run CTS for Replicant 4.2 (thanks to phh on irc.freenode.net#replicant for the help ;-) It is running in the basement and making funny noises. There are two devices under test: a SIII with a samsung firmware running 4.3 and a https://tehnoetic.com/ SII running Replicant 4.2.

RE: CTS and CI for replicant - Added by Loic Dachary 7 months ago

https://source.android.com/compatibility/cts/setup.html#screen_storage reads "If the device has SIM card slots, plug in an activated SIM card to each slot. If the device supports SMS, each SIM card should have its own number field populated.". There is a filesystem on the SIM, it can contain contacts and SMS and the phone number. If it's not possible to do that using settings (which is most likely the case for newer phones) https://github.com/lytsing/MyPhoneNumber may work but requires system permissions.

This is required for CTS to run SMS tests so that it can send SMS to itself.

RE: CTS and CI for replicant - Added by Loic Dachary 7 months ago

RE: CTS and CI for replicant - Added by Wolfgang Wiedmeyer about 1 month ago

Thanks for looking into this and documenting the CTS usage in the wiki!

Do you know if CyanongenMod has been using the CTS or if nowadays LineageOS has an interest in using it? I didn't hear so far that they are using it. My fear is that a lot of or maybe most of the failed tests will be about parts of the code that should actually be fixed in LineageOS. In general, I agree that it would be very helpful to have something like this in place, but it's probably necessary to work with LineageOS to get the failures fixed that should be fixed on their side. And then maybe getting them to adapt the CTS in their development workflow. Then if tests fail, we can be sure that something is going wrong on our end and we need to fix it.

I had a look at the test results. They look like a lot of tracebacks. Is there a summary at some point in the xml? Or an overview of the test cases that did run and their results?

RE: CTS and CI for replicant - Added by Loic Dachary about 1 month ago

Hey ! Last I checked (january IIRC) LineageOS has no interest in running CTS (nor did CyanogenMod). Even though, CTS is the only automated test in existence and not using it means more human testing is required and the regressions it could catch won't be. There is no need for it to be perfect, it is useful as it is. The major blocker, IMHO is the lack of test culture rather than the lack of proper tooling. It's extremely difficult to convince developers to use tests on a project that started without any kind of automated testing.