Encourage new apps enable reproducible build
-
- View options
Reproducible build can solve many problem of F-Droid builds. We should encourage new apps using reproducible build, especially apps without native code.
Attributes
Status
Assignees
Labels
Parent
Weight
Milestone
Iteration
Dates
Start: None
Due: None
Health status
Time tracking
Related to
Activity
- Edited by linsui
Until now, there are some issues may prevent the apk from reproducible build:
- The files in the apk is not ordered. It seems the disorderfs is not necessary. Apks built on MacOS have this problem.
- The apk is signed with gradle. See fdroidserver#1051 (closed)
If the unziped files are identical but the apk is not then it may suffer the above problem. We can use zipinfo to check that.
https://github.com/mikepenz/AboutLibraries/issues/784 The file generated by AboutLibraries has timestamp.
- Edited by linsui
!11999 (merged) !12194 (merged)
The
assets/dexopt/baseline.profmfile changes randomly. https://issuetracker.google.com/issues/231837768The profile can be disabled with
tasks.whenTaskAdded {task -> if(task.name.contains("ArtProfile")) { task.enabled = false } }prebuild: sed -i -e '/android {/i tasks.whenTaskAdded {task -> if(task.name.contains("ArtProfile")) { task.enabled = false }}' build.gradleor
tasks.whenTaskAdded { if(name.contains("ArtProfile")) { enabled = false } }prebuild: sed -i -e '/android {/i tasks.whenTaskAdded { if(name.contains("ArtProfile")) { enabled = false }}' build.gradle.kts
@obfusk Is that possible to migrate old apps to reproducible build? Can we sign the upstream key with F-Droid key directly?
We currently support RB using either
Binaries:(grab upstream APK, publish that if the signature is valid for our unsigned APK after copying) or signatures in metadata (patch extracted signature onto our unsigned APK, publish both that and one signed by us if the APK verifies).I'd say the latter has better security guarantees, but it is more work and doesn't support automatic updates.
fdroidserver#1013 suggests publishing an APK signed by us as well with
Binaries:.I think that's a good idea, but not something we should do without carefully considering the implications (both positive and negative and potential mitigations).
- IMO having both a signature by us and by upstream is good:
- upstream-signed APKs allow updating from e.g. play store version: good.
- upstream may not keep their signing key as safe as we do (though they may keep it safer of course).
- there could be upstream-signed APKs with non-free components or spyware, which the user (or someone else) can now update to: bad.
- users can choose which key they trust more, upstream or ours: good.
- IMO signatures in metadata is preferable from a security standpoint, though it obviously has drawbacks.
- IMO publishing an APK signed by us as well with
Binaries:should be opt-in, not enabled for all apps already usingBinaries:. - we currently don't check which key upstream APKs are signed with AFAIK (fdroidserver#1055 (closed)).
- there can be some unwanted extra data in the APK Signing Block (fdroidserver#1056) which I'd like to remove if possible from upstream APKs, which APKs signed by us would not have.
- IMO having both a signature by us and by upstream is good:
upstream may not keep their signing key as safe as we do (though they may keep it safer of course). users can choose which key they trust more, upstream or ours: good.
I doubt if we should consider this risk. It's not our responsibility. If the users choose to install a random apk they are already at the risk. Such users won't know which source is more trustable and they will feel confused. They can always choose to trust upstream more when they need to switch to install the apk from upstream. The current UI is already very awful. There are just two apks with the same version with no explaination. Droid-ify has better UI which displays the signature but I still don't know which one is signed by whom.
there could be upstream-signed APKs with non-free components or spyware, which the user (or someone else) can now update to: bad.
If upstream want to add malware to the app, they can just do that. We don't audit every update.
there can be some unwanted extra data in the APK Signing Block (fdroidserver#1056) which I'd like to remove if possible from upstream APKs, which APKs signed by us would not have.
We should ask upstream removing them.
Having an (opt-in) version of
Binaries:which publishes an APK signed by us as well should allow us to "migrate [existing] apps to reproducible build", so that would be niceBut we can't migrate the existing users. Is it possible to migrate existing users with key rotation?
The current UI is already very awful.
True :( At least the website tries to label them as "signed by f-droid" and "signed by upstream".
It's not our responsibility.
Maybe not. But I do prefer giving users the choice. The UI can be fixed.
If upstream want to add malware to the app, they can just do that. We don't audit every update.
True. But malware in the source would be harder to hide (even if the damage may already be done).
Whereas an APK with malware added that's only installed on the phones of a small set of targeted users is much harder to detect.
We should ask upstream removing them.
Agreed that would be the best place to remove it.
But we can't migrate the existing users. Is it possible to migrate existing users with key rotation?
I think it might be, but I've not looked into how key rotation actually works yet. It would presumably require manual work by Ciaran though, unless we can somehow integrate it into the signing process. I don't know. I also don't know if that's a one-time thing or if it requires continued use of the old key to provide proof of rotation.
I thought a user who understand the choice won't install a random apk.
There are other ways for a malicious APK to be installed than the user installing random APKs.
For example, now that gplay requires developers to hand over their signing keys, a malicious APK could be distributed via that route without the developer ever knowing.
How should we deal with "flaky" RB (i.e. apps that have tiny differences between builds, but do produce identical builds if you try several times)?
NewPipe had this issue, not sure if it's fixed by now.
Obviously not something we want to support. Fixing the cause is definitely better.
But it might be better to be able to choose to build multiple times (with a maximum of course) until RB succeeds for some apps that can't be easily fixed and for which it's not really an option to just not use RB (e.g. if it used to work for older versions).
- Edited by FC (Fay) Stegerman
CRLF issues: !11690 (comment 1166721148)
Workaround: https://github.com/obfusk/reproducible-apk-tools
Does it make sense to have a
postbuild:fixup step for cases like this?
This might work: https://github.com/obfusk/reproducible-apk-tools
Can't test it without a test case though.
Thanks! There are two testcases: !12116 (merged) and https://github.com/fumiama/copymanga/releases
Unfortunately, I can't test that one any more since upstream has replaced the APK that was build on macOS.
And in this case I have no other APK to compare it to.
For CopyManga: https://gitlab.com/linsui/fdroiddata/-/jobs/3366335463/artifacts/raw/tmp/top.fumiama.copymanga_26.apk and https://github.com/fumiama/copymanga/releases/download/v2.0.beta14/CopyManga_2.0.beta14_sorted.apk. It seems the sort method is different. In apk the permission should also be sorted.
- Edited by FC (Fay) Stegerman
As far as I can tell:
- macOS builds are ordered differently each time, the ordering is both different from Linux and not deterministic/reproducible.
- Linux builds are still not really sorted either, but the ordering is deterministic/reproducible.
I have no idea what determines the order the files end up in.
RB works if we sort our APK as well:
$ repro-apk sort-apk top.fumiama.copymanga_26.apk x.apk $ apksigcopier compare CopyManga_2.0.beta14_sorted.apk --unsigned x.apk && echo OK OK - Edited by FC (Fay) Stegerman
Maybe we can make something like this work?
srclibs: - reproducible-apk-tools@v0.1.1 postbuild: - $$reproducible-apk-tools$$/sort-apk.py $$OUTPUT$$ sorted.apk - mv sorted.apk $$OUTPUT$$Or this (if we install the Python package in the srclib build)?
srclibs: - reproducible-apk-tools@v0.1.1 postbuild: - repro-apk sort-apk $$OUTPUT$$ sorted.apk - mv sorted.apk $$OUTPUT$$ - Edited by FC (Fay) Stegerman
The files with
-rw-rw-rw-permission are before files with-rw----.Ah. Interesting. But not the only difference. I'm guessing they were added by a different library/tool than the other files. They are also marked
unxinstead offat. - Edited by linsui
I prefer to make the apk built on MacOS has the same order as the apk built on Linux without changing both apks.
The order is weird. I saw this:
-rw-r--r-- 0.0 unx 56 b- defN 81-Jan-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties -rw-r--r-- 0.0 unx 6529 b- stor 81-Jan-01 01:01 assets/dexopt/baseline.prof -rw-r--r-- 0.0 unx 348 b- stor 81-Jan-01 01:01 assets/dexopt/baseline.profm -rw-r--r-- 0.0 unx 13607524 b- stor 81-Jan-01 01:01 classes.dex -rw-r--r-- 0.0 unx 2549392 b- stor 81-Jan-01 01:01 classes2.dex -rw-r--r-- 0.0 unx 26152 b- defN 81-Jan-01 01:01 assets/open_source_licenses.htmlThe open_source_licenses.html is generated seperately so maybe that's why it's below classes.dex.
I prefer to make the apk built on MacOS has the same order as the apk built on Linux without changing both apks.
Sure. But I have no idea how the order is determined. It's certainly not as simple as "sorted". So I don't think we can duplicate the order using an external tool. The only way to get the same order is to make the order used by the build itself deterministic like it is on Linux.
I'm guessing there are several parts of the build which generate some files and append them to the APK.
Quite possibly the files in each "batch" are sorted before adding (though that's just each individual batch, not the APK as a whole).
On Linux, these parts run in a deterministic order, but for some reason they don't on macOS. Hence the differences.
Might be worth reporting as a bug in the build system. But I have no idea where in the build system.
I doubt the apk is built incremently.
Might be worth reporting as a bug in the build system.
Yes, please. At least they can sort the apk.
-rw---- 2.4 fat 2338 b- defN 81-Jan-01 01:01 kotlin/reflect/reflect.kotlin_builtins -rw---- 0.0 fat 5816 b- defN 81-Jan-01 01:01 AndroidManifest.xml -rw---- 0.0 fat 368 b- defN 81-Jan-01 01:01 res/-W.xml -rw---- 0.0 fat 1396 b- defN 81-Jan-01 01:01 res/-Y.xmlI thought you are right, it's sorted in batch.
AFAIK
disorderfsis originally a tool to find bugs caused by the filesystem returning things in nondeterministic order by causing that more predictably.But it can also be used to force deterministic order to avoid such bugs.
For example, in Python
os.listdir()andglob.glob()will not sort the list of file names they return. The ordering depends on the filesystem. And can thus vary between different systems or builds.This can easily cause non-reproducible builds.
We probably have some nondeterministic behaviour in
fdroidserver: fdroidserver!991 (closed)I sort the apk built by fdroidserver for now. Hopefully postbuild can make it less verbose. !12138 (merged)
- Edited by FC (Fay) Stegerman
Maybe you can also test it on Bitcoin Wallet which uses disorderfs.
If someone has an unsigned APK built w/odisorderfsfor me, I can test it.Or you can just try building it (e.g. in CI) usingsort-apkinstead ofdisorderfs.The Bitcoin Wallet signed APKs have sorted entries (apart from the v1 signature, but that should not be in the unsigned APK), so it might just work.
Unless the padding for ZIP alignment differs between using
disorderfsand sorting after building.So far it's been impossible to get the padding exactly the same w/o sorting both APKs with
sort-apk. Unless the original APK is not aligned at all, it tends to be different when the APK entries had a different order (and thus a different offset) before sorting.Because of this,
sort-apkforcefully recreates the padding even if the entry is already aligned (since being aligned doesn't mean the padding is identical) to make its output as deterministic as possible. So disorderfs does the same thing? Maybe you can also test it on Bitcoin Wallet which uses disorderfs.
It looks like Bitcoin Wallet uses
disorderfsto makeresources.arscreproducible; that's the only thing that changed when I built it w/odisorderfs, the ordering of files in the APK was the same.Quoting myself from #fdroid-dev:
idk. maybe it is filesystem related. or that's part (but not all) of it. the most obvious difference is that res/* files are ordered (relative to each other) in the APK in our builds on Debian but completely random on macOS/Fedora. but e.g. the position of classes.dex and AndroidManifest.xml also differs between macOS/Fedora and Debian; but the position on macOS and Fedora seems to be the same. it's really odd.
e.g. Debian adds classes.dex, classes2.dex, AndroidManifest.xml, resources.arsc in that order; on Fedora/macOS the relative order of those 4 files is exactly the reverse.
(and that's two completely different apps)
These diffs ignore
res/*files, which are disordered (relative to each other) on Fedora & macOS, but ordered (relative to each other) on Debian; JAR signature files also ignored here.Catima: Fedora vs Debian
@@ -1,6 +1,11 @@ -Archive: app-release-2.21.0.apk +Archive: me.hackerchick.catima_115.apk Length Date Time Name --------- ---------- ----- ---- + 55 1981-01-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties + 3900076 1981-01-01 01:01 classes.dex + 262912 1981-01-01 01:01 classes2.dex + 12940 1981-01-01 01:01 AndroidManifest.xml + 1481356 1981-01-01 01:01 resources.arsc 1714 1981-01-01 01:01 DebugProbesKt.bin 6 1981-01-01 01:01 META-INF/androidx.activity_activity-ktx.version 6 1981-01-01 01:01 META-INF/androidx.activity_activity.version @@ -54,6 +59,8 @@ 6 1981-01-01 01:01 META-INF/androidx.viewpager_viewpager.version 6 1981-01-01 01:01 META-INF/androidx.window_window.version 6 1981-01-01 01:01 META-INF/com.google.android.material_material.version + 54 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler + 52 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.internal.MainDispatcherFactory 926 1981-01-01 01:01 kotlin/annotation/annotation.kotlin_builtins 3685 1981-01-01 01:01 kotlin/collections/collections.kotlin_builtins 200 1981-01-01 01:01 kotlin/coroutines/coroutines.kotlin_builtins @@ -62,15 +69,8 @@ 2301 1981-01-01 01:01 kotlin/ranges/ranges.kotlin_builtins 2338 1981-01-01 01:01 kotlin/reflect/reflect.kotlin_builtins 34000 1981-01-01 01:01 okhttp3/internal/publicsuffix/publicsuffixes.gz - 54 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler - 52 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.internal.MainDispatcherFactory - 1481356 1981-01-01 01:01 resources.arsc - 12940 1981-01-01 01:01 AndroidManifest.xml - 55 1981-01-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties - 262912 1981-01-01 01:01 classes2.dex - 3900076 1981-01-01 01:01 classes.dexQuillpad: macOS vs Debian
NB: the position of
assets/dexopt/baseline.profdiffers between different builds on macOS.@@ -1,8 +1,12 @@ -Archive: quillpad-app-defaultFlavor-release.apk +Archive: io.github.quillpad_8-2.apk Length Date Time Name --------- ---------- ----- ---- - 732 1981-01-01 01:01 assets/dexopt/baseline.prof 55 1981-01-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties + 732 1981-01-01 01:01 assets/dexopt/baseline.prof + 5040432 1981-01-01 01:01 classes.dex + 363520 1981-01-01 01:01 classes2.dex + 12972 1981-01-01 01:01 AndroidManifest.xml + 1232792 1981-01-01 01:01 resources.arsc 1714 1981-01-01 01:01 DebugProbesKt.bin 6 1981-01-01 01:01 META-INF/androidx.activity_activity-ktx.version 6 1981-01-01 01:01 META-INF/androidx.activity_activity.version @@ -81,6 +85,8 @@ 5 1981-01-01 01:01 META-INF/com.google.dagger_dagger.version 5 1981-01-01 01:01 META-INF/com.google.dagger_hilt-android.version 5 1981-01-01 01:01 META-INF/com.google.dagger_hilt-core.version + 54 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler + 52 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.internal.MainDispatcherFactory 66 1981-01-01 01:01 build-data.properties 600 1981-01-01 01:01 kotlin-tooling-metadata.json 926 1981-01-01 01:01 kotlin/annotation/annotation.kotlin_builtins @@ -93,14 +99,5 @@ 218 1981-01-01 01:01 okhttp3/internal/publicsuffix/NOTICE 36752 1981-01-01 01:01 okhttp3/internal/publicsuffix/publicsuffixes.gz 24470 1981-01-01 01:01 org/commonmark/internal/util/entities.properties - 54 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler - 52 1981-01-01 01:01 META-INF/services/kotlinx.coroutines.internal.MainDispatcherFactory - 1232792 1981-01-01 01:01 resources.arsc - 12972 1981-01-01 01:01 AndroidManifest.xml - 363520 1981-01-01 01:01 classes2.dex - 5040432 1981-01-01 01:01 classes.dex- Edited by FC (Fay) Stegerman
Since building catima w/
disorderfschanges nothing about the order of the ZIP entries I think we can rule out the filesystem alone being the cause. Also: catima upstream APKs for
2.21.0and2.21.1have the exact same order of ZIP entries.So do the F-Droid APK for
2.21.0and the one I just built in CI for2.21.1.So Fedora differs from Debian, but the order for each is stable between different builds (unlike on macOS).
Maybe Debian is patching some library or java itself for reproducibility? That might explain differences w/ Fedora and macOS... but not Windows being the same.
@linsui are you sure sorting is actually needed for CopyManga? Because I realised the APK you built in CI matches the non-sorted one from upstream:
$ apksigcopier compare CopyManga_2.0.beta14.apk --unsigned top.fumiama.copymanga_26.apk && echo OK OKAnd RB works if I disable sorting and use the non-sorted upstream APK (obfusk/fdroiddata@f8fc15b9).
Also: the last build on the build server failed because upstream didn't sign the
_sorted.apk.The non-sorted APK is signed w/ signflinger, so maybe that was the issue and not the ZIP entry order?
@TheLastProject it would be interesting to see if building w/
disorderfson Fedora does change anything.- Edited by FC (Fay) Stegerman
Suggested: using RB for verification (but not publishing) during build.
sudo: - apt-get update - apt-get install -y wget - apt-get install -y -t bullseye-backports apksigner apksigcopier output: app/build/outputs/apk/release/app-release-unsigned.apk build: - gradle assembleRelease - wget -O upstream.apk https://github.com/CatimaLoyalty/Android/releases/download/v$$VERSION$$/app-release.apk - apksigcopier compare upstream.apk --unsigned app/build/outputs/apk/release/app-release-unsigned.apkNot sure if
stable-backportsis enabled by default (forapksigcopier);whether this requires settingoutput:;whetherapksigner(needed forcompare) is on$PATH(if not, justapt-getit, or usecopyandcmpinstead).
@TheLastProject @linsui I'm not sure we should be doing the RB verification like this tbh
I don't like downloading the upstream APK during build. It could theoretically be accidentally published if there's a bug in
fdroidserver(or the recipe), though downloading it to~or/tmpinstead of the build dir would make this a bit less likely. As would removing it after comparing. But still.An alternative would be for upstreams (in this case @TheLastProject) to publish extracted signatures as part of the release. Though that's also not ideal IMO.
We actually have no need for the upstream signature in this scenario, we're just making sure we built something identical, not publishing an APK signed by upstream.
My proposal would be for upstreams to publish the
sha256sumfor the unsigned APK as part of their release. We can then download that and compare it to the APK built by us. The unsigned APKs should be bit-by-bit identical after all. And @TheLastProject is going to build on the CLI and sign manually anyway, so this small extra step should be easy to add IMO.Of course, we can use the
apksigcopiervariant when upstream can't or won't provide thesha256sumof the unsigned APK for some reason and we still want to verify the build is identical.So I propose we'd use something like this instead:
output: app/build/outputs/apk/release/app-release-unsigned.apk build: - gradle assembleRelease - sha256sum="$( curl -L https://github.com/CatimaLoyalty/Android/releases/download/v$$VERSION$$/SHA256SUM | cut -c-64 | tr -cd '0-9a-f' )" - sha256sum -c <<< "$sha256sum app/build/outputs/apk/release/app-release-unsigned.apk"
I'd rather not upload the unsigned APK to GitHub because I just know some people are going to try to install it and complain it doesn't work.
Uploading the SHA could work with some kind of name to keep people from trying to use it on the signed APK. Is there already some "default" name in the Reproducible Builds community for the checksum of an unsigned build to keep end-users from confusing it with the checksum they'll want to use (which is the one for the signed build)?
I'd rather not upload the unsigned APK to GitHub because I just know some people are going to try to install it and complain it doesn't work.
Agreed. Wasn't suggesting that :)
Is there already some "default" name in the Reproducible Builds community for the checksum of an unsigned build to keep end-users from confusing it with the checksum they'll want to use (which is the one for the signed build)
Not that I'm aware; someone can ask on rb-general if you really want to know :)
Of course, RB is a lot easier when signatures are not embedded like they are with APKs, so most projects use external signatures (when they can). In that case there's no such thing as signed build vs unsigned build.
You can call it something like
unsigned-apk-sha256sumif you prefer, but I would hope aSHA256SUMSfile like this:fb684187dfe04d185444d0a43fa282dd074abcd42e47006cf2dc1ca7ee78cb69 app-release-unsigned.apkwhich clearly provides the checksum for an unsigned APK would not be mistaken for anything else.
You can always add an entry for
app-release.apkas well:fb684187dfe04d185444d0a43fa282dd074abcd42e47006cf2dc1ca7ee78cb69 app-release-unsigned.apk 0ff630889d6833347000455ae1fc6f81f698b295da374968ff91ed8d51ea4047 app-release.apk
I haven't really looked at how exactly key rotation works yet and thus don't know whether it might allow us to switch existing apps to RB (though I suspect it would need admin work from Ciaran even if it does).
I'll have to investigate key rotation eventually since I want to support it in
apksigtool, but it might take a while before I get to it.- Edited by FC (Fay) Stegerman
Edit: fixed now :)
Eh... so we now have 4 RB apps that build OK but will not be published until the signing server is updated because they usesignflinger:com.github.bmx666.appcachecleanercom.rafapps.earthviewformuzeiio.github.quillpadorg.joinmastodon.android
- Edited by FC (Fay) Stegerman
We could ask upstreams to temporarily sign w/o v1 (just v2/v3) as a workaround if it doesn't seem the signing server will be updated soon. That should allow the apps to be published. Though it means they can't be used on android < 7. They might be more willing to do that than switch to
apksigner.
I used this bash one-liner to identify APKs v1-signed w/
signflinger.for apk in *.apk; do test "$( apksigtool parse-v1 --json "$apk" 2>/dev/null | jq -r .manifest.built_by )" = Signflinger && echo "$apk"; doneOddly, it listed
CopyManga_2.0.beta14_sorted.apkas well, butapksigcopier extractdoesn't show adifferences.json. Not sure what happened there tbh, but it should publish fine then.
Overview of apps using reproducible builds: #2844 (closed)
@linsui for NewPipe I thought we could do something like this in the recipe (instead of modifying
fdroidserver):sha256sum="$( curl -L https://github.com/TeamNewPipe/NewPipe/releases/download/v$$VERSION$$/SHA256SUM | cut -c-64 | tr -cd '0-9a-f' )" identical=no for i in {1..6}; do gradle clean assembleRelease if sha256sum -c <<< "$sha256sum app/build/outputs/apk/release/app-release-unsigned.apk"; then identical=yes break fi done test "$identical" = yesThat might allow it to be published in 1 build cycle. Though we should obviously only use this for apps that build quickly enough that we can afford the extra time from retrying.
NB: I'm assuming here they'd also be willing & able to publish the
sha256sumof the unsigned APK, otherwise we can useapksigcopier.- Edited by FC (Fay) Stegerman
For NewPipe we publish both apks, one signed by F-Droid and the other signed by upstream.
Not atm, signatures are no longer being added to
fdroiddata. But yes, that was the method used.Good point!
I don't think metadata signatures are available in the build VM though, so using the
sha256suminstead ofapksigcopierwould probably still be preferable.I don't know if the apk signed by F-Droid will be published if the verification fails.
No. It's in the flow chart from the RB docs
- Edited by FC (Fay) Stegerman
Working example of retrying + verifying RB:
- versionName: 0.1.7 versionCode: 8 commit: 52c0633a68f4d89dd6df2d4c0bbfbf9ff269d828 subdir: app sudo: - apt-get update - apt-get install -y wget - apt-get install -y -t bullseye-backports apksigcopier output: build/outputs/apk/release/app-release-unsigned.apk build: - unsigned_apk=build/outputs/apk/release/app-release-unsigned.apk - wget -O ~/upstream.apk https://github.com/jroddev/android-oss-release-tracker/releases/download/$$VERSION$$/android-oss-release-tracker-release-v$$VERSION$$.apk - sha256sum_signed="$( sha256sum ~/upstream.apk | cut -c-64 )" - mkdir ~/extracted_meta - apksigcopier extract ~/upstream.apk ~/extracted_meta - rm ~/upstream.apk - reproducible=no - for i in {1..6} - do gradle clean assembleRelease - sha256sum "$unsigned_apk" - if apksigcopier patch ~/extracted_meta "$unsigned_apk" ~/patched.apk - then if sha256sum -c <<< "$sha256sum_signed $HOME/patched.apk" - then reproducible=yes - break - fi - fi - rm -f ~/patched.apk - done - rm -fr ~/extracted_meta ~/patched.apk - test "$reproducible" = yesSee !12251 (closed)
To rotate keys, apksigner[0] seams to need both keystores. If the f-droid app-specific-key won't be used again/elsewhere, it could be send to the developer to rotate keys. (Probably not a good solution from a security standpoint.) Otherwise it is probably doable with a custom script and sending back and forth a few files, but I think it needs to be done for every new app version (at least if people should be able to upgrade from every old version).
[0] https://developer.android.com/studio/command-line/apksigner#usage-rotate
Thanks!
I still haven't looked into all the details, but based on a quick look:
Key rotation is possible with APK Signature Scheme v3 (Android >= 9), but
apksignerindeed seems to require access to both keystores (i.e. both private keys) every time you sign an APK.However, that may be a limitation in
apksigner, not in the v3 scheme itself.I think the v3 scheme might allow us to construct a proof of rotation saying upstream's key replaces ours using only our private key and their certificate (which is public).
I do not immediately see how generating that proof requires access to both keys (it signs the new cert using the old key). Or why it could not be re-used to sign multiple APKs.
So hopefully we could create such a proof (using the F-Droid private key for that app to sign upstream's cert) and give it to upstream so they can sign APKs that Android (>= 9) will accept as updates for APKs signed with our key (except that
apksignerdoesn't seem to support this).If indeed the v3 scheme actually allows that, I might be able to support that use case in
apksigtool, but that won't be any time soon.
If it is not possible to sign with the new priv key then the old priv key in order to rotate the keys : a solution may be to create a shared key for dev and fdroid for the transition. FDroid sign the apk with fdroid key and shared key then the dev sign with shared key and dev key. (I haven't tested)
build-tools31.0.0has brokenzipalign: !12459 (comment 1249610931)- Edited by FC (Fay) Stegerman
Looks like
31.0.0and32.0.0are broken:b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-debian-package.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-29.0.2.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-29.0.3.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-30.0.0.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-30.0.1.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-30.0.2.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-30.0.3.apk cd58b36cf94315c461e3c5f60cf3e5a471a8147d zipaligned-with-build-tools-31.0.0.apk cd58b36cf94315c461e3c5f60cf3e5a471a8147d zipaligned-with-build-tools-32.0.0.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-32.1.0-rc1.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-33.0.0.apk b18e63d662cf493ff79fccfcbf3708e0260ade08 zipaligned-with-build-tools-33.0.1.apkThey are adding 4 bytes of zero padding to several files where none is needed:
entry resources.arsc: - extra (entry)=b'\x00\x00\x00\x00' + extra (entry)=b'' So we should
apt install -y zipaligninstead when RB w/reproducible-apk-toolsfails because of those versions.There should be a line in the build log showing which
zipalignwas used if it's using the latestbuild-toolsit can find in the SDK because it's not on$PATH:[FOUND] /opt/android-sdk/build-tools/31.0.0/zipalign
Yupp. And as said in another thread: I happily share the notes made during our journey. I'll try a few more RBs the next days I guess to settle what I've learned and iron out my notes; feel free to remind me when and where I should put them (including my little Bash function to automate local checks).
@IzzySoft I slightly modified your function and put it in a snippet: https://gitlab.com/-/snippets/2487899
- Edited by Izzy
@obfusk looks a bit compacter
/usr/bin/diffhere as I have set updiffas an alias to a script not supporting your special parameters ;) (git diff --no-indexwhich in turn usesdiff-so-fancyas a pager; much easier to read but then does not support all parameters ofdiff
I've blocklisted those versions in
reproducible-apk-toolsnow.Just understood that a day later. "blocklisted" is a nice term and much clearer than "blacklisted" – retroperspectively
Does it means that if the upstream apk is built with those build-tools we also need to use them.
I don't think so; at least I don't think
zipflingeruseszipalignfrombuild-tools, but does the zip alignment itself (and doesn't have the bug those versions ofzipaligndo).So there might be rare cases where we need to use the broken
zipalignbecause it's used by upstream, but I don't think that should be the default even if those versions ofbuild-toolsare used. We'll need real-world data to be sure how common that is.I guess the blocklist works
From !12495 (merged):
[SKIP BROKEN] 31.0.0 [FOUND] /opt/android-sdk/build-tools/30.0.3/zipalign
@IzzySoft I added an
apkdifffunction to the snippet to unpack the APKs in a temporary directory anddiff -rthem.So you can investigate whether something is a CRLF issue or not :)
$ apkdiff SimpleMP-Beta-1.2.apk com.lighttigerxiv.simple.mp_4.apk diff -r /tmp/tmp.kZNPOqpicF/a/META-INF/androidx.databinding_viewbinding.version /tmp/tmp.kZNPOqpicF/b/META-INF/androidx.databinding_viewbinding.version 1c1 < 7.4.0 \ No newline at end of file --- > 7.3.1 \ No newline at end of file diff -r /tmp/tmp.kZNPOqpicF/a/META-INF/com/android/build/gradle/app-metadata.properties /tmp/tmp.kZNPOqpicF/b/META-INF/com/android/build/gradle/app-metadata.properties 2c2 < androidGradlePluginVersion=7.4.0 --- > androidGradlePluginVersion=7.3.1 Binary files /tmp/tmp.kZNPOqpicF/a/assets/dexopt/baseline.prof and /tmp/tmp.kZNPOqpicF/b/assets/dexopt/baseline.prof differ Binary files /tmp/tmp.kZNPOqpicF/a/assets/dexopt/baseline.profm and /tmp/tmp.kZNPOqpicF/b/assets/dexopt/baseline.profm differ Binary files /tmp/tmp.kZNPOqpicF/a/classes.dex and /tmp/tmp.kZNPOqpicF/b/classes.dex differ Binary files /tmp/tmp.kZNPOqpicF/a/classes2.dex and /tmp/tmp.kZNPOqpicF/b/classes2.dex differ Binary files /tmp/tmp.kZNPOqpicF/a/classes3.dex and /tmp/tmp.kZNPOqpicF/b/classes3.dex differ diff -r /tmp/tmp.kZNPOqpicF/a/kotlin-tooling-metadata.json /tmp/tmp.kZNPOqpicF/b/kotlin-tooling-metadata.json 4c4 < "buildSystemVersion": "7.5", --- > "buildSystemVersion": "7.4",Not sure why you need
/usr/bin/diff. It works for me even if I aliasdiffto something else.Thanks, added here as well! And I have no idea why I have to use
/usr/bin/diff. Leaving out the path didn't work for me. Maybe the full construct gives you a clue, let me check:$ alias diff alias diff='git diff --no-index' $ cat ~/.gitconfig[core] pager = diff-so-fancy | less --tabs=4 -RFX [diff-so-fancy] stripLeadingSymbols = falseIf I understand that correctly, the alias just routes everything trhough to
git diff(works with other aliases as well, no need to specify$@) – so it would run that and pass the output togit-so-fancy. The only place the parameters might be not understood would begit diffthen. You probably have a newer version of that and those options were added there already; I have 2.25.1 running here.The difference is the order in which I quickly tried it:
$ function foo() { diff -r "$1" "$2"; } $ alias diff=echo $ foo X Y [diff output]$ alias diff=echo $ function foo() { diff -r "$1" "$2"; } $ foo X Y -r X YFYI: you can avoid using the alias by using
\diff, no need for/usr/bin/diff:)- Edited by FC (Fay) Stegerman
You can also turn it into a script in e.g.
~/bininstead of a function.Separate scripts don't use aliases from the calling shell.
\diffAh, that it was. I knew there was a shortcut, didn't remember which – and instead of searching for it (what terms?) decided to simply put the full path
Separate scripts don't use aliases from the calling shell.
Doesn't help unless they use a different shell. Aliases are defined via
~/.bashrc, so scripts would pick them up. But 2 good ideas, thanks: escaping, or change the order when defining.Aliases are defined via
~/.bashrc, so scripts would pick them up.That should not be the case. Aliases should never be defined in non-interactive shells; which is why most
.bashrcfiles contain something like this at the top:# If not running interactively, don't do anything case $- in *i*) ;; *) return;; esac
JDK differences (?!)
IMO we need to make sure people are consistently using (Open)JDK 11 in Android Studio for RB.
TaskManager v1.1 vs CI
Using
dexdump -a -d -f -handfix-dexdump.shI get:--- TaskManager-v1.1.dump 2023-01-31 01:36:52.993107385 +0100 +++ com.nima.taskmanager_2.dump 2023-01-31 01:36:53.589111651 +0100 @@ -1,27 +1,27 @@ -Processing 'TaskManager-v1.1/classes.dex'... -Opened 'TaskManager-v1.1/classes.dex', DEX version '037' +Processing 'com.nima.taskmanager_2/classes.dex'... +Opened 'com.nima.taskmanager_2/classes.dex', DEX version '037' DEX file header: magic : 'dex\n037\0' -checksum : 2d5189ea -signature : 07f1...cfba -file_size : 11449480 +checksum : 07cce728 +signature : af70...2f24 +file_size : 11449472 header_size : 112 link_size : 0 link_off : 0 (0x000000) -string_ids_size : 64362 +string_ids_size : 64361 string_ids_off : 112 (0x000070) type_ids_size : 10168 -type_ids_off : 257560 (0x03ee18) +type_ids_off : 257556 (0x03ee14) proto_ids_size : 14275 -proto_ids_off : 298232 (0x048cf8) +proto_ids_off : 298228 (0x048cf4) field_ids_size : 22027 -field_ids_off : 469532 (0x072a1c) +field_ids_off : 469528 (0x072a18) method_ids_size : 61701 -method_ids_off : 645748 (0x09da74) +method_ids_off : 645744 (0x09da70) class_defs_size : 9006 -class_defs_off : 1139356 (0x11629c) -data_size : 10021932 -data_off : 1427548 (0x15c85c) +class_defs_off : 1139352 (0x116298) +data_size : 10021928 +data_off : 1427544 (0x15c858) access_flags : 1537 (0x0601) static_fields_size : 0 @@ -1112599,24 +1112599,24 @@ VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Landroidx/room/EntityInsertionAdapter<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Landroidx/room/EntityDeletionOrUpdateAdapter<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 0 } names={ "__db" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "note" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "note" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lcom/nima/taskmanager/data/Note;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lcom/nima/taskmanager/data/Task;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "id" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "id" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Ljava/lang/String;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 } names={ "arg0" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 } names={ "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "note" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "note" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lcom/nima/taskmanager/data/Note;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lcom/nima/taskmanager/data/Task;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Lkotlinx/coroutines/flow/Flow<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Task;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Lkotlinx/coroutines/flow/Flow<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Note;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Ljava/util/List<" "Ljava/lang/Class<" "*>;>;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "id" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "id" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Ljava/lang/String;" "Lkotlin/coroutines/Continuation<" "-" "Lcom/nima/taskmanager/data/Task;" ">;)" "Ljava/lang/Object;" } - VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "arg1" } + VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 16 16 } names={ "task" "continuation" } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(" "Lcom/nima/taskmanager/data/Task;" "Lkotlin/coroutines/Continuation<" "-" "Lkotlin/Unit;" ">;)" "Ljava/lang/Object;" } Class descriptor : 'Lcom/nima/taskmanager/data/TaskDao_Impl;'TaskManager v1.1.1: OpenJDK 11 vs 17
Fixed
dexdump:--- com.nima.taskmanager_3.dump 2023-01-31 01:53:27.820230456 +0100 +++ com.nima.taskmanager_3-jdk17.dump 2023-01-31 01:53:27.312226819 +0100 @@ -1,10 +1,10 @@ -Processing 'com.nima.taskmanager_3/classes.dex'... -Opened 'com.nima.taskmanager_3/classes.dex', DEX version '037' +Processing 'com.nima.taskmanager_3-jdk17/classes.dex'... +Opened 'com.nima.taskmanager_3-jdk17/classes.dex', DEX version '037' DEX file header: magic : 'dex\n037\0' -checksum : ebfa81f3 -signature : 39f6...e136 -file_size : 11452960 +checksum : fa9dcf47 +signature : 7431...c37b +file_size : 11452976 header_size : 112 link_size : 0 link_off : 0 (0x000000) @@ -20,7 +20,7 @@ method_ids_off : 646020 (0x09db84) class_defs_size : 9021 class_defs_off : 1139660 (0x1163cc) -data_size : 10024628 +data_size : 10024644 data_off : 1428332 (0x15cb6c) access_flags : 1537 (0x0601) @@ -755651,7 +755651,7 @@ direct_methods_size : 2 virtual_methods_size: 0 - VISIBILITY_SYSTEM Ldalvik/annotation/MemberClasses; value={ Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$SingletonCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$Builder; } + VISIBILITY_SYSTEM Ldalvik/annotation/MemberClasses; value={ Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$Builder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$SingletonCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCBuilder; } Class descriptor : 'Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC;' Access flags : 0x0011 (PUBLIC FINAL) @@ -759718,6 +759718,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$note" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -759830,6 +759831,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -759942,6 +759944,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760169,6 +760172,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$id" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760296,6 +760300,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Task;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760497,6 +760502,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Note;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Note;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760667,6 +760673,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760845,6 +760852,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760957,6 +760965,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$note" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; }TaskManager v1.1: OpenJDK 11 vs 17
Fixed
dexdump:--- com.nima.taskmanager_2.dump 2023-01-31 01:44:42.744470837 +0100 +++ com.nima.taskmanager_2-jdk17.dump 2023-01-31 01:44:42.208467000 +0100 @@ -1,10 +1,10 @@ -Processing 'com.nima.taskmanager_2/classes.dex'... -Opened 'com.nima.taskmanager_2/classes.dex', DEX version '037' +Processing 'com.nima.taskmanager_2-jdk17/classes.dex'... +Opened 'com.nima.taskmanager_2-jdk17/classes.dex', DEX version '037' DEX file header: magic : 'dex\n037\0' -checksum : 07cce728 -signature : af70...2f24 -file_size : 11449472 +checksum : 7047b0a0 +signature : 3413...bd50 +file_size : 11449488 header_size : 112 link_size : 0 link_off : 0 (0x000000) @@ -20,7 +20,7 @@ method_ids_off : 645744 (0x09da70) class_defs_size : 9006 class_defs_off : 1139352 (0x116298) -data_size : 10021928 +data_size : 10021944 data_off : 1427544 (0x15c858) access_flags : 1537 (0x0601) @@ -755431,7 +755431,7 @@ direct_methods_size : 2 virtual_methods_size: 0 - VISIBILITY_SYSTEM Ldalvik/annotation/MemberClasses; value={ Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$SingletonCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$Builder; } + VISIBILITY_SYSTEM Ldalvik/annotation/MemberClasses; value={ Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$Builder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$SingletonCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCImpl; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ServiceCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewModelCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ViewWithFragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$FragmentCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityCBuilder; Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC$ActivityRetainedCBuilder; } Class descriptor : 'Lcom/nima/taskmanager/DaggerTaskManagerApplication_HiltComponents_SingletonC;' Access flags : 0x0011 (PUBLIC FINAL) @@ -759498,6 +759498,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$note" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -759610,6 +759611,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -759722,6 +759724,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -759949,6 +759952,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$id" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760076,6 +760080,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Task;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760277,6 +760282,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Note;" ">;>;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()" "Ljava/util/List<" "Lcom/nima/taskmanager/data/Note;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760447,6 +760453,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lcom/nima/taskmanager/data/Task;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$_statement" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760625,6 +760632,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$task" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } @@ -760737,6 +760745,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=0 name=null VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/lang/Object;" "Ljava/util/concurrent/Callable<" "Lkotlin/Unit;" ">;" } VISIBILITY_SYSTEM Ldalvik/annotation/MethodParameters; accessFlags={ 32784 4112 } names={ "this$0" "val$note" } + VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "()V" } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; } VISIBILITY_SYSTEM Ldalvik/annotation/Throws; value={ Ljava/lang/Exception; }WikiAnime v1.0.1: OpenJDK 11 vs 17
Interestingly, CI builds of WikiAnime are identical with OpenJDK 11 and OpenJDK 17.
I haven't tested OpenJDK 8.Fix (?!)
RE https://github.com/NimaKhajehpour/TaskManager/issues/8
I'm not sure why this worked tbh, since it doesn't actually change the JDK used to compile the app.
It does change the minimum version required, so it could be that Android Studio was using Java 8 and changed to 11.
diff --git a/app/build.gradle b/app/build.gradle index 98cf071..b22a938 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -55,11 +55,11 @@ android { } } compileOptions { - sourceCompatibility JavaVersion.VERSION_1_8 - targetCompatibility JavaVersion.VERSION_1_8 + sourceCompatibility JavaVersion.VERSION_11 + targetCompatibility JavaVersion.VERSION_11 } kotlinOptions { - jvmTarget = '1.8' + jvmTarget = '11' } buildFeatures { compose trueUpstream said
so the jdk is 1.8
Which doesn't seem to have been possible, since I cannot build the commit for which RB failed with OpenJDK 8 in CI:
java.lang.UnsupportedClassVersionError: dagger/hilt/android/plugin/HiltGradlePlugin has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0Nowadays Android Studio defaults to using Amazon Corretto. Might that have been related? https://aws.amazon.com/corretto/
(I don't know how relevant it is, just leaving this here in case it might be relevant information)
- Edited by FC (Fay) Stegerman
Unless you explicitly define a different toolchain (which I haven't seen anyone do so far), Gradle "uses the same Java version for running Gradle itself and building JVM projects".
Note that
sourceCompatibility/targetCompatibilityare about "which language version of Java your source files should be treated as" and "the version of byte code the compiler generates", not which JDK version is used to compile the project (though obviously you can't compile it with a lower version than used in those options).
- Edited by p
Extra data in zip entry
I built one APK manually with
gradlew assembleFreeReleaseand one APK using a similar process but withfdroid build. This is the only difference:$ ./repro_apk/diff_zip_meta.py ../sigcp_tk.hack5.treblecheck_5000006.apk ~/Code/StudioProjects/TrebleCheck/app/build/outputs/apk/free/release/app-free-release.apk --- ../sigcp_tk.hack5.treblecheck_5000006.apk +++ /home/penn/Code/StudioProjects/TrebleCheck/app/build/outputs/apk/free/release/app-free-release.apk entry lib/arm64-v8a/libbinderdetector.so: extra (entry): - 35d9b00000100000000000000000000000000000000000000000000000000000000000000000 + 0000000000000000000000000000000000000000000000000000000000000000000000000000 $So close and yet so far... !12522 (comment 1258858871)
Sounds like this. #2816 (comment 1249740966)
- Edited by p
Sign what exactly? I tried signing the unsigned APK but it doesn't seem to have been zipaligned by AGP, so it doesn't match up at all. I tried to zipalign (before resigning) but it didn't seem to do anything.
IMHO the issue is in https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/apksigcopier.py.
- Edited by p
Input header:
b'PK\x03\x04\x00\x00\x00\x00\x00\x00!\x08!\x02\xe6\xa36\x08\x80\x11\x00\x00\x80\x11\x00\x00"\x008\x01lib/arm64-v8a/libbinderdetector.so\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'Aligned header output from apksigcopier.py:
b'PK\x03\x04\x00\x00\x00\x00\x00\x00!\x08!\x02\xe6\xa36\x08\x80\x11\x00\x00\x80\x11\x00\x00"\x00\xb4\x00lib/arm64-v8a/libbinderdetector.so5\xd9\xb0\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'At https://cs.android.com/android/platform/superproject/+/master:build/make/tools/zipalign/ZipEntry.cpp;l=211?q=extra&ss=android%2Fplatform%2Fsuperproject:build%2Fmake%2Ftools%2Fzipalign%2F we can see that zipalign simply adds zeros to the end of the extra field to make it the right size. However, https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/apksigcopier.py#L382 adds some "D935" field to the extras. I don't understand why, and it appears to be completely undocumented.
I've opened https://github.com/obfusk/apksigcopier/issues/84.
adds some "D935" field to the extras. I don't understand why, and it appears to be completely undocumented.
Because that's what
apksignerdoes.Unfortunately, it produces wrong results for APKs signed by Gradle that have both a virtual entry and contain a
.sofile (because the virtual entry is not present in the unsigned APK and thus the.sogets realigned when it it added).Should now be fixed on the
fixesbranch.You can build an unsigned apk and sign it with apksigner.
For devs that don't want to sign manually with
apksigner:You can pretty easily integrate
apksignerwithgradle; you'd have to adapt that code a bit as it's meant to re-sign after modifying the APK.Another option available right now would be to add the annoying virtual entry causing all this to the unsigned APK during the F-Droid build; that script currently requires
apksigcopierfrom thefixesbranch (or thev1.1.1I'm hoping to release soon -- but no guarantees), but that's not a problem in the F-Droid build recipe, it's just the F-Droid signing server that won't be updated soon.
@linsui @IzzySoft @licaon-kter I released
reproducible-apk-toolsv0.2.2just now.As @linsui pointed out, "old androdns is not reproducible because some apis are deprecated".
Build log: https://gitlab.com/linsui/fdroiddata/-/jobs/3715446881
Diff:
@@ -2,9 +2,9 @@ Opened 'classes.dex', DEX version '035' DEX file header: magic : 'dex\n035\0' -checksum : 71f52cbf -signature : f4cb...a8ec -file_size : 2909052 +checksum : 43ae0358 +signature : d985...541c +file_size : 2909080 header_size : 112 link_size : 0 link_off : 0 (0x000000) @@ -20,7 +20,7 @@ method_ids_off : 310080 (0x04bb40) class_defs_size : 2206 class_defs_off : 502264 (0x07a9f8) -data_size : 2336196 +data_size : 2336224 data_off : 572856 (0x08bdb8) access_flags : 1 (0x0001) @@ -3398,6 +3398,7 @@ VISIBILITY_SYSTEM Ldalvik/annotation/EnclosingClass; value=Landrodns/android/leetdreams/ch/androdns/R; VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=25 name="attr" + VISIBILITY_RUNTIME Ljava/lang/Deprecated; Class descriptor : 'Landrodns/android/leetdreams/ch/androdns/R$attr;' Access flags : 0x0011 (PUBLIC FINAL) @@ -12948,6 +12949,8 @@ VISIBILITY_SYSTEM Ldalvik/annotation/EnclosingClass; value=Landrodns/android/leetdreams/ch/androdns/R; VISIBILITY_SYSTEM Ldalvik/annotation/InnerClass; accessFlags=25 name="styleable" + VISIBILITY_RUNTIME Ljava/lang/Deprecated; + VISIBILITY_RUNTIME Ljava/lang/Deprecated; Class descriptor : 'Landrodns/android/leetdreams/ch/androdns/R$styleable;' Access flags : 0x0011 (PUBLIC FINAL)I tried building with
Android SDK Platform 28 (revision: 4)instead ofAndroid SDK Platform 28 (revision: 6), but that also fails.Build log: https://gitlab.com/obfusk/fdroiddata/-/jobs/3715961564
I would very much like to know where it's getting current info on what's deprecated even though we seem to be using all the old versions of the dependencies.
Which I assume would not mark it as deprecated as it presumably wasn't when they were current.
I implemented my own
zipinfoin Python: https://github.com/obfusk/reproducible-apk-tools#zipinfopyWith the
-eoption (that the original doesn't have), it shows the CRC32 as well.Which is very useful for RB, since we can now replace this hack:
$ diff -Naur <( zipinfo -v "$upstreamAPK" | grep -E 'CRC|^ [^: ]*$' ) \ <( zipinfo -v "$fdroidAPK" | grep -E 'CRC|^ [^: ]*$' ) @@ -1233,7 +1233,7 @@ res/iw1.xml 32-bit CRC value (hex): 5b72ebd5 res/iz.json - 32-bit CRC value (hex): 70461ebb + 32-bit CRC value (hex): 0be1a128 res/j-.9.png 32-bit CRC value (hex): d42327b8 res/j9.9.png @@ -1719,9 +1719,9 @@ META-INF/kotlinx_coroutines_core.version 32-bit CRC value (hex): 10dbda9d META-INF/services/j8.x - 32-bit CRC value (hex): 1c65fd6d + 32-bit CRC value (hex): 9892eade META-INF/services/kotlinx.coroutines.internal.k - 32-bit CRC value (hex): 1e234334 + 32-bit CRC value (hex): b3bfb91d kotlin-tooling-metadata.json 32-bit CRC value (hex): f19c6b7f kotlin/annotation/annotation.kotlin_builtinswith this (note that
res/iz.jsononly differs in CRC32 here):$ diff -Naur <( repro-apk zipinfo -e "$upstreamAPK" ) <( repro-apk zipinfo -e "$fdroidAPK" ) @@ -1,5 +1,5 @@ -Archive: Myne-v1.1.0.apk -Zip file size: 3956506 bytes, number of entries: 872 +Archive: com.starry.myne_2.apk +Zip file size: 3944216 bytes, number of entries: 872 -rw-r--r-- 0.0 unx 56 b- 52 defN 1981-01-01 01:01:02 e3c609fd META-INF/com/android/build/gradle/app-metadata.properties -rw-r--r-- 0.0 unx 6832 b- 6832 stor 1981-01-01 01:01:02 bf491634 assets/dexopt/baseline.prof -rw-r--r-- 0.0 unx 622 b- 622 stor 1981-01-01 01:01:02 9c7cdba8 assets/dexopt/baseline.profm @@ -617,7 +617,7 @@ -rw---- 0.0 fat 178 b- 178 stor 1981-01-01 01:01:02 c56f26da res/ii.9.png -rw---- 0.0 fat 624 b- 323 defN 1981-01-01 01:01:02 7acf7c87 res/iw.xml -rw---- 0.0 fat 796 b- 435 defN 1981-01-01 01:01:02 5b72ebd5 res/iw1.xml --rw---- 0.0 fat 74139 b- 9762 defN 1981-01-01 01:01:02 70461ebb res/iz.json +-rw---- 0.0 fat 74139 b- 9762 defN 1981-01-01 01:01:02 0be1a128 res/iz.json -rw---- 0.0 fat 184 b- 184 stor 1981-01-01 01:01:02 d42327b8 res/j-.9.png -rw---- 0.0 fat 194 b- 194 stor 1981-01-01 01:01:02 822df711 res/j9.9.png -rw---- 0.0 fat 2800 b- 1006 defN 1981-01-01 01:01:02 d9ab006c res/j9.xml @@ -860,8 +860,8 @@ -rw---- 2.0 fat 7 b- 9 defN 1981-01-01 01:01:02 8ba8a876 META-INF/com.google.dagger_hilt-core.version -rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 10dbda9d META-INF/kotlinx_coroutines_android.version -rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 10dbda9d META-INF/kotlinx_coroutines_core.version --rw---- 2.0 fat 6 b- 8 defN 1981-01-01 01:01:02 1c65fd6d META-INF/services/j8.x --rw---- 2.0 fat 6 b- 8 defN 1981-01-01 01:01:02 1e234334 META-INF/services/kotlinx.coroutines.internal.k +-rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 9892eade META-INF/services/j8.x +-rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 b3bfb91d META-INF/services/kotlinx.coroutines.internal.k -rw---- 2.0 fat 626 b- 280 defN 1981-01-01 01:01:02 f19c6b7f kotlin-tooling-metadata.json -rw---- 2.0 fat 926 b- 559 defN 1981-01-01 01:01:02 d36c8b5b kotlin/annotation/annotation.kotlin_builtins -rw---- 2.0 fat 3685 b- 1521 defN 1981-01-01 01:01:02 f41c30cf kotlin/collections/collections.kotlin_builtins @@ -872,4 +872,4 @@ -rw---- 2.0 fat 2395 b- 1265 defN 1981-01-01 01:01:02 13816860 kotlin/reflect/reflect.kotlin_builtins -rw---- 2.0 fat 218 b- 149 defN 1981-01-01 01:01:02 df32c7fc okhttp3/internal/publicsuffix/NOTICE -rw---- 2.0 fat 37730 b- 37745 defN 1981-01-01 01:01:02 e33f7b7e okhttp3/internal/publicsuffix/publicsuffixes.gz -872 files, 6838492 bytes uncompressed, 3839158 bytes compressed: 43.9% +872 files, 6838490 bytes uncompressed, 3839156 bytes compressed: 43.9%I may have found a bug in the original
zipinfo: https://github.com/obfusk/reproducible-apk-tools/issues/9
Was looking at the PNG stuff on https://f-droid.org/docs/Reproducible_Builds/#potential-sources-of-unreproducible-builds for a18913af
But both methods break compilation.
Script compilation error: Line 12: aaptOptions { cruncherEnabled = false } ^ Unresolved reference: cruncherEnabled 1 errorand
Script compilation errors: Line 15: vectorDrawables.generatedDensities = [] ^ Val cannot be reassigned Line 15: vectorDrawables.generatedDensities = [] ^ Type mismatch: inferred type is Array<???> but MutableSet<String>? was expected Line 15: vectorDrawables.generatedDensities = [] ^ Unsupported [Collection literals outside of annotations] 3 errors...so I need to set something else besides these?
Also although we use repro-tools to fix newlines, they still error out for
kotlinx.coroutines.android.AndroidDispatcherFactoryandkotlinx.coroutines.android.AndroidExceptionPreHandlerUpdated recipe:
- versionName: '1.24' versionCode: 25 commit: 70e48b9db23b7ca515e8e96c5070e2957c42f3fc subdir: app gradle: - yes srclibs: - reproducible-apk-tools@v0.2.3 prebuild: - sed -e 's/\s\+"/implementation "/g' ../buildSrc/src/main/kotlin/com/example/util/simpletimetracker/Deps.kt > deps.gradle postbuild: $$reproducible-apk-tools$$/inplace-fix.py --zipalign fix-newlines $$OUT$$ 'META-INF/services/*'Confirmation that using a different JDK (jetbrains) can cause different .dex files even if it's the same version (11): !13099 (comment 1398050643)
Google RB bugs snippet: https://gitlab.com/-/snippets/2500252
Thanks!
If the incremental build produces different results, than that's a bug in Android Studio / AGP. Might be good to report that to Google if it happens with recent AGP (please ping me if you do).
So far, we've seen more RB issues caused by AS defaulting to using a different JDK than the OpenJDK 11/17 used by F-Droid builds. I don't think we've had a confirmed case of cleaning before building fixing RB yet (only some cases where it didn't help IIRC, and we do know that older AGP will cause ordering differences in the APK/ZIP when building with AS). But yeah, building from the command line or at least cleaning before building does reduce the chance of this happening.
- Edited by Licaon_Kter
eg. upstream build with 17, we tried 11 and failed, we build with 17 too, and success
/LE: upstream files still say 1.8: https://github.com/Razeeman/Android-SimpleTimeTracker/blob/v1.28/app/build.gradle.kts#L45-L46 but we've seen this "upgrade" pattern for months now where upstream jumps to 17 yet they don't even know it, because AS is doing it.
And if you scroll up two threads from this one:
Confirmation that using a different JDK (jetbrains) can cause different .dex files even if it's the same version (11): !13099 (comment 1398050643)
Can I run f-droid build tool (one that tries to make a reproducible build and checks it at end) locally on my machine ?
You could set up your own build server. There's a docker image for that too, but I haven't used it myself. Usually the
fdroiddatagitlab CI is close enough to the "real" build servers, and easier to use. You can test with that, even if you don't open an MR.If you want to manually check whether two APKs are reproducible, you can use
apksigcopier.
Another weird bug fixed by newer AGP: !13414 (comment 1502121334)
- Edited by FC (Fay) Stegerman
Unreproducible (as in doesn't match anything
zlibshould produce) compressed data :/ AS seems to require cleaning cache for RB increasingly often :/
https://github.com/lighttigerXIV/SimpleMP-Compose/issues/17#issuecomment-1706200807
