opengapps / apkcrawler Goto Github PK

The have the .33 versions of the docs.editors :-)

The playstore crawler is offered the correct vercode for each Chrome sdk24 apk for all CPU's

• arm: 278512402
• arm64: 278512452
• x86: 278512412
• x86_64: 278512462

... but when attempting to download each apk the request results in a HTTP500 error

cc: @therealssj

Per opengapps/opengapps#70,

@Lekensteyn 's tool (https://github.com/Lekensteyn/make-gapps-zip) could be used to extend apkcrawler's functionality to getting files from Nexus images.

apks.org often have multiple versions of each APK, often including less popular ones, downloading from their website could help getting a more complete APK set.

Running the playstorecrawler with the latest playstore APK (com.android.vending, ver=7.1.12.I-all [0] [PR], vercode=80711200 - link ) causes "message.payload.deliveryResponse.appDeliveryData.downloadAuthCookie" googleplay.py:327 to be None.

This propagates from googleplay.py:208 where, "message.payload.deliveryResponse.status" == 2

This code still works as expected when running against older playstore versions (com.android.vending, ver=7.0.18.H-all [0], vercode=80701800 - link )

cc: @therealssj

Since we now have the super package, and the AndroidTV package is in the works, the set of packages should probably be extended to cover these new apps :-)

VERY IMPORTANT AND CRITICAL BUG:
Currently hyphens crash the whole compare.
It should deal with ( and ) as found in e.g. GMS, Google Play Games and Google Camera.

Example URL with detailed version history: http://www.androidapksfree.com/version-history/google-photos-apk-version-history/

Should probably become part of apkdownload-helper

to improve matching of having and/or having not a certain specific version.

Are there plans to add f-droid support to the crawler?

Crawlers iterate over self.report.dAllApks.keys() to get all apkids they should search for, but this includes also .beta and .leanback.
We should make a function that returns the apkids list in a cleaned-up manner.

Also have sometimes rare APKs

download URLs look like: http://freeapkpro.co/file/download-maps-apk.apk

but often the downloads only result in zero bytes.

Some aptoide stores "die" because their user doesn't upload any new APKs anymore.
Maybe the crawler could indicate if a store has not had any updates for e.g. more than 2 months; suggesting to suspend it from our repo list. (we could e.g. comment the line with that store)

Enable the possibility to use (public) proxies during the crawling of aptoide, avoiding the delay limits (and skip the initial-delay request in that situation)

The intermediate output of apkhelper can also be useful to search for missing APKs on Google. Could we find a way to output this in a convenient way?

https://apkpure.com

NOTE: Some apps are in a new format called XAPK which combines the normal APK and OBB files into a zip file. Don't think any GApps have OBB but good to keep in mind that XAPK files can just be extracted to get the APK

To prevent crawling too many pages of the really big stores

For com.google.android.inputmethod.latin the architecture versioncode is now also part of the versionname. Meanwhile they did drop the architecture name itself from the versionname.
We should account for this variable part of the versionname (also in the opengapps build code)

Also we should keep an eye out if this will be applied to the other keyboard-apps too.

When downloading an APK to pass it to add_sourceapp.sh it would be useful to be able to let the APK be downloaded to e.g. a temporary location.

apk-dl.com often has a less popular version of the app.
Downloading from their website should be relatively trivial.

like http://apk.land/apk/google-docs-161521133

Is there any way to provide a bit more detailed tutorial of how to use this apk crawler? I would be glad if there is.

Maybe check out these projects, whether they would be useful:
https://github.com/gpp256/APK-Crawler
https://github.com/mssun/android-apps-crawler

Gets barely updates, so really low priority.

If an APK is ok:
malware.status:"scanned"
and
malware.reason.signature_validated.status:"passed"
malware.reason.signature_validated.signature_from:"market"
malware.reason.scanned.status:"passed"

But if:
malware.status:"warn" the APK is not signed

and more research is necessary:
malware.reason.signature_validated.status:"failed"
in combination with:
malware.reason.signature_validated.signature_from:"market"
means that the APK is incomplete and should simple be ignored at all cost

but if it is combined with:
malware.reason.signature_validated.signature_from:"user" the APK might be complete but from e.g. a factory image, but it can also be a completely fake signature.
In that case the APK can be saved to err.filename.apk.err and output only stderr with big warnings that this APK is untrusted and potentially really harmful, but that it could be an interesting app from e.g. an Android One device.

Is there any way to crawl different apks not only opengapps support apks?

@mfonville:
I found another source: http://google-drive.en.uptodown.com/android
They have a very simple download URL: http://google-drive.en.uptodown.com/android/download
and here they show all (old) versions: http://google-drive.en.uptodown.com/android/old

Example link: http://dlapks.com/com.google.android.apps.docs/

a new "to be downloaded"-url would be added to the apkdownload-helper instance;
the helper would only do 1 download at a time per host
e.g.:
so 1 at aptoide can be parallel with 1 at apk-dl
but not 2 at aptoide at the same time

It could work with passing the avi-object (that contains the URL that needs to be downloaded) and we could make optional fields like "Stream=False" and "Header=''" to pass the necessary tricks.

It might be a bit harder to parse since it is in Farsi, but they do offer often quite rare versions of the APKs.
Luckily the page URLs are very simple, with just the package name.
Full version info is provided on the page nd there is a regular download button. So it should be easy.

The last number after the dot but before the architecture is variable per sdk level; we should treat it as such to be future proof (this also applies to opengapps buildscripts)

Execute a query with the full identifying name of the package, eg:
http://helper.mgccw.com/nclient/sjson/detail/detailInfo.htm?apkId=com.google.android.apps.docs

You'll get some json data back, including a field apkPath and versionCode.

Within versionCode you can find the exact version of the APK.

Attach the value of apkPath to http://download.mgccw.com
This will give you the URL to download the APK.

APKs from the Nexus Play have been added to the repo thus starting the path towards leanback support by Open GApps. The apkcrawler needs to be update to support pulling these from various sources

When using Python 3, the script doesn't work.

Terminal Output

[bgiesing@bgiesing-TOSHIBA opengapps]$ ./add_sourceapp.sh $(python ~/opengapps/apkcrawler/apkcrawler.py docs)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
    msg = self.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
    return fmt.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 565, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.4/logging/__init__.py", line 326, in getMessage
    msg = str(self.msg)
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 79, in __str__
    import urlparse
ImportError: No module named 'urlparse'
Call stack:
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 387, in <module>
    main(sys.argv[1:])
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 350, in main
    logging.debug(ai)
Message: <__main__.ApkInfo object at 0x7fae73853e10>
Arguments: ()
Traceback (most recent call last):
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 387, in <module>
    main(sys.argv[1:])
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 353, in main
    maxVersionByName = sorted(ai.versions)[-1]
TypeError: unorderable types: ApkVersionInfo() < ApkVersionInfo()

Log

2015-09-04 20:59:44,815 INFO/main(11255): Fetching Information for: Docs
2015-09-04 21:03:36,701 INFO/main(11373): Fetching Information for: Docs
2015-09-04 21:05:48,403 INFO/main(11386): Fetching Information for: Docs
2015-09-04 21:16:48,791 DEBUG/main(11682): ['Docs']
2015-09-04 21:16:48,791 INFO/main(11682): Fetching Information for: Docs
2015-09-04 21:16:48,806 DEBUG/getAppVersions(11682): Requesting2: http://www.apkmirror.com/apk/google-inc/docs/
2015-09-04 21:16:50,072 DEBUG/printDictionary(11682): "Docs 1.4.332.17.30"         - /apk/google-inc/docs/docs-1-4-332-17-30-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.33"         - /apk/google-inc/docs/docs-1-4-332-17-33-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.34"         - /apk/google-inc/docs/docs-1-4-332-17-34-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.35"         - /apk/google-inc/docs/docs-1-4-332-17-35-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.36"         - /apk/google-inc/docs/docs-1-4-332-17-36-2-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.44 (arm64)" - /apk/google-inc/docs/docs-1-4-332-17-44-arm64-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.33"         - /apk/google-inc/docs/docs-1-4-352-08-33-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.34"         - /apk/google-inc/docs/docs-1-4-352-08-34-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.35"         - /apk/google-inc/docs/docs-1-4-352-08-35-3-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.09.30"         - /apk/google-inc/docs/docs-1-4-352-09-30-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.09.33"         - /apk/google-inc/docs/docs-1-4-352-09-33-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.34"         - /apk/google-inc/docs/docs-1-4-352-09-34-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.35"         - /apk/google-inc/docs/docs-1-4-352-09-35-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.36"         - /apk/google-inc/docs/docs-1-4-352-09-36-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.44 (arm64)" - /apk/google-inc/docs/docs-1-4-352-09-44-arm64-android-apk-download/

HTML

http://s000.tinyupload.com/index.php?file_id=39298603559286732019

@NicholasBuse @mfonville

http://www.apkmirror.com/apk/google-inc/google-play-services/ sorts by versioncode (not upload date) and only shows the highest 15 apks.

If all 15 have the same same version, then additionally crawl: http://www.apkmirror.com/uploads/?app=google-play-services

The 2nd page sorts by upload date but shows 30 apks

If a vername is provided for a certain (whitelist) of apps, it can safely be assumed that we know what the vercode would have no surprises, and that if the vername matches our local version the APK doesn't need to be downloaded.