flathub / linux-store-backend

Backend for linux-store-frontend

Note: this backend has been replaced in flathub.org by this one (Sept. 2020)

Linux Store backend

This is a simple Spring Boot App implementing a backend for linux-store-frontend:

• Implements a system to get the list of apps and runtimes published in flatpak repos like Flathub and others
• Stores the information in a relational database
• Offers a simple REST API to get this information

Requirements

For the backend:

• Java 7+ (java-8-oracle recommended)
• maven
• appstream-appdata-java compiled and present in your local maven repository

For the script appstream-extractor (info below):

• flatpak
• ostree
• tar
• unzip

How appstream information is obtained

The list of apps and runtimes published in a flatpak repo is obtained extracting the contents of the appstream branch.

This is done using the provided script appstream-extractor. It can be run like this:

# Add flathub repo
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

# Run the extractor script as root manually. 
# Once it works it can be run periodically in a cron job
 ./appstream-extractor 

The script will:

• Extract the appstream data to /var/lib/appstream-extractor/export-data
• Update the file /var/lib/appstream-extractor/appstream-extractor.info with information about the last exported data (ostree commit, ...)

linux-store-backend will parse this information periodically and store it in the database.

Testing linux-store-backend

You can run the backend using an in-memory database from the command line:

mvn install
mvn spring-boot:run -Dspring.profiles.active=TEST 

Once the service is running go to http://localhost:8080/v1/apps/update to import the appstream data exported with the script appdata-extractor.

Finally, fetch the data at http://localhost:8080/v1/apps

Improving linux-store-backend

This is just a proof of concept implemented with Spring Boot. Look at ApiController.java and follow the code to undertand how it works.

TODO:

• Improve appstream-extractor to manage multiple repos
• Import and store appdata translations
• Import and store appdata for all arches (arm, ...)
• Search apps by name, keywords, categories, ...
• Search themes
• Search runtimes
• Pagination
• Activity: events, ...
• Integration with GitHub to obtain extra info (publisher name and photo, ...)
• Improve performance (ehcache, ...)
• Rate Limiting

linux-store-backend's Issues

Expose runtime for apps in the API

Rational

We would like the know the number of apps that use a particular runtime (eg how many have transitioned to freedesktop 18.08). If the flathub api exposed the runtime and runtime version as a property in https://flathub.org/api/v1/apps/ then we could generate such charts.

Proposal

Add "runtime" and "runtime-version" fields to each app in https://flathub.org/api/v1/apps/ which could have the values "org.freedesktop.Platform" and "18.08".

exception updating appdata

Got this on the new deployment:

jan 22 14:23:45 webapps.flathub.org podman[5076]: 2019-01-22 14:23:45.431 ERROR 6 --- [pool-2-thread-1] o.flathub.api.service.UpdateServiceImpl  : Error updating info for app de.akaflieg_freiburg.cavok.add_hours_and_minutes
jan 22 14:23:45 webapps.flathub.org podman[5076]: java.lang.NullPointerException: null
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.freedesktop.appstream.AppdataComponent.findReleaseInfoByMostRecentAndLangIsDefault(AppdataComponent.java:595) ~[appstream-appdata-0.4.2.jar!/:na]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.service.UpdateServiceImpl.setReleaseInfo(UpdateServiceImpl.java:537) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.service.UpdateServiceImpl.updateAppInfo(UpdateServiceImpl.java:275) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.service.UpdateServiceImpl.updateRepoInfo(UpdateServiceImpl.java:212) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.service.UpdateServiceImpl.updateFlathubInfo(UpdateServiceImpl.java:110) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.scheduledtasks.ScheduledTasks.updateFlathubInfo(ScheduledTasks.java:40) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.scheduledtasks.ScheduledTasks$$FastClassBySpringCGLIB$$fadb339e.invoke(<generated>) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) [spring-tx-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) [spring-tx-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) [spring-aop-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.flathub.api.scheduledtasks.ScheduledTasks$$EnhancerBySpringCGLIB$$1f87e864.updateFlathubInfo(<generated>) [classes!/:0.6.5]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) [spring-context-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.3.14.RELEASE.jar!/:4.3.14.RELEASE]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
jan 22 14:23:45 webapps.flathub.org podman[5076]: at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

Use a local PostgreSQL database for PRO environment

The postgresql database is not running in docker in the PRO environment. The conf file must be changed to make the backend work property

App shown twice

0 A.D. is shown twice at http://45.55.104.129/apps

I'm not sure if there's an issue with the manifest and the flathub.json file at https://github.com/flathub/com.play0ad.zeroad or if this is a bug in this backend.

Add categories to api/v1/apps

Need a way to get categories for all apps, very tedious to go through each category.

https://flathub.org/api/v1/apps/category/Utility

https://flathub.org/api/v1/apps/category/Game

Parse appdata for other architectures besides x86_64. Expose arch info in the API

This is required for https://github.com/jgarciao/linux-store-frontend/issues/39

Deprecate appstream-extractor and use provided gzip files

Instead of using the appstream-extractor we could use the gzip files that are in flathub.org:

• https://dl.flathub.org/repo/appstream/x86_64/appstream.xml.gz
• https://dl.flathub.org/repo/appstream/i386/appstream.xml.gz
• https://dl.flathub.org/repo/appstream/arm/appstream.xml.gz
• https://dl.flathub.org/repo/appstream/aarch64/appstream.xml.gz

Add suport for <url> tags inside <release>

Since appstream 0.12.5 (released in Jan 2019) the <release> tag can have a <url> tag inside

https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-releases

Each release tag may have a description tag as child, containing a brief description of what is new in the release. The description tag is structured as described in .
A release may also have an url tag as child. The release url should point to detailed release notes that explain the changes made in this particular release. The url tag may have a type property with details as the only currently allowed value. If the type is missing, an URL type of details is implicitly assumed.

Some apps like Cheese already use this tag and website shows (null) in the release description:

Title field in .flatpakref files uses ID instead of friendly name

As far as I can tell from the code, the .flatpakref files generated by the backend should have something like:

Title=Geary from flathub where the "Geary" part of the string is pulled from the <name> tag of the AppData file.

What actually seems to be happening is that we end up with the ID of the application there, so we end up with things like:
Title=org.gnome.Geary from flathub

I've tried with a selection of different .flatpakref files downloaded from flathub and this seems to be consistent across all of the ones I've downloaded. But as far as I can tell, the code makes sense and should be working, so I'm not sure where it's going wrong.

Resolution of the program version 2.4.1

The resolucion of the program is low and the icons are pixalated, i install the driver for nvidia card in ubuntu 18.04 and continues the same. I download the version 2.4.3 app and works properly. How i can fixed the resolution of the program. No video, no other thing the resolution of the program in the screen.

thanks

Error obtaining io.liri.Text information from the API ("Could not find acceptable representation")

There is an error with this API call: https://flathub.org/api/v1/apps/io.liri.Text

Tue Dec 04 11:49:16 GMT 2018 There was an unexpected error (type=Not Acceptable, status=406). Could not find acceptable representation

Adding a slash at the end it works properly: https://flathub.org/api/v1/apps/io.liri.Text/

I think the Spring framework is having problems identifying the parameter because it ends with ".Text"

Here are some hints to fix it:

• https://stackoverflow.com/a/46994545
• https://stackoverflow.com/questions/28466207/could-not-find-acceptable-representation-using-spring-boot-starter-web
• http://stick2code.blogspot.com/2014/03/solved-orgspringframeworkwebhttpmediaty.html

flathub/io.liri.Text#1

Add "Description" field to api/v1/apps api

https://flathub.org/apps/details/com.mojang.Minecraft

The description would be this text "Join people all over the world playing Minecraft, one of the most popular videogames around! Your own imagination is the limit in this fun game where you get to build the world around you. Play with friends, or alone, as you adventure, gather, build, and battle in an incredible land of your very own. For the best experience, we recommend you create an account and purchase online at https://minecraft.net/en-us/store before installing this app. ** Requires internet."

Proposed:
Add "Description" field to this api
https://flathub.org/api/v1/apps/

{ "flatpakAppId": "com.mojang.Minecraft", "name": "Minecraft", "summary": "Create your own world in one of the most popular video games", "iconDesktopUrl": "/repo/appstream/x86_64/icons/128x128/com.mojang.Minecraft.png", "iconMobileUrl": "/repo/appstream/x86_64/icons/64x64/com.mojang.Minecraft.png", "currentReleaseVersion": "", "currentReleaseDate": null, "inStoreSinceDate": "2019-05-01T10:30:11.625Z", "rating": 0, "ratingVotes": 0 }

Doesn't handle scalable icons

The Flash player only ships with a scalable icon, and the backend is incapable of converting it to a PNG for the Flathub website to be able to use it.

See flathub/com.adobe.Flash-Player-Projector#2

appsByCategory are not sorted alphabetically

provide build bots with checks for publishing app on the Web frontend

Sometimes apps get added to the repository but are not visible on the Web interface. Currently, xournalpp and fontforge are such apps. There is probably something wrong with the appdata, but it remains unclear what is it. The builds went find and the bot was happy.

It would be nice if the bot would complain that the app is not going to be published in the Web interface, so I think that it would be good to port the checks the server side does to the build bots. And keep them in sync.

@jgarciao

Only date shown, not version

The appdata of this app contains a release entry with the version attribute only:
https://github.com/flathub/io.designer.GravitDesigner/blob/master/io.designer.GravitDesigner.appdata.xml

But on the Flathub website, only a date is shown:
https://flathub.org/apps/details/io.designer.GravitDesigner

Use the not compressed appstream.xml file in appstream-extractor

Use the uncompressed appstream.xml file provided by recent versions of flatpak

Document API

Apologies if I've missed it, but it doesn't look like the REST API is documented anywhere. Looking at the code, I was able to figure out:

• /api/v1/apps (list all apps)
• /api/v1/apps/category/Game
  • List of available categories?
• /api/v1/apps/collection/recently-updated
• /api/v1/apps/com.play0ad.zeroad (more info on a single application)

Some apps have (null) in the description

~ $ curl https://flathub.org/api/v1/apps/com.github.ztefn.haguichi | jq '.description'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2726    0  2726    0     0   9303      0 --:--:-- --:--:-- --:--:--  9272
"<p>Join, create and manage Hamachi networks in a breeze.</p>\n<p>Features:</p>\n<ul>\n<li>Searchable, sortable and collapsible network list</li>\n<li>Sidebar for easy access to details and actions</li>\n<li>Desktop notifications</li>\n<li>Fully customizable commands</li>\n<li>Backup and restore your Hamachi configuration folder</li>\n</ul>\n<p>(null)</p>\n

More info at flathub/com.github.ztefn.haguichi@d6ecad1#r36562683

mvn install Error

When I run mvn install , I get error likes:

Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 34.517 sec <<< FAILURE! - in org.flathub.api.service.ApiServiceImplTest
findAllApps(org.flathub.api.service.ApiServiceImplTest) Time elapsed: 1.232 sec <<< FAILURE!
java.lang.AssertionError:
Expecting actual not to be empty
at org.flathub.api.service.ApiServiceImplTest.findAllApps(ApiServiceImplTest.java:37)

can you help me?

this is my environment and process:

0 oracle jdk8 and mvn 3.6
1 git clone project https://github.com/flathub/linux-store-backend and https://github.com/jgarciao/appstream-appdata-java
2 jump to appstream-appdata-java and run mvn install
3 flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
4 In linux-store-backend run bash scripts/appstream-extractor.sh
5 In linux-store-backend run mvn install

disable showSQL in prod

linux-store-backend/src/main/resources/application-PRO.yml

Line 24 in c2370f4

showSql: true

Use icons and flatpakref files provided by flathub.org

We don't need to obtain the app icons or generate the flatpakref files as they are already present in the flathub infra:

Examples:

• https://dl.flathub.org/repo/appstream/x86_64/icons/64x64/com.play0ad.zeroad.png
• https://dl.flathub.org/repo/appstream/com.play0ad.zeroad.flatpakref

Provide app's appdata problems in the api

UpdateServiceImpl print in the logs the appdata problems found in some apps by AppdataValidator, like missing icons, etc.

It would be useful add this info in the api, so it could be shown in a specialized page in the website to help developers fix the problems with their apps

Related to #21