arxcis / flatstat

Present stats from Flathub's Github-repositories.

Hosted at https://arxcis.github.io/flatstat/

How to generate the stats

GITHUB_PERSONAL_ACCESS_TOKEN=<token> make all

Folder structure

# Run commands
Makefile

# Commands that generate data
cmd/
    make-apps.js
    make-changelog.js
    make-count.js
    make-metafiles.js

# Data-folder contains generated data
data/
    apps.json
    changelog.json
    count.json
    metafiles.json

# Shared javascript files
lib/

# HTML Pages
about/index.html
library/index.html
graphs/index.html
index.html

flatstat's Issues

Achievements calculation

Looking at the achievements list, I see Steam at the top, are these stars?

Improved how history is stored on disk

Today

export default [{
  "appID": "org.gnome.Recipes",
  "ext": "json",
  "branch": "master",
  "stargazerCount": 3,
  "history": [
    {
      "ext": "json",
      "date": "2020-03-12T00:33:28Z",
      "x11": true,
      "fallbackX11": false,
      "wayland": true,
      "filesystem": false,
      "filesystemHome": false,
      "filesystemHost": false,
      "device": false,
      "deviceAll": false,
      "pulseaudio": true,
      "finishArgs": true
    },

Future

export default [{
  "appID": "org.gnome.Recipes",
  "ext": "json",
  "branch": "master",
  "stargazerCount": 3,
  "history": [
    {
      "ext": "json",
      "date": "2020-03-12T00:33:28Z",
      "finishArgs": {
        "--device": ["dri", "all"],
        "--filesystem": ["̃~/config", "home", "host"],
        "--socket": ["x11", "fallback-x11", "wayland", "pulseaudio"],
       }
    },

No new data since november

It seems like there was no new data since november?

You might want to use the api of the website https://flathub.org/api/v2/docs for querying summaries as that also includes firefox etc that are not on the github.

But can't give any stability gurantees on the above api.

Add --filesystem-portal achievement

Description

• Because it is fun to see how many apps actually use this portal.
• The --filesystem-portal-achievement replaces the current --no-filesystem-achievement.

Question

• How do we detect that an app uses the --filesystem-portal?

Add --pipewire-with-fallback achievement

Description

• Add --pipewire-with-fallback for audio, in the same fashion as --wayland-with-fallback for window.
• The --pipewire-with-fallback-achievement replaces the current --no-pulseaudio-achievement

Question
As of today, 2020-17, there is no known way to earn this achievement. Does anyone know how a flatpak may implement --pipewire-with-fallback ?

Pipewire is planned to be integrated into Fedora 33 and Fedora 34, ref https://blogs.gnome.org/uraeus/2020/09/04/pipewire-late-summer-update-2020/. I am assuming there will be a implementation of a pipewire-portal, or socket, shortly thereafter.

Add changelog list 🔴

• List changes chronological order, latest change at the top

🕳️ org.signal.Signal opened a portal                 --pipewire-with-fallback   2021-01-20
🏅 com.valvesoftware.Steam earned an achievement     --filesystem-portal        2020-11-20
🏅 com.valvesoftware.Steam earned an achievement     --no-filesystem-all        2020-11-16
💥 com.valvesoftware.Steam punched a hole            --filesystem-all           2020-11-15

Add --finish-args list 🏁

• List all current --finish-args.
• Sort them by frequency.
• This should be the new landing page for the site
• Make new cmd/finishArgs.js and data/finishArgs.json
• New icon 🏁🏁🏁🏁🏁

Use Git LFS for large db-files

The files in the db/ foldder should get the git lfs treatment, to avoid large diffs.

Update periodically

It'd be really helpful if Flatstat updates periodically. How about weekly? If it's too much, then maybe monthly?

This is great!

Not an issue, but a plaudit: this site is great! Fascinating charts.

Rank by number of "holes" (permissions requested) instead of stars

Having a rank of apps based on how many permissions each one require, would be much more interesting.

Use JSON.parse and YAML.parse when parsing metafiles, instead of regex

YAML.parse() libs

url	version	contributors	commits	PRs	issues	latest commit	unpacked size
https://github.com/nodeca/js-yaml	v3.14.0	68	1232	14	14	25 days ago	291 kB
https://github.com/eemeli/yaml	v2.0.0-1	11	906	2	11	26 days ago	445 kB

• Both are well maintained.
• js-yaml looks better overall.

Move changelog into a separate changelog.html-page

Description

The changelog deserves it's own top-level /changelog-page, instead of being integrated into the /apps-page, like today:

Here are some ideas about different changelog-filters:

• New apps
• Additions
• Removals
• 2020
• 2019
• 2018
• Last month
• Last year

Detect if pipewire is enabled inside Flatpak

Hypothesis
Detecting if wayland is enabled in a Flatpak is easy. We just check if we find the flag:

--socket=wayland

As far as I know there is no --socket=pipewire-flag. Correct me if I am wrong 🙏

As far as I can tell from looking through .json|.yaml-files, pipewire-support can be detected if we find the flag:

--filesystem=xdg-run/pipewire-0

Example: https://github.com/flathub/org.musescore.MuseScore/blob/master/org.musescore.MuseScore.yaml#L34

Questions

• Is this the official way to enable the Pipwire-socket inside a Flatpak?
• @TheEvilSkeleton Can you confirm anything?
• Are there official documentation explaining this?
• Is there an equivalent to --socket=fallback-x11 in the Pipewire world?

Make metafiles.json smaller size by only storing 1 history entry per month

When drawing the graphs, we only need a resolution of 1 commit per metafile, per month, and we only need the last commit from each month. Given this, we might as well only store at most 1 commit per month, in the metafile history. Doing this has the potential to reduce the size of metafiles.json, which currently clocks in at:

$ wc metafiles.js 
 309960  574479 7241230 metafiles.js

More than 7 million bytes and 309k lines!

Sort /library/#achievements and /library/#holes by usage

Today
We have an unordered list, with no information about usage.

Suggestion
Show usage of each hole/achievement, and sort by usage.

Add --persist=FILENAME achievement

Discussion
I am struggling to understand the description in the Flatpak CLI reference https://docs.flatpak.org/en/latest/flatpak-command-reference.html#flatpak-override

--persist=FILENAME
If the application doesn't have access to the real homedir, make the (homedir-relative) path FILENAME a bind mount to the corresponding path in the per-application directory, allowing that location to be used for persistent data. This updates the [Context] group in the metadata. This option

Is it for persisting FILENAME to ~/.var/apps/com.example.App/FILENAME

What does --persist=. mean? I assume it means "persist everything"? com.valvesoftware.Steam uses --persist=.

Here is the count of persist-usage across all apps 2020-11-21:

I assume it is NOT a hole, because it seems very sandboxed, even if it is writing to the hosts filesystem.

It looks like --persist should be the preferred sandbox-alternative, compared to asking for --filesystem=xdg-data or --filesystem=~/.local/share/.

Maybe it should be an achievement to use --persist ?

If an app switches from APP_ID.json -> APP_ID.yaml, we only track the APP_ID.json-history, and the APP_ID.yaml-history is ignored

From reddit https://www.reddit.com/r/flatpak/comments/ju8x1c/graph_how_x11_and_wayland_are_doing_across_all/gceukrc?utm_source=share&utm_medium=web2x&context=3

I think I found another problem: Gedit seems to get classified as a non-GUI app (x11, fallbackX11 and wayland are all false in the latest entry in the history array). I think the cause of the problem is that the gedit manifest was switched from JSON to YAML. And in that situation, your code seems to only find the commits that modify the JSON file, the latest one of those is the one where it gets deleted, so everything is set to false.

From db/flathub/metadata.json
gedit is indeed wrongly classified as a non-gui app, in the latest history entry

{
  "index": 88,
  "name": "org.gnome.gedit",
  "ext": "json",
  "branch": "master",
  "stargazerCount": 2,
  "history": [
    {
      "date": "2020-05-06T20:53:37Z",
      "x11": false,
      "fallbackX11": false,
      "wayland": false,
      "filesystem": false,
      "filesystemHome": false,
      "filesystemHost": false,
      "device": false,
      "deviceAll": false,
      "pulseaudio": false,
      "finishArgs": false
    },

And gedit has indeed changed its metadata format from .json --> .yaml

• The current yaml file -> https://github.com/flathub/org.gnome.gedit/blob/master/org.gnome.gedit.yml
• The commit adding the .yaml file -> flathub/org.gnome.gedit@9ea459d
• The commit removing the .json file -> flathub/org.gnome.gedit@494c436

Update stats