makedeb / mprweb Goto Github PK

Hosting platform for the makedeb Package Repository (MPR)

License: GNU General Public License v2.0

Python 77.71% Mako 0.05% Makefile 0.21% Shell 1.33% HTML 17.20% JavaScript 1.23% Dockerfile 0.08% Jsonnet 0.11% Jinja 0.08% SCSS 2.00%

hacktoberfest

mprweb's Introduction

mprweb

mprweb is a hosting platform for the makedeb Package Repository (MPR), a fork of aurweb, the underlying platform that powers the Arch User Repository (AUR). The scripts contained in the repository can be built using makedeb and installed using Debian package managers such as dpkg and apt.

The mprweb project includes:

A web interface to search for packaging scripts and display package details.
An SSH/Git interface to submit and update packages and package meta data.
Community features such as comments, votes, package flagging and requests.
Editing/deletion of packages and accounts by Trusted Users and Developers.
Area for Trusted Users to post MPR-related proposals and vote on them.

Directory Layout

aurweb: aurweb Python modules, Git interface and maintenance scripts
conf: configuration and configuration templates
static: static resource files
templates: jinja2 template collection
doc: project documentation
po: translation files for strings in the aurweb interface
schema: schema for the SQL database
test: test suite and test cases
upgrading: instructions for upgrading setups from one release to another
web: PHP-based web interface for the MPR

Documentation

What	Link
Installation	INSTALL
Testing	test/README.md
Git	doc/git-interface.txt
Maintenance	doc/maintenance.txt
RPC	doc/rpc.txt
Docker	doc/docker.md

Testing

See test/README.md for details on dependencies and testing.

Translating

If you'd like to help with translations for the makedeb project, visit the Weblate page.

mprweb's People

Contributors

Stargazers

Watchers

Forkers

hiddeninthesand wizard-28 kavplex

mprweb's Issues

Update package archives more frequently than every five minutes

Currently package archives are updated every five minutes. This is quite slow, and can lead to fair amount of delay for when things like a packages version are seen as updatable by programs like Tap. Considering the size of the MPR, we should be able to comfortably decrease that time to something like a minute or maybe even 30 seconds.

Hide maintainer's email address from PKGBUILD files

Introduction

PKGBUILD specifications (understandably) require maintainers to add an email address so that they can be reached in case of issues.

Unfortunately, this means that spam bots can find the email addresses and add them to their lists. Once an address has been added to one of these lists, it's unlikely it will ever stop getting spam.

What probably won't work

Replacing @ with at, and other methods of obfuscation. Bots are already prepared to deal with these methods.

Proposals

Proposal 1 - Proxy addresses

This would be a simple way of protecting the maintainer's email address while ensuring the maintainer can still be contacted.

This would involve adding the necessary capabilities to MPR to allow maintainers the use of an email proxy service, provided by the MPR itself. Maintainers could then change the target email address in their profile and would change when packages change hands.

It could look like [email protected]

Proposal 2 - MPR provides an email address

MPR could provide email addresses to maintainers with just enough space to receive emails concerning their maintained software.

This could be automated so that creating an account in the MPR automatically sets up an account for you accessible using Rouncube/whatever it is popular these days.

Proposal 3 - No email address required

The simplest of all. Removing the requirement of adding your email address to the body of the PKGBUILD file and instead provide a "Contact" link on MPR which directs users to a form for people to report issues/problems.

A variation of this proposal could be changing the requirement from "Email address must be public" to "A form of contact must be provided" and allow users to request contact via Matrix or other platforms. This solution is not as neat and tidy as the others.

Copied from https://github.com/makedeb/mprweb-historic/issues/3

Some packages cannot be marked out of date

I'm aware that some of the devs are already aware of this issue, but I didn't see it marked down so I'm taking the opportunity to. There's a bug among some MPR packages that make them incapable of being marked out of date, one of which being Android Studio. If extra information is required, I'll try my best to provide it.

[Bug] Pages have broken favicon

All MPR pages under https://mpr.makedeb.org/cgit/ link to the wrong favicon, causing it to not work.

Internal Server Error upon attempting to hide email address

Steps to reproduce:

Log in to mprweb
Go to account settings
Check "Hide Email Address" and enter password where required
Click button to save, and be met with ISE

Not sure what's going on here, but I'll be happy to provide information where I can.

Can't configure the ssh connection

I have created an mpr key as and set up the config as follows

Host mpr.makedeb.org
  IdentityFile ~/.ssh/mpr
  User mpr

I have added the mpr.pub in the configuration, but I still can't connect via ssh. What else could be wrong?

Removal of DEBUG info

Remove the debug info from prod

➜  bitwise-git git:(master) git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 2 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 217 bytes | 217.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0
remote: warning: .SRCINFO unchanged. The package database will not be updated!
remote: 10:38:07 DEBUG | aurweb.db: DBName(mprweb): mprweb
To ssh://mpr.hunterwittenborn.com/bitwise-git.git
   d2b7e7a..4371669  master -> master
➜  bitwise-git git:(master)

HTTP Git interface isn't working

Attempting to run git clone 'https://mpr.hunterwittenborn.com/tap' or anything similar results in a 502 error from Git. The same result happens regardless if the .git suffix is at the end of the URL or not.

@gamer4life1 reported that the SSH interface for Git is working just fine in the support rooms, so it appears to be an issue with ~~the string used to check for Git URLs~~ (more testing is revealing an issue with possibly the git or smartgit container).

Create custom UI for Git web interface

Cgit is quite limited in how we can customize it, so we should replace Cgit with a custom Git interface in the web UI.

Point `mpr.hunterwittenborn.com` to same IP as `mpr.makedeb.org`

The MPR used to be hosted at https://mpr.hunterwittenborn.com, so we should point that URL to the MPR server.

The web interface already redirects correctly, but we do that redirect from the homeserver instead of the MPR server itself, so SSH still doesn't connect properly.

Overlapping text on package pages

The sources URLs and the git clone URLs overlap on package pages, like on https://mpr.makedeb.org/packages/electronmail-bin

Unable to log in to mprweb

Steps to reproduce:

Open https://mpr.hunterwittenborn.com/login?next=/
Enter Credentials
Encounter a 500 internal server error

The only error mentioned is that it's "Unable to generate a unique session ID in 36 iterations."

TU Vote feature is spamming mails until you participate in the vote

one reminder per day should be more than enough

Pressing Git clone URL does not copy to clipboard

When pressing to ssh git clone url it does try to open it instead of copying it to the clipboard as it should.

Add support for API keys

We'd like to start adding authenticated calls to the API interface, so we should start by adding a way to create and manage API keys.

[Feature Request] Delete self-deleted comments

Currently, self-deleted comments are treated like TU deleted comments, in which they aren't actually deleted from the mprweb server, but merely hidden to everyone but the TU and the original poster (with the ability to be unhidden). This is somewhat problematic from a privacy perspective, so an option to truly delete self-deleted comments would be appreciated.

Remove PHP code

[Feature Request] GitHub repository has no issue forms

Exactly as the title says, the mprweb GitHub repository currently has no issue forms.

icons for the comment actions do not show up properly

API keys aren't hashed

Looking at the implementation I just added, I realized that we aren't hashing API keys, when we should to protect against incidents such as security breaches.

[Bug] When in git view, wrong clone URL is shown

Clone URL is reported as localhost, example at https://mpr.makedeb.org/cgit/aur.git/?h=librewolf-bin

RSS does not show author or icon

currently

before the mprweb Python update at least the authors did work

Change the list page number position for package lists

When we go to, https://mpr.makedeb.org/packages and start scrolling till the page ends then we have to go back up to go to the next page in the list.
Add page number list at the bottom as well

Add padding between sections on API and MPR Archives pages

Remove Groups table from db

This is a feature used in pacman packages, and doesn't translate to dpkg or APT.

Add a ?type=meta endpoint to the RPC interface

Copied from https://gitlab.archlinux.org/archlinux/aurweb/-/issues/179

I recently discovered that the GitHub API has an endpoint to get some metadata about their servers among other info.

It would be cool if the aurweb RPC interface had a feature like such as well, which could list things such as the instance's SSH fingerprints and the statistics for the server (which are seen on the homepage in the bottom right):

curl 'https://aur.archlinux.org/rpc/?type=meta'

{
  "ssh_fingerprints": {
    "ed25519": "SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4",
    "ecdsa": "SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI",
    "rsa": "SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8"
  },
  "statistics": {
    "packages": 73084,
    "orphan_packages": 8679,
    "packages_added_past_7_days": 237,
    "packages_updated_past_7_days": 1568,
    "packages_updated_past_year": 25152,
    "packages_never_updated": 18336,
    "registered_users": 89307,
    "trusted_users": 59
  }
}

Automatic of date notifications aren't working

The cron scripts for automatic out of date notifications currently isn't working. Furthermore, it doesn't look like some of the out-of-date database fields for packages are being cleared when a package is updated.

Add Cgit redirects for new Git UI

Links for the old Cgit UI could be active across the web, so we should add some NGINX redirects to handle the new locations of the PKGBUILDs.

re-enablle previous UI's option of selecting multiple packages and doing a common operation on all

in earleir ui we had an option to select multiple packages and do some operation like give vote or retract vote or disown and such operations. request to implement it in this ui as well.

[Feature Request] Add a security policy

If the MPR ever scales, it'll be important to have a proper, and more importantly, secure way to disclose vulnerabilities.

Set up an email for disclosing vulnerabilities
Publicly share a GPG key for encrypting disclosure

Add button to install packages via MPR helpers

It would be nice to have a button that could bring up a window that showed instructions for installing packages with various MPR helpers.

Archives at `/*.gz` aren't returning any data

Endpoints like /packages-meta-ext-v1.json.gz and /users.gz currently aren't returning any data.

Add Repology out of date tracking

Repology is a fantastic source of information on package versions. We should add support to automatically mark packages as out of date when Repology tells us the MPR version of a package is out of date.

This should also be an opt-in feature though, since Repology could get things mixed up with common package names.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update all dependencies (Authlib, Hypercorn, Jinja2, Markdown, SQLAlchemy, Werkzeug, actions/checkout, aiofiles, aiohttp, alembic, asgiref, autoflake, bcrypt, beautifulsoup4, black, bleach, coverage, email-validator, fakeredis, fastapi, feedgen, filelock, flake8, gunicorn, highlight.js, httpx, isort, itsdangerous, lxml, makedeb-srcinfo, mysqlclient, orjson, posix-ipc, prometheus-fastapi-instrumentator, protobuf, pygit2, pytest, pytest-asyncio, pytest-cov, pytest-tap, pytest-xdist, python-multipart, redis, requests, sentry-sdk, uvicorn)

Detected dependencies

docker-compose

docker-compose.mpr-override.yml

docker-compose.yml

dockerfile

Dockerfile

github-actions

.github/workflows/pr.yml

actions/checkout v3

html

templates/partials/head.html

highlight.js 11.5.0

pep621

pyproject.toml

setup-cfg

setup.cfg

aiofiles ==22.1.0

aiohttp ==3.8.3

asgiref ==3.5.2

bcrypt ==4.0.1

beautifulsoup4 ==4.11.1

bleach ==5.0.1

email-validator ==1.3.0

fakeredis ==1.10.0

feedgen ==0.9.0

httpx ==0.23.0

itsdangerous ==2.1.2

lxml ==4.9.1

makedeb-srcinfo ==0.5.2

orjson ==3.8.1

protobuf ==4.21.9

pygit2 ==1.10.1

python-multipart ==0.0.5

redis ==4.3.4

requests ==2.28.1

paginate ==0.5.6

sentry-sdk ==1.10.1

alembic ==1.8.1

mysqlclient ==2.1.1

Authlib ==1.1.0

Jinja2 ==3.1.2

Markdown ==3.4.1

Werkzeug ==2.2.2

SQLAlchemy ==1.4.43

uvicorn ==0.19.0

gunicorn ==20.1.0

Hypercorn ==0.14.3

mysql-connector ==2.2.9

prometheus-fastapi-instrumentator ==5.9.1

pytest-xdist ==3.0.2

filelock ==3.8.0

posix-ipc ==1.0.5

pyalpm ==0.10.6

fastapi ==0.86.0

autoflake ==1.7.7

black ==22.10.0

flake8 ==5.0.4

isort ==5.10.1

coverage ==6.5.0

pytest ==7.2.0

pytest-asyncio ==0.20.1

pytest-cov ==4.0.0

pytest-tap ==3.3

Check this box to trigger a request for Renovate to run again on this repository

the packages section ui improvement request for current search dialog boxes

the current search criteria box takes up a lot of space. make it hidden by default and tagglable to show it or something similar.

Move everything to a single config file

Add support for required commit signing

It'd be nice if there was a way to force all commits for a package to be signed by the package's maintainer. This could be nice so that maintainer's can add an extra layer of security that others won't be able to push to their package's repositories.

Add support for Matrix notifications

We should add support to receive things like notifications on Matrix

Redo site with new CSS

The makedeb homepage is using a certain style that I'd like to start using across the MPR and docs so that end users have a more cohesive experience.

We're probably gonna completely redesign the layout of the homepage too while we're at it, as well as make the site mobile-friendly.

Routes to finish:

New routes:

/about - Instance information, such as SSH keys, number of packages, and other stats
/pkgstats - Instance containing information such as out of date packages and the list of the user's packages
/pkgbase/{name}/git - Git information about packages

Post-port steps (to do after routes are finished, but before merging):

Fix unit tests
~~Remove uneeded templates~~ ^{Will do later}
~~Move all templates back to {% block %} system.~~ ^{Will do later}

Disowning package fails when package has had an orphan request that got rejected previoulsy

Image of the error:

How to reproduce

select some package
issue an orphan request
go into requests as TU
close / reject orphan request
go back to the package and press disown package
press disown
get error as shown above

Expected behaviour would be that disowning a package should always work regarding of any pending or rejected orphan requests.

Add package CI/button to add to PrebuiltMPR

We talked in Matrix about adding a button when logged in and viewing your package, that triggers the Talus API (preferably authenticated). This API automatically adds a package to @PrebuiltMPR's build system (via /add endpoint) and pushes it to its repo. It also will have an endpoint that returns the ci badge, so that it can be displayed.
This will act as a CI for packages as well as being an auto-build system. It can help maintainers by quickly informing them (runs every 24h) whether their packages have broken.

Thanks,
Leo