utrs2 / utrs

Unblock Ticket Request System (v2) used on Wikimedia projects

UTRS 2.0

Getting UTRS2 working in a dev environment

1. Clone into a directory
2. composer install
3. cp .env.example .env
4. edit the DB details in .env
5. php artisan key:generate
6. php artisan migrate:fresh

The /public subdirectory is the webroot. You can either use php artisan serve to run a standalone webserver, or point your webserver of choice at /public

Configuring wikis

Currently, most details about supported wikis are stored in config/wikis.php, but a transition to storing them in the database is in progress. For this reason, you need to use php artisan utrs-maintenance:sync-wikis-to-database to synchronize changes in that file to the database for some details to work correctly.

Building frontend assets

The frontend is built using Bootstrap 4. It uses purgecss to remove unused classes to reduce bundle size. This can create complications for development.

First, install NPM dependencies using npm install. Then, you can use Laravel Mix (webpack wrapper) to build assets:

• Use npm run watch to start Webpack mix in development watch mode. That mode does not use PurgeCSS and it re-builds all assets when it detects a change. This is most useful when working with JavaScript or when modifying the SCSS files.
• Use npm run dev to do one build in the development mode. This mode does not use PurgeCSS. This is most useful when you are working on the frontend, as you have access to all classes Bootstrap has to offer.
• Use npm run prod to do a production build. This build uses PurgeCSS and URL versioning. Due to legacy reasons a production build should be commited to the repository for pull requests. This may change soon, see issue #240.

Seeding

UTRS has a suite of seeds that can be used to automatically generate sample data for your development environment.

To reset your database, and create data based on seeds, you can always type

php artisan migrate:fresh --seed

Resetting your database will get rid of your user account and all of its permissions. You will find few .env variables to help with this:

# for seeding:
DEVELOPER_USER_NAME=""
DEVELOPER_USER_GRANT_ADMIN=""
DEVELOPER_USER_GRANT_TOOLADMIN=""
DEVELOPER_USER_GRANT_DEVELOPER=""

If you set DEVELOPER_USER_NAME to your user name, that user will automatically created. You can then set DEVELOPER_USER_GRANT_ADMIN and DEVELOPER_USER_GRANT_TOOLADMIN to "enwiki" and then you wll be granted admin and tooladmin access there. Finally, if you set DEVELOPER_USER_GRANT_DEVELOPER to true, you will receive the global developer permission which grants you full access to the interface.

Setting up MediaWiki integration for testing

API calls

You have a couple of options here. Either

Create a bot password for your own testing MediaWiki installation

Use Special:BotPasswords to create a bot password. If you have configured e-mail sending to mediawiki, access to that is useful.

In .env, set MEDIAWIKI_USERNAME to the bot password username and MEDIAWIKI_PASSWORD to the password. Also set WIKI_URL_GLOBAL, WIKI_URL_ENWIKI, and WIKI_URL_PTWIKI to be http://your-mediawiki/w/api.php.

Create a bot password on Beta Cluster

This is useful because Beta Cluster is really similar to beta. Use Special:BotPasswords on some wiki to create a bot password (they are global).

In .env, set MEDIAWIKI_USERNAME to the bot password username and MEDIAWIKI_PASSWORD to the password. Also set

• WIKI_URL_GLOBAL=https://meta.wikimedia.beta.wmflabs.org/w/api.php
• WIKI_URL_ENWIKI=https://en.wikipedia.beta.wmflabs.org/w/api.php
• WIKI_URL_PTWIKI=https://es.wikipedia.beta.wmflabs.org/w/api.php # no ptwiki on betacluster, let's use eswiki instead

OAuth

You need a wiki with Extension:OAuth installed. It's easier if you use the same wiki used with API calls.

Use the following settings:

• Protocol version: 1.0a
• Callback URL: http://utrs.test/oauth/callback change this depending on your setup
• "Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix." should be yes
• Types of grants being requested: "User identity verification only"

If you set it as for yourself only you don't need to get it approved.

Set up .env:

OAUTH_CALLBACK_URL="http://utrs.test/oauth/callback" # change this also depending on your setup

OAUTH_BASE_URL="https://meta.wikimedia.beta.wmflabs.org"
OAUTH_CLIENT_ID="some-client-id"
OAUTH_CLIENT_SECRET="some-client-secret"

Job executor (for production)

For this to work, you need to have your own urls set up for config/wikis.php
This application requires a job queue to verify blocks and do other critical and regualar tasks. Below is a guide to setting up:

1. Ensure jobs work with php artisan queue:work
2. Add the following block at /etc/systemd/system/utrs-jobrunner.service, example, replacing /path/tos:

DO NOT CHANGE ANY OF THESE VALUES EXCEPT THE PATHS

[Unit]
Description=UTRS job runner

[Service]
Restart=always
User=www-data
Group=www-data
ExecStart=/path/to/artisan queue:work
TimeoutStopSec=20s

[Install]
WantedBy=multi-user.target

3. sudo systemctl daemon-reload
4. sudo systemctl enable utrs-jobrunner.service
5. sudo systemctl start utrs-jobrunner.service

utrs's Issues

Verify account named block targets

As an issue in the old UTRS, other people would file appeals on behalf of blocked users to impersonate them. We need wikiactions.py to send them a confirmation code through wikimail before we open the appeal. We may already have our first struggle with this.

User links

Add user links (talk, contribs, find block, block log, etc)

Slack Logging

Using log stacking, I'd like us to return to the original idea of remote notification logging remotely so we can see issues before they stay too long in production.

https://laravel.com/docs/7.x/logging#building-log-stacks

Add user management (master task)

Right now all permission modification needs to happen through the database. It's now time that the basics are down that we change that. Honestly, get creative with how you want to handle the form, but do realize a user can have multiple db entries in the permissions table (wiki being the only proper distinction), and that there are even duplicate entries right now (which will need a scripted sql patch of sorts to go with the pull request for this issue.

Tasklist:

• Make sure multiple entries in the permissions table are handled
• Create a mysql script of sorts to remove current duplicates (should be a separate issue)
• Prevent duplicate user verifications which cause duplicate inserts into permissions table (#88)
• Properly handle who can manage users and which permissions they can add
• Make sure a reason for user changes is obtained on submission
• Add a logs table entry for the change
• Show all previous userrights changes log (should be a separate issue)
• Only allow changing of non-wiki based userrights: Developer, tooladmin & privacy (Developers can add all; tooladmin can add privacy for wikis in which they are a tool admin.
• Show wiki-based rights read-only
• Create laravel job for verifying current onwiki permissions on a regular basis (should be a separate issue - HIGH PRIORITY, before this issue is dealt with) (#88)

There is likely more, so i'll modify this as we go through it and add/replace tasks as necessary.

Add UI to quickly find an appeal given it's number or username/ip

Can't create account

I am an admin. I received an e-mail for account confirmation. When I follow the link, I get a message "Uh-oh. UTRS has suffered an internal exception and can't continue to serve your request."

Users can't see their own comments

On publicappeal.blade.php blocked users can't see their own comments.

Fix:

• Change public comment submission logging to have user=-1
• Create a script that can be run on the DB to set user = -1 where user = 0 and reason != NULL and action = comment
• Add a username for -1 to handlingAdminObject

Convert appeal table updater to a Laravel Job

For cross-wiki support, this should support configuring a differently named page for different wikis (maybe linked via a Wikidata item?)

Differentiate between appeals who have been sent a verification email and those who dont

The appeal view should display a different message for users who have been sent a verification email (verify_token is not null) and users who can't be emailed for some reason or haven't been emailed yet.

Autoblock IDs taken as accounts

In the verify block script, autoblocks, or the block number is taken as an account name and doesn't get verified. They all start with "#" so that is the identifier.

Enable lost appeal key recovery

Currently on the front page we have a big red button that says "Forgot Appeal Key". For appeals whose creator is confirmed (#27 #33) it should be possible to recover (reset?) the appeal key by sending an e-mail to the wiki user.

Add notice about appeals to user talk pages

OAuth Integration

Tacking for OAuth integration.

Upgrade utrs-production-beta to Debian 10

Keeping with eventually depreciating versions on WMF Cloud, we should be upgrading to Debian 10.

Hide ban target from tool administrator

Sometimes we need to ban valid usernames that are oversighted on enwiki as a whole. These "targets" from the db column should be hidden from non-privacy and non-OSers.

There needs to be a field in the database to be able to hide this, and a log entry needs to be generated that is protected.

Add ban management (master task)

We are already at the point in which bans need to be able to be made, and 3 bans have already been issued though the database. Tool Administrators need to be able to start making them and not just viewing them.

Bugs already existing in the code:

• Bans are applied regardless of expiry (#84)
• Bans need the ability to be hidden from tooladmin (#82)

Bugs that may get introduced (please comment on each in issue saying you checked it):

• The visible log table on the blade file needs to hide protected entries (#82)
• Modifying the block should not show non-privacy/OS the protected target (#82)
• Regex may be sanitized being put into the database

Features required:

• Valid regex validation for usernames
• IP range selection with a dropdown of /16 to /64 for both IPv4 & IPv6
• Logs of all changes to the ban on the view and/or modify page (whatever is created)
• Expiry set by phrases like "2 weeks" (OR if not, a calendar dropdown that formats into a timestamp in textbox)
• Additional suppression checkbox w/proper logging (#82)

Required logging:

• Each ban create request
• Each ban modify request

Add return to open button on appeals

Items that are in the tool admin or checkuser queues can't be returned without DB interference right now...don't know how I missed this.

Page for own profile

We need a page that allows users to:

• Reset their password & forgot password (using wikimail if any email is sent)
• View the wikis and permissions on each wiki they have been assigned.

This is seperate from user management as it's tooladmin only there

Validate blockids to username/IP

Right now, people can get around a UTRS block by using the block number to evade syntax detection of a username ban, acting as if they are autoblocked. Autoblocks can remain as such, but anything that has a username or IP associated to who is the block target needs to have appeal for switched to that upon validation.

Can't appeal an IP block: "The POST method is not supported for this route. Supported methods: GET, HEAD. "

Steps to reproduce:

• Be logged out
• On the front page, click "Appeal IP block"
• Fill in some IP address and some reason and submit

Expected result:

• appeal is created

Actual result:

• an error appears, saying "The POST method is not supported for this route. Supported methods: GET, HEAD."

Allow users to have personal templates in addition to the global ones

Create {{UTRS2}} template for wiki

Some implementation of {{Ombox}} where status changes can be made.

This is for eventually putting the unblock templates on User talkpages again.

Show previous appeals in a table

Previous appeals by user and by ip should be listed on the appeal directly, not including itself.

Searching for account appeals only has to worry about appealfor in appeals table.
Searching for other IP appeals has to worry about both appealfor and hiddenip in the appeals table. It would also be prudent to search in the logs table under:
{ user = 0, objecttype = appeal, action = create, ip = {ip} }

Can I have you do this @supertassu ? I'm also going to add you to the project so we can add you to assign.

Full search interface

Expanding from #149:

I'll approve this. My only note is this should not be the end of this enhancement, we need a seperate search page for when it comes to CU searches or crossing related appeals from IPs/hidden IPs safely.

Add useful links to navbar to logged in users / users with permissions

The current navbar is empty. We could add some useful links such as appeal list or template/user/ban management.

This is much easier when #106 is merged as that allows using @can('viewAny', \App\Template::class) styled statements in blade files.

Update to Laravel 8

Currently we are using Laravel 6.18.35 (as of 2020-09-08) even thru 8 is the newest version. I'll do the upgrade when I have time.

Fix viewing of old appeals

Currently viewing old appeals results in a 500. I have a strong feeling it is caused by this as e-mail might be null:

utrs/resources/views/appeals/oldappeal.blade.php

Line 22 in b2bf722

<br />Email Domain: ******{{ '@' }}{{explode("@",$info['email'])[1]}}

Setup all instances of wiki to be loaded from the database

As we intend this project to be in the long term available to multiple wikis, easier configuration should be setup through the database. This may also require an interface to add things (developer only) to database without having to dig for it.

It doesn't seem too far of a stretch as appeals have wikis set, the options can be spawned on the views, and the Apis can be set to rotate based on $wiki.

Determine what needs to be removed from old UTRS database

Any private data columns like email columns should be permanently removed from the old database.

Allow per-wiki customization of permissions required to read or respond to appeals

This is much easier once #106 is merged

Page titles

There are no HTML page titles at this time.

Force comment reason when deferring to CU

Right now when someone defers to CU, they can do it without comment, leaving the checkuser confused. A comment needs to be forced and throw into the system under the question "What would you like the checkuser to review in this appeal?".

Make log entry reference type a proper Laravel polymorphic relationship

See https://laravel.com/docs/7.x/eloquent-relationships#one-to-many-polymorphic-relations. In short, this will make dealing with log entries much easier

• Rename columns
  • referenceobject -> model_id
  • objecttype -> model_type
• Change usages of old column names in code
  • Adjust user class in #106
• Add relationship methods to Log and Appeal
• Change column values in code
  • model_type
    • 'appeal' -> Appeal::class
    • 'Template' -> Template::class
• Change old values to reflect changes above

Fix appeal commenting from blocked user

Currently, if you are the blocked user and add a comment to your appeal, you will be redirected to logged in user's view of appeal giving you a 403. This a) is a bad user experience and b) can cause confusion about if your comment was submitted or not.

The problem here will be redirecting back to POST endpoint, more specifically the same problem as at #33 (comment). I'm not sure what redirect()->back() does with that (not an option at that other case linked below), I'll take a look at that when I have time.

Force HTTPS

Forcing HTTPS isn't working despite the attempt made in #14

Viewing closed appeals

Currently I can't see closed appeals - I get an error message about not having the right permissions. Admins at least should be able to see closed appeals for background on an editor and the context of their appeal.

Modify IP should contain modify wiki also

People can screw up which wiki. Instead of having to refile, allow them to modify the wiki.

Create alpha server

alpha-utrs.domain should be made available for developers to test changes as a group or so no one HAS to spawn a local environment to test (though that is optimal). This is some that is needed for #12 to be passed. That said, there are complications.

• Only tool roots can have command line access
• A web-interface allowing the changing of what branch alpha displays would be required
• Only authenticated users could access this interface

@supertassu Should we be creating another repo with a mini-laravel application that could handle this? Or do we have other ideas?

Fix unit testing for PRs

See https://github.com/UTRS2/utrs/actions/runs/105414200

Format links on block reasons

It would be nice to format wikitext links in block reasons. For example, instead of "[[WP:Spam|Spamming]] links to external sites" it would say "Spamming links to external sites".

Bonus points for linking into templates, for example instead of saying "{{checkuserblock-account}}" it would say "{{checkuserblock-account}}".

This is kinda related to #48 but at the same time it kinda isn't so filing a separate ticket. Support for other wikis than enwiki is needed, so I'd like to get config/wikis.php currently present in #33 to get merged into master before working on this to avoid duplicating stuff.

Translation support

Especially if UTRS is used on other wikis or for global (b)locks, it should be possible to translate UTRS into more than one language. Laravel has it's own built in translation system, using it is probably a good idea. Using translatewiki.net for editing non-source languages (not English) is also an idea worth exploring.

Initially prioritizing this as medium as this is much easier to implement when there aren't that many phrases, feel free to change.

Use Laravel Queues/Jobs instead of a Python script

Laravel has a feature named Queues which allows to run PHP jobs in the background either on regular intervals or based on user actions. I think that it would make sense if the current wikiactions table/script would be converted to use triggered jobs instead. This would allow to reduce code duplication and make on-wiki actions more instant.

• Block existence verification (#33)
• User permission checking (#88)
• Appeal table updating (#141)
• Account verification (not needed when #12 is merged)
• Private data cleanup (#380)
• Close expired NOTFOUND appeals

Add Developer view of appeal logs

When an appeal is brought up, at the bottom, devs should have a table of all logs related to the appeal ( [ objecttype=>'appeal', referenceobject=> appealid ]; select user, referenceobject, reason, action, ip, ua, protected, timestamp).

This will give a clearer view of things without having to go DB dumpster diving.

Auto backups

After today's issue with the database, I want to do regular updates. This seems like a good platform build into laravel:

https://github.com/spatie/laravel-backup

Reply to user status options

When an admin is replying to the user, they should have additional options to change the status.

• AWAIT_REPLY - for those that are awaiting reply from the blocked user. These should then be hidden from the main list.
• DECLINE/APPROVE/EXPIRE - So another button doesn't have to be clicked.
• Just reply to the user like it is now.

Setting Priority: High as this really can dig into the time someone takes to review unblocks if they aren't flagged AWAIT_REPLY. And this was a UTRS 1 feature.

Allow usage of some formatting in comments or replies

ref WT:UTRS#Templates.

Technically speaking it would be easy to allow HTML formatting in comments/templates but I'd like to not do that. Either we could allow sanitized HTML via a library like html-sanitizer or allow using some markup language like https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet or even Wikitext.

Environment type can crash ip functions

A lot of the app uses $ip = $request->server('HTTP_X_FORWARDED_FOR'); to determine the IP. This is great for a our production environment, as it's what it should be, but not for other productions or any locals.

Using a new use_xff config var to fix this.

Bans are applied regardless of expiry

Currently AppealController when the appeal is submitted, it checks for bans. It doesn't check the expiry date for validity. 0000-01-01 00:00:00 is indefinite.

Decommission UTRS 1

While we aren't ready for this just yet, its good to make this a known project goal (maybe a milestone too).

Pre-Requirements:

• Current tool administrators for enwiki need to sign off that all templates have been transferred
• Old appeals needs to be fully functional (#47 , possibly more)
• All private data needs to be ensured it's removed from the database #63
• Alpha needs to be made available #61 ❓
• Change [[utrs:]] interwiki prefix and https://en.wikipedia.org/wiki/Template:UTRS
• Get accepted OAuth consumer for new domain

Day of requirements:

• All UTRS 1 data should be transferred to a read-only state for archive purposes
• Sandbox needs to be configured as the "Alpha" server w/OAuth setup for ACC OAuth wiki to save duplication
• Playground needs to be configured as the "beta" server w/OAuth setup for WMF wikis
• Beta domain needs to be transferred to the regular domain
• Possibly implement new domain name (under new wmcloud.org (https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage)?)
• Shutdown and destroy utrs-production-2
• File Phab task to note we no longer need the floating IP (not that we do even now)

Clean up (IP) appeal creation code

I'm going to do a hacky copy paste to get it running, but it really should be cleaned up. @supertassu can I get you to create a new issue to clean it up?

Originally posted by @dqwiki in #34 (comment)