decred / politeia

politeia

Politeia is a system for storing off-chain data that is both versioned and timestamped, essentially “git, a popular revision control system, plus timestamping”. Instead of attempting to store all the data related to Decred’s governance on-chain, we have opted to create an off-chain store of data that is anchored into Decred’s blockchain, minimizing its on-chain footprint.

The politeia stack is as follows:

~~~~~~~~ Internet ~~~~~~~~~
            |
+-------------------------+
|      politeiawww        |
+-------------------------+
            |
+-------------------------+
|        politeiad        |
+-------------------------+
            |
~~~~~~~~ Internet ~~~~~~~~~
            |
+-------------------------+
|        dcrtimed         |
+-------------------------+

Core software:

• politeiad - Reference server daemon. Data layer.
• politeiawww - Web backend server; depends on politeiad. User layer.

The politeiawww APIs and politeiawww client can be treated as stable. All other APIs and libraries should be treated as unstable and subject to breaking changes.

See the politeiad README for instructions on building and running politeiad.

See the politeiawww README for instructions on building and running politeiawww.

politeia's Issues

[www] Ref client should send file digests

2 issues:

• Reference client doesn't send the digest along with the files
• Server does not return an error for missing digest

[www] Make static assets actually point to a directory

When writing the router I added a file browser and it points to the binary location. Make this point to an actual directory.

[www] Lazy load inventory from politeiad

www shouldn't required politeiad to be running when it starts up, so inventory shouldn't be loaded on startup but on the first proposal-related request.

Admin actions should probably be protected with voting

Administrative actions (like publishing/censoring a proposal) should probably not be absolutely available to a single admin level user, because it opens up potential for abuse. Similar to Stack Overflow moderation, it's probably better if an administrative action (especially a non-reversible action) can only be performed after a few different users have voted to take it.

It might be beneficial in the long run to eventually adopt a few other ideas from Stack Overflow's moderation system, like different levels of moderator permissions and elections for new moderators, to utilize the community to make Politeia more self-sustaining.

[www] Need a separate route for creating a new session

Currently the / route creates a new session and returns route info, but /v1/user/me is used by the client to check if a session already exists. So the order the client must make the calls is:

1. /v1/user/me to check if user has session
2. / to get the route info ({ route: "/v1" }) and create a session

That ordering doesn't make sense, because the client has to know which version of the API to use before getting that information from the API. We need a separate endpoint to create a new session.

[www] Add policy RPC

Add a RPC that returns policy settings of www. Use GET route /api/v1/policy and reply with a JSON message like so:
type PolicyReply struct {
MaxImages uint
MaxImageSize uint
MaxMD uint
MaxMDSize uint
ValidMIMETypes []string
}

[gui] Add node.js frontend to visualize politeiawww

Please add a node.js GUI that transforms politeiawww JSON to pretty pretty UX/UI.

Users should not be able to see unvetted proposals

Currently unreviewed and censored proposals are accessible via direct link to the public, and this could be a liability if a malicious user uploads illegal images. Unvetted proposals should only be accessible to admin users and direct links to unvetted proposals (when accessed by non-admins) should the censorship token & status. We could return 404 for those proposals, but if we want to implement something like #97, then that information could be shared in a publicly accessible way through the proposal details page.

[www] Derive title from index.md in proposal

Currently the proposal title is not part of the merkle and therefore signature, so it's possible for a user to submit a perfectly valid proposal to Politeia with a bad/offensive title, have it censored, and be able to make a case to others that the valid proposal was unfairly censored.

To address this, the title will be embedded in the index.md file as the first line with text by the client, and www should extract it, strip any markdown characters (if present), and then set it on the ProposalRecord before sending to politeiad.

[policy] Need to come up with expected proposal format

Before the real go-live, we need to come up with a standard format (or at least a minimum set of information) that is expected of all proposals.

[politeiad] Rework errors

Politeiad is returning 400 Bad Request for errors due to user input, but the error message is a string rather than a numeric error code, which would be easier for www to process. Errors should be reworked to return JSON with 400 responses with an error code and, optionally, a human-readable error message.

unused vars

politeiad/backend/gitbe/git.go:108:6:warning: stderrError declared but not used (unused)
politeiad/backend/gitbe/git.go:89:6:warning: stdoutError declared but not used (unused)

Title should not be sent for inaccessible proposals

When a regular user accesses proposal details for unreviewed/censored proposals, the server does not send the markdown or images. It should also not send the title.

[policy] Figure out how to do proposal submission payments

We have had some vague ideas on how to do payments when submitting a proposal. Let's debate and nail down the method in this issue.

[www] Add new route

Add a route that adds a new unvetted proposal to the repo. This is also the right location to start enforcing some policy. We need a max file size, max files and files types (make these config settings with sane defaults).

Currently, politeiad allows only text and png. We should add svg to that mix.

Don't forget to add the new proposal to the www cache!

[www] [release-blocker] Can't access /proposals/{token} directly

You can't access the /proposals/{token} url directly by copying/pasting into the browser or clicking a link, because the route doesn't correctly support url params.

Example: https://test-proposals.decred.org/proposals/5a90228603c98f194a3c8813be9017330043146b389be6cc551348cbe4c32aed

[ui] Add front page

The front page is a list of getvetted input. We should probably make the proposals clickable so that we can drill down on them. Ultimately we need to render the md with graphics in a reasonable manner.

[politeiad] Merkle verification sometimes fails when running on Windows

Merkle verification can fail for newly submitted proposals when running on Windows with the following git config:

git config --global core.autocrlf true

Steps to reproduce:

1. Set the above config.
2. Submit a new proposal with \r\n characters.
3. Try to access the proposal details.

Politeiad should set core.autocrlf to false locally within both vetted and unvetted repos to avoid this issue where git automatically converts between \r\n and \n.

[www] Submitting user needs to be sent with each proposal

Currently there is no way for the client to know which user created which proposal. We need www to support user-proposal associations in the database, and to send the user id (and other identifying information) along with each proposal record.

[www] Add a policy for proposal title limit

Politeiad rejects proposal names that are greater than 80 characters. There should be a policy in www for this. www is also validating what characters are allowed, so we should have a separate policy that includes the regex of supported characters as well.

Runtime error when trying to fetch proposal that doesn't exist

$ politeia -v -testnet -h 127.0.0.1 getunvetted 959c855726c75270ef435c67c5f7fa86458084f3dde62e4e9462b46b15ec2147
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x6e85d7]

goroutine 1 [running]:
github.com/decred/politeia/api/v1.Verify(0xc04243e340, 0xb, 0xc04243e350, 0x9, 0x34d3653afe46033, 0x455d02561df587f, 0x7f53a817b75154ac, 0xc95b2b1797abd27b, 0x7f0054865f7ae017, 0xf0acaf7ba5fe8cc2, ...)
        C:/projects/gopath/src/github.com/decred/politeia/api/v1/v1.go:96 +0x407
main.getUnvetted(0x0, 0x0)
        C:/projects/gopath/src/github.com/decred/politeia/cmd/politeia/politeia.go:487 +0x9e1
main._main(0xc04202bf70, 0x4044ab)
        C:/projects/gopath/src/github.com/decred/politeia/cmd/politeia/politeia.go:782 +0x3ca
main.main()
        C:/projects/gopath/src/github.com/decred/politeia/cmd/politeia/politeia.go:798 +0x2d

[www] Add vetted route

In order to render the main page we need the getvetted route. Please do something similar as getunvetted. We may want to create a separate map for unvetted proposals.

[ui] Add new proposal page

This is where we are submitting the proposal. We probably need to show a text box where the user can enter md and upload some files and do a "view" to see it rendered. I am open to other ideas.

I think GitHub has nailed the md editing quite well so let's borrow some ideas from them. Maybe there is even some code out there.

[www] Change unit tests to test the full routes

Use https://godoc.org/net/http/httptest to test routes which include the logic within www.go instead of the current backend_* tests.

Need to implement a reason for proposal censorship

@davecgh mentioned this has been discussed, so I'm just creating an issue to capture it.

Currently there is no support for the admin providing a reason to the user for censoring a proposal.

[ui] Add proposal lists from mockup

Add the default/user/admin proposal lists from the mockup. The unreviewed admin list should have the ability to Publish/Censor.

Need a tool to verify a proposal has been submitted to Politeia

There is no way to verify that the signature provided as part of the censorship token data is signed by Politeia.

If a user says his proposal has been unjustly censored by Politeia, it would be nice to have a CLI or some other tool for others to easily verify his claim.

[www] Revise the signup flow

• Split ProcessNewUser into UpdateUserVerification and InsertNewUser within backend.go
• Change webserveraddress to webhost

[www] Add email verification for new users

Use (make sure to glide get it): https://godoc.org/github.com/dajohi/goemail

• If email credentials are empty DON'T send email verification
• If email credentials are partially filled in FAIL
• If email credentials are filled in assume it is working correctly, or if feeling adventures ensure mail server is working and FAIL if it isn't.

[politeiad] ~ doesn't expand properly

in at least politeiad.conf

[www] Make template a file

#72 kills me on the inside and we need to move this to a file. This must not be in a relative path to the binary. I suggest we put it in the same directory as the config file and look for it, by default, there.

Proposal list pagination

We need to be able to paginate api endpoints for proposal lists.

I recommend the approach reddit uses.

A list is sorted in some way (let's assume new)

To get a page, I specify the limit (max number of items returned) and an optional start/end point.

You could have a default limit of 20.

For these examples, assume the list is sorted by newest first.

To get a page of i items I fetch

/vetted?limit=20

To get the next oldest page, I would fetch:

/vetted?limit=20&before=<idofthelastiteminpreviouspage>

A meta key can be added to the response as a sibling to proposals

{
  proposals: [...]
  meta: { total: 9123 }
}

[ui] Add captcha during user signup

I hate capthca as much as the next guy but we need to prevent robots and floods.

An idea was thrown out there to require a user to pay a nominal fee to sign up. I kind of like that idea as well.

Thoughts?

[www] Add caching for Inventory command

The politeiad Inventory command is very expensive time and bandwidth wise. We, therefore, need to cache proposal records at start-of-day. At this time we should shoot for an implementation that fires off the Inventory command and then places the vetted and unvetted branches in memory use a map with the censorship token as the key). Currently, the Inventory call does not filter and will always reply with everything it has. That is a separate issue that will be addressed once we get the code to work.

Let's not cache to disk yet because reconcile code is always hard.

[www] Add a policy for comment size limit

We should have a new policy representing the comment size limit, either in bytes or characters that is sent in /v1/policy response.

protect /signup/new

[www] Add support for pagination to the proposal list routes

Pagination should be done like reddit's API, with support for the following:

• after: the id of the last proposal on the previous page
• before: used for navigating backwards through the list in the UI, it's the id of the first proposal on the previous page
• limit: (optional) the number of proposals to fetch in the page

[nginx] Hitting a proposal url directly doesn't return response data

I think this is an nginx issue; when I open https://test-proposals.decred.org and navigate to a proposal, it correctly opens. But if I go to the address bar and press enter to navigate directly to the same proposal, it sometimes returns 200 OK for the request but there's no response data.

[www] Fix /setstatus route

/proposals/{token}/setstatus needs to be fixed to instead be /proposals/{token}/setstatus/{proposalstatus} to have consistent parameters.

Loop in signup?

browser was stuck in what seems like a loop.. hitting server every ~200ms

2017-10-19 15:07:02.110 [INF] PWWW: 70.114.160.212 via 10.135.7.135:48530 GET / HTTP/1.1                             
2017-10-19 15:07:02.301 [TRC] PWWW: GET / HTTP/1.1          
Host: test-proposals.decred.org                             
Accept: */*                   
Accept-Encoding: gzip, deflate, br                          
Accept-Language: en-US,en;q=0.8                             
Connection: upgrade           
Cookie: _gorilla_csrf=MTUwNzIyMDE0OXxJa3BLUkRVME5rZEtLM0pJUkRONFNHSkpiWEYxV2xSQlpsUndRMnBJZUZGeFZWaFZPRWQzVHpOMFVVRTlJZ289fK8mI9mx3K8Wbzywd3HwAtar3dRKDyMwsQ7O_I7TlwPl
Dnt: 1                        
Referer: https://test-proposals.decred.org/user/signup      
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
X-Forwarded-For: 70.114.160.212                             


2017-10-19 15:07:02.301 [INF] PWWW: 70.114.160.212 via 10.135.7.135:48532 GET / HTTP/1.1                             
2017-10-19 15:07:02.473 [TRC] PWWW: GET / HTTP/1.1          
Host: test-proposals.decred.org                             
Accept: */*                   
Accept-Encoding: gzip, deflate, br                          
Accept-Language: en-US,en;q=0.8                             
Connection: upgrade           
Cookie: _gorilla_csrf=MTUwNzIyMDE0OXxJa3BLUkRVME5rZEtLM0pJUkRONFNHSkpiWEYxV2xSQlpsUndRMnBJZUZGeFZWaFZPRWQzVHpOMFVVRTlJZ289fK8mI9mx3K8Wbzywd3HwAtar3dRKDyMwsQ7O_I7TlwPl
Dnt: 1                        
Referer: https://test-proposals.decred.org/user/signup      
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
X-Forwarded-For: 70.114.160.212                             


2017-10-19 15:07:02.475 [INF] PWWW: 70.114.160.212 via 10.135.7.135:48534 GET / HTTP/1.1                             
2017-10-19 15:07:02.654 [TRC] PWWW: GET / HTTP/1.1          
Host: test-proposals.decred.org                             
Accept: */*                   
Accept-Encoding: gzip, deflate, br                          
Accept-Language: en-US,en;q=0.8                             
Connection: upgrade           
Cookie: _gorilla_csrf=MTUwNzIyMDE0OXxJa3BLUkRVME5rZEtLM0pJUkRONFNHSkpiWEYxV2xSQlpsUndRMnBJZUZGeFZWaFZPRWQzVHpOMFVVRTlJZ289fK8mI9mx3K8Wbzywd3HwAtar3dRKDyMwsQ7O_I7TlwPl
Dnt: 1                        
Referer: https://test-proposals.decred.org/user/signup      
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
X-Forwarded-For: 70.114.160.212                             


2017-10-19 15:07:02.654 [INF] PWWW: 70.114.160.212 via 10.135.7.135:48536 GET / HTTP/1.1                             
2017-10-19 15:07:02.825 [TRC] PWWW: GET / HTTP/1.1          
Host: test-proposals.decred.org                             
Accept: */*                   
Accept-Encoding: gzip, deflate, br                          
Accept-Language: en-US,en;q=0.8                             
Connection: upgrade           
Cookie: _gorilla_csrf=MTUwNzIyMDE0OXxJa3BLUkRVME5rZEtLM0pJUkRONFNHSkpiWEYxV2xSQlpsUndRMnBJZUZGeFZWaFZPRWQzVHpOMFVVRTlJZ289fK8mI9mx3K8Wbzywd3HwAtar3dRKDyMwsQ7O_I7TlwPl
Dnt: 1                        
Referer: https://test-proposals.decred.org/user/signup      
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
X-Forwarded-For: 70.114.160.212                             


2017-10-19 15:07:02.826 [INF] PWWW: 70.114.160.212 via 10.135.7.135:48538 GET / HTTP/1.1    

[www] Clean up www and backend

• Create a new UserError error type in backend.go (see https://github.com/decred/politeia/blob/master/politeiad/backend/gitbe/git.go#L17-L28 for reference) and use that to represent all user errors.
• Add logic in www.go to check the error type returned by functions in backend.go and decide on the appropriate action. Internal (real) errors should be logged with the originating IP address. User errors should be returned as part of the JSON. Both errors should return an appropriate HTTP status error code.

[www] Need validation for emails test vs [email protected]

Currently we're only performing strict text comparison of emails, so you can hit /v1/user/new to spam one email address by using different text variations which will deliver to the same email address. For example:

[email protected]
[email protected]
[email protected]
[email protected]
...

[www] Add a redis backend

Currently, we stuff everything in a local leveldb. It is an interface and we should add a redis option in order to be able to horizontally scale and be more container-y.

[www] Get user's IP using the X-Forwarded-For header

logging.go currently uses http.Request.RemoteAddr, but that apparently won't work behind nginx, so fetch it using the X-Forwarded-For header. Example: https://stackoverflow.com/a/26223739

[www] Server should report invalid mime type to UI

If you create a JPG/BMP and save as a PNG and upload it with a proposal, the server will return a 500 error. It should return a valid invalid-mime-type error code along with the detected mime type of the file.

Add startup failure when webserveraddress config isn't populated

If mailhost, mailuser, and mailpass are populated, webserveraddress should be populated too.

Multiple cookie values

We sometimes end up with the browser sending multiple cookie values.

It's not yet clear if this is a client or server issue.

[verification tool] Add JSON in- and output

We need to add a JSON mode to the verification tool. The idea is that we should be able to spit out the JSON once a proposal is submitted and then feed that directly into the tool for verification.

We probably need something like -jsonin to import the JSON submission reply. When that flag is set -k -m etc should cause the tool to abort. We should only do one or the other.

We also need a -jsonout flag that replies in some dort of JSON format. We can then use this tool as a backend to some GUI.

use templates.

All files required need to be read into a global variable on startup
See https://golang.org/pkg/html/template/#Must

Move all templates to templates/ directory.

[www] Code refactoring for www.go

Refactor www.go and backend.go so that functions in backend.go are smaller and more cohesive and www.go handles the incoming commands and creates the outgoing replies.