mozilla / contribute.json

A JSON schema for open-source project contribution data.

contribute.json

NOTE: This project is decommissioned and is no longer active. We will leave the repository here along with the JSON schema and the code for the website, but the website itself has been taken down and the contributejson.org domain now redirects to the repository on GitHub.

A JSON schema for open-source project contribution data.

This is currently a proposal and is not yet stable. Suggestions and pull-requests welcome.

This is the current draft. I'm presenting the schema as an example instance using mozilla/bedrock as the subject. Previous discussion can be found in the comments of the original gist.

{
    // required
    "name": "Bedrock",
    "description": "The app powering (most of) www.mozilla.org.",
    "repository": {
        "url": "https://github.com/mozilla/bedrock",
        "license": "MPL2",
        // optional
        "type": "git",
        "tests": "https://ci.mozilla.org/job/bedrock/",
        "clone": "https://github.com/mozilla/bedrock.git"
    },

    // optional
    "participate": {
        "home": "https://wiki.mozilla.org/Mozilla.org",
        "docs": "http://bedrock.readthedocs.org/",
        // optional
        "mailing-list": "https://www.mozilla.org/about/forums/#dev-mozilla-org",
        "irc": "irc://irc.mozilla.org/#www",
        "irc-contacts": [
            "pmac",
            "jgmize",
            "malexis",
            "cmore"
        ],
        "chat": {
            "url": "irc://irc.mozilla.org/#www",
            "contacts": [
                "pmac",
                "jgmize",
                "malexis",
                "cmore"
            ]
        }
    },
    "bugs": {
        "list": "https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&product=www.mozilla.org",
        "report": "https://bugzilla.mozilla.org/enter_bug.cgi?product=www.mozilla.org&component=Bedrock",
        "mentored": "https://bugzilla.mozilla.org/buglist.cgi?f1=bug_mentor&o1=isnotempty&query_format=advanced&bug_status=NEW&product=www.mozilla.org&list_id=10866041"
    },
    "urls": {
        "prod": "https://www.mozilla.org",
        "stage": "https://www.allizom.org",
        "dev": "https://www-dev.allizom.org",
        "demo1": "https://www-demo1.allizom.org"
    },
    "keywords": [
        "python",
        "less-css",
        "django",
        "html5",
        "jquery"
    ]
}

Validation

We're currently using the JSON Schema standard and we publish our schema at schema.json.

You can, for example, use the json-schema-validator to validate your own contribute.json against this schema.

Flask app

There's a server-side app that is currently available on https://www.contributejson.org

Running the Flask app locally

# clone from the main repo
git clone https://github.com/mozilla/contribute.json.git

# go into the directory
cd contribute.json

# using virtualenv wrapper, create a new virtual environment for the project.
mkvirtualenv contribute.json

# intall the requirements
pip install -r requirements.txt

# install the npm dependencies
npm install

# generate the CSS files (use `grunt watch` to work on the LESS files)
grunt less

# You can run the app with
DEBUG=true python app

open http://localhost:5000/

NB! Most of the functionality is built as an AngularJS app.

Sentry

To use Sentry you just need to create an environment variable called SENTRY_DSN and it will be used.

contribute.json's Issues

Change to static site?

What if we changed this from a flask app running on Heroku to a GitHub pages static page that uses a JS schema validator to provide the schema validation function?

No place for translation contribution in the schema

I'm not sure yet where this info should go, but I believe it would be beneficial to call out specifically translation contribution URLs and contacts. I was originally thinking maybe it belongs under the "participate" key, but now maybe a top level "localization" or "translation" key is called for.

Make clear how to use this as a site owner

Not entirely clear here (examples would help, maybe?) what one would do as a site owner to make this happen. Put it into contribute.json at the root of the site when it's deployed (what about projects that aren't deployed)? Or into the repository root?

Make a distinct note about this in webdev-bootcamp

At the time of writing, @Osmose is working on a refactored webdev-bootcamp that has a mention about contribute.json.

We ought to have a whole paragraph about it there and how to add it.

Incorrect hrefs in examples page

Links in the contribute.json example page (https://contribute.paas.allizom.org/examples) have incorrect hrefs. For example, in the Kuma example...

<a class="ng-binding" ng-href="project._url" href="project._url">
    https://developer.mozilla.org/contribute.json
</a>

<a href="link.url" ng-href="link.url">
     <!-- ngIf: link.label=='repository' -->
     <!-- ngIf: link.label=='prod' --><span class="ng-binding ng-scope" aria-hidden="true" data-icon="" ng-if="link.label=='prod'">https://developer.mozilla.org/</span><!-- end ngIf: link.label=='prod' -->
</a>

contributejson.org seems to redirect back to mozilla.org homepage

STR:

1. Go to https://www.contributejson.org

`contribute.json`'s `contribute.json` out of date;)

Just a friendly hint:

contribute.json/contribute.json

Lines 15 to 18 in a1b54b6

	"irc": "irc://irc.mozilla.org/#webdev",
	"irc-contacts": [
	"peterbe",
	"pmac"

IRC has been replaced with Matrix, the project's own contribute.json needs updating with current chat location.

Add "under_active_development" property

I'd like to suggest a new root property: under_active_development.

It should signify that a project claims to be active and would like to get new contributors. Where it becomes useful is for sites that are no longer under active development; those sites can set it to false, and tools can in turn remove them from their listings or mark them as inactive or whatever they want.

It was suggested in IRC that inactive sites just remove their contribute.json file, but I feel like it's useful, at least from a historical perspective, to have the data in a contribute.json file still available, even if the site is inactive. I also think we shouldn't attach semantic meaning to the existence or non-existence of a contribute.json file.

Thoughts?

Sample JSON should show correct SPDX licence

This is published to landing page:

contribute.json/app/app.py

Line 30 in a1b54b6

"license": "MPL2"

For #76 this should better be SPDX:

   "license": "MPL-2.0"

Fixed in own .json in #116, missing from sample/homepage.

How can we turn the example schema to an actual schema

Suppose it's Friday afternoon and I want to write a validator that checks your.com/contribute.json URL.

If we write down the schema in a way that denotes required and optional it might work:
E.g.

{
    "name": "REQUIRED",
    "description": "REQUIRED",
    "repository": {
        "_parent": "REQUIRED",
        "type": "REQUIRED",
        "url": "REQUIRED",
        "license": "REQUIRED",
        "tests": "OPTIONAL",
        "clone": "OPTIONAL"
    },
    "participate": {
        "_parent": "OPTIONAL",
        "home": "OPTIONAL",
        "docs": "OPTIONAL",
        "mailing-list": "OPTIONAL",
        "irc": "OPTIONAL"
    },

With that it shouldn't be too hard to write a validator that checks that all keys whose value is "REQUIRED" is there and for the rest of the validator we could spit out: "Have you considered adding..." and/or "The following keys were not recognized..."

Joining with humans.txt and security.txt

There are similar projects to this like http://humanstxt.org/ and https://securitytxt.org/.
Would be great if they are all compatible with each other or even develop one standard that works for all.

Add raven to track errors

We're going to need a Sentry DSN in Heroku.

Canonical license identifiers/names

Identifiers for licenses can be written in many different ways - "GPLv3 or later", "GPLv3", "GPL3", ...

To put a stop to identifier proliferation in contribute.json from the start, there could be a preference for identifiers from the SPDX licenses list, https://spdx.org/licenses/ - and warning by the validator if the license doesn't match any of those.

P.S. this means that most existing contribute.json files should be changed from using MPL2 to MPL-2.0 - see https://spdx.org/licenses/MPL-2.0.html

P.P.S. and yes, the WTFPL is part of this list :)

TypeError - Object of type 'bytes' is not JSON serializable

See https://sentry.prod.mozaws.net/operations/contributejsonorg/issues/4704082/

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  File "app/app.py", line 136, in post
    content = json.loads(request.data)
  File "json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

TypeError: Object of type 'bytes' is not JSON serializable
  File "flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 35, in reraise
    raise value
  File "flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "app/app.py", line 140, in post
    'response': request.data,
  File "flask/json/__init__.py", line 321, in jsonify
    dumps(data, indent=indent, separators=separators) + '\n',
  File "flask/json/__init__.py", line 179, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "flask/json/__init__.py", line 81, in default
    return _json.JSONEncoder.default(self, o)
  File "json/encoder.py", line 180, in default
    o.__class__.__name__)

Examples of Use page broken

This comment broke the "Examples of Use" page. (thank you git bisect!)
cc @koddsson

At the time of writing you can see the problem on https://contribute.paas.allizom.org/examples
Something to do with the overflow I guess.
cc @schalkneethling

validator not working on mozilla.org or contributejson.org

Seemed like a reasonable thing to experiment with -- I got a failure to load every time.

So then I tried your site:
https://contributejson.org
which navigated to
https://www.contributejson.org/https:%252F%252Fwww.contributejson.org%252F
and displayed the following errors in window:

Sorry.
Something went horribly wrong. The request to the server couldn't even complete. Check the Web Console for possible additional errors. 

The web console showed:

Loading failed for the <script> with source “https://mozorg.cdn.mozilla.net/tabzilla/tabzilla.js”.
www.contributejson.org:312
<!DOCTYPE html>
	<html>
	  <head>
		<meta name="viewport" content="width=device-width, initial-scale=1">
		<meta charset="utf-8">
		<title>Application Error</title>
		<style media="screen">
		  html,body,iframe {
			margin: 0;
			padding: 0;
		  }
		  html,body {
			height: 100%;
			overflow: hidden;
		  }
		  iframe {
			width: 100%;
			height: 100%;
			border: 0;
		  }
		</style>
	  </head>
	  <body>
		<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
	  </body>
	</html> 503
controllers.js:208:17

A database of known working URLs

So the current https://contribute.paas.allizom.org/ site has the ability to remember recent submissions and list them. However, that's stored in cache (rather long termish).

How about we maintain a simple txt file here in the repo as the database. The app can download that periodically. That'd be a list of known working good examples of established sites.

What do you think?

Non-IRC chat references

Currently the schema has in the participate section both irc and irc-contacts.

As per this announcement, IRC is being replaced inside Mozilla with Matrix. This will make it impossible for Mozilla projects to satisfy the demands of the observatory which demands an IRC channel and contacts.

The easiest would be to just add matrix and matrix-contacts in a similar fashion, under participate, for example:

"participate": {
  "matrix": "#matrix:matrix.org",
  "matrix-contacts": [
    "@someaccount:matrix.org",
    "@someaccount:mozilla.org"
  ],
}

However, possibly a more future proof way would be to introduce an array of chat rooms as objects? Something like this, an example for the Matrix project:

"participate": {
  "chat": [
    {
      "location": "irc://chat.freenode.net/#matrix",
      "contacts": [
        "someaccount1",
        "someaccount2"
      ]
    },
    {
      "location": "matrix:room/matrix:matrix.org",
      "contacts": [
        "@someaccount1:matrix.org",
        "@someaccount2:mozilla.org"
      ]
    }
  ]
}

Not entirely sure what the keys would be. The Matrix URI scheme proposal is based on MSC2312 which is yet to be accepted.

Opening for discussion, happy to PR something as well if there is consensus.

Validate via direct input doesn't work

I paste a json in textarea and click validate button and nothing happens. Other options work as promised.

Referencing multiple licenses

Are there any thoughts on referencing multiple licenses in contribute.json?

For https://www.gimp.org/contribute.json I went with

"license": "GPL-3.0+, LGPL-3.0+"

The GIMP application code is licensed under the GNU GPL version 3 or any later, the library code (used by e.g. plug-ins) is licensed under the Lesser GNU GPL version 3 or any later.

Schema page

A page that explains the schema.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Make site pretty

The current site is functional but it's ugly and doesn't smell at all of being a Mozilla initiative.

Adoption by non-Mozilla projects?

Hi,

came across contribute.json when I was testing https://www.gimp.org with the Mozilla Observatory (i.e. via https://wiki.mozilla.org/Security/Guidelines/Web_Security#contribute.json ).

Do you expect adoption of this standard by non-Mozilla projects?

And if a project did (or a critical mass of them), how would the data they provide be accessible to users - do you plan to offer a directory for arbitrary projects where users can search/browse/find projects?

repository->type is excessive and almost always useless

The repository->type ought to be optional. It's almost always going to be "git" this and the next couple of years. Also, it's less likely to matter unless there's a repository->clone.

I'm not suggesting we delete it. Just that we make it optional.

Travis CI free usage ends Dec 3; mozilla repos should switch to other CI platforms

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Improve the site's score on the HTTP Observatory

https://observatory.mozilla.org/analyze.html?host=www.contributejson.org

Currently a D

UnboundLocalError: local variable 'content' referenced before assignment

https://sentry.prod.mozaws.net/operations/contributejsonorg/issues/4843923/

UnboundLocalError: local variable 'content' referenced before assignment
(4 additional frame(s) were not displayed)
...
  File "app/app.py", line 149, in post
    'response': content,

UnboundLocalError: local variable 'content' referenced before assignment

git.mozilla.org https clone urls don't validate

Reference:
https://www.bugzilla.org/contribute.json

The repo clone url listed there comes up as invalid when I try to run this on the validator on https://contribute.paas.allizom.org , but I assure you it is correct:

dave@rmbp:temp $ git clone https://git.mozilla.org/bugzilla/bugzilla.git
Cloning into 'bugzilla'...
remote: Counting objects: 93013, done.
remote: Compressing objects: 100% (23526/23526), done.
remote: Total 93013 (delta 67833), reused 89185 (delta 65193)
Receiving objects: 100% (93013/93013), 19.04 MiB | 5.64 MiB/s, done.
Resolving deltas: 100% (67833/67833), done.
Checking connectivity... done.

contribute.json should have a contribute.json

I would think that, even though the schema is not finalized, this project itself should have a contribute.json

TypeError on /examples.json

See https://sentry.prod.mozaws.net/operations/contributejsonorg/issues/4703600/

TypeError: startswith first arg must be bytes or a tuple of bytes, not str
  File "flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 35, in reraise
    raise value
  File "flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "app/app.py", line 210, in get
    if line and not line.startswith('#'):

Consider registering /contribute.json with .well-known

http://tools.ietf.org/html/rfc5785

SystemExit on URL downloads that take forever

See https://sentry.prod.mozaws.net/operations/contributejsonorg/issues/4704095/

The URL submitted was http://100.100.100.200/latest/meta-data/ which probably takes "forever" to download and would eventually fail. But before it can happen, Heroku's gunicorn running killed it. We could catch this sooner.

Remove tabzilla.js and replace with HTML

I'm not sure what it takes but whatever the tabzilla.js does before you click to expand, should be kept as HTML and just link to www.mozilla.org. Then we should remove tabzilla.js entirely.

Site not good on mobile

This is what it looks like if you pinch-zoom-out all the way.

CC @schalkneethling @craigcook @Osmose

TypeError on load-example

See https://sentry.prod.mozaws.net/operations/contributejsonorg/issues/4703964/

TypeError: Unicode-objects must be encoded before hashing
  File "flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 35, in reraise
    raise value
  File "flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "app/app.py", line 223, in get
    cache_key = 'project_%s' % hashlib.md5(url).hexdigest()

Ability to validate by text or by file upload

Currently we support validating by a URL.
We should support by text and by file too.

participate->home is hard to make required

Many of my projects that I would welcome contributions on (from staff or non-staff) don't even have a wiki page. Usually there's the github page or the docs page that I'd send people to.

The few projects I do have a wiki on are usually out of date.

In fact, I have projects that would welcome some contributions that don't even have docs. E.g. github.com/mozilla/airmozilla doesn't have docs.

Can we make the whole participate optional?

Syntax highlight JSON with https://highlightjs.org/

https://highlightjs.org/ seems the best tool for the job these days. There's also https://github.com/pc035860/angular-highlightjs which might make things easier.

Validator doesn't match Mozilla security guidelines

According to https://wiki.mozilla.org/Security/Guidelines/Web_Security#contribute.json, the "urls" attribute is required. However, the schema that is validated at contributejson.org doesn't list that as required, so the HTTP Observatory won't give you the points for a contribute.json that is ruled valid here.

contributejson.org's contribute.json lacks a participation section

It would be great to have at least an IRC room (participation -> irc) listed, even if it's not in its own unique room. IRC contacts (participation -> irc-contacts) might also be nice, if somebody wants to be that person.

Thanks!

https://www.contributejson.org/contribute.json has an incorrect prod URL (paas.m.o)