freecodecamp / open-api

freeCodeCamp's open-api Intiative

open-api

This project is currently being refactored, performance and API will be unstable.

About

open-api is a graphQL API that will serve multiple purposes:

• serve the frontend of freeCodeCamp
• an implementation of the freeCodeCamp's open-data policy
• allow developers to build applications around the freeCodeCamp's eco-system and its open data sets

Urls

environment	url	method
staging	https://hxtsoafqna.execute-api.us-east-1.amazonaws.com/stage/api	GET
staging	https://hxtsoafqna.execute-api.us-east-1.amazonaws.com/stage/graphql	POST
production
production

Contributing

We welcome pull requests 🎉! Please follow these steps to contribute.

Updating certificates

Tokens are verified using public keys, each tenant will have their own certificate containing the public key.

Certificates are stored either on developer laptops in .env files, or in an environment variable JWT_CERT for deployments. We use Travis for deployments, and scripts/deploy.sh will pick either JWT_CERT_STAGE or JWT_CERT_PROD and export it as JWT_CERT. This will be picked up and deployed by Serverless.

Certificates are base64 encoded to prevent encoding issues. This works around the fact that Travis uses Bash to export environment variables, which causes issues with newlines and other characters have a special meaning in shells.

To add a new certificate, download it as a .pem file, and base64 encode it. Use yarn encode-file if you want a convenient script:

▶ yarn encode-file ~/Downloads/freecodecamp-dev.pem
yarn run v1.6.0
$ node scripts/base64encode.js /Users/ojongerius/Downloads/freecodecamp-dev.pem
Original contents:

-----BEGIN CERTIFICATE-----
MIIDDzCCAfegAwIBAgIJGHAmUeq9oGcAMA0GCSqGSIb3DQEBCwUAMCUxIzAhBgNV
<SNIP>
zIPPbMj9c6D7tETg2ZeHEthScPsgoPSHXxYu5N9ImoY/KLjDD5Nk364e0M+ZT8rF
rbgxgxHNJH92enBwsqrq7CWi2Q==
-----END CERTIFICATE-----

Base64 encoded (copy this):

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tDQpNSUlERHpDQ0FmZWdBd0lCQWdJSkdIQW1VZXE5b0djQU1B
<SNIP>
MzY0ZTBNK1pUOHJGDQpyYmd4Z3hITkpIOTJlbkJ3c3FycTdDV2kyUT09DQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tDQo=
✨  Done in 0.23s.

And copy the base64 encoded string to your destination.

Deployment

Deployment is normally done by CI.

Manual Deployment

If you want to do a manual deployment:

Configure your AWS credentials, see https://serverless.com/framework/docs/providers/aws/guide/credentials

Ensure that you have the serverless package install globally

yarn global add serverless

Assert that the stages configured in serverless.yml in line with what you'd like to deploy to, and run:

serverless --stage=YOUR_STAGE_HERE deploy

Getting an API key

TBD

License

Copyright (c) 2018 freeCodeCamp.

The computer software in this repository is licensed under the BSD-3-Clause.

open-api's Issues

Create contributing file

Too early to do now, but let's not forget to create a guide when this project is in a shape where it's ready for other contributors.

• Add contributing guide on lines of the main repo, but minimal.
• Add contacts for general queries.
• Add policy to report security issues. have raised #146
• Add instructions on creating a PR (for ex: PR must come from forks, etc.)

Setup CI

@raisedadead just copy the https://github.com/freeCodeCamp/freeCodeCamp/blob/staging/.travis.yml?

• Configure travis.yml
• Add Travis badge

Snyk report

Thanks for opening up access to Snyk @raisedadead!

There is one low severity vulnerability found; Prototype pollution via the "hoek" module. ( https://snyk.io/vuln/npm:hoek:20180212 )

There are bunch of libraries using it, I've raised a PR for one, but there are a few others and chasing that all up does not seem like a wise use of my time.

Does Snyk support whitelisting of "blessed" vulnerabilities, or do you adjust the sev level and ignore say "low"? @freeCodeCamp/open-api how do you normally handle these reports in the freeCodeCamp org?

General report: https://snyk.io/test/github/freecodecamp/open-api?severity=high&severity=medium&severity=low

prepublish should be prepublishOnly

open-api/package.json

Line 25 in 2652e5c

"prepublish": "yarn snyk-protect",

This is creating conflicts in yarn.lock and should ideally be removed or migrated

Update package name and possibly repository name too.

Currently the package name is open-data and repository name is open-api.

I propose we change both. It's too early to decide, but this is an issue to remind us to decide this before we launch anything.

Relax rules for commit messages

I like the rules for commit messages, but please can we relax them.

My proposal:

$1($2): $3

$1: One of feat fix chore docs tests
$2: Any single word, any case
$3: Any string, any case

Example:

feat(types): Expand User type and apply 'on-the-fly' migrations

Reasoning:

I like to keep my commit messages consistent with the code is touches. If I fix a bug in someBuggyFunction, I want to reference it in my commit message. Or if I am working with types, I want to use StartCase in the commit message to reference the type I have worked on, which uses StartCase in the code base.

chore: planning and discussions.

Close and move issues from https://github.com/freeCodeCamp/freeCodeCamp/projects/5 to this tracker.

We should still be using the project board above, while reporting and discussing the threads here.

Discuss deployment strategy

@raisedadead whats your normal strategy here? Function as a service could be cost effective, but it looks like Digital Ocean does not support it.

Implement strategy against denial of service attacks

If we are going to use GraphQL this will be a little more complicated than using REST.

Things to consider:

• Query depth
• Timeouts
• Query complexity
• Rate limiting based on server time vs query complexity

This article provides a good background:

https://github.com/howtographql/howtographql/blob/master/content/graphql/advanced/4-security.md

Githubs approach: https://developer.github.com/v4/guides/resource-limitations/

Schema generation

We can create GraphQL schemas from Mongoose schema. I love the idea of making sure the GraphQL schema and DB model is in sync.
However, after the initial migration we won't have to rely on that one MongoDB instance anymore, and this design makes less sense.

Figuring out a way forward should inform #37

Docker Setup

We should have a docker setup for this early on.

Query Auth0 for accountLinkId in directives if missing

It is possible for clients to send a token for an existing user that does not have the accountLinkId, even though the user has one assigned to them and an entry in the database.

• User logs in/creates an account. The token sent to create the account has no accountLinkId
• As part of creating an account, we assign a uuid as the accountLinkId and update the user record held at Auth0
• Subsequent requests from this user still holds a token with no accountLinkId
• Here lies a potential never-ending loop of multiple accounts for the same user, until the user logs out and in again to gain their new token.

There is scope for setting the accountLinkId over in Auth0, so every token has an accountLinkId from the very first login.

Do we still need to handle the case for if we end up setting it here because it is missing? I think it would lead to some crappy UX having to invalidate the token and having new users log out and in again.

I am going to raise a PR that will add the current rules we have set in Auth0 to this repo. Just so we all have some visibility of them and what they are doing.

Configure monitoring

How is monitoring set up at the moment, what do other freeCodeCamp project do? Can and if so, are we happy to re-use this?

/cc @freeCodeCamp/open-api

Current status of the availability of an API key?

Hi @roelver how are you?

The current readme of the project has a paragraph that shows as follows:

Getting an API key

All requests must have a API-key in the request. In this stage there is no online resource to generate an API-key. If you want an API key for your app, please ask for it on the FreeCodeCamp/DataScience Gitter. The API-key is specified in the query string as key=.

This is driving traffic into the room when we haven't been able to give a clear answer about the existence of the API.

Shouldn't that message be changed into something like "on-hold" if the API key is not currently available?

Or should we provide those keys?

Set up SNS notifications for critical alarms

Definition of done: critical alerts create a phone call to team members.

This could be possible by having critical alarms firing of separate SNS topics that have a Twilio webhook as subscriber.

I've seen people create Lambas to connect to Twilio when they fire, but that kind of defeats the purpose, we want to know when Lambdas are on 🔥

Warning: this will be less sophisticated than services like Pagerduty, VictorOps etc, having a schedule, and escalations is well out of scope for this issue.

/cc @freeCodeCamp/open-api did I miss anything, and concerns? Is this a blocker for our first release?

Switch to Yarn

@raisedadead proposed to switch to Yarn. Some discussion took place in PRs, which I'll link here for posterity.

Original discussion took place in #59, I mentioned:

I'd be happy to start using Yarn!

I guess the use case is a little less compelling now that NPM has a lock file. However the speed difference still looks compelling when node_modules and a lockfile are present -judging by some of the results at https://github.com/pnpm/node-package-manager-benchmark .
@Bouncey anything to add?

And @Bouncey added:

yarn does things that are nice like --prune automatically, which is manual only in npm. Plus you get shorter script commands: npm run format becomes yarn format

Yay for yarn!

Stack traces reference the webpack bundle

{ TypeError: Cannot read property 'authorization' of undefined
    at checkAuthAndResolve (/home/stuart/open-api/.webpack/service/handler.js:485:33)
    at users (/home/stuart/open-api/.webpack/service/handler.js:660:109)
    # ...
}

Not much of a problem whilst I am developing the feature that is throwing errors, but not very helpful for future developers potentially having to cross-ref the bundle to the source manually.

Expected output:

{ TypeError: Cannot read property 'authorization' of undefined
    at checkAuthAndResolve (/home/stuart/open-api/graphql/resolvers/checkResolvers.js:10:47)
    at users (/home/stuart/open-api/graphql/typeDefs/User/index.js:5:17)
    # ...
}

Configure logging

Where do our logs go, what do other freeCodeCamp project do? Can and if so, are we happy to re-use this?

/cc @freeCodeCamp/open-api

Architecture / data sources

Hi there,

We need to make some decisions on how to move forward on this project, and what the scope is going to be. There are a few options to discuss, they should probably be broken apart and judged by their merits, but let's see what options there are first:

We expose GraphQL via this project and operations are resolved..

• using REST calls served by loopback in the freecodecamp site.
• using REST calls served by loopback in this project, models are kept in sync by pushing them into a shared library. Data lives in the current MongoDB server and is read from and written to from both this project and the freecodecamp website.
• using code generated from loopback models that are kept in sync by pushing them into a shared library. Data lives in the current MongoDb server and is read from and written to from both this project and the freecodecamp website.
• we recreate all models by hand in open-api. Just kidding.
• ... your solution here.

One party pooper is that I've seen is that the REST API return responses that I did not expect. An example is hitting http://127.0.0.1:3001/api/users, I happily received a 200 OK, but useless HTML. I've not explored why this is set up like it is, could someone give me some context?

Lambda warming

A "cold" Lambda takes a while to start. A common technique is to to have a scheduled events invoke the Lambda. This will make AWS start a container and process, which it will then keep around in case another invocation will follow. Those subsequent invocations will be much faster, as all set up has already been done.

Keeping Lambdas warm in a low traffic is important to keep things responsive. However it is not needed if there is enough traffic to have warmed containers around at any time.

Before we spend more time investigating the best way to keep Lambdas warm, could someone provide insights on the frequency and volume of traffic we have, to see if we need to worry about this at all?

@raisedadead, @Bouncey could you shed some light on this?

2 approaches the first approach would be more cost efficient as the warming invocation could have a shorter running time, and hence cost, than an health endpoint invocation. The other plus is that there is no API gateway hit involved either, saving more cost:

• A serverless plugin that creates a scheduled Lambda that can invoke other Lambdas: https://github.com/FidelLimited/serverless-plugin-warmup . A blog post about this plugin: https://serverless.com/blog/keep-your-lambdas-warm . This is kind of approach is referred to in an AWS answer in https://forums.aws.amazon.com/thread.jspa?threadID=234124
• Have a monitoring endpoint hit an health endpoint (like /health) to monitor statistics. This invocation would keep the endpoint warm, although there would be a bigger chance of a normal client request then trigger a cold start while the monitoring endpoint is hitting the health endpoint.

Rename test file

./graphql/resolvers/directive.test.js ----> ./graphql/resolvers/directives.test.js

Feat: User type Outstanding tasks

• remove or improve deleteGDPRUser
• agree if tests should have uncaught errors

From #55 (comment)

Question about this API

Is this still a working project?

I am working on a project with some other campers and we are trying to find an easy way to figure out how many certificates (if any) a FCC user has achieved. Is it possible for us to use this API for that purpose?

Explore testing with Jest

As recommended in https://github.com/withspectrum/spectrum/blob/alpha/docs/backend/api/testing.md

See https://github.com/facebook/jest

@QuincyLarson @raisedadead @Bouncey any options on testing here, lessons learned that I could build upon?

As an existing user, I want to see what information freeCodeCamp holds on me

This should be in the form of a JSON file.

Possible separate lambda function?

Setup Snyk

• Setup repo on snyk.io
• Add badge
• Add npm run prepare-production to be executed only in deploy environments

Only run commit linting in PR builds

It's good to have commit linting in PRs, but has little value in branch builds, specially a deployment build.

We could test the value of ${TRAVIS_PULL_REQUEST} and only run commitlint-travis if it is set to true.

Decide on authentication

• for interactions from a user perspective. This could be the freeCodeCamp website or, for example, software like a IDE plugin that would pull current challenges in, and submit solutions on their behalf
• for 3rd parties that want to build other functionality -an example being leaderboards

Related issue suggesting to : freeCodeCamp/freeCodeCamp#16104 (limited to Collaborators, so I am unable to chip in 😞 )

I think we should consider using auth0, they provide a secure and scalable solution, and they have a unlimited free tier for open source projects. I doubt we could do a better job than auth0, and it would unblock the team to work on projects.

The library support looks great, see https://auth0.com/opensource They support social providers, passwordless, and MFA, see https://auth0.com/docs/identityprovider for an overview.

/cc @raisedadead @QuincyLarson @BerkeleyTrue

Enforce conventional commits

As done in FCC monolith.

Performance verification

We need to verify, rather early than later, that performance between our Lambdas and our current MongoDB server is acceptable. If it is not, we could consider moving our MongoDB server to AWS, or see what performance a hosted solution would give us.

@freeCodeCamp/open-api any thoughts on how we can verify this? Stand up another Mongo server in DO, , let open-api staging communicate with it an measure the latency? Do we want to compare Learn -> open-api -> Mongo with www.freecodecamp.org?

Support pagination

As part of #25 I'm looking at ways to protect ourselves against harmful queries. One of the easier ways to prevent queries from generating excessive load is to set a limit to the amount of items a query return can return. Setting a limit on the amount of results returned requires us to support pagination, hence this issue.

/cc @Bouncey, @raisedadead

As an existing user, I want to be able to delete all my information from freeCodeCamp

Nothing should remain in the database for the user being deleted

As a new user, I want to create a new account

This account must be linked to the corresponding auth0 account by uuid

Configure / setup caching

See http://graphql.org/learn/caching for background. Current implementations are inMemory and memcache. As per the docs it's probably wise to start with in memory. There is a request open to use Redis, but progress on it is unclear: https://github.com/apollographql/apollo-engine-js/issues/55 .

Configure performance tracing

See announcement blogpost at https://dev-blog.apollodata.com/exposing-trace-data-for-your-graphql-server-with-apollo-tracing-97c5dd391385 and the docs at https://www.apollographql.com/docs/engine/performance.html . It' would be good to start recording this data .

If we run from Lambda, we'll need a standalone container to keep track of state, see https://www.apollographql.com/docs/engine/setup-node.html#lambda and https://www.apollographql.com/docs/engine/setup-lambda.html

• Configure tracing
• Enable compression to make up for the extra data sent for each request

Update License files

The current license is MIT ( https://github.com/freeCodeCamp/open-api/blob/staging/LICENSE ) and mentions "feathers", so I'm guessing it's a copy.
freeCodeCamp uses BSD 3-Clause (https://github.com/freeCodeCamp/freeCodeCamp/blob/staging/LICENSE.md) for the code.

There is another license file https://github.com/freeCodeCamp/open-api/blob/staging/LICENSE.md that seems to pertain to the the dataset, and refers to http://opendatacommons.org/licenses/odbl/1.0/

@raisedadead do you know what to do here? Are there any guidelines or strong opinions here?

Authorisation

We have decided on authentication in #23. We'll now need to think about authorisation.

Authentication can be done using Auth0. Both /graphql and the graphcool IDE endpoints should be approachable without authentication, and only queries and mutations needing authentication should require it.

User groups

• unauthenticated users -this might not be available for a while or at all
• 3rd parties acting on behalf of users
• 3rd parties authenticating with API keys
• logged in users, most likely via the site

Authorisation

There are plenty of options here as the specification does not dictates a solution.

• Business logic layer: the recommendation of the GraphQL best practices doco (http://graphql.org/learn/authorization/)
• Chaining resolvers, see https://github.com/thebigredgeek/apollo-resolvers and the corresponding talk at Apollo Day https://www.youtube.com/watch?v=xaorvBjCE7A
• GraphQL shield is a project separating permission logic from the application logic allowing you to pass your resolvers and a permission object into it's API https://github.com/maticzav/graphql-shield
• Directives, see https://blog.graph.cool/graphql-directive-permissions-authorization-made-easy-54c076b5368e based on a talk by Ryan Chenkie at GraphQL summit: https://www.youtube.com/watch?v=4_Bcw7BULC8
  #25

A related thread full of interesting viewpoints ardatan/graphql-tools#313

Deploy toy GraphQL server using Lambda

Deploy a simple Lambda serving GraphQL, using Travis. Initially against a small staging database.

• Deploy a Lambda from Travis to AWS
• Deploy a small staging database, mLab was suggested

Implement MVP queries and mutations for events

Definition of done: queries and mutations needed by the events project are supported by open-api.

docs: add contributing guide

• Add contributing guide on lines of the main repo, but minimal.
• Add contacts for general queries.
• Add policy to report security issues.
• Add instructions on creating a PR (for ex: PR must come from forks, etc.)

Switch to Yarn broke deployments

Failing build: https://travis-ci.org/freeCodeCamp/open-api/builds/370905160
Commit: 06d028e

@Bouncey / @raisedadead I'm happy to look at this tomorrow if you don't get around to fixing it.

Use dataloader

Dataloader is a project from Facebook and allows removing pressure from the database: https://github.com/facebook/dataloader .

As per the README, it does not replace shared application level caches. It's meant to be used per request, and prevent multiple DB calls when one will do. See https://github.com/facebook/dataloader/blob/master/README.md.

Premature optimisation for now, but will come in handy when we are in need for performance improvements.

See https://github.com/withspectrum/spectrum/tree/alpha/api/loaders for a JS implementation.

Lock down IAM permissions

Related issues:

• serverless/serverless#588 (comment)
• serverless/serverless#1439
• serverless/serverless#1674

Related blog posts:

• https://serverless.com/blog/abcs-of-iam-permissions/
• https://serverless-stack.com/chapters/customize-the-serverless-iam-policy.html

Add task runner for offloading load from npm run scripts

request for an API key

Hey, i have posted in freecodecamp gitter group but haven't got a response. I am wondering if i could receive an API which i will use for my exercise in creating a leader camp board with extended functionalities. My private email is [email protected]. Would appreciate your reply.

Thanks!

As a freeCodeCamp user, I want to programatically retrieve my current challenge

As an authenticated codeCamp user, I would to programatically retrieve the current challenge.

Configure AWS Cloudwatch alarms when service limits are hit

Add AWS Cloudwatch notifications that send out SMS messages to the dev team whenever an AWS service hits a limit. For example, whenever Lambda hits 5,000 calls per second

CI : build notification webhook URL should be under variable

https://github.com/freeCodeCamp/open-api/blob/staging/serverless.yml#L37

This is going deliver payloads from all forks

Announce deployment on Gitter channels

We deploy automatically, to make this more transparent it would be great to announce deployment in 2 channels:

• For staging deployments, announce it in freecodecamp/open-api
• For production (master) deployments, announce in both freecodecamp/open-api and freecodecamp/contributors

Create integration test env that handles auth protected routes correctly