GithubHelp home page GithubHelp logo

earthlab / abc-classroom Goto Github PK

View Code? Open in Web Editor NEW
28.0 28.0 21.0 745 KB

Tools to automate github classroom and autograding workflows

Home Page: https://abc-classroom.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 99.60% Shell 0.17% Makefile 0.23%
earth-analytics education

abc-classroom's People

Contributors

betatim avatar codydunne avatar jedbrown avatar kcranston avatar nkorinek avatar pyup-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

abc-classroom's Issues

Add Files To assignment Template Repo

Below are items that we can work on now associated with populating the template repo. I'd like the abc-create-template-repo (not a good function call name but just a start) to automatically create the readme and .gitignore like jed does. let's add these as helpers in a

create-repo or a git module?? i like jed's approach but i'd like to customize it to use the config file.

  • Add Files to template repo
    ABC-Classroom approach: abc-classroom has a list of files that you want to add that you provide in the config file. It then moves those files over into the template repo. This puts the onus on the instructor to create the file which is ok.

Jed's approach: Jed's approach is interesting because

ADD GITIGNORE FILE

  • i could see a config variable that if populated, would use jed's function but rather than adding dstore and such by default, it grabs a default list of items from the config file that a user could easily add to and writes the .gitignore file.

we could populate that with

  • .DStore
  • .ipynb_checkpoints

so it would function like jeds. but it could also be added to if need be.

ADD README FILE

I think we could similarly use jed's add_readme function but again make it a bit more flexible to grab information from the config file based upon the assignment name.
Similar to the .gitignore about, we'd want it to create the readme but provide optional text that a user could create via a yaml element associated with the assignment. if they dont' create text something default would be populated like what jed has.

just in case, i think we should have the option of turning "off" the auto add readme file with a yaml element that is boolean (optional and defaults to TRUE).

The big difference with this approach vs abc classroom is abc classroom has a list of files and it looks for those files. this approach creates the files for the user and allows them to customize them (or not). This seems like a nice middle ground.

Update ABC-classroom to use pygithub rather than github3.py

pygithub is more active and has many more users. github3 hasn't been updated in over a year.

Let's consider moving to pygithub

Below is some quick code that creates a repo in the earth analytics edu org!
The one thing that i couldn't find examples for was authenticating and creating a token which does exist now via abc-init. but it seems like this should be doable via subprocess if need be altho i did see an authenticator in pygithub just no good examples of calling it.

from github import Github

# using username and password
g = Github(gh_username, password )
g = Github("token-string-here")
organization = g.get_organization("earth-analytics-edu")

# Get all repos and print them out for the earth analytics edu org
for repo in organization.get_repos():
    print(repo.name)

organization.create_repo(
        name="test-template",
        allow_rebase_merge=True,
        auto_init=False,
        description="yaas done from the API",
        has_issues=True,
        has_projects=False,
        has_wiki=False,
        private=True,
       )

This can then be used to

  1. authenticate
  2. create token with github
  3. create a new repo.

We can potentially use other parts of jed's scripts OR abc-classroom to create and initialize the local repo. there is an init function here in abc-classroom.

Auto-grading Things We'd like to check

Below is a list of things that we'd like to check for in autograding.

  • Run the entire notebook. Does all of the code in the notebook run?
    • if it all doesn't run, they lose points (boolean T/F)
  • All required python packages should be listed at the top of the notebook in the first code chunk cell.
  • Pep8 code format: one way to do this may be to write the notebook out as a .py file and run... flake8? This won't check for capital comments however like this # Capitalize comments
  • Can we test to ensure there aren't extra outputs in the notebook?
  • Can we spellcheck the notebook?

Spurious github3.py log messages

Below an issue I might file with github3.py if I cna't work out why this is happening. Tim is seeing a whole bunch of info level log messages from github3.py which shouldn't appear.


There are two places (here and here) in the library where log messages are emitted that come "out of nowhere" for a user. When using github3 as a library to automate some tasks on GitHub I was surprised to see the messages. What I don't understand is why I see these.

My library (from which I call github3.py) doesn't import logging anywhere, the code in github3 uses a package specific logger, and the default log level is warning so the info messages should not appear.

I searched for logging, logger and log to see if I could find relevant discussion but couldn't.

Version Information

  • The version of Python you're using: Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38)
  • The version of pip you used to install github3.py: 18.0
  • Versions of github3.py (1.2.0), requests (2.19.1), uritemplate (3.0.0), and dateutil not installed

Minimum Reproducible Example

I can't reproduce it with a minimal example. This is part of the mystery.

a new feature :) All student directories disappear when you select a date BEFORE any of the deadlines.

This is a clever feature. I wonder if we want to stop a user from doing this by mistake!!?? Maybe if a user selects a date runing nbauthor --date that is before now we can ask them if that is their intended task?

Or better yet what if it doesn't ever remove lessons but we can add a remove assignment if we want. that way removing things is more purposeful. I will look more into this tomorrow but i was surprised by all of my student/* dirs disappearing when i chose 2018-01-01 as a date by mistake

Close outstanding instructor PRs

When running nbdistribute check there are no open PRs from the instructor in a student's repository.

If there is an open one close it before creating a new one.

Student Notebook issues

The code below provides a solution for a student plot. At the bottom it has this

### DO NOT REMOVE LINE BELOW ###
ax1 = nb.convert_axes(plt)

I believe that we need to grab the axis object and not call plt.show() but i'm not 100% sure. I am curious why that code disappeared for the first PLOT assignment cell of homework 5 but not for the second.

# For PLOT 1&2: use the data/colorado-flood/precipitation/805325-precip-dailysum-2003-2013.csv file

# PLOT 1: a plot of precipitation from 2003 to 2013
# DO NOT USE plt.show() anywhere in this cell

# BEGIN SOLUTION

f = "data/colorado-flood/discharge/06730200-discharge-daily-1986-2013.txt"
discharge = pd.read_csv(f,
                        skiprows=23,
                        header=[1, 2],
                        sep='\t',
                        parse_dates=[2])
discharge.columns = discharge.columns.droplevel(1)
discharge = discharge.set_index(["datetime"])

monthly_max_all = discharge.resample("M").max()
monthly_max = monthly_max_all['1990':'2014']

# Create scatter plot
fig, ax = plt.subplots(figsize=(10, 8))
ax.scatter(x=monthly_max.index,
           y=monthly_max["17663_00060_00003"],
           color="purple")
ax.set(xlabel='Date', ylabel='Daily Precipitation (inches)',
       title='Homework plot 1: Precipitation-Boulder, CO\n 2013-2014')

# END SOLUTION

### DO NOT REMOVE LINE BELOW ###
ax1 = nb.convert_axes(plt)
  • Can you show me how to customize what a student sees? This cell is visually confusing to a student. i wasn't sure where to add my homework code. I am curious. is there a way to embed running python code in the metadata of a notebook or something? If not, can we add some comments to these cells and then add more spaces to the cells where we want students to add their homework? I'd suggest a blank cell or two above the cell where the answer should be and then the answer cell. it seems to remove all additional spaces.

screen shot 2018-12-15 at 5 52 47 pm

  • When i run the cell with check("time-series-homework-wk-5/q-a34ccf2.py") it doesn't know how to find that function. Where is that function and are there standard imports that belong at the top of all notebooks?

  • here is the cell for plot 2. it seems to retain that call to grab the plot.

screen shot 2018-12-15 at 5 56 36 pm

  • Can we customize the comment on cells where the solutions are hidden

screen shot 2018-12-17 at 3 35 27 pm

I found the code in nbclean but am unsure if there is the ability to parse each cell. The alternative would be to allow for one comment to persist at the top of the cell.

  • Is there any way to maintain code in a cell - for instance like below grabbing the plot element. i think we need to do that as plt.show could break things. i'm not sure however.

screen shot 2018-12-17 at 4 01 06 pm

Update: USER ERROR - I fixed this by using `###` rather than a single `#`

Circle CI failure

hey @betatim so i submitted a test partially complete assignment over the weekend her. See last commit: 3de3e428081703ba5e1b546191c8c21f0bc4c2ce

Circled Ci failed to run it LOOKS like it was hung up in the ok module

Here is my assignment
https://github.com/lwasser/autograded-course-starter/blob/master/master/week-05/time-series-homework-wk-5.ipynb

I think i may need to clean it up more but am not sure where we landed with the points and grading. Can you kindly have a look so i better understand what's going on with circle ci?

Current Errors I'm Getting with CL Calls

Hey @betatim just wanted to document some of the issues i'm having with the CLI stuff.
I totally see how this is coming together. it's just not working for me quite yet so i'm hoping we can talk through this together today!!

  • ## Initialize Connection Returns: 422 Validation Failed
    nbinit
    I get failed validation for some reason.
MacBook-Pro-4:autograded-course-starter leah-su$ nbinit
GitHub username: lwasser
Password for lwasser: 
Traceback (most recent call last):
  File "/Users/leah-su/anaconda3/bin/nbinit", line 11, in <module>
    sys.exit(init())
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/grading/__main__.py", line 56, in init
    two_factor_callback=two_factor)
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/github3/api.py", line 26, in deprecation_wrapper
    return func(*args, **kwargs)
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/github3/api.py", line 59, in authorize
    client_secret)
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/github3/github.py", line 462, in authorize
    json = self._json(self._post(url, data=data), 201)
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/github3/models.py", line 156, in _json
    raise exceptions.error_for(response)
github3.exceptions.UnprocessableEntity: 422 Validation Failed
  • ## Connect to Course

nbdistribute --template
note: this course i think exists but if it exists can it fail gracefully please?

MacBook-Pro-4:autograded-course-starter leah-su$ nbdistribute --template
Using /Users/leah-su/Documents/github/2-autograding/autograded-course-starter/student to create the student template.
Loading configuration from config.yml
Creating template repository.
Traceback (most recent call last):
  File "/Users/leah-su/anaconda3/bin/nbdistribute", line 11, in <module>
    sys.exit(distribute())
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/grading/__main__.py", line 131, in distribute
    config['github']['token'])
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 747, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'github'

Setup student dir

nbauthor --date 2018-01-01
this works well so far! haven't tested working on a notebook yet.

Distribute assignment

This is still failing and i'm not sure why.

MacBook-Pro-4:autograded-course-starter leah-su$ nbdistribute 
Using /Users/leah-su/Documents/github/2-autograding/autograded-course-starter/student to create the student template.
Loading configuration from config.yml
Fetching work for betatim...
Traceback (most recent call last):
  File "/Users/leah-su/anaconda3/bin/nbdistribute", line 11, in <module>
    sys.exit(distribute())
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/grading/__main__.py", line 144, in distribute
    token=config['github']['token'])
  File "/Users/leah-su/anaconda3/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 747, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'github'

Creating student assignments

I am missing something here. i just made new assignments but i only see one week1 folder in my student dir.

MacBook-Pro-4:autograded-course-starter leah-su$ nbauthor --date 2018-12-01
Processing /Users/leah-su/Documents/github/2-autograding/autograded-course-starter/master/week1/homework.ipynb
Processing /Users/leah-su/Documents/github/2-autograding/autograded-course-starter/master/week-03/time-series-homework.ipynb
Processing /Users/leah-su/Documents/github/2-autograding/autograded-course-starter/master/week-04/time-series-homework-2.ipynb
Inspect `/Users/leah-su/Documents/github/2-autograding/autograded-course-starter/student/` to check it looks as you expect.
MacBook-Pro-4:autograded-course-starter leah-su$ 

Student PR's - nbdistribute & nbauthor should be able to focus on distributing one assignment at a time or one file?

I pretended that i was a student. i downloaded week 5. but i don't think i made any purposeful changes. Regardless, when i just pushed week 6, i ended up with a potential conflict where git asked me to change my edits to week 5.

screen shot 2018-12-18 at 4 03 54 pm

Because this is inherit to github, i think we should manage nbauthor and nbdistribute so that you can push a single assignment at a time. And once that is pushed, you can't update it unless there is some clever way to update something a student is already working on without a conflict. I currently am not sure how to accept the new assignment without stashing or deleting /checking out my work to week 5.

Support environment.yml

If there is a environment.yml teach CircleCI to setup the environment and run stuff in it.

late grade policy implementation

Implement a way to assign grades using a late grade policy implemented via the config. This could be built as follows:

  1. implement the policy via config.yml: 24hours late, 10% dock, 48 hours late, 20% dock
  2. Add a yaml exception for students who need an extension like
- lwasser
     due: 2018-10-15

NOTE: I could see an issue where a student submits something (maybe even something they don't want graded) after the deadline. Do need a way to flag whether to grade a student using the commits that are AFTER the deadline or not?


via email we discussed

Can we modify the deadline for a student if need be via a yaml file?
Could we apply penalties to late assignments โ€“ like

-   10% off after if submitted within 0- 24 hrs of deadline
- 20% 24-48, etc?

Penalties like this should be straightforward.

For per student things I'd go with handling it via command-line
arguments. So you'd run `nbgrade --student betatim --penalty 0
-assignment week1` to grade betatim's work for the week1 assignment
without any penalty. My impression is that handling individua cases
like this will be the exception and that each case will require some
different combo of penalty, assignment, student etc.

This is how I see the trade-off: Being able to specify things on the
command-line gives you full flexibility, but also means you have to
type it out each time. Configuring via the config file means easier to
repeat but more limited because we have to design what kinds of
modifications are possible up front. What do you think?

I could imagine this part happening at the command line and would involve a yaml file with individual student deadlines (only for exceptions, otherwise all would be the same) I might have 1-3 exceptions for any given assignment on average. Often less.

Meta issue on creating documentation

We will create a clean, new, simple example repository (the course-starter repo) that people can use to start a new course. It will contain:

  • two or three simple notebooks/assignments that show off how to create private/public tests, different points per assignment
  • explanation of what can and can't be in different cells (depending on what tag they have)
  • markdown explanation of how to create tests, comments, stuff
  • annotated but with gaps config.yml that will be filled out by readers when they follow the quick start

Create a way to have a "dummy-student" directory locally that you can "distribute" to and then "grade". It would allow the instructor to pretend to be a student, attempt the assignment, see what the assigned grade is.


There will be a quick start guide (hackmd to get going) to walk people through the process of creating a new class based on the quick start repository.

  • will walk reader through the process of starting from the course-starter repo and end with them having a fully working course
  • instructions for simulating the grading/rendering of the student notebook

This quick start guide will be part of the documentation (RTDs) that is in this repository.


There will be additional documentation about the components in this repository as well.

Two steps per notebook in CricleCI

Create two steps per notebook in circle CI config one for papermill and one for nbconvert. Otherwise we won't get an artifact for failed notebooks as circle stops executing a step once a command fails.

Default editor for `nbdistribute`

When asking the user for a commit message we check the EDITOR and VISUAL environment variables to try and figure out what (command-line?) editor the user prefers. If we can't find anything vi is used. There is no particularly good reason to use vi as fallback.

There are two things to do:

  • are there more variables or places we can check to find out what editor the user would like us to use?
  • decide what the default editor should be

Handling Student Rosters - issues to address

There are a few ways to handle rosters in our current grading workflow. Here are some of hte issues

  1. If you download the roster from github classroom, it has an identification column. However nbgrader needs an id column. you can rename the identification column to ID however several issues arise when you do that.
  • Could nbgrader be updated to accept both id and identification? OR should we add a roster cleanup step that adds all required columns to that github downloaded roster? it could just be a little helper function
  1. If you create your own roster, there is no template for what that should look like.

@jlpalomino can you kindly add some clarification around the pieces needed in a roster? i think we just need to streamline our scripts to ensure all of the pieces are there. Will a first and last name ever be available in the github classroom roster? ie if you add me as a student will my first name and last name be there?

Get started with classroom!

  • go through contributing.rst file, and setup dev envt. Does it work?
  • see if you can get abc-init to work. If it does work for you, document how it works in the hackmd document (just add a new section)
    then update the docstrings associated with the functions that it uses

Branches build up in student directories

Currently each time i merge a PR, the branches in that repo build up (please see image below). I think we need to clean up the branch created each time by the PR. otherwise it could get difficult to manage. i am not sure what the best approach to this would be. One option could be to clean up the previous branch each time a new pr is created? so then there is always just one branch that is there unless the student deletes it when they merge the PR. Or could the PR come from the template repo rather than from a branch within the student's repo? that would be the cleanest option. i'm just unclear as to what is happening with this PR process.
screen shot 2018-12-18 at 4 10 35 pm

Only add specified assignments/files

Right now abc-author adds all assignments to the student directory. Similarly abc-distribute will generate a PR that modifies everything.

Instead we want to have abc-author assignment1 which only adds assignment1 related files to the already existing student/ directory. abc-author -d assignment1 would remove the assignment from the student/ directory again.

abc-distribute (w/o arguments) would generate a PR based on everything in student/ (copy over whole student/ directory). abc-distribute assignment1 would only create a PR with changes for that assignment (copy over only the assignment specific sub-directory from student/).

One thing that will be tricky to figure out is the .circleci/config.yaml which we will need to edit as we do this.

GitHub token creation/nbinit

When running nbinit check localconfig.yaml for existing token and reuse it.

If there is no token then use something unique in the note for creating the token as we can't create two tokens with the same name.

Scrub token from terminal output

Scrub all print outs/tracebacks so that they do not contain access tokens.

This probably means we need to wrap all calls in try: except: blocks where we customise the printouts. Currently some of the git clone calls will reveal the users personal access token in the log. They might then copy&paste that to a help forum or similar.

Refactoring the gh-classroom / nbgrader approach

@jlpalomino and I spent some time working through the workflow for github classroom.

Essentially, we'd like to take the scripts here and turn them into components for abc classroom that can be run at the CLI.

The high level workflow is outlined below. From this we will have several issues and questions to address before we can do further work! The one SIGNIFICANT chance from those scripts will be using a config.yml file like we use here in abc-classroom. More on that will be specified below.

Proposed Config File

Below we begin to proposed the structure of a config file. This config file would map closely to abc-classroom's existing functionality however, it would have some additional components as well.

config.yml assumptions:

- org_name:
- course_name:
- assignments:
  # List all assignments as they are created here. 
  - assignment-name-one
      - deadline:
      - ???

# Additional files that will always be added to the student assignment repository
# In this example the file called `student_README.md` will be copied to the
# student's repository as `README.md`
- assignment_repo_files:
  - README.md: student_README.md
  - environment.yml: environment.yml
  - .gitignore: .gitignore

The above is a start to the config file use. Note that the original config also included student GH usernames. there will need to be an independent discussion surrounding how the student roster is populated.

2. Create a Classroom

2a. Create a nbgrader class repo / directory

  • This is the directory where the nbgrader DB and such is housed. This step is soley nbgrader functionality but we might consider wrappers around this step and step 2b below

2b. Create a GitHub Classroom Class.

Questions Surrounding Creating a Class

  • Can we use the API to create a new classroom or does this step need to be manual? I suspect the answer to this is "no" but i asked anyway!! it's ok to have a manual step in our process. it's a small step and only performed once.
  • Meta classroom question - could steps 1a and 1b be merged via a wrapper with a config file IF you can use the API for GH classroom?? see this question here

UPDATE: I DON'T BELIEVE THAT WE CAN AUTOMATE CREATING A GITHUB CLASS. This is ok as it's a one time step.

3. Create Assignment (Nbgrader)

Here the instructor needs to create an assignment that will be distributed to students. It involves several steps.

  • Instructor builds notebook with tests
  • Instructor validates notebook with nbgrader
  • instructor assigns the assignment which creates a copy of it in the release folder.

These tasks can all be implemented using nbgrader. It would be good for us to document this in our docs just to have it here. We have some of this documented in our hackmd document. Earth Lab can develop these docs.

4. Create / Distribute Assignment (Github)

The steps here are contingent upon another question - I think a wrapper is created that both creates the assignment via nbgrader and produces the github repo all in one step. But you could also do these things independently to make this more versatile. we have all of the information needed at this step to implement both.

This should be possible. as a baby step we have a abc-create-assignment

Last Steps

I suggest that for now we refactor our scripts to implement the steps below using the config file which will remove many of the arguments that are used over and over again. i think each function will just need to call the assignment name and the rest should work... More soon on the roster issue which is a whole different can of worms.

The last 4 steps are:

  1. collect assignments
  2. Grade assignments
  3. Generate Feedback
    8 Push feedback to students

Different points per cell

Tag a cell with a name bob1 and a tag for number of points p=15. When creating the student/teacher version of the notebook we create a JSON file for the teacher in which we look up how much a question named bob1 is worth.

nbdistribute -- fails when student repo doesn't exist

previous there was an issue with nbdistribute that i thought i'd fixed with a pr -- but now i'm getting another error here - it's returning the appropriate message but still trying to clone their repo.

Fetching work for jlpalomino...
Student jlpalomino does not have a repository for this course, maybe they have not accepted the invitation yet? Skipping them for now.
Traceback (most recent call last):
  File "/Users/leah-su/anaconda3/envs/earth-analytics-python/bin/nbdistribute", line 11, in <module>
    load_entry_point('grading', 'console_scripts', 'nbdistribute')()
  File "/Users/leah-su/Documents/github/2-autograding/grading-workflow-experiments/grading/__main__.py", line 253, in distribute
    token=config['github']['token'])
  File "/Users/leah-su/Documents/github/2-autograding/grading-workflow-experiments/grading/github.py", line 68, in fetch_student
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/Users/leah-su/anaconda3/envs/earth-analytics-python/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://[email protected]/earth-analytics-edu/ea-spring-2019-jlpalomino.git']' returned non-zero exit status 128.

Document abc-init with a .rst file in the docs folder.

Create an rst file that overviews:

  • What does this command line call do?

  • what does it require
    -- git needs to be setup locally
    -- will ask for username and password

  • Output:

    • token in your home directory

You can use the github.rst file to flesh out these docs!

Private & public tests, don't seem to run and are visible to students in a confusing way

  • I believe that private tests are somehow rendering as a cell still. It's a little confusing to have this visible to the student in the notebook. We should discuss how private tests run and are shown to the students.

I think i'd prefer that we tag cells to be graded with something unique that then gets mapped when grading locally as one option? perhaps there are other options too!

screen shot 2018-12-17 at 8 08 06 pm

  • Also - i'm not sure how to access my solution when grading. ie how to you build private tests. Right now it just cleans the solution from the notebook but where does that solution go and how do we use it to grade a notebook?

  • Finally private & public tests do not seem to run in my student notebook. yet they work in my master notebook. Oddly i don't even get an error i just get nothing when i run that cell.

As an example i just completed and submitted a homework assignment. And it just doesn't seem to run any tests locally public or private. please see here:

Create template repo -- should use config file

The create template repo should do the following

  • it should use the config file to access a list of files that belong in all template repos.
    • Those files need to be created or stored somewhere?? might need to talk this through a bit more. abc-classroom may create a template readme i can't remember
  • It should use the config to determine where to store the template repo
  • it should use the config file to find the homework name that it is going to use to name the repo and then it should add -template to that name. abc classroom does do this.

New name

This package needs a better name, especially as it grows up and starts being useful instead of a collection of experiments.

My current proposal: "teaching and learning with notebooks" shortened to tln (or tlnb). Commands would become tln-distribute and tln-author instead of nbdistribute and nbauthor.

It is descriptive, can be shortened to a TLA (three letter acronym) and for vanity contains the initials of Tim and Leah ๐Ÿ˜‚

Start adding tests

We don't have any tests yet. Step one is figuring out how to do this nicely, step two is to write some tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.