GithubHelp home page GithubHelp logo

tomcatling / black-nb Goto Github PK

View Code? Open in Web Editor NEW
50.0 3.0 11.0 154 KB

Runs black on code cells in a Jupyter notebook

Home Page: https://pypi.org/project/black-nb/

License: MIT License

Python 87.92% Jupyter Notebook 12.08%

black-nb's Introduction

The uncompromising code formatter, for Jupyter notebooks

Build Status Code Style Code Style

black-nb applies black to Jupyter notebooks.

Much of the code is taken from the original black project and the behaviour is intentionally similar.

Installation

pip install black-nb

Usage

To apply black to all code cells in notebooks under the current directory:

black-nb .

To clear cell outputs in addition to reformatting:

black-nb --clear-output .

To check if notebooks pass black and additionally have no output (files will be unchanged):

black-nb --clear-output --check .

To reformat all *.ipynb files below ./, excluding paths matching */outputs/* or */.ipynb_checkpoints/*:

black-nb --exclude '/(outputs|\.ipynb_checkpoints)/' .

Command Line Options

black-nb doesn't provide many options. You can list them by running black-nb --help:

Usage: black-nb [OPTIONS] [SRC]...

  The uncompromising code formatter, for Jupyter notebooks.

Options:
  -l, --line-length INTEGER  How many characters per line to allow.  [default:
                             88]
  --check                    Don't write the files back, just return the
                             status.  Return code 0 means nothing would
                             change.  Return code 1 means some files would be
                             reformatted.  Return code 123 means there was an
                             internal error.
  --include TEXT             A regular expression that matches files and
                             directories that should be included on recursive
                             searches.  An empty value means all files are
                             included regardless of the name.  Use forward
                             slashes for directories on all platforms
                             (Windows, too).  Exclusions are calculated first,
                             inclusions later.  [default: \.ipynb$]
  --exclude TEXT             A regular expression that matches files and
                             directories that should be excluded on recursive
                             searches. An empty value means no paths are
                             excluded. Use forward slashes for directories on
                             all platforms (Windows, too). Exclusions are
                             calculated first, inclusions later.  [default: /(
                             \.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|_build
                             |buck-out|build|dist|\.ipynb_checkpoints)/]
  --extend-exclude  TEXT     Like --exclude, but adds additional files and
                             directories on top of the excluded ones.
                             (Useful if you simply want to add to the default)
  --force-exclude   TEXT     Like --exclude, but files and directories matching
                             this regex will be excluded even when they are
                             passed explicitly as arguments.
  --stdin-filename  TEXT     The name of the file when passing it through stdin.
                             Useful to make sure Black will respect --force-exclude
                             option on some editors that rely on using stdin.
  -q, --quiet                Don't emit non-error messages to stderr. Errors
                             are still emitted, silence those with
                             2>/dev/null.
  -v, --verbose              Also emit messages to stderr about files that
                             were not changed or were ignored due to
                             --exclude=.
  --clear-output             Clear cell output as part of formatting.
  --config FILE              Read configuration from PATH.
  -h, --help                 Show this message and exit.

Copyright

Copyright ยฉ 2019 Tom Catling, Liam Coatman.

black-nb is distributed under the terms of the MIT licence.

black-nb's People

Contributors

harshad16 avatar jtmiclat avatar liamcoatman avatar sentewolf avatar tlinhart avatar tomcatling avatar whitead avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

black-nb's Issues

Removing isort

I couldn't get isort to agree with itself locally and on CI, so i removed it from the nox build validation. I don't think it's adding much, so i recommend we remove the dependency from pyproject.toml and move on.

use as jupyter presave hook

is there any guidance for using black-nb as a presave hook in jupyterlab?

it would be great if that can be automated by running a command or script

thank you

Incompatability with Black 21.4b0

The following error occurs when I run Run black-nb --clear-output --check . when version 21.4b0 of black is used:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.9/x64/bin/black-nb", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/black_nb/cli.py", line 142, in cli
    target_versions=black.PY36_VERSIONS,
AttributeError: module 'black' has no attribute 'PY36_VERSIONS'

If I fix black to version 20.8b1 this issue does not occur.

Note that new installs of black-nb will automatically use black 21.4b0 and therefore result in errors in new projects.

PyPI release

Dear team,

I have observed that there are two new releases for this project, but the last PyPI release still points to v0.4. Is it possible to upload the releases there too?

Thanks!

License Update

We should include the MIT license originally shipped with black, with modifications listed under the copyright as described here

Incompatible with current release of black

Traceback (most recent call last):
  File "./env/bin/black-nb", line 8, in <module>
    sys.exit(cli())
  File "./env/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "./env/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "./env/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "./env/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "./env/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "./env/lib/python3.9/site-packages/black_nb/cli.py", line 195, in cli
    sources = black.get_sources(
TypeError: get_sources() missing required keyword-only argument 'root'

black-nb 0.7
black 23.11.0

Why pinned requirements?

The install_requires passed to setup in setup.py uses specific versions (e.g. black==18.9b0). Is there a salient reason for this? It would help us downstream users if these could be minimum requirements (e.g. black>=18.9b0) so that we aren't forced to downgrade in order to use black-nb.

black-nb does not preserve encoding

black-nb can sometimes destroy a IPYNB file when this file contains UTF8 accents.

I work on a Ipynb with UTF8 accents. This corresponds to the before.ipynb.txt Notebook:

before.ipynb.txt

Then I execute black-nb on it:

(C:\Users\myname\AppData\Local\Continuum\anaconda3-2019-03) Z:\myworkingdirectory>black-nb Untitled.ipynb
    1 cell left unchanged.
All done!
1 file left unchanged.

No cell was apparently changed. However, the new notebook:

after.ipynb.txt

cannot be opened:

image

This is because the cell contains a "รฉ" which requires UTF8 encoding.

image

But the new file has ANSI encoding.

image

black-nb==0.3.0 is incompatible with black==20.8b1 (latest)

Title should be pretty self-descriptive. I'm using the Anaconda installation of Python 3.7.6 in a Windows machine. If I run:

pip install black==20.8b1 black-nb==0.3.0

and then run black-nb . in a directory (can be empty), I get this error:

Traceback (most recent call last):
  File "c:\users\danmill\miniconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\danmill\miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\users\danmill\Miniconda3\Scripts\black-nb.exe\__main__.py", line 7, in <module>
  File "c:\users\danmill\miniconda3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\danmill\miniconda3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\danmill\miniconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\danmill\miniconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\danmill\miniconda3\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\danmill\miniconda3\lib\site-packages\black_nb\cli.py", line 171, in cli
    black.gen_python_files_in_dir(
AttributeError: module 'black' has no attribute 'gen_python_files_in_dir'

black[jupyter]

I believe black has separated its jupyter part into black[jupyter], so maybe the dependency for this project should be black[jupyter] instead?

Compatibility issue with latest Black release (22.1.0)

When using v0.6 with the latest Black version (22.1.0), I get this error:

$ black-nb --clear-output .
Traceback (most recent call last):
  File "/home/user/tmp/project/venv/bin/black-nb", line 8, in <module>
    sys.exit(cli())
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user/tmp/project/venv/lib/python3.8/site-packages/black_nb/cli.py", line 194, in cli
    sources = black.get_sources(
  File "src/black/__init__.py", line 626, in get_sources
TypeError: 'NoneType' object is not subscriptable

Broken compat with Black

psf/black@6310a40 changed the internal get_sources() black function that this package uses.

I know black supports formatting notebooks now, but it's really convenient to have formatting and checking for execution counts / output with the same tool ๐Ÿ˜ฌ .

Tests

We should test:

  • notebook discovery with combinations of --include/--exclude
  • clearing output
  • preservation of final ';'
  • expected behaviour with with line/cell magic

Testing exclude/include

I don't think test_include_exclude, test_empty_include, test_empty_exclude are testing the CLI.

Indented comments with trailing question mark trigger Internal Error

This has been fun to chase down!

I thought the popular INTERNAL ERROR: Black produced different code on the second pass of the formatter. had been fixed in last black release, but after ugprading, I realized the one I was hitting was due to black-nb alone.

It has to do with the hide_magic / reveal_magic functions. An indented comment that ends in a question mark will trigger the contains_magic tests and then make the 2nd pass validation tests fail.

This notebook reproduces the issue:

https://gist.github.com/jaimergp/46f74cffcecc4e38ce4471e32cf5f374

I had to modify a function to get more insights on the actual error:

def assert_stable(dst: str, mode: black.FileMode = black.FileMode(),) -> None:
    new_dst = format_str(dst, mode=mode)
    if dst != new_dst:
        print(black.diff(dst, new_dst, "1st pass", "2nd pass"))  # I added this <-----------
        raise AssertionError(
            "INTERNAL ERROR: Black produced different code on the second pass "
            "of the formatter. Check diff above."
        ) from None

My workaround so far is to either remove the trailing question marks or adding something else after it, but it should be addressed here as a bug, ideally.

Thanks!

Readme Update

Let's git rid of the ๐Ÿ““ emoji and include a PePy badge: Downloads . It would also be nice to have a logo and use the title as a tagline, as in the black project.

Adapt for use in pre-commit

We use black as a pre-commit hook, and it would be nice to also be able to use black-nb as a pre-commit hook. Since this project mirrors black's CLI, all that should be needed is to add a .pre-commit-hooks.yaml file to this repository.

More details can be seen in pre-commit's docs, and the black pre-commit hook PR.

Deal with `?` IPython syntax

A line like np.sum? is valid in a Jupyter notebook, and prints the documentation for np.sum. Currently black-nb will fail to format a cell with lines like this. We need to fix this.

Add a recursive option

[ENHANCEMENT]
Add a -r option to format all notebooks in the given path and its subdirectories.

The question mark in comment results in extra spaces on the second pass.

Copied from the thread
Repro steps:

  1. Create the minimal jupyter notebook, which will contain one cell with the function ands a comment, containing question mark;
def example_fn(param):
    """
    param: str    what to print?
    """
    print(param)
  1. Close the file and run
black-nb  Repro.ipynb

The error will be shown:
error: cannot format /path/to/nb/Repro.ipynb: INTERNAL ERROR: Black produced code that is not equivalent to the source on pass 1. Please report a bug on https://github.com/psf/black/issues. This diff might be helpful: ${TEMP}/blk_y4fjze4u.log

The diff will show that at the second pass extra whitespaces were added:

--- src
+++ dst
@@ -23,11 +23,11 @@
       body=
         Expr(
           value=
             Constant(
               value=
-                '###MAGIC###    param: str    what to print?',  # str
+                '###MAGIC###        param: str    what to print?',  # str
             )  # /Constant
         )  # /Expr
         Expr(
           value=
             Call(

If the question mark will be removed, the notebook will pass

Example notebook:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c38d2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def example_fn(param):\n",
    "    \"\"\"\n",
    "    param: str    what to print?\n",
    "    \"\"\"\n",
    "    print(param)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
  • contains_magic makes no consideration for whether or not it's looking at a string. This explains why the issue doesn't happen if you remove the question mark. I'm not familiar with jupyter, so I'm not sure if this is intended behavior or not.

  • As a consequence of 1, black-nb will add a magic comment block to the beginning of the line after 0 spaces, which is not appropriate for a docstring. Black will correctly try to indent it, adding 4 spaces. When the magic is removed, those 4 spaces remain, which accounts for the diff you are seeing.

Docstrings

docstrings need to be reviewed/updated.

Adding pylint

I removed pylint from build validation to get things working, but we should add it back in now that we're in a stable state.

Add a short flag for `--clear-output`

black-nb fills what was a big hole in my toolbox: thanks @tomcatling and @liamcoatman! Most of the time when I use it, I want to both format code cells and clear the outputs, so I use the --clear-output flag. This is a bit long to type a lot though. What do you thing about adding a synonymous short flag such as -c? (although perhaps -c should be synonymous with --check, and an alternative used for --clear-output).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.