chilipp / docrep

A Python Module for intelligent reuse of docstrings

docrep - A Python Module for intelligent reuse of docstrings

docs
tests
package

What's this?

Welcome to the documentation repetition module docrep! This module targets developers that develop complex and nested Python APIs and helps them to create a well-documented piece of software.

See the documentation on readthedocs for a small tutorial.

Installation

Installation simply goes via pip:

$ pip install docrep

or via conda:

$ conda install -c conda-forge docrep

or from the source on github via:

$ python setup.py install

Disclaimer

Copyright 2021 Philipp S. Sommer, Helmholtz-Zentrum Geesthacht

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

docrep's Issues

Sections with trailing spaces are not identified

If you have a trailing space behind the section (e.g. "Parameters " instead of "Parameters"), docrep will not recognize the section. This should be fixed.

Example:

import docrep
d = docrep.DocstringProcessor()

@d.get_sections(base='test')
def test(i):
    '''Some test function
    
    Parameters 
    ----------
    i: int   
        some param
    '''
    pass

print(d.params["test.parameters"])  # gives ''

Unable to remove multiple params using docrep.delete_params()

Hello, thanks for this package - I'm finding it really helpful.

I'm not sure if this is a feature request, a bug, or unclear documentation, but I'm having trouble removing more than one parameter from the DocstringProcessor.

From the documentation:

The new docstring without the selected parts will be accessible as base_key + '.no_' + '|'.join(params), e.g. 'original_key.no_param1|param2'.

Here's my workflow so far.

import docrep
docstrings = docrep.DocstringProcessor()

@docstrings.get_sections(
    base='base_values',
    sections=docstrings.param_like_sections,
)
def my_base_func():
    ...

docstrings.delete_params('base_values.parameters', 'method')
# Make a copy because we don't want to delete 'base_period' from the main DocstringProcesser.
core_params = copy(docstrings)
core_params.delete_params('base_values.parameters', 'base_period')

@core_params.dedent
def new_func():
    """My new func.

    Parameters
    ------------
    %(base_values.parameters.no_base_period|method)s
    """
    ...

So my issue is that I'm following the docs to try and remove both parameters "base_period" and "method" but I've had no luck in doing that using the workflow described above. If it's an issue in my implementation rather than a bug, then a clearer example in the docs for dropping multiple parameters would be much appreciated.

Additional point:

• Having to first do .delete_params() and then add extra syntax in the docstring seems like duplication of work. What's stopping configuring the API so the user doesn't have to even call .delete_params() on the docstring processor, instead just specifying which parameters they want to keep directly in the docstring such as "%(base_values.parameters.param1|param2)s".

Thanks

python 2 compatibility for class inheritance

We are using docrep in https://github.com/dask/dask-jobqueue project to avoid docstring duplication. However, we've got a problem with python 2, see dask/dask-jobqueue#35.

I see in #2 that you may have solutions for this?

Deprecated regex

I am reporting this deprecation warning:

docrep.py:24: DeprecationWarning: Flags not at the start of the expression '(?<!%)(%%)*%(?!%)   ' (truncated)
    \((?P<key>(?s).*?)\)# key enclosed in brackets""", re.VERBOSE)

Line 24 is:

substitution_pattern = re.compile(
    r"""(?<!%)(%%)*%(?!%)   # uneven number of %
        \((?P<key>(?s).*?)\)# key enclosed in brackets""", re.VERBOSE)

issue with GPL license

Hello and thanks for providing this excellent and useful package! We use it in xgcm: https://github.com/xgcm/xgcm

Some users of xgcm conducted a license audit and found that docrep (a xgcm dependency) is licensed under GPL (xgcm/xgcm#308). These users can't use GPL-licensed code. Many commercial users can't use GPL-licensed code at all, which is why most of the scientific python stack avoids it.

Was GPL with its "copyleft" attributes a deliberate choice for docrep? If not, I was wondering if you would be open to changing your license to a "permissive" license like MIT or Apache.

Thanks for considering this question.

Add custom sections

Thanks for the great work! I'm trying to use docrep in my project but cannot, for the life of me, find a way to add a custom section to the text_sections attr.

I have tried many different things, but none of them work. I always get a KeyError when I add the section called Rules to the docstring. How do I add a new custom section to text_sections and also keep the original ones (ex., Examples, Notes)?

This is my code:

import docrep
docstrings = docrep.DocstringProcessor()

@docstrings.get_sections(
    base='do_something_great',
    sections=[
        "Parameters",
        "Rules",
    ],
)
def do_something_great(x, y):
    """Do something

    More description.

    Parameters
    ----------
    x :str
        is x
    y :str
        is y

    Rules 
    -----
    *characteristic*: Important rule for this characteristic. It is such
        an important characteristic.

    Returns
    -------
    instances_of_characteristic: int
        Number of times the *characteristic* was observed
    """
    pass


@docstrings.with_indent(4)
def do_more(*args, **kwargs):
    """Do even more stuff

    Even more description

    Parameters
    ----------
    %(do_something_great.parameters)s

    Rules
    -----
    %(do_something_great.rules)s

    Returns
    -------
    instances_of_characteristics : int
        Number of times many important characteristics are found.
    """
    pass

help(do_more)

How can I add the Rules section to the text_sections so that I can use it in do_more?

Reuse docstring from Class doc

I tried a bit more and again I'm stuck.
I usually document the __init__ method in the class docstring rather than in __init__ itself.

So I tried

import docrep
d = docrep.DocstringProcessor()


@d.get_sectionsf("A")
@d.dedent
class A():
    """
    Docstring from A

    Parameters
    ----------
    a : int
        Some value passed to _func.
    """
    def __init__(self, a):
        self._func(a)

    @d.dedent
    def _func(self, a):
        """
        Doc from _func

        Parameters
        ----------
        %(A.parameters)s
        """
        print(d.params)

        return

This gives:

SyntaxWarning: 'A.parameters' is not a valid key!
  @d.dedent

However, d.params has:

'A.parameters': 'a : int\n    Some value passed to _func.', 'A.other_parameters': ''

What works instead is just to document the __init__ method.
But old habits and stuff...

Is there a way around this, or am I again just not seeing how it works?
I guess the f in get_sectionsf is for function, but I don't see the difference to a class, which also has __doc__?
Best,
Thorben

Works on classes in different files?

I've looked at the answer you gave in issues #1, and your code snippet indeed works...it even works on deeper inheritance (e.g. methods in class C can get doc strings from class A).

BUT, when I try to do it on my project, where, A, B and C are each in different files it does not work. I have confirmed this on the simple test cases in issue #1. If you put A and B in different files (called docrep_A and docrep_B), then when you do from docrep_B import B and b = B() on the command line it says: SyntaxWarning: 'A.my_method.parameters' is not a valid key! @docstrings.dedent

Do you have any thoughts on this?

Issue when using docrep with instance methods

When using docrep to reuse an instance method's docstring, I found that the order of the methods matter. The method from which the docstring is extracted has to be defined before the method that uses its docstring.

I guess it makes sense, but it took me by surprise because when dealing with classes I usually don't think of methods in a linear way (as in procedural programming). I usually think of each method in the API as independent from each other in terms of the user calling it.

Is this intended or just something for which there is no workaround? I would prefer not to move the methods around in the class, but it wouldn't be the end of the world. I usually write the more general method that uses other private methods and just after it I add the methods that are used by the first method.

If it is intended or unavoidable then it should probably be documented.

Some code to illustrate this:

This will work:

from docrep import DocstringProcessor
DocstringProcessor.text_sections = ["Rules", "Notes", "Warnings"]
docstrings = DocstringProcessor()

class W():
    @docstrings.get_sections(base="silent_u", sections=["Parameters", "Rules"])
    @docstrings.dedent
    def _check_silent_u(self):
        """
        Count the number of silent letters *u*.

        Parameters
        ----------
        x : int
            some param

        Rules 
        -----
        *u*: when preceded by a *g* or *q* and
            followed by an *i* or *e*. These follow the pattern *gui*, *gue*,
            *que*, *qui*. 

        Returns
        -------
        silent_u_count: int
            Number of silent letters *u* in the word
        """
        pass

    @docstrings.dedent(4)
    def _check_silent_letters(self):
        """
        Count the number of silent letters.

        Silent letters are graphemes that do not have a phonemic representation.
        In Spanish, silent letters are *h*s and *u*s that meet certain criteria. 

        %(silent_u.parameters)s

        Rules
        -----
        %(silent_u.rules)s

        Returns
        -------
        silent_letter_count : int
            Number of silent letters.
        """
        pass

print(W._check_silent_letters.__doc__)

But this will not work:

from docrep import DocstringProcessor
DocstringProcessor.text_sections = ["Rules", "Notes", "Warnings"]
docstrings = DocstringProcessor()

class W():
    @docstrings.dedent(4)
    def _check_silent_letters(self):
        """
        Count the number of silent letters.

        Silent letters are graphemes that do not have a phonemic representation.
        In Spanish, silent letters are *h*s and *u*s that meet certain criteria. 

        %(silent_u.parameters)s

        Rules
        -----
        %(silent_u.rules)s

        Returns
        -------
        silent_letter_count : int
            Number of silent letters.
        """
        pass

    @docstrings.get_sections(base="silent_u", sections=["Parameters", "Rules"])
    @docstrings.dedent
    def _check_silent_u(self):
        """
        Count the number of silent letters *u*.

        Parameters
        ----------
        x : int
            some param

        Rules 
        -----
        *u*: when preceded by a *g* or *q* and
            followed by an *i* or *e*. These follow the pattern *gui*, *gue*,
            *que*, *qui*. 

        Returns
        -------
        silent_u_count: int
            Number of silent letters *u* in the word
        """
        pass

print(W._check_silent_letters.__doc__)

Keep/delete_params features and optional docstring typehints

Hi @Chilipp ,
thanks for making docrep, I find it really useful!
I have 3 small requests, all somewhat related to keep_params/ delete_params:

1. would it be possible to include alias argument (optional) to access the modified docstring? With some check if it's specified so that it doesn't overwrite existing keys.

d.keep_params("foo.parameters", "bar", "baz", "quux", alias="name")
# foo.parameters.name can be used instead of foo.parameters.bar|baz|quuux
d.keep_params("foo.parameters", "bar", "baz", "quux", alias=None)
# same as it is right now: foo.parameters.bar|baz|quuux
d.keep_params("foo.parameters", "bar", "baz", alias="name")
# would raise an error, alias "name" is already present

2. can keep_params and deleta_params be used as decorators? I'd like to keep things in place rather than invoking d.keep_params somewhere below the functions

@d.keep_parameters("foo.parameters", "bar")
@d.get_sections(base="foo", sections=["Parameters"])
def foo(bar, baz):
   """
   ...
   """
   pass

3. can typehints in the docstrings be made optional? I always use https://pypi.org/project/sphinx-autodoc-typehints/, so including them in the docstring is not necessary. However, not including them doesn't currently work with {keep,delete}_params.

def foo(bar: int, baz: str) -> None:
   """
   Quux.

   Parameters
   ----------
   bar
       The bar.
   baz
       The baz.
   """
   pass

I think 1. and 2. are fairly easy to adjust and I can make a PR for them.

As for 3., I somewhat naively though it might be as simple as modifying the relevant part of the regex from : to :? (or similar), however that's not the case, since some tests were failing (esp. the complex ones, where there was a mixture of keeping the annotations in the docstrings and omitting them [which is really discouraged]).
Maybe you have an idea how to best approach this?

blank lines between parameters

Does docrep require the parameters in the docstring to be written without a blank line separating them? It seems like a blank line is taken as the end of the parameter section.

Would it be possible to allow blank lines such that the raw docs look a little neater?

Module or __init__ docstrings reference

Hello, top library... only one i found that solve something a need.

I have one use case i don't know how to solve:

I have python library - I want to document library with referencing modules.

I found how to use docrep in classes and functions, is there a way how to use it in module docstrings (I dont know how to use decorator on module...)

Example

module_1.py

"""
module_1
----------

This fancy module has such a features
"""

init.py

"""
This library modules:

<Somehow get module docstrings of module 1>
"""

Solution can be opposite - document all modules in init and then reference from modules

Why?

In my case there is most copy pasters. Good to be able type once.

Docstring inheritance

Hey!

There is a new feature that I would like to discuss for docrep and I'd very much like your feedback (pinging @jgostick, @mennthor, @amnona, @lesteve). It's about docstring inheritance (i.e. the docrep.DocstringProcessor inserts parameters based on the base class).

Wouldn't it be nice to have something like

class A:

    def my_method(self, a=1):
        """Do something awesome

        Parameters
        ----------
        a: int
            A number

        Returns
        -------
        int
            The changed number
        """"
        return a + 1


@docstrings.resolve_inheritance
class B:

    @docstrings.inherit_parameters
    @docstring.inherit_returns
    def my_method(self, a=1):
        """Do something better"""
        return a + 2

and then have B.my_method the same docstring as A.my_method? inherit_parameters and inherit_returns would mark the method to be processed then in resolve_inheritance (Note: this separate treatment is necessary as, during class compilation, B is not yet defined and we therefore cannot resolve that B inherits A within inherit_parameters).

I'd like to discuss with you, how this should be set up (besides the implementation discussed above). Here are some ideas that I have and I highly appreciate your feedback and suggestions 😃

Specifying the base class

We could extend this by optionally specifying the base class, e.g. via

def funca(a=1):
    """"Do something

    Parameters
    ----------
    a: int
        The input
    """"
    return a+ 1

@docstrings.resolve_inheritance
@docstrings.inherit_parameters(funca)
def funcb(a=1):
    """Do something else"""
    return a + 2

This would then make it available for functions as well.

Replace arguments

We could also replace arguments in the sub class method (see alsp #16), e.g.

def funca(a=1, b=2):
    """"Do something

    Parameters
    ----------
    a: int
        The input (Default 1)
    b: int
        Another input (Default 2).
    """"
    return a+ b

@docstrings.resolve_inheritance
@docstrings.inherit_parameters(funca)
def funcb(a=3, b=2):
    """Do something else

    Parameters
    ----------
    a: int
        The input (Default 3)"""
    return a + 2

Specify parameters

I would in general use inspect.getfullargspec and match the parameters of the function with the parameters of base. But it should also have the possibility to specify them manually, e.g. to resolve stuff like *args, **kwargs. Here is an example

@docstrings.resolve_inheritance
@docstrings.inherit_parameters(funca, params=['a'])
def funcb(*args):
    """Do something else

    Parameters
    ----------
    b: int
        The input (Default 3)"""
    pass

funcb in this case should have used the docstring of parameter a although it is not in the argspec of funcb.

Python 2: for class docstrings '__doc__' of 'type' objects is not writable

Snippet reproducing the problem:

import docrep

docstrings = docrep.DocstringProcessor()


@docstrings.get_sectionsf('Base')
class Base(object):
    """ Base short description

    Base long description

    Parameters
    ----------
    param1 : int
    param2 : int
    """
    def __init__(self, param1, param2):
        pass


@docstrings.with_indent(4)
class Derived(Base):
    """ Derived short description.

    Derived long description.
    %(Base.parameters)s
    """
    pass

Full traceback:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/test-docrep.py in <module>()
     20 
     21 @docstrings.with_indent(4)
---> 22 class Derived(Base):
     23     """ Derived short description.
     24 

/home/local/lesteve/miniconda3/envs/scratch27/lib/python2.7/site-packages/docrep/__init__.pyc in replace(func)
    374                 func = func.im_func
    375             func.__doc__ = func.__doc__ and self.with_indents(
--> 376                 func.__doc__, indent=indent, stacklevel=4)
    377             return func
    378         return replace

AttributeError: attribute '__doc__' of 'type' objects is not writable

We found a work-around of doing something like this instead:

class Derived(Base):
   __doc__ = docstrings.with_indents(""" Derived class etc ... """)

Is that what the recommended approach?

More a question than an issue.

Apologies if this isn't the right forum for this.

If I do %(my_func.parameters)s % meta it's fine, but the bare %(my_func.parameters) gives

...docrep\__init__.py", line 105, in safe_modulo
    return s % meta
ValueError: unsupported format character '?' (0xa) at index 139

I'm just not sure what's happening and if there's something I should change in my code. For example escaping literal % characters.

Feature requests

Hi Philipp
We have started using docrep 'officially' in our project (OpenPNM). It's now merged into dev and will be release to master by the end of March.

I have it working pretty well, but there are two behaviors/features I wish it had. I apologize for being demanding, but here they are:

1. It would be great if individual parameters in subclasses over-rode those from the parent automatically. For instance, if I have the following method on a parent or abstract class:

def some_awesome_method(self, param1):
    r"""
    Parameters
    ------------
    param1 : int
         This parameters controls blah and it's default is None
    """

And I subclass this method on a child class:

def some_awesome_method(self, param1):
    r"""
    Parameters
    ------------
    param1 : int
         This parameters controls the size of the timestep and it's default is **10**
    """

In the above method I have indicated a different default value for the child class, and might also want to be more specific in the text, for instance. It would be REALLY helpful is the child class method just overwrote the parent class one, without me having to think about it. I know you have functionality for removing items from the parameter list, but this requires too much knowledge of docrep on the part of developers.

2. I understand that you only support 'numpydoc' valid headings (i.e. Parameters), but would it be possible to allow users to put arbitrary headings in? I use Spyder to for developing and Sphinx for doc generation, and both do not seem to mind having arbitrary headings (i.e. Transient Parameters, Reaction Parameters). This would allow me to breakup the parameter sections in the parent classes and then pick & choose which sections I show in subclasses.

Request 1 is far more important that 2.

License question

Will it be ok to copy the code and modify/use it in a BSD 3 license program we are developing (calour)?

Doesn't work with classes(?)

I tried to use it between inherited classes.
E.g. class B inherits from A and overrides a function with some extra args, so I want to use the same docstring for the identical arguments and just add in the extras.

If I use it like in your example and just wrap it inside classes, the second class can't find the keys defined in the first one (which seems reasonable, as it is a new scope): ... is not a valid key!.

How would this work/does this work with docrep?

Inherited docstrings not rendering in many places

It works fine when I do ctrl-i at the python command line in Spyder, but not from within the editor. I'm now hearing that is does not work on either in PyCharm or VSCode. I guess you've added some magic that works in spyder but is not standard protocol? Any suggestions?

Insert generated text into docstring?

This is more a feature request than an issue...

I would like to inject some programmatically generated text into my docstrings, and based on the awesomeness of docrep, I"m sure you know how to do it.

For instance, consider the following docstring:

"""
This is a function docstring

"{0}"

More docstring
""".format(('insert this'))

This is a valid python string and the text will be inserted at the {0} location. But in spyder and sphinx, this does not work. Since docrep is able to inject text that DOES work with spyder and sphinx, I wonder if you have any suggestions?

BTW, my actually use case is to inject a generated table of functions into the docstring for a module init, so that an up-to-date list of all functions can be displayed in spyder / sphinx without me having to manually maintain the list as functions are added/removed/renamed etc.