mkdocstrings / pytkdocs

Load Python objects documentation.

pytkdocs

Load Python objects documentation.

Installation

With pip:

pip install pytkdocs

With pipx:

python3.7 -m pip install --user pipx
pipx install pytkdocs

With conda:

conda install -c conda-forge pytkdocs

Usage

pytkdocs accepts JSON on standard input and writes JSON on standard output.

Input format:

{
  "objects": [
    {
      "path": "pytkdocs",
      "new_path_syntax": false,
      "members": true,
      "inherited_members": false,
      "filters": [
        "!^_[^_]"
      ],
      "docstring_style": "google",
      "docstring_options": {
        "replace_admonitions": true
      }
    }
  ]
}

Output format:

{
  "loading_errors": [
    "string (message)"
  ],
  "parsing_errors": {
    "string (object)": [
      "string (message)"
    ]
  },
  "objects": [
    {
      "name": "pytkdocs",
      "path": "pytkdocs",
      "category": "module",
      "file_path": "/media/data/dev/pawamoy/pytkdocs/src/pytkdocs/__init__.py",
      "relative_file_path": "pytkdocs/__init__.py",
      "properties": [
        "special"
      ],
      "parent_path": "pytkdocs",
      "has_contents": true,
      "docstring": "pytkdocs package.\n\nLoad Python objects documentation.",
      "docstring_sections": [
        {
          "type": "markdown",
          "value": "pytkdocs package.\n\nLoad Python objects documentation."
        }
      ],
      "source": {
        "code": "\"\"\"\npytkdocs package.\n\nLoad Python objects documentation.\n\"\"\"\n\nfrom typing import List\n\n__all__: List[str] = []\n",
        "line_start": 1
      },
      "children": {
        "pytkdocs.__all__": {
          "name": "__all__",
          "path": "pytkdocs.__all__",
          "category": "attribute",
          "file_path": "/media/data/dev/pawamoy/pytkdocs/src/pytkdocs/__init__.py",
          "relative_file_path": "pytkdocs/__init__.py",
          "properties": [
            "special"
          ],
          "parent_path": "pytkdocs",
          "has_contents": false,
          "docstring": null,
          "docstring_sections": [],
          "source": {},
          "children": {},
          "attributes": [],
          "methods": [],
          "functions": [],
          "modules": [],
          "classes": []
        }
      },
      "attributes": [
        "pytkdocs.__all__"
      ],
      "methods": [],
      "functions": [],
      "modules": [
        "pytkdocs.__main__",
        "pytkdocs.cli",
        "pytkdocs.loader",
        "pytkdocs.objects",
        "pytkdocs.parsers",
        "pytkdocs.properties",
        "pytkdocs.serializer"
      ],
      "classes": []
    }
  ]
}

Command-line

Running pytkdocs without argument will read the whole standard input, and output the result once.

Running pytkdocs --line-by-line will enter an infinite loop, where at each iteration one line is read on the standard input, and the result is written back on one line. This allows other programs to use pytkdocs in a subprocess, feeding it single lines of JSON, and reading back single lines of JSON as well. This mode was actually implemented specifically for mkdocstrings.

Configuration

The configuration options available are:

• new_path_syntax: when set to true, this option forces the use of the new object path syntax, which uses a colon (:) to delimit modules from other objects.

• filters: filters are regular expressions that allow to select or un-select objects based on their name. They are applied recursively (on every child of every object). If the expression starts with an exclamation mark, it will filter out objects matching it (the exclamation mark is removed before evaluation). If not, objects matching it are selected. Every regular expression is performed against every name. It allows fine-grained filtering. Example:

  • !^_: filter out every object whose name starts with _ (private/protected)
  • ^__: but still select those who start with two _ (class-private)
  • !^__.*__$: except those who also end with two _ (specials)

• members: this option allows to explicitly select the members of the top-object. If True, select every members that passes filters. If False, select nothing. If it's a list of names, select only those members, and apply filters on their children only.

• inherited_members: true or false (default). When enabled, inherited members will be selected as well.

• docstring_style: the docstring style to use when parsing the docstring. google, restructured-text¹ and numpy².

• docstring_options: options to pass to the docstring parser.

  • replace_admonitions boolean option (default: true). When enabled, this option will replace titles of an indented block by their Markdown admonition equivalent: AdmonitionType: Title will become !!! admonitiontype "Title".
  • trim_doctest_flags boolean option (default: true). When enabled, all doctest flags (of the form # doctest: +FLAG and <BLANKLINE>) located within python example blocks will be removed from the parsed output.

  The google docstring style accepts both options. The numpy style only accepts trim_doctest_flags. The restructured-text style does not accept any options.

¹: reStructured Text parsing is in active development and is not feature complete yet.
²: The following sections are currently not supported : Notes, See Also, Warns and References.

Details on new_path_syntax

Example:

New syntax	package.module:Class.attribute
Old syntax	package.module.Class.attribute

• If there is a colon is an object's path, pytkdocs splits the path accordingly, regardless of the value of new_path_syntax.
• If there isn't a colon, and new_path_syntax is false, pytkdocs uses the old importing behavior.
• If there isn't a colon, and new_path_syntax is true, pytkdocs uses the new importing behavior and therefore considers that the path points to a module.

pytkdocs's Issues

Idea: link builtin types and exceptions to their resp. docs

Example:

builtin_urls = {
    "int": "https://docs.python.org/3/library/functions.html#int",
    "str": "https://docs.python.org/3/library/functions.html#func-str",
    ...,  # other builtin types

    "OverflowError": "https://docs.python.org/3/library/exceptions.html#OverflowError",
    "ImportError": "https://docs.python.org/3/library/exceptions.html#ImportError",
    ...  # other exceptions
}

This data would allow us to use links in the Type column of parameters, return value and exceptions.

This could also be extended to support:

• the standard library
• the package's own objects (see #12)
• cross-documentation references (🚀) if we maintain a global mapping across all packages (😓). This could be a dedicated (and exciting) project on its own. Maybe it already exist? It would be huge, repetitive work, but could be automated a bit by crawling docs?

[FeatureRequest] Support the numpy docstring format

The numpy docstring format is very popular in the scientific python community.

The numpy docstring format is supported by the numpydoc and napoleon sphinx extensions.

print statements in the loaded code break the output

Describe the bug
I just thought about the case where the user could have added print statements that are executed at import time. In that case pytkdocs JSON output would be broken/corrupted.

To Reproduce

$ echo -e '"""Module docstring."""\n\nprint("finished loading")\n' > corrupt.py
$ echo '{"objects": [{"path": "corrupt"}]}' | poetry run pytkdocs | jq
parse error: Invalid literal at line 1, column 9

Expected behavior
The output of the imported code should be discarded.

System (please complete the following information):

• pytkdocs version 0.3.0
• Python version: irrelevant
• OS: irrelevant

Additional context
Possibly related to rdilweb/docs@cc20660 and mkdocstrings/mkdocstrings#91

Add Enum values

Is your feature request related to a problem? Please describe.

When documenting Enums, I will usually want to show the value of the enum, rather than just it's reference, to show the valid options.

Describe the solution you'd like

Rather than duplicating the enum in the docstring, I would prefer if a subclass of enum.Enum is encountered, the value of each attribute is shown next to it. Bonus: the value is color-highlighted with python syntax.

Describe alternatives you've considered

1. Documenting the Enum completely in markdown:

ServiceCancelReasons

An Enum defining the valid Cancellation reasons for a Service

CUSTOMER_REQUEST = "Customer Request"
INCORRECT_DATE_OR_TIME = "Incorrect Date Or Time"
OTHER = "Other"

2. Documenting each attribute with duplicating the value:

class MaintenanceStateIds(IntEnum):
    """Valid IDs for Maintenance states."""

    TEMPORARY = 1
    """Temporary: 1"""
    TENTATIVE = 2
    """Tentative: 2"""
    SCHEDULED = 3
    """Scheduled: 3"""
    CANCELLED = 4
    """Canceled: 4"""
    COMPLETED = 5
    """Completed: 5"""

3. Documenting using a combination of the above, duplicating the values in the top-level docstring.

class ServiceCancelReasons(str, Enum):
    """Valid reasons to cancel a service.

    ```python
    CUSTOMER_REQUEST = "Customer Request"
    INCORRECT_DATE_OR_TIME = "Incorrect Date Or Time"
    OTHER = "Other"
    ```
    """

    CUSTOMER_REQUEST = "Customer Request"
    INCORRECT_DATE_OR_TIME = "Incorrect Date Or Time"
    OTHER = "Other"

Additional context

Ideally, I could do something like the following:

class MaintenanceStateIds(IntEnum):
    """Valid IDs for Maintenance states."""

    TEMPORARY = 1
    """Maintenance is in a temporary state, before scheduling or confirming any time slots"""
    TENTATIVE = 2
    """Maintenance slot is being tentatively held on the calendar"""
    SCHEDULED = 3
    """Maintenance has been confirmed for a particular time slot on the calendar to be executed"""
    CANCELLED = 4
    """Maintenance was cancelled before execution"""
    COMPLETED = 5
    """Maintenance was completed, status is dependent on states of services"""

class MaintenanceStateNames(str, Enum):
    """Valid names for Maintenance states."""

    TEMPORARY = "Temporary"
    """Maintenance is in a temporary state, before scheduling or confirming any time slots"""
    TENTATIVE = "Tentative"
    """Maintenance slot is being tentatively held on the calendar"""
    SCHEDULED = "Scheduled"
    """Maintenance has been confirmed for a particular time slot on the calendar to be executed"""
    CANCELLED = "Canceled"
    """Maintenance was cancelled before execution"""
    COMPLETED = "Completed"
    """Maintenance was completed, status is dependent on states of services"""

and end up with the results (achieved through some quick HTML hacking):

Incorrectly dedenting class documentation (non PEP-257-compliant)

Leading whitespace on class comments is not stripped correctly when the first line of text is on the same line as the triple-quotes.

A class like the following:

class A:
     """This is a class.
     
     More words here.
     """
     pass

ends up having whitespace stripped to look like this:

This is a class.

    More words here.

whereas this is expected:

This is a class.

More words here.

The intended behaviour is specified in PEP-257: Docstring Conventions > Handling Docstring Indentation and implemented in inspect.cleandoc (which is called by inspect.getdoc)

This is only a problem for classes, since elsewhere pytkdocs does use inspect.getdoc

"OSError: could not get source code" - caused by class?

Hi,
I am unable to build docs for one file.
If I understand those errors correct than they might be caused by the following class definition:

class GSS(NamedTuple):
    """
        Helper Class to specify the return type of gss
    """

    gss: float
    nA: float
    nB: float
    nAB: float

Is there any way to solve this?

poetry run mkdocs build
INFO    -  Cleaning site directory 
INFO    -  Building documentation to directory: /mnt/c/Users/demo/Documents/repos/myownpackage/site 
Traceback (most recent call last):
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocs/__main__.py", line 163, in build_command
    ), dirty=not clean)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocs/commands/build.py", line 270, in build
    nav = config['plugins'].run_event('nav', nav, config=config, files=files)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocs/plugins.py", line 94, in run_event
    result = method(item, **kwargs)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocstrings/plugin.py", line 110, in on_nav
    root_object = self.documenter.get_object_documentation(import_string)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocstrings/documenter.py", line 305, in get_object_documentation
    root_object = self.get_module_documentation(obj)
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocstrings/documenter.py", line 323, in get_module_documentation
    root_object.add_child(self.get_class_documentation(member, module))
  File "/mnt/c/Users/demo/Documents/repos/myownpackage/.venv/lib/python3.6/site-packages/mkdocstrings/documenter.py", line 362, in get_class_documentation
    source = inspect.getsourcelines(actual_member)
  File "/usr/lib/python3.6/inspect.py", line 955, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python3.6/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code
Makefile:64: recipe for target 'docs' failed
make: *** [docs] Error 1

Raises section parsing of Google-style docstrings fails with multiple exceptions

d6561f8 stopped build errors in regards to this. But the results of a raises section with multiple exceptions (like this) end up like:

stdout is not restored when printing a traceback

Describe the bug
If an exception is raised in line-by-line mode, the standard output descriptor is not restored, so the JSON error is not printed, and it makes mkdocstrings hang since there's no output to read.

Expected behavior
stdout should be properly restored when an exception is caught in line-by-line mode.

Additional context
We should move the code responsible for discarding/restoring stdout into the main function of the cli module, so we can restore it in the except handler, and in the finally clause.

Treat "space" lines as empty lines in docstrings

Depending on the editor used, trailing spaces will be removed upon saving. When they are not removed, and a line at the end of a Parameters section contains 8 spaces, mkdocstrings will try to split it on : to get the param name and description, which will throw an exception.

So, lines composed only of spaces in docstrings should be considered empty lines.

Attribute of type logging.Logger only shows logging in HTML

def get_logger(level: str) -> logging.Logger:
    logger = logging.getLogger("my_package")
    logger.setLevel(logging.getLevelName(level.upper()))
    return logger


log: logging.Logger = get_logger("debug")
"""The logger used throughout this package."""

...though the return type of get_logger is correctly displayed as Logger.

Support default values for pydantic fields

Is your feature request related to a problem? Please describe.
Pydantic models can be documented via the new Attributes category. However, the default values are missing from the table.

Describe the solution you'd like
For this example,

from pydantic import BaseModel

class M(BaseModel):
   number: int = 2

pytkdocs should add 2 as the default for the field number.

Document docstring parsing limitations

Example:

"""
Args:
    obj: instance of [HelloWorld][1].

Some text.

[1]: https://url.to.helloworld/documentation
"""

This will not work because obj's description is isolated from the rest in the structured data, so the [1] is not available when mkdocstrings renders it, meaning the link will not be rendered.

Maybe we could parse such refs ([1], [link id], etc.) so mkdocstrings could append them to every Markdown piece it renders.

Docstring parser base

EDIT: we're not planning on outsourcing anymore.

Candidates/inspiration:

• https://github.com/rr-/docstring_parser
• https://github.com/rsokl/custom_inherit/blob/master/src/custom_inherit/_doc_parse_tools/napoleon_parse_tools.py
• https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/napoleon/docstring.py

[BUG] Don't apply parent docstring to overridden member when not selecting inherited members

Describe the bug
Individual attribute docstrings are re-applied to inherited class attributes even when not selecting inherited members. It doesn't feel right when autodocumenting a class and its subclasses in the same module, because then the attribute is documented in each one of them with the same docstring.

To Reproduce

class Base:
    VAR = 0
    """My variable."""

class Child(Base):
    VAR = 1

Screenshot

Expected behavior
When not selecting inherited members, I'd like the overwritten members to stay docstring-less, so they can be skipped with show_no_doc: false.

System (please complete the following information):

• pytkdocs 0.5.x

Mock unavailable libraries

• mkdocstrings version: [0.10.3]

Hi ,

I would like do to document a project that makes heavy use of external libraries where I don't have access to the sources. Does mkdocstrings offer something similar to mocks in sphinxs/autodoc
autodoc_mock_imports = ["django","td"]

Thanks in advance

Co-authoring

Hi @StephenBrown2, @cs01,

I'm creating this repo to move the code of mkdocstrings responsible for loading the documentation.

Since you were contributors, I want to add you as co-authors on the first commit, following this method: https://stackoverflow.com/questions/7442112/how-to-attribute-a-single-commit-to-multiple-developers

I want to make sure you agree to it 🙂

You can react with a thumbs up if you do!

For the "why" I'm moving this part of the code, I'll explain it in the Architecture issue 🙂

Support setting argument/return types in the docstring

Not everybody uses (or wants to use) type hints in their code. Not supporting setting types in the docstrings limits the user base of mkdocstrings to those that do.

Would you be amenable to adding support for setting the types in the docstring? It would probably make most sense to follow the Google style as done with sections.

Thanks for the consideration and the promising package.

Enable the inheritance of classmethod for docstrings

Currently the docstrings for inherited classmethods are completely ignored, this forces the user make trivial reimplementation of methods such as __init__ just to show proper documentation.

Original disscutions: mkdocstrings/mkdocstrings#78 (comment)

Cannot compute top package's path of a native namespace package

Describe the bug
Native namespace packages (without __init__.py modules) trigger the following kind of error when attempting to retrieve the top package's path:

ERROR   -  mkdocstrings.handlers.python: Collection failed: <module 'bot' (namespace)> is a built-in module
Traceback (most recent call last):
  File "/home/dadyarri/projects/bjorn/venv_bjorn/lib/python3.8/site-packages/pytkdocs/cli.py", line 179, in main
    print(json.dumps(process_json(line)))
  File "/home/dadyarri/projects/bjorn/venv_bjorn/lib/python3.8/site-packages/pytkdocs/cli.py", line 116, in process_json
    return process_config(json.loads(json_input))
  File "/home/dadyarri/projects/bjorn/venv_bjorn/lib/python3.8/site-packages/pytkdocs/cli.py", line 98, in process_config
    serialized_obj = serialize_object(obj)
  File "/home/dadyarri/projects/bjorn/venv_bjorn/lib/python3.8/site-packages/pytkdocs/serializer.py", line 143, in serialize_object
    relative_file_path=obj.relative_file_path,
  File "/home/dadyarri/projects/bjorn/venv_bjorn/lib/python3.8/site-packages/pytkdocs/objects.py", line 185, in relative_file_path
    top_package_path = Path(inspect.getabsfile(top_package)).parent
  File "/usr/lib/python3.8/inspect.py", line 720, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/usr/lib/python3.8/inspect.py", line 696, in getsourcefile
    filename = getfile(object)
  File "/usr/lib/python3.8/inspect.py", line 659, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module 'bot' (namespace)> is a built-in module

To Reproduce
Simply try to collect documentation for a native namespace package.

Expected behavior
No exception. We should find another way to determine the top package's path in that case.

Reference: mkdocstrings/mkdocstrings#79 (comment)

Discuss selection options

We'll try to list here to selection option that would be useful.

• option to unselect children of a certain category: submodules, nested classes, etc.

Add Examples section in Google parser

I think it would be quite useful to add an Example/Examples section as we have here.

Allow ignoring objects

Sometimes, I want to document an entire tree, but not specific modules. It would be nice if you could put something like @pytkdocs-ignore in the docstring and have pytkdocs, well, ignore it! This would be nice and helpful in situations where I don't want documentation to appear for a specific class at all.

Use attribute's type if no annotation found

Is your feature request related to a problem? Please describe.
Sometimes it's a bit redundant to add a type annotation. See for example copier-org/copier#203 (comment)

Describe the solution you'd like
Instead of using None or _empty, we could make sure to always fallback to the attribute's type with type(attr).
Typically done here: https://github.com/pawamoy/pytkdocs/blob/master/src/pytkdocs/loader.py#L699

Class attribute's type not detected properly when class is decorated

Describe the bug
It seems pytkdocs has trouble detecting a class attribute's type when the class is decorated.

To Reproduce
See an example here: copier-org/copier#203 (comment). The description is correctly parsed from the docstring, but the type is not used.

Expected behavior
The type (cli.Flag in the example) should appear in the Attributes table.

Screenshots
If you are using pytkdocs through mkdocstrings and if relevant, please attach a screenshot.

System (please complete the following information):

• pytkdocs version 0.6.0
• Python version: 3.6
• OS: Linux

Get descriptions for dataclasses fields

Is your feature request related to a problem? Please describe.
Currently descriptions of dataclasses fields will not be picked up.

Describe the solution you'd like
Pick up dataclasses fields descriptions. This will require to parse the code to find either comments or docstrings following the field.

Could be done alongside #30.

Support f-strings for attribute docstrings

Could be nice to be able to have this code (note the f-string for the log object):

import logging

DEFAULT_LOG_LEVEL = "debug"

def get_logger(level: str) -> logging.Logger:
    logger = logging.getLogger("my_package")
    logger.setLevel(logging.getLevelName(level.upper()))
    return logger

log: logging.Logger = get_logger(os.environ.get("MY_PACKAGE_LOG_LEVEL", DEFAULT_LOG_LEVEL))
f"""
The logger used throughout this package.

Initialized with a default level of {DEFAULT_LOG_LEVEL},
or using the environment variable `MY_PACKAGE_LOG_LEVEL`.
"""

We would need to match the f-string nodes as well when parsing attributes and their docstrings.

Spec: data extraction (code, module/class/function docstrings, attribute-docstrings, attribute-comments)

I want to try and clarify what is possible, what we want to support for documentation of attributes, and how to implement it. This is mostly me writing my thoughts to try and structure them. I'll update this post gradually.

What is possible

Attributes can be described

• in the Attributes section of a module docstring, a class docstring, or an __init__ method doctring (google-style)
• individually
  • with a triple-quoted string immediately after the declaration
  • with comments #: above the declaration
  • with a trailing comment #:

Attributes can be annotated

• with type annotations NAME: TYPE [= VALUE]
• with type hints in comments # type: TYPE
• by prefixing the attribute description in docstrings: TYPE: DESCRIPTION.

I'm not sure if a type and a description can be combined in a single comment: #: TYPE: DESCRIPTION or #: DESCRIPTION # type: TYPE.

What we want to support

Descriptions and annotations in comment are appealing, but I think comments have zero chance of being accessible via introspection, even in the far future of Python. If we were to only and entirely parse the source code to generate our data, then why not. But it's not the case, so I don't think supporting comments is worth the effort. Annotations will be retrieve from type annotations and docstrings, and descriptions will be retrieved from docstrings sections (like google-style's Attributes) or individual docstrings below declarations.

How to implement that

• Type annotations of module or class level attributes can be obtained using typing.get_type_hints(module_or_class) (it's using module_or_class.__annotations__ internally IIUC). No parsing of the code required 🎉
• Type annotations for instance-level attributes must be parsed from the source code: the __init__ method doesn't store any __annotations__, and get_type_hints on a method/function returns type annotations of its signature, not its body.
• Descriptions of module-level attributes can be parsed in the module docstring, or in individual docstrings.
• Descriptions of class-level attributes can be parsed in the class docstring, or in individual docstrings.
• Descriptions of instance-level attributes can be parsed in the class docstring, the __init__ method docstring, or in individual docstrings.

What we must be able to do

• parse the source code of a module or class to find docstrings below attributes declarations
• parse the source of code of __init__ methods to find documented instance attributes and their annotations
• pass module-level annotations while parsing module docstrings
• pass class-level annotations, instance-level annotations, and __init__ signature while parsing class docstrings
• pass instance-level annotations and __init__ signature while parsing __init__ methods docstrings.

The flow of operations while loading

• modules
  1. parse the module level code to find individual docstrings
  2. parse the module docstring (passing the previous data + get_type_hints(module) as context)
  3. use the previous data when iterating on members
• classes
  1. parse the code class level and (if it exists) the __init__ method code, to find individual docstrings and annotations of instance-attributes (in __init__)
  2. parse the class docstring (passing the previous data + get_type_hints(class) as context)
  3. use the previous data when iterating on members
• __init__ methods
  1. parse the method code to find individual docstrings and annotations of instance-attributes
  2. parse the method docstring (passing the previous data as context)
  3. no members to iterate on here

Things to cache?

As we can see, we will read the same source code several times. This will be done through inspect.getsource. The source is already read to populate the source value of objects. Maybe it's possible to do some caching here.

The __init__ methods code will also be parsed more than once. We could cache the found descriptions and annotations.

Technical details

To find attribute docstrings: search for pairs of ast.Assign / ast.AnnAssign and ast.Expr, where the value of the expression is ast.Str, ast.Constant, then recurse on nodes like ast.If, ast.IfExp, ast.Try, ast.With or ast.ExceptHandler, and again on their node.handlers, node.orelse and node.finalbody.

Updates to make to the docstrings parsers

When parsing a docstring, a parser should also be able to accept data like the following:

{
    # both description and real-type annotation
    "attribute1": {"description": "Some description", "annotation": int},

    # or only description
    "attribute2": {"description": "Some description"},

    # or only parsed-type annotation
    "attribute3": {"annotation": "int"},

    # additionally, for classes, we would have a level key
    # to distinguish class-attributes from instance-attributes
    "attribute4": {"annotation": str, "level": "class"},
    "attribute4": {"annotation": "str", "level": "instance"},
    "attribute4": {
        "class": {"annotation": str},
        "instance": {"annotation": "str"}
    }
}

I wonder if it's really necessary to pass descriptions since a docstring is written to add descriptions. They could be used to record a warning/error though, or the parser configured to use one or the other in priority.

parser not picking up methods of enums if not explicitly told to

Describe the bug
The parser is not picking up methods of enums, only their attributes when told to parse a enum subclass, e.g. Direction.
However, if instructed to parse a method of that enum directly (e.g. Direction.to_vector) it will get picked up correctly.

To Reproduce

class Direction(IntEnum):
    """This enum represents a direction in the 2d plane."""

    X = 0
    """X direction"""

    Y = 1
    """Y direction"""

    def to_vector(self) -> np.ndarray:
        """Turns a direction into its corresponding vector.

            Returns:
                the vector representation
        """
        if self == Direction.X:
            return np.array([1, 0])
        if self == Direction.Y:
            return np.array([0, 1])

Expected behavior
I expected to get the attributes as well as the methods.

System (please complete the following information):

• pytkdocs version 0.7
• Python version: 3.7.7
• OS: MacOS 10.15.4 (19E287)

pytkdocs collects incorrect path and no docstring for type-aliases

Describe the bug
When collecting type-alias attributes, pytkdocs in some cases incorrectly collects a path under the typing module. This seems to happen when the type alias is an alias to a type in the typing module, and not for example an alias for a user defined type.

To Reproduce
When defining a type alias in my_module.py

from typing import Union


IntOrStr = Union[str, int]
"""
A type alias that I wish to document with mkdocstrings.
"""

running the following:

$ echo '{"objects": [{"path": "my_module.IntOrStr"}]}' | pytkdocs

Produces

{"loading_errors": [], "parsing_errors": {}, "objects": [{"name": "IntOrStr", "path": "typing.IntOrStr", "category": "attribute", "file_path": "/Users/sune/.pyenv/versions/3.7.5/lib/python3.7/typing.py", "relative_file_path": "python3.7/typing.py", "properties": [], "parent_path": "typing", "has_contents": true, "docstring": "", "docstring_sections": [], "source": {}, "children": {}, "attributes": [], "methods": [], "functions": [], "modules": [], "classes": [], "type": "None"}]}

Expected behavior
Since the type-alias is just a module level attribute, I expect pytkdocs to collect the path as my_module.IntOrStr, and to collect the appropriate docstring as well in order that I can document it with mkdocstrings

Screenshots
An example from a project using mkdocstrings to document a type alias of Union[...]

System (please complete the following information):

• pytkdocs 0.6.0
• Python version: 3.7
• OS: macos 10.15.4

Use a JSONEncoder

...instead of this ugly serializer module.

Add support for django models

Is your feature request related to a problem? Please describe.
It would be nice to be able to document Django models without adding docstrings, just like pydantic models.

Describe the solution you'd like
Add similar support for Django models

Describe alternatives you've considered
Adding docstrings to every model field, I don't really like it because usually it means duplicating verbose_name value.

Support for examples section

Hi! I would like to start by saying this project has made mkdocs usable for our team and we are SO excited about finally ditching RST! I cannot emphasize this enough: Thank. You.

The only thing we are missing is that, currently, mkdocstrings doesn't support an Examples section, which normally include doctest formatted blocks. For example:

def example_function(arg1, kwarg=None) -> object:
    """
    Example function to demonstrate how APIs are rendered

    Parameters:
        arg1 (dict): Some description for this argument.
            This type (in parenthesis) is ignored.
        kwarg: Some more descriptions

    Returns:
        A description for the returned value

    Examples:

        >>> 2 + 2 == 4
        False

    """
    pass

gets rendered as:

The style is not bad, but it could be even better! Notice how pytest is able to parse the block just fine.

I can get what I want by using this other docstring:

def example_function(arg1, kwarg=None) -> object:
    """
    Example function to demonstrate how APIs are rendered

    Parameters:
        arg1 (dict): Some description for this argument.
            This type (in parenthesis) is ignored.
        kwarg: Some more descriptions

    Returns:
        A description for the returned value

    __Examples__

    ```python
    >>> 2 + 2 == 4
    False

    ```
    """
    pass

, but I find it a bit annoying (in order of importance):

1. Manually hardcoding the Examples heading style
2. Having to specify the code fences
3. Needing an extra blank line before the closing triple backticks so pytest can get the doctests correctly.

Maybe a good compromise would be to just add Examples to the recognized headers? That way I could write:

def example_function(arg1, kwarg=None) -> object:
    """
    Example function to demonstrate how APIs are rendered

    Parameters:
        arg1 (dict): Some description for this argument.
            This type (in parenthesis) is ignored.
        kwarg: Some more descriptions

    Returns:
        A description for the returned value

    Examples:

        ```python
        >>> 2 + 2 == 4
        False

        ```
    """
    pass

, which currently does not work as intended:

I'd be happy to help and contribute a PR if needed, but if I understood correctly, you are currently refactoring the backend and I don't know if this is the best moment to add a feature.

Thanks a lot!

Allow picking up inherited members for classes

See mkdocstrings/mkdocstrings#78.

We should change

for member_name, member in class_.__dict__.items():

...to

for member_name, member in inspect.getmembers(class_):

...based on configuration options.

We must also distinguish inherited members from regular ones, to be able to apply different filters.

[BUG] 0.5.0 AttributeError Tuple has no attribute id

Describe the bug

Traceback (most recent call last):
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/cli.py", line 193, in main
    output = json.dumps(process_json(line))
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/cli.py", line 114, in process_json
    return process_config(json.loads(json_input))
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/cli.py", line 91, in process_config
    obj = loader.get_object_documentation(path, members)
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/loader.py", line 232, in get_object_documentation
    root_object = self.get_class_documentation(leaf, members)
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/loader.py", line 333, in get_class_documentation
    merge(attributes_data, get_class_attributes(cls))
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/parsers/attributes.py", line 127, in get_class_attributes
    return combine(get_module_or_class_attributes(nodes[0].body), type_hints)
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/parsers/attributes.py", line 85, in get_module_or_class_attributes
    names = [target.id for target in assignment.targets]
  File "/home/runner/work/integrations-core/integrations-core/.tox/docs/lib/python3.8/site-packages/pytkdocs/parsers/attributes.py", line 85, in <listcomp>
    names = [target.id for target in assignment.targets]
AttributeError: 'Tuple' object has no attribute 'id'

Screenshots

https://github.com/DataDog/integrations-core/runs/751798110?check_suite_focus=true

Support Pydantic model attributes

I'm using Pydantic for several models in my code, and I would like mkdocstrings to handle them. The issue is, class attributes get sucked up into a __fields__ attribute with separate __annotations__, and so they get filtered out by filter_name_out due to the global_filters.

Example __dict__.items() on a Pydantic class with two fields (debugged from just before https://github.com/pawamoy/mkdocstrings/blob/ca82f6964e223b7d8b0a3d0e0fe4d9ea8f4296ce/src/mkdocstrings/documenter.py#L340):

class_.__dict__.items(): (
    dict_items(
        [
            ('__config__', <class 'pydantic.main.Config'>),
            ('__fields__', {
                'start': ModelField(name='start', type=DateTime, required=True),
                'end': ModelField(name='end', type=DateTime, required=True)
            }),
            ('__field_defaults__', {}),
            ('__validators__', {
                'start': [<pydantic.class_validators.Validator object at 0x10257df10>],
                'end': [<pydantic.class_validators.Validator object at 0x10257df10>]
            }),
            ('__pre_root_validators__', []),
            ('__post_root_validators__', []),
            ('__schema_cache__', {}),
            ('__json_encoder__', <staticmethod object at 0x10260e978>),
            ('__custom_root_type__', False),
            ('__module__', 'mypackage.models.available_times'),
            ('__annotations__', {
                'start': <class 'pendulum.datetime.DateTime'>,
                'end': <class 'pendulum.datetime.DateTime'>
            }),
            ('__doc__', '### TimeSlot\n\n    A slot of time in which a maintenance may be scheduled.\n    '),
            ('__lt__', <function TimeSlot.__lt__ at 0x1026152f0>),
            ('parse_times', <classmethod object at 0x10260e828>),
            ('__weakref__', <attribute '__weakref__' of 'TimeSlot' objects>),
            ('__abstractmethods__', frozenset()),
            ('_abc_registry', <_weakrefset.WeakSet object at 0x10260e908>),
            ('_abc_cache', <_weakrefset.WeakSet object at 0x10260ea58>),
            ('_abc_negative_cache', <_weakrefset.WeakSet object at 0x10260eac8>),
            ('_abc_negative_cache_version', 42)
        ]
    )
)

A crude implementation for an older version of pydantic was put forward here: pydantic/pydantic#638 but not accepted into Pydantic proper. Perhaps mkdocstrings could take some of the logic and parse through the field attributes?

Update: I should probably also note that I started down this path by attempting to add some docstrings to the attributes as suggested by mkdocstrings docs, but then got this error:

  File "/Users/step7212/git/rack/emcee/.nox/docs-3-6/lib/python3.6/site-packages/mkdocstrings/documenter.py", line 437, in node_to_annotated_names
    name = node.target.attr
AttributeError: 'Name' object has no attribute 'attr'

Changing node.target.attr to node.target.id fixed that for me avoided the error, but I'm not sure how relevant it is, just means I can't docstring my pydantic models (yet) since it's filtering the dunder-attrs.

Support more objects properties

• generators, coroutines, awaitable, ...see inspect.is...
• decorators?
• metaclass, dataclass
• optional (parameters with default values), kinda already done with the type annotations

CollectionError swallows exception

Is your feature request related to a problem? Please describe.
I've noticed when collecting, pytkdocs swallows errors from the module in question. For example if there is an issue importing a module, It returns that:

AttributeError: module 'module.x.y' has no attribute 'z'

However this usually points to something amiss in the module (unmet dependency). If i open up a cli and try to import the module by hand i can see whats going on.

Describe the solution you'd like
It would be nice if pytkdocs could bubble the underlying exception through to CollectionError for easier diagnosis.

Return members in the same given order

Is your feature request related to a problem? Please describe.
Feature asked in mkdocstrings: mkdocstrings/mkdocstrings#102 (comment)

Describe the solution you'd like

• pytkdocs should re-order the root object children using the given members order.
• For this we'll need to stop casting members as a set, and keep it as a list.
• Before returning the root object, we'll then need to re-order them.
• Let's make it the default, not configurable behavior (sorting is more a rendering issue, we shouldn't implement rendering configuration in pytkdocs)

Describe alternatives you've considered
It's easier to do it in pytkdocs than re-sorting in mkdocstrings: the renderer and collector options are separated, it would be messy to use the collector options in the renderer, and I don't want to duplicate the members option in the renderer (nor users want to copy/paste it).

Add option to move the __init__ docstring to the class docstring

Usually, editors will not help writing the parameters documentation into the class' docstrings but rather in the __init__ method. But everybody knows that __init__ is used when instantiating an object. We rarely call it directly. So maybe it would make more sense to move the parameters documented in the __init__ docstring into the class' docstring.

Pick attributes without docstrings

Just like we still select objects with no docstrings, we could also pick attributes without docstrings.

The user can then configure the plugin to render them or not.

vague unpack parsing errors parsing google docstrings.

I'm using google docstrings format for my docstrings and I'm often getting:

annotation, description = exception_line.split(": ")
ValueError: not enough values to unpack (expected 2, got 1)

It's really difficult to debug when the traceback only says which file is causing the error. Would be nice to include at least an aproximate where the fault actually is.

Error when handling **kwargs parameters

Describe the bug
I get the following error when I document **kwargs parameter in my docstring with the two **.

line 216, in read_parameters_section
    signature_param = self.signature.parameters[name]
KeyError: '**kwargs'

To Reproduce

def function(a, **kwargs):
    """
    Args:
        a: a parameter.
        **kwargs: kwarg parameters
    """
    return None

Information (please complete the following information):

• OS: Debian
• mkdocstrings version: [0.7.0]

Additional context
This seems to come from the inspect module as inspect.signature(function).parameters would return:

mappingproxy({'a': <Parameter "a">, 'kwargs': <Parameter "**kwargs">})

and the **kwargs key is not present but kwargs is.

If decorator is used, function disappears from mkdocstrings documentation

Describe the bug

When a decorator is used before a function, the function does NOT appear in the documentation:

To reproduce, try placing this as a decorator before any function:

import functools

@functools.lru_cache(maxsize=1024)
def my_function(some_arg):
    print()

Screenshots
When this decorator is used, the function does NOT appear in the documentation:

When the decorator is not used, the function DOES appear in the documentation:

System (please complete the following information):

• pytkdocs version: latest
• Python version: 3.7
• OS: Mac OS X

[BUG] Individual docstrings are not dedented

⬆️

Refactor the attributes parser

Is your feature request related to a problem? Please describe.
The attributes parser code is messy, and not tested. We're currently parsing attributes and their docstrings separately from the rest of the loading, and I think this could be improved to be more integrated in the loading flow, but also more decoupled from it.

Describe the solution you'd like
Several things can be improved:

• as @shyamd suggested in #25, we could return structured data instead of Attribute instances to decouple the parser from the objects/loader.
• instead of parsing the whole module, we could take advantage of inspect's getsource methods to parse only attributes for a given object. This could decrease performance though, as inspect might not cache the contents of a file. Maybe its methods are efficient though.
• as seen in typed-argument-parser, we could use a lexer/tokenizer instead of the ast module to parse the code. With this we could maybe even bring support for comments, but I'm still not sure I want to have it.
• we could combine this refactor with #11 (pick attributes without docstrings) and #31 (support dataclasses fields descriptions)

We could have a function that accepts a module or class, and return all its attributes, in the form of a list of dicts.

[BUG] Failing on a class with `__getattr__` method

Describe the bug

I upgraded to pytkdocs 0.5.1 and got this error:

ERROR   -  mkdocstrings.handlers.python: Collection failed: ('No route with that name.', '__wrapped__')
Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/pytkdocs/cli.py", line 193, in main
    output = json.dumps(process_json(line))
  File "/venv/lib/python3.8/site-packages/pytkdocs/cli.py", line 114, in process_json
    return process_config(json.loads(json_input))
  File "/venv/lib/python3.8/site-packages/pytkdocs/cli.py", line 91, in process_config
    obj = loader.get_object_documentation(path, members)
  File "/venv/lib/python3.8/site-packages/pytkdocs/loader.py", line 230, in get_object_documentation
    root_object = self.get_module_documentation(leaf, members)
  File "/venv/lib/python3.8/site-packages/pytkdocs/loader.py", line 311, in get_module_documentation
    root_object.add_child(self.get_module_documentation(leaf))
  File "/venv/lib/python3.8/site-packages/pytkdocs/loader.py", line 295, in get_module_documentation
    child_node = ObjectNode(member, member_name, parent=node)
  File "/venv/lib/python3.8/site-packages/pytkdocs/loader.py", line 43, in __init__
    self.obj: Any = inspect.unwrap(obj)
  File "/usr/lib/python3.8/inspect.py", line 520, in unwrap
    while _is_wrapper(func):
  File "/usr/lib/python3.8/inspect.py", line 511, in _is_wrapper
    return hasattr(f, '__wrapped__')
  File "/venv/lib/python3.8/site-packages/beancount/web/bottle_utils.py", line 25, in __getattr__
    return self.mapper_function(name)
  File "/venv/bin/bottle.py", line 409, in build
    if not builder: raise RouteBuildError("No route with that name.", _name)
bottle.RouteBuildError: ('No route with that name.', '__wrapped__')
 
ERROR   -  mkdocstrings.extension: Could not collect 'beancount.web' 
ERROR   -  Error reading page 'api_reference/beancount.web.md': ('No route with that name.', '__wrapped__') 
Traceback (most recent call last):
  File "build.py", line 52, in <module>
    sys.exit(cli())
  File "/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/mkdocs/__main__.py", line 140, in serve_command
    serve.serve(
  File "/venv/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 141, in serve
    config = builder()
  File "/venv/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 136, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "/venv/lib/python3.8/site-packages/mkdocs/commands/build.py", line 274, in build
    _populate_page(file.page, config, files, dirty)
  File "/venv/lib/python3.8/site-packages/mkdocs/commands/build.py", line 174, in _populate_page
    page.render(config, files)
  File "/venv/lib/python3.8/site-packages/mkdocs/structure/pages.py", line 183, in render
    self.content = md.convert(self.markdown)
  File "/venv/lib/python3.8/site-packages/markdown/core.py", line 263, in convert
    root = self.parser.parseDocument(self.lines).getroot()
  File "/venv/lib/python3.8/site-packages/markdown/blockparser.py", line 90, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "/venv/lib/python3.8/site-packages/markdown/blockparser.py", line 105, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "/venv/lib/python3.8/site-packages/markdown/blockparser.py", line 123, in parseBlocks
    if processor.run(parent, blocks) is not False:
  File "/venv/lib/python3.8/site-packages/mkdocstrings/extension.py", line 147, in run
    data: Any = handler.collector.collect(identifier, selection)
  File "/venv/lib/python3.8/site-packages/mkdocstrings/handlers/python.py", line 204, in collect
    raise CollectionError(result["error"])
mkdocstrings.handlers.CollectionError: ('No route with that name.', '__wrapped__')

Here's the relevant source code:

https://github.com/beancount/beancount/blob/2.3.0/beancount/web/bottle_utils.py#L24-L25

System (please complete the following information):

• pytkdocs version: 0.5.1
• Python version: 3.8
• OS: Arch Linux

Dataclasses attributes get picked up twice

Describe the bug
Dataclasses attributes are picked up twice, because they appear in the direct members, and then we iterate on __dataclass_fields__ again.

To Reproduce

@dataclass
class FrozenPerson:
    """I'm a frozen person."""

    say: str = "Oh no"
    """Say attr."""

Expected behavior
Attributes are picked up only once.

Screenshots

System (please complete the following information):

• pytkdocs 0.6.0
• Python version: 3.7
• OS: Linux

Guess object type when not annotated

I think it would be quite easy to get an object's annotation with annotation = type(obj) when it is not annotated.

Store dotted path for annotations types that are within the package

For example, if my function declares a return type of MyClass, then we should detect that it's referencing the class mypackage.mymodule.MyClass and store that information somewhere. It could allow us to write links to objects instead of just strings!

Pydantic code sometimes set a docstring to None

Describe the bug
If a pydantic field doesn't have a docstring, the docstring is not set to "", making the parser fail.

To Reproduce
Try to build the documentation of pytkdocs. It will hang because of a combination of this issue and #36

Expected behavior
Docstring should always be a string.

Additional context
Maybe we could simply accept None in the parsers and return immediately if it's None, rather than making sure the docstring is always set to "" as default.