oprypin / mkdocs-gen-files

MkDocs plugin to programmatically generate documentation pages during the build

mkdocs-gen-files

Plugin for MkDocs to programmatically generate documentation pages during the build

pip install mkdocs-gen-files

Continue to the documentation site.

Usage

Activate the plugin in mkdocs.yml (scripts is a required list of Python scripts to execute, always relative to mkdocs.yml):

plugins:
  - search
  - gen-files:
      scripts:
        - gen_pages.py  # or any other name or path

Then create such a script gen_pages.py (this is relative to the root, not to the docs directory).

import mkdocs_gen_files

with mkdocs_gen_files.open("foo.md", "w") as f:
    print("Hello, world!", file=f)

This added a programmatically generated page to our site. That is, the document doesn't actually appear in our source files, it only virtually becomes part of the site to be built by MkDocs.

Continue to the documentation site.

mkdocs-gen-files's Issues

[Question] What process runs the `make_pages.py` script?

In short, I'm looking for the command that triggers the make_pages.py to improve my error handling when building with docker.

For context, I'm using the make_pages.py script to git clone multiple repos when I build the docs. I'm using the structure below, which works for repos that exist, and correctly fails for repos that don't exist, but can't handle private repos. I'd like to keep some repos private until I'm ready to publish, and just rebuild at that time.

import mkdocs_gen_files
from pathlib import Path
import subprocess

for REPO in my_list:
  if not Path(REPO).is_dir():
    try:
        subprocess.run(
            f"git clone https://github.com/{ORG}/{REPO}.git /main/".split(
                " "
            ),
            check=True,
        )
       # Other mv/rm commands removed for simplification
    except subprocess.CalledProcessError:
        pass  

If the repo doesn't exist, my except works well. If it's private, however, I see the following in my docker console

fatal: could not read Username for 'https://github.com/': No such device or address

I'm trying to track down the parent process to replicate the error better except the error type. If I try launching make_pages from a python terminal, it prompts me to enter in credentials.

Add py.typed to distribution

See: https://www.python.org/dev/peps/pep-0561/

For mypy users to get type checking when coding against this library, a py.typed file should be vendored with the distribution.

For Poetry, see: https://blog.whtsky.me/tech/2021/dont-forget-py.typed-for-your-typed-python-package/

python's match syntax is not supported

After many tests, we found that whenever python's use of match was used, an error was reported，You can repeat it. thanks！！

Path separator mismatch on Windows

Hello @oprypin 🙂
Still enjoying gen-files a lot!

I've just encountered the following issue on Windows (sorry, work...): gen-files warns me about files not existing.
It seems it's just a mismatch with path separators using backslashes on one side vs. forward slashes on the other:

self._edit_paths: {
    'reference/package/index.md': 'src\\package\\__init__.py',
    ...
}
# while
page.file.src_path: 'reference\\package\\index.md'

...so this condition is always false:

mkdocs-gen-files/mkdocs_gen_files/plugin.py

Line 44 in baf5a5e

if page.file.src_path in self._edit_paths:

I see that you are normalizing the path here:

mkdocs-gen-files/mkdocs_gen_files/editor.py

Line 70 in baf5a5e

self.edit_paths[_normpath(name)] = edit_name and str(edit_name)

Maybe separators should not be replaced, or maybe the later comparison should wrap them in Path again or something.

What do you think?

Feature request: Passing arguments to the scripts.

Why

I like using gen-files, and I am using it with literate-nav in multiple projects. However, only 1-5 lines of the script code are unique to each project. These lines could easily be replaced by passing arguments to the scripts instead.

Proposal

Use white space to separate the arguments just like when invoking with python in the shell, e.g. python3 docs/api_reference/.generate.py my_python_package. An example mkdocs.yml:

...
plugins:
  - gen-files:
      scripts:
        - docs/.index.py my_main_readme_file my_contributing_file my_install_file
        - docs/api_reference/.generate.py my_python_package
        - docs/examples/.generate.py my_examples_folder
...

Similar to shell invocation, if the scripts themselves have spaces in their names you would need to quote the script, e.g. :

...
plugins:
  - gen-files:
      scripts:
        - "\"docs/api_reference/.generate the api.py\" my_python_package"
...

I think this is acceptable, since the user already is/should be wary of using spaces in file names.

With arguments passing, each of my projects would have identical scripts but unique arguments in the mkdocs.yml.

Note that the beginning . in the script names is just to make mkdocs not copy the python scripts over to site.

Dynamic use while rendering Markdown pages

Hey @oprypin!

I'm working on a simple Markdown extension that allows to execute code blocks and inject their output in the page.
I'm building a small gallery of examples, and am currently trying to write a diagram example:

```python exec="true" show_source="tabbed-right"
import tempfile
from contextlib import suppress
import mkdocs_gen_files
from diagrams import Diagram, setdiagram
from diagrams.k8s.clusterconfig import HPA
from diagrams.k8s.compute import Deployment, Pod, ReplicaSet
from diagrams.k8s.network import Ingress, Service

with suppress(FileNotFoundError):
    with Diagram("Exposed Pod with 3 Replicas", show=False) as diagram:
        diagram.render = lambda: None
        net = Ingress("domain.com") >> Service("svc")
        net >> [Pod("pod1"),
                Pod("pod2"),
                Pod("pod3")] << ReplicaSet("rs") << Deployment("dp") << HPA("hpa")

        with mkdocs_gen_files.open(f"img/{diagram.filename}.png", "wb") as fd:
            fd.write(diagram.dot.pipe(format="png"))

output_html(f'<img src="../img/{diagram.filename}.png" />')
```

The issue is that, by the time this code is executed, FilesEditor._current is None again, so opening a file with mkdocs_gen_files.open triggers the creation of another instance of FilesEditor without a temporary directory attached. It means that files I create this way end up in my docs dir, which is unwanted.

So I attach a temporary directory myself:

        editor = mkdocs_gen_files.editor.FilesEditor.current()
        with tempfile.TemporaryDirectory(prefix="mkdocs_gen_files_") as tmp_dir:
            editor.directory = tmp_dir
            with editor.open(f"img/{diagram.filename}.png", "wb") as fd:
                fd.write(diagram.dot.pipe(format="png"))

But then it seems the file is not added to the final site because I'm getting a 404 😕

Copying/generating MD files

how can i include existing MD files in the final generated documentation?

Example:
Let's say I have projects/project1/code.py and projects/project1/README.md

Ideally the documentation created and nav would look like this:

• projects
• • project1
• -- README
• -- classname
• --- Method1()

I am currently able to generate the above, minus the README

literate-nav + section-index + gen-files

Hello @oprypin, hope you're doing well 🙂

I'm trying to generate a nav with gen-files, to be used by literate-nav, with a section index. Is it possible?
What is the gen-files equivalent of this nav?

* [Borgs](borgs/index.md)
    * [Bar](borgs/bar.md)
    * [Foo](borgs/foo.md)

I'm currently trying this:

"""Generate the code reference pages and navigation."""

from pathlib import Path

import mkdocs_gen_files

nav = mkdocs_gen_files.Nav()

for path in sorted(Path("src").glob("**/*.py")):
    module_path = path.relative_to("src").with_suffix("")
    doc_path = path.relative_to("src", "griffe").with_suffix(".md")
    full_doc_path = Path("reference", doc_path)

    # --------------> INTERESTING PART <-------------------
    parts = list(module_path.parts)
    if parts[-1] == "__init__":
        parts = parts[:-1]
    elif parts[-1] == "__main__":
        continue
    nav_parts = list(parts)
    print(nav_parts, doc_path)
    nav[nav_parts] = doc_path
    # --------------> END OF INTERESTING PART <------------

    with mkdocs_gen_files.open(full_doc_path, "w") as fd:
        ident = ".".join(parts)
        print("::: " + ident, file=fd)

    mkdocs_gen_files.set_edit_path(full_doc_path, path)

with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
    nav_file.writelines(nav.build_literate_nav())

But it gives me this nav:

...and prints this to stdout:

INFO     -  Building documentation...
['griffe'] __init__.md
['griffe', 'cli'] cli.md
['griffe', 'collections'] collections.md
['griffe', 'dataclasses'] dataclasses.md
['griffe', 'docstrings'] docstrings/__init__.md
['griffe', 'docstrings', 'dataclasses'] docstrings/dataclasses.md
['griffe', 'docstrings', 'google'] docstrings/google.md
['griffe', 'docstrings', 'markdown'] docstrings/markdown.md
['griffe', 'docstrings', 'numpy'] docstrings/numpy.md
['griffe', 'docstrings', 'parsers'] docstrings/parsers.md
['griffe', 'docstrings', 'rst'] docstrings/rst.md
['griffe', 'docstrings', 'utils'] docstrings/utils.md
['griffe', 'encoders'] encoders.md
['griffe', 'extended_ast'] extended_ast.md
['griffe', 'extensions'] extensions/__init__.md
['griffe', 'extensions', 'base'] extensions/base.md
['griffe', 'loader'] loader.md
['griffe', 'logger'] logger.md
['griffe', 'visitor'] visitor.md

I tried replacing the __init__.md filenames by index.md, but it gives the same result with "Index" in the nav instead of "init".

root is relative to config file.

it seems paths are relative to mkdocs config file, e.g. mkdocs.yml. Not sure if this is intentional. It is however consistent with other plugins.

I would modify the language in the readme.

Generate index.md warning

I want to use this plugin to generate a table of contents for my documentation, but I get the following warning:

WARNING - Documentation file 'index.md' contains a link to 'xx/index.html' which is not found in the documentation files.

Is it possible to change the timing of the plugin execution to eliminate this kind of warning?

Proposed feature: Skip N levels when building literate nav

I've worked on projects where the core codebase was in a subdirectory (e.g., src/project/), which resulted in additional clicks to go from top -> api reference -> src -> project -> relevant files

I propose an optional build_literate_nav parameter that would skip the first N levels as follows:

class Nav:
    ...
    def build_literate_nav(self, indentation: int | str = "", skip_levels : int = 0) -> Iterable[str]:
        """Builds a file suitable for https://github.com/oprypin/mkdocs-literate-nav, as a sequence of lines to be written into a file.

        For an example, see https://mkdocstrings.github.io/recipes/#generate-a-literate-navigation-file
        """
        if isinstance(indentation, int):
            indentation = " " * indentation
        for item in self.items():
            if item.level < skip_levels: 
                continue
            adjusted_level = item.level - skip_levels
            line = item.title
            if line.startswith(self._markdown_escape_chars):
                line = "\\" + line
            if item.filename is not None:
                line = f"[{line}]({item.filename})"
            yield indentation + "    " * adjusted_level + "* " + line + "\n"

In my case, this would permit me to include nav.build_literate_nav(skip_levels=2) in my page-generator script to remove the additional clicks mentioned above

If this is of interest, I'd be happy to open a PR

Thanks for all your work on this project!

`mkdocs-awesome-pages` doesn't jive with generated docs

👋 Loving the plugin!

I filed lukasgeiter/mkdocs-awesome-pages-plugin#78 against that repo, so I'll file one here too.

Essentially, generated pages (and navs) don't work well together.

How to use these plugins on readthedocs？

I'm using mkdocs-gen-files，mkdocs-literate-nav and section-index to generate docs for my project，however default script in this document can only create a random path (such as C:\Users\……\AppData\Local\Temp\mkdocs_utm0v2w3) and put files into it，so how to let readthedocs to use those files？Or I can only use it on local environment？
@oprypin @aahnik @davfsa

Client side redirects from anchors

Hi! I am trying to use this plugin for client side redirects with '#' in. I've got the basics working:

import mkdocs_gen_files

redirect_map = {
    "reference/about.md": "../glossary/#test-1",
}

redirect_template = """
<script type="text/javascript">
    window.location.href = "{link}";
</script>
<a href="{link}">Redirecting...</a>
"""

for page, link in redirect_map.items():
    with mkdocs_gen_files.open(page, "w") as fd:
        print(redirect_template.format(link=link), file=fd)

However, it doesn't work if the redirect is from an anchor tag:

redirect_map = {
   "reference/glossary/#test-1": "../about/" ,
}

It generates a page: glossary/#test-1 (I guess as intended) Is there a way to handle # in the original link?

Thanks for any help!

Using markdown snippets with mkdocs-gen-files

I wanted to insert a generated file with markdown snippet syntax --8<--. But it seems as if the files are not found. Probably when trying to include the files content the required pages are not generated yet. Am I missing any configuration or is this just not possible currently.

Best regards

Unable to run script with arguments

I am currently running a command to produce documentation and I observed that is not possible to pass arguments.

  - gen-files:
      scripts:
        - python3 -m ansiblelint -L -f docs

How to generate all files in an appointed directory？

I'm using your mkdocs-gen-files，mkdocs-literate-nav and section-index to generate docs for my project，however default script in this document can only create a random path (such as C:\Users\……\AppData\Local\Temp\mkdocs_utm0v2w3) and put files into it，so how to change the script and mkdocs.yml to appoint a directory such as root/docs？
Thanks for your reply.

Generated images are not persistent with build/gh-deploy

Hi mkdocs-gen-files devs!

Thank you very much for this useful plugin. I'm close to publishing my first documentation based on mkdocs, but I have a problem probably based on missing understanding of the underlying of your (Material for MkDocs or even basic MkDocs) plugin:

I'm generating images (SVG files) beside having gen_ref_pages.py which generates the Code reference of my library. The SVG files are created under an assets/images folder in docs. I want to use the command mkdocs gh-deploy --force in the end such that the images appear in the assets/images folder. But somehow there is only a favicon.png.

I tried it with mkdocs_gen_files like seen here:

diag: diagram.Diagram = model.by_uuid(uuid).context_diagram
with mkdocs_gen_files.open(f"{str(dest / diag.name)}.svg", "w") as fd:
    print(diag.as_svg, file=fd)

and while debugging I see that the SVGs are generated at the right location. But somehow during build there are no SVGs kept. Are the SVGs deleted again or is MkDocs ignoring them? Reading through this and this didn't help unfortunately. I hope someone of you knows how to fix this or gives a helpful hint 😃 .

Thanks in advance.

Scripts relative to mkdocs.yml file?

It would be great to be able to specify scripts relative to the mkdocs.yml file.

Use case

In my mkdocs.yml file:

plugins:
  - gen-files:
      scripts:
        - gen_pages.py

If the current working directory is the folder containing mkdocs.yml, this works:

mkdocs build

But if I want to run this from somewhere else, this does not work:

mkdocs build -f /absolute/path/to/mkdocs.yml

This is very cumbersome for all kinds of CI/CD situations. It would be nice for mkdocs-gen-files to be consistent with the rest of the ecosystem, where everything is relative to the mkdocs.yml file.

Interaction bug with mkdocs-awesome-pages if you edit all the *md files in a directory

I am using both this plugin and https://github.com/lukasgeiter/mkdocs-awesome-pages-plugin

As part of that plugin, many of our directories have .pages files in them containing metadata for the plugin to build the nav.

If I happen to use gen-files to edit all of the .md files in one of these directories, then awesome-pages can no longer read the .pages file in that same directory. Example error message:

INFO     -  NavEntryNotFound: Nav entry "howto" not found. [/home/nphilbrook/repos/<REDACTED>/.pages]
              File "/usr/local/lib/python3.10/dist-packages/mkdocs_awesome_pages_plugin/navigation.py", line 127, in _nav
                result = _make_nav_rec(meta.nav)
              File "/usr/local/lib/python3.10/dist-packages/mkdocs_awesome_pages_plugin/navigation.py", line 124, in _make_nav_rec
                warnings.warn(warning)

If I only edit some of the .md files in this directory, awesome-pages works as expected.

I have been skimming the source code to see if I can spot the bug easily, but ran out of time. Something about the way the gen-files 'virtual' directory is interacting with mkdocs and awesome-pages.

Add changelog

Hi, thanks for making this package! Would you be open to a PR adding a changelog? I plan to use the keep a changelog style.

Plugin not found

When I try and run mkdocs with the following config file I get an error:
Error: Invalid Plugins configuration

site_name: qtcodes

theme:
  name: "material"

plugins:
  - search
  - gen-files:
    scripts:
    - docs/gen_ref_pages.py    
  - mkdocstrings:

Thoughts?

Add `-> None` to `Nav.__init__()`

Hi, thanks for this really useful library!

I get an error from mypy when using Nav:

docs/gen_ref_pages.py:10: error: Call to untyped function "Nav" in typed context [no-untyped-call]

It is due to the fact that an __init__() method with no args and no return type is considered untyped.

Would you accept a PR adding -> None: for the method.

Cheers!