pyscaffold / pyscaffoldext-custom-extension

🚧 PyScaffold extension to generate custom user extensions

pyscaffoldext-custom-extension

PyScaffold extension that lets you create your own custom extensions.

Description

This extension was written to help users interested in developing their own extension for PyScaffold. It configures your project so that you can start writing your extension logic and tests right away, taking care of all the wiring required to conform to PyScaffold's needs.

Let's say you want to create an extension named notebooks that creates a notebooks folder with some template Jupyter notebook. After having installed this extension with:

pip install pyscaffoldext-custom-extension

you will be able to just use it with:

putup --custom-extension notebooks

This will create a typical PyScaffold project template with some modifications:

• the topmost namespace will be pyscaffoldext to have a unified namespace for PyScaffold extensions,
• assures that the package (as pip/PyPI sees it) is named pyscaffoldext-notebooks in setup.cfg,
• sets the correct install_requires as well as the options.entry_points parameters in setup.cfg,
• automatically activates the extensions --no-skeleton, --pre-commit, --cirrus and since we want clean-coded, high-quality extensions,
• creates a extension.py module holding a class which serves you as a template for your extension,
• adds basic unit tests checking that the invocation of your extension works and that it complies with our flake8 code guidelines,
• provides a modified README.rst indicating that this is a PyScaffold extensions and how to install it.

Making Changes & Contributing

This project uses pre-commit, please make sure to install it before making any changes:

pip install pre-commit
cd pyscaffoldext-custom-extension
pre-commit install

It is a good idea to update the hooks to the latest version:

pre-commit autoupdate

Please also check PyScaffold's contribution guidelines,

Note

For more information about PyScaffold and its extension mechanism, check out https://pyscaffold.org/.

pyscaffoldext-custom-extension's Issues

Issue while pip install

Whenever I install this package I get the following from pip

Installing collected packages: configupdater, pyscaffold, pyscaffoldext-custom-extension
  Attempting uninstall: configupdater
    Found existing installation: ConfigUpdater 2.0
    Uninstalling ConfigUpdater-2.0:
      Successfully uninstalled ConfigUpdater-2.0
  Attempting uninstall: pyscaffold
    Found existing installation: PyScaffold 4.0.1
    Uninstalling PyScaffold-4.0.1:
      Successfully uninstalled PyScaffold-4.0.1
Successfully installed configupdater-1.1.3 pyscaffold-4.0 pyscaffoldext-custom-extension-0.6.1

Seems to uninstall the latest version from PyScaffold and then putup command is not working any more:

putup --a-python-template test                                                                                                                                                                  
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/bin/putup", line 5, in <module>
    from pyscaffold.cli import run
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/cli.py", line 13, in <module>
    from . import api, templates
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/api.py", line 9, in <module>
    from . import actions, info
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/actions.py", line 20, in <module>
    from . import info, repo
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/info.py", line 30, in <module>
    from .templates import ScaffoldOpts, licenses, parse_extensions
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/templates/__init__.py", line 14, in <module>
    from .. import dependencies as deps
  File "/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/pyscaffold/dependencies.py", line 9, in <module>
    from setuptools_scm.version import VERSION_CLASS
ImportError: cannot import name 'VERSION_CLASS' from 'setuptools_scm.version' (/usr/local/Caskroom/miniconda/base/envs/ds-template/lib/python3.8/site-packages/setuptools_scm/version.py)

I guess should be as easy as increase the requirements.
Cheers

Have an extension call other extensions

Hiya,

I'm trying to write an extension that call other extensions, I've seen the the include command but I'm a confused on when I need to use it. Do I register it in the actions pipeline?

tox 4.0.17+ breaks test_custom_extension on windows

First mentioned here: tox-dev/tox#2776 (comment)

tox 4.0.17 (specifically tox-dev/tox@bf87bea) reliably breaks our custom extension built with Jenkins on Windows (python 3.8).

I bisected tox all the way between 4.0.16 (which was good) and 4.3.5.
No code changed and no python deps changed apart from tox.

Log from tox==4.3.5:

tmpfolder = WindowsPath('C:/Users/zSvcbuildmaster01/AppData/Local/Temp/pytest-of-unknown/pytest-99/test_generated_extension0/tmpqbuyf27f')

...
Error while running command:
['D:\\\\jenkins\\\\workspace\\\\pyscaffoldext-mything_master\\\\.tox\\\\test-windows\\\\Scripts\\\\python.EXE', '-m', 'tox']
{'stderr': -2, 'universal_newlines': True}

test-windows: internal error
Traceback (most recent call last):
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\session\\cmd\\run\\single.py", line 45, in _evaluate
    tox_env.setup()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\tox_env\\api.py", line 248, in setup
    self._setup_env()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\tox_env\\python\\runner.py", line 106, in _setup_env
    super()._setup_env()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\tox_env\\python\\api.py", line 198, in _setup_env
    self.ensure_python_env()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\tox_env\\python\\api.py", line 205, in ensure_python_env
    self.create_python_env()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\tox\\tox_env\\python\\virtual_env\\api.py", line 129, in create_python_env
    self.session.run()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\run\\session.py", line 42, in run
    self._create()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\run\\session.py", line 49, in _create
    self.creator.run()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\create\\creator.py", line 153, in run
    self.create()
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\create\\via_global_ref\\builtin\\via_global_self_do.py", line 89, in create
    src.run(self, self.symlinks)
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\create\\via_global_ref\\builtin\\ref.py", line 151, in run
    method(self.src, dest)
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\util\\path\\_sync.py", line 33, in copy
    ensure_safe_to_do(src, dest)
  File "D:\\jenkins\\workspace\\pyscaffoldext-mything_master\\.tox\\test-windows\\lib\\site-packages\\virtualenv\\util\\path\\_sync.py", line 23, in ensure_safe_to_do
    dest.unlink()
  File "C:\\Python38\\lib\\pathlib.py", line 1324, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError 5] Access is denied: 'D:\\\\jenkins\\\\workspace\\\\pyscaffoldext-mything_master\\\\.tox\\\\test-windows\\\\Scripts\\\\python.exe'
test-windows: FAIL ! in 0.3 seconds

Create files that are part of the .gitignore

I'm trying to create a rather simple extension that creates a starting .vscode/settings.json for projects to initialise some basic vscode settings and that sort of thing. However, the .vscode folder is in the default .gitignore (which shouldn't change) so the init_git action will raise an error due to ignored files. See below:

pyscaffold.exceptions.ShellCommandException: The following paths are ignored by one of your .gitignore files:
           .vscode/settings.json
           Use -f if you really want to add them.

I've tried add an action after the init_git action, but that doesn't create the file. Is there a way to do this?

Add proper docstrings

Add docstrings to the sourcecode following the Google documentation style like done in PyScaffold.
We should not see docstrings like this any more:

    """
    Enforce the naming convention of PyScaffold extensions.
    The project name must start with 'pyscaffoldext-' and
    the package name shouldn't contain the redundant
    'pyscaffoldext_' in the beginning of the name.
    :param struct:
    :param opts:
    :return:
    """

It seems you are using PyCharm. You can change the automatic generated docstring (:param struct: etc.) to the Google style in your settings btw.

Also add tox to pyscaffoldext-custom-extension itself

Currently there is no tox.ini within this repo. Since we propose this as a best practice we should also include it in all our extensions. The pre-commit and tox extensions then also need to be added in the [pyscaffold] section of setup.cfg.

Generate tests that combine pre-commit and namespace

At some point, we probably should also include a namespace here --namespace my.ns to ensure the files being generated by the new extension do not break pre-commit for the worst case scenario. I suppose it is very easy to forget considering namespaces when developing a new extension.

pyscaffoldext-custom-extension/src/pyscaffoldext/custom_extension/templates/test_custom_extension.template

Line 36 in 1119dfc

args = ["--no-config", "--venv", "--pre-commit", EXT_FLAG, "my_project"]

See discussion in:
pyscaffold/pyscaffoldext-dsproject#11
pyscaffold/pyscaffoldext-dsproject#12

Make sure project name is `pyscaffoldext-myname`

I think it is a best practice to also name custom extensions pyscaffoldext-my-extension. This is currently not enforced. Maybe one could check if opts['project].startswith('pyscaffoldext-') and add it accordingly if not provided. This would then comply with our current documentation and makes it extremely helpful and easy to find extensions with PyPI.

Provide a LINESEP constant in extension by default

As I just stumpled on a bug that only occurred on Windows I think it makes sense to have by default in everyextension.py following code so that people are aware of using LINESEP instead of os.linesep.

# Use LINESEP instead of os.linesep since '\n' is used in templates
LINESEP = '\n'

small README change

just so that I can put issue in pull request

Generated package is not working properly with PyScaffold

I've created and installed a package using this generator and when I tried to run it, I got this error:

putup --ds-python-template test                                                                                                                                                                  (ds-template) (*master+1) 19:01:53
ERROR: There was an error loading 'ds_python_template'.
    Please make sure you have installed a version of the extension that is compatible
    with PyScaffold 4.0.1. You can also try unininstalling it.

Generate tests combining pre-commit and namespace

At some point, we probably should also include a namespace here --namespace my.ns to ensure the files being generated by the new extension do not break pre-commit for the worst case scenario.

pyscaffoldext-custom-extension/src/pyscaffoldext/custom_extension/templates/test_custom_extension.template

Line 36 in 1119dfc

args = ["--no-config", "--venv", "--pre-commit", EXT_FLAG, "my_project"]

It is very easy to forget considering namespaces when developing a new extension, so the best is to automatically remind ourselves via testing.

See discussion in:
pyscaffold/pyscaffoldext-dsproject#11
pyscaffold/pyscaffoldext-dsproject#12

new .template files are not being picked up in build/install

When using custom-extension to generate a new extension i followed the examples, particularly the dsproject example. When I came to add .template files these are somehow not being picked up in the build/install and therefore my custom extension is failing with a

FileNotFoundError: [Errno 2] No such file or directory: '/opt/homebrew/lib/python3.9/site-packages/pyscaffoldext_generative-0.0.post1.dev1+g12e1872.d20220111-py3.9.egg/pyscaffoldext/generative/templates/lint_sh.template'

Steps to reproduce:

OS: Macos Monterey v12.0 (M1)
Python: Python 3.9.9

I verified similar behaviour in the dsproject:

1, git clone dsproject

2, Add a new .py and a new .template file to the src/pyscaffoldext/dsproject/templates folder.

3, run python3 setup.py install

4, Check the generated SOURCES.txt file

Expected: Both .py and .template files are added to this list

Actual: only the new .py file is added.

In dsproject the pre-existing .template files are added as expected but I have been unable to find where this filter or these specific files are specified. Its very possible I am doing something wrong here but it isnt clear from the documentation which might benefit from an example of adding custom templates.

Thanks.

On the road to PyScaffold 4.0

I have created a new pyscaffold-v4 branch containing the code prepared to run with v4.
The idea is to do the same for all the extensions (and hopefully having migrated this one first will help, so we can do a forced update + git difftool).

@FlorianWilhelm, I have noticed that you were working in a v0.6a1 tag. Will that v0.6 still be released (and in that case v0.7 will be the version compatible with PyScaffold v4)? Or the newest changes will be absorbed into a v0.6?