davidvujic / python-polylith

Tooling support for the Polylith Architecture in Python.

✨ Python tools for the Polylith Architecture ✨

A repo containing tooling support for the Polylith Architecture in Python.

What's Polylith? 🤔

Polylith is an architecture (with tooling support) originally built for Clojure. The code in this repo brings Polylith to Python.

From the official Polylith Architecture docs:

... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ...

Polylith is using a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, serverless functions and microservices.

Documentation 📚

Have a look at the Python-specific documentation. You will find installation, setup, usage guides and more.

Polylith for Python? 🐍

This repo contains a Poetry plugin, a CLI that enables support for Hatch, PDM, Rye and Pantsbuild. There's a Hatch Build Hook plugin and a PDM build hooks to fully support these tools.

• a Poetry Plugin
• a CLI
• a Hatch Build Hook
• a PDM Build Hook for projects
• a PDM Build Hook for the workspace

The Poetry plugin adds Polylith specific tooling support to Poetry. The CLI adds tooling support for Polylith, and enables different kinds of Package & Dependency Management tools (such as Hatch and PDM).

The Hatch Build Hook adds support for building Libraries from Polylith with Hatch (and Rye, using hatchling as a build backend by default).

The PDM Build Hook for projects add support for building apps, services and libraries from Polylith using PDM. The PDM Build Hook for the workspace makes the virtual environment aware of the way Polylith organizes code (i.e. the bases and components folders).

Use cases

Microservices and apps 👍

The main use case is to support having one or more microservices (or apps) in a Monorepo, and share code between the services.

Libraries

Polylith for Python has support for building libraries to be published at PyPI, even if it isn't the main use case. More details about how to package libraries in the docs about Packaging & deploying.

✨ Examples ✨

There's example Polylith repositories for:

• Poetry
• Hatch
• PDM
• Rye
• Pants

The repositories are example Python setups of the Polylith Architecture. You will find examples of sharing code between different kind of projects, and developer tooling setup such as mypy and the venv.

Videos

• Python with the Polylith Architecture - an overview (about 15 minutes)
• Python Poetry Polylith Plugin - the tooling support & commands (about 13 minutes)
• The Developer Experience
  • developing a Dad Joke Service with Polylith (about 27 minutes)
  • the Polylith Dev Experience using Hatch and Rye (about 7 minutes)
• The standalone Polylith CLI - An intro to the polylith-cli - tooling support for Polylith with Python and Hatch

Talks

• Python Web Conference 2023 - Microservices, Monolith, Monorepos: the differences & how nicely Polylith solves the trade offs - A Fresh Take on Monorepos in Python (about 36 minutes)

Podcasts

• Talk Python To Me: Monorepos in Python

Articles

• The last Python Architecture you will ever need?
• A Fresh Take on Monorepos in Python
• A simple & scalable Python project structure
• Aws CDK App with polylith code architecture by Yoel Benítez Fonseca
• GCP Cloud Functions with Python and Polylith
• Python FastAPI Microservices with Polylith
• Kafka messaging with Python & Polylith
• Runestone Monorepo and Server Structure by Minh-Thao Pham
• Python Monorepo Visualization

Repo Visualization

A visualization of this repo (that itself is a Polylith workspace) using the poly info command.

python-polylith's Issues

poly libs: fails detecting library names with alias that differ

Describe the bug
A clear and concise description of what the bug is.
Example:
the library python-youtube is imported as pyyoutube. When running polyl libs, the library is reported as missing.

Possible solution: using difflib.

Example:

In [1]: import difflib

In [2]: difflib.get_close_matches("pyyoutube", ["requests", "python-youtube", "starlette", "fastapi"])
Out[2]: ['python-youtube']

v1.5.0 release is broken for Python < 3.12

Describe the bug
The latest version of polylith-cli released on Github has a syntax error.

To Reproduce
Download the latest source release, check line 16 in polylith_cli/polylith/check/report.py.

Does this project use CI/CD? If not, I would strongly encourage doing so to make releases more dependable. Due to the difference between the code in the repo and the code on Pypi I can only assume that this latest release was a mistake of some sort, but there is no audit trail showing how the release was built and uploaded.

poly libs is confused by some packages

Describe the bug
When I run poetry poly libs in my repo it complains for several of the projects about dateutil and humps and pkg_resources

Could not locate all libraries in book_server. Caused by missing dependencies?
🤔 dateutil, humps, pkg_resources

To Reproduce
run poetry poly libs

Expected behavior
The packages mentioned are in the pyproject.toml file so I would not expect there to be a problem

Desktop (please complete the following information):

• OS:
• Python version:
• Poetry version:

Additional context

humps is provided by the pyhumps package
dateutil is provided by the python-dateutil package

It may be that it is just confused because the package name doesn't match? Or maybe I'm misunderstanding something?

Poly check can't understand pillow dependency

Describe the bug
poetry poly check returns 🤔 Cannot locate PIL in my_package

To Reproduce
Steps to reproduce the behavior:

1. Have an import like from PIL import Image
2. Have a dependency like pillow = "^10.0.0"
3. Run poetry poly check

Error Log

poetry poly check -vvv                     
Loading configuration file .../pypoetry/config.toml
🤔 Cannot locate PIL in my_package
ℹ my_package is importing skimage, PIL, numpy
Expected behavior
PIL == pillow

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: M1 MacOS Ventura 13.5.2
• Python version: 3.9.16
• Poetry version: 1.6.1

Additional context
I thought it was interesting that it does find scikit-image even though its import is skimage. Though I guess that was fixed #83

Poly sync not working properly for projects in sub-directories

Describe the bug

• I'd like to be able to organize projects in sub-directories under the projects folder
• poetry poly info correctly finds all projects, even those in nested directories
• I can run poetry poly sync to have bricks added to pyproject.toml
• But the generated path is always ../../bases

To Reproduce
Steps to reproduce the behavior:

1. Create a project in a sub-directory under projects
2. Run poetry poly sync
3. Run poetry build-project within the respective project

Expected behavior
Relative paths to components/bases should be correct – f.x. ../../../bases for a project residing in projects/sub-directory/project_a

Desktop (please complete the following information):

• OS: Mac OS 14.3.1
• Python version: 3.12.2
• Poetry version: 1.8.2

The command "poly" does not exist.

Hi @DavidVujic

Just an installation issue I noticed on my machine (Mac M1) when setting up a new repo for polylith using your plugin.

I was creating a new repo with git init and poetry init. I already have the two plugins installed as per the installation instructions for my previous polylith repos. In particular the installation commands both suggest that the plugins are already installed. For example

> poetry self add poetry-multiproject-plugin  

The following packages are already present in the pyproject.toml and will be skipped:

  • poetry-multiproject-plugin

If you want to update it to the latest compatible version, you can use `poetry self update`.
If you prefer to upgrade it to the latest available version, you can use `poetry self add package@latest`.

Nothing to add.

The same goes for the other poetry plugin

So this should all work find right?

The issue

Well I was finding that the poly command was not registering with poetry. So if I try any of the poly commands I get an error

> poetry poly create ...

The command "poly" does not exist.

The fix

I found that the fix required me to run

poetry self update

This seemed to fix the issue and now I can create a workspace. I wanted to flag as this was a bit unexpected and I wasn't quite sure why it happened. If others have the same issue try poetry self update in the new project repo.

Matt

Open points to discussion

Hi @DavidVujic, I have been playing with python-polylith, and I am truly impressed. It is really close to how I think that ideal Python project would look like (at least for me). I have two points I would like to discuss:

1. Decoupling of base, components and external packages require some testing of the resulting artefact (whl or sdist). Because it is super simple to forget to include some brick or external package and find out that app is not working in deploy or even in prod. Since I am using Docker for all packages, I am trying to mitigate this risk in the Dockerfile:

ENV TEST_CODE="import sys\ntry: import namespace.package \nexcept Exception as e: sys.exit(e)"

COPY ./dist/$WHEEL /tmp/
RUN pip install --no-cache-dir --upgrade /tmp/$WHEEL --target ${TARGET} && \
    echo -e $TEST_CODE | python

But it is less than ideal. This is not a case for the compiled languages, I guess. Think the best way would be to running tests in the fresh venv where wheel artefact is installed. But again, the problem is with relative paths and specification of test that should be executed. Do you think this should be somehow supported by multiproject plugin? Or it is more up to CI/CD to manage this? I am more keen to CI/CD path, but it would be good to warn users that this might be tricky (especially with bigger projects with multiple dependencies).

2. In monorepos you usually have "universe of dependencies". If I get it, in polylith every project is totally independent to each other. So in one project you can use e.g. pandas==1.5.0 but in the other pandas==1.1.0. But it can also mean that same component in both the projects can behave differently (since it is using different external dependency). Any idea how to tackle this?

Thanks so much for this!!!

Can't trace an exception

Describe the bug
Whenever I run poetry poly libs, or poetry poly check I get this error: 'charmap' codec can't decode byte 0x98 in position 880: character maps to <undefined>
I have no idea what is wrong and where to look.

To Reproduce
I'm not sure this is reproducible.

Expected behavior
A more detailed description of the error, preferably with indication of which file is faulting.

Desktop (please complete the following information):

• OS: Win10
• Python version: 3.10.5
• Poetry version: 1.3.2

Additional context
Just moved a multifaceted project to polylith structure, and for starters assigned everything to one project . I started moving things around (that is, creating new bases/components and moving code there from legacy ones, then removing legacy ones). This wasn't happening at the beginning.

Aliases configuration file

Is your feature request related to a problem? Please describe.
The current way of providing aliases via the --alias flag does not provide any reliable way to share or replicate aliases between team members.

Describe the solution you'd like
My current workaround is to maintain the aliases in a file inside of the repo (.aliases). It would be good if the tool supported this natively.

Describe alternatives you've considered
I feel like it would be a good idea if the poly CLI either could accept a file containing newline-separated aliases directly, or if it had a configuration setting in a TOML file somewhere where one could set the aliases. I would like to hear your opinion on whether this configuration should happen in a separate file (i.e $USER/.poly.toml or $(pwd)/poly.toml) or whether it should be located in the development project pyproject file.

Hope you like my idea! I could implement this and PR, i would just need to hear your preference between the above alternatives.
P

In-brick pyproject.toml support

In Clojure polylith we have a separate deps.edn for each brick, which allows us to not repeat dependencies in dev deps.edn and project deps.edn — they are merged from all bricks' deps.edn file. Is it possible to have this in python polylith and multiproject?

For example I want to have a component log which depends on a logging library loguru, and this is specified in the in-component pyproject.toml. When I include this in a project, loguru will be automatically installed, like normal library dependency.

poly diff --short reports project folder name, and project name from the pyproject.toml

Describe the bug
The poetry poly diff --short command wrongfully lists two projects changed, when there is only one.

This happens when the actual folder name and the project name in the toml file is different. This can be reproduced in the actual python-polylith repo.

To Reproduce
Steps to reproduce the behavior:

1. Change code in one or more components of this repo.
2. run poetry poly diff --short
3. reported changes are for poetry_polylith_plugin,poetry-polylith-plugin

Expected behavior
Only one project should be reported. Preferably the project folder name, and not the name that can be found in the pyproject.toml file.

can't develop under editable mode in rye

when run rye run pytest the error messages are as following:
.venv/lib/python3.11/site-packages/profitcube/backtest/statistics.py:5: in <module> from profitcube.data.reader import get_jindata_reader .venv/lib/python3.11/site-packages/profitcube/data/reader/__init__.py:1: in <module> from .core import JinDataQuantReader, get_jindata_reader .venv/lib/python3.11/site-packages/profitcube/data/reader/core.py:5: in <module> from profitcube.data.ddb.ddb_client import create_session_from_env, create_session .venv/lib/python3.11/site-packages/profitcube/data/ddb/ddb_client.py:13: in <module> from profitcube.utils.singleton import SingletonClass .venv/lib/python3.11/site-packages/profitcube/utils/__init__.py:1: in <module> from profitcube.utils import core .venv/lib/python3.11/site-packages/profitcube/utils/core.py:12: in <module> from profitcube.data.reader import get_jindata_reader
In above error message, files's are in .venv dir which should be in the source code dir.
I found that in poetry mode the editable mode is OK. but in rye it's not.
I also find out:
If I use poetry install to update the .venv, then rye run pytest work well too. It's insteresting!

Thank you very much!

poly sync command misses out on some bricks

Describe the bug
When importing a brick like this from namespace import brick it looks like the sync command fails to identify the brick as missing.

However, it will locate and identify it as missing when importing it like this: from namespace.brick import the_function

Current behavior:
The sync command only checks one level. If more than one brick is added, that in turn imports another new brick, it will only identify the first brick. Running the command again will register the second one.

Expected behavior
Expecting that the sync command will recognize all ways of importing a Polylith brick.

there are some errors using rye

raise FileNotFoundError(msg)
FileNotFoundError: Forced include not found: /private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-via-sdist-fpf28p9w/profitcube-0.1.0/components/profitcube/abilities

ERROR Backend subprocess exited when trying to invoke build_wheel

Could you please help me, thank you!

Blank README.md in each new brick on create

Is your feature request related to a problem? Please describe.
Working through an example I am finding myself wanting to add a README.md to each component and base (and possibly also project) to make the documentation a bit easier.

Describe the solution you'd like
By default it would be nice if the new folder when you do poetry poly create ... added a README.md with the name of the brick as the title. It could be set up as a feature that can be switched on an off in the config.

Describe alternatives you've considered
I can of course do this manually. It's a useful prompt and feature though I think makes the scope of a brick clearer if you can document it in the directory itself.

Here is an example of a README.md I found myself writing for one of my bricks

# fetch_data component

This is a simple component that simulates fetching data from an external API. Specifically the current implementation here uses the Faker library to create some geo positioning data for each fetch call.

Linting on brick imports in VSCode

Is your feature request related to a problem? Please describe.
I'm loving exploring Polylith and the poetry tool and I think it has legs to be very helpful in the sort of systems I am building.

One small niggle when developing in VSCode is that the imports have linting errors in some circumstances.

The issue is that in this framework we do imports from the python package rather than from the directory structure in the brick structure. For example this is a directory structure where my workspace namespace is fruity

- bases
  - fruity 
    - grapefruit
- components
  - fruity
    - banana
    - pear

Let's say I want to use the banana component in my grapefruit base, the Polylith import I need to use is from the workspace namespace ...

# bases/grapefruit/core.py
from fruity.banana import banana
...

At the start of a project this generates a linting error as the fruity.banana path is not found in the directory structure. This makes sense as this import is not going to be a from a directory import but is going to be a from the built poetry package import.

When I made my correct packages imports in the master root pyproject.toml and did a poetry build it built the package and installed it. Now the linting issues disappear as the package is now installed.

Describe the solution you'd like

There are three issues here I think:

1. I could see this was going to happen but some people may be confused or do the absolute imports which would be incorrect. I think this could be fixed with some examples and documentation. I am happy to contribute to this if that is useful.

2. If I run tests or similar now, these imports will be be using what was last built in poetry build. That might be out of date with my current code and cause confusion. Doing a poetry build as part of your pre-run and pre-test activities will be important but may be easy to forget

3. If I want to import banana into pear, both of which are components. The correct import does in fact work. What is now unclear to me when I run my code is whether the poetry banana is being used or the directory banana version. They should be the same of course.

I think the most important thing is to find a way to include the poetry build into the workflow so that it stays up to date. I don't know how best to do this without getting in the way of the development workflow but wanted to raise the issue as there may be better ideas.

Describe alternatives you've considered
I also wondered if some sort of VSCode some sort of plugin could use the root pyproject.toml packages to suppress linting errors, but I think this is not the real issue. The real issue is that you need to have done a build.

H

Sync should respect --quiet flag

Describe the bug
poetry poly sync --quiet should not have any output.

To Reproduce
Steps to reproduce the behavior:

1. Open a terminal window
2. Go to root of the project
3. Run the command poetry poly sync --quiet
4. See output where there should be none.

Expected behavior
poetry poly sync --quiet should have no output.

Additional context
It doesn't appear that --quiet is respected by any poetry poly command, but to me sync is one of the most likely scenarios where someone would want this functionality.

[Feature] Optionally generate `py.typed` stub file, while creating bases & components

Is your feature request related to a problem? Please describe.

Pyright (strict) keeps complaining of missing typing stub file.

Describe the solution you'd like

poetry poly create ... can generate an empty py.typed stub file along side __init__.py and core.py.

Describe alternatives you've considered

This could be poly's default behaviour or invoked by a CLI option.

Add support for Python 3.12 when analyzing source code (poly libs)

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Add support for identifying the stdlib and usage of it in source code for Python 3.12.

Describe the solution you'd like
A clear and concise description of what you want to happen.
Add 3.12 specific stdlib news and removed in the libs component (the stdlib.py module).

poetry poly create base will allow names with dashes

Describe the bug
poly create base --name my-base-with-dashes will create (sim for components)

To Reproduce
as above
the imports in the generated py files are incorrect

Expected behavior
Sub '-'s for '_'s

poly delete

Is your feature request related to a problem? Please describe.

Poly command supports diff/info/libs/sync/check/create but not delete.

Describe the solution you'd like

Could delete be added?

Describe alternatives you've considered

For now I use rm -fr to remove directory. If that is correct solution maybe docs could be updated

poetry scripts break when building with --with-top-namespace

Describe the bug
When building a wheel build-project together with the --with-top-namespace argument, all the package includes in all the components/packages are modified so they are namespaced with name of the top-namespace argument. But the console scripts are not modified.

This breaks the scripts. In order to not break the scripts, they need to be set to whatever the top-namespace argument will be in advance, thus limiting the utility of dynamically modifying the namespace during build.

To Reproduce
Consider the following pyproject.toml

[tool.poetry]
name = "qaccess"
version = "0.0.0"

packages = [
    {include = "my_package", from = "src"},
]

[tool.poetry.dependencies]
python = ">=3.10,<3.13"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
test_entrypoint = "my_package:main"

When I build this project w/ poetry build-project --with-top-namespace=xyz, I get a wheel. All the libraries except for the console script are now under the xyz namespace.

When the wheel is unzipped, we have-

$ cat qaccess-0.0.0.dist-info/entry_points.txt 
[console_scripts]
test_entrypoint=my_package:main

When the wheel is installed and the test_entrypoint is called, it gives-

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:992, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1004, in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'my_package'

This happens because my_package is not inside the xyz namespace.

poly create: produces invalid Python code when the --name contains a path

Describe the bug
The command will create a component with imports, but with invalid Python, when using the create command like this: poetry poly create component --name=foo/bar

The output will be something like:

from top_ns.foo/bar import core

Described in #45

Expected behavior
The command should generate python files with correct import statements, such as:

from top_ns.foo.bar import core

Improve functionality of "components" (specification of external dependencies & wheel builds)

Thank you @DavidVujic for your very interesting talk at PyCon, was a joy to attend!

This project could be nice to organize code in a monorepo while also having a drastically lower onboarding cost compared to tools such as Bazel or Pants (although they try to solve slightly different things). Currently, using this tool is a bit blocked or would be cumbersome because of some design decisions about the "components" of a polylith repository:

Specification of external dependencies:

Say you'd have a bunch (aka dozens - hundreds) of components which all individually could have a bunch of external dependencies. These components now are consumed by a bunch of different projects. As far as I understand it, the projects would now to be responsible to specify the external dependencies of the components in their corresponding pyproject.toml's. This leads to a lot of repitition and also potential sources of errors.

Could there be a mechanism of the components to specify their external dependencies? This would move this responsibility away from the projects and also would have the great benefit that running poetry lock on a project which depends on components with conflicting external depencies would fail.

Wheel builds of dependencies:

Sometimes, even when there is already a nicely maintained monorepo, it can be very helpful to have the code of a component available as a wheel in an internal registry. As far as I know, there is no mechanism to build a wheel from a component, correct? This obviously would also need the capability of the component to be able to specify its external dependencies, otherwise one would have a wheel in the package registry for which it's unclear that it depends on.

Thank you very much again for this package! Please understand this purely as a question whether this could in principle be possible or would even be an intended use case for this project.

CircleCI: add steps to test code in different Python versions

Test the code in different Python versions, to make sure the syntax used works for Python 3.8 and above.

Libs command is not working as expected

Describe the bug
Hi David, I apologize for late answer. I am truly excited about the new feature of lib validation. I have just been playing with it, and it is magical, but I have some problems with it:

• if I am not mistaken, the lib command is comparing bricks dependencies with a root poetry.lock. I would expect to compare it to a project's poetry.lock. Think you can have third-party dependency in the root pyproject.toml, but not in the project's pyproject.toml -> built artefact does not contain all the required dependencies.
• unfortunately, it is not working with Poetry custom dependency groups. https://python-poetry.org/docs/master/managing-dependencies/#adding-a-dependency-to-a-group. The reason is probably this filter

  python-polylith/components/polylith/poetry/commands/libs.py

  Line 31 in 27c451e

  return {p.name for p in packages if p.category == "main"}

  . Seems that Poetry is setting category = "dev" if you add dependency to a group. Those groups should be most likely used for development, but they are really practical with a polylith since you do not have to always install all the dependencies.

To Reproduce
Steps to reproduce the behavior:

1. In python-polylith-example run poetry remove fastapi.
2. poetry poly libs is reporting that fastapi is missing, even though it is present in projects/my_fastapi_project/poetry.lock
3. Run the command poetry add fastapi --group fastapi
4. poetry poly libs is still reporting missing fastapi.

Expected behavior
See a description.

Desktop (please complete the following information):

• OS: MacOS
• Python version: 3.11.2
• Poetry version: 1.3.2

Exit codes in libs and check commands

Describe the bug
If I run poetry poly libs or poetry poly check and some libraries or bricks are missing, I would expect that last exit code is different to 0. That can help to e.g. CI/CD pipeline decide whether a project is OK or not.

Thanks a lot!

Testing

This is a great project @DavidVujic! I don't have a feature request, it's more of a question: would you be open to contributions to the unit tests?

`tdd` theme doesn't work with `pdm`

Describe the bug
It seems that the tdd theme seup is broken with pdm.

To Reproduce
Steps to reproduce the behavior:

mkdir foopdm && cd foopdm
pdm init -n --backend pdm-backend minimal
# add `pdm-polylith-workspace` hook
touch README.md
pdm add -d polylith-cli
pdm run poly create workspace --name poly
pdm run poly create component --name foo
pdm run poly sync
pdm install
pdm run python -c "from poly import foo"

Results in:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'poly'

Expected behavior
The same behavior as poetry:

mkdir foopoe && cd foopoe
poetry init
poetry poly create workspace --name poly
poetry poly create component --name foo
poetry poly sync
poetry install
poetry run python -c "from poly import foo"  # no error

Desktop (please complete the following information):

• OS: linux
• Python version: 3.12
• Poetry version: 1.7.1
• Pdm version: 2.12.3

verbose output for `poetry poly sync`

Is your feature request related to a problem? Please describe.
I just spent quite some time figuring out why poetry poly sync was adding dependencies of project_a to project_b even though project_b should not depend on them. The issue was in code, but poetry poly sync doesn't give any help in figuring out where a dependency is coming from. -vvv doesn't print anything more than without the verbose flag.

Describe the solution you'd like
It'd be nice if -v on poetry poly sync could print where a dependency is coming from. Alternatively poetry poly check could show this information. Pretty-printing brick_imports would probably already be enough.
As a start, it'd already help to just know what component a dependency is coming from. It'd be awesome if it could show code locations as well.

Additional context
For a bit more context, in code there was a migrations component that contains alembic migrations code. There is general migrations code, that doesn't depend on anything, and then different x_migrations subdirectories for migrations of different other components. The individual migrations depend on the components in question, but the generic migrations code does not. The individual migrations folder also don't contain an __init__.py, so they're not exported with the component itself, only referenced in alembic.ini configuration. So there actually was no direct dependency on the code in question. But since poetry poly uses glob('*.py'), it still found the python files and calculated imports from there. This made debugging the issue harder.

That component looks like

|- __init__.py
|- alembic.ini
|- core.py
|- env.py 
L_ a_migrations
  L_ migrations
    |- __init__.py
    |- env.py   <-- this depends on a component only needed for project_a
    L_ versions
        |- <migrationsfile>.py

I didn't want to break this up into further components to not clutter the hierarchy more than needed, and didn't expect poly to pick up on the env.py in the subfolder (since there's no __init__.py).

But I don't think this necessarily needs to be changed, technically the migrations subfolder code does depend on the project_a component. Though it is more of an indirect dependency. I wonder if having a way to ignore a dependency with something like a # poly: ignore-import comment, could be a helpful addition as well. It might help in some cases when migrating from non-polylith code to polylith.

poetry poly libs command rendered strange in Windows PowerShell

Describe the bug
A clear and concise description of what the bug is.
Described in #73

The poetry poly libs command sometimes render output in a confusing way on Windows and in PowerShell, possibly in the PyCharm terminal.

Investigate if there is anything that can be solved. Suggestion: a later version of rich?

adding a poly deps command

Currently working on implementing a Python version of the poly deps command, in a similar way as in the Clojure version.

This command will visualize the bricks and the dependencies between them.

Idea: the top level pyproject.toml file in sync with the individual projects

Is your feature request related to a problem? Please describe.
Keep the top level pyproject.toml file in sync with the individual projects.

Describe the solution you'd like
Suggestions:
This could be solved by a new command, or a combination of existing commands.

Additional context
Idea originates from DavidVujic/python-polylith-docs#1

poly diff reports changes on non-brick code

Describe the bug
A clear and concise description of what the bug is.
The test folder (when structured in the same way as the bricks) is reported as changed in the view when running poetry poly diff.

This can be confusing, since it is not a brick that is used in the projects.

To Reproduce
Steps to reproduce the behavior:

1. Make changes in a unit test.
2. Run poetry poly diff
3. the test folder is reported in the table view.

Expected behavior
The poly diff command should only highlight changes in bricks and map them to usage in the projects.

poly info: exclude cache files from list of bricks

Describe the bug
Accidentally listing a __pycache__ folder among bricks, when running the poetry poly info command.

To Reproduce
Steps to reproduce the behavior:

1. put a __pycache__ folder in components/<my-top-namespace> folder
2. run poetry poly info

Expected behavior
The command should not list cache files or folders.

Screenshots

Commands "check" and "libs" confused by __future__

Describe the bug
Running the poetry poly libs or poetry poly check produces an error when importing from the __future__ module.

To Reproduce
Steps to reproduce the behavior:

1. Create an empty poetry project and a poly workspace,
2. Create a component with poetry poly create component,
3. Add import line from __future__ import annotations to the core.py file,
4. Run poetry poly libs command.

Small reproduction repository is available here.

Error Log

# poetry poly libs   

Libraries summary for development

Libraries used in bases: 0
libraries used in components: 1

  brick   libraries  
 ────────────────────
  test    __future__

Could not locate all libraries in poetry-polylith-repro. Caused by missing dependencies?
🤔 __future__

Expected behavior
The __future__ module is not a third-party module and should be treated the same as any built-in modules.

Desktop (please complete the following information):

• OS: Windows
• Python version: 3.11
• Poetry version: 1.5.0.dev0

Rename the repo to Python-Polylith

This repo will be a monorepo containing more than the Poetry Plugin. That's why I think it will make sense renaming the repo.

poly info command does not list components under lower level namespaces

Is your feature request related to a problem? Please describe.
Hi @DavidVujic, first of all this is such a great project, this way of organising python code makes so much sense in many situations.

The issue I came across is related to using multiple namespaces, under the top level namespace (or org_name) as mentioned in #45. This is something I think could be very useful for my use case.

But, when creating something like the example below...

├── README.md
├── bases
├── components
│   └── org_name
│       ├── namespace_1
│       │   ├── component_1
│       │   │   ├── __init__.py
│       │   │   └── core.py
│       │   ├── component_2
│       │   │   ├── __init__.py
│       │   │   └── core.py
│       │   └── component_3
│       │       ├── __init__.py
│       │       └── core.py
│       └── namespace_2
│           ├── component_1
│           │   ├── __init__.py
│           │   └── core.py
│           └── component_2
│               ├── __init__.py
│               └── core.py
├── development
├── poetry.lock
├── projects
├── pyproject.toml

Using poetry poly create component --name=namespace_1/component_1... And then running poetry poly info, I get:

Workspace summary

projects: 0
components: 2
bases: 0
development: 1

  brick                                                                       development
 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  namespace_1                                                                      -
  namespace_2                                                                      -

Describe the solution you'd like
Would be nice poly info could list all the components detected under "sub namespaces". Same would apply to bases as well. For example, would like a summary something like the one below. Hopefully this makes sense?

Workspace summary

projects: 0
components: 5
bases: 0
development: 1

  brick                                                                       development
 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  namespace_1                                                                      -
      component_1                                                                  -
      component_2                                                                  -
      component_3                                                                  -
  namespace_2                                                                      -
      component_1                                                                  -
      component_2                                                                  -

Additional context
Following on #45.

How to handle dependecies?

I have seen the Polylith concept before but have not used it

Seeing you had this Poetry plugin and I had a new monorepo to structure I decided to give it a go

I am a bit confused about how to handle dependencies

I can see there is a root pyproject.toml and then individual ones for each project

Should I poetry add all deps at the root level and then use poetry poly sync to let the plugin apply only relevant deps to the project pyproject.toml files? i.e. I shouldn't manage the project files manually myself at all?

Support for different git tag patterns (stable-*, release-* etc.)

It would be great to be able to add multiple git tag patterns and check for changes since each of then (like "release-*" tag). At the moment I have 6 docker images recompiling and pushing to repo on each change. Something like this would let me only compile what changed since last release:

Maybe instead of this:

[tool.polylith]
namespace = "ufy"
git_tag_pattern = "stable-*"

We could have:

[tool.polylith.git_tag_patterns]
stable="stable-*"
release="release-[0-9][0-9][0-9][0-9]-*"

And then have poetry poly diff --since release?

This can even be a non-breaking change. The user can still use git_tag_pattern but adding the separate section would override it.

Relase version 1 of the Poetry Plugin

Releasing a new version containing the new source code structure and supporting Poetry 1.2.

poly info command: sort order of projects

Describe the bug
The ordering of projects should be alphabetically, when running poetry poly info in a workspace with several projects.

To Reproduce
Steps to reproduce the behavior:

1. run poetry poly info
2. The table with bricks and projects will display projects in a seemingly random order.

Expected behavior
The list of projects should be sorted alphabetically

Screenshots

Allow using `tests`

Thank you for creating these plugins. I've become not only a fan but also an advocate of this. There's a bit of learning curve for sure but after that I feel I've a solution for better managing the structure of python mono-repos.

Is your feature request related to a problem? Please describe.

Please allow tests along with test where both are subjective but tests is more aligned when including unit-tests, functional and integ tests in the repo.

poly check command

A poly check command to analyze the code in projects: check for missing dependencies and packages.

Suggested approach: this feature can be built on top of the newly added check-project command of the Multiproject Plugin.

There's more info and suggestions in #41

Multiple namespaces?

Hello -- thanks for putting together the polylith repo here as a reference. I started to use it for a small project to create a client library around an API, and also create a singer tap for that api.

One thing I found myself wanting to do is create more than one namespace underneath the components/ and bases/ directories. I'm not sure if this is supported or encouraged at this time, both because the workspace.toml at the top level looks like it wants a single value for tool.polylith.namespace, and if I try to run poetry poly create workspace --name my_new_namespace --theme loose again, I get the following error: [Errno 17] File exists: '/Users/britz/myprojects/simple-garden/bases'.

Is creating multiple namespaces discouraged? Am I thinking about something the wrong way when I want to be able to do that?

Poly diff to list changed bases/components for CI testing

Is your feature request related to a problem? Please describe.
poetry poly diff -s is an amazing tool for a CI environment, but when working in a large project where tests for components/bases can become rather extensive, it would be nice to be able to get a simple list of those which were updated as well. (Rather than having to run all tests)

Describe the solution you'd like
Something like:
poetry poly diff bases --> base1 base3
poetry poly diff components --> component2 component4

Describe alternatives you've considered
Maybe -s flag, and/or bases/components as flags rather than an argument.

Nice to have would be a list of test paths, so you could just run :

poetry run pytest `poetry poly diff --test-paths`

Additional context

# Test projects by using their self-defined dependencies.
for val in `poetry poly diff -s`
do
    poetry install --sync -C ./projects/${val}
    poetry install --only=dev
    for base in `poetry poly diff bases -C ./projects/${val}`
    do
        poetry run pytest ./test/proj/bases/${base}
    done
done

cd test/proj/components && poetry run pytest `poetry poly diff components`

Tooling support for third-party dependencies (i.e. "libraries" from PyPI)

Is your feature request related to a problem? Please describe.
Investigate and try out ways to make it simple to add third-party dependencies used by components and needed to build artifacts from projects.

Related to #41 and #6

PDM support

Since PDM seems to have taken off lately and also provides a cleaner integration with pyproject.toml (PEP 517, 621), is there any plan about adding Polylithic support for PDM?

error in rye build --whell

[project]
name = "profitcube"
version = "0.1.0"
description = "jinniuai's investment power store"
authors = [
    { name = "markqiu", email = "[email protected]" }
]
dependencies = [
    "jinja2>=3.1.3",
    "typer>=0.12.2",
    "rich>=13.7.1",
    "numpy>=1.26.4",
    "pydantic>=2.6.4",
    "loguru>=0.7.2",
    "pandas>=2.2.1",
    "pandera>=0.18.3",
    "dolphindb>=3.0.0.0",
    "mootdx>=0.11.4",
    "openpyxl>=3.1.2",
    "mytt>=2.9.3",
    "python-dotenv>=1.0.1",
    "simple-toml-configurator>=1.2.2",
    "croniter>=2.0.3",
    "statsmodels>=0.14.1",
    "pendulum>=3.0.0",
    "pyfinance>=1.3.0",
    "pydantic-extra-types>=2.6.0",
    "backtrader @ https://pypi.jinniuai.com/packages/backtrader/backtrader-1.9.76.123.2.tar.gz",
    "tomli-w>=1.0.0",
    "dynaconf>=3.2.5",
]
readme = "README.md"
requires-python = ">= 3.11"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.rye]
managed = true
dev-dependencies = [
    "polylith-cli>1.5.0",
    "pytest>=8.1.1",
    "ruff>=0.3.7",
    "mypy>=1.9.0",
]


[tool.hatch.build]
dev-mode-dirs = ["components", "bases", "development", "."]


[tool.polylith.bricks]
"bases/profitcube/cli" = "profitcube/cli"
"bases/profitcube/config" = "profitcube/config"
"components/profitcube/core" = "profitcube/core"
"components/profitcube/abilities" = "profitcube/abilities"
"components/profitcube/backtest" = "profitcube/backtest"
"components/profitcube/resources" = "profitcube/resources"
"components/profitcube/utils" = "profitcube/utils"
"components/profitcube/data" = "profitcube/data"
"components/profitcube/可视化" = "profitcube/可视化"
"components/profitcube/services" = "profitcube/services"

[tool.hatch.metadata]
allow-direct-references = true

[tool.ruff]
exclude = [
  ".git",
  ".github",
  "__pycache__",
  ".mypy_cache",
  ".ruff_cache",
  "dist",
  ".venv",
  "./development/*.py",
]
line-length = 120

[tool.ruff.lint]
select = ["E4", "E7", "E9", "F", "I"]

[tool.rye.scripts]
ability-create = { call = "profitcube.cli:app" }

reproduce command:
rye build --wheel --sdist or rye build --whell
get the following errors:

building profitcube
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for sdist...
* Building sdist...
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for wheel...
* Building wheel...
Traceback (most recent call last):
  File "/Users/qiucheng/.rye/self/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/Users/qiucheng/.rye/self/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiucheng/.rye/self/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/build.py", line 58, in build_wheel
    return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 155, in build
    artifact = version_api[version](directory, **build_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 455, in build_standard
    for included_file in self.recurse_included_files():
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 176, in recurse_included_files
    yield from self.recurse_selected_project_files()
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 180, in recurse_selected_project_files
    if self.config.only_include:
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/config.py", line 806, in only_include
    only_include = only_include_config.get('only-include', self.default_only_include()) or self.packages
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 240, in default_only_include
    return self.default_file_selection_options.only_include
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiucheng/.rye/py/[email protected]/lib/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/rc/m01mh1mn3m99t4879pf_wbk40000gn/T/build-env-qar9yi40/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 228, in default_file_selection_options
    raise ValueError(message)
ValueError: Unable to determine which files to ship inside the wheel using the following heuristics: https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

The most likely cause of this is that there is no directory that matches the name of your project (profitcube).

At least one file selection option must be defined in the `tool.hatch.build.targets.wheel` table, see: https://hatch.pypa.io/latest/config/build/

As an example, if you intend to ship a directory named `foo` that resides within a `src` directory located at the root of your project, you can define the following:

[tool.hatch.build.targets.wheel]
packages = ["src/foo"]

ERROR Backend subprocess exited when trying to invoke build_wheel

By the way: rye build --sdist can be successful completed.

Any example to deploy a project using docker?

The Lego bricks idea behind Poly lith is awesome. Can anyone provide an example that deploys a project using Docker?