stefanbras / polugins Goto Github PK

It can either be done inline as it is now, with actual python imports to specify which namespaces to add OR it could be specified externally in pyproject.toml or something, similar to how the entry points are collected. (Entry points could be used in the same way as external packages, but that can be a bit annoying. See the the README.)

On one hand, it's annoying to have to use metadata if you only care about registering the namespaces and not typing. On the other hand, the type generation will be way more cumbersome if we have to try and parse arbitrary python code to resolve imports.

Right now we can't generate types for namespaces registered through either an import path as string or directly (because we'd have to scan through the code and parse the AST and all kind of too-much-work stuff)

However we could maybe just dump the types at runtime, if for example an ENV var was set. Then you could set it as a flag when running pytest or something, or have a standalone script that just registers namespaces.

My idea is to just create a simple CLI tool that either generates or extract the original polars type stubs and then just adds namespaces to them with some AST magic.

Similar to this: https://stackoverflow.com/questions/75835580/how-to-configure-mypy-for-polars-api-extension)

Notably the limitation the only namespaces referenced through .toml files, env vars or package metadata is collected.

We're using pip index versions polars to get the available versions when generating the type stubs, but this does not show yanked versions.

We probably still want those.

If someone generates types but nothing is added to them, it's probably because they're not correctly registering.

Various test improvements

• Nox
• Coverage
• CLI testing
• Polars api extension context - there's no way to unregister namespaces, so every test is dirty

If a namespace registers itself, it will give the user warning. Since the namespaces often are external code you can't control this and it's then annoying to get warnings about it

It's failing because I forgot to update a lock file (and maybe because poetry had a new version published?)

hey

sorry for the delay in trying this - just gave it a go, and got

ValueError: Type stubs for version 0.20.3 does not exist. This is usually because the version has been yanked. Feel free to create an issue if you want to generate types for this version

I ran polugins stubs from the root of polars-xdt

I apparently broke it when splitting up the package.

The dependencies usually use the latest version, but can be loosened.

• importlib-metadata = "^6.6.0"
• importlib-resources = "^5.12.0"
• typing-extensions = "^4.6.3"

is there a way to keep up with the weekly releases of polars ;)

olugins_type_gen.cli.MissingVersionException: Type stubs for version 0.20.10 does not exist. This is usually because the version has been yanked or because it's new. Feel free to create an issue if you want types for this version.

• Release
• Test
• Lint
• Make PR when new polars version is released

I've tried making a Python file t.py with

from polugins import register_namespaces
import polars as pl
from polars_xdt import ExprXDTNamespace

register_namespaces(
    expression_namespaces={
        'xdt': ExprXDTNamespace,
    },
  )

pl.col('a').xdt.format_localized('%a')

If I type check it, I get:

$ mypy t.py
t.py:1: error: Skipping analyzing "polugins": module is installed, but missing library stubs or py.typed marker  [import-untyped]
t.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
t.py:11: error: "Expr" has no attribute "xdt"; maybe "dt"?  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)

If I run it, I get:

$ python  t.py
/home/marcogorelli/tmp/.venv/lib/python3.10/site-packages/polugins/_types.py:35: UserWarning: Overriding existing custom namespace 'xdt' (on 'Expr')
  return pl.api.register_expr_namespace(name)(namespace)  # type: ignore

Am I doing something wrong?

Right now I just put in imports in the type stubs, but these are only there at type checking-time and not at run time.

And give a warning about the types maybe mismatching at that they should update that package.

I'm still not sure what the best interface is, so lets just put in lots of ways and see what sticks.

We could have a env var structure like

POLUGINS_<class>_<name> = <module_path>:<NamespaceCLass>

For example

POLUGINS_lazyframe_custom = mypackage.submodule:CustomNameSpace

Would add the namespace CustomNameSpace under pl.LazyFrame.custom.

Then register_namespaces should have a register_env_vars: bool too

And when I do so use argparse instead of the current poor per... minimal CLI implementation.

Would be nice if the PR bot for type stubs included a diff between the pr and the latest version.