consider a client connection as an alternative to being an extension about postgresml

[montanalow]

I agree, it'd be great if PostgresML could run in a Python virtual environment, distinct from the OS Python. Unfortunately, Postgres doesn't currently provide a mechanism for individual Python extensions to specify a Python runtime, although the version of Python used can be configured at build time for Postgres.

There are a few ways we could avoid dependency on the OS system Python (listed in my order of preference):

1. Implement venv (or similar) support in Postgres core.
2. Reimplement PostgresML as a C (abi compatible) extension independent of Python.
3. Package PostgresML as a Python process running in a container, and have a non Python extension call out to it.

Option 1 is great, because all Postgres extensions would benefit not just PostgresML, but I think this is a Postgres issue, not a PostgresML issue.

Option 2 is something we're considering, but the primary motivation will be around resource usage from real world benchmarks. I think we'll likely do this one feature or bottleneck at a time on a case by case basis if and when it makes sense. Completely removing all Python dependency from the extension would be more of a rewrite or port, which I'm not keen to commit to all at once.

Option 3 seems like something we'd consider if the ML part of PostgresML became significantly more important than the Postgres part, and we wanted a cleaner separation. Currently, I think Postgres is an integral part of PostgresML, and will be for the foreseeable future, so this is not something we're pursuing.

I'm going to close this issue, because we have no immediate plan to start working toward it's resolution, but would appreciate continued discussion if the community has additional thoughts.

[fdr]

I don't think you addressed my proposal as I intended to convey it.

This project already does more or less what is necessary, via ./pgml-extension/tests/plpy.py, for testing.

What you would have is a variant that would have some module state to be able to connect to a postgres instance and adapt any result sets so the plpy and, say, psychopg2 (or whatever) result sets had the same interface.

As far as I can tell, the extension as such is limited to fairly simple queries and result sets, making both my proposal, and the testing methodology, feasible.