Comments (11)
@killme2008 @discord9 I remember that in an offline discussion, it's thrown that we might finally use PyO3 only.
Is there any concrete reason and plan? IIRC we initially add RustPython to decouple the dependency to system python interpreter. But now we think it's a fair requirement?
from greptimedb.
@killme2008 @discord9 I remember that in an offline discussion, it's thrown that we might finally use PyO3 only.
Is there any concrete reason and plan? IIRC we initially add RustPython to decouple the dependency to system python interpreter. But now we think it's a fair requirement?
The main reason is that RustPython is far from production currently, and Python and PyO3 are developing rapidly.
Using PyO3 and native Python, the user can use all the community resources such as open-source projects and documents, etc. and don't have to worry about any compatibility issues.
from greptimedb.
@killme2008 can we first dropping the code related to RustPython and recommend to use the PyO3 backend version?
For PyO3 backend version, the major challenge is supported range of Python version. @messense do you have some inputs in this topic? We're a binary rather than a lib, so we may not release multiple artifacts to cover the py versions like OpenDAL. Instead, we're trying to use abi3 to expand the supported versions range. But we still lack knowledge in how to move forward in this direction.
from greptimedb.
Embedding python interpreter is an underexplored area in pyo3, pyo3-ffi
does not support dynamic loading yet: PyO3/pyo3#2668.
An alternative could be to build your own
.so
which implements the Python plugin engine (and uses PyO3's existing link model), and then your binary could dynamically load that.so
if a Python plugin is requested.
Will this work for greptimedb?
from greptimedb.
@messense Thanks for your information. Let me think of it ...
Cross-ref: https://pyo3.rs/v0.21.0/building-and-distribution
from greptimedb.
@messense we may not use embedded python interpreter
, but to ensure that the binary can work with multiple python versions well. We currently configure it as:
greptimedb/src/script/Cargo.toml
Line 57 in d5a948a
But yet to test against multiple Python version.
from greptimedb.
@killme2008 can we first dropping the code related to RustPython and recommend to use the PyO3 backend version?
Found that we currently depend on RustPython's AST parsing function to extra metadata from decorators:
@coprocessor(args=[], returns=["value"], backend="pyo3")
def test_numpy() -> vector[i64]:
try:
import numpy as np
import pyarrow as pa
except ImportError as e:
# Python didn't have numpy or pyarrow
print("Warning: no pyarrow or numpy found in current python", e)
return vector([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,])
from greptime import vector
v = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,])
v = pa.array(v)
v = vector.from_pyarrow(v)
v = vector.from_numpy(v.numpy())
v = v.to_pyarrow()
v = v.to_numpy()
v = vector.from_numpy(v)
return v
from greptimedb.
@killme2008 can we first dropping the code related to RustPython and recommend to use the PyO3 backend version?
Found that we currently depend on RustPython's AST parsing function to extra metadata from decorators:
@coprocessor(args=[], returns=["value"], backend="pyo3") def test_numpy() -> vector[i64]: try: import numpy as np import pyarrow as pa except ImportError as e: # Python didn't have numpy or pyarrow print("Warning: no pyarrow or numpy found in current python", e) return vector([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,]) from greptime import vector v = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,]) v = pa.array(v) v = vector.from_pyarrow(v) v = vector.from_numpy(v.numpy()) v = v.to_pyarrow() v = v.to_numpy() v = vector.from_numpy(v) return v
RustPython's parser is now a standalone crate, which is battle-tested in Real Python Usecase like Python Lint Puff
so it's intended to be kept, while the RustPython Runtime can be dropped, might consider first disable RustPython backend in code, and clean up code later then.
P.S. I mean ruff
sorry
from greptimedb.
No. ruff
seems to write their own crate.
The API of RustPython has changed a lot, ref - RustPython/Parser#119.
I generally doubt if such source edit approach the way to go. Maybe a Python UDF like PL/Python or a Python worker that can connect to the SQL server a good alternative.
from greptimedb.
No.
ruff
seems to write their own crate.The API of RustPython has changed a lot, ref - RustPython/Parser#119.
I generally doubt if such source edit approach the way to go. Maybe a Python UDF like PL/Python or a Python worker that can connect to the SQL server a good alternative.
https://github.com/astral-sh/ruff/blob/f779babc5f725ccfe9c4e08b26d1abe54166619d/CONTRIBUTING.md?plain=1#L641
This one though it's also a fork, and you are very likely correct, the only advantages now by having a embedded interpreter is just saving serde and network cost, which I didn't benchmark, but could very well be insignificant since pyhton itself is already slow enough
from greptimedb.
Supersedes by #3777.
from greptimedb.
Related Issues (20)
- Workflow run 'report-ci-status' failed HOT 1
- Metainfo of the inverted index may overflow, causing the index to become unavailable HOT 2
- Workflow run 'Nightly CI' failed HOT 1
- Configure CI to run fuzz tests in distributed mode with MinIO HOT 2
- improve observability for procedure
- Fuzz insert test failed HOT 10
- Fuzz alter logical table failed HOT 1
- panic when executing a flow job
- Write requests may stall too long when the write buffer is full HOT 1
- Make auto-create/alter table through gRPC put request optional
- Support "show status" HOT 2
- Support to drop multiple tables HOT 1
- failed to write data using prometheus remote_write HOT 2
- Implement an information schema table to inspect flow node's internal state. HOT 2
- Workflow run 'Release' failed HOT 3
- [Tracking issue] open regions in batches
- Checking if database is used currently when dropping database HOT 1
- Workflow run 'Release' failed HOT 1
- Workflow run 'Release' failed HOT 1
- Metadata upgrade tool
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from greptimedb.