kyligence / kylinpy

Apache Kylin Python Client Library

Apache Kylin Python Client Library

Apache Kylin Python Client Library is a python-based Apache Kylin client.

Any application that uses SQLAlchemy can now query Apache Kylin with this Apache Kylin dialect installed.

Installation

The easiest way to install Apache Kylin Python Client Library is to use pip:

pip install kylinpy

alternative, install by offline tarball package:

# download from https://pypi.org/project/kylinpy/#files
pip install kylinpy-<version>.tar.gz

Apache Kylin dialect for SQLAlchemy

Any application that uses SQLAlchemy can now query Apache Kylin with this Apache Kylin dialect installed.

You may use below template to build DSN to connect Apache Kylin:

kylin://<username>:<password>@<hostname>:<port>/<project>?<param1>=<value1>&<param2>=<value2>

DSN Fields	Default	Allow omitted
username	null	false
-----------------------------	-----------------	-----------------------
password	null	false
-----------------------------	-----------------	-----------------------
hostname	null	false
-----------------------------	-----------------	-----------------------
port	7070	true
-----------------------------	-----------------	-----------------------
project	null	false

DSN query string config is as follows

Fields	Default Value	Optional value	Description
is_ssl	0	0|1	Is the Kylin cluster enabled for https
-----------	------------------	-----------------	------------------
prefix	/kylin/api	string	Kylin cluster API prefix
-----------	------------------	-----------------	------------------
timeout	30	integer > 0	HTTP timeout with Kylin cluster
-----------	------------------	-----------------	------------------
version	v1	v1v4	v1 == using Apache Kylin API v2 == using Kyligence Enterprise 3 API v4 == using Kyligence Enterprise 4 API
-----------	------------------	-----------------	------------------
is_pushdown	0	0|1	If enabled, viewing a project table will use the hive source table
-----------	------------------	-----------------	------------------
is_debug	0	0|1	Whether to enable debug mode

From SQLAlchemy access Apache Kylin

$ python
>>> import sqlalchemy as sa
>>> kylin_engine = sa.create_engine('kylin://ADMIN:KYLIN@sandbox/learn_kylin?timeout=60&is_debug=1')
>>> results = kylin_engine.execute('SELECT count(*) FROM KYLIN_SALES')
>>> [e for e in results]
[(4953,)]
>>> kylin_engine.table_names()
[u'KYLIN_ACCOUNT',
 u'KYLIN_CAL_DT',
 u'KYLIN_CATEGORY_GROUPINGS',
 u'KYLIN_COUNTRY',
 u'KYLIN_SALES',
 u'KYLIN_STREAMING_TABLE']

From Pandas access Apache Kylin

$ python
 >>> import sqlalchemy as sa
 >>> import pandas as pd
 >>> kylin_engine = sa.create_engine('kylin://ADMIN:KYLIN@sandbox/learn_kylin?timeout=60&is_debug=1')
 >>> sql = 'select * from kylin_sales limit 10'
 >>> pd.read_sql(sql, kylin_engine)

From Superset access Apache Kylin

Now you can configure the DSN in your application to establish the connection with Apache Kylin.

For example, you may install Apache Kylin Python Client Library in your Superset environment and configure connection to Apache Kylin in Superset

then you may be able to query Apache Kylin one table at a time from Superset

you may also be able to query detail data

Alternatively, you may also be able to query multiple tables from Apache Kylin by using SQL Lab in Superset.

kylinpy's Issues

'NoneType' object is not callable error with superset = 0.29.0rc7

2019-04-04 09:25:34,763:ERROR:root:'NoneType' object is not callable
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/superset/sql_lab.py", line 183, in execute_sql
db_engine_spec.execute(cursor, query.executed_sql, async_=True)
File "/usr/local/lib/python3.6/site-packages/superset/db_engine_specs.py", line 375, in execute
cursor.execute(query)
File "/home/superset/.local/lib/python3.6/site-packages/kylinpy/kylindb.py", line 48, in execute
] for row in resp['results']]
File "/home/superset/.local/lib/python3.6/site-packages/kylinpy/kylindb.py", line 48, in
] for row in resp['results']]
File "/home/superset/.local/lib/python3.6/site-packages/kylinpy/kylindb.py", line 47, in
for (col, cell) in enumerate(row)
File "/home/superset/.local/lib/python3.6/site-packages/kylinpy/utils/kylin_types.py", line 39, in kylin_to_python
return KylinType.get(str(_type).upper())(s) if s else s
TypeError: 'NoneType' object is not callable

Kylinpy can't work

after type "pip install --upgrade kylinpy",the CRT display

root@dp-kf-graphdb-2:~/kylinpy# pip install --upgrade kylinpy
Downloading/unpacking kylinpy from https://files.pythonhosted.org/packages/68/16/343f8e07f89b21967f1b2afada8f106e4faa0602226e1aeac6faee28f276/kylinpy-2.8.3.tar.gz#sha256=fda35dbbc2ac4e26ae02e5566a39684b05827a666c83613c58da384f7ed06024
Downloading kylinpy-2.8.3.tar.gz
Running setup.py (path:/tmp/pip_build_root/kylinpy/setup.py) egg_info for package kylinpy
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
warnings.warn(msg)

warning: manifest_maker: MANIFEST.in, line 2: 'recursive-exclude' expects <dir> <pattern1> <pattern2> ...

Installing collected packages: kylinpy
Found existing installation: kylinpy 0.0.999dev
Can't uninstall 'kylinpy'. No files were found to uninstall.
Running setup.py install for kylinpy
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
warnings.warn(msg)

warning: manifest_maker: MANIFEST.in, line 2: 'recursive-exclude' expects <dir> <pattern1> <pattern2> ...

Successfully installed kylinpy
Cleaning up...

means install successful ,but type the command "kylinpy",

error display
:root@dp-kf-graphdb-2:~/kylinpy# kylinpy
kylinpy: command not found

no idea the reason ,the document dosen't show more infro to debug

Create Model feature is needed

• Desc:

In the support areas, we have to do lots of "reproduce" work, which requires us to rebuild the model and cube with our customer's metadata. Always, these steps waste lots of time. so, we need a way to create model and cube through command line.

• Typical Scenario:

we get the Model_desc/kylin_sales_model.json, generated by the command
$KYLIN_HOME/bin/metastore.sh backup
and we want to use the kylinpy CLI like this:
kylinpy -h kap.kapdemo.com -P 7070 -u ADMIN -p KYLIN --project learn_kylin --api1 --debug -f Model_desc/kylin_sales_model.json model
and as a result, we want a new model built in kylin. YOU MAY THINK THIS IS AN ANOTHER WAY TO CLONE A MODEL

control ACL feature is needed

Kylin & KAP's ACL part is important, and should be considered to add into kylinpy.

• Column ACL REST API

• Row ACL REST API

• TABLE ACL REST API

• PROJECT ACL REST API

Typically, it should support both batch processing & single processing.
For the batch processing, it should use a json file as the input, which should have the same structure with the json file that returns by the "GET" method.

TypeError while connecting superset and kylin

my kylin is 3.0, superset is 0.28.1, python is 3.7, kylinpy is 2.4.1
when I test connection, it gives exceptions like:
kylinpy.client.exceptions.InternalServerError:
TypeError: str returned non-string (type NoneType)

Get cube structure failure

当cube内包含相同名表, 而schemab不同时, 获取cube结构信息会失败

The result set size is default to 50000 and can not be changed.

The 'execute' method of the 'Cursor' class passes only query but not params to the 'query' method of the 'Project' class. The query method set 'limit' default to 50000. The result set size will be less than or equal to 50000. This is not we want when the result set size is larger than 50000.

Adding https support for client

Expose Rowkey dimensions in cube describe

Issue with Kylin Integration with superset

am trying to configure Kylin connection with Superset. I have configured kylin package and my connection seems to be ok. However, it doesn't show me any tables, at the bottom of the page.
I checked the logs and it tries to call the below Rest API -
############################################################################
method: GET
url: http://XXXX.XXXX:80/kylin/api/tables_and_columns?project=projectName
headers: {'User-Agent': 'Kylin Python Client', 'Authorization': 'Basic someEncodedPassword'}
body: None
############################################################################
I have tried the above url with Postman, and it returns me table and column names.

Moreover, if I go to Sql Editor in Sql Lab it shows me Database name, schema & Tables. But as soon as I click on any of the table, it gives an error " startswith first arg must be bytes or a tuple of bytes, not str"

Attached screenshots for same

Not sure, if its related to kylin package or with Superset. Any suggestions will be appreciated

got kylinpy.exceptions.KylinQueryError when use Query.first or Query.filter

First i got an engine and a session form kylin and create table like this class Date(Base): __table__ = Table('DIM_DATE', metadata, schema='WEB_AD', autoload=True) __mapper_args__ = { 'primary_key': [Table('DIM_DATE', metadata, schema='WEB_AD', autoload=True).c.DATE_KEY] }.When i start a query from this table if i use operator like first() or filter(), i got an error like this kylinpy.exceptions.KylinQueryError: Encountered "= %" at line 3, column 38. Was expecting one of: <EOF> "ORDER" ... "LIMIT" ... "OFFSET" ... "FETCH" ... "GROUP" ...
There are some code i used result = session.query(Date.YEAR,Date.MONTH).filter(Date.DATE_KEY == 20161231).first()

Unable to connect Apache kylin to superset

I'm running Superset on docker on windows, and kylin is runnning on another machine both are in same virtual network. I have installed kylinpy on docker container to enable connection from kylin on superset but I'm unable to connect. I'm getting following error:
(builtins.NoneType) None [SQL: _compose_select_body() takes 7 positional arguments but 9 were given] (Background on this error at: https://sqlalche.me/e/14/dbapi)

SQLALCHEMY URI uri that i used is given below:
kylin://ADMIN:KYLIN@<IP-address-machine-running-kylin>:7070/learn_kylin

Create cube feature is needed

• Desc:

just like #1 , we need a new feature to create cube through kylinpy

• Typical Scenario:

we get the cube_desc/kylin_sales_model.json, generated by the command
$KYLIN_HOME/bin/metastore.sh backup
and we want to use the kylinpy CLI like this:
kylinpy -h kap.kapdemo.com -P 7070 -u ADMIN -p KYLIN --project learn_kylin --api1 --debug -f Model_desc/kylin_sales_model.json cube
and as a result, we want a new cube built in kylin. YOU MAY THINK THIS IS AN ANOTHER WAY TO CLONE A CUBE

Needs LRU-like cache when the request is same

Incorrect statement try to convert kylin return boolean to sql boolean

Kylin return boolean as string "true" and "false". Kylinpy try to use dict to covert it to python bool value.

the dict can be found in utils/kylin_types.py line 21

...
BOOLEAN=bool,
...

in line 31, this map will cause follow action

bool_value = bool(str)

However, this won't work, in python, bool("true") is True, bool("false") is True still.

I have change line 21 in my private kylinpy like this:

BOOLEAN=lambda x: x in ["True","true","TRUE"],

It works.

But I'm sure we need consider a better solution later.

Support Session header

refactor http client

1. only http request and response
2. config kylinpy from kylinpy.Kylinpy class
3. ut converage

kylinpy.errors.KylinError or kylinpy.exceptions.KylinError?

After installation with pip install --upgrade kylinpy, I can import KylinError like this:
from kylinpy.errors import KylinError
However, if install kylinpy from the code with python setup.py install, Import Error will be thrown, unless I change this to from kylinpy.exceptions import KylinError:

When I use the 'case when' in select, it returns an extra space

like this "SELECT CASE WHEN org=1 THEN 'SAB' ELSE 'CD' END AS SCALE, from_date FROM xx GROUP BY from_date , CASE WHEN org=1 THEN 'SAB' ELSE 'CD' END"
i hope the result is "CD" or "SAB", but it returns "CD " or "SAB"

Adding cert support when using https

client._request not support client cert conf

python3：error：TypeError: the JSON object must be str, not 'bytes'

when test the kylin connection,python 2.7 is normal.but in python 3, the error was thrown:
==========================[QUERY]===============================

2018-02-07 16:06:15,319:ERROR:root:the JSON object must be str, not 'bytes'
Traceback (most recent call last):
File "/usr/local/webserver/venv/lib/python3.5/site-packages/superset/views/core.py", line 1508, in testconn
return json_success(json.dumps(engine.table_names(), indent=4))
File "/usr/local/webserver/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2128, in table_names
return self.dialect.get_table_names(conn, schema)
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/sqla_dialect.py", line 91, in get_table_names
return conn.connection.connection.get_table_names().get('data')
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/kylinpy.py", line 128, in wrapper
obj = request(self, *args, **kwargs)
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/kylinpy.py", line 252, in get_table_names
return [t['table_NAME'] for t in self.tables_and_columns()['data']]
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/kylinpy.py", line 128, in wrapper
obj = request(self, *args, **kwargs)
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/kylinpy.py", line 192, in tables_and_columns
return self.client.fetch(endpoint='tables_and_columns', params=params)
File "/usr/local/webserver/venv/lib/python3.5/site-packages/kylinpy/kylinpy.py", line 94, in fetch
dumps = json.loads(fd.read())
File "/usr/local/python3/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

What is the difference between pykylin and kylinpy?