to_td function does not work from "requests" library v0.16.0 about pandas-td

[chezou]

As takuti said in #8 (review), it seems the root cause is pyopenssl doesn't support bytesarray yet and conda installs requests with pyOpenSSL by default. So we hit this issue pyca/pyopenssl#622.

I think we can approach this issue, but I also recommend to use pytd instead.

[aodag]

i found not only bytearray causes this problem, array.array too.

[chezou]

Close since the root cause has been fixed on td-client-python treasure-data/td-client-python#59

[edit]
Released td-client 0.14.0. https://pypi.org/project/td-client/0.14.0/

You can try with pip install -U td-client
[/edit]

@kamikaseda @hadrianhu Can you tell me if you still have the issue with the master version of td-client-python? Feel free to reopen it if this issue happens again.

[yyuu]

I've released 0.9.0 on PyPI with including dependency fixes (#8). Please give it a try and confirm if the problem has solved.

[kamikaseda]

Confirmed it worked. Thank you.

[kamikaseda]

I'd like to close this issue because #8 is enough for now.

[hadrianhu]

@yyuu, I are getting the same error when using anaconda distribution, I have confirmed the same script works fine on standard python distro.

Running on anaconda

Verified Requests Version

(base) bash-3.2$ pip show requests
Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /Users/ivan/anaconda3/lib/python3.7/site-packages
Requires: urllib3, chardet, idna, certifi
Required-by: Sphinx, pandas-td, conda-build, anaconda-project, anaconda-client
(base) bash-3.2$ 

Verified pandas_td version

(base) bash-3.2$ pip show pandas_td
Name: pandas-td
Version: 0.10.0
Summary: Pandas extension for Treasure Data
Home-page: https://github.com/treasure-data/pandas-td
Author: Treasure Data, Inc.
Author-email: [email protected]
License: Apache License 2.0
Location: /Users/ivan/anaconda3/lib/python3.7/site-packages
Requires: pandas, six, certifi, td-client, requests, pytz, futures, tzlocal
Required-by: 
(base) bash-3.2$

Script Error

pandas_td.to_td(df, 'ivan_zen_analaytics.boom', con1, if_exists='replace', index=False)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/pandas_td/td.py", line 714, in to_td
    uploader.upload_frame(frame, chunksize)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/pandas_td/td.py", line 409, in upload_frame
    self._upload(self._gzip(self._pack(chunk)))
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/pandas_td/td.py", line 399, in _upload
    elapsed = self.client.import_data(self.database, self.table, 'msgpack.gz', data, data_size, unique_id)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/tdclient/client.py", line 602, in import_data
    return self.api.import_data(db_name, table_name, format, bytes_or_stream, size, unique_id=unique_id)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/tdclient/import_api.py", line 39, in import_data
    with self.put(path, bytes_or_stream, size, **kwargs) as res:
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/tdclient/api.py", line 275, in put
    response = self.send_request("PUT", url, body=stream, headers=headers, decode_content=True, preload_content=False)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/tdclient/api.py", line 359, in send_request
    return self.http.urlopen(method, url, body=body, headers=headers, **kwargs)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/urllib3/poolmanager.py", line 323, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/ivan/anaconda3/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/ivan/anaconda3/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/ivan/anaconda3/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/ivan/anaconda3/lib/python3.7/http/client.py", line 1055, in _send_output
    self.send(chunk)
  File "/Users/ivan/anaconda3/lib/python3.7/http/client.py", line 981, in send
    self.sock.sendall(d)
  File "/Users/ivan/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 327, in sendall
    while total_sent < len(data):
TypeError: object of type 'int' has no len()

[chezou]

I confirmed the issue still happens with the following environment using miniconda3 Docker image:

(base) root@c379604d7ea3:/# conda --version
conda 4.5.12
(base) root@c379604d7ea3:/# python --version
Python 3.7.1
(base) root@c379604d7ea3:/# pip freeze
asn1crypto==0.24.0
backcall==0.1.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
conda==4.5.12
cryptography==2.4.2
decorator==4.4.0
futures==3.1.1
idna==2.8
ipython==7.4.0
ipython-genutils==0.2.0
jedi==0.13.3
msgpack==0.6.1
numpy==1.16.2
pandas==0.24.2
pandas-td==0.10.0
parso==0.4.0
pexpect==4.7.0
pickleshare==0.7.5
prompt-toolkit==2.0.9
ptyprocess==0.6.0
pycosat==0.6.3
pycparser==2.19
Pygments==2.3.1
pyOpenSSL==18.0.0
PySocks==1.6.8
python-dateutil==2.8.0
pytz==2019.1
requests==2.21.0
ruamel-yaml==0.15.46
six==1.12.0
td-client==0.13.0
traitlets==4.3.2
tzlocal==1.5.1
urllib3==1.24.1
wcwidth==0.1.7

[hadrianhu]

@yyuu have you had a chance to take a look

[aodag]

Using pyopenssl in urllib3 is turned on by the such a below codes.

from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()

reuqests calls above code from version 2.16, caused the problem of this issue.

import urllib3
import requests
import certifi

# from urllib3.contrib import pyopenssl
# pyopenssl.inject_into_urllib3()


http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs=certifi.where())
body = b"abc"
body = bytearray(body)
r = http.request('POST', 'https://example.com', body=body)
print(r.status)
print(r.data)

[aodag]

inject_into_urllib3 sets urllib3.utils.IS_PYOPENSSL to True,
and pyopenssl accepts memoryview that contains bytearray.

import urllib3
import urllib3.utils
import certifi

from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()


http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED',
    ca_certs=certifi.where())
body = b"abc"
body = bytearray(body)
if urllib3.util.IS_PYOPENSSL:
     body = memoryview(body)
r = http.request('POST', 'https://example.com', body=body)
print(r.status)
print(r.data)

[chezou]

Confirmed working with continuumio/anaconda3:2019.03 Docker image.

(.venv) ➜  pandas-td git:(master) docker run -it --rm -e TD_API_KEY=$TD_API_KEY -e TD_API_SERVER=$TD_API_SERVER continuumio/anaconda3:2019.03 /bin/bash
(base) root@dd13f47f7a98:/# pip install pandas-td
Collecting pandas-td
  Downloading https://files.pythonhosted.org/packages/21/c3/ad0b140364cd68fce61c6d2d74c0a6048b870d9f7293e1433af0289fe4fb/pandas_td-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from pandas-td) (2018.9)
Collecting tzlocal (from pandas-td)
  Downloading https://files.pythonhosted.org/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz
Requirement already satisfied: pandas>=0.16.0 in /opt/conda/lib/python3.7/site-packages (from pandas-td) (0.24.2)
Collecting td-client>=0.4.0 (from pandas-td)
  Downloading https://files.pythonhosted.org/packages/47/53/c3259c70e56563cebd36661dfdbd769f331f8d0f579964a2d9a4a26dd1e0/td_client-0.14.0-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 662kB/s
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from pandas-td) (2019.3.9)
Requirement already satisfied: requests>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from pandas-td) (2.21.0)
Collecting futures>=3.0.3 (from pandas-td)
  Downloading https://files.pythonhosted.org/packages/cc/26/b61e3a4eb50653e8a7339d84eeaa46d1e93b92951978873c220ae64d0733/futures-3.1.1.tar.gz
Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from pandas-td) (1.12.0)
Requirement already satisfied: numpy>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.16.0->pandas-td) (1.16.2)
Requirement already satisfied: python-dateutil>=2.5.0 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.16.0->pandas-td) (2.8.0)
Requirement already satisfied: urllib3 in /opt/conda/lib/python3.7/site-packages (from td-client>=0.4.0->pandas-td) (1.24.1)
Requirement already satisfied: msgpack in /opt/conda/lib/python3.7/site-packages (from td-client>=0.4.0->pandas-td) (0.6.1)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests>=2.21.0->pandas-td) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests>=2.21.0->pandas-td) (3.0.4)
Building wheels for collected packages: tzlocal, futures
  Building wheel for tzlocal (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/15/ae/df/a67bf1ed84e9bf230187d36d8dcfd30072bea0236cb059ed91
  Building wheel for futures (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/f3/f9/c7/4fbf1faa6038faf183f6e3ea61f17a5f7eea5ab9a1dd7753fd
Successfully built tzlocal futures
Installing collected packages: tzlocal, td-client, futures, pandas-td
Successfully installed futures-3.1.1 pandas-td-0.10.0 td-client-0.14.0 tzlocal-1.5.1
(base) root@dd13f47f7a98:/# python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas_td as td
>>> import pandas as pd
>>> import numpy as np
>>> con = td.connect()
>>> df = pd.DataFrame(np.random.rand(3, 3), columns=['x', 'y', 'z'])
>>> td.to_td(df, 'aki.test_pandastd', con, if_exists='replace', index=False)