RequestError: Timeout about pygsheets

[nithinmurali]

Hi, i have tested using your xlsx sheets. I dont think there is much i can change in code.- I think it is happening because currently the GOOGLE_SHEET_CELL_UPDATES_LIMIT is set to 50,000 which i think is a bit bigger and sometimes will take more time than the timeout period. So I can see 3 solutions, either you can increase the Timeout(currently 10), or increase the number of retries (default = 1) , or you can reduce the GOOGLE_SHEET_CELL_UPDATES_LIMIT. I tested writing both your sheets with all the 3 solutions, and it seems to run without any error.

changes to be made for all the three cases are given below,

pygsheets.client.GOOGLE_SHEET_CELL_UPDATES_LIMIT = 10,000
wks.set_dataframe(df, (1, 1), fit=True)

changing number of retries

gc = pygsheets.authorize(outh_file=oauth_file, retries=5)

for changing timeout

import httplib2
http_client = httplib2.Http( timeout=50)
gc = pygsheets.authorize(outh_file=oauth_file, http_client=http_client)

[jainraje]

I'm closing this as I rewrote all the code and it appears to be working.

[jainraje]

I'm reopening as I've run more tests and I'm observing a timeout failure:

test sequence:

• read xlsx file into a DF using pandas
• write the DF to google sheet

same test sequence is used to write 2 DFs to google sheets.
DF1 = 3796 rows x 71 cols
DF2 = 7273 row x 71 cols

Summary of test results:

test #1:
write 3796 row x 71 col DF to wk => consistently pass every time

test #2:
write 7273 row x 71 col DF to wks => consistently fail every time with timeout error but correct data is present in the google sheet

here is trace of failing case:

---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py in _execute_request(self, spreadsheet_id, request, batch)
    382                 try:
--> 383                     response = request.execute()
    384                 except Exception as e:

/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/_helpers.py in positional_wrapper(*args, **kwargs)
    132                     logger.warning(message)
--> 133             return wrapped(*args, **kwargs)
    134         return positional_wrapper

/Users/rajeev/anaconda/lib/python3.5/site-packages/googleapiclient/http.py in execute(self, http, num_retries)
    832           http, num_retries, 'request', self._sleep, self._rand, str(self.uri),
--> 833           method=str(self.method), body=self.body, headers=self.headers)
    834 

/Users/rajeev/anaconda/lib/python3.5/site-packages/googleapiclient/http.py in _retry_request(http, num_retries, req_type, sleep, rand, uri, method, *args, **kwargs)
    159       exception = None
--> 160       resp, content = http.request(uri, method, *args, **kwargs)
    161     # Retry on SSL errors and socket timeout errors.

/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/transport.py in new_request(uri, method, body, headers, redirections, connection_type)
    174                                 clean_headers(headers),
--> 175                                 redirections, connection_type)
    176 

/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/transport.py in request(http, uri, method, body, headers, redirections, connection_type)
    281                          redirections=redirections,
--> 282                          connection_type=connection_type)
    283 

/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py in request(self, uri, method, body, headers, redirections, connection_type)
   1321                 else:
-> 1322                     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
   1323         except Exception as e:

/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py in _request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
   1071 
-> 1072         (response, content) = self._conn_request(conn, request_uri, method, body, headers)
   1073 

/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py in _conn_request(self, conn, request_uri, method, body, headers)
   1024             try:
-> 1025                 response = conn.getresponse()
   1026             except (http.client.BadStatusLine, http.client.ResponseNotReady):

/Users/rajeev/anaconda/lib/python3.5/http/client.py in getresponse(self)
   1197             try:
-> 1198                 response.begin()
   1199             except ConnectionError:

/Users/rajeev/anaconda/lib/python3.5/http/client.py in begin(self)
    296         while True:
--> 297             version, status, reason = self._read_status()
    298             if status != CONTINUE:

/Users/rajeev/anaconda/lib/python3.5/http/client.py in _read_status(self)
    257     def _read_status(self):
--> 258         line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    259         if len(line) > _MAXLINE:

/Users/rajeev/anaconda/lib/python3.5/socket.py in readinto(self, b)
    575             try:
--> 576                 return self._sock.recv_into(b)
    577             except timeout:

/Users/rajeev/anaconda/lib/python3.5/ssl.py in recv_into(self, buffer, nbytes, flags)
    936                   self.__class__)
--> 937             return self.read(nbytes, buffer)
    938         else:

/Users/rajeev/anaconda/lib/python3.5/ssl.py in read(self, len, buffer)
    798         try:
--> 799             return self._sslobj.read(len, buffer)
    800         except SSLError as x:

/Users/rajeev/anaconda/lib/python3.5/ssl.py in read(self, len, buffer)
    582         if buffer is not None:
--> 583             v = self._sslobj.read(len, buffer)
    584         else:

timeout: The read operation timed out

During handling of the above exception, another exception occurred:

RequestError                              Traceback (most recent call last)
<ipython-input-57-7c10e4e4c03b> in <module>()
      7 #sh.add_worksheet(wks_name, rows=len(df) + 1, cols=len(df.columns))
      8 wks = sh.worksheet_by_title(wks_name)
----> 9 wks.set_dataframe(df, (1, 1), fit=True)

/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py in set_dataframe(self, df, start, copy_index, copy_head, fit, escape_formulae)
    732                         row[i] = "'" + str(row[i])
    733         crange = format_addr(start) + ':' + end
--> 734         self.update_cells(crange=crange, values=values)
    735 
    736     def get_as_df(self, has_header=True, index_colum=None, start=None, end=None, numerize=True, empty_value=''):

/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py in update_cells(self, crange, values, cell_list, majordim)
    387         body['majorDimension'] = majordim
    388         body['values'] = values
--> 389         self.client.sh_update_range(self.spreadsheet.id, body, self.spreadsheet.batch_mode)
    390 
    391     def update_col(self, index, values, row_offset=0):

/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py in sh_update_range(self, spreadsheet_id, body, batch, parse)
    337         final_request = self.service.spreadsheets().values().update(spreadsheetId=spreadsheet_id, range=body['range'],
    338                                                                     valueInputOption=cformat, body=body)
--> 339         self._execute_request(spreadsheet_id, final_request, batch)
    340 
    341     def sh_batch_clear(self, spreadsheet_id, body, batch=False):

/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py in _execute_request(self, spreadsheet_id, request, batch)
    386                         raise
    387                     if i == self.retries-1:
--> 388                         raise RequestError("Timeout")
    389                     # print ("Cant connect, retrying ... " + str(i))
    390                 else:

RequestError: Timeout

[jainraje]

Thank you for your prompt attention to fix this bug. When is your next planned official release? Before the release, how can i get my hands on the fix?

[nithinmurali]

Hi, you can get the fix by installing from staging branch

pip install https://github.com/nithinmurali/pygsheets/archive/staging.zip.

It would be great if you could confirm this fixes your issue. I will bump it for next release.

[jainraje]

FYI, to install I had to adjust your above procedure to include --upgrade:
pip install --upgrade https://github.com/nithinmurali/pygsheets/archive/staging.zip

I re-ran both test cases. Both failed with below error:

wks.set_dataframe(df, (1, 1), fit=True)

File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 735, in set_dataframe
self.update_cells(crange=crange, values=values)
File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 389, in update_cells
self.client.sh_update_range(self.spreadsheet.id, body, self.spreadsheet.batch_mode)
File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py", line 359, in sh_update_range
for batch_start in range(0, num_rows, batch_length):
TypeError: 'float' object cannot be interpreted as an integer

[nithinmurali]

ahh, my bad 6152f08 . Try now.

[jainraje]

i'm not expert w/ github, can yu please update so I can install using the same method as before:

pip install --upgrade https://github.com/nithinmurali/pygsheets/archive/staging.zip

[nithinmurali]

--upgrade should work. Else just uninstall and install again.

pip uninstall pygsheets
pip install --upgrade https://github.com/nithinmurali/pygsheets/archive/staging.zip

[jainraje]

Actually I installed. I'm getting a different error now:

wks.set_dataframe(df, (1, 1), fit=True)

File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 735, in set_dataframe
self.update_cells(crange=crange, values=values)
File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 389, in update_cells
self.client.sh_update_range(self.spreadsheet.id, body, self.spreadsheet.batch_mode)
File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py", line 366, in sh_update_range
body['range'] = title+'!'+format_addr(value_range_start, output='label')+':'+format_addr(value_range_end, output='label')
File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/utils.py", line 110, in format_addr
raise InvalidArgumentValue
pygsheets.exceptions.InvalidArgumentValue

[jainraje]

FYI, if you have another set of changes you wanted tested, please let me know. No problem for me. Thx, === Rajeev Jain [email protected] On Jun 7, 2017, at 12:28 PM, Nithin Murali <[email protected]> wrote:

…

--upgrade should work. Else just uninstall and install again. pip uninstall pygsheets pip install --upgrade https://github.com/nithinmurali/pygsheets/archive/staging.zip — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#84 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOe_-RBJin65SeJqbnGHtJqxRQb9V_oIks5sBvnFgaJpZM4Nkmrc>.

[nithinmurali]

Thanks, Can you check now?

[jainraje]

That seems to fix the issue. No errors and data in the sheet look OK for both test cases. I'll continue to run this and will convey back my observations (good or bad). thank you for your good and prompt attention on this!

[jainraje]

I am still observing pygsheets.exceptions.RequestError: Timeout

Please re-open.

Here is a failure trace:
Writing to google sheet

Traceback (most recent call last):
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py", line 414, in _execute_request
    response = request.execute()
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/googleapiclient/http.py", line 835, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/googleapiclient/http.py", line 162, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py", line 1322, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py", line 1072, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/httplib2/__init__.py", line 996, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/Users/rajeev/anaconda/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/Users/rajeev/anaconda/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/Users/rajeev/anaconda/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/Users/rajeev/anaconda/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/Users/rajeev/anaconda/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/Users/rajeev/anaconda/lib/python3.5/ssl.py", line 899, in sendall
    v = self.send(data[count:])
  File "/Users/rajeev/anaconda/lib/python3.5/ssl.py", line 869, in send
    return self._sslobj.write(data)
  File "/Users/rajeev/anaconda/lib/python3.5/ssl.py", line 594, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "update_google_sheets_v1.py", line 94, in <module>
    main()
  File "update_google_sheets_v1.py", line 79, in main
    google_sheets_write_df(technicals, spreadsheet_name, tech_sheet)
  File "../rjlib/utils_app_v1.py", line 460, in google_sheets_write_df
    wks.set_dataframe(df, (1, 1), fit=True)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 735, in set_dataframe
    self.update_cells(crange=crange, values=values)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/worksheet.py", line 389, in update_cells
    self.client.sh_update_range(self.spreadsheet.id, body, self.spreadsheet.batch_mode)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py", line 370, in sh_update_range
    self._execute_request(spreadsheet_id, final_request, batch)
  File "/Users/rajeev/anaconda/lib/python3.5/site-packages/pygsheets/client.py", line 419, in _execute_request
    raise RequestError("Timeout")
pygsheets.exceptions.RequestError: Timeout

[nithinmurali]

How big is the df you are trying to write? Is it occurring consistently? try reducing the GOOGLE_SHEET_CELL_UPDATES_LIMIT to a lower value.

pygsheets.client.GOOGLE_SHEET_CELL_UPDATES_LIMIT=5000

Please not that timeout can occur due to network errors too.

[jainraje]

The df is 7273 rows x 70 cols. Same that passed before but now failing.

My test cases are using 2 DFs:

DF1 = 3796 rows x 71 cols
DF2 = 7273 row x 71 cols

when writing DF2 it appears the 1st attempt fails. but the 2nd attempt passes (sometimes). I don't believe the timeout to be a network issue (but it could). My sense is there is still a bug of some sort (logic, race condition or other) still present.

if you send me your email I'll email you the xlsx file I'm using for DF2. With the file, you can test precisely my same test case. Could be helpful for you in terms of debugging.

[nithinmurali]

Hi, having your xlsx would certainly help me debug the issue. my email is imnmfotmal [at] gmail.

[jainraje]

xlsx files have been sent.

[jainraje]

thx for reviewing and testing my files, i'll try out your suggestions. feel free to close the ticket. i'll report back should the suggestions for not work.

[jainraje]

FYI, I'm using your 1st suggestion and no more crashes have been observed (at least so far):

pygsheets.client.GOOGLE_SHEET_CELL_UPDATES_LIMIT = 10000
wks.set_dataframe(df, (1, 1), fit=True)