kaliiiiiiiiii / cdp-socket

socket for handling chrome-developer-protocol connections

💻 Tech Stack:

Support Me

📊 GitHub Stats:

---

---

Star History

cdp-socket's Issues

requirements.txt: remove dev dependencies

these belong to a separate file, something like requirements-dev.txt

CDP-Socket/requirements.txt

Lines 5 to 7 in 9188210

	# dev
	twine
	setuptools

Task was destroyed but it is pending! error

Full error:

Task was destroyed but it is pending!
task: <Task pending name='Task-247' 
coro=<SingleCDPSocket._handle_callback.<locals>.async_handle() done, 
defined at C:\Users\PC\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\socket.py:178> 
wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]>>

Code

import asyncio
from selenium_driverless import webdriver
from selenium_driverless.scripts.network_interceptor import NetworkInterceptor

options = webdriver.ChromeOptions()

async def task(driver):
    await driver.get('https://github.com', wait_load=False)

async def RequestBlocker():
    async with await webdriver.Chrome(options=options) as driver:
        while True:
            await driver.get('https://google.com')
            driver3 = await driver.new_window('tab')
            async with NetworkInterceptor(driver3) as interceptor:
                await task(driver3)
                await asyncio.sleep(2)
                async for data in interceptor:
                    if data.request.url == "https://collector.github.com/github/collect":
                        await data.fail_request('TimedOut')
                        print('Request Blocked')
                        break
                await driver3.close()


asyncio.run(RequestBlocker())

Minimal code to reproduce:

import asyncio
from selenium_driverless import webdriver
from selenium_driverless.scripts.network_interceptor import NetworkInterceptor

async def main():
    async with await webdriver.Chrome() as driver:
        while True:
            tab = await driver.new_window('tab')
            async with NetworkInterceptor(tab) as interceptor:
                asyncio.ensure_future(tab.get('https://httpbin.org/get', wait_load=False))
                async for data in interceptor:
                    await data.fail_request('TimedOut')
                    print('Request Blocked')
                    break
                await tab.close()

asyncio.run(main())

OS: Windows 10 22H2
Chrome Version: 124.0.6367.62
selenium_driverless.version: 1.9.1
cdp_socket.__version: 1.2.7
Python version: 3.11

EDIT by @kaliiiiiiiiii

• add minimal code

python site-packages should be treated as read-only

  File "/lib/python3.10/site-packages/cdp_socket/utils/utils.py", line 119, in launch_chrome
    os.makedirs(data_dir_path, exist_ok=True)
PermissionError: [Errno 13] Permission denied: '/lib/python3.10/site-packages/cdp_socket/files/tmp'

CDP-Socket/src/cdp_socket/utils/utils.py

Lines 116 to 119 in 4bae35c

	if not data_dir_path:
	data_dir_path = f"{cdp_sock_path()}files/tmp/data_dir"
	if not os.path.exists(data_dir_path):
	os.makedirs(data_dir_path, exist_ok=True)

CDP-Socket/src/cdp_socket/utils/utils.py

Lines 60 to 61 in 4bae35c

	def cdp_sock_path():
	return os.path.dirname(cdp_socket.__file__) + "/"

quickfix:

def cdp_sock_path():
    return os.environ.get("HOME", "/tmp") + "/.cache/cdp-socket"

similar issue: kaliiiiiiiiii/Selenium-Driverless#122

cdp_socket.exceptions.CDPError: {'code': -32600, 'message': 'Duplicate id in protocol request'}

"C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 382, 
in text return await self.get_property("textContent") File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 371, 
in get_property return await self.execute_script(f"return obj[arguments[0]]", name) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 913, 
in execute_script return await self.exec(script, *args, max_depth=max_depth, serialization=serialization, File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 202, 
in exec res = await self.exec_raw(script, *args, max_depth=max_depth, File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 152, 
in exec_raw raise e File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 145, 
in exec_raw res = await self.target.execute_cdp_cmd("Runtime.callFunctionOn", args, timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\target.py", line 1125, 
in execute_cdp_cmd result = await self.socket.exec(method=cmd, params=cmd_args, timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\socket.py", line 85, 
in exec res = await asyncio.wait_for(self._responses[_id], timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\asyncio\tasks.py", line 445, 
in wait_for return fut.result() 
cdp_socket.exceptions.CDPError: {'code': -32600, 'message': 'Duplicate id in protocol request'} 

You suggested possible issue is here:

CDP-Socket/src/cdp_socket/socket.py

Lines 71 to 77 in 4813479

	_id = [self._req_count][0]
	_dict = {'id': _id, 'method': method}
	if params:
	_dict['params'] = params
	await self._ws.send(json.dumps(_dict))
	self._req_count += 1
	return _id

setup.cfg: LICENSE.md should be LICENSE

CDP-Socket/setup.cfg

Line 3 in 9188210

license_files = LICENSE.md

install says

warning: no files found matching 'LICENSE.md'

High memory consumption

High memory consumption There is something wrong with the class, if a program runs in a loop and downloads data non-stop, the program's memory consumption increases every second.

Sample program:

from selenium_driverless import webdriver
from selenium_driverless.types.by import By
import asyncio


async def main():
    options = webdriver.ChromeOptions()
    async with webdriver.Chrome(options=options) as driver:
        while True:
            await driver.get('https://pudelek.pl')
            await driver.sleep(0.5)
            source = await driver.page_source
            await asyncio.sleep(3)
            

asyncio.run(main())

After several minutes of running the program, the memory usage of the python.exe process increases by several MB, and after several dozen minutes even up to 2 GB.

Python 3.10.6
Windows 11
Chrome 119.0.6045.124