hmaker / python-cdp

Python client and types generator for the Chrome DevTools Protocol (CDP)

Python CDP

Currently supports CDP r1179426 (Chrome 117).

Python CDP Generator (shortened to PyCDP) is a library that provides Python wrappers for the types, commands, and events specified in the Chrome DevTools Protocol.

The Chrome DevTools Protocol provides for remote control of a web browser by sending JSON messages over a WebSocket. That JSON format is described by a machine-readable specification. This specification is used to automatically generate the classes and methods found in this library.

Installation

You can install this library as a dependency on your project with:

pip install git+https://github.com/HMaker/python-cdp.git@latest

Change the git tag @latest if you need another version. To install for development, clone this repository, install Poetry package manager and run poetry install to install dependencies.

Usage

If all you want is automate Chrome right now, PyCDP includes a low-level client for asyncio and twisted:

import asyncio
from pycdp import cdp
from pycdp.browser import ChromeLauncher
from pycdp.asyncio import connect_cdp

async def main():
    chrome = ChromeLauncher(
        binary='/usr/bin/google-chrome', # linux path
        args=['--remote-debugging-port=9222', '--incognito']
    )
    # ChromeLauncher.launch() is blocking, run it on a background thread
    await asyncio.get_running_loop().run_in_executor(None, chrome.launch)
    conn = await connect_cdp('http://localhost:9222')
    target_id = await conn.execute(cdp.target.create_target('about:blank'))
    target_session = await conn.connect_session(target_id)
    await target_session.execute(cdp.page.enable())
    # you may use "async for target_session.listen()" to listen multiple events, here we listen just a single event.
    with target_session.safe_wait_for(cdp.page.DomContentEventFired) as navigation:
        await target_session.execute(cdp.page.navigate('https://chromedevtools.github.io/devtools-protocol/'))
        await navigation
    dom = await target_session.execute(cdp.dom.get_document())
    node = await target_session.execute(cdp.dom.query_selector(dom.node_id, 'p'))
    js_node = await target_session.execute(cdp.dom.resolve_node(node))
    print((await target_session.execute(cdp.runtime.call_function_on('function() {return this.innerText;}', js_node.object_id, return_by_value=True)))[0].value)
    await target_session.execute(cdp.page.close())
    await conn.close()
    await asyncio.get_running_loop().run_in_executor(None, chrome.kill)

asyncio.run(main())

the twisted client requires twisted and autobahn packages:

from twisted.python.log import err
from twisted.internet import reactor, defer, threads
from pycdp import cdp
from pycdp.browser import ChromeLauncher
from pycdp.twisted import connect_cdp

async def main():
    chrome = ChromeLauncher(
        binary='C:\Program Files\Google\Chrome\Application\chrome.exe', # windows path
        args=['--remote-debugging-port=9222', '--incognito']
    )
    await threads.deferToThread(chrome.launch)
    conn = await connect_cdp('http://localhost:9222', reactor)
    target_id = await conn.execute(cdp.target.create_target('about:blank'))
    target_session = await conn.connect_session(target_id)
    await target_session.execute(cdp.page.enable())
    await target_session.execute(cdp.page.navigate('https://chromedevtools.github.io/devtools-protocol/'))
    async with target_session.wait_for(cdp.page.DomContentEventFired):
        dom = await target_session.execute(cdp.dom.get_document())
        node = await target_session.execute(cdp.dom.query_selector(dom.node_id, 'p'))
        js_node = await target_session.execute(cdp.dom.resolve_node(node))
        print((await target_session.execute(cdp.runtime.call_function_on('function() {return this.innerText;}', js_node.object_id, return_by_value=True)))[0].value)
    await target_session.execute(cdp.page.close())
    await conn.close()
    await threads.deferToThread(chrome.kill)

def main_error(failure):
    err(failure)
    reactor.stop()

d = defer.ensureDeferred(main())
d.addErrback(main_error)
d.addCallback(lambda *args: reactor.stop())
reactor.run()

You also can use just the built-in CDP type wrappers with import pycdp.cdp on your own client implementation. If you want to try a different CDP version you can build new type wrappers with cdpgen command:

usage: cdpgen <arguments>

Generate Python types for the Chrome Devtools Protocol (CDP) specification.

optional arguments:
  -h, --help            show this help message and exit
  --browser-protocol BROWSER_PROTOCOL
                        JSON file for the browser protocol
  --js-protocol JS_PROTOCOL
                        JSON file for the javascript protocol
  --output OUTPUT       output path for the generated Python modules

JSON files for the CDP spec can be found at https://github.com/ChromeDevTools/devtools-protocol/tree/master/json

Example:

cdpgen --browser-protocol browser_protocol.json --js-protocol js_protocol.json --output /tmp/cdp

You can then include the /tmp/cdp package in your project and import it like the builtin CDP types.

Updating built-in CDP wrappers

The update-cdp.sh script generates the builtin CDP wrappers, the pycdp.cdp package, by automatically fetching CDP protocol specifications from the ChromeDevTools repostitory.

To generate types for the latest version:

./update-cdp.sh

To generate types for a specific version, you must provide full commit hash:

./update-cdp.sh 4dd6c67776f43f75bc9b19f09618c151621c6ed9

P.S. Don't forget to make it executable by running chmod +x update-cdp.sh

Implementation of a CDP client

The pycdp.cdp package follows same structure of CDP domains, each domain is a Python module and each command a function in that module.

Each function is a generator with a single yield which is a Python dict, on the CDP wire format, containing the message that should be sent to the browser, on resumption the generator receives the message from browser:

import cdp

# Get all CDP targets
command = cdp.target.get_targets() # this is a generator
raw_cdp_request = next(command) # receive the yield
raw_cdp_response = send_cdp_request(raw_cdp_request) # you implement send_cdp_request, raw_cdp_request is the JSON object that should be sent to browser
try:
    command.send(raw_cdp_response) # send the response to the generator where raw_cdp_response is the JSON object received from browser, it will raise StopIteration
    raise RuntimeError("the generator didnt exit!") # this shouldnt happen
except StopIteration as result:
    response = result.value # the parsed response to Target.get_targets() command
print(response)

For implementation details check out the docs.

---

PyCDP is licensed under the MIT License.

---

python-cdp's Issues

Add to_json method to CDP events

Hi,

All the python classes have the from_json method. The ones that represent CDP data types also have a to_json method, but the events don't have. I would like a complete coverage of json conversion for the python objects. I think the generation script can be modified to achieve this requirement.

At the moment, I don't see this to be a difficult task. I am happy to make the contribution myself.

GG for the project

Remove code duplication between asyncio and twisted clients

We need to abstract event-loop dependent features so we can reuse common code between asyncio and twisted clients.

кент

cdpgen not generating util.py file

after executing a cdpgen --browser-protocol devtools-protocol/browser_protocol.json --js-protocol devtools-protocol/js_protocol.json --output cdp/ command, script does not create a util.py which used in cdp/accessibility.py.

Used environment:
MacOS
Poetry

Here is my traceback:

Traceback (most recent call last):
  File "/Users/turbokach/Dev/betbot_cdp/script.py", line 2, in <module>
    import cdp
  File "/Users/turbokach/Dev/betbot_cdp/cdp/__init__.py", line 6, in <module>
    from . import (accessibility, animation, audits, background_service, browser, css, cache_storage, cast, console, dom, dom_debugger, dom_snapshot, dom_storage, database, debugger, device_orientation, emulation, event_breakpoints, fetch, headless_experimental, heap_profiler, io, indexed_db, input_, inspector, layer_tree, log, media, memory, network, overlay, page, performance, performance_timeline, profiler, runtime, schema, security, service_worker, storage, system_info, target, tethering, tracing, web_audio, web_authn)
  File "/Users/turbokach/Dev/betbot_cdp/cdp/accessibility.py", line 12, in <module>
    from .util import event_class, T_JSON_DICT
ModuleNotFoundError: No module named 'cdp.util'

Experimental fields

Is there an option to properly handle experimental (and not optional) fields (that may or may not be present in implementation as far as I understand that correctly)?
Currently it evaluates directly as json['field'] which throws 'key not found' exception, breaking the whole thing down.

cdp.page.DomContentEventFired event listener not catching an event

async with target_session.wait_for(cdp.page.DomContentEventFired): does not catch an event on tab navigation using a python types generated by cdpgen.
last used protocol commit: ChromeDevTools/devtools-protocol@5caaeb9
but seems like this bug came a few commits before.

using a README.md example code:

import asyncio
# from pycdp import cdp  # here is an import from default pycdp types
import cdp  # here we imported cdp which was generated by cdpgen
from pycdp.browser import ChromeLauncher
from pycdp.asyncio import connect_cdp

async def main():
    chrome = ChromeLauncher(
        binary='/usr/bin/google-chrome', # linux path
        args=['--remote-debugging-port=9222', '--incognito']
    )
    # ChromeLauncher.launch() is blocking, run it on a background thread
    await asyncio.get_running_loop().run_in_executor(None, chrome.launch)
    conn = await connect_cdp('http://localhost:9222')
    target_id = await conn.execute(cdp.target.create_target('about:blank'))
    target_session = await conn.connect_session(target_id)
    await target_session.execute(cdp.page.enable())
    await target_session.execute(cdp.page.navigate('https://chromedevtools.github.io/devtools-protocol/'))
    # you may use "async for target_session.listen()" to listen multiple events, here we listen just a single event.
    async with target_session.wait_for(cdp.page.DomContentEventFired):
        dom = await target_session.execute(cdp.dom.get_document())
        node = await target_session.execute(cdp.dom.query_selector(dom.node_id, 'p'))
        js_node = await target_session.execute(cdp.dom.resolve_node(node))
        print((await target_session.execute(cdp.runtime.call_function_on('function() {return this.innerText;}', js_node.object_id, return_by_value=True)))[0].value)
    await target_session.execute(cdp.page.close())
    await conn.close()
    await asyncio.get_running_loop().run_in_executor(None, chrome.kill)

asyncio.run(main())