micropython-magic's Introduction

micropython-magic

These Jupyter magic methods allow MicroPython to be used from within any Jupyter Notebook or JupyterLab (formerly IPython Notebook) The magics make use of the mpremote tool to enable communication with the MCUs

This allows:

Mixing of Host and MCU Code ( and languages if you wish)
Creating graphs of the data captured by MCU sensors
create re-uasable sequences ( download/compile firmware - flash firmware - uploade code - run expiriment - same outcome)
Create and execute tests that require orchestration across multiple MCUs and hosts
Rapid Prototyping
Capturing the results and outputs of your expiriments in a consistent way
Mixing documentation with code

A few of the possibilities

Live Plot of the cpu temperature	Visualize the memory map of the MCU
	Memory allocation of the MCU over time

For the source please refer to the samples folder

Installation

create and activate a venv python3 -m venv .venv
pip install -U "micropython-magic"
or install directly into your notbook environment/kernel using the '%pip' magic by running
- %pip install -U "micropython-magic"

Recommended : install stubs for your MCU of choice

Install stubs for MicroPython syntax checking pip install micropython-rp2-stubs (or your port of choise)

Usage

1) Create a notebook

install your desired notebook environment:
create a new notebook

2) Load the magic

%load_ext micropython_magic

This can also be configured once to always load automatically ( see below)

3) add a cell with some code to run on the MCU

# %%micropython  
from machine import Pin
led = Pin(25, Pin.OUT)
led.value(1)

The %%micropython cell magic will instruct Jupyter to run the code on the connected MCU

4) enable code highlighting for MicroPython

%pip install micropython-esp32-stubs==1.20.0.*
# installs the stubs for MicroPython syntax checking (one time install per environment)

# %%micropython  
from machine import Pin
led = Pin(25, Pin.OUT)
led.value(1)

This allows for syntax highlighting and code completion of MicroPython code. Tested in VSCode with

Python extension
Pylance extension

More Examples

Please refer to the samples folder for more examples

install - install the magic
board_control - basic board control
board_selection. - list connected boards and loop through them
device_info - Get simple access to port, board and hardware and firmware information
WOKWI - Use MicroPython magic with WOKWI as a simulator (no device needed)
Plot rp2 CPU temp - create a plot of the CPU temperature of a rp2040 MCU(bqplot)
Display Memory Map - Micropython memory map visualizer
Plot Memory Usage - plot the memory usage of a Micropython script running on a MCU over time

Automatically load the magic on startup

In order to automatically load the magic on startup, you can add the following to your ipython_config.py file:

create a ipython profile
- ipython profile create
- add the following to the configuration file (.../.ipython/profile_default/ipython_config.py)
```
c = get_config()

c.InteractiveShellApp.extensions = [
    'micropython_magic'
]
```

Configuration options

Configuration can be done via the %config magic

%config MicroPythonMagic

    MicroPythonMagic(Magics) options
    ------------------------------
    MicroPythonMagic.loglevel=<UseEnum>
        Choices: any of ['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR']
        Current: <LogLevel.WARNING: 'WARNING'>
    MicroPythonMagic.timeout=<Float>
        Current: 300.0

# example
%config MicroPythonMagic.loglevel = 'TRACE'

loglevel : set the loglevel for the magic ( default WARNING)
timeout : set the timeout for the mpremote connection ( default 300 seconds - 5 minutes)

Development and contributions

The most welcome contributions are :

Testing on different platforms (OS) but also different Jupyter environments ( Jupyter Notebook, JupyterLab, VSCode)
Provide additional sample notebooks
Help add documentation (preferably in a notebook or .md file)
Share this with other people that may be interested in this.

See current status and on Github

micropython-magic's People

Contributors

Stargazers

Watchers

Forkers

iot49

micropython-magic's Issues

add board lookup

Check -unix implementation

https://github.com/v923z/micropython-ulab/blob/pid2/docs%2Fulab-pid.ipynb

Show backtrace when a micropython exception occurs on the device?

if it is possible to get the full backtrace when a micropython exception occurs on the device?

I think the main challenges are :

parsing the full/relevant MCU exception
exposing the traceback to Jupyter as log messages
providing the MCU exception information to Jupyter
limiting/reducing the amount of irrelevant trace information (

current limited implementation:
https://github.com/Josverl/micropython-magic/blob/Josverl/issue21/src/micropython_magic/octarine.py#L366

allow mounting a local folder during cell magic

syntax something like %%micropython --mount <path>

cmdline :: mpremote mount <path> <rest of cmdline>

Update --info code to handle V1.22.0. and new descriptions

version has an additional dot
CPU is not alway correctly determined ( double with ) ( see SPIRAM) below
change board --> description

port	arch	family	board	cpu	version	mpy	ver	serial_port
esp32	xtensawin	micropython	Generic ESP32 module with SPIRAM with ESP32	SPIRAM	1.22.0	v6.2	v1.22.0.	COM4
samd	armv7emsp	micropython	Wio Terminal D51R with SAMD51P19A	SAMD51P19A	1.22.0	v6.2	v1.22.0.	COM10
rp2	armv6m	micropython	Arduino Nano RP2040 Connect with RP2040	RP2040	1.22.0	v6.2	v1.22.0.	COM12

allow 'info.serialport' Notation

avoid needing to use dictionary access info.["serialport"] --> info.serialport

perhaps by using a Map as in : https://stackoverflow.com/a/32107024/7909427

FileNotFoundError exception when trying to execute micropython code

I'm running Jupyter via the VS Code extension and just started trying out your package. I can load the extension without issue, but as soon as I try to run some micropython code, it fails trying to run the mpremote command:

Below is the full exception output. I tried running the mpremote command manually on the command line and it seemed to run without issue, so perhaps this is an issue within VS Code and its access to the file system?

I'm running on MacOS version 14.2.1.

Hopefully we can get this worked out, I'm excited to use this extension to help debug one of my micropython projects!

{
	"name": "FileNotFoundError",
	"message": "Failed to start m",
	"stack": "---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/interactive.py:100, in ipython_run(cmd, stream_out, timeout, shell, hide_meminfo, store_output, log_errors, tags, follow)
     99 try:
--> 100     process = subprocess.Popen(
    101         cmd,
    102         shell=False,
    103         stdout=subprocess.PIPE,
    104         stderr=subprocess.PIPE,
    105         start_new_session=False,
    106     )  # ,  universal_newlines=True)
    107 except FileNotFoundError as e:

File /opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py:1026, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group)
   1023             self.stderr = io.TextIOWrapper(self.stderr,
   1024                     encoding=encoding, errors=errors)
-> 1026     self._execute_child(args, executable, preexec_fn, close_fds,
   1027                         pass_fds, cwd, env,
   1028                         startupinfo, creationflags, shell,
   1029                         p2cread, p2cwrite,
   1030                         c2pread, c2pwrite,
   1031                         errread, errwrite,
   1032                         restore_signals,
   1033                         gid, gids, uid, umask,
   1034                         start_new_session, process_group)
   1035 except:
   1036     # Cleanup if the child failed starting.

File /opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py:1950, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group)
   1949         err_msg = os.strerror(errno_num)
-> 1950     raise child_exception_type(errno_num, err_msg, err_filename)
   1951 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: 'mpremote connect auto resume  run /var/folders/zz/4_j849n104j0ztbqh1mc34100000gn/T/tmpb667n9to.py'

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
Cell In[8], line 1
----> 1 get_ipython().run_cell_magic('micropython', '', 'from micropython import const\
TEST_CONST = const(5)\
')

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2517, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2515 with self.builtin_trap:
   2516     args = (magic_arg_s, cell)
-> 2517     result = fn(*args, **kwargs)
   2519 # The code below prevents the output from being displayed
   2520 # when using magics with decorator @output_can_be_silenced
   2521 # when the last Python token in the expression is a ';'.
   2522 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/octarine.py:211, in MpyMagics.micropython(self, line, cell)
    209 if not cell:
    210     raise UsageError(\"Please specify some MicroPython code to execute\")
--> 211 output = self.MCU.run_cell(
    212     cell, timeout=args.timeout, follow=args.follow, mount=args.mount
    213 )

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/mpr.py:108, in MPRemote2.run_cell(self, cell, timeout, follow, mount)
    105     run_cmd = f'mount \"{Path(mount).as_posix()}\" ' + run_cmd
    107 # TODO: detect / retry / report errors copying the file
--> 108 result = self.run_cmd(
    109     run_cmd,
    110     stream_out=True,
    111     timeout=timeout,
    112     follow=follow,
    113 )
    114 # log.info(_)
    115 # log.info(f.name, \"copied to device\")
    116 Path(f.name).unlink()

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/mpr.py:61, in MPRemote2.run_cmd(self, cmd, auto_connect, stream_out, shell, timeout, follow)
     59         log.warning(f\"cmd is not a string: {cmd}\")
     60 log.debug(cmd)
---> 61 return ipython_run(
     62     cmd, stream_out=stream_out, shell=shell, timeout=timeout or self.timeout, follow=follow
     63 )

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/interactive.py:108, in ipython_run(cmd, stream_out, timeout, shell, hide_meminfo, store_output, log_errors, tags, follow)
    100     process = subprocess.Popen(
    101         cmd,
    102         shell=False,
   (...)
    105         start_new_session=False,
    106     )  # ,  universal_newlines=True)
    107 except FileNotFoundError as e:
--> 108     raise FileNotFoundError(f\"Failed to start {cmd[0]}\") from e
    110 assert process.stdout is not None
    111 if follow == False:
    112     # we do not need to follow the output of the command
    113     # just return

FileNotFoundError: Failed to start m"
}

better output for cell magics

normal cells show the eval of the last (non-blank) line
(its complicated to do so , but it allows for simple use)

the %%micropython cell magic does not do this

# %%micropython
msg = "Hello World"
msg

implementation option :

take the last line
check if we can eval that in Cpython ( or use the ipython methods for that ?)
( if syntax error ) then execute the entire cell as normal ( not beatiful, but effective )
otherwise
- remove the last line
- run the first part of the cell ( if any)
- eval the last line : mpremote resume eval "msg"
  combine the results

try mimetype : "text/x-python" for --info output

text/plain

text/x-python

pretty colors - but extra spaces .... Not sure if it is worth it

run mpy-cross on cell before execution

add a --mpy or --mpy-cross option to %%micropython that cross compiles the cell before executing it on the mcu

can mpremote run a foo.mpy file ?
Waiting for micropython/micropython#8744
what version of mpy-cross to use
- assume most recent version,
- allow version to be specified

add line magic --bootloader

    @argument("--bootloader", action="store_true", help="make the device enter its bootloader")
....
        elif args.bootloader:
            self.MCU.run_cmd("bootloader")

Store outcome of test runs per port / board in the cell metadata

%mpy --info --store
can store all CPU info in the notebook metadata

cell magic %% micropython --store

test outcomes ( OK / Failure or full details could then be stored per cell
This makes testing the same code , and collecting the information across different ports / version much simpler
how to store the data - reliably - no formal APIs ?
https://github.com/christianp/jupyter-interactions-site/blob/master/get_metadata.py
Need a way to report
need a way to specify the distinguishing factor ( port / version / board et )
how to clear the stored metadata

cell %%magic appears to prevent pylance from running in that cell

If there is no way to resolve this, then the magics make little sense as they would require learners to need to choose between

being able to run the cell's code on a micropython device
OR
having type checking and stub lookup from locally installed stubs for micropython

Note: that GH copilot still proposes code in the cells

ref:

microsoft/vscode-jupyter#13327

Recommend Projects

josverl / micropython-magic Goto Github PK