Comments (8)
@timothymillar Thanks for linking to your filter!
I'm not sure at this point if I want to include filters within Codebraid itself. I've looked into Jupyter conversion some more, and it appears that filters can only handle a very limited set of cases.
However, I think it could be worthwhile to start collecting Codebraid-related filters. I can create a Codebraid wiki page for filters and link to your filter from there for now. Then perhaps at some point it may be worth creating a separate repo for filters.
from codebraid.
I've done some research and experimentation, and have a proposal for how to proceed with this.
There is a subset of Codebraid features that maps exactly to a Jupyter notebook, so I think it makes sense to start with support for that and then gradually add support for other things (in some form) from there. So to start with, implementation is simpler at the cost of some documents failing to convert.
Since Pandoc already has ipynb support and Codebraid is based on manipulating the Pandoc abstract syntax tree (AST), I think the easiest way forward is to convert a Codebraid Pandoc Markdown document into Pandoc's Markdown representation of a Jupyter notebook. Then Pandoc can create the actual notebook from there. Converting from Jupyter notebook to Codebraid is just the reverse, again with Pandoc's Markdown notebook as an intermediary.
I've done a few experiments, and this looks like a straightforward process for the subset of Codebraid features that maps exactly to a Jupyter notebook. I can add support for this as I have time, and then that will provide a starting point for working on the more difficult cases as they are needed. It's possible that many people who want to convert back and forth won't need the features that are especially difficult to translate.
@mwouts I can plan on adding options to the codebraid
executable that will convert a Codebraid document into Pandoc's Markdown representation of a notebook and also to ipynb, plus options to convert from those formats back to Codebraid. For jupytext, would you prefer the capability to use the relevant part of Codebraid as a library rather than using the codebraid
executable via subprocess? If so, are there any particular features you need or that are useful? Either way, Codebraid would run Pandoc as a subprocess to handle some of the conversions.
from codebraid.
A filter to convert to/from jupytext and Jupyter notebooks could be nice. I've thought about the possibility occasionally, but haven't actually attempted anything.
I can't invest any significant time in this, given my other projects and commitments. I'm happy to answer questions about how Codebraid works to help in the creation of filters, and might have time to help with converting a basic subset of Codebraid features to/from jupytext/Jupyter.
There are a few questions that would be useful to consider in thinking about this:
- What do you gain by jupytext/Jupyter conversion?
- I'm guessing that the main thing is the interactive notebook editor. Is there anything else?
- If it is just for the notebook editor, it may also be worth thinking about ways to enhance Codebraid to make editing work better.
- What do you lose by jupytext/Jupyter conversion?
- If all you are using is code blocks with
.cb.nb
, with a Jupyter kernel, for a single language, with a single session, without any special display settings, then it is possible to convert to/from jupytext/Jupyter without losing anything. Writing filters to handle that case would probably be easy, and could be worthwhile. This probably wouldn't have any particular advantages compared to other Jupyter-notebook-as-Markdown formats. But it could be useful if you're using Codebraid for lots of additional documents and want to have everything in the same format. - Codebraid has lots of features that can't be converted to a Jupyter notebook. I've listed some of the things that are difficult below. You could figure out a way to convert to/from Jupyter without losing anything, but to do that, you would probably have to put a lot of things in specially marked code blocks (or something similar), since there isn't really a way to render them within a Jupyter notebook. At this point, the value of the notebook as an editor may start to become limited. Here are some things that are difficult to represent in a notebook:
- Codebraid can run code with the built-in system, and that can't necessarily be replaced with a Jupyter kernel.
- Within a single document, Codebraid can use multiple Jupyter kernels and multiple separate sessions per kernel. You would probably just have to choose one session for one language to be executable interactively, and would have to convert all other executable code into specially marked static code blocks.
- Codebraid can run inline code, within paragraphs. I think you can get a limited subset of these capabilities by using notebook extensions.
- Codebraid can show code and its output in separate locations in a document, with either appearing first.
- Codebraid allows customization of what a Jupyter kernel displays. I suppose there might be a way to use Jupyter cell tags to handle this...that might save the display information after you convert back to Codebraid, but probably wouldn't be able to affect the display in the notebook.
- If all you are using is code blocks with
from codebraid.
While I am aware of all the limitations of jupyter compared to codebraid, I think having an interactive environment where you can execute chunks and see the result would be very powerful (basically compilation vs. REPL).
Right now I can only think of jupyter and its popularity. Leveraging it, even for a subset of codebraid, would add a lot of value.
from codebraid.
I'm also interested in codebraid+jupytext but I'm not sure I have quite the same use-case as @teucer
My use-case is leveraging jupytexts command-line tools to convert scripts (.py, .R, .sh etc) to markdown.
I want to prototype small workflows in a plain scripts then convert to a markdown file and execute code in notebook mode for documentation purposes.
Previously I had used Pweave for this but It seems to be unmaintained (for now) and I like that codebraid can work without jupyter kernels.
I have implemented one-way conversion from jupytexts markdown* to a codebraid 'notebook' using a pandoc filter that inserts the .cb.nb
class into every code block.
It also checks for jupytext style metadata specifying a jupyter kernel and (if present) specifies the kernel in the first code block.
*EDIT: the filter works on pandoc ast (of course) so it can convert from any format supported by pandoc including .ipynb.
Pandoc can then be used to convert back to a notebook.
The filter is here
It can be used like so:
pandoc <file> --filter cbnb.filter.py --to markdown ...
@gpoore would you be interested in a PR to add something like this?
from codebraid.
Hello @gpoore , thank you for considering this. I'll give my view point on a few questions, hopefully that can help:
What do you gain by jupytext/Jupyter conversion?
As you mention, there are two direct benefits:
- Open/edit codebraid documents as notebooks in Jupyter
- Convert codebraid documents from/to other formats (notebooks & scripts)
If we work on this, an indirect benefit will be a good certification on the round trip between .ipynb
and codebraid
documents - when we add a new format to Jupytext, we add a series of tests and example notebooks to make sure that no information is lost on round trips.
Codebraid has lots of features that can't be converted to a Jupyter notebook
I agree. We will have to determine how to represent these features in a notebook, even if they are not active there (until someone develops a Jupyter extension to activate them). Let me also note that we have had the same challenge for the R Markdown and for the Myst-Markdown formats. In practice, my target is that the same document should run, as much as possible, both in Jupyter and in the natural renderer (for instance, in R Markdown we have to comment out Jupyter magic commands because knitr
can't run them).
If you want to go further, I propose that someone with a good knowledge of codebraid
writes two Python functions, one for converting a codebraid
document to a Jupyter notebook (using new_notebook
, new_code_cell
etc from nbformat.v4.nbbase
), and the other for doing the reverse. With that I could easily test the round trip at a larger scale, and then propose a version of Jupytext extended to codebraid
. What do you think?
from codebraid.
Hi @gpoore , these are great news!
I think I'd prefer to use codebraid
as a library (as we do for the md:myst
format) rather than calling it via subprocess (as we do for the md:pandoc
format), as in my experience it is faster and easier to debug.
To integrate codebraid
into jupytext
, I'll need two functions, one to convert codebraid
text (a string) to nbformat
notebooks (I mean, what you get when you do nbformat.read("notebook.ipynb", as_version=4)
), and the other to do the opposite.
I would also have use of the following:
codebraid
's format version number (maybe you want it to be equal to codebraid's version number?), and the minimal version number that the installed version ofcodebraid
can read. For ourmd
format this is for instance: https://github.com/mwouts/jupytext/blob/bc1b15935e096c280b6630f45e65c331f04f7d9c/jupytext/formats.py#L73-L85- a function that can tell me if a string is a
codebraid
document, likematches_mystnb
for themyst
format at https://github.com/mwouts/jupytext/blob/bc1b15935e096c280b6630f45e65c331f04f7d9c/jupytext/formats.py#L289-L290 - one or more sample
codebraid
documents to activate the round-trip tests on them
from codebraid.
Not having followed the entire discussion, I still would like to share my use-case:
For multiple Python-related classes, I am using RISE to present, in an iteractive fashion, aspects of the language and libraries. RISE has the great advantage of allowing ad-hoc variations of code ("What would happen if we changed this to ....?"), which adds a lot of value to the audience, IMO.
For preparation and reviewing, I provide lecture materials in multiple formats, namely
- a PDF document,
- an HTML version for use inside a Learning Management System, and
- PDF versions of the slides, using a Beamer template.
All this is being produced from a version-controlled, authoritative Markdown document per unit using a sophisticated Pandoc workflow with filters etc.
Now, Jupytext does a pretty good job of creating and keeping in sync a Markdown and an *.ipynb
version of the main document, but, by design (and to the best of my knowledge), strips off all output cells and there is no way to preserve them.
But for lecture notes, you typically want to show the code snippets and their effects.
Despite quite some research, I do not have an ideal solution; starting from the *.ipynb
document via Pandoc to Markdown is painful, as e.g. references (@foo2022a
) are escaped and inside fenced divs etc.; one would have to use a complicated filter to clean up.
It is a lot easier starting from the Jupytext-provided Markdown document, except for a few quirks and the missing output cells.. IMO, a straightforward solution would be for Jupytext to add CodeBraid classes to PandocCode blocks, as follows:
Jupytext Output:
```python
for i in range(3):
print(i)
```
With CodeBraid classes:
```{.python .cb-run .cb-nb}
for i in range(3):
print(i)
```
The same could be done for Bash
and other supported types of code blocks.
From an implementation point of view, this could be handled by
- a Pandoc filter that simply adds the CodeBraid classes to code blocks, or
- by Jupytext as an option for Markdown output, which I would prefer.
The actual execution of CodeBraid could be left to the workflow that processes the Markdown representation; IMO, there is no need to hard-wire the two components.
As for interactivity: I would not spend too much effort on this, because CodeBraid and Jupyter cover very different parts of the spectrum - for a highly interactive class, changing and running code snippets as we go is much better in Jupyter Notebook than in any more advanced Markdown-Workflow (I would also not use Quarto in here.).
Apologies if this is lengthy and a bit off topic; I really appreciate your efforts on the CodeBraid and Jupytext sides; they help tremendously in doing better in teaching and research! Hence, a huge thank you to @gpoore and @mwouts!
from codebraid.
Related Issues (20)
- Pandoc Version Read Incorrectly HOT 1
- Errors with pandoc 2.11 HOT 3
- Search path for Python executable on Windows
- --citeproc Pandoc option HOT 1
- Failure when specifying local MathJax HOT 2
- Use of inline html code characters appears to break source file parsing HOT 2
- Working with multiple source files w/ .cb.nb HOT 2
- can't run codebraid with new version of jupyter-client HOT 4
- What is the status of per-chunk caching? Is it supported/planned? HOT 8
- `codebraid` can't handle raw latex HOT 8
- Custom executable / custom language HOT 4
- How to use the Julia jupyter kernel to generate plots in output HOT 4
- Does codebraid work with Python Anaconda environments? Trouble plotting with python from jupyter notebook HOT 7
- Question about removing extraneous markdown characters from output HOT 2
- --only-code-output doesn't support example=true
- Overly aggressive caching HOT 3
- Markdown strings don't render normally HOT 2
- Feature request: Force execution of all code cells without adding `.cb-run` and `.cb-nb` HOT 3
- Feature request: override stdout code block language HOT 2
- host a website HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codebraid.