simonw / markdown-to-sqlite

CLI tool for loading markdown files into a SQLite database

markdown-to-sqlite

CLI tool for loading markdown files into a SQLite database.

YAML embedded in the markdown files will be used to populate additional columns.

Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...

For example:

$ markdown-to-sqlite docs.db documents file1.md file2.md

Breaking change

Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.

markdown-to-sqlite's Issues

Separated columns for headers?

Thanks for this tool and datatasette wonderful ecosystem!

Sometimes we deal with collections of documents having the same structure (eg .docx) that can be transform to markdown thanks to pandoc for naming one other great tool.

Would it be possible to generate SQLite columns for headers (h1, h2...) beside paragraphs for easier comparison of recurring content?

h1	h2	h3	content
h1 title	h2 subtitle	h3 subsubtitle	markdown paragraph

Might be out of scope though. Thx

how to use YAML for extra fields?

In the readme you mention something about embedded yaml and extra fields, but I cannot see anything documented about it. How does it work? What do I need? I would love to have more data in my database instead of having to alter the generated database directly.

I love this tool, thank you!

TypeError generated for markdown files with tables

There may be other instances where this TypeError failure occurs, but the markdown attached (its-science.md) will generate the following error:

Traceback (most recent call last):
  File "/Users/ryan/Documents/ryancheley.com/venv/bin/markdown-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/ryan/Documents/ryancheley.com/venv/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ryan/Documents/ryancheley.com/venv/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/ryan/Documents/ryancheley.com/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ryan/Documents/ryancheley.com/venv/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/ryan/Documents/ryancheley.com/venv/lib/python3.9/site-packages/markdown_to_sqlite/cli.py", line 27, in cli
    doc = {
TypeError: 'str' object is not a mapping

Should be `markdown-to-sqlite DBNAME PATHS`, not `markdown-to-sqlite PATHS DBNAME`

For consistency with other x-to-sqlite tools. I'll release this as 1.0 to signify the breaking change.

Invalid value for PATHS

I am trying to generate a database to run inside docker. However, while in my terminal this works fine

$ markdown-to-sqlite blog.db posts "./_blog/**/*.md"

when it runs inside the docker I get the following error:

Error: Invalid value for 'PATHS...': Path './_blog/**/*.md' does not exist.
Error: error building at STEP "RUN markdown-to-sqlite blog.db posts "./_blog/**/*.md"": error while running runtime: exit status 2

And I am not quite sure why it does not. I guess it has to do with the shell it runs on (on my machine I am running zsh while the docker one is just sh) but I do not know how work around it.

Any and all help will be much appreciated!