compilerla / intake-html-table

Intake plugins for HTML tables.

intake-html-table

Intake plugin for HTML tables.

Installation

pip install -e git+https://github.com/compilerla/intake-html-table@main#egg=intake-html-table

Or

git clone https://github.com/compilerla/intake-html-table
cd intake-html-table/
pip install -e .

Usage

See examples/notebook.ipynb or view on nbviewer for more.

From an intake catalog

Use the html_table driver to read data from HTML tables. Pass additional kwargs to pandas.read_html():

metadata:
  version: 1
sources:
  table:
    description: Read from an HTML table with id=data, skipping the first 2 rows
    driver: html_table
    args:
      urlpath: "https://example.com/"
      attr:
        id: data
      skiprows: 2

Use the apache_dir driver to read a catalog from an Apache Server directory:

metadata:
  version: 1
sources:
  ncei:
    description: National Centers for Environmental Information data catalog
    driver: apache_dir
    args:
      urlpath: "https://www.ncei.noaa.gov/data/"

Tests

Run the test suite (from the root of the repository):

coverage run -m pytest

To view the coverage report with indicators for untested (missed) lines:

coverage report -m

To upate the README badge from the latest test run:

coverage-badge -f -o tests/coverage.svg

The -f argument ensures the existing badge is overwritten.

Tests also run via GitHub Action on events against the main branch.

intake-html-table's Issues

Auto-generate and update coverage badge

We have the coverage-badge package available, which generates an SVG from the most recent coverage run, with a simple command:

coverage-badge -f -o tests/coverage.svg

How to run this automatically at some point when new code is introduced?

See #9 and #10 for more history.

Publish to PyPI

We should have proper pip install support.