bloomberg / ipydatagrid

Fast Datagrid widget for the Jupyter Notebook and JupyterLab

ipydatagrid

Fast Datagrid widget for the Jupyter Notebook and JupyterLab

Menu

• Usage and Examples
• Installation
• Development installation
• Contributions
• License
• Code of Conduct
• Security Vulnerability reporting

Usage and Examples

A fully-featured DataGrid interface

Highly performant and fully integrated with ipywidgets

Customize the way data is represented in your grid using a variety of renderers

Enjoy a sophisticated selections model with two-way data binding

Conditional formatting powered by Vega Expressions

Tutorial and example notebooks can be found in the /examples directory.

Installation

If using JupyterLab, ipydatagrid requires JupyterLab version 3 or higher.

You can install ipydatagrid using pip or conda:

Using pip:

pip install ipydatagrid

Using conda:

conda install -c conda-forge ipydatagrid

If you are using Jupyter Notebook 5.2 or earlier, you may also need to enable the nbextension:

jupyter nbextension enable --py [--sys-prefix|--user|--system] ipydatagrid

NOTE: For examples using Scales from bqplot to work as intended, the bqplot notebook and lab extensions must be installed as well. See the bqplot repo for installation instructions:

Development installation

For a development installation:

git clone https://github.com/Bloomberg/ipydatagrid.git
cd ipydatagrid
conda install ipywidgets=8 jupyterlab
pip install -ve .

Enabling development install for Jupyter notebook:

jupyter nbextension install --py --symlink --sys-prefix ipydatagrid
jupyter nbextension enable --py --sys-prefix ipydatagrid

Enabling development install for JupyterLab:

jupyter labextension develop . --overwrite

Note for developers: the --symlink argument on Linux or OS X allows one to modify the JavaScript code in-place. This feature is not available with Windows.

If you are changing TypeScript code you can watch for code changes and automatically rebuild using

jlpm watch

in one terminal and

jupyter lab

(or jupyter notebook or similar) in another.

Contributions

We ❤️ contributions.

Have you had a good experience with this project? Why not share some love and contribute code, or just let us know about any issues you had with it?

We welcome issue reports here; be sure to choose the proper issue template for your issue, so that we can be sure you're providing the necessary information.

Before sending a Pull Request, please make sure you read our Contribution Guidelines.

License

Please read the LICENSE file.

Code of Conduct

This project has adopted a Code of Conduct. If you have any concerns about the Code, or behavior which you have experienced in the project, please contact us at [email protected].

Security Vulnerability Reporting

If you believe you have identified a security vulnerability in this project, please send email to the project team at [email protected], detailing the suspected issue and any methods you've found to reproduce it.

Please do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports private until we've had an opportunity to review and address them.

ipydatagrid's Issues

py2vega future

cc. @kaiayoung @mbektasbbg @supriyakhandekar @ibdafna

@supriyakhandekar 's issue on py2vega made me rethink the py2vega scope QuantStack/py2vega#36

Vega expressions are expressions, hence they do not support assignments, for-loops, while-loops, if-conditions.

I wanted to go around it and added fake assignment and if-condition support in py2vega. Which I thought was nice at first. But I fear there would be feature requests for for-loops, while-loops, class definitions and such, which will be far from trivial to implement. Its complexity could easily increase. Also, the pseudo if-condition support might bite users more than once.

I feel like we should maybe reduce the py2vega scope, and remove Python functions support (as well as assignment support). Supporting only lambdas and Python code strings (actual expressions) as input, which means only supporting code that can be executed with a Python eval call. This would make it clearer to the user what py2vega can and cannot do.

Please tell me what you think.

Come with an automatic way of setting the minimum grid size

Seems we generate extra space at the bottom of short grids:

FWIW I just checked on the binder link provided by the repo and I get the same effect, so at least you should be able to reproduce it easily...

We do have a workaround for it:

The issue you are seeing with the blank space is the default canvas size. We don't have an intelligent way (yet) of determining a good default size for a given grid, so they all have the same one, and I agree it does look a bit odd for smaller grids. The good news is that this is a layout property and as such you pass it as a parameter to the grid constructor:

from ipydatagrid import DataGrid
import pandas as pd

df = pd.DataFrame({"A":[1,2,3], "B":[4,5,6]})
grid = DataGrid(df, layout={"height":"120px", "width":"200px", "align-content":"center"})
grid

This issue is to track in finding an automated way of setting the grid size to a just-right size.

Exposing selection to Python

Should we expose the selection to Python? And if yes, how should we expose the selection to Python?

1- Should we expose the same structure as it is on the front-end? (list of selections, a selection being the indices of the upper-left cell and the lower-right cell)

2- Should we expose a list of cell indices for all the selected cells?

Header menu icons

I am finding menu icons taking space unnecessarily and confusing. It is looking like descending sorting state icon (by the way I think we need separate icons at least for asc, desc, filtered states).

Why dont we show the menu icon on hover and in a different shape than arrow down? and show an additional state icon when a filter/sort applied. What do you think?

Enhance readme including some gifs

a visual help in the Readme part is powerful
The little brother of ipydatagrip -ipysheet- includes some nice gifs in the Readme part. It's so easy to get an idea of the capabilities of the repo just with 3 seconds look at it.

Filtering multi index data always returns an empty grid.

The transform executors are not currently handling multi index data well. There is not a convention for this in Google Sheet, but we could look to pandas for some cues on how we may want this to function

Filter menu needs to support JLab themes

The interactive filtering menu still has fixed styling, and needs to use JLab CSS variables instead.

Testing

Do we have any strong opinions about JS testing frameworks? I've got a few PR's left on the filtering/sorting, but will probably shift to focus on testing after that. I've been looking at Jest, but was curious if there were any CI or other reasons that another one may work better for our needs?

@martinRenou @SylvainCorlay @mbektasbbg

Different (Text)Renderers based on column dtype?

Is your feature request related to a problem? Please describe.
Submitting a dict of TextRenderers for every column can be frustrating when many columns require the same renderer settings (as I imagine many people face). For example, in my case, I have several string type columns and many float type columns, but it is very manual to write out a dict for every column when only two different dict values are needed. The alternative I have come up with at the moment is to select the columns of one type by name from df.columns and use dict(zip(selected_columns, len(selected_columns)*[TextRenderer(...)])). While messy, it seems to work. But it is not robust to creation / deletion of columns and generally is very manual.

Describe the solution you'd like
It would be wonderful to be able to input to the DataGrid call to use one renderer for each column dtype. E.g. use one for all columns of dtype string and another for all columns of float dtype.

Describe alternatives you've considered
I suppose you could provide a single renderer for the entire grid (which I know is possible, but can't find any documentation on how). In this renderer, you would provide details on how to handle strings, floats, ints, etc as you desire and then as the renderer encounters each column, it can render appropriately. I'm not sure if this would be better. I think the best solutions would also continue to provide flexibility when needed to handle columns of the same dtype differently. E.g. it would be best if you could apply a certain rendering to all float type columns and then apply a different rendering to a specified float type column afterwards.

Automatic column widths?

Is your feature request related to a problem? Please describe.
No. This just seems like a very useful feature that would vastly enhance user experience with varying columns and many of them. It is frustrating to have to provide a dictionary of column widths when there are dozens and the required widths change depending on dynamic data inputs.

Describe the solution you'd like
Ideally, it would be best to have a simple boolean flag (e.g. auto_column_size) in the ipydatagrid.DataGrid() call.

Describe alternatives you've considered
One (hopefully intermediate) alternative I considered would be to at least provide a "get_column_sizes" method from a current data grid where I have manually shrunk and expanded the different columns to get a good fit. It would be great to at least be able to get these and input it into future versions of the grid. In my case, I have the grid in an ipywidgets layout and dynamically update it based on other widgets. Thus, having this dict of column widths would prove very useful, allowing me to set it once and then reuse it in future.

voila / ipyvuetify /ipydatagrid issues

I am running a notebook (Jupyter lab) in a Jupyter hub and I got the following situation:

I imported ipydatagrid as such import ipydatagrid.
Ipydatagrid ended not being used in the whole notebook.

Now as it is seen on the left when running the notebook the GUI (ipyvuetify) is rendered.
But on the right as can be seen voila DOES NOT RENDER THE IPYVUETIFY COMPONENTS.

Now if I comment out the import ipydatagrid line at the beginning then the result is that voila renders:

This was after i did this:

Ctrl-select row does not toggle selection

It only makes the selection marker display as darker.

Large dataframe tabulation - slow initialisation/visualisation

Hi

I have a use case where I want to be able to display large dataframes in an interactive table.
My definition of 'large' is a frame of > 200k rows; 50 columns.

I have tried other table widgets (qgrid, ipyaggrid) - both struggle to render this dataframe size. ipyaggrid in particular takes an incredibly long time to render the table.

I was hoping for better performance from ipydatagrid -> unfortunately performance appears very similar. Other than trying perspective-python (which I can't currently build :( ), are there any suggestions for improving performance?

Example of datagrid using cell renderer that renders an ipywidget

Can an example notebook be provided demonstrating the embedding of an ipywidget within a cell ?

E.g. DatePicker (for date cells) ? ipyupload for a file selection cell ? checkbox for a toggle cell ?

Warning when creating a datagrid

I get the following warning when creating a table:

/home/martin/miniconda3/envs/ipydatagrid/lib/python3.7/site-packages/pandas/io/json/_table_schema.py:82: UserWarning: Index name of 'index' is not round-trippable
  warnings.warn("Index name of 'index' is not round-trippable")

Error displaying widget: model not found

after installing ipydatagrid and cloning the repo and trying to run the DataGrid example I get the following error:
Error displaying widget: model not found

every time a cell ends trying to display the data grid.

How can I found out what is exactly the widget missing?

And I reinstalled ipywidgets and ipydatagrid

Extra blank space at the bottom of the widget

FWIW I just checked on the binder link provided by the repo and I get the same effect, so at least you should be able to reproduce it easily...

Conditional fromatting

cc. @SylvainCorlay @kaiayoung @mbektasbbg @maartenbreddels @wolfv

Here are some thoughts for conditional formatting:

1. Sandboxing

I wonder if we should use sandboxing techniques for conditional formatting.

I see some cons using Web Workers like it is done in ipysheet:

• It can be slow (~500 ms for styling cells of a sheet containing ~50 cells). That would slow down scrolling.
• We ask the user to provide JavaScript code, or Python code if using pscript. In the case of using pscript: it is not valid Python code, it has no access to the global scope of the Notebook, so it can be error-prone for the user.
• The execution of the function by the worker is asynchronous, but PhosphorJS needs a synchronous function: phosphorjs/phosphor#403

2. Check the code sanity

I thought about checking the code sanity. That would be done once, and then we can execute the code synchronously without using sandboxing (Web Workers or iframes). But I am not able to find an open-source sanitizer, and using a third-party library or custom code for sanitizing might pose security/certifiability issues.

3. Restricting user possibilities

My favorite solution (it is also more work) would be to restrict the user possibilities by providing customizable renderers,
e.g.

• a ColorMap renderer that provides defaults color maps (like does bqplot)
• a TernaryOperator for which the user would provide a basic operator like for the filter (>, >=, contains, ==, ...), a specific value, the value when the condition is true/false.

API ideas:

datagrid = JSONGrid(data, background_color=ColorMap('viridis'))
datagrid = JSONGrid(data, background_color=TernaryOperator('>', 0, 'white', 'red')) 

No scrollbars when size set with Layout()

It appears that the grid's scrollbars don't work when height/width values are set with the ipywidgets Layout() item. Many producers will be setting a custom size to work with the UI of their app, so we'll need to address this. I also ran into this issue when I had protoypyed a phosphorJS datagrid widget last year, and vaguely recall that I had solved it by getting the bounding box of the the div that ipywidgets.Layout() controls the size of, then directly setting that height/width on the grid's phosphor widget CSS.

from ipydatagrid import DataGrid
import pandas as pd
import numpy as np
import json

df = pd.DataFrame(np.random.randn(20,20))
DataGrid(
    data=json.loads(df.to_json(orient='table')),
    # Comment out the line below, and the horizontal scrollbars will work as intended
    layout={'height': '150px', 'width': '400px'}
)

Context menu icon not displayed the first time

The first time I render the table, the context menu icons are not displayed. Only when I trigger a resize the icon gets displayed, as shown in the following screenshot:

I wonder what the issue could be, maybe font-awesome was not loaded yet when the table gets displayed the first time?

Optimizing Python <-> JavaScript communication for streaming data

cc. @kaiayoung @mbektasbbg

When using ipywidgets, it is not supported to change a subset of the data. e.g.:

my_data_grid.data[26][32] = 3.4

will not send the proper message to the JavaScript side for it to update the grid view. For this to work, it is needed to assign the entire data again. Which means doing:

old_data = my_data_grid.data
old_data[26][32] = 3.4
# Making a copy of the data so that ipywidgets sees the change...
my_data_grid.data = old_data[:]

This might affect performances drastically for streaming data on big grids...

But there is a way around this. If we provide means to update the data using custom comm messages, we could have very decent performances. Which means doing something like:

class DataGrid(DOMWidget):
    [...]

    def update_cell(self, row, column, value):
        self.send({'action': 'update_cell', args: [row, column, value]})
    
    def update_row(self, row, value):
        self.send({'action': 'update_row', args: [row, value]})
    
    def update_column(self, column, value):
        self.send({'action': 'update_column', args: [column, value]})

and on the JavaScript side (this is pseudo code, not tested):

this.model.on('msg:custom', (message) => {
  if (message.action === 'update_cell') {
    const [row, column, value] = message.args;
    this.model.data[row, column] = value
  }

  [...]
});

Hit testing fails on corner-header menu icon

Just noted that the menu icon on the corner header isn't being hit tested, which we probably missed with all of the changes to bring in the selection/event model.

@mbektasbbg

How to deselect a selected row ?

Clicking on a datagrid row -> selects the row (assuming selection is set to 'row') - this is natural expected UI behaviour.

However clicking on a selected row - does not toggle the selection - this is natural expected UI behaviour.
How should one deselect a selected row ? (I've tried various combinations of key modifiers and mouse buttons).

Update py2vega version

We should use py2vega 0.5.0 and use the new Variable class to expose the cell variable.

Also, we should not expose x, y, height, width parameters.

What should do the DataGrid.view method Python side?

cc. @kaiayoung @mbektasbbg

Currently, the DataGrid.view method returns a new DataGrid which is the result of the view. As discussed last Thursday, maybe the behavior should be that it transforms the DataGrid output in-place, and we could have a way to revert the transformations.

Should we support both approaches?

Format numeric columns

Often I find myself in need of a way to format columns in a datagrid, such as setting the number of decimals.

I'd like the datagrid to take as (optional) input a dict of columns and their respective formats, similar to column_widths.
As far as I can see, that is not currently possible.

The only alternative I have considered is to use the built-in round() function prior to creation of the datagrid.

Conditional Formatting improvement ideas

Those are some improvements ideas following this PR: #10

• Add support for replacing a renderer, will be fixed by #14
• Support for AND and OR operators for the Predicate class. The OR operator is already implicitly implemented (when providing a list of predicates).
• If there is a change in the renderer model, only the region/column affected by this renderer should be repainted We now let phosphor handle the repaint.
• We should compute Scales domain automatically, given the data. This could be an optional feature, first because it could be slow for huge data, second because in the case of streaming data we cannot really know if the domain will stay the same.

Advanced ideas:

• Implement a custom AST grammar for compiling renderers from Python functions?
• Use vega-expression for conditional formatting functions

Large JSON files on the repo

We should consider using git lfs for large JSON files like trees.json. Also we should take care that those large JSON files are not shipped during releases.

License for cars.json and trees.json

Would anyone happen to have more information on the source of these files in the example notebooks? We may need to include licenses or be able to demonstrate that the data is in the public domain as part of the internal OSS process.

@SylvainCorlay @martinRenou

Pandas support

Maybe the Datagrid data trait could support Pandas dataframes as input automatically, by converting them to a JSON table schema using json.loads(df.to_json(orient='table'))

Header menu should move with grid when page scrolls

Something to look into after MVP

Include gif in github Readme.

Include gif in github Readme. The new 1.1.0 release include nice features as shown in this tweet/gif. Please include the gif in the girhub Readme. The same goes for a gif showing wrapping text in a cell. This is so powerful for millions of excel users that it would be very convenient to be shown straight away in Readme.

https://twitter.com/Itayd/status/1417362579215634437?s=19

Feature req: Editable by column

Would it be possible to make the editable flag take a dictionary of column names with corresponding flag on a per-column basis?

example custom renderer in examples/BoolRenderer.ipynb not rendering

Describe the bug
example custom renderer in examples/BoolRenderer.ipynb not rendering as expected.
instead it looks like generic placeholders.

To Reproduce
Launch the examples/BoolRenderer.ipynb on binder from the main Github readme.md page

Screenshots of bug

Expected behavior
Expected to see cells rendered as per the commented descriptions here:

Environment (please complete the following information):
launch binder on chrome

examples/CellEditing.ipynb - on_cell_changed does not fire from interactive edit

When running examples/CellEditing.ipynb....

on_cell_changed() does fire when datagrid.set_cell_value() is called to change the grid.

on_cell_changed() does NOT fire when cell selected interactively and cell value changed

I also note that the data member datagrid.data does reflect the cell change - after datagrid.set_cell_value() is called
It does NOT reflect the cell change when the cell is edited interactively.

Filtering not working for boolean data.

This is probably a quick fix, but we may also want to create some custom filtering options in the operator menu for boolean data. Sample code:

import pandas as pd
import numpy as np
import json
import random
from ipydatagrid import DataGrid, TextRenderer, Expr
n = 5
df = pd.DataFrame({
    'Value 1': [random.choice([True, False]) for i in range(n)],
    'Value 2': [random.choice([True, False]) for i in range(n)],
    'Value 3': [random.choice([True, False]) for i in range(n)],
    'Value 4': [random.choice([True, False]) for i in range(n)],
})

jsontable = json.loads(df.to_json(orient='table'))
DataGrid(data=jsontable)

Custom cell renderer

I'd like to create a custom renderer, but I'm not quite sure where to start. Would you mind giving some pointers?

I was hoping to create a renderer which would have two key features:

• Possible to draw custom graphics in the background (similar to how the BarRenderer works)
• Ability to update the shown values in the cell (and only specific cells) by direct javascript callback, and without syncing to python. Reason for this is because it is a performance-sensitive application with many fast updates.

JLAB v3.0.7

error installing jupyter labextension for ipywidgets

Describe the bug
Installation of ipywidgets jupyter labextension fails.

To Reproduce
Steps to reproduce the behavior:

1. $ pip install ipywidgets
2. $ jupyter labextension install @jupyter-widgets/jupyterlab-manager ipydatagrid
3. The following error occurred:

An error occured.
ValueError: "ipydatagrid" is not a valid npm package
See the log file for details:  /var/folders/z1/nhct2gps3y94tkbwqvvd7y_jw6p1l8/T/jupyterlab-debug-dse9w2rn.log

Expected behavior
I expected the installation to succeed without error messages

Screenshots
jupyterlab-debug-dse9w2rn.log

Environment (please complete the following information):

• Operating System and Version: Mac OS X 10.14.6
• Python version: 3.8.5 (anaconda3-2020.11)
• Jupyter version: $ jupyter --version

jupyter core     : 4.6.3
jupyter-notebook : 6.1.4
qtconsole        : 4.7.7
ipython          : 7.19.0
ipykernel        : 5.4.3
jupyter client   : 6.1.7
jupyter lab      : 2.2.6
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.0.8
traitlets        : 5.0.5

Additional context
Please advise

Sorting not reflected in selection checks

Describe the bug

When a sort is applied in the GUI, this is not reflected in the selection checks.

To Reproduce
Steps to reproduce the behavior:

1. Create a random dataset
2. Display an ipydatagrid of that dataset
3. In the GUI, sort by a given column
4. Select, and then observe the output of selected_cell_values, which reflect the original sorting

Expected behavior
I would expect that the returned values would be reflective of the in-GUI sorted values, or that they would not be available

Screenshots

Environment (please complete the following information):

• Ubuntu 20.04
• Python 3.9, ipydatagrid from conda-forge

Use `on_cell_change` (or other callbacks) for data input validation?

Is your feature request related to a problem? Please describe.

Given that a user can input a new value into a cell, is it possible to use the callback infrastructure to prevent invalid inputs such as out of bounds numbers or invalid types?

Describe the solution you'd like
A clear and concise description of what you want to happen.

Some way to do input validation on cells.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
N/A
Additional context
Add any other context or screenshots about the feature request here.
N/A

Command mode in jupyterlab

Describe the bug
Thanks for making a fantastic grid-widget! I am not sure if this is a bug or intended, but posting it anyway. After having made a selection in a grid, it seems that going into command mode in jupyter-lab does not work.

To Reproduce
Steps to reproduce the behavior:

1. Display a datagrid in jupyter lab
2. Select any of the cells
3. Press escape-key
4. Try using a command-mode button, such as add cell above with "a". Observe that nothing happens.

Expected behavior
Command mode work as usual.

Use py2vega for the filtering/sorting actions?

@kaiayoung I can try to do it if that's fine with you

How to add or remove a row?

Its great to see this library open sourced!

I have not found in the examples a way to add or remove rows from a grid.
I am using the grid for a user to enter weights of a different assets, but I would the user to be able to add more assets or remove them.

Changes to editable cells from the front-end are not percolating down to the back-end

Describe the bug
Changes to editable cells from the front-end are not percolating down to datagrid.get_visible_data(). This seems to be a regression from 0.2.17.

To Reproduce
Steps to reproduce the behavior:

1. Run:

from ipydatagrid import DataGrid
import pandas as pd

df = pd.DataFrame(data={"A":[1,2,3],"B":[4,5,6]})

datagrid = DataGrid(df, selection_mode="cell", layout={"height":"100px"}, editable=True)
datagrid

2. Click on a cell and change its value
3. Run:

datagrid.get_visible_data()

4. The resulting DataFrame does not include the change made to the cell.

Expected behavior
The resulting DataFrame should include the change made to the cell.

Environment (please complete the following information):

• Ubuntu Linux 18.04 on WSL
• Chrome
• JupyterLab3
• ipydatagrid 1.0.1

Additional context
Regression seems to stem from the Python side following the release code review

Scrollbar button icons are not showing up

Is it possible to add search functionality?

just came across this new widget and it looks awesome! there are many datagrid solutions available in Jupyter but each has there own drawback - this looks like a no compromise solution for every datagrid need. I can't wait to use it in my next project! One feature that i'd love to see:

Is your feature request related to a problem? Please describe.
When reviewing data in a datagrid, I find a general search bar to be very useful, and often a quicker route to finding what you want as compared to column filters

Describe the solution you'd like
I'd like the datagrid to have a optional search bar UI. Ideally with the following options:

• 1no search bar for the whole datagrid
• optionally adding a search bar for a given column heading

Describe alternatives you've considered
The column filters already implemented provide some of the required functionality.

Additional context
AG-Grid (and ipyaggrid) provide this functionality and it works really well. Something directly equivalent to that would be ideal.

How to reference Multi-index in row dimension?

Is your feature request related to a problem? Please describe.

I looked at the ConditionalFormatting.ipynb and I can see how to reference a Multi-index column, but how would you do the same for a Multi-index row?

Describe the solution you'd like
Some documentaation on how to reference a Multi-index row.

Describe alternatives you've considered
N/A

Additional context
N/A

.uniqueValues() on View doesn't support all cell regions

.uniqueValues() assumes the cell region is body, though there may be use cases in which it would need to return values for the corner-header. All of the values should be unique, but in the case of a categorical field used as primary key, users may want to use the interactive filtering to only show certain categories.

Is it possible to wrap text in a cell? (i.e. several lines)

When rendering cells containing large texts is it possible to wrap text in several lines?