GithubHelp home page GithubHelp logo

webgrid's Introduction

WebGrid

image

image

image

Introduction

WebGrid is a datagrid library for Flask and other Python web frameworks designed to work with SQLAlchemy ORM entities and queries.

With a grid configured from one or more entities, WebGrid provides these features for reporting:

  • Automated SQL query construction based on specified columns and query join/filter/sort options
  • Renderers to various targets/formats
    • HTML output paired with JS (jQuery) for dynamic features
    • Excel (XLSX)
    • CSV
  • User-controlled data filters
    • Per-column selection of filter operator and value(s)
    • Generic single-entry search
  • Session storage/retrieval of selected filter options, sorting, and paging

Installation

Install using `pip`:

pip install webgrid

Some basic internationalization features are available via extra requirements:

pip install webgrid[i18n]

A Simple Example

For a simple example, see the Getting Started guide in the docs.

Running the Tests

Webgrid uses Tox to manage testing environments & initiate tests. Once you have installed it via pip install tox you can run tox to kick off the test suite.

Webgrid is continuously tested against Python 3.6, 3.7, and 3.8. You can test against only a certain version by running tox -e py38-base for whichever Python version you are testing.

webgrid's People

Contributors

bchopson avatar bladams avatar colanconnon avatar ethanwillis avatar guruofgentoo avatar jsparksman avatar matthiaswh avatar mtbrock avatar pytrumpeter avatar rsyring avatar tjlevel12 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

webgrid's Issues

Consider reducing number of appveyor jobs

Is it necessary to test the base and i18n versions of the tests for every Python version for both 32 and 64 bit Pythons? This results in 12 runs of the test suite.

Unless we have code that changes based on architecture, it seems like we could cut down on the permutations here with something like I did in Keg:

  matrix:
    # Pre-installed Python versions, which Appveyor may upgrade to
    # a later point release.
    - PYTHON: "C:\\Python27"
      TOXENV: py27-{base,lowest,i18n}

    - PYTHON: "C:\\Python35"
      TOXENV: py35

    - PYTHON: "C:\\Python36"
      TOXENV: py36

    - PYTHON: "C:\\Python37"
      TOXENV: py37-{base,lowest,i18n}

Make the permutation runs on the version bookends and keep the stuff in the middle only testing the base run.

Consider rendering dates/times in HTML `time` tags

HTML5 has a time tag that's meant to represent dates and times semantically. It is better for indexing and accessibility. But it's also convenient for hiding machine-readable formats in the datetime attribute and displaying it differently.

<time datetime="20160309T130222.3022Z">March 9th</time>

Don't use column names/labels as CSS classes

Nick and I spent almost an hour tracking this one down: One of my grid columns was coming from our LookupMixin which has a column named label. I was showing that label on webgrid and you could filter it. But webgrid used the column name label as the class on the filter HTML. Bootstrap also has a label class. So you can imagine the styling was all wrong.

Basically this was an extremely surprising bug and we really should not be doing this. We should derive some highly unlikely name for our CSS classes instead of copying directly from the column names.

HTML characters are not escaped in multi-select filter

If a TextFilter is switched to a multi-select (via the multi-select toggle), values containing HTML characters are not escaped properly.

In the issue I ran into, one of my column values was '<Unassigned>', which resulted in the following rendered HTML:

<li>
    <label>
        <input type="checkbox" name="selectItemv1(class_name)" value="7">
        <unassigned></unassigned>
    </label>
</li>

Removed unused parameters from the URL

One of the artificats of having all the fields in the form but just hidden with CSS, is that our URLs get every possible parameter, making them very long. Example:

example.com/some-grid?session_key=kTVfyGmGQk4C&op%28date%29=eq&v1%28date%29=07%2F07%2F2018&v2%28date%29=&op%28event_code%29=&v1%28event_code%29=&op%28race_num%29=&v1%28race_num%29=&op%28race_status%29=&op%28archive_post_time_open_utc%29=&v1%28archive_post_time_open_utc%29=&v2%28archive_post_time_open_utc%29=&op%28bet_release_time_utc%29=&v1%28bet_release_time_utc%29=&v2%28bet_release_time_utc%29=&op%28wagers_requested_utc%29=&v1%28wagers_requested_utc%29=&v2%28wagers_requested_utc%29=&op%28archive_locked_message_utc%29=&v1%28archive_locked_message_utc%29=&v2%28archive_locked_message_utc%29=&datagrid-add-filter=&sort1=bet_release_time_utc&sort2=&sort3=&onpage=1&perpage=100

In that particular example, I only have three things set specifically: a filter, a sort, and a custom perpage value. I'd love it if only the fields with non-default values showed up in the URL.

Allow writing in xlsx format

xls has a hard upper limit of 65,000 rows. Some projects have many more than that displayed in a grid. Switching to xlsx will bump that limit to over 1million. While that is still an upper limit, that is much much higher.

add a post-init method

It would be helpful to have a method on the grid called at the end of the init process (i.e. after the columns are set up for the instance).

Improve license declaration for pypi

I got this email recently:

Hello!

You have recently released a new version of WebGrid. Thanks! It will
probably make a lot of people happy. There is a tiny issue with your
package on PyPI though: it is a bit difficult to guess the exact
license you use.


It is very clear that you use one of the BSD licenses. PyPI allows you
to use the following classifier:

    License :: OSI Approved :: BSD License

But it is not very precise. You can also use the "license" field in
setuptools, but there is no "fixed" format, so you could write "BSD2",
"BSD-2", "BSD-2-Clause", etc. This makes it hard for automated tools
to properly guess which BSD license you are using.

May I suggest using the aforementioned classifier *and* an SPDX
identifier (https://spdx.org/licenses/) in the "license" field?


Thanks,
Cyril Roelandt

(This email was automatically generated, but you can answer it! A real
human being will read your answer.)

Add the ability to set a `column_group` on a column

  • Update renderer
  • Get columns set up

Slack reference

class SentTimeGroup(ColumnGroup):
	label = 'Sent Time (s)
class RacesGrid(Grid):
    RaceIdColumn('Id', ents.Race.id)
    Column('Max', some_aggregate, filters.NumericFilter, group=SentTimeGroup)

Randy Syring [1:56 PM]
There is a lot of wasted space in the header cells b/c of the need to repeate "Sent Time (s)" and "Receive Time (s)" in columns.
What if we created the ability to set a column_group on a column so that the grid looked like: example

Matt Lewellyn [2:00 PM]
adding that to webgrid or rolling it into your project would be fairly trivial, I think

Bill Adams [2:01 PM]
That would require updates to the renderers. I'm not sure how difficult it would be to get that property passed through on the column to the renderer
It could possibly be done without any updates to webgrid

Randy Syring [2:02 PM]
My thinking was to roll it into webgrid.

Matt Lewellyn [2:03 PM]
I agree it might be useful. The renderer piece of it should be the trivial part. Could be more of a task to get the columns set up, with the way webgrid creates those for the grid instance

Reduce URL size

Grids tend to generate very large URLs when filters are applied. Some larger grids have gotten to the point where the URL is overflowing uwsgi's default buffer sizes.

To reduce the size of the URL, when the filter form is submitted, it should only include fields that are actually set.

BREAKING: refactor renderer connections

Currently, with the way Excel exports work, format-specific information has a lot of fingers reaching into the main grid code.

  • there are methods in the framework plugins and the HTML renderer directly named for an xls export
  • columns have xls-specific init parameters and methods
  • Grid.set_renderers sets direct attributes for the renderers (e.g. Grid.html, Grid.xls)

This needs to get refactored. When #21 happens, the first point above should be addressed in a non-breaking manner. The other items will break compatibility and should be reserved for a more major revision:

  • Grid should namespace renderer attributes set in set_renderers
  • the xls-specific methods in columns will need to be thought through a bit more. Renderer-specific operations should be handled within the renderer, which may mean passing an inner class of the renderer to the column to keep track of those things.

support arrow dates

Excel export and filter functionality do not understand how to export dates that are arrow objects.

Add default `edit-link` and `delete-link` css classes

Keg-Auth users grid depends on the edit-link and delete-link classes, but no default CSS is supplied for them. Without any css it's hard to tell that the actions are there. I added these CSS classes to make the actions work as expected.

.edit-link:before {
    position: relative;
    padding: 2px;
    font-family: "FontAwesome";
    content: "\f040";
}

.delete-link:before {
    position: relative;
    padding: 2px;
    font-family: "FontAwesome";
    content: "\f1f8";
}

TypeError: '<=' not supported between instances of 'datetime.datetime' and 'NoneType'

TypeError: '<=' not supported between instances of 'datetime.datetime' and 'NoneType'
(7 additional frame(s) were not displayed)
...
File "commonbwc/lib/views.py", line 220, in manage_assign_vars
dg = self.manage_init_grid()
File "commonbwc/lib/views.py", line 236, in manage_init_grid
return self.grid_init()
File "commonbwc/lib/views.py", line 96, in grid_init
grid.apply_qs_args()
File "webgrid/init.py", line 802, in apply_qs_args
v2,
File "webgrid/filters.py", line 604, in set
if self.value1 <= self.value2:

TypeError: '<=' not supported between instances of 'datetime.datetime' and 'NoneType'

forced filters vs default filters

Some filters are necessary to be present, in order to reduce the amount of data a grid is processing. Otherwise the results could take a very long time to complete. In other cases, they are simply user helpers which should be allowed to be cleared.

What we need is the concept of a "forced" filter, introduced by having is_forced=True in the filter constructor. Forced filters will need to require a default op, and in the UI would remove the possibility of disabling them.

"Default" filters would then need the grid to tell them if they should be processed with the default op. If the user has selected other filters, the default should be allowed to lapse.

Support case-insensative searches

Postgres search will be case sensitive by default. MSSQL will not. I believe in most cases, an insensitive search would be preferred.

apply_qs_args() ends up with dict instead of MultiDict

  File "agariadm/views/private.py", line 54, in make_grid
    g.apply_qs_args()
  File "webgrid/__init__.py", line 792, in apply_qs_args
    v1 = args.getlist(filter_v1_qsk)
  AttributeError: 'dict' object has no attribute 'getlist'

I can't say I really understand all this code anymore, but the only thing that makes sense in this code:

session_args = self.get_session_store(args, session_override)

is that the result of self.get_session_store() can be a normal dictionary under some circumstance. Code later on is assuming a MultiDict, hence the exception when getlist() is called.

give better context in exceptions during filter setup

OptionsFilterBase can give an exception ValueError: value_modifier argument set to "auto", but the options set is empty and the type can therefore not be determined. But there is no easy way to determine which filter on a grid is causing the issue. The exception should include the name of the filter.

enable python3 testing for blazeweb

Tests on the blazeweb testing app are relegated to python 2. I believe this happened because when webgrid was updated to support python 3, blazeweb did not support that yet.

Add helpers for testing grids

There should be a base testing class for testing webrid grids. We use the following in one of our projects:

import re
import urllib.parse

from blazeutils.spreadsheets import workbook_to_reader
import flask
import flask_login
from pyquery import PyQuery
import pytest
import sqlalchemy


def query_to_str(statement, bind=None):
    """This function is copied directly from sqlalchemybwc.lib.testing

        returns a string of a sqlalchemy.orm.Query with parameters bound
        WARNING: this is dangerous and ONLY for testing, executing the results
        of this function can result in an SQL Injection attack.
    """
    if isinstance(statement, sqlalchemy.orm.Query):
        if bind is None:
            bind = statement.session.get_bind()
        statement = statement.statement
    elif bind is None:
        bind = statement.bind

    if bind is None:
        raise Exception('bind param (engine or connection object) required when using with an '
                        'unbound statement')

    dialect = bind.dialect
    compiler = statement._compiler(dialect)

    class LiteralCompiler(compiler.__class__):
        def visit_bindparam(
                self, bindparam, within_columns_clause=False,
                literal_binds=False, **kwargs
        ):
            return super(LiteralCompiler, self).render_literal_bindparam(
                bindparam, within_columns_clause=within_columns_clause,
                literal_binds=literal_binds, **kwargs
            )

    compiler = LiteralCompiler(dialect, statement)
    return 'TESTING ONLY BIND: ' + compiler.process(statement)


class GridBase(object):
    grid_cls = None
    filters = ()
    sort_tests = ()

    @classmethod
    def setup_class(cls):
        cls.user = User.testing_create()

        if hasattr(cls, 'init'):
            cls.init()

    def assert_in_query(self, look_for, **kwargs):
        pg = self.get_session_grid(**kwargs)
        query_str = query_to_str(pg.build_query())
        assert look_for in query_str, '"{0}" not found in: {1}'.format(look_for, query_str)

    def assert_not_in_query(self, look_for, **kwargs):
        pg = self.get_session_grid(**kwargs)
        query_str = query_to_str(pg.build_query())
        assert look_for not in query_str, '"{0}" found in: {1}'.format(look_for, query_str)

    def assert_regex_in_query(self, look_for, **kwargs):
        pg = self.get_session_grid(**kwargs)
        query_str = query_to_str(pg.build_query())

        if hasattr(look_for, 'search'):
            assert look_for.search(query_str), \
                '"{0}" not found in: {1}'.format(look_for.pattern, query_str)
        else:
            assert re.search(look_for, query_str), \
                '"{0}" not found in: {1}'.format(look_for, query_str)

    def get_session_grid(self, *args, **kwargs):
        flask_login.login_user(kwargs.pop('user', self.user), force=True)
        g = self.grid_cls(*args, **kwargs)
        g.apply_qs_args()
        return g

    def get_pyq(self, grid=None, **kwargs):
        pg = grid or self.get_session_grid(**kwargs)
        html = pg.html()
        return PyQuery('<html>{0}</html>'.format(html))

    def get_sheet(self, grid=None, **kwargs):
        pg = grid or self.get_session_grid(**kwargs)
        xls = pg.xls()
        return workbook_to_reader(xls).sheet_by_index(0)

    def check_filter(self, name, op, value, expected):
        qs_args = [('op({0})'.format(name), op)]
        if isinstance(value, (list, tuple)):
            for v in value:
                qs_args.append(('v1({0})'.format(name), v))
        else:
            qs_args.append(('v1({0})'.format(name), value))

        def sub_func(ex):
            url = '/?' + urllib.parse.urlencode(qs_args)
            with flask.current_app.test_request_context(url):
                if isinstance(ex, re.compile('').__class__):
                    self.assert_regex_in_query(ex)
                else:
                    self.assert_in_query(ex)
                self.get_pyq()  # ensures the query executes and the grid renders without error

        def page_func():
            url = '/?' + urllib.parse.urlencode([('onpage', 2), ('perpage', 1), *qs_args])
            with flask.current_app.test_request_context(url):
                pg = self.get_session_grid()
                if pg.page_count > 1:
                    self.get_pyq()

        if self.grid_cls.pager_on:
            page_func()

        return sub_func(expected)

    def test_filters(self):
        if callable(self.filters):
            cases = self.filters()
        else:
            cases = self.filters
        for name, op, value, expected in cases:
            self.check_filter(name, op, value, expected)

    def check_sort(self, k, ex, asc):
        if not asc:
            k = '-' + k
        d = {'sort1': k}

        def sub_func():
            with flask.current_app.test_request_context('/?' + urllib.parse.urlencode(d)):
                self.assert_in_query('ORDER BY %s%s' % (ex, '' if asc else ' DESC'))
                self.get_pyq()  # ensures the query executes and the grid renders without error

        def page_func():
            url = '/?' + urllib.parse.urlencode({'sort1': k, 'onpage': 2, 'perpage': 1})
            with flask.current_app.test_request_context(url):
                pg = self.get_session_grid()
                if pg.page_count > 1:
                    self.get_pyq()

        if self.grid_cls.pager_on:
            page_func()

        return sub_func()

    @pytest.mark.parametrize('asc', [True, False])
    def test_sort(self, asc):
        for col, expect in self.sort_tests:
            self.check_sort(col, expect, asc)

    def assert_table(self, table, grid=None, **kwargs):
        d = self.get_pyq(grid, **kwargs)

        assert len(d.find('table.records thead th')) == len(table[0])
        for idx, val in enumerate(table[0]):
            assert d.find('table.records thead th').eq(idx).text() == val

        assert len(d.find('table.records tbody tr')) == len(table[1:])
        for row_idx, row in enumerate(table[1:]):
            len(d.find('table.records tbody tr').eq(row_idx)('td')) == len(row)
            for col_idx, val in enumerate(row):
                read = d.find('table.records tbody tr').eq(row_idx)('td').eq(col_idx).text()
                assert read == val, 'row {} col {} {} != {}'.format(row_idx, col_idx, read, val)

    def expect_table_contents(self, expect, grid=None, **kwargs):
        d = self.get_pyq(grid, **kwargs)
        assert len(d.find('table.records tbody tr')) == len(expect)

        for row_idx, row in enumerate(expect):
            td = d.find('table.records tbody tr').eq(row_idx).find('td')
            assert len(td) == len(row)
            for col_idx, val in enumerate(row):
                assert td.eq(col_idx).text() == val

profile to log

Log the grid name, filter/sort/page settings, and the performance for the three queries (count, data, totals).

Enable via option on the grid.

Webgrid 2.0

This is the big tracking issue for webgrid 2.0:

  • Public Documentation
  • Streamlined control flow and clearer abstractions
  • Updated UI

Query exceptions possibly being passed

I had an attribute on the Grid class that had a bare column on the entity that caused and exception but the exception was never raised making it difficult to troubleshoot the problem.

Kick off only one Appveyor job for PRs

by setting this in the appveyor.yml:

# If a branch has a PR, don't build it separately.  Avoids queueing two appveyor runs for the same
# commit.
skip_branch_with_pr: true

expose an absolute static path

Some apps would like to expose static assets through the web server (Apache, nginx) rather than the app. To do this, we need to expose a static path the app can use to copy assets to its staticly-served folder. And, the manager/templates will need to support having a base static path to make sure the assets are referenced with the proper URLs (depends on the manager).

handle renderer limits

Two limit-related issues wrt renderers:

  • some renderers, like Excel, will have inherent limitations such as row counts (~65k for xls, and ~1m for xlsx)
  • usually, users only render very large recordsets unintentionally (i.e. forgot to set/apply a filter, didn't notice the large number of total records, etc.)

So, we need to do two things: have the user confirm large exports, and throw a more useful exception when a hard limit is transgressed for the renderer:

  • have a property on the renderer that checks the appropriate criteria and returns True/False.
    • calling the renderer when can_render is False should raise an exception (named something like RenderLimitExceeded)
    • developers may also use the property to control whether export links are shown, and provide information to the user as to how to bring the grid into an "exportable" state (e.g. apply more filters)
  • place a unconfirmed_export_limit on the grid, set on the base grid to 10k
    • setting the attribute to None should disable the check
    • webgrid's JS should use the attribute's value to have a confirm on export links, comparing to the current record count on the grid

Replace page "X of X" select box with input

Page [1] of 10

Where [1] is an input tag and 10 is just a span or something.

On the rare occasion when you might load a grid with many thousands of pages, the number of items in the select box can cause so much HTML to be generated that it breaks rendering and/or prevents the page from being fully loaded.

conditional display for columns

Columns should be made to conditionally displayed through permissions, app settings, etc. We could do a ConditionalColumn, but I'd rather be able to simply pass a lambda to a Column of any type to determine whether it would display.

Should also look at render_in and make sure it can handle lambdas.

use pipenv for CI setup

The wheelhouse got removed because the most recent set of wheels did not support python 3.6-3.7. CI is currently just pulling latest dependency versions from PyPI, which is fine until something breaks. We should set up pipenv for this, as we do in some other libraries.

Prefer to use string labels over column expressions when applying order_by

SQLAlchemy in version 1.1.5 no longer uses expression labels when generating the ORDER BY clause. I believe this has happened before and was eventually fixed but it seems like a recurring regression.

For more consistent query generation, I think it would be a good idea to check for the presence of a string label on the column's expression and use that as the preferred argument to order_by here and then falling back to the expression if no label is provided.

Maybe hide "Export to Excel" link if xlwt isn't installed

From HLLAPI and RaceBetter GridView:

        if g.export_to == 'xls':
            raise ImmediateResponse(g.xls.as_response())

Can result in:

AttributeError: 'NoneType' object has no attribute 'as_response'

when xlwt isn't installed. Would it be better if we simply hid the link for exporting to Excel if the xls renderer isn't available?

deprecation warnings

renderers.py:18: DeprecationWarning: The import 'werkzeug.Href' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.urls import Href' instead.
renderers.py:18: DeprecationWarning: The import 'werkzeug.MultiDict' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.datastructures import MultiDict' instead.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.