level12 / webgrid Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
SQLAlchemy in version 1.1.5 no longer uses expression labels when generating the ORDER BY
clause. I believe this has happened before and was eventually fixed but it seems like a recurring regression.
For more consistent query generation, I think it would be a good idea to check for the presence of a string label on the column's expression and use that as the preferred argument to order_by
here and then falling back to the expression if no label is provided.
HTML5 has a time
tag that's meant to represent dates and times semantically. It is better for indexing and accessibility. But it's also convenient for hiding machine-readable formats in the datetime
attribute and displaying it differently.
<time datetime="20160309T130222.3022Z">March 9th</time>
If a TextFilter is switched to a multi-select (via the multi-select toggle), values containing HTML characters are not escaped properly.
In the issue I ran into, one of my column values was '<Unassigned>'
, which resulted in the following rendered HTML:
<li>
<label>
<input type="checkbox" name="selectItemv1(class_name)" value="7">
<unassigned></unassigned>
</label>
</li>
The wheelhouse got removed because the most recent set of wheels did not support python 3.6-3.7. CI is currently just pulling latest dependency versions from PyPI, which is fine until something breaks. We should set up pipenv for this, as we do in some other libraries.
TypeError: '<=' not supported between instances of 'datetime.datetime' and 'NoneType'
(7 additional frame(s) were not displayed)
...
File "commonbwc/lib/views.py", line 220, in manage_assign_vars
dg = self.manage_init_grid()
File "commonbwc/lib/views.py", line 236, in manage_init_grid
return self.grid_init()
File "commonbwc/lib/views.py", line 96, in grid_init
grid.apply_qs_args()
File "webgrid/init.py", line 802, in apply_qs_args
v2,
File "webgrid/filters.py", line 604, in set
if self.value1 <= self.value2:
TypeError: '<=' not supported between instances of 'datetime.datetime' and 'NoneType'
This is the big tracking issue for webgrid 2.0:
Currently, with the way Excel exports work, format-specific information has a lot of fingers reaching into the main grid code.
Grid.set_renderers
sets direct attributes for the renderers (e.g. Grid.html
, Grid.xls
)This needs to get refactored. When #21 happens, the first point above should be addressed in a non-breaking manner. The other items will break compatibility and should be reserved for a more major revision:
Grid
should namespace renderer attributes set in set_renderers
Grids tend to generate very large URLs when filters are applied. Some larger grids have gotten to the point where the URL is overflowing uwsgi's default buffer sizes.
To reduce the size of the URL, when the filter form is submitted, it should only include fields that are actually set.
From HLLAPI and RaceBetter GridView:
if g.export_to == 'xls':
raise ImmediateResponse(g.xls.as_response())
Can result in:
AttributeError: 'NoneType' object has no attribute 'as_response'
when xlwt
isn't installed. Would it be better if we simply hid the link for exporting to Excel if the xls renderer isn't available?
xls has a hard upper limit of 65,000 rows. Some projects have many more than that displayed in a grid. Switching to xlsx will bump that limit to over 1million. While that is still an upper limit, that is much much higher.
by setting this in the appveyor.yml:
# If a branch has a PR, don't build it separately. Avoids queueing two appveyor runs for the same
# commit.
skip_branch_with_pr: true
Truncate long fields (like description), potentially adding a clickable ellipsis that expands the field.
Log the grid name, filter/sort/page settings, and the performance for the three queries (count, data, totals).
Enable via option on the grid.
Excel export and filter functionality do not understand how to export dates that are arrow objects.
A csv export option so we can support downloading big datasets
Page [1] of 10
Where [1] is an input tag and 10 is just a span or something.
On the rare occasion when you might load a grid with many thousands of pages, the number of items in the select box can cause so much HTML to be generated that it breaks rendering and/or prevents the page from being fully loaded.
Tests on the blazeweb testing app are relegated to python 2. I believe this happened because when webgrid was updated to support python 3, blazeweb did not support that yet.
I got this email recently:
Hello!
You have recently released a new version of WebGrid. Thanks! It will
probably make a lot of people happy. There is a tiny issue with your
package on PyPI though: it is a bit difficult to guess the exact
license you use.
It is very clear that you use one of the BSD licenses. PyPI allows you
to use the following classifier:
License :: OSI Approved :: BSD License
But it is not very precise. You can also use the "license" field in
setuptools, but there is no "fixed" format, so you could write "BSD2",
"BSD-2", "BSD-2-Clause", etc. This makes it hard for automated tools
to properly guess which BSD license you are using.
May I suggest using the aforementioned classifier *and* an SPDX
identifier (https://spdx.org/licenses/) in the "license" field?
Thanks,
Cyril Roelandt
(This email was automatically generated, but you can answer it! A real
human being will read your answer.)
OptionsFilterBase can give an exception ValueError: value_modifier argument set to "auto", but the options set is empty and the type can therefore not be determined
. But there is no easy way to determine which filter on a grid is causing the issue. The exception should include the name of the filter.
Can be the same as: https://github.com/level12/keg/blob/master/license.txt
Some apps would like to expose static assets through the web server (Apache, nginx) rather than the app. To do this, we need to expose a static path the app can use to copy assets to its staticly-served folder. And, the manager/templates will need to support having a base static path to make sure the assets are referenced with the proper URLs (depends on the manager).
I had an attribute on the Grid class that had a bare column on the entity that caused and exception but the exception was never raised making it difficult to troubleshoot the problem.
Flake8 settings in tox.ini currently have complexity set pretty high.
It would be helpful to have a method on the grid called at the end of the init process (i.e. after the columns are set up for the instance).
There should be a base testing class for testing webrid grids. We use the following in one of our projects:
import re
import urllib.parse
from blazeutils.spreadsheets import workbook_to_reader
import flask
import flask_login
from pyquery import PyQuery
import pytest
import sqlalchemy
def query_to_str(statement, bind=None):
"""This function is copied directly from sqlalchemybwc.lib.testing
returns a string of a sqlalchemy.orm.Query with parameters bound
WARNING: this is dangerous and ONLY for testing, executing the results
of this function can result in an SQL Injection attack.
"""
if isinstance(statement, sqlalchemy.orm.Query):
if bind is None:
bind = statement.session.get_bind()
statement = statement.statement
elif bind is None:
bind = statement.bind
if bind is None:
raise Exception('bind param (engine or connection object) required when using with an '
'unbound statement')
dialect = bind.dialect
compiler = statement._compiler(dialect)
class LiteralCompiler(compiler.__class__):
def visit_bindparam(
self, bindparam, within_columns_clause=False,
literal_binds=False, **kwargs
):
return super(LiteralCompiler, self).render_literal_bindparam(
bindparam, within_columns_clause=within_columns_clause,
literal_binds=literal_binds, **kwargs
)
compiler = LiteralCompiler(dialect, statement)
return 'TESTING ONLY BIND: ' + compiler.process(statement)
class GridBase(object):
grid_cls = None
filters = ()
sort_tests = ()
@classmethod
def setup_class(cls):
cls.user = User.testing_create()
if hasattr(cls, 'init'):
cls.init()
def assert_in_query(self, look_for, **kwargs):
pg = self.get_session_grid(**kwargs)
query_str = query_to_str(pg.build_query())
assert look_for in query_str, '"{0}" not found in: {1}'.format(look_for, query_str)
def assert_not_in_query(self, look_for, **kwargs):
pg = self.get_session_grid(**kwargs)
query_str = query_to_str(pg.build_query())
assert look_for not in query_str, '"{0}" found in: {1}'.format(look_for, query_str)
def assert_regex_in_query(self, look_for, **kwargs):
pg = self.get_session_grid(**kwargs)
query_str = query_to_str(pg.build_query())
if hasattr(look_for, 'search'):
assert look_for.search(query_str), \
'"{0}" not found in: {1}'.format(look_for.pattern, query_str)
else:
assert re.search(look_for, query_str), \
'"{0}" not found in: {1}'.format(look_for, query_str)
def get_session_grid(self, *args, **kwargs):
flask_login.login_user(kwargs.pop('user', self.user), force=True)
g = self.grid_cls(*args, **kwargs)
g.apply_qs_args()
return g
def get_pyq(self, grid=None, **kwargs):
pg = grid or self.get_session_grid(**kwargs)
html = pg.html()
return PyQuery('<html>{0}</html>'.format(html))
def get_sheet(self, grid=None, **kwargs):
pg = grid or self.get_session_grid(**kwargs)
xls = pg.xls()
return workbook_to_reader(xls).sheet_by_index(0)
def check_filter(self, name, op, value, expected):
qs_args = [('op({0})'.format(name), op)]
if isinstance(value, (list, tuple)):
for v in value:
qs_args.append(('v1({0})'.format(name), v))
else:
qs_args.append(('v1({0})'.format(name), value))
def sub_func(ex):
url = '/?' + urllib.parse.urlencode(qs_args)
with flask.current_app.test_request_context(url):
if isinstance(ex, re.compile('').__class__):
self.assert_regex_in_query(ex)
else:
self.assert_in_query(ex)
self.get_pyq() # ensures the query executes and the grid renders without error
def page_func():
url = '/?' + urllib.parse.urlencode([('onpage', 2), ('perpage', 1), *qs_args])
with flask.current_app.test_request_context(url):
pg = self.get_session_grid()
if pg.page_count > 1:
self.get_pyq()
if self.grid_cls.pager_on:
page_func()
return sub_func(expected)
def test_filters(self):
if callable(self.filters):
cases = self.filters()
else:
cases = self.filters
for name, op, value, expected in cases:
self.check_filter(name, op, value, expected)
def check_sort(self, k, ex, asc):
if not asc:
k = '-' + k
d = {'sort1': k}
def sub_func():
with flask.current_app.test_request_context('/?' + urllib.parse.urlencode(d)):
self.assert_in_query('ORDER BY %s%s' % (ex, '' if asc else ' DESC'))
self.get_pyq() # ensures the query executes and the grid renders without error
def page_func():
url = '/?' + urllib.parse.urlencode({'sort1': k, 'onpage': 2, 'perpage': 1})
with flask.current_app.test_request_context(url):
pg = self.get_session_grid()
if pg.page_count > 1:
self.get_pyq()
if self.grid_cls.pager_on:
page_func()
return sub_func()
@pytest.mark.parametrize('asc', [True, False])
def test_sort(self, asc):
for col, expect in self.sort_tests:
self.check_sort(col, expect, asc)
def assert_table(self, table, grid=None, **kwargs):
d = self.get_pyq(grid, **kwargs)
assert len(d.find('table.records thead th')) == len(table[0])
for idx, val in enumerate(table[0]):
assert d.find('table.records thead th').eq(idx).text() == val
assert len(d.find('table.records tbody tr')) == len(table[1:])
for row_idx, row in enumerate(table[1:]):
len(d.find('table.records tbody tr').eq(row_idx)('td')) == len(row)
for col_idx, val in enumerate(row):
read = d.find('table.records tbody tr').eq(row_idx)('td').eq(col_idx).text()
assert read == val, 'row {} col {} {} != {}'.format(row_idx, col_idx, read, val)
def expect_table_contents(self, expect, grid=None, **kwargs):
d = self.get_pyq(grid, **kwargs)
assert len(d.find('table.records tbody tr')) == len(expect)
for row_idx, row in enumerate(expect):
td = d.find('table.records tbody tr').eq(row_idx).find('td')
assert len(td) == len(row)
for col_idx, val in enumerate(row):
assert td.eq(col_idx).text() == val
Slack reference
class SentTimeGroup(ColumnGroup):
label = 'Sent Time (s)
class RacesGrid(Grid):
RaceIdColumn('Id', ents.Race.id)
Column('Max', some_aggregate, filters.NumericFilter, group=SentTimeGroup)
Randy Syring [1:56 PM]
There is a lot of wasted space in the header cells b/c of the need to repeate "Sent Time (s)" and "Receive Time (s)" in columns.
What if we created the ability to set acolumn_group
on a column so that the grid looked like: example
Matt Lewellyn [2:00 PM]
adding that to webgrid or rolling it into your project would be fairly trivial, I think
Bill Adams [2:01 PM]
That would require updates to the renderers. I'm not sure how difficult it would be to get that property passed through on the column to the renderer
It could possibly be done without any updates to webgrid
Randy Syring [2:02 PM]
My thinking was to roll it into webgrid.
Matt Lewellyn [2:03 PM]
I agree it might be useful. The renderer piece of it should be the trivial part. Could be more of a task to get the columns set up, with the way webgrid creates those for the grid instance
Keg-Auth users grid depends on the edit-link
and delete-link
classes, but no default CSS is supplied for them. Without any css it's hard to tell that the actions are there. I added these CSS classes to make the actions work as expected.
.edit-link:before {
position: relative;
padding: 2px;
font-family: "FontAwesome";
content: "\f040";
}
.delete-link:before {
position: relative;
padding: 2px;
font-family: "FontAwesome";
content: "\f1f8";
}
If you specify a default_op
when creating a Filter
it will incorrectly adjust the has_filters
property of a grid when initially loading the grid.
renderers.py:18: DeprecationWarning: The import 'werkzeug.Href' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.urls import Href' instead.
renderers.py:18: DeprecationWarning: The import 'werkzeug.MultiDict' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.datastructures import MultiDict' instead.
Postgres search will be case sensitive by default. MSSQL will not. I believe in most cases, an insensitive search would be preferred.
One of the artificats of having all the fields in the form but just hidden with CSS, is that our URLs get every possible parameter, making them very long. Example:
example.com/some-grid?session_key=kTVfyGmGQk4C&op%28date%29=eq&v1%28date%29=07%2F07%2F2018&v2%28date%29=&op%28event_code%29=&v1%28event_code%29=&op%28race_num%29=&v1%28race_num%29=&op%28race_status%29=&op%28archive_post_time_open_utc%29=&v1%28archive_post_time_open_utc%29=&v2%28archive_post_time_open_utc%29=&op%28bet_release_time_utc%29=&v1%28bet_release_time_utc%29=&v2%28bet_release_time_utc%29=&op%28wagers_requested_utc%29=&v1%28wagers_requested_utc%29=&v2%28wagers_requested_utc%29=&op%28archive_locked_message_utc%29=&v1%28archive_locked_message_utc%29=&v2%28archive_locked_message_utc%29=&datagrid-add-filter=&sort1=bet_release_time_utc&sort2=&sort3=&onpage=1&perpage=100
In that particular example, I only have three things set specifically: a filter, a sort, and a custom perpage
value. I'd love it if only the fields with non-default values showed up in the URL.
Two limit-related issues wrt renderers:
So, we need to do two things: have the user confirm large exports, and throw a more useful exception when a hard limit is transgressed for the renderer:
can_render
is False should raise an exception (named something like RenderLimitExceeded
)unconfirmed_export_limit
on the grid, set on the base grid to 10k
It would be nice to be able to search all columns in a grid from one field. A simple partial implementation is here https://github.com/level12/fis-climate/blob/6e665ed036d55a08ea21403d359028532785335c/climate/libs/grids.py#L49 but improvements could be made to search on other column types
Nick and I spent almost an hour tracking this one down: One of my grid columns was coming from our LookupMixin
which has a column named label
. I was showing that label on webgrid and you could filter it. But webgrid used the column name label
as the class on the filter HTML. Bootstrap also has a label
class. So you can imagine the styling was all wrong.
Basically this was an extremely surprising bug and we really should not be doing this. We should derive some highly unlikely name for our CSS classes instead of copying directly from the column names.
Some filters are necessary to be present, in order to reduce the amount of data a grid is processing. Otherwise the results could take a very long time to complete. In other cases, they are simply user helpers which should be allowed to be cleared.
What we need is the concept of a "forced" filter, introduced by having is_forced=True in the filter constructor. Forced filters will need to require a default op, and in the UI would remove the possibility of disabling them.
"Default" filters would then need the grid to tell them if they should be processed with the default op. If the user has selected other filters, the default should be allowed to lapse.
50e12bc2dd7a#L127 will break if you pass a list
. It will turn render_in=['html']
into (['html'],)
which is not what you want because 'html' in (['html'],)
will fail.
Currently, WebGrid is tested against sqllite only. There will be grid helper classes added for #68 that aid in testing against postgres and SQL Server backends.
File "agariadm/views/private.py", line 54, in make_grid
g.apply_qs_args()
File "webgrid/__init__.py", line 792, in apply_qs_args
v1 = args.getlist(filter_v1_qsk)
AttributeError: 'dict' object has no attribute 'getlist'
I can't say I really understand all this code anymore, but the only thing that makes sense in this code:
Line 755 in de9903a
is that the result of self.get_session_store()
can be a normal dictionary under some circumstance. Code later on is assuming a MultiDict
, hence the exception when getlist()
is called.
Columns should be made to conditionally displayed through permissions, app settings, etc. We could do a ConditionalColumn, but I'd rather be able to simply pass a lambda to a Column of any type to determine whether it would display.
Should also look at render_in
and make sure it can handle lambdas.
Is it necessary to test the base and i18n versions of the tests for every Python version for both 32 and 64 bit Pythons? This results in 12 runs of the test suite.
Unless we have code that changes based on architecture, it seems like we could cut down on the permutations here with something like I did in Keg:
matrix:
# Pre-installed Python versions, which Appveyor may upgrade to
# a later point release.
- PYTHON: "C:\\Python27"
TOXENV: py27-{base,lowest,i18n}
- PYTHON: "C:\\Python35"
TOXENV: py35
- PYTHON: "C:\\Python36"
TOXENV: py36
- PYTHON: "C:\\Python37"
TOXENV: py37-{base,lowest,i18n}
Make the permutation runs on the version bookends and keep the stuff in the middle only testing the base run.
XLS does not support worksheet names longer than 31 characters and the default name generated can easily be longer. The result is an uncaught exception from xlwt.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.