dz0ny / pyramid_straw

Makes debugging SQLAlchemy in Pyramid a joy.

Pyramid Straw

The first tool you should use to analyze whats going on with your database, when developing pyramid apps.

---

Installing

Include project after debug_toolbar.

pyramid.includes =
    pyramid_straw.profiler
    pyramid_straw.panel

Configuration

Project consists of two parts

Config options

pyramid_straw.only_paths
pyramid_straw.report_hook
pyramid_straw.disable_inspect

pyramid_straw.only_paths

If you want to limit what is backtraced for each query (defaults to package where it is included). You can override pyramid_straw.only_paths.

For example:

pyramid_straw.only_paths = ['/home/dz0ny/ebn/src/']

pyramid_straw.report_hook

And if you want to send reports somewhere else you can use report hook that needs to be callable and accepts data dict for each query. This would be little expensive in production so beware.

For example:

def report_hook(data):
    requests.post(config.my_report_url, json=data)

config.add_settings({
    'pyramid_straw.report_hook': report_hook
})

pyramid_straw.disable_inspect

You can also disable query inspection with setting disable_inspect to false. For example:

config.add_settings({
    'pyramid_straw.disable_inspect': False
}) ```

pyramid_straw's Issues

add PEV

Analytics based on overtime statistics

NOTE: This issue is still on progress, don't bother to read further yet.

Motivation:
Analytical view SQLAlchemy in pyramid_debugtoolbar does not provide statistical/historical view on sql/functions queries/calls which make it impossible to track general tendentious and to find and investigate extremes (and possibly anomalies).

Idea:

1. Implement functionality for gathering descriptive statistics to associate flowing parameters with query/call: mean, average deviation, median, mode...
2. Implement functionality that would gather query explains and profiling of call stack (if possible) for cases that exit allowed deviation.

Implementation notes:

• all sql query explains and call stack profiling logic should be as close to pyramid_debugtoolbar as much possible so it could be integrated into its code base.
• view function is a main source of events (and in context of pyramid_debugtoolbar the only one) so all queries probably should be associated with it.
• the statistic gathering functionality should be based pyramid_debugtoolbar and provide simple view to represent statistical information for each view (mean, average deviation, median, mode, ...) and extreme cases
• as a storage endpoint for analytical information should be used a DB that support OLAP cube (or at least have group by and aggregation functionality - if sqlite allow it, would be great to use it as a starter).
• for each event source provide box plot for better representation (image taken from http://rstudio.owlmetabolomics.com:8031/AgingAnalysis/):
  
• provide accumulative statistic of time spend(load) for each event source per hour/day/week/mount

Consider replacing PEV with simpler tool for visualization

Internal https://github.com/AlexTatiyants/pev calculations doesn't seem to be that complicated and came goes to visualization part, but the tool itself appeared to be quite hard to integrate with and it is quite complicated to extend or modify. PEV was fine as POC but for as a project dependency it could become quite hard to maintain it. The other thing is that it's really slow - it takes couple of seconds to load the app before it start working, also the size of static files for that app is 8MB which is kind of insane...

support for sqlite, mysql, etc...

Wasn't able to pyramid_straw with sqlite, on every request it was giving strange trace-back, I guess it should be handle somehow, at least with providing sane trace-back that pyramid_straw doesn't support any DB backends except psql

PEV won't reneder until user perform action on input fields

Impossible to click submit button on pev's new plan form until type something in execution plan field, even if it has some value.

PEV renders in iframe in modal window, all data that is received from backed populates in following way 3c5fd01
which doesn't seem to trigger pev's event listeners, probably to fix that it's required to broadcast keypress event for textarea inputs.