GithubHelp home page GithubHelp logo

oslandia / qgeologis Goto Github PK

View Code? Open in Web Editor NEW
28.0 12.0 6.0 1.09 MB

Migrated to: https://gitlab.com/Oslandia/qgis/QGeoloGIS

License: GNU General Public License v2.0

Python 99.48% Makefile 0.52%
qgis plot python geology borehole well-logs

qgeologis's Introduction

QGeoloGIS plugin

This project allows the visualization of logs of drilling wells or boreholes and time series.

It is based on QT and the QGIS rendering engine to plot series of measurements. This allows:

  • the use of the rich symbology engine of QGIS to display underground data
  • having decent display performances, since QGIS is optimized to quickly render geometries possibly made of lots of points

Currently three types of data are handled:

  • stratigraphy data, where a polygon is defined by a depth range and a pattern fill is given by a rock code
  • continuous series of data that represent data sampled continuously underground (a sample every centimer for instance). This could also be reused to plot time series.
  • scatter plots of data

Example in a QGIS application

See the corresponding video

How to use it as a standalone plugin

Install the plugin in the QGIS plugin directory and enable it. You can install it by typing make deploy in the main folder.

It requires a configuration that describes what is the base layer that displays measure points and how to access the different measure layers.

Some sample resources are available in the sample folder. Apart from the small toy dataset itself, you can find an example QGIS project file and the associated configuration file, which may be loaded from the plugin menu.

qgeologis's People

Contributors

delhomer avatar lbartoletti avatar troopa81 avatar vmora avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

qgeologis's Issues

Allow the update of a column / row configuration

It is only possible to add a new column data for now.

Additionally, it is not very clear what should happen when an invalid column has been configured. It is easy to add an invalid column that keep causing errors until the project is reloaded.

QGIS 3 PostgreSQL's arrays are slow

QGIS3 handles natively PostgreSQL arrays: they are converted into QList and then into Python lists with PyQt.

However, it appears it is very slow.

Here is a short benchmark:

1/ create a database with the 3 following tables:

create table t_array as select 1 as id, array_agg(random()) as values from generate_series(1,100000);
create table t_string as select 1 as id, array_to_string(array_agg(random()),',') as values from generate_series(1,100000);
-- simulate a bytea of 100000 8-bytes floating point numbers
create table t_bytea as select 1 as id, decode(repeat('0102030405060708', 100000), 'hex') as values;

2/ Load them in QGIS and run:

import time
import array

l_array = QgsProject.instance().mapLayersByName("t_array")[0]
l_string = QgsProject.instance().mapLayersByName("t_string")[0]
l_bytea = QgsProject.instance().mapLayersByName("t_bytea")[0]
start = time.time()
for f in l_array.getFeatures():
    assert len(f["values"])==100000
print("array", time.time() - start)
start = time.time()
for f in l_string.getFeatures():
    tt = [None if x == 'NULL' else float(x) for x in f["values"].split(",")]
    assert len(tt) == 100000
print("string", time.time() - start)
start = time.time()
for f in l_bytea.getFeatures():
    tt = array.array("d", f["values"].data())
    assert len(tt) == 100000
print("bytea", time.time() - start)

I have the following results (in seconds):

array 9.508360624313354
string 0.04486846923828125
bytea 0.0058481693267822266

Removed columns reapear

If I remove columns with the trash icon, it works, but if I select a new collar, the deleted columns reapear.

Modifying timeseries symbology raises an AttributeError

When I try to "edit the row style" of a timeseries, I can design my own symbology, however when I validate it, the following error is raised:

AttributeError: 'TimeSeriesWrapper' object has no attribute 'styles' 
Traceback (most recent call last):
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/default/python/plugins/BDLHES/QGeoloGIS/main_dialog.py", line 283, in on_styles_updated
    styles = self.__view.styles()
AttributeError: 'TimeSeriesWrapper' object has no attribute 'styles'

Anyway, the symbology is applied to the graph, but it seems impossible to save it through the configuration.

Linked to #34 ?

TimeSeries View config crash

I try to use QGeoloGIS on a ready-to-use QGIS project:

QGIS3_PROFILE=my_profile make deploy
QGIS3_PROFILE=my_profile qgis my_qgis_project.qgs

Then on QGIS, something goes wrong when clicking on the TimeSeries View button. I get the following trace:

AttributeError: 'PlotConfig' object has no attribute 'get_dict' 
Traceback (most recent call last):
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/qgis_plugin.py", line 106, in 
    self.view_timeseries.triggered.connect(lambda: self.on_view_plots("timeseries"))
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/qgis_plugin.py", line 143, in on_view_plots
    dialog = MainDialog(self.__dock, plot_type, self.__config, layer, self.iface)
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/main_dialog.py", line 263, in __init__
    self.__update_selected_features()
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/main_dialog.py", line 270, in __update_selected_features
    self.__view.set_features(self.__layer.selectedFeatures())
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/main_dialog.py", line 198, in set_features
    self.update_view()
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/main_dialog.py", line 208, in update_view
    self.__config.get_timeseries())
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/main_dialog.py", line 73, in load_plots
    add_function(data, title, uom, station_name=feature_name, config=cfg)
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/qgeologis/timeseries_view.py", line 309, in add_data_row
    self._update_data_row(data, config)
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/qgeologis/timeseries_view.py", line 331, in _update_data_row
    if config and 'min' in config.get_dict() and config['min'] is not None:
AttributeError: 'PlotConfig' object has no attribute 'get_dict'

I got this result with v1.3.0, however no problem with v1.2.0.

Classified / data defined symbology

For "regular" plots (not the special case of stratigraphy), data are rendered by considering one QGIS feature for the whole dataset.

It means we render a multipoint, a multilinestring ... or a polygon.

The problem with this current approach is that we cannot have a symbology that changes with each data point. No classification / graduation / data defined, etc.

I am facing a use case where I need to distinguish between different kinds of data based on their attribute.

I then plan to modify the current behavior and render one QGIS feature for each data point. But I see the following drawbacks:

  • if we want to keep the "polygon" rendering mode, we need to keep one feature = all the data points
  • having one feature per data point may add some performance issues.

So, in the end, I think I will:

  • add a symbology option "render each data point separately"
    • if enabled the user can choose a more complex symbology type ("classification", "graduated", "rule-based", etc.), but it only works for "points" and "lines" rendering, not "polygon"
  • if disabled (default) we keep the current rendering mode

This should however add some code complexities.

@vmora @troopa81 @delhomer Any comments ?

Export config to .json

After running a config export, I got the following bug:

KeyError: 'timeseries' 
Traceback (most recent call last):
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/qgis_plugin.py", line 214, in on_export_config
    export_config(self.__config, filename)
  File "/home/rdelhome/.local/share/QGIS/QGIS3/profiles/cea/python/plugins/QGeoloGIS/config.py", line 209, in export_config
    for layer_cfg in config[subkey]:
KeyError: 'timeseries'

Graph selection is broken when the scrollbar is moved

With a dialog with more graphs than what is possible to display at once, and the scrollbar has been scrolled to see graphs at the bottom, the graph selection by left click is shifted.

Capture d’écran de 2020-01-20 11-27-45
here, "Pression totale" is selected when "Time" is clicked

Root layer (re)loading

Allow to load or reload the layer configured as the root one in the configuration file

Include time series

The PlotItem class has been designed to display both vertical "XZ" data and "XY" horizontal data.
We usually plot log data vertically and time series horizontally.
And thinking about it, some could want to represent log data horizontally (why not ?)

The project is called "qgis_well_log", should it be renamed so that we understand it covers both types of data ?

@troopa81 @vmora

QGIS 3 port

The widget is currently QGIS 2.x only.

Make it compliant with Python 3 / QT5 / QGIS3.

Step by step:

  • Make the code python3 compatible
  • Use qgis.PyQt rather than PyQt4 or PyQt5
  • Adapt QGIS API calls

This would normally also allow to get rid of numpy as a dependency, since Python3 has support for memoryview + struct.

Code quality improvement

Ways to improve the code state:

  • add docstrings (see #30 )
  • use flake8 and black (add a git commit hook ?)
  • add tests !
  • add a CI

Add docstrings (and type annotations ?)

The project is growing and the overall readability is not really enhancing ...

At the minimum add docstrings to document function usage and parameter types.

Maybe we should investigate the use of type annotations ? at least to highlight areas where the typing is a bit too much dynamic or not clear (It is already not very clear which functions take a regular dict from those which take PlotConfig)

Configuration refactoring

Summary of a small design sprint with @vmora and me, on possible evolutions to the configuration part of QGeoloGIS.

Here are some propositions to resolve some needs:

Save the ordering of columns (resp. rows for timeseries)

Current situation

Columns are stored in different arrays:

  • stratigraphy_config
  • log_measures
  • imagery_data
    So it is not very clear how to order logs vs. stratigraphy or vs. imagery

Proposed change

Have two objects of high-level logs and timeseries that store source definitions as JSON arrays.
It will merge stratigraphy, log_measures and imagery_data into the same common array logs

A new key type allows the distinction between stratigraphy, logs, imagery and so on.

Store the symbology

Add a configuration key symbology that stores the raw QGIS QML symbology of each column / row

Remove the concept of "sub-selection"

The current code is designed to work with some data sources that have multiple measures stored in the same table (e.g. chemical analysis).

It is based on a logic of filtering and grouping on data:

  • the "sub selection" combobox of the data selection dialog scans the source layer for unique values on a given field and allows the user to select one value
  • then the selected value is used to filter data from the source layer

Capture d’écran de 2020-01-20 16-13-14

It was initially thought to allow flexibility in the configuration of data sources when the list of distinct values cannot be defined a priori.

However it adds too much complexity to the code base and does not fit very well with the simple idea of "1 measure = 1 data source".

Proposed change

Remove the concept of "sub selection".

Each single measure should be configured as a separate data source, with the possibility to add a QGIS filter (with the configuration key filter)

Replace layer IDs by (uri, name, provider)

Current situation

The configuration uses layer IDs to reference data sources.
Problem: it makes the configuration tied to a specific QGIS project file.

Proposed change

Change layer IDs to a triple (name, URI, provider)

This will also allow a trivial import / export function to have a separate configuration file (that can be put on git easier)

Merged plots

In order to merge different plots on the same column (or row), the configuration will allow to specify for each column an array of data sources.

See the complete example.

Additional metadata on graphs

  • minimum value,
  • maximum value,
  • font title
  • font size
  • column / row width in pixels

Configuration example

# The "main" layer of points, to be replaced by (name, uri, provider)
"collar_7e814841_4cc9_495a_8ef6_9229fcfffb5e" :
{
                  "id_column": "id",
                  "name_column": "id",
                  # new keys for fonts
                  "font": Helvetica,
                  "font_size": 14,
                  "logs":[
                    {
                       # width (in pixels) of the column
                        "width": 200,
                        "name": "Stratigraphy",
                        # each data has one or more "series"
                        series = [
                        {
                        "provider": "postgres", 
                        "uri":"dbname=....",
                        "type": stratigraphy,
                        "feature_ref_column": "hole_id",
                        "depth_from_column": "from_",
                        "depth_to_column": "to_",
                        "formation_code_column": "comments",
                        "rock_code_column": "code",
                        "formation_description_column": "comments",
                        "rock_description_column": "code",
                        "style": "formation_style.xml",
                        # new key to specify whether a column is visible or not
                        "visible": true
                    }],
                    {
                        "name": "Radiometry",
                        "width": 200,
                        "visible": false,
                        "series": [
                            {
                                "source": "radiometry_9d3b823d_d1b6_47a8_a1c2_dd0895a7262e",
                                "uom": "Gamma",
                                "feature_ref_column": "hole_id",
                                # min and max values
                                "min": null,
                                "max": null,
                                # new filter key
                                "filter":"element='Au'",
                                "type": "instantaneous_log",
                                # serialized QGIS symbology
                                "symbology": "xml_string",
                                "event_column": "from_",
                                "value_column": "gamma",
                                "visible": false
                                # possible addition to blend different plots during the merge
                                "blending_mode": "multiply"
                            }
                            ]

                    },
                    {
                        "name": "Image",
                        "source": "radiometry_9d3b823d_d1b6_47a8_a1c2_dd0895a7262e",
                        "feature_ref_column": "hole_id",
                        "type": "imagery_log",
                        "event_column": "from_",
                        "value_column": "gamma",
                        "visible": false
                    },

                  ],
                  "timeseries": []

Tasks

  • Write the UI part for the configuration of timeseries
  • Add a function to import / export the configuration from / to a project
  • Remove the concept of "subselection"
  • Refactor with the new configuration proposed here

Classified symbology does not work

I use a stratigraphic column with just mandatory values.

I can choose categorized symbology, I can chose a column, classification gives me classes allright (sees different values) but the displayed column has a single color (the one for other values).

Save symbology

At the moment column symbology can only be saved as a separate .qml file, it would be nice to keep the user's configuration in the project file.

Zoom too fast with touchpad

The zoom behavior is different between a mouse with a wheel and a touchpad scroll event.
With the touchpad it is wayyy too fast.

Add support for image data

Allow to have a new column type of image data.

Should it be a reference to an image on disk ? Or an image object (embedded in a database for instance) ?

Display artifacts

capture d ecran de 2019-02-28 15-24-20

With a strong zoom, we can see that there is a rounding issue somewhere: in this example with a sample on each integer and a scale that begins not on an integer, a blank space is displayed. There should be a partial line to the previous sample

Error with Statigraphy when formation code is an int

TypeError: width(self, str, length: int = -1): argument 1 has unexpected type 'int' 
Traceback (most recent call last):
  File "/home/vmo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/QGeoloGIS/qgeologis/stratigraphy.py", line 116, in paint
    w = fm.width(formation_code)
TypeError: width(self, str, length: int = -1): argument 1 has unexpected type 'int'

Ergonomic refactor

I'm planning to make some major modifications on QGeoloGIS to improve usability:

  • Remove QGIS 2 compatibility (QGIS 2 is no longer supported) and fix style file management for QGIS 3
  • Make the QGeoloGIS window unique and dockable. Its content will be updated according to the current identified feature
  • Load the QGeoloGIS configuration from project file (and later be able to edit and save this configuration)
  • Replace the "data to add" list with a checkboxes list that manage the visibility for a given data (it won't be possible to add the same data several times anymore)
  • Save column order in project file
  • Use layerid instead of uri+provider when configuring plot. Then, it will be possible to use virtual field in plot.

I'm planning to propose a PR in the days to come for these modifications.

Just for the record, I think it will also be interesting to have in QGeoloGIS:

  • Connect to selection to display and compare plot from different feature
  • Edit configuration from IHM

@mhugo @vmora

Configuration Export and Time Series

Hello, good morning.

Unfortunately I have to deal with 2 problems with the QGeologis plugin (1.3.1).
1.) Export Configuration
I tried exporting the configuration of a continuous log (which is working well) via 'Plugins --> QGeologis --> Export configuration to ...'. But I received the following error message:

KeyError: 'timeseries'
Traceback (most recent call last):
File "C:/Users/Martin/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\QGeoloGIS-master\qgis_plugin.py", line 214, in on_export_config
export_config(self.__config, filename)
File "C:/Users/Martin/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\QGeoloGIS-master\config.py", line 210, in export_config
for layer_cfg in config[subkey]:
KeyError: 'timeseries'

I am a bit confused because I didn't do anything in the configuration with times series.

2.) Time Series
I tried creating the plot of a time series but it didn't work. Assuming that I need to configure the type 'log measures instantaneous' getting a time series plot I defined an event column which contains the day of the measurement as numerical value (1, 2, 3, 4, ....) and a column with the measured values (e.g. 2, 5, 9,..). Obviously these definitions are not correct. I received some error messages but I am not an expert so I didn't really understood what they mean. Maybe an example of a table with a time series in the correct format would help.

Regards

Martin

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.