3liz / qgis-wfsoutputextension

QGIS Server plugin to add Output Formats to WFS GetFeature request.

wfsOutputExtension

QGIS Server Plugin to add additional output formats to WFS GetFeature request.

Demo in Lizmap Web Client:

Description

wfsOutputExtension is a QGIS Server Plugin. It extends OGC Web Feature Service capabilities. It adds output formats to WFS GetFeature request.

It adds:

• CSV
• ESRI ShapeFile as ZIP file
• Geopackage
• GPX
• KML
• MapInfo TAB as ZIP file
• MIF/MID File as ZIP file
• ODS, the datatable
• XLSX, the datatable

Installation

We recommend to use QGIS-Plugin-Manager :

# After you have installed and setup qgis-plugin-manager
cd /opt/qgis/plugins
qgis-plugin-manager install wfsOutputExtension
# Check rights
# Restart QGIS Server

For more details :

• Read the documentation of QGIS Server.
• Read AtlasPrint install process because it's similar.

Debug on production

It's possible to set DEBUG_WFSOUTPUTEXTENSION to TRUE or 1, the plugin will not remove temporary files on the disk.

Tests

Using the docker stack to test the plugin :

make test

qgis-wfsoutputextension's Issues

Save SHP with .sld- and .qml-files to preserve styles.

Saving as shape doesn't preserve styles from the project. Please add .sld and .qml to the zip-file.

On Win32: temporary files attempted to be removed before process completed, causing 'The process cannot access the file because it is being used by another process' error

Description

On Windows Server 2019, the current version (v1.7.1) causes the data in the requested output format to have 0 bytes and have errors in the server log.

Context

It appears that the plugin tries to remove the temporary files before the process is completed. This results in the following error: The process cannot access the file because it is being used by another process.

The problem goes away when the env var DEBUG_WFSOUTPUTEXTENSION is set to 1 but this is undesirable in the long term as temporary files would never be removed.

We run QGIS Server as FastCGI on IIS.

A more complete log output:

15:06:15 INFO WfsOutputExtension[8448]: WFS request to get format CSV
15:06:15 INFO WfsOutputExtension[8448]: Fetching XSD : HTTP code 200, request ?SERVICE=WFS&VERSION=1.0.0&REQUEST=DescribeFeatureType&TYPENAME=Features_of_Interest_-_Polygons&OUTPUT=XMLSCHEMA
15:06:15 INFO WfsOutputExtension[8448]: xsd_for_layer ran in 0.0s
15:06:15 INFO WfsOutputExtension[8448]: Temporary GML file is C:\Windows\Temp\QGIS_WfsOutputExtension\gml_features_1684299974.5488436.gml|option:FORCE_SRS_DETECTION=YES
15:06:15 INFO WfsOutputExtension[8448]: Temporary csv file is C:/Windows/Temp/QGIS_WfsOutputExtension/to-csv-VkLONr.csv
15:06:15 INFO WfsOutputExtension[8448]: Sending the output file
15:06:15 INFO WfsOutputExtension[8448]: send_output_file ran in 0.17s
15:06:15 WARNING [8448]: Cannot send error after headers written
15:06:15 CRITICAL Server[8448]: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\Temp\\QGIS_WfsOutputExtension\\gml_features_1684299974.5488436.gml'

• QGIS Server FULL version (with 3 numbers) : v3.28.4
• OS on the server : Windows Server 2019 Datacenter

Problem with QGIS 3.16 (new LTR)

Description

After an update of QGIS server to the new LTR (QGIS 3.16) the plugin doesn't works on my server

Context

• QGIS Server FULL version (with 3 numbers) : 3.16.4
• OS on the server : Ubuntu 18.04

install on nginx

Hi, I use qgis server 3.10 via spawn-fcgi and multiwatch, LWC 3.3 on a congig debian buster/nginx. I'm trying to install the plugin:

• unzipped plugin in the folder /opt/qgis-server/plugins
• "fastcgi_param QGIS_PLUGINPATH /opt/qgis-server/plugins;" add in the virtual host conf (in location /qgisserver {})
• qgis server and nginx restarted
  Nothing changes in the getcapabilities, do I forget something ?
  Thanks

Add FlatGeobuf support

FlatGeobuf is becoming a popular format and it would be nice to have it available as an output format.

It may be just as simple as adding the following to definitions.py:

    Fgb = Format(
        content_type='application/x-fgb',
        filename_ext='fgb',
        force_crs=None,
        ogr_provider='FlatGeobuf',
        ogr_datasource_options=(),
        zip=False,
        ext_to_zip=(),
    )

Adding the above code block works in our setup (Windows 2019 Server).

Happy to provide a PR if needed.

Temporary space not left after not enought space error

Description

When we have not enought space in the apache temp filesystem, the temporary files are not deleted.

Context

Qgis server is using with apache2 and fcgi.
The plugin variable debug is with the false value
When we call the plugin for a large file, send with a SHP outputFormat for example, the plugin create a gml file, then he converts this one in shp and finally in a zip file.
All theses files are deleted when no error and the final file is send, but when an error is occuring, the plugin stop, send a partial file and don't delete the temporary files.
We must delete manually.

• QGIS Server FULL version (with 3 numbers) :
  Qgis Server 3.16.4

• OS on the server :
  Debian 10.8

Reduce log output for WfsOutputExtension.sendResponse

First of all, very cool plugin, thanks a lot 👍

This has been reported previously as a side remark in #24 but might be worth an own issue:

With QGIS_SERVER_LOG_LEVEL = 0

there are hundreds of lines for each output request which are bloating the server logs:

INFO WfsOutputExtension[1728]: WfsOutputExtension.sendResponse

Would be nice to reduce the amount of messages, maybe one line of these would be sufficient :-)

output format according spatial or non-spatial data tables

hi

as suggested by @Gustry
3liz/lizmap-web-client#1623 (comment)

the output formats must be available according to the type of table

thanks

Dxf format

Extend export format types with dxf.

Field format altered text to numeric when exporting in csv/xlsx

Description

When downloading a layer in csv or xlsx, type of field is changed. From text to numeric.
Very annoying on insee code cause dep Aisne : initially 02*** became 2***

Please, look at this example : https://geocatalogue.hautsdefrance.fr/geonetwork/srv/fre/catalog.search;jsessionid=54BF46D02CD9744DD1523E68AB154BD1?node=srv#/metadata/4b5f8e1b-de37-47cd-9203-37a59f318b09

Context

• QGIS Server FULL version (with 3 numbers) : QGIS-Server-3.16.14
• OS on the server : Linux Debian 10 Buster

Error message opening xlsx file

Hi, we have installed the release 1.2.3 with Qgis server 3.4 and Lizmap WC 3.3.3.
When we download xlsx files from the WebGIS and open them with MS Office 365, the error message in the image below is returned. It means that there is a problem with some part of the file content and if you think that the source is reliable Excel will try to restore it.
If you click “yes” the file opens and everything seems ok.. but this message is a problem for users. Do you have any idea about the reason? Do you know if this can be fixed?

Requests with SRS=EPSG:3006 returns wrong bounding box

If I do a request with
OUTPUTFORMAT=GEOJSON&BBOX=191346.62,6212573.2,191454.58,6212671.06574&SRS=EPSG:3008
I correctly get a geojson with "bbox": [ 14.16327484, 56.03422478, 14.16502169, 56.035113]

But if I reproject the bounding box to EPSG:3006: OUTPUTFORMAT=GEOJSON&BBOX=447862.11,6210205.5,447972.13,6210300.9735&SRS=EPSG:3006
Then I get a geojson with a totally different bounding box "bbox": [ 18.26650159, 55.92242813, 18.26836047, 55.92335068]

Critical error when writing to QTemporaryFile

I'm using QGIS Server 3.14.16 with wfsOutputExtension 1.4.1 and I'm getting the following error in the QGIS log:

CRITICAL wfsOutputExtension[2656]: Creation of data source failed (OGR error: It seems a file system object called '[my QGIS temp folder]/request-WFS-qzMlsS.csv' already exists.)

It seems that writing works when I edited wfs_filter.py to not open the QTemporaryFile (temporary.open() which I think is what creates the file).

The error occurs with all output formats. Any idea what might prevent wfsOutputExtension from overwriting the file?

Installation returns no attribute 'classFactory' error

Hello, masters
I tried to install from plugin manager (Qgis 3.6.1), but there is something wrong. The phyton error log returns:

File "C:/OSGEO4~1/apps/qgis/./python\qgis\utils.py", line 335, in startPlugin
plugins[packageName] = package.classFactory(iface)
AttributeError: module 'qgis-wfsOutputExtension-master' has no attribute 'classFactory'

any idea?

export files - wrong file name

hi

lizmap exports the files with wrong file name encoding (e.g.: export_Bocas_de_Incêndio.xxx in case of Bocas de Incêndio/layer name in lizmap).

the same problem when the user downloads the PDF file (printer tool)

3liz/lizmap-web-client#1475

thanks

WFS with output format "shp" fail to return a zip 9 times out of 10

On QGIS server 3 with the extension installed.
If I run multiple times the following CURL command :

curl -o myshape.zip http://qgisserver.XYZ.com/?exp_filter=ZZZZ&typeName=YYY&VERSION=1.0.0&SERVICE=WFS&outputFormat=SHP&REQUEST=GetFeature&MAP=/path/to/this.qgs

I will get about 9 times (out of 10) a bad ZIP file which cannot be unzip and which is too large with the following response header :

Content-Type: text/xml; charset=utf-8

And about 1 times (out of 10) a good ZIP file with the following response header :

Content-Type: application/x-zipped-shp

No the good name and extension in the end of the file

Hello,

I've installed the plugin V1.7 on Lizmap 3.5.1 and Qgis-server 3.22.

The export work, but the name and the extension of the export are not good. exemple :

f2f487be-d23b-4b1e-a8c1-e422b8711a3d

But when a re-write manually the entire url : Server & query, it work well.

Have an idea?

Thanks.

[QGIS 3.0.1] Metadata Error QGIS Version

Hi guys,
for QGIS 3.0.1 this is plugin is broken. "Is incompatible with this version of QGIS Plugin designed for QGIS 2.8 - 2.99"

I change the qgis minimum version in metadata.txt but then give an error of import PyQt4 module probable need code migration.

Regards
João

Unreleased space after deleting files

Description

This issue follow the #37
When we have not enought space in the apache temp filesystem, the temporary files are now deleted in the 1.7.0 version but some space are unrealesed by the qgis_maps process.

Context

Qgis server is using with apache2 and fcgi.
When we call the plugin for a large file, send with a SHP outputFormat for example, the plugin create a gml file, then he converts this one in shp and finally in a zip file.
All theses files are deleted when no error and the final file is send, and when an error is occuring, the plugin stop, send a corrupt file and delete the temporary files but the space is not integrally unrealesed, qgis_maps process keep some space with the deleted files.
We must restart apache to unrealsed the space.

An extract of the lsof command :
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
qgis_maps 61436 www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61440 qgis_maps www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61441 QXcbEvent www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61442 Qt\x20bea www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61443 QDBusConn www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61444 QCA::KeyS www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61468 www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61472 qgis_maps www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61473 QXcbEvent www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61474 Qt\x20bea www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61475 QDBusConn www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61476 QCA::KeyS www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)

• QGIS Server FULL version (with 3 numbers) :
  Qgis Server 3.16.15

• OS on the server :
  Debian 10.8

writeAsVectorFormat is now deprecated

I have a warning saying it's deprecated.

Use writeAsVectorFormatV2

wfsoutputExtension windows server compatibility

is this extension is compatible with windows server?

if so how to enable this?

Regards

Is a stable relase planned?

Hi there,

Nice plugin, thanks!

Is there a reason why the plugin is still flagged as experimental after 5 years? Do you plan to release it as stable?
It's for a friend 😉

Remowe wfsOutputExtension from QGIS Desktop plugins list?

Description

From QGIS Desktop, Plugins manager dialog, it is possible to install the wfsOutputExtension plugin but as far as I understand, this is a QGIS Server only plugin. Moreover, when you open QGIS after the install on desktop, you always get

WfsOutputExtension is plugin for QGIS Server. There is nothing in QGIS Desktop.

Context

• QGIS Server FULL version (with 3 numbers) :
• OS on the server :

Remove temporary files

Description

There is nothing to remove temporary files.

• Temporary GML, it looks easy, as soon as we have the QgsFileVectorWritter in the new format, we can remove the GML file.
• Temporary target extension, need to check for that one

Encoding csv export

Is there any setting for the encoding (utf8, unicode) when exporting the attribute tables as csv? I'm working with french data, so a lot of accents...

[CDATA[<text]] in exported xlsx tables

When there are special characters in text fields (e.g. < or &) in exported excel tables we find <![CDATA[<text]] (no error message is returned opening xlsx file).

New export formats not showing in drop down menu

I installed the plugin and it seems to load normally. However I only get the standard export formats (GML, GeoJSON) in the drop down menu. I restarted Apache multipe times but nothing changed.

Here are some of the log file contents:

Loading the plugin

11:52:24 INFO WfsOutputExtension[961696]: WfsOutputExtension : tempdir : /tmp/QGIS_WfsOutputExtension  
11:52:24 INFO Server[961696]: Server plugin wfsOutputExtension loaded!  
11:52:24 INFO wfsOutputExtension[961696]: SUCCESS - init

Accessing a WFS layer

11:52:29 INFO Server[961696]: TYPENAME:wirtschaftliche_einheiten  
11:52:29 INFO Server[961696]: VERSION:1.0.0
11:52:29 WARNING [961696]: Service WFS 1.0.0 not found, returning default
11:52:29 INFO Server[961696]: WFS Request parameters:
11:52:29 INFO Server[961696]:  - OUTPUTFORMAT : GeoJSON
11:52:29 INFO Server[961696]:  - PROPERTYNAME : geometry,Kassenzeichen
11:52:29 INFO Server[961696]:  - TYPENAME : wirtschaftliche_einheiten
11:52:29 INFO Server[961696]:  - GEOMETRYNAME : extent
11:52:29 INFO Server[961696]:  - VERSION : 1.0.0
11:52:29 INFO WfsOutputExtension[961696]: WfsOutputExtension.sendResponse
//... see comment below
11:52:29 INFO WfsOutputExtension[961696]: WfsOutputExtension.responseComplete
11:52:29 INFO Server[961696]: Request finished in 156 ms

The line WfsOutputExtension.sendResponse I got over 1200 times for each request. Seems pretty high...

Fix name tag on GH Action

Description

Compare the last 2 tags about v prefix

Export Format don't appear in Lizmap

Hello,

I installed the HelloWorld et the wfsOutputExtension plugins and configured the fcgid conf file of apache2 ("/etc/apache2/conf-available/serve-cgi-bin.conf") like describes in the documentation.

When I try the url on the Hello World plugin (http://151.236.35.125/cgi-bin/qgis_mapserv.fcgi?SERVICE=Hello), it works and I've a response.
But when I try the export on lizmap, I just have GeoJson and GML formats ...

Qgis version : 3.4.13
Lizmap : 3.3.6
Lizmap plugin : 3.2.13
Qgis-server : 3.4.15

Thanks for your help.

Installation error

I follow the instructions in the README file and I encountered the following error after apache2 restart:

WARNING Python error[15902]: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 309, in loadPlugin
    __import__(packageName)
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 685, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'qgis-wfsOutputExtension-1'

16:34:41 WARNING Couldn't load plugin 'qgis-wfsOutputExtension-1.2.2'[15902]: Couldn't load plugin 'qgis-wfsOutputExtension-1.2.2'


ModuleNotFoundError: No module named 'qgis-wfsOutputExtension-1'


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 309, in loadPlugin
    __import__(packageName)
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 685, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'qgis-wfsOutputExtension-1'




Python version: 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]


QGIS version: 3.4.15-Madeira Madeira, e83d02e274


Python Path:

/usr/share/qgis/python//.local/share/QGIS/QGIS3/profiles/default/python/opt/qgis-server/plugins//.local/share/QGIS/QGIS3/profiles/default/python/plugins/usr/share/qgis/python/plugins/usr/lib/python36.zip/usr/lib/python3.6/usr/lib/python3.6/lib-dynload/usr/local/lib/python3.6/dist-packages/usr/lib/python3/dist-packages

If I change the name of the folder from qgis-wfsOutputExtension-1.2.2 to qgis-wfsOutputExtension I do not encountered errors but I do not see the new export formats available from lizmap.

Any suggestion?

Does not work on newer versions of QGIS Server

Hi, Does this plugin work on the latest versions of QGIS Server?

I'm trying to use this plugin on QGIS Server versions 3.16 and 3.18 with LWC but without success.
I tested the HelloServer plugin and it worked.

My WFS GetCapabilities does not show other options for export, only geojson/gml.

I already cleaned the temp/lizmap

I don't know how to install older versions of QGIS Server.

My specs:
Debian 10
LWC 3.3.12
QGIS SERVER 3.16

Thank you

MIME-Type is not correct : application/zip for ODS files

Apparently the exported ods file is unzipped, so it cannot be opened directly by LibreOffice.

Use the new API for QGIS Server for QGIS 3.24

Related to qgis/QGIS#45158

Disable GPX export for polygons

Hello,
It would be interesting to add the gpx format to the list in order to easily load the data into a GPS.

'gpx':{
        'contentType': 'application/gpx+xml',
        'filenameExt': 'gpx',
        'forceCRS': 'EPSG:4326',
        'ogrProvider': 'GPX',
        'ogrDatasourceOptions':None,
        'zip': False,
        'extToZip': []
    },

Thanks

Additional outputFormats with WFS when using describeFeatureType

WFS only supports text/xml through gml when using describeFeatureType-request. It would be useful to be able to use additional output formats.

For making QGIS Server compatible with more clients and operations, it would be great if it could accept additional variants, e.g;

application/json (Media Type used by GeoServer)

Unreleased space after deleting files

Description

This issue follow the #37
When we have not enought space in the apache temp filesystem, the temporary files are now deleted in the 1.7.0 version but some space are unrealesed by the qgis_maps process.

Context

Qgis server is using with apache2 and fcgi.
When we call the plugin for a large file, send with a SHP outputFormat for example, the plugin create a gml file, then he converts this one in shp and finally in a zip file.
All theses files are deleted when no error and the final file is send, and when an error is occuring, the plugin stop, send a corrupt file and delete the temporary files but the space is not integrally unrealesed, qgis_maps process keep some space with the deleted files.
We must restart apache to unrealsed the space.

An extract of the lsof command :

COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
qgis_maps 61436 www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61440 qgis_maps www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61441 QXcbEvent www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61442 Qt\x20bea www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61443 QDBusConn www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61436 61444 QCA::KeyS www-data 20r REG 254,7 681464714 29 /tmp/QGIS_WfsOutputExtension/gml_features_1649074437.4294026.gml (deleted)
qgis_maps 61468 www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61472 qgis_maps www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61473 QXcbEvent www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61474 Qt\x20bea www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61475 QDBusConn www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)
qgis_maps 61468 61476 QCA::KeyS www-data 20r REG 254,7 681464714 37 /tmp/QGIS_WfsOutputExtension/gml_features_1649074453.1260824.gml (deleted)

• QGIS Server FULL version (with 3 numbers) :
  Qgis Server 3.16.15

• OS on the server :
  Debian 10.8

Add .cpg file for Shapefile output

Currently there's no .cpg file included for the Shapefile output ZIP to specify the codepage for identifying the characterset to be used:

https://geo.kreis-viersen.de/ows/osm-daten?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.1.0&TYPENAME=wertstoffhoefe_nrw&SRSNAME=EPSG:25832&OUTPUTFORMAT=SHP

How can you install on a QGIS SERVER installation in windows with OSGeo4W?

How can you install on a QGIS SERVER installation in windows with OSGeo4W?