pramsey / pgsql-ogr-fdw Goto Github PK

I always thought ogr throws in a geometry column even if the file source is not spatial. It turns out it doesn't or at least doesn't seem to in all drivers.

I just queried an excel spreadsheet (XLS) (which requires you compile GDAL with the Freexl driver) - https://www.gaia-gis.it/fossil/freexl/index and the output of ogr_fdw_info did not work. It kept on complaining about casting geometry to date.

Once I removed the geometry column from the definition, it worked great.
Evidentally same issue with at least some CSV files.

I grabbed this one from here:
http://catalog.data.gov/dataset/2010-federal-stem-education-inventory-data-set

and ogr_fdw_info gave me this:

CREATE SERVER myserver
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'C:\fdw_data\Consumer_Complaints.csv',
    format 'CSV' );

CREATE FOREIGN TABLE consumer_complaints (
  fid integer,
  geom geometry,
  complaint_id varchar,
  product varchar,
  sub_product varchar,
  issue varchar,
  sub_issue varchar,
  state varchar,
  zip_code varchar,
  submitted_via varchar,
  date_received varchar,
  date_sent_to_company varchar,
  company varchar,
  company_response varchar,
  timely_response_ varchar,
  consumer_disputed_ varchar )
  SERVER myserver
  OPTIONS ( layer 'Consumer_Complaints' );

Which gave error

ERROR:  column "geom" of foreign table "consumer_complaints" converts OGR "String" to "geometry"

Once I dropped the geometry column from the table definition, doing

  select * from consumer_complaints limit 10;
  select product, count(*) from consumer_complaints
GROUP BY product;

both worked great. The file is kinda biggish so doing the summary took 4 seconds, but not bad considering the file is 52 MB.

I was trying a query with this:

http://download.geofabrik.de/north-america/us/district-of-columbia.html

 ogr_fdw_info -s "C:\fdw_data\gisdata\district-of-columbia-latest.osm" -l lines

which outputs this:

     CREATE SERVER myserver
      FOREIGN DATA WRAPPER ogr_fdw
      OPTIONS (
        datasource 'C:\fdw_data\gisdata\district-of-columbia-latest.osm',
        format 'OSM' );

    CREATE FOREIGN TABLE lines (
      fid integer,
      geom geometry,
      osm_id varchar,
      name varchar,
      highway varchar,
      waterway varchar,
      aerialway varchar,
      barrier varchar,
      man_made varchar,
      z_order integer,
      other_tags varchar )
      SERVER myserver
      OPTIONS ( layer 'lines' );

However when I go to query the table it created, it complains that z_order is a string and not an integer.

So I changed the definition to integer and notice, it seems the z_order is missing and what ends up showing in z_order is the other_tags value.

I checked to see if perhaps a gdal issue by running this query is is using the same gdal library as my ogr_fdw

ogrinfo "C:\fdw_data\gisdata\district-of-columbia-latest.osm" lines | more

And that looked right, z_order values were there and data structure agrees with ogr_fdw

Layer name: lines
Geometry: Line String
Feature Count: -1
Extent: (-77.120100, 38.791340) - (-76.909060, 38.996030)
Layer SRS WKT:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        TOWGS84[0,0,0,0,0,0,0],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9108"]],
    AUTHORITY["EPSG","4326"]]
osm_id: String (0.0)
name: String (0.0)
highway: String (0.0)
waterway: String (0.0)
aerialway: String (0.0)
barrier: String (0.0)
man_made: String (0.0)
z_order: Integer (0.0)
other_tags: String (0.0)
OGRFeature(lines):4681186
  osm_id (String) = 4681186
  name (String) = (null)
  highway (String) = residential
  waterway (String) = (null)
  aerialway (String) = (null)
  barrier (String) = (null)
  man_made (String) = (null)
  z_order (Integer) = 3
  other_tags (String) = (null)
  LINESTRING (-77.1082606 38.935738,-77.1082557 38.9358361,-77.1082555 38.9359244,-77.1082552 38.9360971,-77.1082544 38.9365412,-77.1082541 38.9366631)

OGRFeature(lines):4725152
  osm_id (String) = 4725152
  name (String) = Texas Avenue Southeast
  highway (String) = secondary
  waterway (String) = (null)
  aerialway (String) = (null)
  barrier (String) = (null)
  man_made (String) = (null)
  z_order (Integer) = 6
  other_tags (String) = "HFCS"=>"Collector","lanes"=>"1","oneway"=>"yes","source:HFCS"=>"District of Columbia (DC GIS)"
  LINESTRING (-76.9386616 38.8891714,-76.9383773 38.8893008,-76.9382276 38.889401,-76.9381042 38.8895367,-76.938026 38.8896185,-76.9379174 38.8896617)

I don't seem to have the same issue if I use the osm.pbf version of the file.

I'm going to try next with a small .osm file to rule out its something specific to this file.

I'll double-check to make sure it's not a library issue, but I'm pretty sure I was compiling with GDAL 2.0.2 or GDAL 2.1 which both support this.

-- So I imported some tiger data:

``

CREATE SERVER svr_tiger FOREIGN DATA WRAPPER ogr_fdw
OPTIONS (
datasource 'E:/GISData/tiger',
format 'ESRI Shapefile'
);

IMPORT FOREIGN SCHEMA ogr_all
FROM SERVER svr_tiger INTO staging;

``
All good so far, but when I go to query bamm!

``
SELECT * from staging.tl_2015_36029_addr limit 10;

``

ERROR: unsupported OGR type "Integer64"
********** Error **********

ERROR: unsupported OGR type "Integer64"
SQL state: XX000

This same exercise works fine on my PostgreSQL 9.5 running ogr_fdw 1.0.1. So I think this is a recent regression.

The census site seems to be having issue of late but here is one of the problem files
tl_2015_36041_addr.zip

Hi,

Can you please release a tarball, so that I can build RPMs of pgsql-ogr.fdw?

Thanks!
Regards, Devrim

When I do a make clean

All the .o files and .dll are gone, but ogr_fdw_info.exe still remains. And compiling as a result doesn't rebuild the .exe. Discovered this when testing my 32-bit and it didn't work.

ogrFeatureToSlot() assumes that the CREATE FOREIGN TABLE statement used is exactly the one suggested by ogr_fdw_info. If trying something else, PG columns receive content from unexpected OGR fields.

Excerpted from 9.5.0 release today

Foreign Data Wrappers (FDWs): These already allow using PostgreSQL as a query engine for other Big Data systems such as Hadoop and Cassandra. Version 9.5 adds IMPORT FOREIGN SCHEMA and JOIN pushdown making query connections to external databases both easier to set up and more efficient.

Is that possible to do with gdal? If it isn't feel free to close this out. It's probably only relevant for some datasources anyway.

I'll try to take a stab at fixing this when I have time.

Lots of warnings and these errors:

`
^
ogr_fdw.c: In function 'ogrGetForeignPaths':
ogr_fdw.c:554:6: error: incompatible type for argument 3 of 'create_foreignscan_path'
)
^
ogr_fdw.c:554:6: error: incompatible type for argument 6 of 'create_foreignscan_path'
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: expected 'Cost' but argument is of type 'struct List *'
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
^
ogr_fdw.c:554:6: warning: passing argument 9 of 'create_foreignscan_path' from incompatible pointer type [enabled by default]
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: expected 'struct Path *' but argument is of type 'struct List *'
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
^
ogr_fdw.c:554:6: error: too few arguments to function 'create_foreignscan_path'
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG9~1.6W6/include/server/optimizer/pathnode.h:90:21: note: declared here
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,

In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: expected 'struct PathTarget *' but argument is of type 'double'
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
^
ogr_fdw.c:554:6: error: incompatible type for argument 6 of 'create_foreignscan_path'
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: expected 'Cost' but argument is of type 'struct List *'
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
^
ogr_fdw.c:554:6: warning: passing argument 9 of 'create_foreignscan_path' from incompatible pointer type [enabled by default]
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: expected 'struct Path *' but argument is of type 'struct List *'
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
^
ogr_fdw.c:554:6: error: too few arguments to function 'create_foreignscan_path'
)
^
In file included from ogr_fdw.h:41:0,
from ogr_fdw.c:29:
c:/MING641/projects/POSTGR1/rel/PG91.6W6/include/server/optimizer/pathnode.h:90:21: note: declared here
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,

`

Evidentally I guess something changed upstream in createForeignPath

My 9.6 regression tests failed but they look like bogus errors because you have your folder path hard-coded in there. I recall you had some variable in there instead.

*** c:/projects/sources/pgsql_ogr_fdw/expected/ogr_fdw.out  Fri Apr 22 10:23:29 2016
--- c:/projects/sources/pgsql_ogr_fdw/results/ogr_fdw.out   Fri Apr 22 10:23:30 2016
***************
*** 2,8 ****
  CREATE SERVER myserver
    FOREIGN DATA WRAPPER ogr_fdw
    OPTIONS (
!     datasource '/Users/pramsey/Code/pgsql-ogr-fdw/data',
      format 'ESRI Shapefile' );
  ------------------------------------------------
  CREATE FOREIGN TABLE pt_1 (
--- 2,8 ----
  CREATE SERVER myserver
    FOREIGN DATA WRAPPER ogr_fdw
    OPTIONS (
!     datasource 'c:/projects/sources/pgsql_ogr_fdw/data',
      format 'ESRI Shapefile' );
  ------------------------------------------------
  CREATE FOREIGN TABLE pt_1 (
***************
*** 76,82 ****
  CREATE SERVER myserver_latin1
    FOREIGN DATA WRAPPER ogr_fdw
    OPTIONS (
!     datasource '/Users/pramsey/Code/pgsql-ogr-fdw/data',
      format 'ESRI Shapefile',
      config_options 'SHAPE_ENCODING=LATIN1' );
  CREATE FOREIGN TABLE e_1 (
--- 76,82 ----
  CREATE SERVER myserver_latin1
    FOREIGN DATA WRAPPER ogr_fdw
    OPTIONS (
!     datasource 'c:/projects/sources/pgsql_ogr_fdw/data',
      format 'ESRI Shapefile',
      config_options 'SHAPE_ENCODING=LATIN1' );
  CREATE FOREIGN TABLE e_1 (

I tried both with SQL Server and MS Access.

using timestamptz or timestamp gives same issue with tryng to map to Date/Time
What shows in the logs is this if I set level to 3 (after your nice debug change)

DEBUG:  converting OFTDateTime '2008-11-26 14:35:00' from OGR
ERROR:  timestamp(133760) precision must be between 0 and 6

I should add that using a date column in SQL Server seems to work fine (so its just the datetime to timestamp or timestamptz)

Sorry, Issue got submitted before text was completed. Please close issue #36.

I am running Windows 7 64-bit, but have posgresql -32-bit installed. I installed ogr_fdw from http://www.postgresonline.com/downloads/fdw_win32_94_bin.zip (not sure who compiled this version for Windows).

I have successfully connected to an old Microsoft Visual Foxpro database using a 32-bit odbc DSN. This same database attaches just fine via odbc to a Microsoft Access 2002 front end. It also appears to work when attached to postgresql via ogr_fdw, except that there seems to be some strange datatype mismatching going on.

In the postgresql examples below flag2 is specified as flag2 varchar(1)and fid is specified as fid integer. These definitions match the specifications in the Visual Foxpro database. When I run the following query, I get a null result, which is incorrect.

Testdb=# SET client_min_messages = debug;
SET
Testdb=# Select fid, desc_int, flag1, flag2
Testdb-# from "table_$" where flag2 = 'N' and fid = 1;
DEBUG: OGR SQL: (flag2 = 'N') AnD (fid = 1)
fid | desc_int | flag1 | flag2
-----+----------+-------+-------
(0 rows)

But if I augment the field variables flag2 and fid in the SQL query (probably causing an automatic internal data cast to char and integer, respectively) it works correctly:

Testdb=# Select fid, desc_int, flag1, flag2
Testdb-# from "table_$" where '' || flag2 = 'N' and fid + 0 = 1;
DEBUG: OGR SQL:
fid | desc_int | flag1 | flag2
-----+----------+-------+-------
1 | ? | N | N
(1 row)

These two queries should return identical results.

A straightforward CAST operation doesn't work:

Testdb=# Select fid, desc_int, flag1, flag2
Testdb-# from "table_$" where cast (flag2 as varchar(1)) = 'N' and
Testdb-# cast (fid as integer) = 1;
DEBUG: OGR SQL: (flag2 = 'N') AnD (fid = 1)
fid | desc_int | flag1 | flag2
-----+----------+-------+-------
(0 rows)

I always get a null result with any data type unless I force a "cast" to the variable(s) through a null operation.

Any Ideas?

Okay just happened to try ogr_fdw querying a spatial datasource on a database that happens to have postgis installed in a schema called "postigs" instead of default public location:

This is querying an OSM pbf file, but I suspect that doesn't matter and yes postgis is in my database search_path.

ALTER EXTENSION postgis
  SET SCHEMA postgis;

-- start a new connection run this

SELECT ST_AsText(geom) from points  limit 1;

-- and I get this
NOTICE:  conversion to geometry called with column type not equal to bytea or geometry

Repeat the same exercise:

ALTER EXTENSION postgis SET SCHEMA public;

-- start a new connection
SELECT ST_AsText(geom) from points  limit 1;

-- and I haz me a reasonable answer with no error

POINT(-77.0887311 38.9672163)

Did you by chance get rid of the quote_ident I put in for import schema? With my fresh pot of ogr_fdw, I can no longer import my OSM files.

`NOTICE: Number of tables to be created 5

ERROR: syntax error at or near "natural"
LINE 22: natural varchar,
^
QUERY: CREATE FOREIGN TABLE multipolygons (
fid integer,
geom geometry,
osm_id varchar,
osm_way_id varchar,
name varchar,
type varchar,
aeroway varchar,
amenity varchar,
admin_level varchar,
barrier varchar,
boundary varchar,
building varchar,
craft varchar,
geological varchar,
historic varchar,
land_area varchar,
landuse varchar,
leisure varchar,
man_made varchar,
military varchar,
natural varchar,
office varchar,
place varchar,
shop varchar,
sport varchar,
tourism varchar,
other_tags varchar
) SERVER osm_dc
OPTIONS (layer 'multipolygons');
********** Error **********

ERROR: syntax error at or near "natural"
SQL state: 42601`

Should be stated that this FDW only supports PG 9.3 or higher.

This is not a bug, but a feature request.

http://michael.otacoo.com/postgresql-2/postgres-9-5-feature-highlight-import-foreign-schema/

I don't remember seeing this notice until recently. I'm guessing it's harmless since the results look okay, but just wanted to make sure:

NOTICE:  unsupported OGR FDW expression type, T_ScalarArrayOpExpr

I get the above when doing something like

SELECT a, b
FROM c
WHERE b NOT IN ('1','2','3');

I assume that just means the query had to be done in postgres because OGR doesn't support IN? Get same notice with IN. This is against an MS Access db.

So I have this access database, and it has numbered fields.

When I use ogr_fdw_info to generate the metadata, I get something like this:

CREATE FOREIGN TABLE rates_2da (
  fid integer,
  geom geometry,
  type varchar,
  weight real,
  202 real,
  203 real,
  204 real,
  205 real,
  206 real,
  207 real,
  208 real,
  224 real,
  225 real,
  226 real )
  SERVER myserver
  OPTIONS ( layer 'Rates_2DA' );

And since numbered fields aren't legal in PostgreSQL, I have to go in and quote all of them. Would be nice if this was automagically done for me :)

By applying OGR_L_SetIgnoredFields to the OGR Layer before reading, we can dramatically reduce the amount of data pulled over the wire in response to a particular query target list.

If the query is just:

SELECT name FROM thetable;

Then the backend should not be pulling all the fields over the wire, but that's the default OGR behaviour and our default too.

There is a knock-on efficiency to this work for update support, in that every update starts with a query. That is,

UPDATE thetable SET name = 'bar' WHERE age = 20;

will first run

SELECT name FROM thetable WHERE age = 20;

to get a list of candidate records. Those records are then iterated on to apply the update back to the foreign source. Without using ignored fields, we have to pull all the data in those records back, even though we're only updating the name column.

This will require careful updates to all functionality to use only the requested fields in slot handling for both read and write.

ogr_fdw_info gives this for help

       ogr_fdw_config -s <ogr datasource>

what is ogr_fdw_config?

This might be an issue in GDAL and not here, but since I see the problem here, I'm complaining here..

I can understand that getting list of layers is slow, but why does querying a given table after you have the layers have to be slow?

For example I was using the ODBC driver to query an Oracle database. Same behavior for SQL Server database too.

It took a whole 11 minutes to query a table with 4 rows.

From memory, I recalled I had the same issue with SQL Server if it had many tables, so I changed my Foreign server to something like below to itemize only the tables I cared about.

CREATE SERVER oracle_svr
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'ODBC:system/whatever@oraserver,GIS.BUILDINGS_STATUS_LOOKUP,GIS.AFFORDABLE_HOUSING',
    format 'ODBC' );

Then it was very fast querying from tables.

I'll try to replicate the same issue in GDAL to see if GDAL is the culprit. It could also be an issue with the ODBC driver and not all data sources.

Issue with this file - http://www.bostongis.com/postgisstuff/london_tube_lines.zip

This is a file I had gotten originally in KML format from http://www.doogal.co.uk/london_stations.php and converted to ESRI shapefile with

    ogr2ogr -f "ESRI Shapefile" london_tube_lines "London tube Lines.kml" -nln london_tube_lines

I was also able to load this shp converted file fine with PostGIS packaged shp2pgsql

However when I run

ogr_fdw_info -s "C:\fdw_data\shp" -l london_tube_lines

I get this:

CREATE SERVER myserver
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'C:\fdw_data\shp',
    format 'ESRI Shapefile' );

CREATE FOREIGN TABLE london_tube_lines (
  fid integer,
  geom geometry,
  name varchar,
  descriptio varchar,
  timestamp date,
  begin date,
  end date,
  altitudemo varchar,
  tessellate ERROR 1: Unsupported GDAL type 'Integer64'

Just breaks at the tessellate

ogrinfo gives me this info about the structure:

Layer name: london_tube_lines
Geometry: 3D Line String
Feature Count: 497
Extent: (-0.611415, 51.378552) - (0.251415, 51.705380)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_84",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
Name: String (80.0)
descriptio: String (80.0)
timestamp: Date (10.0)
begin: Date (10.0)
end: Date (10.0)
altitudeMo: String (80.0)
tessellate: Integer64 (10.0)
extrude: Integer64 (10.0)
visibility: Integer64 (10.0)
drawOrder: Integer64 (10.0)
icon: String (80.0)

So it seems ogr_fdw_info is at a loss what to do with this Integer64 datatype and gives up.

Firstly, thank you for this wrapper.
In one of our cases, the problem : X/Y are inverted. Perhaps a OGR error about WFS version (1.0.0, 1.1.0) ? The version is it transmitted/exploited ?

CREATE SERVER gesteau
FOREIGN DATA WRAPPER ogr_fdw
OPTIONS (
datasource 'WFS:http://gesteau.eaufrance.fr/service?SERVICE=WFS&VERSION=1.1.0',
format 'WFS' );
...
OPTIONS ( layer 'sage' )

Okay I built with your latest commit: f93a7e0

While this doesn't cause an issue for flat files or WFS services I tested, it seems to have completely broken my SQL Server queries I access via ODBC.

Count always returns 0 and doing any kind of select returns no records.

Another feature request and not sure this is feasible so dismiss if not.

I assume this is possible to infer from ogr since as I recall when I import with ogr it does create typmoded geometries.

Currently -- ogr_fdw_info gives me something like this:

CREATE FOREIGN TABLE mbta_node (
  fid integer,
  geom geometry,
  objectid integer,
  station varchar,
  line varchar )
  SERVER myserver
  OPTIONS ( layer 'MBTA_NODE' );

and then I got to go look at the source to figure out I can do this geom geometry(POINT,26986), but
unfortunately though I can get away with this and it registers correctly in geometry_columns,

my geometries when I do ST_SRID(geom) are still 0. So my spatial joins are kind of ugly cause I have to do a ST_SetSRID then possibly and additional ST_Transform or create a view wrapper around my foreign table.

It would be nice if it both the ogr_fdw could infer the SRID and also if ogr_fdw_info could produce the corresponding meta data.

Just tried building pgsql-ogr-fdw against recently released PostgreSQL 9.5beta1, and sadly it doesn't even get out of the compile gate.

ogr_fdw.c: In function 'ogr_fdw_validator':
ogr_fdw.c:317:30: warning: variable 'layer' set but not used [-Wunused-but-set-variable]
  const char *source = NULL, *layer = NULL, *driver = NULL;
                              ^
ogr_fdw.c: In function 'ogrGetForeignPlan':
ogr_fdw.c:590:26: error: too few arguments to function 'make_foreignscan'
                          fdw_private);
                          ^
In file included from ogr_fdw.h:42:0,
                 from ogr_fdw.c:29:
e:/jenkins/postgresql/rel/pg9.5w64gcc48/include/server/optimizer/planmain.h:46:21: note: declared here
 extern ForeignScan *make_foreignscan(List *qptlist, List *qpqual,
                     ^
ogr_fdw.c:591:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^

It it possible to convert the utility program ogd_fdw_info into a PL/pgSQL function residing in the OGR_FDW extension ? And let this function create the fdw table directly ?

Something like:
SQL> CREATE EXTENSION OGR_FDW;

SQL> SELECT OGR_FDW_CREATE_TABLE(' ... OGR-def....', 'layername', 'my_new_table');
or
SQL> SELECT OGR_FDW_CREATE_TABLE('...OGR-def....','layername','my_new_table',MultiPolygon,4326);

The last call would create the table with a predefined typemod on the geometry column.

Is it just me or does make installcheck just not finished yet?

I do
make installcheck

after make install, and it creates the contrib_regression database, but it fails to install the extension and ends with error:

(using postmaster on localhost, port 5444) ============== dropping database "contrib_regression" ============== DROP DATABASE ============== creating database "contrib_regression" ============== CREATE DATABASE ALTER DATABASE ============== running regression test queries ============== test ogr_fdw ... The system cannot find the path specified.

If I go into the the contrib_regression database it created and then run

CREATE EXTENSION ogr_fdw;

that works fine so I assume I have all the dependencies installed it needs.

Hi,

I execute ogr_fdw_info succesfully:

D:\Program Files\PostgreSQL\9.3\bin> .\ogr_fdw_info.exe -s "ODBC:SQLPROD" -l "dbo.MCOMERC"

OUT:
Warning 1: No column definitions found for table 'dbo.VeladoresA├▒oActual', layer not usable.

CREATE SERVER myserver
FOREIGN DATA WRAPPER ogr_fdw
OPTIONS (
datasource 'ODBC:SQLPROD',
format 'ODBC' );

CREATE FOREIGN TABLE dbo_mcomerc (
fid integer,
codtipoexp varchar,
a_o integer,
m_propietario_local varchar )
SERVER myserver
OPTIONS ( layer 'dbo.MCOMERC' );

If I execute the CREATE SERVER statement in Postgres I get:

********** Error **********

ERROR: unable to connect to data source "ODBC:SQLPROD"
SQL state: HV00D

My PostgreSQL version: "PostgreSQL 9.3.5, compiled by Visual C++ build 1600, 64-bit"

My PostGIS version: "POSTGIS="2.2.0 r14208" GEOS="3.5.0-CAPI-1.9.0 r4090" PROJ="Rel. 4.9.1, 04 March 2015" GDAL="GDAL 2.0.0, released 2015/06/14" LIBXML="2.7.8" LIBJSON="0.12" RASTER"

OS: Windows 8.1

Any help or advice would be appreacited..

Greetings, did the following using ogr_fdw_info and got the following:

C:\Program Files\PostgreSQL\9.4\bin>ogr_fdw_info.exe -s MSSQL:server=JPS-DB2-GN-TEST;database=ElectricGIS;uid=postgres;pwd=postgres -l dbo.servicetransformer

CREATE SERVER myserver
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'MSSQL:server=JPS-DB2-GN-TEST;database=ElectricGIS;uid=postgres;pwd=postgres',
    format 'MSSQLSpatial' );

CREATE FOREIGN TABLE dbo_servicetransformer (
  fid integer,
  geom geometry,
  premisesnumber varchar,
  transformerfacilityid varchar,
  x real,
  y real,
  lastmodified timestamp )
  SERVER myserver
  OPTIONS ( layer 'dbo.ServiceTransformer' );


C:\Program Files\PostgreSQL\9.4\bin>

Copied it verbatim into pgadmin. then ran select * from dbo_servicetransformer limit 50 which gives me the following error:

ERROR: unable to connect to layer to "dbo.ServiceTransformer"
SQL state: HV00J
Hint: Error initializing the metadata tables : [Microsoft][ODBC SQL Server Driver][SQL Server]The specified schema name "dbo" either does not exist or you do not have permission to use it.

All other connections using ogr to the same mssql server work (i.e all non ogr_fdw connections) and clearly ogr_fdw_info works, so I'm wondering if the database user (postgres) is making something go wonky. My initial thought is that it is ignoring the uid and password in the connection string thus throwing the permissions error.

Be prepared, we're going to have to change the foreign scan thingy for 9.5 yet again. Let's wait till the next beta release though so people can build against 9.5beta1 tar balls.

http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5043193b78919a1bd144563aadc2f7f726549913

found this out from adunstan/file_text_array_fdw#2

I'll have to try this again to see if I can consistently replicate. I was able to crash my postgres backend

(My version is older that here, but this should work too) - http://www.massdot.state.ma.us/planning/Main/MapsDataandReports/Data/GISData/RoadInventory.aspx

I was able to crash my backend by running this query on 2 separate simultaneous connections and cancelling one query - when cancelling the backend crashes.

SELECT count(*) from road_inv where fromcity = 131; ```

This is running on windows PostgreSQL 9.3.5 32-bit with mingw compiled pgsql-ogr-fdw

Though I haven't been able to consistently replicate the issue to my disappointment.

Can we have a tagged release so have release that works with 9.5.

I'm sure you probably know this, so consider it a feature enhancement request.
I noticed the TDS FDW driver has a similar issue, though it just returns null instead of throwing an error, which is less nice.

Here is a really lame example to demonstrate with your sample data.

SELECT name, age, (SELECT geom FROM pt_two AS t where t.name = p.name limit 1) As geom 
FROM 
pt_two As p;

Give error:

ERROR:  got into ogrDeparseParam code
********** Error **********

ERROR: got into ogrDeparseParam code
SQL state: XX000

Is it possible to use this in Windows?

Ubuntu 14.04 LTS
gdal-config --version : 1.11.2
psql --version : psql (PostgreSQL) 9.3.6

The extension built and installed normally.
When trying to enable the ogr_fdw extension, I am running into an error:

create extension ogr_fdw;
ERROR: could not load library "/usr/lib/postgresql/9.3/lib/ogr_fdw.so": /usr/lib/postgresql/9.3/lib/ogr_fdw.so: undefined symbol: OGR_FD_GetGeomFieldCount

I accidentally posted this to the wrong ticket. Might as well add it as bug now that you have a pseudo working installcheck

make installcheck sort works now :)

The only issue remaining is I had to manually create an expected and an sql folder for it to work.

OGR doesn't really have a good idea of what the SRID of a geometry coming in is, but pgsql does, so apply that info from the table typmod if possible?

Just pulled the latest and I don't recall having this issue before. Now the installcheck gives:

CREATE EXTENSION ogr_fdw;

• ERROR: OGR FDW only works with UTF-8 databases

This is fine except some people might not have server initialized to UTF-8. Like my windows dev install, defaults to WIN1252. Wondering if it would make sense to create the database with template0 and then enforce creation with UTF-8 so that people can test even if their cluster does not have UTF-8 as the default.

I get

Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () when using ogr_fdw_info on the above system. I have tried with both XLS files and SHP files with the same result. I can't get the actual data wrapper to work either - the message is ERROR: unable to connect to data source "...." SQL state: HV00D

I tried running ogr_fdw_info as root to no avail. The ogr_fdw_info binary doesn't seem to be installed anywhere - not sure if that is a problem?

I could maybe try a custom build of a newer version of GDAL to see if that helps.

I know I'm really being petty here, but I suspect other package maintainers will complain, so best you here it first from your ol' pal.

Here: https://github.com/pramsey/pgsql-ogr-fdw/releases you have release as 0.1

BUT the extension install reads 1.0.

There is a bit of a disconnect here no? Now since you can't go backwards, you should just call it 1.0.1 or something

I am running Windows 7 64-bit, but have posgresql -32-bit installed. I installed ogr_fdw from http://www.postgresonline.com/downloads/fdw_win32_94_bin.zip (not sure who compiled this version for Windows).

I have successfully connected to an old Microsoft Visual Foxpro database using a 32-bit odbc DSN. This same database attaches just fine via odbc to a Microsoft Access 2002 front end. It also appears to work when attached to postgresql via ogr_fdw, except that there seems to be some strange datatype mismatching going on.

In the postgresql examples below flag2 is specified as flag2 varchar(1) and fid is specified as fid integer. These definitions match the specifications in the Visual Foxpro database. When I run the following query, I get a null result, which is incorrect.

Testdb=# SET client_min_messages = debug;
SET
Testdb=# Select fid, desc_int, flag1, flag2
Testdb-# from "table_$" where flag2 = 'N' and fid = 1;
DEBUG: OGR SQL: (flag2 = 'N') AnD (fid = 1)
fid | desc_int | flag1 | flag2
-----+----------+-------+-------
(0 rows)

I am trying to understand the different levels of encoding/recoding in ogr_fdw.

I have a dataset which I can load into my database using regular ogr2ogr :

export PGCLIENTENCODING=latin1
ogr2ogr -overwrite -f "postgresql" pg:"dbname=opendata" -nln matrikelkort.jordstykke_1084 /vsizip//vsicurl/ftp://testuser_ogr_fdw:[email protected]/matrikeldata/matrikelkort/SHAPE/1084_SHAPE_UTM32-EUREF89.zip/1084_SHAPE_UTM32-EUREF89/MINIMAKS/BASIS/JORDSTYKKE.shp JORDSTYKKE

Note: The datasource is free and open data from Denmark. I created the user/pass for this issue to make it easier to reproduce.
You can create your own user/pass here if you like.

My database is UTF8 and the data looks fine when querying records with latin1 characters:

SELECT elavsnavn, 
       elavskode, 
       matrnr 
FROM   matrikelkort.jordstykke_1084 
WHERE  elavskode = 2000154 
LIMIT  1 

Returns

Which is perfect!

Now that I have tested and confirmed I can load the data correctly with ogr2ogr, I will set up a FOREIGN TABLE like this:

DROP SERVER IF EXISTS matrikel_jordstykke CASCADE;
CREATE SERVER matrikel_jordstykke
   FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
  datasource '/vsizip//vsicurl/ftp://testuser_ogr_fdw:[email protected]/matrikeldata/matrikelkort/SHAPE/1084_SHAPE_UTM32-EUREF89.zip/1084_SHAPE_UTM32-EUREF89/MINIMAKS/BASIS/JORDSTYKKE.shp',
  format 'ESRI Shapefile');

DROP FOREIGN TABLE IF EXISTS matrikelkort.jordstykke;
CREATE FOREIGN TABLE matrikelkort.jordstykke (
  fid integer,
  geom geometry(POLYGON, 25832),
  uuid character varying,
  feat_id real,
  feat_kode real ,
  feat_type character varying,
  elavsnavn character varying,
  elavskode double precision,
  matrnr character varying,
  kms_sagsid real,
  kms_journr character varying,
  skelsagsid real,
  supmsagsid real,
  komnavn character varying,
  komkode integer,
  sognnavn character varying,
  sognkode integer,
  regionnavn character varying,
  regionkode integer,
  retskrnavn character varying,
  retskrkode integer,
  moderjord real,
  regareal real,
  arealbereg character varying,
  vejareal real,
  vejarealbe character varying,
  vandarealb character varying,
  faelleslod character varying,
  esr_ejdnr real,
  sfe_ejdnr real,
  sfe_sagsid real,
  sfe_dato date,
  sfe_journr character varying,
  sfe_note character varying,
  land_note character varying,
  arealtype character varying,
  dq_index real,
  registdato date,
  geomdato date,
  publidato date)
  SERVER matrikel_jordstykke
  OPTIONS ( layer 'JORDSTYKKE' );

Now, I make the same query on the FOREIGN TABLE:

SELECT elavsnavn, 
       elavskode, 
       matrnr 
FROM   matrikelkort.jordstykke 
WHERE  elavskode = 2000154 
LIMIT  1; 

Returns

Which is not so good, because the data is missing for the column elavsnavn.
I was hoping to get around this issue really quick by changing the client encoding like i I did when loading the data with regular ogr2ogr and rerun the query:

SET CLIENT_ENCODING TO 'latin1';

Which returns an error:

ERROR: invalid byte sequence for encoding "UTF8": 0xf8

I am not exactly sure what is happening anymore.

My questions are:

1. What is the lifecycle of encoding/recoding using ogr_fdw ?
2. Is it possible to pass in GDAL configuration options to the FOREIGN SERVER? I would like to experiment with disabling recoding on the ogr2ogr side with either: setting
   SHAPE_ENCODING to an empty string "" OR using the -oo (open option) ENCODING ="" in the shapefile
   driver to avoid any recoding

Hi,
I couldn't make attribute filter work here, could you provide an example for this in readme file?
Cheers
Régis

It should be possible to notice an && operator in the deparse stage and find a spatial literal argument (???) to emplace in the OGR layer. There may be some difficulties though:

• If the spatial filter is wrapped up in functions, it might not show up as a const node in the plan, eg: ST_SetSRID(ST_GeomFromText('')) could emit a literal in the end, but not show as a literal in the original planning phase when we generate the OGR filters

They broke your code again :)

ogr_fdw.c: In function 'ogr_fdw_handler':
ogr_fdw.c:170:29: warning: assignment from incompatible pointer type [enabled by default]
  fdwroutine->GetForeignPlan = ogrGetForeignPlan;
                             ^
ogr_fdw.c: In function 'ogr_fdw_validator':
ogr_fdw.c:317:30: warning: variable 'layer' set but not used [-Wunused-but-set-variable]
  const char *source = NULL, *layer = NULL, *driver = NULL;
                              ^
ogr_fdw.c: In function 'ogrGetForeignPaths':
ogr_fdw.c:520:6: warning: passing argument 8 of 'create_foreignscan_path' from incompatible pointer type [enabled by default]
      NIL));   /* no fdw_private data */
      ^
In file included from ogr_fdw.h:41:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/pathnode.h:82:21: note: expected 'struct Path *' but argument is of type 'struct List *'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^
ogr_fdw.c:520:6: error: too few arguments to function 'create_foreignscan_path'
      NIL));   /* no fdw_private data */
      ^
In file included from ogr_fdw.h:41:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/pathnode.h:82:21: note: declared here
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^
ogr_fdw.c: In function 'ogrGetForeignPlan':
ogr_fdw.c:602:1: error: too few arguments to function 'make_foreignscan'
 );
 ^
In file included from ogr_fdw.h:42:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/planmain.h:46:21: note: declared here
 extern ForeignScan *make_foreignscan(List *qptlist, List *qpqual,
                     ^
ogr_fdw.c:605:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
<builtin>: recipe for target `ogr_fdw.o' failed
make: *** [ogr_fdw.o] Error 1
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2  -I/projects/gdal/rel-2.0.1w64gcc48/inclu
ogr_fdw.c: In function 'ogr_fdw_handler':
ogr_fdw.c:170:29: warning: assignment from incompatible pointer type [enabled by default]
  fdwroutine->GetForeignPlan = ogrGetForeignPlan;
                             ^
ogr_fdw.c: In function 'ogr_fdw_validator':
ogr_fdw.c:317:30: warning: variable 'layer' set but not used [-Wunused-but-set-variable]
  const char *source = NULL, *layer = NULL, *driver = NULL;
                              ^
ogr_fdw.c: In function 'ogrGetForeignPaths':
ogr_fdw.c:520:6: warning: passing argument 8 of 'create_foreignscan_path' from incompatible pointer type [enabled by default]
      NIL));   /* no fdw_private data */
      ^
In file included from ogr_fdw.h:41:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/pathnode.h:82:21: note: expected 'struct Path *' but argument is of type 'struct List *'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^
ogr_fdw.c:520:6: error: too few arguments to function 'create_foreignscan_path'
      NIL));   /* no fdw_private data */
      ^
In file included from ogr_fdw.h:41:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/pathnode.h:82:21: note: declared here
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^
ogr_fdw.c: In function 'ogrGetForeignPlan':
ogr_fdw.c:602:1: error: too few arguments to function 'make_foreignscan'
 );
 ^
In file included from ogr_fdw.h:42:0,
                 from ogr_fdw.c:29:
c:/MING64~1/projects/POSTGR~1/rel/PG9~2.5W6/include/server/optimizer/planmain.h:46:21: note: declared here
 extern ForeignScan *make_foreignscan(List *qptlist, List *qpqual,
                     ^
ogr_fdw.c:605:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^

I noticed the issue after I upgraded to PostgreSQL 9.5rc1 from PostgreSQL 9.5beta2 and my queries with ogrfdw just crashed the backend.

I haven't investigated a fix for this yet.

The extension builds fine, but the helper meta data generator doesn't compile under my mingw-w64 win32 gcc 4.8.0.

Gives error:

ogr_fdw_info.c: In function 'main':
ogr_fdw_info.c:39:9: warning: variable 'bflag' set but not used [-Wunused-but-set-variabl
     int bflag, ch;
         ^
ogr_fdw_info.c: In function 'strlaunder':
ogr_fdw_info.c:121:2: error: 'for' loop initial declarations are only allowed in C99 mode
  for(int i = 0; str[i]; i++)
  ^
ogr_fdw_info.c:121:2: note: use option -std=c99 or -std=gnu99 to compile your code
<builtin>: recipe for target `ogr_fdw_info.o' failed
make: *** [ogr_fdw_info.o] Error 1

I'm not sure if this is limited to just excel and open office workbooks. I suspect it's not but those are the only ones I commonly edit. Anyway what happens is this:

If I run a query against a Foreign table of a worksheet, then edit the file in Excel or whatever, and then try to rerun the query in the same connection as the original query, I get a weird failure like.

ERROR:  unable to connect to data source "//S/E$/Survey150513.xlsx"
HINT:  GotoFileOffset failed
********** Error **********

ERROR: unable to connect to data source "//S/E$/Survey150513.xlsx"
SQL state: HV00D
Hint: GotoFileOffset failed

To work around the issue, I have to start a new postgres connection.

Now that you support not bringing in all columns, it would be great if we had a feature to exclude certain named columns. For example FID for many data sources such as spreadsheets and ODBC sources is always included, and it's a made up thing that wasn't even in my original data. I'd like to get rid of it since it sometimes doesn't even play well with querying like fid >2 returns nothing.

The format would be something like below and this is a real use case I have in mind. ESRI stuffs in these useless area and length columns I don't need cause I can always recompute if I really wanted them.

CREATE SERVER svr_shps
   FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (datasource '/data',format 'ESRI Shapefile'); 

IMPORT FOREIGN SCHEMA ogr_all FROM SERVER svr_shps INTO staging
 OPTIONS(exclude_columns 'fid, shape_area, shape_leng');

If it's not too much trouble, can we have a --formats or -f or whatever you want to call it option to ogr_fdw_info similar to what ogrinfo has?