opengeospatial / geopackage

An asciidoc version of the GeoPackage specification for easier collaboration

GeoPackage Standard

==========

GeoPackage is an open, standards-based, platform-independent, portable, self-describing, compact format for transferring geospatial information. The GeoPackage standard describes a set of conventions for storing the following within an SQLite database:

• vector features
• tile matrix sets of imagery and raster maps at various scales
• attributes (non-spatial data)
• extensions

These capabilities are built on a common base and an Extension Mechanism is described to provide implementors a way to include additional functionality in their GeoPackages.

This OGC® Encoding Standard defines the schema for a GeoPackage, including table definitions, integrity assertions, format limitations, and content constraints. The allowable content of a GeoPackage is entirely defined in this standard.

For more information about GeoPackage, including implementations and sample data, go to the public page at http://www.geopackage.org. An HTML version of the standard is available at http://www.geopackage.org/spec/. The asciidoc source for the standard is in the spec/core/ folder.

About

This GitHub repository was originally extracted from the Microsoft Word version of the Candidate GeoPackage Standard version 0.8 released for public comment on August 6, 2013. With this repository the OGC invites collaboration and comments directed at the development and enhancement of this candidate standard.

The repo tracks the latest version of the standard as it evolves. Pull requests for fixes are appreciated, and new functionality will still be considered even though version 1.0 has been adopted. The spec is done in asciidoc a format supported by GitHub, similar to markdown but with some features that make it better for specifications, like automatic section numbering.

Editor: Jeff Yutzler

Editor Emeritus: Paul Daisey

Contributing

The contributor understands that any contributions, if accepted by the OGC Membership, shall be incorporated into the formal OGC GeoPackage standards document and that all copyright and intellectual property shall be vested to the OGC.

The GeoPackage Standards Working Group (SWG) is the group at OGC responsible for the stewardship of the standard, but is working to do as much GeoPackage work in public as possible.

The Geopackage SWG currently has the following email lists:

• [email protected] -- sign up here
  • Public List, very little traffic.
  • Open archives
  • No Intellectual Property items should be discussed here.
• [email protected] -- sign up here
  • Private List
  • Access controlled archives
  • Requires OGC membership and Observer Agreement to protect IPR (intellectual property rights)

SWG Chair: Jeff Yutzler

SWG Vice Chair: Tracey Birch

Editing and commenting

The GeoPackage SWG is accepting public comments and suggested revisions to the standard via GitHub. This is the first time OGC has supported this mechanism for public comment and review. To suggest changes to the standard please fork the repository and submit a pull request with the desired changes. Please make one pull request per logical requested change and be sure to include a comment in the PR with any justification or reasoning on why the change is needed.

For more general comments (that don't include actual text changes to the spec) please create a GitHub issue for that topic.

Complex changes and feature requests must go through the change request process. The details entered in the change request form will help the SWG adjudicate and prioritize the request.

For more detailed guidance, or if you are new to GitHub, see the Process page for additional information on editing.

geopackage's Issues

/base/core/contents/data/table_def test case has wrong table ref

 Pass if the column names and column definitions in the returned CREATE TABLE statement, including data type, nullability, default values and primary, foreign and unique key constraints match all of those in the contents of C.2Table 18. Column order, check constraint and trigger definitions, and other column definitions in the returned sql are irrelevant.

However C.2 does not contain Table 18. That reference should probably be Table 21. You could add a space before Table as well.

Also note that Table numbers do not appear in http://opengis.github.io/geopackage/#table_definition_sql version.

PRAGMA foreign_key_check();

On my SQLite 3.8.0.2, running this query per the file integrity section (1.1.1) of the document returns a syntax error, not an empty result set.

Typo in /opt/features/geometry_columns/data/data_values_geometry_type_name

Verrify should be Verify

Wrong test in /opt/features/geometry_columns/data/data_values_z and /opt/features/geometry_columns/data/data_values_m

SELECT z FROM gpkg_geometry_columns WHERE z NOT IN (1,2,3)

should be

SELECT z FROM gpkg_geometry_columns WHERE z NOT IN (0,1,2)

Same for m

Spec overview

We should have some sort of overview / 'guide' to the spec. This could be a new document in the spec folder, or maybe just a richer readme.

Need for dual endianness in geometry BLOB format ?

Relates to 2.1.3.1.1. BLOB Format

What is the point of having a bit for endianness ? It would be far more simple to impose that the multi-bytes values in the header are in litte-endian (all nowadays popular platforms Intel & (most) ARM are little-endian)

The benefit is :

1. applications that do not care of portability and must run only on little-endian hosts have nothing to do (not a good reason)
2. applications that do care about portability on little endian and big endian hosts can deal with byte-swapping at compile-time. No need for run-time check. Makes code smaller and faster.

While we are it, let impose that the WKB encoding itself uses little-endian order.

Move intro paragraphs from Annex B to front

The introductory background material moved to Annex B as a side effect of moving other informative material out of the front. I think it needs to move back to a preface, or to 1 Base.

Axis ordering

I see no statement in the spec about which axis ordering the coordinates in a geometry blob should respect. Concretely, for a vector table that has a SRID of EPSG:4326, should the x be the longitude and the y the latitude as the usual GIS practice, or should it follow the EPSG axis ordering that is latitude, longitude. §1.1.2.1.2 points to http://www.epsg-registry.org/report.htm?type=selection&entity=urn:ogc:def:crs:EPSG::4326&reportDetail=long&title=WGS%2084&style=urn:uuid:report-style:default-with-code&style_name=OGP%20Default%20With%20Code that advertizes the latitude, longitude ordering.

The same applies for all mentions to x and y in gpkg_contents, gpkg_tile_matrix_set, etc

/base/core/spatial_ref_sys/data_values_default not traceable to requirement

Further to #38 (closed, but not fixed), the /base/core/spatial_ref_sys/data_values_default test requires that the srs_id (nee srid) and organization_coordsys_id (nee auth_srid) are both 4326, but the need for these to match is not traceable to the requirement (which doesn't specify any constraint on what the srs_id should be - it only says "Req 11: The gpkg_spatial_ref_sys table in a GeoPackage SHALL contain a record for
organization EPSG or epsg [B3] and organization_coordsys_id 4326 [13][14] for WGS-84..."

In addition, the specific WKT for SRS 4326 is not traceable to the REQ 11 (it may become traceable via the links to EPSG web sites, but would need at least a version number for the data set).

Table 41 constraints use old column name

Table 41 has example tile pyramid triggers. None of those triggers work, because they all use "t_table_name" in the WHERE clause, but t_table_name doesn't exist as a column in gpkg_table_matrix.

The column name should probably be "table_name"

Add annexes for real

Convert all annexes to markdown

add contributing organizations

From Jeff Harrison:

'I think the GeoPackage GitHub needs a list of Contributing organizations, like the regular spec. I looked bit couldn't see this anywhere...'

Need detailed description of spatial index

If I'm not wrong, the spatial index is defined as "use SQLite rtree extension", which is not specific for a standard - what version/revision exactly, where is description of the rtree format etc, especially as rtree is not standard part of SQLite, and it is not included in many deployments: most notably it is not in Android and iOS builds so far. Here should be binary-level description of it, so even after 20 years this specification could be used to decode GeoPackage files. It is already now hard to find SQLite rtree implementation code, and rtree.c seems to be the only place where binary description could be deducted.

More hyperlinks throughout

Since we're publishing for the web it'd be good to have it use the web well, with real hyperlinks throughout. One of the places that would be best done is the normative spec references, and bibliography stuff. These footnote in to mostly hyperlinks. They should instead link straight out.

The other thing to link more is the 'clauses'. The spec refers to different clauses pretty often, would be good for people to click right there, instead of having to scroll to find it themselves. Tables similarly would be good to go to directly, but not sure how easy that is with github.

/reg_ext/tiles/tile_encoding_nitf/data/nitf_ext_name refers to webps

Not testable if no logged webps should be Not testable if no logged nitfs

Table 38 contraints inconsistent with Table 15 scopes

Table 15 says "catalog" but Table 38 uses "catalogue".

Presumably Table 15 takes precedence, since Table 38 is only informative.

If updating Table 38, the abort message could be modified to be "... | taxonomy | software | ..." instead of "... | taxonomy software | ..."

Typo in REQ 83

 Req 83: The definition column value in a gpkg_exteinsons row SHALL contain or reference the text that results from documenting an extension by filling out the GeoPackage Extension Template in Annex I.

The table name should be gpkg_extensions

/opt/tiles/tiles_encoding/data/mime_type_png refers to (deleted) gpkg_tiff, gpkg_nitf

The test contains:

(SELECT table_name FROM gpkg_contents WHERE data_type = 'tiles' AND
table_name NOT IN (SELECT table_name from gpkg_extensions WHERE extension_name IN ('gpkg_webp’,’gpkg_tiff’,’gpkg_nitf’)))

However gpkg_tiff and gpkg_nitf are not documented.

This also occurs in /opt/tiles/tiles_encoding/data/mime_type_jpeg

Add annexes

We need to add all the annexes. My suggestion is to do them under geopackage/spec/annexes/ So that you see the folder of the extras from the spec page, but they don't fill up things there.

They'll need to be formatted, hopefully we can get a number of hands on it to make it light work.

SQL definition in Table 27 does not reflect defaults of Table 9

Table 9 specifies default values for zoom_level, matrix_width, matrix_height, tile_width, tile_height, pixel_x_size and pixel_y_size.

These defaults are not included in Table 27.

Which is correct?

Table 26 (gpkg_tile_matrix_set) has missing comma

CREATE TABLE gpkg_tile_matrix_set (
  table_name TEXT NOT NULL PRIMARY KEY,
  srs_id INTEGER NOT NULL
  min_x DOUBLE NOT NULL,
  min_y DOUBLE NOT NULL,
  max_x DOUBLE NOT NULL,
  max_y DOUBLE NOT NULL,
  CONSTRAINT fk_gtms_table_name FOREIGN KEY (table_name) REFERENCES gpkg_contents(table_name),
  CONSTRAINT fk_gtms_srs FOREIGN KEY (srs_id) REFERENCES gpkg_spatial_ref_sys (srs_id)
);

is not legal SQL. There needs to be a comma at the end of the srs_id line

What are the georeferenced coordinates of the origin of a tile matrix set ?

"§2.2.7.1.2. Table Data Values" mentions "GeoPackages follow the most frequently used conventions of a tile origin at the upper left and a zoom-out-level of 0 for the smallest map scale “whole world” zoom level view, as specified by WMTS [16]. The tile coordinate (0,0) always refers to the tile in the upper left corner of the tile matrix at any zoom level, regardless of the actual availability of that tile."

But that doesn't really tell which georeferenced coordinate correspond to the upper-left corner of tile (0,0). In WMTS, such information is explicitely specified as in a element.

In practice, this is often for a SRID = 4326, this must be (-180,90), and for SRID = 3857, this must be (-20037500,20037500) or (-20037508.34 20037508.34).

Perhaps this information could be computed from min_x, max_y in the gpkg_tile_matrix_set table and information from gpkg_tile_matrix, but that would likely require a full scan of the user table to find the top-left most tile.

I feel like there's a lack of specification somewhere...

Another closely related issue : do all the tile level of a tile pyramid have the same georeferenced coordinates for their tile (0,0) ? In WMTS, this is not necessarily the case ( being a property of and not ), which is an implementation annoyance.

Test case /base/core/spatial_ref_sys/data/table_def not implementable

"A.1.1.2.1.1 Table Definition
Test Case ID: /base/core/spatial_ref_sys/data/table_def
Test Purpose: Verify that the spatial_ref_sys table exists and has the correct definition.
Test Method:
1) SELECT sql FROM sqlite_master WHERE type = 'table' AND tbl_name =
'gpkg_spatial_ref_sys'
2) Fail if returns an empty result set
3) Pass if column names and column definitions in the returned CREATE TABLE
statement in the sql column value, including data type, nullability, default values and
primary, foreign and unique key constraints match all of those in the contents of C.1
Table 19. Column order, check constraint and trigger definitions, and other column
definitions in the returned sql are irrelevant.
4) Fail otherwise.
Reference: Clause 1.1.2.1.1 Req 6:
Test Type: Basic"

That C.1 Table 19 reference does not include nullability, default values, foreign key or unique key constraints. This information is also not included in Table 2.

Further, this test requires that this be implemented as a table, however Req 6 allows this to be implemented as an updateable view.

Also, the test purpose should read gpkg_spatial_ref_sys, not spatial_ref_sys.

Suggest changing the pass condition to match the requirements of Req 6, Table 2 and Table 19.

Section 2.4 - non-GPKG columns

(this should be labeled under metadata, however I am unable to assign labels).

Can non-GPKP columns be part of gpkg_metadata (I suppose this applies to most tables in GPKG). Imagine a use case where one parses an ISO XML document values into columns. Can the gpkg_metadata table have additional columns or does this break the spec?

Better 'process' page

Before public release we should have the 'process' sorted for how we tell people to contribute comments and suggested changes. Can put this on the process.md page.

REQ 30 inconsistent with REQ 22

The design of gpkg_geometry_columns is inconsistent (or perhaps just redundant) with the feature data model described in REQ 30.

If there is only one geometry column, then the REQ 22 text "one row record for each geometry column in each vector feature user data table " doesn't make sense. Similarly, the gpkg_geometry_columns primary key being across (table_name, column_name) is not required because table_name would be sufficient.

More generally, this restriction is unfortunate in that it prevents "fall-back" implementations for extended (non-linear) geometry.

Unclear relation between zoom_level and pixel_x_size, pixel_y_size

AFAIHU, there's no explicit relation stated between zoom_level and pixel_x_size, pixel_y_size.

Req 29 mentions "zoom level pixel sizes for that table SHALL vary by powers of 2 between adjacent zoom levels in the tile matrix metadata table", but it is unclear what "adjacent" means.

To tell the truth, at a moment, I thought the values of zoom_level were power-of-two, but I finally understanded that they are just a normal integer sequence like 0,1,2,... for classical pyramids.

Req 42 could be formulatted more explicitely : pixel_x_size * pow(2, zoom_level) and pixel_y_size * pow(2, zoom_level) must be constant for all rows of gpkg_tile_matrix_metadata that refer to the same t_table_name value.

update to 0.8 version of spec

I think I used the last version of the 0.7 series. Should check the changes that were made for 0.8.

@rajrsingh any chance you can do this? I'm less good with word, and I don't actually have word (libre office instead, which sometimes doesn't work as well).

Test case /base/core/spatial_ref_sys/data_values_default is bogus

The subject test has the wrong table name and wrong column headings.

Table 10 md_standard_uri

(this should be labeled under metadata, however I am unable to assign labels).

The default value for md_standard_uri is shown as http://schemas.opengis.net/iso/19139/. Should this be http://www.isotc211.org/2005/gmd as per the namespace URI for ISO metadata? Are we trying to communicate a webpage for the metadata type, or something tighter?

Is it assumed that all metadata is XML-based (which would make mime_type somewhat redundant)?

Some elaboration on this column would be valuable. Something like "if XML, then the value is the XML namespace URI of the metadata"

Conformance tests mention "GPB"

A.2.1.2.1.1 BLOB Format
i. Fail if the first three bytes of each gc are not “GPB”

--> should be " the first two bytes of each gc are not “GP” "

And other "GPB" references afterwards in the document

Heading A.1.1.3 corrupted

Looks like the "Test Type:" part from the previous clause got merged into the header

Annex N typo in title

Trigers should be Triggers

/opt/extension_mechanism/extensions/api/api_geopackage_sqlite_config has incorrect test

SELECT sqlite_compileoption_used('SQLITE_ENABLE_RTREE_') will always return 0, but that means the test will always fail.

Perhaps you mean SELECT sqlite_compileoption_used('SQLITE_ENABLE_RTREE') which would match Table 15.

Adapt preamble to this draft

@rajrsingh - could you take the lead on making sure we have all the OGC stuff right? Like we got this preamble, which should be incorporated and probably updated somewhere. Perhaps directly in the readme?

The following GitHub document has been extracted from the OGC GeoPackage candidate standard current working draft, and is a revision of the earlier (January) draft released for public comment. This is an open document and the OGC invites collaboration and comments directed at the development and enhancement of this section of the GeoPackage candidate standard. Please see http://www.opengeospatial.org/node/1756 for more information including instructions on how to provide comments on the draft specification to OGC. The contributor understands that any contributions, if accepted by the OGC Membership, shall be incorporated into the formal OGC GeoPackage standards document and that all copyright and intellectual property shall be vested to the OGC.

Add logos to README and elsewhere

For public release we need to get appropriate OGC logos. The most obvious (and prominent) place is in the readme.

The logo files can go in the images folder created in #2

About "A feature table SHALL have only one geometry column"

Let admit this restriction (although it is not really in the "trend". For example OGR will soon support multiple geometry columns : http://trac.osgeo.org/gdal/wiki/rfc41_multiple_geometry_fields )

The gpkg_geometry_columns Table Definition SQL in Table 24 seems to be less restrictive :
CONSTRAINT pk_geom_cols PRIMARY KEY (table_name, column_name)

If you want to enforce a single geometry column per table, the primary key should only be on the table_name

Is it wanted as a provision for a future extension of the spec to support multiple geometry columns per feature table ?

§2.3.3.1.2. Table Data Values : Enumerated values

It is not clear to me what the following sentence means :
"""The case sensitive value column contains an enumerated legal value for constraint_type "enum" """

What is concretely "an enumerated legal value" ? A single value ? But that doesn't really have any interest. I would rather imagine it as being the list of all possible enumerated values possible for the colum. Are they comma separated ?

An example would be welcome.

/reg_ext/tiles/tile_encoding_tiff/data/tiff_ext_name refers to webp

Not testable if no logged webps should be Not testable if no logged tiffs

Conflict between a GeoPackage SQLite Extension and libspatialite

The fact that GeoPackage has dropped the Spatialite geometry blob will likely cause an integration nightmare for an application (e.g. GDAL/OGR) that intends supporting both GeoPackage and spatialite databases.

The reason is that both libspatialite and the GeoPackage SQLite exension (e.g. the Luciad implementation) will try to register a ST_MinX() (and other similar ST_ functions) SQLite function, but depending on the order the sqlite extensions are loaded, one will win over the other one, and will only be able to process the format of geometry blob it supports. So only spatialite or exclusively only geopackage could be handled.

How to workaround that ? Several ideas :

• The easier solution : use the spatialite binary blob, so that any of the implementation can be used. Although I'm not sure it will always work. For example ST_Tranform() needs to query the spatial_ref_sys table to do the reprojection. The spatialite implementation requires the proj4text column in spatial_ref_sys for that, but if it operates on a GeoPackage database this column will likely not exist, causing ST_Transform() to fail.
• or make both concurrent implementation detect if the database is well the relevant one before registering their functions, i.e. a GeoPackage SQLite function would test the presence of a spatialite table (e.g. spatial_ref_sys with a proj4text column) and would avoid loading its function, and similarly libspatialite would test the presence of gpkg_ table
• or decorate GeoPackage SQL function with a "gpkg_" prefix

All the above is a bit speculative, since I did not actually try, but I do think the issue is plausible.

Tile matrix definition refers to missing (deleted?) clauses

Table 10 defines the tile_data column as:

Of an image MIME type specified in clauses 2.2.4, 2.2.5, 3.2.2, 3.2.3, 3.2.4

However there is no clause 3.2.3 or 3.2.4.

Add images to spec

There are a few images to add to the spec, like the big UML diagram, and the geometry classes stuff. These can go directly in github. I'd recommend an images folder, on geopackage/spec/images (could also go under just geopackage/images), where these go, and then from markdown can just refer to them.

Unclear description for f_table_name of gpkg_tile_matrix_metadata table

The description for the f_table_name column of gpkg_tile_matrix_metadata table (2.2.6.1.1. Table Definition) says : "{RasterLayerName}_tiles"

This is unclear to me. Is it the name of a "Tile Matrix User Data" table ? Is the "_tiles" suffix a requirement for such a table name ?

Table 10 md_scope

(this should be labeled under metadata, however I am unable to assign labels).

Must the scope name be case-sensitive? I'm guessing yes, however if wouldn't hurt to make this more explicitly stated?

/base/core/spatial_ref_sys/data_values_required refers to wrong table in purpose

/base/core/spatial_ref_sys/data_values_required test case refers to spatial_ref_sys instead of gpkg_spatial_ref_sys.

superset of current standard

it would be nice if geopackage could be a superset of the current defacto standard for sqlite simple features which is already supported by qgis, gdal, tilemill etc. browsing the spec quickly it looks like the main incompatibilities (besides, obviously, new features) are things like table names.

The standard could be written such that geopackage level 1 is the current sqlite implementation, (possibly with some details filled in if you can't help yourself) while geopackage level 2 would be the closer to what is proposed here adding raster day and indices on top of the current thing.

This would make it significantly easier to get it adopted by the community as instead of coming out with a 3rd (or 4th depending on when arcmap 10.2 ships) sqlite based format, it's instead just formalizing what's already in use and taking it to the next level.

Annex C has mixed informative / normative content

Annex C claims to be normative, but some of the content is purely informative (e.g. the SF/SQL and SQL/MM views). It would be easier to demonstrate compliance if the informative material was removed or relocated.

Format issues and PR's for Editor

Issues and pull requests should be formatted in a way that the GeoPackage spec editor can easily process.

Section A.3 is missing

The Abstract Test Suite section on the extensions (A.3) is missing from the asciidoc version of the spec.

Test Case ID: /base/core/spatial_ref_sys/data_values_required cannot fail

Test Case ID: /base/core/spatial_ref_sys/data_values_required cannot fail, because of combination of the FK constraint on gpkg_contents and NOT NULL constraints on gpkg_spatial_ref_sys. That probably means that REQ 12 should be removed (or reworded as a constraint).