cbick / gtfs_sql_importer Goto Github PK

Using the current BART gtfs (http://bart.gov/dev/schedules/google_transit.zip), it happens to have two trailing white space lines in the stops table and the transfers table which causes script to stop uploading tables beyond there. When I deleted the white space it fixed the problem. I don't think most feeds have these trailing spaces, but maybe the program should end at the last full line.

These errors go away when the whitespace is deleted and the program is re-run...

BEGIN
ERROR:  missing data for column "stop_name"
CONTEXT:  COPY gtfs_stops, line 48: ""
ERROR:  current transaction is aborted, commands ignored until end of transaction block
invalid command \.
ERROR:  syntax error at or near "AirBART"
LINE 1: AirBART|AirBART|NULL|SHUTTLE|NULL|3|http://www.bart.gov/guid...
        ^
invalid command \.
ERROR:  syntax error at or near "WKDY"
LINE 1: WKDY|1|1|1|1|1|0|0|20120701|20140101
        ^
invalid command \.
ERROR:  syntax error at or near "SAT"
LINE 1: SAT|20120116|1
        ^
...

gtfs_SQL_importer fails on importing utf-8 with BOM files.
The GTFS specification states utf-8 BOM is ok.
Files coming from a Microsoft environment are likely to have UTF8 w/ BOM.

Workaround: add a command to clean the BOM from the gtfs files before running the script.
LANG=C LC_ALL=C sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- path/to/gtfs_files/*
(from https://unix.stackexchange.com/questions/381230/how-can-i-remove-the-bom-from-a-utf-8-file )

Able to set up tables but fails to populate when it hits errors with agency_id:
ERROR: column "agency_id" of relation "gtfs_agency" does not exist

Tried with different feeds with and without an agency_id and some work while others dont. SEPTA has no agency_id and worked, MTA Maryland had a 1 and worked. Arlington had a 1 and failed. TriMet had nothing and failed.

GTFS feeds are running into errors about stop_id even when the stop_id is coded properly in the files.

No errors listed earlier in this log about gtfs_stops table.

ERROR:  column "stop_id" of relation "gtfs_stops" does not exist
invalid command \.
ERROR:  syntax error at or near "83"
LINE 1: 83|51001|Ballston Metro, Fairfax Dr, EB @ N Stafford St, NS|...
        ^
invalid command \.
ERROR:  syntax error at or near "1"
LINE 1: 1_41|1|41|Columbia Pike-Ballston-Court House|3|http://realti...
        ^
invalid command \.
ERROR:  syntax error at or near "1"
LINE 1: 1|0|0|0|0|0|1|0|20120423|20131231
        ^
invalid command \.
ERROR:  syntax error at or near "1"
LINE 1: 1|20121008|1
        ^
invalid command \.
ERROR:  current transaction is aborted, commands ignored until end of transaction block
ERROR:  current transaction is aborted, commands ignored until end of transaction block

Hi,
I was wondering how much change would it require from this code to import it to SQLite instead of SQL. (I don't know much about DBMS structures)

Thanks