cbick / gtfs_sql_importer Goto Github PK
View Code? Open in Web Editor NEWImport GTFS data to SQL
Home Page: http://cbick.github.com/gtfs_SQL_importer/
License: MIT License
Import GTFS data to SQL
Home Page: http://cbick.github.com/gtfs_SQL_importer/
License: MIT License
gtfs_SQL_importer fails on importing utf-8 with BOM files.
The GTFS specification states utf-8 BOM is ok.
Files coming from a Microsoft environment are likely to have UTF8 w/ BOM.
Workaround: add a command to clean the BOM from the gtfs files before running the script.
LANG=C LC_ALL=C sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- path/to/gtfs_files/*
(from https://unix.stackexchange.com/questions/381230/how-can-i-remove-the-bom-from-a-utf-8-file )
GTFS feeds are running into errors about stop_id even when the stop_id is coded properly in the files.
No errors listed earlier in this log about gtfs_stops table.
ERROR: column "stop_id" of relation "gtfs_stops" does not exist
invalid command \.
ERROR: syntax error at or near "83"
LINE 1: 83|51001|Ballston Metro, Fairfax Dr, EB @ N Stafford St, NS|...
^
invalid command \.
ERROR: syntax error at or near "1"
LINE 1: 1_41|1|41|Columbia Pike-Ballston-Court House|3|http://realti...
^
invalid command \.
ERROR: syntax error at or near "1"
LINE 1: 1|0|0|0|0|0|1|0|20120423|20131231
^
invalid command \.
ERROR: syntax error at or near "1"
LINE 1: 1|20121008|1
^
invalid command \.
ERROR: current transaction is aborted, commands ignored until end of transaction block
ERROR: current transaction is aborted, commands ignored until end of transaction block
Hi,
I was wondering how much change would it require from this code to import it to SQLite instead of SQL. (I don't know much about DBMS structures)
Thanks
Using the current BART gtfs (http://bart.gov/dev/schedules/google_transit.zip), it happens to have two trailing white space lines in the stops table and the transfers table which causes script to stop uploading tables beyond there. When I deleted the white space it fixed the problem. I don't think most feeds have these trailing spaces, but maybe the program should end at the last full line.
These errors go away when the whitespace is deleted and the program is re-run...
BEGIN
ERROR: missing data for column "stop_name"
CONTEXT: COPY gtfs_stops, line 48: ""
ERROR: current transaction is aborted, commands ignored until end of transaction block
invalid command \.
ERROR: syntax error at or near "AirBART"
LINE 1: AirBART|AirBART|NULL|SHUTTLE|NULL|3|http://www.bart.gov/guid...
^
invalid command \.
ERROR: syntax error at or near "WKDY"
LINE 1: WKDY|1|1|1|1|1|0|0|20120701|20140101
^
invalid command \.
ERROR: syntax error at or near "SAT"
LINE 1: SAT|20120116|1
^
...
Able to set up tables but fails to populate when it hits errors with agency_id:
ERROR: column "agency_id" of relation "gtfs_agency" does not exist
Tried with different feeds with and without an agency_id and some work while others dont. SEPTA has no agency_id and worked, MTA Maryland had a 1 and worked. Arlington had a 1 and failed. TriMet had nothing and failed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.