tulsawebdevs / django-multi-gtfs Goto Github PK
View Code? Open in Web Editor NEWDjango app to import and export General Transit Feed Specification (GTFS)
Home Page: http://tulsawebdevs.org/
License: Apache License 2.0
django-multi-gtfs's People
Contributors
Stargazers
Watchers
Forkers
cheekybastard kantorv ricardosasilva nicopace brendancol jtszalay erickmendonca alenfant probabble biznixcn juyrjola miraculixx edusegzy powersurge360 vpistis francot vsidou code4maine jwhitlock jclgoodwin alaw005 polyglotopenstreetmap bileto sipp11 aherneyeldarb nirgilboa hjtamaral raekkeri nnsevadjango-multi-gtfs's Issues
Maybe remove the 'Inbound-Outbound' labels
I've found the choices=(('0', 'Outbound'), ('1', 'Inbound')) argument of the direction attribute of the Trip model (
django-multi-gtfs/multigtfs/models/trip.py
Line 123 in 0c9d9ae
More specifically, in some feeds, such as good old SEQ, 0 does not always mean outbound, and 1 does not always mean inbound. In that feed, some direction 0 routes are inbound and some are neither inbound nor outbound, being cross-town routes. And that's not breaking the GTFS rules as far as i can tell, because 0 = Outbound and 1 = Inbound are only example interpretations in the spec (see https://developers.google.com/transit/gtfs/reference#trips_fields).
With that in mind, i vote for removing the 'Inbound-Outbound' labels associated with directions 1 and 0.
Can't export a non-ascii feed in Python 2
When exporting a stop with a bad description (utf-8 "The Delta Caf�\xc2\x82"), export fails:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/Users/john/src/django-multi-gtfs/multigtfs/management/commands/exportgtfs.py", line 80, in handle
feed.export_gtfs(out_name)
File "/Users/john/src/django-multi-gtfs/multigtfs/models/feed.py", line 158, in export_gtfs
klass._filename, content.count('\n') - 1,
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 72255: ordinal not in range(128)
Under Python 2, content is a UTF-8 encoded byte string, and '\n' is a unicode string. content.count('\n') attempts to decode it as an ascii string first. Under Python 3, both content and '\n' are unicode strings.
Convert to GeoDjango
django-multi-gtfs is using decimal fields for lat/long. We did this because GeoDjango has additional database requirements. However, GeoDjango is the right tool for the job.
Possible Steps:
- Add GeoDjango to requirements
- Migrate decimal fields to the correct spatial fields (one or more South migrations)
- Update installation docs w/ references to external docs
Migration error
I switched to GeoDjango, upgraded multigtfs via pip on my local machine, ran the multigtfs database migrations, and all is well. Then i tried to do the same thing on my remote machine but with a fresh PostGIS database and ran into errors.
More specifically, i got the following error. Any ideas about what i'm doing wrong?
Thanks.
$ python manage.py migrate multigtfs
Running migrations for multigtfs:
- Migrating forwards to 0010_initial.
> multigtfs:0001_initial
> multigtfs:0002_auto__chg_field_fare_transfers
> multigtfs:0003_convert_fare_transfers
> multigtfs:0004_auto_add_geopoint_fields
> multigtfs:0004_initial
FATAL ERROR - The following SQL query failed: CREATE TABLE "agency" ("id" serial NOT NULL PRIMARY KEY, "feed_id" integer NOT NULL, "agency_id" varchar(255) NOT NULL, "name" varchar(255) NOT NULL, "url" varchar(200) NOT NULL, "timezone" varchar(255) NOT NULL, "lang" varchar(2) NOT NULL, "phone" varchar(255) NOT NULL, "fare_url" varchar(200) NOT NULL)
The error was: relation "agency" already exists
Error in migration: multigtfs:0004_initial
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/araichev/lib/python2.7/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/home/araichev/lib/python2.7/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 232, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 307, in migrate_many
result = self.migrate(migration, database)
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 132, in migrate
result = self.run(migration, database)
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 116, in run
return self.run_migration(migration, database)
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 86, in run_migration
migration_function()
File "/home/araichev/lib/python2.7/south/migration/migrators.py", line 62, in <lambda>
return (lambda: direction(orm))
File "/home/araichev/lib/python2.7/multigtfs/migrations/0004_initial.py", line 21, in forwards
('fare_url', self.gf('django.db.models.fields.URLField')(max_length=200, blank=True)),
File "/home/araichev/lib/python2.7/south/db/generic.py", line 47, in _cache_clear
return func(self, table, *args, **opts)
File "/home/araichev/lib/python2.7/south/db/generic.py", line 361, in create_table
"columns": ', '.join([col for col in columns if col]),
File "/home/araichev/lib/python2.7/south/db/generic.py", line 282, in execute
cursor.execute(sql, params)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/araichev/webapps/gtfs_explorer_1/lib/python2.7/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "agency" already exists
Update to Feb 17, 2014 GTFS Spec
Google updated the GTFS spec on Feb 17th 2014 (see date at bottom):
https://developers.google.com/transit/gtfs/reference
Need to review changes, update doc strings, and add fields as needed.
Changes include:
- Trips: Added wheelchair_accessible and bikes_allowed (this has been in place since Oct 2012 ).
Handle BOM on import, used in Sydney feed
Andrew Harvey posted about the Transport for NSW (Sydney, NSW, Australia) GTFS feed in the Google group:
https://groups.google.com/forum/#!topic/transit-developers/NlYiJDfwSNc
It is available here:
Downloading requires registering for an account and agreeing to terms of service.
It is a huge feed, 227 MB compressed. I can't run the Google verification script on it, since I can't load the whole thing into an in-memory SQLite database.
It fails to import into multigtfs 0.4.0. This is probably because the files start with a BOM. This breaks both Python 2 and Python 3. It thinks the BOM is part of the first columns name. In agency.txt, this means the agency_id column is taken as extra data, and agency_id (optional) is blank. So, duplicate detection rejects all but the first line.
"Enter valid json data" when editing fields in django admin
When editing any of the GTFS fields in django admin, the "extra_data" field gets a validation error saying "This field is required". I have found a workaround entering '{"somestuff": "to_avoid_validation_errors"})' instead of the default value of ' {}', but it would be good to fix the problem.Thanks!
Handle empty lines
The routes.txt file in https://www.transportforireland.ie/transitData/google_transit_buseireann.zip has an extra newline character at the end, which is misinterpreted as an actual row resulting in django.db.utils.IntegrityError: NOT NULL constraint failed: route.rtype.
Link routes to default agency on import
The agency_id field is optional for feeds with only one agency. This means that routes are not directly associated within an agency on import as there appears to be no relationship .
Suggest that on import check to see if there is only one agency present. If there is only one agency then assign it a default agency_id (e.g. 1) and update the route records to reference this new agency.
I don't fully understand the import process but it would appear that a good option might be to update the "import_gtfs" function in the feed model (immediately before update geometries)? I attempted adding a a default value of 1 to agency_id and Agency and agency_id in Route but this didn't work.
Autonaming breaks for feeds with calendar_dates.txt
When importing a feed with a calendar_dates.txt (The "TriMet Style") with the management command, the import fails with a nasty error when it tries to rename the feed at the very end. The rest of the import seems to work flawlessly.
When providing a name via the -n switch, the name generation is skipped and the error is avoided.
It appears that the Service objects are being created, but with Service.start_date and Service.end_date both nulled. When the importer gets to this part
django-multi-gtfs/multigtfs/management/commands/importgtfs.py
Lines 84 to 92 in f104c09
the service exists, but service.start_date is None which causes the strftime to puke out this error:
INFO - Import completed in 2512.7 seconds.
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/mharvey/Envs/portal-rewrite/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/Users/mharvey/Envs/portal-rewrite/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/mharvey/Envs/portal-rewrite/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/mharvey/Envs/portal-rewrite/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/Users/mharvey/Envs/portal-rewrite/lib/python3.6/site-packages/multigtfs/management/commands/importgtfs.py", line 92, in handle
name += service.start_date.strftime(' starting %Y-%m-%d')
AttributeError: 'NoneType' object has no attribute 'strftime'
I think the better behavior would be to check for an existing FeedInfo and use the start/end dates from that, then fall back to Service, and if the start_date is None or the Service doesn't exist, then finally fall back to the default name with the date on which the feed was imported.
Migration error 2
Using multigtfs 0.3.3 on my local machine works great. Setting it up on my production web server on Webfaction gives me the migration error below. Any tips on what's going wrong?
$ python manage.py migrate multigtfs
Running migrations for multigtfs:
- Migrating forwards to 0016_add_20140217_optional_fields.
> multigtfs:0001_initial
> multigtfs:0002_auto__chg_field_fare_transfers
> multigtfs:0003_convert_fare_transfers
> multigtfs:0004_auto_add_geopoint_fields
> multigtfs:0005_migrate_lat_lon_to_point
> multigtfs:0006_auto_del_lat_lon_fields
> multigtfs:0007_rm_geography_points
Error in migration: multigtfs:0007_rm_geography_points
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 232, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 307, in migrate_many
result = self.migrate(migration, database)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 132, in migrate
result = self.run(migration, database)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 116, in run
return self.run_migration(migration, database)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 86, in run_migration
migration_function()
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 62, in <lambda>
return (lambda: direction(orm))
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/multigtfs/migrations/0007_rm_geography_points.py", line 26, in forwards
db.alter_column('shape_point', 'point', field)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 47, in _cache_clear
return func(self, table, *args, **opts)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 509, in alter_column
"type": self._db_type_for_alter_column(field),
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/postgresql_psycopg2.py", line 93, in _db_type_for_alter_column
return self._db_positive_type_for_alter_column(DatabaseOperations, field)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 1135, in _db_positive_type_for_alter_column
super_result = super(klass, self)._db_type_for_alter_column(field)
File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 435, in _db_type_for_alter_column
return field.db_type()
TypeError: db_type_for_postgis() takes exactly 2 arguments (1 given)
exportgtfs should use compression if available
If zlib is available, then exportgtfs should create a zip file with compression. See ZipFile docs.
Update sample test feeds
One of the test feeds from the OSS Transit TimeTable Publishing Application is an invalid feed, but is used for round-trip testing. It will result in an import error once issue #7 is fixed.
We need new feed(s) for testing purposes, that cover some of the optional files that are common in current GTFS development. They still should be minimal.
do not override db_table
Can you explain the reasoning for why db_table is being overridden in all the models instead of using the default of app_label + model name?
Add shape distance units attribute to Feed class?
I can't find a place in the GTFS reference where one can specify the distance units used in the feed's shape_dist_traveled field. Knowing a feed's distance units is, of course, essential for calculating distance stats.
If there is no standard place for units in the specification, it would handy to add a 'units' attribute to the multigtfs Feed class that notes the distance units used in the feed, as supplied by the user upon calling importgtfs, say. Perhaps this attribute could be a CharField field with defalut='km' for kilometers.
What do you all think?
Choice should be characters for Frequency field
In Frequency, the exact_times is a CharField, but the choices are integers. This results in an error when saving:
Select a valid choice. 0 is not one of the available choices.
Support generating validation report
Google provides an open source feed validator:
https://code.google.com/p/googletransitdatafeed/wiki/FeedValidator
It is not easy to call the code from django-multi-gtfs (it is built around loading the data from a zip file), but it should be easy to write a task that export the feed to a temp file, runs the feed validator as a subprocess, and imports the resulting HTML into the database. This would simplify my own GTFS export process.
Look into speeding up feed imports
I think it would be worthwhile for someone, possibly me, to spend some time figuring out how to speed up feed imports. As it stands, a 165MB feed (21MB zipped) such as Southeast Queensland's (http://translink.com.au/sites/default/files/assets/resources/about-translink/reporting-and-publications/open-data/gtfs/SEQ.zip) takes about 9--12 hours to load into a PostGIS database. (I'll time it more accurately next time i do the import.) Is that slow? Seems like it, but i don't know much about databases.
Handle quotation marks
The trips.txt file in this feed https://www.transportforireland.ie/transitData/google_transit_expressbus.zip has some odd lines like
46-860-y11-1,"Mo-Fr#1","33.Mo-Fr.46-860-y11-1.2.I","46-860-y11-1.2.I","Aston Quay, stop 4720", "1"
The extra space means "1" is used as the direction_id, which causes an error (DataError: value too long for type character varying(1)).
Couldn't install with pip within a virtualenv
I tried to install multigtfs via pip within a virtualenv and failed. See the log below.
Maybe the 'build/' directory is not what it should be? After the attempted installation, my build directory '/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/build/' contains 'Users/' and 'pip-egg-info/ '. And 'Users/' contains 'johnw/.virtualenvs/django-multi-gtfs/lib/python2.7/site-packages/multigtfs'.
Any help would be appreciated.
/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/bin/pip run on Fri Jun 7 10:33:34 2013
Downloading/unpacking multigtfs
Getting page https://pypi.python.org/simple/multigtfs/
URLs to search for versions for multigtfs:
-
https://pypi.python.org/simple/multigtfs/
Analyzing links from page https://pypi.python.org/simple/multigtfs/
Found link https://pypi.python.org/packages/source/m/multigtfs/multigtfs-0.2.2.tar.gz#md5=762568d94cd572aae4fbe327e3509cd6 (from https://pypi.python.org/simple/multigtfs/), version: 0.2.2
Found link https://pypi.python.org/packages/any/m/multigtfs/multigtfs-0.2.5.macosx-10.8-intel.tar.gz#md5=19bd85697c319a80cfc3eb6442171a82 (from https://pypi.python.org/simple/multigtfs/), version: 0.2.5.macosx-10.8-intel
Found link https://pypi.python.org/packages/source/m/multigtfs/multigtfs-0.2.4.tar.gz#md5=16509ece047157940b24e15109fc2e54 (from https://pypi.python.org/simple/multigtfs/), version: 0.2.4
Found link https://pypi.python.org/packages/source/m/multigtfs/multigtfs-0.2.5.tar.gz#md5=e894b44ee5abd966ac80549780a989a9 (from https://pypi.python.org/simple/multigtfs/), version: 0.2.5
Found link https://pypi.python.org/packages/source/m/multigtfs/multigtfs-0.2.3.tar.gz#md5=cfb4db09b8005f161e73596ae11169f4 (from https://pypi.python.org/simple/multigtfs/), version: 0.2.3
Using version 0.2.5.macosx-10.8-intel (newest of versions: 0.2.5.macosx-10.8-intel, 0.2.5, 0.2.4, 0.2.3, 0.2.2)
Downloading from URL https://pypi.python.org/packages/any/m/multigtfs/multigtfs-0.2.5.macosx-10.8-intel.tar.gz#md5=19bd85697c319a80cfc3eb6442171a82 (from https://pypi.python.org/simple/multigtfs/)
Running setup.py egg_info for package multigtfsTraceback (most recent call last):
File "<string>", line 16, in <module>IOError: [Errno 2] No such file or directory: '/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/build/multigtfs/setup.py'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 16, in
IOError: [Errno 2] No such file or directory: '/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/build/multigtfs/setup.py'
Command python setup.py egg_info failed with error code 1 in /Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/build/multigtfs
Exception information:
Traceback (most recent call last):
File "/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", line 139, in main
status = self.run(options, args)
File "/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", line 266, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1057, in prepare_files
req_to_install.run_egg_info()
File "/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 236, in run_egg_info
command_desc='python setup.py egg_info')
File "/Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/util.py", line 662, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command python setup.py egg_info failed with error code 1 in /Users/raichev/Dropbox/.virtualenvs/django_gtfs_analyzer/build/multigtfs
"String or unicode input unrecognized as WKT EWKT, and HEXEWKB"
stops.txt in https://si.metzmetropole.fr/fiches/opendata/gtfs_current.zip has some lines like this:
10258,TANEU,TANNEURS,,+49.119799,+06.182966,1,
The + sign prevents PostGIS from recognising POINT(+06.182966 +49.119799) as valid input.
Use raw_id_fields in admin
When there is a lot of data, the Django admin becomes slow. Using custom admin classes with raw_id_fields will speed it up. For example, stops could use raw_id_fields for feed, zone, and parent station.
When re-importing a feed using importgtfs the feed is duplicated in the database instead of updated
As the title says, is this the expected behavior? If so, on a production environment how do you update a feed? I am only trying to avoid deleting it first and import it after.
Error importing transfers.txt with invalid stop IDs
Greetings.
Using mult-gtfs v0.4.3, i tried to import this GTFS feed for Fort Lauderdale: http://transitfeeds.com/p/broward-county-transit/49/latest/download and got the following error.
Traceback (most recent call last):
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
psycopg2.IntegrityError: null value in column "point" violates not-null constraint
DETAIL: Failing row contains (249219, 54, 169, , , , null, , , null, , null, , {}).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/multigtfs/management/commands/importgtfs.py", line 73, in handle
feed.import_gtfs(gtfs_feed)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/multigtfs/models/feed.py", line 109, in import_gtfs
count = klass.import_txt(table, self) or 0
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/multigtfs/models/base.py", line 249, in import_txt
fields[name_map[column_name]] = val_map[column_name](value)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/multigtfs/models/base.py", line 155, in get_instance
**kwargs).id
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/manager.py", line 157, in create
return self.get_queryset().create(**kwargs)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/query.py", line 322, in create
obj.save(force_insert=True, using=self.db)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/query.py", line 1514, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
cursor.execute(sql, params)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/utils/six.py", line 658, in reraise
raise value.with_traceback(tb)
File "/home/araichev/.virtualenvs/bingo/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "point" violates not-null constraint
DETAIL: Failing row contains (249219, 54, 169, , , , null, , , null, , null, , {}).
After some debugging, i discovered that the feed is faulty: transfers.txt contains stop IDs not present in stops.txt.
Fair enough, but it appears that multigtfs is trying to create the stops in transfers.txt that it can't find in stops.txt, and that causes the error; see https://github.com/tulsawebdevs/django-multi-gtfs/blob/v1.0.0/multigtfs/models/base.py#L153.
That seems like a bug to me, because transfers.txt will never contain enough data, e.g. stop geography, to properly create stop objects.
I'd favor a warning, such as, "Encountered stop ID in transfers.txt, which is not present in stops.txt. Skipping this stop."
What do you think?
Now, i didn't test the feed import using multigtfs v1.0.0, but i do see that the offending code block linked above is the same as in version 0.4.3.
Thanks for your attention.
Add initial public views of data
It would be useful to see the GTFS data in a webbrowser for non-admins. If done with Tastypie, we might get a API for free as well. The first version could be maps + data + links between data items.
Import route_directions.txt
I understand that this is a non-standard extension to GTFS, but it seems to be adopted by several agencies (Mainly TriMet, but others are including it as well) and the information within is extremely useful. Here is a sample.
Is there opposition to including this as something that would be imported if it's present in the archive abut ignored if it's not?
Looking at the code, I'd be happy to take a stab at implementing this and send a PR for review.
In class FeedInfo, change attribute ``version`` to have max char length 255
The current max char length of 20 is too short for some feeds, such as http://www.gtfs-data-exchange.com/agency/basin-transit-service/.
Bug in muligtfs 0.3.1 migrations?
Hi John, i just installed multigtfs 0.3.1, started from a fresh PostGIS database, synced the database, and then ran python manage.py migrate multigtfs and got the following error
Running migrations for multigtfs:
- Migrating forwards to 0015_populate_field_route_geometry.
> multigtfs:0001_initial
> multigtfs:0002_auto__chg_field_fare_transfers
> multigtfs:0003_convert_fare_transfers
> multigtfs:0004_auto_add_geopoint_fields
> multigtfs:0005_migrate_lat_lon_to_point
> multigtfs:0006_auto_del_lat_lon_fields
> multigtfs:0007_rm_geography_points
FATAL ERROR - The following SQL query failed: ALTER TABLE "shape_point" ALTER COLUMN "point" TYPE geometry(POINT,4326), ALTER COLUMN "point" SET NOT NULL, ALTER COLUMN "point" DROP DEFAULT;
The error was: column "point" cannot be cast automatically to type geometry
HINT: Specify a USING expression to perform the conversion.
Error in migration: multigtfs:0007_rm_geography_points
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 232, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 307, in migrate_many
result = self.migrate(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 132, in migrate
result = self.run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 116, in run
return self.run_migration(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 86, in run_migration
migration_function()
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 62, in <lambda>
return (lambda: direction(orm))
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/multigtfs/migrations/0007_rm_geography_points.py", line 14, in forwards
db.alter_column('shape_point', 'point', self.gf('django.contrib.gis.db.models.fields.PointField')())
File "/usr/local/lib/python2.7/site-packages/south/db/generic.py", line 47, in _cache_clear
return func(self, table, *args, **opts)
File "/usr/local/lib/python2.7/site-packages/south/db/generic.py", line 536, in alter_column
flatten(values),
File "/usr/local/lib/python2.7/site-packages/south/db/generic.py", line 282, in execute
cursor.execute(sql, params)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/araichev/.virtualenvs/gtfs_explorer_develop/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column "point" cannot be cast automatically to type geometry
HINT: Specify a USING expression to perform the conversion.
Bug in shape geometry
I was playing around with the Bowen GTFS feed (https://data.qld.gov.au/dataset/general-transit-feed-specification-gtfs-qconnect/resource/999196bf-8815-47b5-ac90-e8caa9a7aee3) and discovered a bug, i think, in the geometry assigned to shape objects: the linestrings aren't following the shape point sequence.
One solution is to replace line https://github.com/tulsawebdevs/django-multi-gtfs/blob/master/multigtfs/models/shape.py#L109 with
points = trip.shape.points.order_by('sequence').values_list('point', flat=True)
Here's an example bug based on the Bowen feed. Consider the following code.
feed = Feed.objects.get(name='Bowen')
route = Route.objects.in_feed(feed).get(short_name='1')
trip = route.trip_set.all()[0]
points = trip.shape.points.order_by('sequence').values_list('point', flat=True)
geometry1 = [p.coords for p in points]
geometry2 = trip.shape.geometry
if geometry1 != geometry2:
print('Unequal!')
print(list(geometry1))
print(list(geometry2))
It produces
Unequal!
[(148.246743818, -20.0129421084), (148.246227635, -20.012129696), (148.245502961, -20.0109899199), (148.244779167, -20.0098528702), (148.244056335, -20.0087158452), (148.243457358, -20.0077723199), (148.243391096, -20.0076930275), (148.243365345, -20.0076932358), (148.243335021, -20.0076870121), (148.24330584, -20.0076745103), (148.243280632, -20.0076567078), (148.243260269, -20.0076363318), (148.243252973, -20.0076253202), (148.241397251, -20.0086776183), (148.240644127, -20.0075343612), (148.240644127, -20.0075343612), (148.241397251, -20.0086776183), (148.243155966, -20.0076803907), (148.243155966, -20.0076803907), (148.243252973, -20.0076253202), (148.243243907, -20.0076097547), (148.243235221, -20.0075815815), (148.243232278, -20.0075526617), (148.243235051, -20.0075238958), (148.243244467, -20.0074962098), (148.243253502, -20.0074811314), (148.243158811, -20.007301033), (148.242524452, -20.0062979717), (148.242144592, -20.0057091332), (148.24187069, -20.0052745594), (148.24187069, -20.0052745594), (148.241540381, -20.0047501429), (148.240978317, -20.003848165), (148.239027616, -20.0049528633), (148.238890113, -20.0047382638), (148.238890113, -20.0047382638), (148.238307378, -20.0038267106), (148.237865093, -20.0031586764), (148.237706276, -20.0028921303), (148.237706276, -20.0028921303), (148.237386791, -20.0023571856), (148.237093742, -20.0018941527), (148.237093742, -20.0018941527), (148.236891672, -20.0015759703), (148.236461129, -20.0008983371), (148.2384994, -20.0013786406), (148.238601531, -20.0014381709), (148.238601531, -20.0014381709), (148.239621019, -20.0020307185), (148.240095063, -20.0022778117), (148.240324778, -20.0024065687), (148.242094433, -20.0010752644), (148.242516258, -20.0006819464), (148.242713316, -19.9996471845), (148.2430853, -19.9958421197), (148.243239065, -19.9943762852), (148.243410338, -19.9934372625), (148.242856975, -19.9927851811), (148.242856975, -19.9927851811), (148.242221706, -19.9920353609), (148.242221706, -19.9920353609), (148.242114889, -19.9919099149), (148.240825937, -19.991782405), (148.240825937, -19.991782405), (148.239691913, -19.9916707724), (148.239691913, -19.9916707724), (148.238849718, -19.991587718), (148.236868882, -19.9913883788), (148.236868882, -19.9913883788), (148.23656949, -19.9913577842), (148.236770194, -19.9895696757), (148.236792574, -19.9893647903), (148.236792574, -19.9893647903), (148.236894282, -19.9884572531), (148.237091715, -19.9867772063), (148.237191768, -19.9862724886), (148.236490429, -19.9865465155), (148.236490429, -19.9865465155), (148.236121933, -19.9866898071), (148.235212908, -19.9844914816), (148.235129991, -19.9843008847), (148.235129991, -19.9843008847), (148.234851, -19.9836570799), (148.233320172, -19.9830174061), (148.23382151, -19.9820088828), (148.23325794, -19.9817611757), (148.23325794, -19.9817611757), (148.231700818, -19.9810775234), (148.231640616, -19.9810506658), (148.231640616, -19.9810506658), (148.230262237, -19.9804466242), (148.230159333, -19.9803816635), (148.230156312, -19.9803869897), (148.230131684, -19.9804133635), (148.230102484, -19.9804333052), (148.230067758, -19.9804467892), (148.230032243, -19.9804548444), (148.229995067, -19.9804547435), (148.229958136, -19.9804465379), (148.229924225, -19.9804330061), (148.229894315, -19.9804132734), (148.229870336, -19.9803864906), (148.229851202, -19.9803571346), (148.229839825, -19.9803234813), (148.229836094, -19.980289133), (148.229838923, -19.9802585665), (148.229851494, -19.9802210527), (148.229870515, -19.9801909227), (148.229894164, -19.9801654238), (148.229921405, -19.9801472318), (148.229924344, -19.9801446072), (148.229958117, -19.9801310975), (148.229993632, -19.9801230424), (148.23003176, -19.9801231689), (148.230067739, -19.9801313488), (148.23010165, -19.9801448806), (148.230132513, -19.9801646389), (148.230160138, -19.9801969276), (148.230992094, -19.9796686835), (148.229891255, -19.9780176508), (148.229891255, -19.9780176508), (148.2298648, -19.9779781836), (148.229001285, -19.9766769244), (148.229001285, -19.9766769244), (148.228740467, -19.9762850567), (148.227385609, -19.974271173), (148.227385609, -19.974271173), (148.227205874, -19.9740040613), (148.227599416, -19.9740408095), (148.226738271, -19.9726305583), (148.22653369, -19.9723023925), (148.22653369, -19.9723023925), (148.226049987, -19.9715259861), (148.225534337, -19.9707027536), (148.225342353, -19.9708102319), (148.225342353, -19.9708102319), (148.224679181, -19.9711798811), (148.225194827, -19.9720031154), (148.225416007, -19.9723812989), (148.225416007, -19.9723812989), (148.225847456, -19.9731202474), (148.226112646, -19.9735563965), (148.226112646, -19.9735563965), (148.226338756, -19.973929305), (148.225742689, -19.973875375), (148.225941608, -19.9741700427), (148.225941608, -19.9741700427), (148.227299465, -19.9761786076), (148.227299465, -19.9761786076), (148.227763014, -19.9768652463), (148.228765242, -19.97837123), (148.228765242, -19.97837123), (148.22888927, -19.9785575289), (148.229921405, -19.9801472318), (148.229924344, -19.9801446072), (148.229958117, -19.9801310975), (148.229993632, -19.9801230424), (148.23003176, -19.9801231689), (148.230067739, -19.9801313488), (148.23010165, -19.9801448806), (148.230132513, -19.9801646389), (148.230160138, -19.9801969276), (148.230174672, -19.980220752), (148.23018605, -19.9802544053), (148.230189781, -19.9802887536), (148.230185864, -19.980323797), (148.230175334, -19.9803568597), (148.230159333, -19.9803816635), (148.230262237, -19.9804466242), (148.231001747, -19.9807711819), (148.231001747, -19.9807711819), (148.231700818, -19.9810775234), (148.233102698, -19.9816930039), (148.233102698, -19.9816930039), (148.23382151, -19.9820088828), (148.233320172, -19.9830174061), (148.234632585, -19.9835655773), (148.234632585, -19.9835655773), (148.234851, -19.9836570799), (148.235212908, -19.9844914816), (148.236283111, -19.9840931028), (148.237121341, -19.9860786233), (148.237191768, -19.9862724886), (148.23808016, -19.9872895983), (148.23808016, -19.9872895983), (148.241909342, -19.9916754631), (148.241909342, -19.9916754631), (148.242114889, -19.9919099149), (148.240825937, -19.991782405), (148.240825937, -19.991782405), (148.239691913, -19.9916707724), (148.239691913, -19.9916707724), (148.238849718, -19.991587718), (148.236868882, -19.9913883788), (148.236868882, -19.9913883788), (148.23656949, -19.9913577842), (148.236488689, -19.9920775196), (148.236478245, -19.9921709697), (148.236478245, -19.9921709697), (148.236418602, -19.9926947995), (148.23628759, -19.9938466894), (148.23628759, -19.9938466894), (148.236264118, -19.9940560513), (148.236139765, -19.9951450317), (148.235886344, -19.99738415), (148.235808278, -19.9980760184), (148.235808278, -19.9980760184), (148.235784994, -19.9982790762), (148.235708693, -19.9989439541), (148.235520594, -20.0006602954), (148.23576377, -20.0007218255), (148.236461129, -20.0008983371), (148.2384994, -20.0013786406), (148.238601531, -20.0014381709), (148.238601531, -20.0014381709), (148.239621019, -20.0020307185), (148.240095063, -20.0022778117), (148.240349707, -20.0026550882), (148.241057685, -20.0038088433), (148.241624513, -20.0047109493), (148.242226084, -20.0056626581), (148.242604011, -20.0062523458), (148.243211146, -20.0072096107), (148.243211146, -20.0072096107), (148.24323848, -20.0072518047), (148.24335176, -20.0074161805), (148.243365207, -20.0074129373), (148.243396673, -20.0074128829), (148.243426997, -20.0074191065), (148.243456206, -20.0074307078), (148.243481387, -20.0074494108), (148.243501749, -20.0074697867), (148.24350812, -20.0074798722), (148.243518112, -20.0074963639), (148.243526825, -20.0075236365), (148.243529768, -20.0075525563), (148.243526967, -20.0075822228), (148.243518505, -20.0076099344), (148.243502475, -20.0076356399), (148.243495616, -20.0076417642), (148.243480864, -20.0076566888), (148.243456477, -20.0076749591), (148.243427516, -20.0076867975), (148.243396812, -20.0076931814), (148.243391096, -20.0076930275), (148.243365345, -20.0076932358), (148.243335021, -20.0076870121), (148.24330584, -20.0076745103), (148.243280632, -20.0076567078), (148.243260269, -20.0076363318), (148.243252973, -20.0076253202), (148.241397251, -20.0086776183), (148.240644127, -20.0075343612), (148.240644127, -20.0075343612), (148.241397251, -20.0086776183), (148.239404677, -20.0098091224), (148.237790375, -20.0107272791), (148.237790375, -20.0107272791), (148.2374188, -20.0109389767), (148.238141611, -20.0120751206), (148.238862472, -20.0132130133), (148.239585275, -20.0143500558), (148.240307134, -20.0154870718), (148.241032704, -20.0166277916), (148.242477181, -20.0158095869), (148.242477181, -20.0158095869), (148.243011705, -20.0155067322), (148.243740238, -20.0166448201), (148.244471801, -20.0177775808), (148.245890147, -20.0169721625), (148.245890147, -20.0169721625), (148.246462506, -20.016647783), (148.245738688, -20.015510731), (148.245256075, -20.0147550972), (148.245256075, -20.0147550972), (148.245010197, -20.0143708482), (148.245899819, -20.013864838), (148.245899819, -20.013864838), (148.246841807, -20.0133295876), (148.246830889, -20.0133121697), (148.246821331, -20.0132812694), (148.24681659, -20.013248696), (148.24682135, -20.0132172795), (148.246830874, -20.0131859911), (148.246847939, -20.0131576094), (148.246854825, -20.0131505845), (148.246800232, -20.0130319506), (148.246743818, -20.0129421084)]
[(148.246743818, -20.0129421084), (148.246800232, -20.0130319506), (148.246854825, -20.0131505845), (148.246847939, -20.0131576094), (148.246830874, -20.0131859911), (148.24682135, -20.0132172795), (148.24681659, -20.013248696), (148.246821331, -20.0132812694), (148.246830889, -20.0133121697), (148.246841807, -20.0133295876), (148.245899819, -20.013864838), (148.245899819, -20.013864838), (148.245010197, -20.0143708482), (148.245256075, -20.0147550972), (148.245256075, -20.0147550972), (148.245738688, -20.015510731), (148.246462506, -20.016647783), (148.245890147, -20.0169721625), (148.245890147, -20.0169721625), (148.244471801, -20.0177775808), (148.243740238, -20.0166448201), (148.243011705, -20.0155067322), (148.242477181, -20.0158095869), (148.242477181, -20.0158095869), (148.241032704, -20.0166277916), (148.240307134, -20.0154870718), (148.239585275, -20.0143500558), (148.238862472, -20.0132130133), (148.238141611, -20.0120751206), (148.2374188, -20.0109389767), (148.237790375, -20.0107272791), (148.237790375, -20.0107272791), (148.239404677, -20.0098091224), (148.241397251, -20.0086776183), (148.240644127, -20.0075343612), (148.240644127, -20.0075343612), (148.241397251, -20.0086776183), (148.243252973, -20.0076253202), (148.243260269, -20.0076363318), (148.243280632, -20.0076567078), (148.24330584, -20.0076745103), (148.243335021, -20.0076870121), (148.243365345, -20.0076932358), (148.243391096, -20.0076930275), (148.243396812, -20.0076931814), (148.243427516, -20.0076867975), (148.243456477, -20.0076749591), (148.243480864, -20.0076566888), (148.243495616, -20.0076417642), (148.243502475, -20.0076356399), (148.243518505, -20.0076099344), (148.243526967, -20.0075822228), (148.243529768, -20.0075525563), (148.243526825, -20.0075236365), (148.243518112, -20.0074963639), (148.24350812, -20.0074798722), (148.243501749, -20.0074697867), (148.243481387, -20.0074494108), (148.243456206, -20.0074307078), (148.243426997, -20.0074191065), (148.243396673, -20.0074128829), (148.243365207, -20.0074129373), (148.24335176, -20.0074161805), (148.24323848, -20.0072518047), (148.243211146, -20.0072096107), (148.243211146, -20.0072096107), (148.242604011, -20.0062523458), (148.242226084, -20.0056626581), (148.241624513, -20.0047109493), (148.241057685, -20.0038088433), (148.240349707, -20.0026550882), (148.240095063, -20.0022778117), (148.239621019, -20.0020307185), (148.238601531, -20.0014381709), (148.238601531, -20.0014381709), (148.2384994, -20.0013786406), (148.236461129, -20.0008983371), (148.23576377, -20.0007218255), (148.235520594, -20.0006602954), (148.235708693, -19.9989439541), (148.235784994, -19.9982790762), (148.235808278, -19.9980760184), (148.235808278, -19.9980760184), (148.235886344, -19.99738415), (148.236139765, -19.9951450317), (148.236264118, -19.9940560513), (148.23628759, -19.9938466894), (148.23628759, -19.9938466894), (148.236418602, -19.9926947995), (148.236478245, -19.9921709697), (148.236478245, -19.9921709697), (148.236488689, -19.9920775196), (148.23656949, -19.9913577842), (148.236868882, -19.9913883788), (148.236868882, -19.9913883788), (148.238849718, -19.991587718), (148.239691913, -19.9916707724), (148.239691913, -19.9916707724), (148.240825937, -19.991782405), (148.240825937, -19.991782405), (148.242114889, -19.9919099149), (148.241909342, -19.9916754631), (148.241909342, -19.9916754631), (148.23808016, -19.9872895983), (148.23808016, -19.9872895983), (148.237191768, -19.9862724886), (148.237121341, -19.9860786233), (148.236283111, -19.9840931028), (148.235212908, -19.9844914816), (148.234851, -19.9836570799), (148.234632585, -19.9835655773), (148.234632585, -19.9835655773), (148.233320172, -19.9830174061), (148.23382151, -19.9820088828), (148.233102698, -19.9816930039), (148.233102698, -19.9816930039), (148.231700818, -19.9810775234), (148.231001747, -19.9807711819), (148.231001747, -19.9807711819), (148.230262237, -19.9804466242), (148.230159333, -19.9803816635), (148.230175334, -19.9803568597), (148.230185864, -19.980323797), (148.230189781, -19.9802887536), (148.23018605, -19.9802544053), (148.230174672, -19.980220752), (148.230160138, -19.9801969276), (148.230132513, -19.9801646389), (148.23010165, -19.9801448806), (148.230067739, -19.9801313488), (148.23003176, -19.9801231689), (148.229993632, -19.9801230424), (148.229958117, -19.9801310975), (148.229924344, -19.9801446072), (148.229921405, -19.9801472318), (148.22888927, -19.9785575289), (148.228765242, -19.97837123), (148.228765242, -19.97837123), (148.227763014, -19.9768652463), (148.227299465, -19.9761786076), (148.227299465, -19.9761786076), (148.225941608, -19.9741700427), (148.225941608, -19.9741700427), (148.225742689, -19.973875375), (148.226338756, -19.973929305), (148.226112646, -19.9735563965), (148.226112646, -19.9735563965), (148.225847456, -19.9731202474), (148.225416007, -19.9723812989), (148.225416007, -19.9723812989), (148.225194827, -19.9720031154), (148.224679181, -19.9711798811), (148.225342353, -19.9708102319), (148.225342353, -19.9708102319), (148.225534337, -19.9707027536), (148.226049987, -19.9715259861), (148.22653369, -19.9723023925), (148.22653369, -19.9723023925), (148.226738271, -19.9726305583), (148.227599416, -19.9740408095), (148.227205874, -19.9740040613), (148.227385609, -19.974271173), (148.227385609, -19.974271173), (148.228740467, -19.9762850567), (148.229001285, -19.9766769244), (148.229001285, -19.9766769244), (148.2298648, -19.9779781836), (148.229891255, -19.9780176508), (148.229891255, -19.9780176508), (148.230992094, -19.9796686835), (148.230160138, -19.9801969276), (148.230132513, -19.9801646389), (148.23010165, -19.9801448806), (148.230067739, -19.9801313488), (148.23003176, -19.9801231689), (148.229993632, -19.9801230424), (148.229958117, -19.9801310975), (148.229924344, -19.9801446072), (148.229921405, -19.9801472318), (148.229894164, -19.9801654238), (148.229870515, -19.9801909227), (148.229851494, -19.9802210527), (148.229838923, -19.9802585665), (148.229836094, -19.980289133), (148.229839825, -19.9803234813), (148.229851202, -19.9803571346), (148.229870336, -19.9803864906), (148.229894315, -19.9804132734), (148.229924225, -19.9804330061), (148.229958136, -19.9804465379), (148.229995067, -19.9804547435), (148.230032243, -19.9804548444), (148.230067758, -19.9804467892), (148.230102484, -19.9804333052), (148.230131684, -19.9804133635), (148.230156312, -19.9803869897), (148.230159333, -19.9803816635), (148.230262237, -19.9804466242), (148.231640616, -19.9810506658), (148.231640616, -19.9810506658), (148.231700818, -19.9810775234), (148.23325794, -19.9817611757), (148.23325794, -19.9817611757), (148.23382151, -19.9820088828), (148.233320172, -19.9830174061), (148.234851, -19.9836570799), (148.235129991, -19.9843008847), (148.235129991, -19.9843008847), (148.235212908, -19.9844914816), (148.236121933, -19.9866898071), (148.236490429, -19.9865465155), (148.236490429, -19.9865465155), (148.237191768, -19.9862724886), (148.237091715, -19.9867772063), (148.236894282, -19.9884572531), (148.236792574, -19.9893647903), (148.236792574, -19.9893647903), (148.236770194, -19.9895696757), (148.23656949, -19.9913577842), (148.236868882, -19.9913883788), (148.236868882, -19.9913883788), (148.238849718, -19.991587718), (148.239691913, -19.9916707724), (148.239691913, -19.9916707724), (148.240825937, -19.991782405), (148.240825937, -19.991782405), (148.242114889, -19.9919099149), (148.242221706, -19.9920353609), (148.242221706, -19.9920353609), (148.242856975, -19.9927851811), (148.242856975, -19.9927851811), (148.243410338, -19.9934372625), (148.243239065, -19.9943762852), (148.2430853, -19.9958421197), (148.242713316, -19.9996471845), (148.242516258, -20.0006819464), (148.242094433, -20.0010752644), (148.240324778, -20.0024065687), (148.240095063, -20.0022778117), (148.239621019, -20.0020307185), (148.238601531, -20.0014381709), (148.238601531, -20.0014381709), (148.2384994, -20.0013786406), (148.236461129, -20.0008983371), (148.236891672, -20.0015759703), (148.237093742, -20.0018941527), (148.237093742, -20.0018941527), (148.237386791, -20.0023571856), (148.237706276, -20.0028921303), (148.237706276, -20.0028921303), (148.237865093, -20.0031586764), (148.238307378, -20.0038267106), (148.238890113, -20.0047382638), (148.238890113, -20.0047382638), (148.239027616, -20.0049528633), (148.240978317, -20.003848165), (148.241540381, -20.0047501429), (148.24187069, -20.0052745594), (148.24187069, -20.0052745594), (148.242144592, -20.0057091332), (148.242524452, -20.0062979717), (148.243158811, -20.007301033), (148.243253502, -20.0074811314), (148.243244467, -20.0074962098), (148.243235051, -20.0075238958), (148.243232278, -20.0075526617), (148.243235221, -20.0075815815), (148.243243907, -20.0076097547), (148.243252973, -20.0076253202), (148.243155966, -20.0076803907), (148.243155966, -20.0076803907), (148.241397251, -20.0086776183), (148.240644127, -20.0075343612), (148.240644127, -20.0075343612), (148.241397251, -20.0086776183), (148.243252973, -20.0076253202), (148.243260269, -20.0076363318), (148.243280632, -20.0076567078), (148.24330584, -20.0076745103), (148.243335021, -20.0076870121), (148.243365345, -20.0076932358), (148.243391096, -20.0076930275), (148.243457358, -20.0077723199), (148.244056335, -20.0087158452), (148.244779167, -20.0098528702), (148.245502961, -20.0109899199), (148.246227635, -20.012129696), (148.246743818, -20.0129421084)]
How to import and use multiple feeds?
How do i import and use multiple feeds, say for two different cities, in one database? I can't figure this out from the current app documentation.
For example, i have GTFS files for Auckland and Sydney. If i import Auckland's via python manage.py importgtfs auckland.zip and then import Sydney's via the analogous command, does the Sydney import overwrite the Auckland one? Is the Auckland one saved as feed 1 and Sydney as feed 2? In the latter case, how would i get all the routes for the two feeds separately within an app or the shell? sydney_routes = Feed.objects.get(feed_id='2').Route.objects.all()?
Personally, i would like the following functionality. For imports from the command line: python manage.py importgtfs my_gtfs_zipfile_or_directory my_name_for_this_feed, where my_name_for_this_feed defaults to feed_my_gtfs_zipfile_or_directory. For use within an app or the shell, i would like to be able to do feed = Feed.objects.get(feed_name) and things like feed_routes = feed.Route.objects.all().
What do you all think?
Why does Trip have a `services` field instead of a `service_id` field (as in the Google spec)?...
...Doesn't each trip have just one service associated with it?
In importgtfs, use agency name for default name
When importing a GTFS feed using ./manage.py importgtfs, the default feed name should be set from the agency name in agency.txt, if included.
@probabble first implemented this in his fork by reading agency.txt before running the rest of the import (see importgtfs.py). I think it would be OK to set the feed name to a default, and then rename it after the data was imported.
Generic support of extra files
Hi,
I need to work manage GTFS feeds containing additional file (stop_external_ids.txt). I see that there is already an issue (#63) regarding other non-standard file.
I suggest to add generic support for extra files: define new model ExtraFile with columns feed, filename, and extra_data, which would handle any extra data. List of extra filenames would be defined in Feed.meta['extra_files'] and list of columns in Feed.meta['extra_files'] as usual.
I'll provide PR, but I want to discuss it first.
Do you have any objections or suggestions? Thank You.
Add Python 3.3 support
Code is fairly simply, so it should be possible for it to be valid 2.7 / 3.3 code, and build universal wheels. To the future!
Difficulty: Moderate
Criteria: Consistant tox.ini updated, tox builds combos of py2.7, py3.3 w/ Django 1.5, 1.6
Some areas that need fixing:
- Relative imports
- String literals
- StringIO imports
- Model's
__unicode__function - print usage
- Update packages in requirements.txt and requirements.dev.txt
Tools:
- https://docs.djangoproject.com/en/dev/topics/python3/
- https://pypi.python.org/pypi/six
- http://docs.python.org/dev/howto/pyporting.html
- http://tox.readthedocs.org/en/latest/
One Strategy:
- Install python 3.3
- Create fresh virtualenv w/ python 3.3
- Update versions in requirements.* and add six
- Bulk edit to add
from __future__ import absolute_import, unicode_literals, print_function - Run
./run_tests.py, fix issues, run again - Update
tox.iniand run tox
Create GTFS difference report
Create a method for showing the common elements and differences between two GTFS feeds.
Difficultly - Hard
Criteria - A text or HTML report would be a great starting place
One strategy: Create a hashing function for a 'row', use to find the identical elements in two feeds and the unique elements. For the unique elements, do a simple matching by GTFS IDs to identify records that changed from one to another. Generate a Markdown report.
importgtfs should work on a directory as well as a zip file
If a GTFS .zip is extracted to a folder (feed/agency.txt, feed/stops.txt, etc.), then importgtfs should be able to import it (./manage.py importgtfs path/to/feed)
Failed to import Portland's November feed
Hi folks,
Today i used multigtfs to import Portland's November feed (available at http://www.gtfs-data-exchange.com/agency/trimet/) and got the error below.
Is the problem that multigtfs doesn't handle feeds, such as Portland, that have a calendar_dates.txt file but no calendar.txt file? Such feeds are acceptable according to the GTFS, as described at https://developers.google.com/transit/gtfs/reference#calendar_dates_fields.
Traceback (most recent call last):
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
psycopg2.IntegrityError: null value in column "start_date" violates not-null constraint
DETAIL: Failing row contains (117, 13, C.404, t, t, t, t, t, t, t, null, null, {}).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/init.py", line 399, in execute_from_command_line
utility.execute()
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/init.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(_args, *_options.dict)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(_args, *_options)
File "/Users/araichev/gtfse/explorer/management/commands/import_feed.py", line 45, in handle
feed.import_gtfs(feed_file)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/feed.py", line 109, in import_gtfs
count = klass.import_txt(table, self) or 0
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/base.py", line 243, in import_txt
fields[name_map[column_name]] = val_mapcolumn_name
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/base.py", line 149, in get_instance
**kwargs).id
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/manager.py", line 157, in create
return self.get_queryset().create(**kwargs)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/query.py", line 322, in create
obj.save(force_insert=True, using=self.db)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/query.py", line 1514, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
cursor.execute(sql, params)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/utils.py", line 99, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/utils/six.py", line 549, in reraise
raise value.with_traceback(tb)
File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "start_date" violates not-null constraint
DETAIL: Failing row contains (117, 13, C.404, t, t, t, t, t, t, t, null, null, {}).
Compress route geometries
In https://github.com/tulsawebdevs/django-multi-gtfs/blob/master/multigtfs/models/route.py instead of making the geometry of a route a multilinestring of the list of its trip geometries, i think it would be better to make it a multilinestring of the set of its trip geometries. The latter doesn't contain duplicates and so will result in a smaller file, which is better when displaying routes on web maps.
Suggestion: add a LineStringField attribute to the Shape class for geo queries
I'm a newbie to GeoDjango, so please correct me if i'm misunderstanding something.
Would it be a good idea to add a LineStringField attribute to the Shape class in shape.py that encodes the geometry of the shape so that then one could make geometric queries such as "Get me all the trips in bounding box blah"?
Amend importgtfs.py to ignore extra columns in feed files
Some feeds e.g. Adelaide, Australia (http://www.gtfs-data-exchange.com/agency/adelaide-metro/) have extra columns in their feed files e.g. 'wheelchair_accessible' in trips.txt. Currently importgtfs.py fails on such feeds. It would be more useful, i reckon, for importgtfs.py to simply ignore these extra columns and import the rest of the columns as usual.
Remove 'verify_exists' option in models files to make compatible with Django 1.5
Shape object is missing in_feed() accessor/manager/thingy
I'd like to get all the shapes in a feed. Given the feed feed, i can do so via feed.shape_set.all(), but using Shape.objects.in_feed(feed) fails. Can this be fixed, perhaps by adding to the gtfs_order tuple in feed.py the item ('shapes.txt', Shape)?
Related to that, Trip.objects.in_feed(feed).all() works, but feed.trip_set.all() fails.
Add user documentation
Add user documentation, hosted on Read the Docs or Python Hosted.
Is migration ``0024_service_dates_optional.py`` included in the v0.4.3 on PyPi?
When i install v0.4.3 via pip, the migration is not included. Am i doing something wrong or is it actually missing?
Explore example does not restrict list views to the selected feed
When multiple feeds are in the database, most list views display all the objects, rather than just the objects in that feed.
A ListView with a feed-aware get_queryset would fix the issue.
Update admin site to include additional list fields and links between models
I have been working on adding additional list fields and links between models in the admin site. This involves adding list_display fields and defining inline lists in admin.py. I am currently working on lookin at a way of overriding standard inlines so will just list related records with links.
The admin site dashboard should also be updated to better reflect the relationships between models.
I will let you know when I have something worth looking at.
Perhaps increase route short name max character length
In routes.py, it might be a good idea to increase the short_name max length to something greater than 10 (line https://github.com/tulsawebdevs/django-multi-gtfs/blob/master/multigtfs/models/route.py#L129). Maybe 50.
The current restriction prevents importing some feeds, e.g. Lake Tahoe (http://www.gtfs-data-exchange.com/agency/tahoe-area-regional-transit/latest.zip).
service_date table doesn't have feed_id field
service_date table doesn't have feed_id field so it's impossible manage multigtfs that use calendar_dates information
How to import a GTFS directory into the database?
Hi folks, this project looks interesting, and i'd like to use it for my home town of Auckland, New Zealand. I followed the README to set up the software. Now how do i import the GTFS zip file (or directory) that i have for Auckland?
Thanks.
django 1.7+ support?
Hi,
I am interested in using your project in some form, but I want to use a more modern version of django. Can you explain why 1.7 and higher is not supported? If there is interest (like mine) in getting it higher, are you interested in outside help (not just feature requests)?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.