jarondl / pygtfs Goto Github PK
View Code? Open in Web Editor NEWA python (2/3) library for GTFS
License: MIT License
A python (2/3) library for GTFS
License: MIT License
Our builds don't exist, we should migrate the matrix of testing to github.
Is there a way to look at stop_times through a function like: stop_times_by_id()? Ex. - I want to answer a question like what are the stop times for the 33rd street stop?
Looking through schedule.py - Is there an issue since stop_times has an underscore or is already plural while the other objects are singular and one word?
For now I am just running a basic loop in Python. Continuing with the earlier example, I am gathering every stop_times object with a stop_id that is equivalent to stops_by_id(33). I hope to hear your thoughts. Thank you in advance!
gtfs2db dies with
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 16: ordinal not in range(128)
It looks like it's trying to "decode" a unicode object. I can provide more detailed info if you want to take this on. I don't want to fix it myself as I might break Python3 support.
The GTFS file I'm trying to consume is available here:
http://ressources.data.sncf.com/explore/dataset/sncf-horaires-des-lignes-transilien/?tab=metas
When agency_id is not set on a route, the import fails. Example feed: https://www.bart.gov/sites/default/files/docs/google_transit_20211001_20220213_v1.zip
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.9.1/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1779, in _execute_context
self.dialect.do_executemany(
File "/home/travis/virtualenv/python3.9.1/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 962, in do_executemany
context._psycopg2_fetched_rows = xtras.execute_values(
File "/home/travis/virtualenv/python3.9.1/lib/python3.9/site-packages/psycopg2/extras.py", line 1270, in execute_values
cur.execute(b''.join(parts))
psycopg2.errors.ForeignKeyViolation: insert or update on table "routes" violates foreign key constraint "routes_feed_id_fkey"
DETAIL: Key (feed_id, agency_id)=(1, None) is not present in table "agency".
From the spec:
Field Name | Type | Presence | Description |
---|---|---|---|
agency_id | ID referencing agency.agency_id | Conditionally Required | Agency for the specified route.Conditionally Required:- Required if multiple agencies are defined in agency.txt. - Optional otherwise. |
Suggested fix: set the agency_id automatically to the one included agency if no agency_id is defined
Hi @jarondl, would you mind publishing the latest available version to PyPi? It's been quite a while since the last one and there's been some significant changes since. Thanks in advance!
The package should be re-tested with various SQLAlchemy versions and the BART dataset:
http://www.bart.gov/sites/default/files/docs/google_transit_20160328_v1.zip
SQLAlchemy 0.7.8 (the minimum version according to setup.py) crashes.
SQLAlchemy 0.9.0 works.
SQLAlchemy 1.0.6 (latest at time of writing) doesn't work anymore.
I have an issue parsing Tisseo GTFS files. Using home-assistant, here's the error I have:
Traceback (most recent call last):
File "/home/kernald/home-assistant/homeassistant/helpers/entity_component.py", line 91, in _setup_platform
discovery_info)
File "/home/kernald/home-assistant/homeassistant/components/sensor/gtfs.py", line 157, in setup_platform
config["origin"], config["destination"]))
File "/home/kernald/home-assistant/homeassistant/components/sensor/gtfs.py", line 176, in __init__
self.update()
File "/home/kernald/home-assistant/homeassistant/components/sensor/gtfs.py", line 215, in update
self._data_source))
File "/home/kernald/.homeassistant/lib/pygtfs/loader.py", line 87, in append_feed
instance = gtfs_class(feed_id=feed_id, **records_as_dict)
File "<string>", line 4, in __init__
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/state.py", line 306, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/home/kernald/.homeassistant/lib/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/kernald/.homeassistant/lib/sqlalchemy/util/compat.py", line 184, in reraise
raise value
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/state.py", line 303, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/home/kernald/.homeassistant/lib/sqlalchemy/ext/declarative/base.py", line 649, in _declarative_constructor
setattr(self, k, kwargs[k])
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 224, in __set__
instance_dict(instance), value, None)
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 701, in set
value, old, initiator)
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 710, in fire_replace_event
self._init_append_or_replace_token())
File "/home/kernald/.homeassistant/lib/sqlalchemy/orm/util.py", line 118, in set_
return validator(state.obj(), key, value)
File "/home/kernald/.homeassistant/lib/pygtfs/gtfs_entities.py", line 57, in in_range
raise ValueError("Empty value not allowed in {0}".format(key))
ValueError: Empty value not allowed in wheelchair_boarding
Relevant home-assistant ticket is here: home-assistant/core#1761
Hi @elad661,
The changes you have made in #17 break Python 2 support. Can you have a look at the build errors https://travis-ci.org/jarondl/pygtfs/jobs/118716436 ?
If you don't have time to debug this, I'll try to solve it when I have time.
Yaron
Traceback (most recent call last):
File "/home/mdminhaz/pycharm-community-2022.3.2/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
coro = func()
File "<input>", line 1, in <module>
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib/python3.10/site-packages/pygtfs/loader.py", line 87, in append_feed
instance = gtfs_class(feed_id=feed_id, **record._asdict())
File "<string>", line 4, in __init__
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/state.py", line 575, in _initialize_instance
with util.safe_reraise():
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/state.py", line 573, in _initialize_instance
manager.original_init(*mixed[1:], **kwargs)
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/decl_base.py", line 2076, in _declarative_constructor
setattr(self, k, kwargs[k])
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 528, in __set__
self.impl.set(
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1261, in set
value = self.fire_replace_event(
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1276, in fire_replace_event
value = fn(
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/events.py", line 2590, in wrap
return fn(target, *arg)
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib64/python3.10/site-packages/sqlalchemy/orm/util.py", line 322, in set_
return validator(state.obj(), key, value)
File "/home/mdminhaz/PycharmProjects/gtfs-practice/venv/lib/python3.10/site-packages/pygtfs/gtfs_entities.py", line 86, in is_float_none
return float(value)
TypeError: float() argument must be a string or a real number, not 'NoneType'
Some GTFS providers seem to have started using extensions to the format. For example, Toronto's GO Transit adds one more value to the list of valid location_type's.
Extensions list is here:
https://support.google.com/transitpartners/answer/2450962?hl=en
Fix for this specific issue is:
gtfs_entities.py: line 166, change list from [None,0,1] to [None,0,1,2]
I added some additional info to the _validate_int_choice function as well, to assist with debugging:
line 60:
assert int_value in int_choice, "%s value outside limits: %r not in %s" % (key,int_value,int_choice[0:100])
At this point, there's little reason to maintain backward compatibility with Python 2.
Moving forward, we should take advantage of some things that came with Python 3.
Could you publish this package on Pypi, so it can be fetched without specifying the github URL?
Hi, I've been trying to get GTFS working via the homeassistant component which uses pygtfs. I'm trying to use Prague's GTFS feed.
I'm on
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
It doesn't have a lot of options, I just gave it the following parameters:
I also tried the human readable names for the stops and tried to unzip the jrdata and give it the folder name instead.. No dice.. Any ideas where the problem might be?
018-12-03 00:13:09 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up platform gtfs
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 128, in _async_setup_platform
SLOW_SETUP_MAX_WAIT, loop=hass.loop)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
return fut.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/homeassistant/components/sensor/gtfs.py", line 180, in setup_platform
pygtfs.append_feed(gtfs, os.path.join(gtfs_dir, data))
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/pygtfs/loader.py", line 89, in append_feed
instance = gtfs_class(feed_id=feed_id, **records_as_dict)
File "", line 4, in init
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/state.py", line 424, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 66, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/util/compat.py", line 249, in reraise
raise value
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/state.py", line 421, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/ext/declarative/base.py", line 748, in _declarative_constructor
setattr(self, k, kwargs[k])
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/attributes.py", line 229, in set
instance_dict(instance), value, None)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/attributes.py", line 710, in set
value, old, initiator)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/attributes.py", line 717, in fire_replace_event
state, value, previous, initiator or self.replace_token)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/sqlalchemy/orm/util.py", line 136, in set
return validator(state.obj(), key, value)
File "/Users/scstraus/.homeassistant/deps/lib/python/site-packages/pygtfs/gtfs_entities.py", line 66, in in_range
"{0} must be in range {1}, was {2}".format(key, int_choice, value))
pygtfs.exceptions.PygtfsValidationError: route_type must be in range range(0, 8), was 800
Source: https://www.data.gouv.fr/fr/datasets/r/47bc8088-6c72-43ad-a959-a5bbdd1aa14f
Failure while writing FeedInfo(feed_publisher_name='Palmbus', feed_publisher_url='https://www.palmbus.fr', feed_lang='fr', feed_start_date=None, feed_end_date=None, feed_version=None)
feed_info.txt contains indeed no dates.
feed_publisher_name,feed_publisher_url,feed_lang,feed_start_date,feed_end_date,feed_version
Palmbus,https://www.palmbus.fr,fr,,,
I am getting an error reading in the schedule.
I have successfully used this gtfs file for routing and service area creating via ArcMap, and I'd like to use your tool to add new routes to my analysis.
I'm unsure how to address the error.
ipython 5.3.0 py36_0
ipython_genutils 0.2.0 py36_0
import pygtfs
sched = pygtfs.Schedule(":memory:")
pygtfs.append_feed(sched, 'full_greater_sydney_gtfs_static.zip')
The output is:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Agency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Stop'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Route'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Trip'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.StopTime'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Service'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ServiceException'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Fare'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FareRule'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ShapePoint'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Frequency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Transfer'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FeedInfo'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Translation'>:
125 records read for <class 'pygtfs.gtfs_entities.Agency'>.
.......38991 records read for <class 'pygtfs.gtfs_entities.Stop'>.
Failure while writing Routes(route_id='1-11T-6-sj2-2', agency_id='700', route_short_name='11T6', route_long_name='Carlingford, then all stations to Clyde', route_desc='Temporary buses', route_type='714', route_color='00B5EF', route_text_color='FFFFFF')
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-9-5ce656f03730> in <module>()
----> 1 pygtfs.append_feed(sched, 'full_greater_sydney_gtfs_static.zip')
C:\ProgramData\Anaconda3\lib\site-packages\pygtfs\loader.py in append_feed(schedule, feed_filename, strip_fields, chunk_size, agency_id_override)
87
88 try:
---> 89 instance = gtfs_class(feed_id=feed_id, **records_as_dict)
90 except:
91 print("Failure while writing {0}".format(record))
<string> in __init__(self, **kwargs)
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\state.py in _initialize_instance(*mixed, **kwargs)
412 except:
413 with util.safe_reraise():
--> 414 manager.dispatch.init_failure(self, args, kwargs)
415
416 def get_history(self, key, passive):
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\util\langhelpers.py in __exit__(self, type_, value, traceback)
64 self._exc_info = None # remove potential circular references
65 if not self.warn_only:
---> 66 compat.reraise(exc_type, exc_value, exc_tb)
67 else:
68 if not compat.py3k and self._exc_info and self._exc_info[1]:
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py in reraise(tp, value, tb, cause)
185 if value.__traceback__ is not tb:
186 raise value.with_traceback(tb)
--> 187 raise value
188
189 else:
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\state.py in _initialize_instance(*mixed, **kwargs)
409
410 try:
--> 411 return manager.original_init(*mixed[1:], **kwargs)
412 except:
413 with util.safe_reraise():
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\ext\declarative\base.py in _declarative_constructor(self, **kwargs)
652 "%r is an invalid keyword argument for %s" %
653 (k, cls_.__name__))
--> 654 setattr(self, k, kwargs[k])
655 _declarative_constructor.__name__ = '__init__'
656
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\attributes.py in __set__(self, instance, value)
222 def __set__(self, instance, value):
223 self.impl.set(instance_state(instance),
--> 224 instance_dict(instance), value, None)
225
226 def __delete__(self, instance):
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\attributes.py in set(self, state, dict_, value, initiator, passive, check_old, pop)
700 if self.dispatch.set:
701 value = self.fire_replace_event(state, dict_,
--> 702 value, old, initiator)
703 state._modified_event(dict_, self, old)
704 dict_[self.key] = value
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\attributes.py in fire_replace_event(self, state, dict_, value, previous, initiator)
709 state, value, previous,
710 initiator or self._replace_token or
--> 711 self._init_append_or_replace_token())
712 return value
713
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\orm\util.py in set_(state, value, oldvalue, initiator)
116 def set_(state, value, oldvalue, initiator):
117 if include_backrefs or not detect_is_backref(state, initiator):
--> 118 return validator(state.obj(), key, value)
119 else:
120 return value
C:\ProgramData\Anaconda3\lib\site-packages\pygtfs\gtfs_entities.py in in_range(self, key, value)
59 else:
60 int_value = int(value)
---> 61 assert int_value in int_choice, "value outside limits"
62 return int_value
63 return in_range
AssertionError: value outside limits
trying to import a GTFS file to mysql using the following and getting a ton of VARCHAR length unspecified errors from sqlalchemy. what am i doing wrong?
import pygtfs
sched = pygtfs.Schedule('mysql://gtfs:gtfs@localhost:5432/gtfs_njt_rail')
pygtfs.append_feed(sched, "/home/anthony/code/pi_transitsign/data/gtfsdata/njt_rail_data")
here's the output
python2 import_rail_mysql.py
Traceback (most recent call last):
File "import_rail_mysql.py", line 17, in
sched = pygtfs.Schedule('mysql://gtfs:gtfs@localhost:5432/gtfs_njt_rail')
File "/usr/lib/python2.7/site-packages/pygtfs/schedule.py", line 36, in init
Base.metadata.create_all(self.engine)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3695, in create_all
tables=tables)
File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1856, in _run_visitor
conn._run_visitor(visitorcallable, element, *_kwargs)
File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1481, in _run_visitor
*_kwargs).traverse_single(element)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 730, in visit_metadata
_is_metadata_operation=True)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 764, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 962, in _execute_ddl
compiled = ddl.compile(dialect=dialect)
File "", line 1, in
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 494, in compile
return self._compiler(dialect, bind=bind, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 26, in _compiler
return dialect.ddl_compiler(dialect, self, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 190, in init
self.string = self.process(self.statement, *_compile_kwargs)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 213, in process
return obj._compiler_dispatch(self, *_kwargs)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 81, in _compiler_dispatch
return meth(self, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 2170, in visit_create_table
(table.description, column.name, ce.args[0])
File "/usr/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 2159, in visit_create_table
and not first_pk)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 213, in process
return obj._compiler_dispatch(self, *_kwargs)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 81, in compiler_dispatch
return meth(self, **kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 2190, in visit_create_column
first_pk=first_pk
File "/usr/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 1961, in get_column_specification
column.type, type_expression=column)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 261, in process
return type._compiler_dispatch(self, *_kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 81, in _compiler_dispatch
return meth(self, *kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 2594, in visit_unicode
return self.visit_VARCHAR(type, **kw)
File "/usr/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2313, in visit_VARCHAR
self.dialect.name)
sqlalchemy.exc.CompileError: (in table '_feed', column 'feed_name'): VARCHAR requires a length on dialect mysql
Hi,
we're using pygtfs as part of Home Assistant [0.91.2] and have received the following error during setup:
File "/usr/local/lib/python3.7/site-packages/pygtfs/gtfs_entities.py", line 74, in in_range
float_value = float(value)
ValueError: could not convert string to float:
We are using the GTFS data set from VRN:
https://www.vrn.de/mam/service/downloads/vrn_gtfs.zip
and are trying to use a route between the following stops:
origin: "de:07332:1015"
destination: "de:07332:1009:1:Bus"
According to the documentation at https://www.home-assistant.io/components/gtfs/ we are opening this issue here.
Thank you for the work you do,
-Jan
I have a GTFS files with a bunch of different agencies, but I'm only interested in the data for one of them.
it'd be very useful for me if there was a mechanism to skip insertion of routes that don't belong to that specific agencies, and after all the routes I want are in and the rest are out, skip insertion of Trips, StopTimes and Shapes that belongs to routes which are not in the DB.
Is there a way to convert the GTFS feed to a database and cache it to a file, so that every time after that I can load it instantly, rather than doing the GTFS -> database conversion (which takes over a minute) each time?
append_feed
returns UnicodeDecodeError: 'utf-8' codec can't decode byte 0xee in position 107: invalid continuation byte
Hi,
While importing https://stibmivb.opendatasoft.com/api/explore/v2.1/catalog/datasets/gtfs-files-production/alternative_exports/gtfszip/ I got a unique constraint error on the translation.txt import.
The issuer added twice the same trans_id and lang.
I found the faulty lines and could import the data, but at the next update I will get again an error if they don't correct the file.
Is there a way to ignore existing trans_id & lang combination in the same file?
I'm afraid that SQLAlchemy doesn't support "OR IGNORE" in session.add().
2402 records read for <class 'pygtfs.gtfs_entities.Translation'>.
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1942, in _exec_single_context
context,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 919, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: translations.feed_id, translations.trans_id, translations.lang
2402 records read for <class 'pygtfs.gtfs_entities.Translation'>.
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1942, in _exec_single_context
context,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 919, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: translations.feed_id, translations.trans_id, translations.lang
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "sensor.py", line 321, in <module>
init(clean)
File "sensor.py", line 149, in init
attributes = getGTFSAttributes()
File "sensor.py", line 72, in getGTFSAttributes
pygtfs = import_gtfs_files()
File "sensor.py", line 66, in import_gtfs_files
pygtfs.append_feed(gtfs, os.path.join(".", data))
File "/home/pi/.local/lib/python3.7/site-packages/pygtfs/loader.py", line 121, in append_feed
schedule.session.flush()
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 4310, in flush
self._flush(objects)
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 4446, in _flush
transaction.rollback(_capture_exception=True)
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 4406, in _flush
flush_context.execute()
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
rec.execute(self)
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 645, in execute
uow,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 98, in save_obj
insert,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1044, in _emit_insert_statements
statement, multiparams, execution_options=execution_options
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
execution_options or NO_OPTIONS,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 517, in _execute_on_connection
self, distilled_params, execution_options
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1649, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
dialect, context, statement, parameters
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
e, str_statement, effective_parameters, cursor, context
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1942, in _exec_single_context
context,
File "/home/pi/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 919, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: translations.feed_id, translations.trans_id, translations.lang
[SQL: INSERT INTO translations (feed_id, trans_id, lang, translation) VALUES (?, ?, ?, ?)]
[parameters: [(1, 'STIB', 'fr', 'STIB'), (1, 'STIB', 'nl', 'MIVB'), (1, 'http://www.stib-mivb.be', 'fr', 'http://www.stib-mivb.be/index.htm?l=fr'), (1, 'http://www.stib-mivb.be', 'nl', 'http://www.stib-mivb.be/index.htm?l=nl'), (1, 'ABBAYE', 'fr', 'Abbaye'), (1, 'ABBAYE', 'nl', 'Abdij'), (1, 'ACACIAS', 'fr', 'Acacias'), (1, 'ACACIAS', 'nl', 'Acacia') ... displaying 10 of 2402 total bound parameter sets ... (1, 'ICHEC', 'fr', 'ICHEC'), (1, 'ICHEC', 'nl', 'ICHEC')]]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
Thanks
I got that king of errors with the last version of SQalchemy
Warning (from warnings module):
File "C:\Python34\lib\site-packages\sqlalchemy\orm\relationships.py", line 2667
for (pr, fr_) in other_props)
SAWarning: relationship 'Trip.feed' will copy column _feed.feed_id to column trips.feed_id, which conflicts with relationship(s): 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id), 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
It would be nice to have the capability to optionally specify a schema to store the data in for databases that support schemas.
@jarondl Let's add a github action to build and publish new releases to PyPI. It's straightforward, but I assume you set up a github secret to contain the API key to publish there.
It looks like this was previously done via Travis but the link there is broken.
When pygtfs tries to parse the following line:
shape_id,shape_pt_lat,shape_pt_lon,shape_pt_sequence,shape_dist_traveled
149554,38.742155,-9.102203,1,
I get the following error:
Failure while writing Shapes(shape_id='149554', shape_pt_lat='38.742155', shape_pt_lon='-9.102203', shape_pt_sequence='1', shape_dist_traveled=None)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-44756ea8a7d9> in <module>
----> 1 pygtfs.append_feed(sched, "data")
...
~/.virtualenvs/thesis/lib/python3.7/site-packages/pygtfs/gtfs_entities.py in is_float_none(self, key, value)
85 def is_float_none(self, key, value):
86 try:
---> 87 return float(value)
88 except ValueError:
89 if value is None or value == "":
TypeError: float() argument must be a string or a number, not 'NoneType'
Looking at the is_float_none
method, it seems you were expecting float(None)
to raise a ValueError
, but it is raising a TypeError
instead.
Use case: datasources update with a certain frequency, from what I have seen this is anywhere between weekly and annually.
Esp. for the high frequent updates there are cases that the new source is already published before the current data expires and there is no overlap. When performing a nightly automated update one can have a situation where the new datasource can not yet be used as its first data is e.g. tomorrow.
Options (not exhaustive)
I get the following error when running gtfs2db
script
gtfs2db append gtfs.zip postgresql://localhost/gtfs_db
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) there is no unique constraint matching given keys for referenced table "stops"
[SQL: '\nCREATE TABLE translations (\n\tfeed_id INTEGER, \n\ttrans_id VARCHAR NOT NULL, \n\tlang VARCHAR NOT NULL, \n\ttranslation VARCHAR, \n\tPRIMARY KEY (trans_id, lang), \n\tFOREIGN KEY(feed_id, trans_id) REFERENCES stops (feed_id, stop_name), \n\tFOREIGN KEY(feed_id) REFERENCES _feed (feed_id)\n)\n\n']
Any idea what the problem might be?
Hi,
Indirectly I'm trying to get your awesome piece of software to work as it was implemented in Home-Assistant :D I'm trying to add the GTFS data for The Netherlands, however, it seems to have some format issues. Could you confirm? The official data is here: http://gtfs.ovapi.nl/new/gtfs-nl.zip
The following error log pops up in home-assistant:
16-04-13 20:21:49 homeassistant.components.sensor: Error while setting up platform gtfs Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/homeassistant/helpers/entity_component.py", line 91, in _setup_platform discovery_info) File "/usr/local/lib/python3.4/dist-packages/homeassistant/components/sensor/gtfs.py", line 157, in setup_platform config["origin"], config["destination"])) File "/usr/local/lib/python3.4/dist-packages/homeassistant/components/sensor/gtfs.py", line 176, in __init__ self.update() File "/usr/local/lib/python3.4/dist-packages/homeassistant/components/sensor/gtfs.py", line 215, in update self._data_source)) File "/home/rob/.homeassistant/lib/pygtfs/loader.py", line 87, in append_feed instance = gtfs_class(feed_id=feed_id, **records_as_dict) File "<string>", line 4, in __init__ File "/home/rob/.homeassistant/lib/sqlalchemy/orm/state.py", line 306, in _initialize_instance manager.dispatch.init_failure(self, args, kwargs) File "/home/rob/.homeassistant/lib/sqlalchemy/util/langhelpers.py", line 60, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/home/rob/.homeassistant/lib/sqlalchemy/util/compat.py", line 184, in reraise raise value File "/home/rob/.homeassistant/lib/sqlalchemy/orm/state.py", line 303, in _initialize_instance return manager.original_init(*mixed[1:], **kwargs) File "/home/rob/.homeassistant/lib/sqlalchemy/ext/declarative/base.py", line 649, in _declarative_constructor setattr(self, k, kwargs[k]) File "/home/rob/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 224, in __set__ instance_dict(instance), value, None) File "/home/rob/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 701, in set value, old, initiator) File "/home/rob/.homeassistant/lib/sqlalchemy/orm/attributes.py", line 710, in fire_replace_event self._init_append_or_replace_token()) File "/home/rob/.homeassistant/lib/sqlalchemy/orm/util.py", line 118, in set_ return validator(state.obj(), key, value) File "/home/rob/.homeassistant/lib/pygtfs/gtfs_entities.py", line 57, in in_range raise ValueError("Empty value not allowed in {0}".format(key)) ValueError: Empty value not allowed in wheelchair_boarding
Thanks so much! 👍
Rob
I am using the library a lot ..i.e. I have a growing set of end users in Home Assistant
As the data is often not (very) good, the logs help me to identify things quicker.
However, pygtfs is not sending out logs but print statements and these are not shown in the Home Assistant logs, also resulting in not showing errors when extracting.
Can you add a proper log output or is this too complex?
2024-05-09 09:14:13.750 DEBUG (MainThread) [custom_components.gtfs2.config_flow] Checkdata pygtfs: extracting with data: {'extract_from': 'url', 'file': 'vv', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'}
Loading GTFS data for <class 'pygtfs.gtfs_entities.Agency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Stop'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Transfer'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Route'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Fare'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FareRule'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ShapePoint'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Service'>:
...
for gtfs_class in gtfs_all:
print('Loading GTFS data for %s:' % gtfs_class)
gtfs_filename = gtfs_class.__tablename__ + '.txt'
Hello! While trying to import data from BART into Home Assistant using https://www.home-assistant.io/integrations/gtfs/ I ran into a SQLAlchemy error. I have confirmed that this error occurs with the latest pygtfs even when home assistant's code isn't in use:
gtfs2db append bart.zip bart.sqlite
Loading GTFS data for <class 'pygtfs.gtfs_entities.Agency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Stop'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Transfer'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Route'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Fare'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FareRule'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ShapePoint'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Service'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ServiceException'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Trip'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Frequency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.StopTime'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FeedInfo'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Translation'>:
1 record read for <class 'pygtfs.gtfs_entities.Agency'>.
182 records read for <class 'pygtfs.gtfs_entities.Stop'>.
99 records read for <class 'pygtfs.gtfs_entities.Transfer'>.
14 records read for <class 'pygtfs.gtfs_entities.Route'>.
2500 records read for <class 'pygtfs.gtfs_entities.Fare'>.
2500 records read for <class 'pygtfs.gtfs_entities.FareRule'>.
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1750, in _execute_context
self.dialect.do_executemany(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 714, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: transfers.feed_id, transfers.from_stop_id, transfers.to_stop_id
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/bin/gtfs2db", line 8, in <module>
sys.exit(main())
File "/opt/homebrew/lib/python3.9/site-packages/pygtfs/gtfs2db.py", line 54, in main
append_feed(schedule, args['<feed_file>'],
File "/opt/homebrew/lib/python3.9/site-packages/pygtfs/loader.py", line 93, in append_feed
schedule.session.flush()
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3298, in flush
self._flush(objects)
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3438, in _flush
transaction.rollback(_capture_exception=True)
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3398, in _flush
flush_context.execute()
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj
_emit_insert_statements(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1094, in _emit_insert_statements
c = connection._execute_20(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1451, in _execute_clauseelement
ret = self._execute_context(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
self._handle_dbapi_exception(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1994, in _handle_dbapi_exception
util.raise_(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1750, in _execute_context
self.dialect.do_executemany(
File "/opt/homebrew/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 714, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: transfers.feed_id, transfers.from_stop_id, transfers.to_stop_id
[SQL: INSERT INTO transfers (feed_id, from_stop_id, to_stop_id, transfer_type, min_transfer_time) VALUES (?, ?, ?, ?, ?)]
[parameters: ((1, 'MCAR', 'MCAR', 2, '20'), (1, 'MCAR', 'MCAR', 2, '240'), (1, 'MCAR', 'MCAR', 2, '240'), (1, 'MCAR', 'MCAR', 2, '20'), (1, 'MCAR', 'MCAR', 2, '20'), (1, 'MCAR', 'MCAR', 2, '20'), (1, 'MCAR', 'MCAR', 2, '20'), (1, 'MCAR', 'MCAR', 2, '20') ... displaying 10 of 99 total bound parameter sets ... (1, 'MONT', 'MONT', 2, '20'), (1, 'MONT', 'MONT', 2, '20'))]
(Background on this error at: http://sqlalche.me/e/14/gkpj)
relevant package version info:
pip3 list | egrep -i "pygtfs|sqlalchemy"
pygtfs 0.1.6
SQLAlchemy 1.4.20
If you install this module in a virtual environment, and install its dependencies with pip, you can use
pip freeze > requirements.txt
Then, we users can install with
pip install -r requirements.txt
Makes deployment easier.
First time using pygtfs, so apologies in advance if I've not yet caught onto a possibly obvious solution...
Using pygtfs 0.1.5 on Ubuntu 16.04 with SQLite 3.11.0, running gtfs2db append carta-sc-us.zip feed.db
fails with TypeError: DeclarativeMeta object got multiple values for keyword argument 'feed_id'
while writing FeedInfo.
Strange as there's only 1 instance of feed_id
and its value is 213
.
Not quite sure where to go from here. Any advice is greatly appreciated.
➜ cat carta-sc-us/feed_info.txt feed_publisher_url,feed_publisher_name,feed_lang,feed_version,feed_license,feed_contact_email,feed_contact_url,feed_start_date,feed_end_date,feed_id http://www.trilliumtransit.com,"Trillium Solutions, Inc.",en,UTC: 16-May-2019 17:24,,[email protected],http://support.trilliumtransit.com,20190516,20200101,213
➜ gtfs2db append carta-sc-us.zip feed.db Loading GTFS data for <class 'pygtfs.gtfs_entities.Agency'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Stop'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Transfer'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Route'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Fare'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.FareRule'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.ShapePoint'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Service'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.ServiceException'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Trip'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Frequency'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.StopTime'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.FeedInfo'>: Loading GTFS data for <class 'pygtfs.gtfs_entities.Translation'>: 1 record read for <class 'pygtfs.gtfs_entities.Agency'>. 881 records read for <class 'pygtfs.gtfs_entities.Stop'>. 2 records read for <class 'pygtfs.gtfs_entities.Transfer'>. 26 records read for <class 'pygtfs.gtfs_entities.Route'>. 6 records read for <class 'pygtfs.gtfs_entities.Fare'>. 64 records read for <class 'pygtfs.gtfs_entities.FareRule'>. ....44472 records read for <class 'pygtfs.gtfs_entities.ShapePoint'>. 8 records read for <class 'pygtfs.gtfs_entities.Service'>. 53 records read for <class 'pygtfs.gtfs_entities.ServiceException'>. 2381 records read for <class 'pygtfs.gtfs_entities.Trip'>. 2381 records read for <class 'pygtfs.gtfs_entities.Frequency'>. ......60829 records read for <class 'pygtfs.gtfs_entities.StopTime'>. Failure while writing FeedInfo(feed_publisher_url='http://www.trilliumtransit.com', feed_publisher_name='Trillium Solutions, Inc.', feed_lang='en', feed_version='UTC: 16-May-2019 17:24', feed_start_date='20190516', feed_end_date='20200101', feed_id='213') Traceback (most recent call last): File "/usr/local/bin/gtfs2db", line 9, in <module> load_entry_point('pygtfs==0.1.6.dev22+gbbc2691', 'console_scripts', 'gtfs2db')() File "/usr/local/lib/python3.5/dist-packages/pygtfs-0.1.6.dev22+gbbc2691-py3.5.egg/pygtfs/gtfs2db.py", line 55, in main chunk_size=int(args['--chunk-size'])) File "/usr/local/lib/python3.5/dist-packages/pygtfs-0.1.6.dev22+gbbc2691-py3.5.egg/pygtfs/loader.py", line 84, in append_feed instance = gtfs_class(feed_id=feed_id, **record._asdict()) TypeError: DeclarativeMeta object got multiple values for keyword argument 'feed_id'
Sorry if I missed this (and I didn't find a mailing list). Looping through entities is very convenient, but I'd like to specify queries for performance. Trying to make a query did not work as expected:
Stop = pygtfs.gtfs_entities.Stop
stops = schedule.session.query(Stop).all()
[...]
ArgumentError: Column-based expression object expected for argument 'order_by'; got: 'stop_sequence', type <type 'str'>
Is there support for specifying SQLalchemy queries is pygtfs? How do I use it?
Thanks.
Elliot
Hi,
I'm having an issue when using the library with my Python 3.3 setup.
Traceback (most recent call last):
File "/home/antonin/PycharmProjects/transit-heatmap/transit-heatmap.py", line 6, in <module>
pygtfs.append_feed(sched, "data/RATP_GTFS_FULL_25-09-2013.zip")
File "/usr/local/lib/python3.3/dist-packages/pygtfs/loader.py", line 45, in append_feed
fd = feed.Feed(feed_filename, strip_fields)
File "/usr/local/lib/python3.3/dist-packages/pygtfs/feed.py", line 48, in __init__
if six.PY2:
AttributeError: 'module' object has no attribute 'PY2'
When looking at my six module's source, I can see that PY2 is indeed not defined in it, only PY3:
# True if we are running on Python 3.
PY3 = sys.version_info[0] == 3
Maybe this should be checked with getattr
, or inside a try;except?
Hi @jarondl, I'm sure it has been a thankless task to maintain this project for the last 8 years (if I'm reading this right). The repo is really valuable to the community. I came across this project as a dependency of Home Assistant and am very thankful for the work you've put into it.
Seeing as the project seems to be lagging behind the GTFS standard and PRs seem to be taking ~1 month to get reviewed, I think the bus factor for this project is too low.
Are you open to adding new maintainers who can help you out?
Some transit operators add non-standard feed columns (which don't exist in the Google Extensions or in the official specs) to their feeds. for example MBTA (this feed: http://www.mbta.com/uploadedfiles/MBTA_GTFS.zip) adds "route_sort_order" to routes.txt
This causes pygtfs to fail, because it doesn't know what to do with these values.
Failure while writing Routes(route_id='Blue', agency_id='1', route_short_name='', route_long_name='Blue Line', route_desc='Rapid Transit', route_type='1', route_url='', route_color='2F5DA6', route_text_color='FFFFFF', route_sort_order='1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/site-packages/pygtfs-0.1.2-py3.4.egg/pygtfs/loader.py", line 76, in append_feed
File "<string>", line 4, in __init__
File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/state.py", line 306, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/usr/lib64/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python3.4/site-packages/sqlalchemy/util/compat.py", line 184, in reraise
raise value
File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/state.py", line 303, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/usr/lib64/python3.4/site-packages/sqlalchemy/ext/declarative/base.py", line 648, in _declarative_constructor
(k, cls_.__name__))
TypeError: 'route_sort_order' is an invalid keyword argument for Route
It'd be better if these kind of errors will be ignored, so pygtfs will be usable even with not-exactly-standard feeds
According to https://gtfs.org/schedule/reference/#calendar_datestxt it's OK to omit calendar.txt when publishing a GTFS feed:
Alternate: Omit calendar.txt, and specify each date of service in calendar_dates.txt. This allows for considerable service variation and accommodates service without normal weekly schedules. In this case service_id is an ID.
However, pygtfs if it encounters such a feed, it'll fill the calendar
and calendar_dates
SQLite tables in such a way that every date occurs on "no day of the week" - every monday,tuesday,thursday and so on are "0" in the SQLite table. The code responsible for this is here: https://github.com/jarondl/pygtfs/blob/master/pygtfs/loader.py#L97
An example feed that - when loaded into a SQLite database with pygtfs - will cause the described issue is here: https://komunikacja.bialystok.pl/cms/File/download/gtfs/google_transit.zip
(Another one is https://eu.ftp.opendatasoft.com/sncf/gtfs/transilien-gtfs.zip - it has calendar.txt, but many calendar_dates.txt with service exceptions are making plenty of entries in "calendar" table where all days of the week are 0) - taken from home-assistant/core#68614
Example SQLite query:
feed_id | service_id | monday | tuesday | wednesday | thursday | friday | saturday | sunday | start_date | end_date | feed_id | service_id | date | exception_type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | P_852 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-10-27 | 2023-10-27 | 1 | P_852 | 2023-10-27 | 1 |
1 | R_852 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-10-28 | 2023-10-28 | 1 | R_852 | 2023-10-28 | 1 |
1 | S_852 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-10-29 | 2023-10-29 | 1 | S_852 | 2023-10-29 | 1 |
1 | S_862 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-11-01 | 2023-11-01 | 1 | S_862 | 2023-11-01 | 1 |
1 | P_863 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-11-02 | 2023-11-02 | 1 | P_863 | 2023-11-02 | 1 |
1 | R_863 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-11-04 | 2023-11-04 | 1 | R_863 | 2023-11-04 | 1 |
1 | S_863 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-11-05 | 2023-11-05 | 1 | S_863 | 2023-11-05 | 1 |
Would it be possible to just set the day of the week based on the ServiceException date, rather than filling the entity with all 0?
In case you're wondering how I came across this, I noticed how home-assistant/core#68614 - an issue related to the HomeAssistant gtfs widget - https://www.home-assistant.io/integrations/gtfs/ - that displays "when's the closest departure" somewhere on a tablet in your smart home.
The issue is that the sqlite db would get generated properly, but the widget would just always fail to find any departure.
This is because the widget's code relies on the following SQL query to find departures:
SELECT trip.trip_id, trip.route_id,
time(origin_stop_time.arrival_time) AS origin_arrival_time,
time(origin_stop_time.departure_time) AS origin_depart_time,
date(origin_stop_time.departure_time) AS origin_depart_date,
origin_stop_time.drop_off_type AS origin_drop_off_type,
origin_stop_time.pickup_type AS origin_pickup_type,
origin_stop_time.shape_dist_traveled AS origin_dist_traveled,
origin_stop_time.stop_headsign AS origin_stop_headsign,
origin_stop_time.stop_sequence AS origin_stop_sequence,
origin_stop_time.timepoint AS origin_stop_timepoint,
time(destination_stop_time.arrival_time) AS dest_arrival_time,
time(destination_stop_time.departure_time) AS dest_depart_time,
destination_stop_time.drop_off_type AS dest_drop_off_type,
destination_stop_time.pickup_type AS dest_pickup_type,
destination_stop_time.shape_dist_traveled AS dest_dist_traveled,
destination_stop_time.stop_headsign AS dest_stop_headsign,
destination_stop_time.stop_sequence AS dest_stop_sequence,
destination_stop_time.timepoint AS dest_stop_timepoint,
calendar.thursday AS yesterday,
calendar.friday AS today,
calendar.start_date AS start_date,
calendar.end_date AS end_date
FROM trips trip
INNER JOIN calendar calendar
ON trip.service_id = calendar.service_id
INNER JOIN stop_times origin_stop_time
ON trip.trip_id = origin_stop_time.trip_id
INNER JOIN stops start_station
ON origin_stop_time.stop_id = start_station.stop_id
INNER JOIN stop_times destination_stop_time
ON trip.trip_id = destination_stop_time.trip_id
INNER JOIN stops end_station
ON destination_stop_time.stop_id = end_station.stop_id
WHERE (calendar.thursday = 1
OR calendar.friday= 1
)
AND start_station.stop_id = "047"
AND end_station.stop_id = "303"
AND origin_stop_sequence < dest_stop_sequence
AND calendar.start_date <= "2023-11-02"
AND calendar.end_date >= "2023-11-02"
ORDER BY calendar.sunday DESC,
calendar.monday DESC,
origin_stop_time.departure_time
LIMIT 172800
https://github.com/home-assistant/core/blob/dev/homeassistant/components/gtfs/sensor.py#L295-L343
The problematic part is the:
WHERE (calendar.thursday = 1
OR calendar.friday= 1
)
since all days of the week are 0, this condition will never match and always return 0 rows.
I'm getting the following error:
Logger: homeassistant.components.sensor
Source: components/gtfs/sensor.py:511
Integration: Sensor (documentation, issues)
First occurred: 7:28:50 PM (1 occurrences)
Last logged: 7:28:50 PM
Error while setting up gtfs platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 231, in _async_setup_platform
await asyncio.shield(task)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/gtfs/sensor.py", line 511, in setup_platform
pygtfs.append_feed(gtfs, os.path.join(gtfs_dir, data))
File "/usr/local/lib/python3.8/site-packages/pygtfs/loader.py", line 81, in append_feed
instance = gtfs_class(feed_id=feed_id, **record._asdict())
File "<string>", line 4, in __init__
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/state.py", line 445, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/state.py", line 442, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py", line 1145, in _declarative_constructor
setattr(self, k, kwargs[k])
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 458, in __set__
self.impl.set(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1070, in set
value = self.fire_replace_event(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1078, in fire_replace_event
value = fn(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/util.py", line 185, in set_
return validator(state.obj(), key, value)
File "/usr/local/lib/python3.8/site-packages/pygtfs/gtfs_entities.py", line 65, in in_range
raise PygtfsValidationError(
pygtfs.exceptions.PygtfsValidationError: location_type must be in range [None, 0, 1, 2], was 3
because the following GTFS file http://data.pid.cz/PID_GTFS.zip has location_type
set to 3.
Within gtfs_entities.py this line should be updated...
_validate_location = _validate_int_choice([None, 0, 1], 'location_type')
To
_validate_location = _validate_int_choice([None, 0, 1, 2], 'location_type')
The location_type field identifies whether this stop ID represents a stop, station, or station entrance. If no location type is specified, or the location_type is blank, stop IDs are treated as stops. Stations may have different properties from stops when they are represented on a map or used in trip planning. The location type field can have the following values:
0 or blank - Stop. A location where passengers board or disembark from a transit vehicle. | |
1 - Station. A physical structure or area that contains one or more stop. | |
2 - Station Entrance/Exit. A location where passengers can enter or exit a station from the street. The stop entry must also specify a parent_station value referencing the stop ID of the parent station for the entrance.**
when trying to build the database using gtfs2db append <feed_file>
with this zip file as the feed (http://www.gtfs-data-exchange.com/agency/bc-transit-victoria-regional-transit-system/latest.zip)... this happens.
Don't know if you intend to update any of this code, but here's something you might not have known.
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Trip.service' will copy column calendar.feed_id to column trips.feed_id, which conflicts with relationship(s): 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Service.trips' will copy column calendar.feed_id to column trips.feed_id, which conflicts with relationship(s): 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Trip.route' will copy column routes.feed_id to column trips.feed_id, which conflicts with relationship(s): 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Service.trips' (copies calendar.feed_id to trips.feed_id), 'Trip.service' (copies calendar.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Route.trips' will copy column routes.feed_id to column trips.feed_id, which conflicts with relationship(s): 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Service.trips' (copies calendar.feed_id to trips.feed_id), 'Trip.service' (copies calendar.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'StopTime.stop' will copy column stops.feed_id to column stop_times.feed_id, which conflicts with relationship(s): 'Trip.stop_times' (copies trips.feed_id to stop_times.feed_id), 'StopTime.trip' (copies trips.feed_id to stop_times.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Stop.stop_times' will copy column stops.feed_id to column stop_times.feed_id, which conflicts with relationship(s): 'Trip.stop_times' (copies trips.feed_id to stop_times.feed_id), 'StopTime.trip' (copies trips.feed_id to stop_times.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Route.feed' will copy column _feed.feed_id to column routes.feed_id, which conflicts with relationship(s): 'Route.agency' (copies agency.feed_id to routes.feed_id), 'Agency.routes' (copies agency.feed_id to routes.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Feed.routes' will copy column _feed.feed_id to column routes.feed_id, which conflicts with relationship(s): 'Route.agency' (copies agency.feed_id to routes.feed_id), 'Agency.routes' (copies agency.feed_id to routes.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Trip.feed' will copy column _feed.feed_id to column trips.feed_id, which conflicts with relationship(s): 'Service.trips' (copies calendar.feed_id to trips.feed_id), 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Route.trips' (copies routes.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id), 'Trip.route' (copies routes.feed_id to trips.feed_id), 'Trip.service' (copies calendar.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Feed.trips' will copy column _feed.feed_id to column trips.feed_id, which conflicts with relationship(s): 'Service.trips' (copies calendar.feed_id to trips.feed_id), 'ShapePoint.trips' (copies shapes.feed_id to trips.feed_id), 'Route.trips' (copies routes.feed_id to trips.feed_id), 'Trip.shape_points' (copies shapes.feed_id to trips.feed_id), 'Trip.route' (copies routes.feed_id to trips.feed_id), 'Trip.service' (copies calendar.feed_id to trips.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'StopTime.feed' will copy column _feed.feed_id to column stop_times.feed_id, which conflicts with relationship(s): 'Trip.stop_times' (copies trips.feed_id to stop_times.feed_id), 'StopTime.stop' (copies stops.feed_id to stop_times.feed_id), 'Stop.stop_times' (copies stops.feed_id to stop_times.feed_id), 'StopTime.trip' (copies trips.feed_id to stop_times.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Feed.stop_times' will copy column _feed.feed_id to column stop_times.feed_id, which conflicts with relationship(s): 'Trip.stop_times' (copies trips.feed_id to stop_times.feed_id), 'StopTime.stop' (copies stops.feed_id to stop_times.feed_id), 'Stop.stop_times' (copies stops.feed_id to stop_times.feed_id), 'StopTime.trip' (copies trips.feed_id to stop_times.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'FareRule.feed' will copy column _feed.feed_id to column fare_rules.feed_id, which conflicts with relationship(s): 'Route.fare_rules' (copies routes.feed_id to fare_rules.feed_id), 'FareRule.route' (copies routes.feed_id to fare_rules.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Feed.fare_rules' will copy column _feed.feed_id to column fare_rules.feed_id, which conflicts with relationship(s): 'Route.fare_rules' (copies routes.feed_id to fare_rules.feed_id), 'FareRule.route' (copies routes.feed_id to fare_rules.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Frequency.feed' will copy column _feed.feed_id to column frequencies.feed_id, which conflicts with relationship(s): 'Trip.frequencies' (copies trips.feed_id to frequencies.feed_id), 'Frequency.trip' (copies trips.feed_id to frequencies.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
C:\Python27\lib\site-packages\sqlalchemy\orm\relationships.py:2667: SAWarning: relationship 'Feed.frequencies' will copy column _feed.feed_id to column frequencies.feed_id, which conflicts with relationship(s): 'Trip.frequencies' (copies trips.feed_id to frequencies.feed_id), 'Frequency.trip' (copies trips.feed_id to frequencies.feed_id). Consider applying viewonly=True to read-only relationships, or provide a primaryjoin condition marking writable columns with the foreign() annotation.
for (pr, fr_) in other_props)
Traceback (most recent call last):
File "C:\Python27\Scripts\gtfs2db-script.py", line 9, in
load_entry_point('pygtfs==0.1.2', 'console_scripts', 'gtfs2db')()
File "build\bdist.win-amd64\egg\pygtfs\gtfs2db.py", line 54, in main
File "build\bdist.win-amd64\egg\pygtfs\loader.py", line 76, in append_feed
File "", line 4, in **init**
File "C:\Python27\lib\site-packages\sqlalchemy\orm\state.py", line 306, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "C:\Python27\lib\site-packages\sqlalchemy\util\langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\state.py", line 303, in _initialize_instance
return manager.original_init(_mixed[1:], *_kwargs)
File "C:\Python27\lib\site-packages\sqlalchemy\ext\declarative\base.py", line 649, in _declarative_constructor
(k, cls_.**name**))
TypeError: 'stop_short_name' is an invalid keyword argument for Stop
Hi,
In the code of loader.py we have
gtfs_tables = {}
for gtfs_class in gtfs_all:
...
gtfs_tables[gtfs_class] = fd.read_table(gtfs_filename,
set(c.name for c in gtfs_class.__table__.columns) - {'feed_id'})
then a few lines later:
for gtfs_class in gtfs_all:
if gtfs_class not in gtfs_tables:
continue
gtfs_table = gtfs_tables[gtfs_class]
To me (correct me if i'm wrong) :
Thx
Regards
Fabien
based on the list here:
https://support.google.com/transitpartners/answer/3520902
What are your thoughts about enabling modifications to an imported feed? For my purposes, it would be nice to issue a command like "delete(routes, route_id=24)" or "delete(stop_times, route_id=15") and have the relevant information removed. It would also be nice to have an interface for adding information.
If there's a more efficient way to accomplish this without writing custom code within pygtfs I'd love to hear about it.
Would you mind helping me with the error below when I try to look at MTA subway data? The link to the GTFS data is https://datamine-history.s3.amazonaws.com/gtfs-2014-09-17-09-31
IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: shapes.feed_id, shapes.shape_id [SQL: 'INSERT INTO shapes (feed_id, shape_id, shape_pt_lat, shape_pt_lon, shape_pt_sequence, shape_dist_traveled) VALUES (?, ?, ?, ?, ?, ?)'] [parameters: ((1, '4..N06R', 40.668897, -73.932942, '0', None), (1, '4..N06R', 40.669399, -73.942161, '1', None), (1, '4..N06R', 40.669847, -73.950466, '2', None), (1, '4..N06R', 40.669922, -73.951706, '3', None), (1, '4..N06R', 40.670083, -73.954713, '4', None), (1, '4..N06R', 40.670096, -73.954983, '5', None), (1, '4..N06R', 40.670118, -73.955258, '6', None), (1, '4..N06R', 40.670157, -73.955542, '7', None) ... displaying 10 of 5001 total bound parameter sets ... (1, 'L..S01R', 40.683552, -73.905577, '260', None), (1, 'L..S01R', 40.683429, -73.905534, '261', None))]
Hi,
I am remodelling the gtfs solution in HomeAssistant and have a use case with a large file from the NL, the sqlite turns into 7Gb.
As this dataset does not contain calendar entries, I need to rewrite the query to compensate for that and since sqlite does not allow outer joins I need to run it twice with an UNION ALL. Due to the large amount of data the query is pretty slow (db browser : 20-23 sec) and I was wondering if I could optimize this wqith indexes. This I will do myself but 2 questions:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.