Goal
The goal is to reduce all import steps (path-prepare, ppr-prepare, parking-prepare, osrm-extract, osrm-contract, etc.) into one single command line call like motis --mode import [+some dataset paths] or motis --init "/import".
One idea would be to use the existing module system, especially the publish/subscribe functionality to accomplish this task. Tasks activate them self when all required input events were triggered. Every task produces an output event specifying what was produces (e.g. the path of the schedule.raw file). A module waiting for multiple input events has to store the output (paths) of those events it already received until it can start.
| Module |
Required Input |
Task |
Output |
Init |
| loader |
{ GTFS } |
gtfs_parser |
schedule.raw |
✅ |
| loader |
{ HRD } |
hrd_parser |
schedule.raw |
✅ |
| osrm |
{ OSM, PROFILE } |
osrm_extract |
PROFILE-graph |
✅ |
| osrm |
{ OSM, PROFILE } |
osrm_contract |
PROFILE-contracted-graph |
✅ |
| path |
{ OSM, bus-contracted-graph, schedule.raw } |
path-prepare |
path-database |
|
| ppr |
{ OSM } |
ppr-prepare |
ppr-graph |
✅ |
| parking |
{ OSM, ppr-graph } |
parking-prepare |
parking-db |
✅ |
| ... |
... |
... |
... |
... |
Configuration
Import settings could be configured within the modules using parameters that make it clear that this is an import-parameter (e.g. parking.import.variable). But: since most modules have only parameters for their "Required Input" (see table column), there will not be many parameters.
Every "Required Input" as well as "Output" is an event publish()ed in the module system.
Initial Events
Initial events published are OSM, GTFS or HRD, and OSRM_PROFILE. There could be different solutions:
- Hardcode those parameters and manually trigger the inital input events
- Make this generic, too and just propagate
FILE_PATH as initial event (attached with e.g. ./data/switzerland.osm.pbf, car.lua, or ./data/gtfs). Every module can register an import operation that discovers whether the filepath is a OSM dataset, GTFS dataset, HRD dataset, etc. If the discovery step identified an OSM dataset, it will trigger the OSM event with the corresponding path and therefore automatically trigger the import steps that require the OSM dataset. To debug why the import did not trigger, we also need a --import.debug=true flag that prints the missing files. Otherwise, it could be hard to find out what's wrong.
The latter solution is more generic.
Progress Documentation
Every process should document its progress using the following event message:
table ImportProgress {
task_name: string;
finished: bool;
progress: int8; // percent [0, 100]
}
The first update always has to be { finished=false, progress=0 } to start showing the progress bar. This should be done before starting the task just to indicate (to the user) that the task is running. The progress bar will be removed if an update has finshed set to true.
To give the user an overview of which modules are already loaded and ready, which modules are in-progress and which modules are waiting for dependencies, every module can update its state (i.e. using the ImportProgress or alternatively a separate structure with enum ModuleImportState { LOADING, WAITING, FINISHED };. The output could look like this:
schedule: finished
osrm: loading
ppr: loading
parking-prepare: waiting for "ppr"
[============> ] 48% osrm-contract
[===> ] 14% ppr
The console status is continuously updated as new progress and module state updates come in. One solution to display multiple progress bars is demonstrated in the p-ranav/indicators library.
Missing Features of the Module System
Update: not needed because all tasks will be executed with one thread
Currently, the module system is not capable of handling RAM requirements. The maximal parallelism can be set using the num_threads parameter but this does not prevent Motis from spawning two tasks that do not fit into RAM together. Maybe, it makes sense to use the same technique that's also used in the ctx::access_schedule class to prevent concurrent (read and write) or (write and write):
- when a task is ready - check if it can be started using a wrapping lambda
- If not ready: put it back in to a "on hold queue"
- when a task finishes: check the "on hold queue" and re-schedule all operations that were on hold, waiting to be executed (this is also done in the wrapping lambda after the actual operation has finished)
Thus, the wrapping lamda would look like:
[&] {
if (ram_usage() > current_op<Data>()->ram_requirement_) {
queue_.emplace_back(current_op<Data>()->shared_from_this());
current_op()->suspend();
// retry...
}
fn();
if (!queue_.empty()) {
unqueue(queue_);
}
}This has to work together with the access scheduler. Maybe it needs to be integrated into the access scheduler?
Update from discussions with @sfahnens and @pablohoch below
Task Execution Conditions
It would be possible to detect that a task has finished its work by checking if the produced database file is there. However, the file might also be there if the task crashed. In this case, the task should be repeated. Unfortunately, the task would not be repeated because the (incomplete) file of the (incomplete/crashed) run prevents this.
To indicate that a task is finished it should put a separate "finished"-file into the filesystem after everything is done. This file should contain everything that is required to check whether the task needs to be repeated because either the settings or the input files (OSM/HRD/GTFS/...) have changed. Since this information comes with the required input events, the file contents may essentially be all input event messages converted to JSON.
So the task execution condition would be:
- either, there is no finished file. So there is no produced output file (e.g. database) or if there is an output file it is probably incomplete or the user deleted the finished file on purpose to trigger the import task to run again
- or there is a finished file:
- if the contents of the finished file are the same as those in the input events that triggered this task to run, there is nothing to do and the output event should be triggered without further ado.
- if the contents of the finished file do not match the input events that triggered this task to run, the task needs to be executed again
The input events for OSM-based tasks should contain a hash of the first 100MB of the OSM file and the OSM filesize to enable checking whether the OSM file has changed. Replacing the OSM file should force all OSM-based import tasks to be executed.
Event Collector
Basically, every import task that has more than one required input needs to wait for several input events and remember all of the message contents. To prevent code duplication, a even_collector class should be introduced:
event_collector ec;
ec.listen("osm", MessageContent_ImportOSM);
ec.listen("schedule", MessageContent_ImportSchedule);
if (import_use_srtm_) { // parameter configured in module settings
ec.listen("srtm", MessageContent_ImportSRTM);
}
ec.task([](std::map<std::string, msg_ptr> const& dependencies) {
std::cout << dependencies.at("osm")->to_json() << std::endl;
std::cout << dependencies.at("schedule")->to_json() << std::endl;
if (import_use_srtm_) {
std::cout << dependencies.at("srtm")->to_json() << std::endl;
}
// TODO: Check finished file.
// TODO: Start import if required.
});
// subscribes to all events listed above
// and executes the task after the last event has arrived
ec.subscribe(registry);Integration into MOTIS
The most favorable solution is to include the import steps into MOTIS.
for (auto const& m : modules) m.init();
publish("/import", /* thread_count= */ 1);
publish("/init");If everything is already preprocessed, the "/import" step should be finished within <1s because it only checks all "finished" files (see above) and sees that everything is already imported. If anything has changed (e.g. the OSM file was replaced), all import tasks that depend on the changed input (and recursively those that depend on steps that depend on changed input) files will be executed again.
Detection of Bad Configurations
There can be bad configurations. One example would be that PPR waits for SRTM data and OSM data but the SRTM dataset was not given as input file. Therefore, the PPR import task would never run. This should be easy to detect if the PPR import task is marked as "required" but did not run after the event queue has been cleared. Thus, we need a step that checks whether all "required" tasks were executed. If a required task was not executed, this could be a bad configuration.
Additionally, the output of the task state ("waiting/loading/finished") should print which events it is waiting for. Therefore, the user can quickly see which input was not fulfilled.
Event Chain Example
This example of triggered events should illustrate the process described above:
FILE_EVENT("input.osm.pbf") (triggered because the MOTIS import configuration listed this file)- task: osm_detection (listening for
FILE_EVENT)
OSM("input.osm.pbf", size, hash of first 100MB) (triggered from osm_detection because the file name matched the OSM file pattern)- all steps that require OSM
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent