wri-cities / payanam Goto Github PK

Ensure that when changes are made in table like:

• stop rename,
• stop re-map,
• stop de-map,
• stop remove,
• stop move

... then the map updates automatically to reflect the changes made. That includes stops and the lines connecting them.

Way of doing this : Define follow-on actions in event listeners for all the mentioned actions.

Standard code for updating stops and lines:

setTimeout(function() {
			mapStops(); 
			if(lineLayer.getLayers().length && map.hasLayer(lineLayer)) {
				map.removeLayer(lineLayer);
				routeLines();
			}
		},500);

there were unnecessary wait-periods before doing this or that. Legacy code from back when this was a standalone app. Removed and streamlined.

We can type in a name and add a stop while route mapping. Previously it was always inserted at the top of the table, then the user had to drag it down to its place.

Now, if another stop is already selected, then the add stop action will insert the new stop just below the selected one. And it will inherit the direction_id of the selected one.

There was an error happening on running Bulk Suggest function.

Traced it to under function routeSuggestFunc in functions.py : if databank df was pre-loaded, then the if condition to check for whether it's loaded or not was failing. Changing that to specifically check for length resolved the error.

Found a much easier-to-implement and fully accurate way of positioning the crosshair exactly at the map center. From https://gis.stackexchange.com/a/90230/44746

Adding services numbers input in Timings page. Make service numbers display in print page and in the table on home page.

There were two things happening if we clicked on an existing stop on the map : The new location (red dot) was getting set there; and if there was any stop there it was being added to our to-reconcile selection as well.

The last part caused unnecessary hassle.

Sample use case: I want to move 4 scattered stop locations to one pre-existing location. I've used the lasso tool to select those 4. Now I want to simply tell the program that their new location should be same as this 5th stop. So I click that. But I don't want to change the route that this 5th stop belongs to - it's already there!

It didn't even make sense from a "want to select it" point of view, because if suppose there were 10 pre-existing stop at the same location, then clicking on it is not going to select all 10. It'll only select the one that has been rendered on top. To select all those 10, we need the lasso tool anyways.

So, disabling that extra click-selecting on map. Now onwards, if you want to select something on the map, the only way is through the lasso tool. We can still click anywhere on the map to set the new location (red dot)

The Bulk Suggestions / Automapping feature is a powerful if time-taking one. At present it's one-or-all. Put in a middle level : run it at a depot level only; leave all the other routes under other depots untouched.

• To simplify the interface for target users, remove many of the extra columns from the routes table and keep only the most relevant columns visible
• Below the table, List options (checkboxes) to enable / disable viewing the extra columns.

This would risk data loss (per-stop timings, stop id's, trip variations, calendar variations), but this feature may be appealing for folks who have a gtfs that's not complicated to begin with and they need to get the data into payanam anyways.

Could start with a separate script like gtfs_creation.py .

anonymous usage stats using Matomo

The stats.json created by reports_creation script cumulatively logs the stats as of every 8 hours under a "history" node. Did some coding to output those historical stats into a CSV as well. This CSV is much easier to plot on a chart etc.

Using Leaflet Hash plugin to make the map's current lat, long, zoom level appear in the URL. Now you can copy the URL and share with a colleague and they will get to see exactly what and where you were. Also, the last parts of the URL can be copied and pasted in another page, ex: from route-mapping page to stops-reconcilliation, and so we can easily jump to the exact location.

Example: https://server.nikhilvj.co.in/payanam/routeMap.html?route=MHRM/530.json#14/17.0548/78.3577

when scanning all routes, drop the auto-suggestions data. It's big, doesn't get used in the reports, and unnecessarily occupies too much memory.

On python side, the config/config.json file was being loaded as a normal dict. That was causing the ordering of keys withing the dict to vary by operating environment. That was fine in the beginning.

But now the databanks loading has been parameterized and their info is stored in config.json as so:

"databanksList": {
        "Databank": "databank/stops-databank.csv",
        "Mapped Stops": "reports/stops_mapped.csv",
        "Automapped Stops": "reports/stops_automapped.csv",
        "Post Offices": "databank/postoffice.csv",
        "Hamlets": "databank/hamlets-hyderabad-region.csv"
    },

(this is my payanam setup.. the on-github instance has just some dummy data pre-loaded)
In the auto-suggest / auto-mapping feature of Payanam, the program now references the config Dict and takes the first databank listed up there as the databank.
Else the multiple databanks are for the front-end side, RouteMapper page (see lower table)

I noticed in production env it was loading the hamlets databank instead of the default one, that's when it hit that the program was loading config/config.json as an unordered Dict and the sequence of keys in this can change. So, loading this as OrderedDict now.

from collections import OrderedDict  # just mentioning here for others looking for how to do this
...
configRules = json.load(open(os.path.join(configFolder,configFile)), object_pairs_hook=OrderedDict)

Like the map and timings icons, add an icon to edit the route in Data-Entry page.

Why : That's the page where we can edit the metadata : route name, bus type etc. Also it's handy for large-scale changes like major re-writing of stops sequence.

To reduce initial visual clutter, the routes overview table won't show many of the columns. But to get to see them if wanted, options with check-boxes are there on the page at bottom. Checking them on or off toggles those columns.

Stops Reconciliation : Allow REVIEW level access for unmapped stops mapping

Default access level needed is ADMIN. That makes sense as there is high risk when assigning a single stop name+location to stops spread across several routes.

The unmapped mode serves a different purpose : to map those stops for the first time. So that can be allowed to mapping interns having REVIEW level access.

Changes that are involved:

• pass an additional "mode" argument to the reconcile API call.
• at python api-handler end, determine the required level of access from the value of this mode argument.
• if mode = unmapped and api-key's access level is "REVIEW" or above, then allow it.
• but for remaining modes (mapped and all for now), reject if access level is below ADMIN.

Use route name instead of json filename when listing / displaying routes

Some routes turn out to be circular or otherwise uni-directional. We should be able to erase off everything on the return journey side. That wasn't happening.

Found a safety mechanism that was causing the bug: if the incoming data block is blank then don't do any change. Again this was legacy code from earlier times. Now with proper backups happening at every change, we need not worry. So, removed that condition-check and allowed blank data to translate to the stops block becoming a blank array.

Make map controls so that user can proceed to next and previous stops from the map itself, without having to click on the table. This will speed up review.

Have to make it work when in fullscreen mode also, so moving all map overlays to custom map controls.

This had to be done at both input and output stages.

Processing data-entry submissions:

row['stop_lat'] = ''.join([c for c in stopParts[1] if c in '1234567890.-'])

Then, in reports_creation script, similar safeguards put in to drop the lat-long value if it's not a proper float number.

                    try:
                        lat = float(stopRow.get('stop_lat',0))
                        lon = float(stopRow.get('stop_lon',0))
                        if lat and lon:
                            points.append([ lat,lon ])
                    except ValueError as e:
                        pass

On python side. This optimizes the performance.
For checking access level and verifying user, two functions are used : userInfo(key) and checkAccess(key, desired).

We were loading the config/access.csv file as a normal df and then querying it by the key to find corresponding user name, email, access level.
But if the target is just one row, then it makes more sense to set the key column as index while reading the file, and then reference the desired values by df.at[key,'column']. It also skips having to create a new df from the querying.

The top table on RouteMapper page that lays out the route :

column for direction_id : instead of a free typing editor as filter, have a dropdown select where user can choose between ALL, UP or DOWN.

Ref: http://tabulator.info/docs/4.2/edit#edit-builtin

new feature 📢 that comes into its own from v1.4.0 onwards

From home page, click a route's print icon - you will be taken to the print page with that route pre-loaded.
Click "Print" and the browser's regular print dialog will appear.

In Chrome / Chromium browser, choose the "Save as PDF" option, and ensure that "background graphics" option is checked on.

Save it, and now you have a PDF routemap ready to print!

Click "Options" in top right box for configuring the print.

• Similar to the map icon to jump to mapping a route, put a clock icon to jump to entering timings for a route
• To display whether timings have been entered for a route or not, show a column with just a green tick mark
• Do the same for frequency

Earlier behaviour : If you click a stop to map it, then the suggested locations appear as blue dots on the map. You can click a "toggle" link at the end of the suggestions listed below the main table to toggle them off.

Problem : Now that we're relying more on the mapped stops databank and the network is quite dense, they are becoming more of a distraction.

Solution : Let the suggestions layer stay off on map if it was wasn't already turned on. User can click the toggle link to turn them on if they want.

Mission: To make it easy to copy-paste in a column of timings from an excel

Some folks write timings as "3.40" instead of "3:40". Fine, let's handle it.

Using regular expressions to validate timings.

Check if string is valid time hh:mm (24h):

/^(2[0-3]|[0-1]?[\d]):[0-5][0-9]$/.test(clean1)

works from 0:00 to 23:59

made a stackoverflow answer related to this here: https://stackoverflow.com/a/56311086/4355695

In the home page (routes overview) and stops reconciliation page, there are features to load a quick preview of just the onward direction of a selected route, as a polyline on the map.

So far we've been using a getRouteLine API call to the backend. At backend, the corresponding function reads the route's json, extracts all the valid lat-longs into a simple array like so:

[ [0,1], [0,2], [1,1], [1,2] ]

and returns that array to front-end, which renders it into a polyline.

The flow is supposed to be straightforward and quick, but when using this on our production deployment, I found that:

• There was quite some latency in getting the job done
• The server would get occupied with many API calls; and this "light" preview ended up being used a lot and very often, thus clogging up the backend server's pipeline.

We don't actually need the python backend to do the job. It does it with lesser code, but even Javascript can directly load the route json as we have been doing for all route-editing, and render the lat-longs.

So I've re-written those functions now (drawLine()) to do direct json loading and not bother the backend API. Result is a much faster experience on the front-end : the "quick" previews are truly loading very quickly.

• At onward and return journey block headers, show the names of first and last stops.
• Give jump links to mapping and data-entry pages for the same route.