asiripanich / emdash Goto Github PK

As part of the auto-labeling functionality, we added the set of automatically inferred labels to each confirmed trip object. These labels are a JSON dict [{labels: {mode_confirm: X, replace_confirm: Y}, p: <probability>}], similar to

[{'labels': {'mode_confirm': 'bike',
   'purpose_confirm': 'exercise',
   'replaced_mode': 'shared_ride'},
  'p': 0.5},
 {'labels': {'mode_confirm': 'bike',
   'purpose_confirm': 'home',
   'replaced_mode': 'shared_ride'},
  'p': 0.5}]

These show up automatically in the data table, but because this is a JSON array, they show up as [(object Object), (object Object)] (see screenshot). Would it be possible to JSON.serialize this field so the deployer can get a sense of what the trip might be even if the user hasn't labeled it? I think @allenmichael099 had an example of applying a javascript transform to a column (for the bike checkinout table).

• Change the "unconfirmed" table column header to "unlabeled"
• Change the other table columns headers to also be meaningful (@shankari add meaningful values)
• Larger font size for the variables in the data plot
• make the display of the sign-in trend a configurable option that is passed in to the emdash::runapp function (similar to the current anon_locations)
• the list of columns in the participant table would be configurable somewhere as an option
  • for our particular use case, we will probably want to hide the first_XXX since we only care about the last_XXX

Using Firebase push nofication. See https://github.com/e-mission/e-mission-server/tree/master/emission/net/ext_service/push.

We thought this was related to my change to map the mode to the z-axis, but @allenmichael099 said that it happens even when the user_id is mapped to the z-axis (as on master).

It is very confusing because you think that the colors are right, but they are wrong.

@asiripanich since you originally implemented the map, maybe you can give us some pointers?

When I use emdash-test-db data with load_trips_start_ndays set to 7,
then within emdash select 2015-07-22 or 2015-07-23 as the first date,
then click 'Load trips data', the trip lines do not display.

Trip lines do display in these cases:

• after clicking 'Load trips data' a second time.
• after 'continuing' when I place a browser() statement in app_server within observeEvent(filtered_trips()$data_filtered(), { before callModule

Especially when the trip data is loading.

Hi @asiripanich congrats for this tools !

I could install without problem the docker image,
FYI I wanted also to install the R package, it worked because I needed to install a lot of linux packages (libudunits2-dev libfontconfig1-dev libssl-dev libcairo2-dev libv8-dev libssl-dev libsasl2-dev plus upgrading postgresql from 12 to 13 because of a version problem with libpq) on my ubuntu 20.04 machine before I could install emdash in R.

Then I loaded the test data you suggested (for the moment I have no e-mission server running, next week maybe), and it works :-) At least the dashboard, and maybe the tables/plots (I didn't try yet).

But the maps don't seem to work.
I get this error :
Couldn't normalize path inaddResourcePath, with arguments: prefix= 'PopupTable-0.0.1';directoryPath= ''
Any idea why ?

#32 (comment)

Currently, emdash downloads all the data each time it is loaded or reloaded. This is inefficient and costly, since cloud providers (e.g., AWS) usually charge for sending data. Here is a solution:

• Only query records created after the most recent refresh.
• Automatic refresh interval can be set in run_app() or as a parameter in the config file.

The program managers plan to use the dashboard to view travel patterns. But if they need to follow up with individual people, they need to know who they are, and the UUID doesn't give that information. Displaying the "user_name" (a random token in our case) allows them to make that linkage.

This should be displayed at least in the participant table, but may be added to the trip table as well for greater ease of use.

@asiripanich I have another question for the table/interactive plot screen,
the table displayed at the page bottom seems fine, however I don't understand how to display a graph, I tried for instance simply to put user_ID in X and n_Trips in Y but nothing is displayed (even if I click on "play").

Originally posted by @PatGendre in #20 (comment)

After #13 by @ahcyip, I now see some trips in the map view which have N/A for all values except the user_id. For example, the trips from the Bay Area to Hawaii (see screenshot below).

That seems wrong. If we look at the trip table for example, there are no entries with N/A

For example, the trip from the Bay Area to Hawaii is row 9 since the start timezone is PDT and the end timezone is HST.
The trip from the Bay Area to Hawaii doesn't show up here because I had my phone off, but is hidden between 38 and 39.

And entries 9, 38 and 39 all have all entries present, including duration, distance and start and end times. Why does this show up as N/A in the map?

• trip table
• participant table

This can be as simple as adding in a 'download' button that changes with the selected table

The current docker build on master fails.

It gets past all of the initial steps correctly,

 => [ 2/10] RUN apt-get update && apt-get install -y default-jre-headless gdal-bin git-core imagemagic  72.9s
 => [ 3/10] RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'li  0.3s
 => [ 4/10] RUN R -e 'install.packages("remotes")'                                                       5.6s
 => [ 5/10] RUN R -e 'install.packages("devtools")'                                                    561.0s
 => [ 6/10] RUN R -e 'remotes::install_github("r-lib/remotes", ref = "97bbf81")'                         6.3s
 => [ 7/10] RUN mkdir /build_zone                                                                        0.3s
 => [ 8/10] ADD . /build_zone                                                                            0.5s
 => [ 9/10] WORKDIR /build_zone                                                                          0.0s

but fails on the last step due to some download errors?

 => ERROR [10/10] RUN R -e 'remotes::install_local(upgrade="never")'                                    71.7s

#14 71.61 Error in download.file(url, destfile, method, mode = "wb", ...) :
#14 71.61   (converted from warning) downloaded length 288991 != reported length 581347
#14 71.61 Error in download.packages(pkgs, destdir = tmpd, available = available,  :
#14 71.61   (converted from warning) download of package ‘e1071’ failed
#14 71.61 Calls: <Anonymous> ... with_envvar -> force -> force -> i.p -> download.packages
#14 71.66 Execution halted

 => ERROR [10/10] RUN R -e 'remotes::install_local(upgrade="never")'                                    71.7s
...
#14 0.436 > remotes::install_local(upgrade="never")
#14 3.671 Installing 109 packages: attempt, backports, base64enc, bit, bit64, bslib, cellranger, checkmate, classInt, colorspace, config, cpp11, credentials, crosstalk, curl, data.table, datamods, DBI, diffobj, dockerfiler, dplyr, DT, e1071, esquisse, farver, flexdashboard, forcats, generics, geojsonsf, geometries, gert, ggplot2, gh, golem, gridExtra, gtable, haven, here, hms, htmltools, htmlwidgets, httpuv, httr, isoband, janitor, jquerylib, jsonify, labeling, later, lazyeval, leafem, leaflet, leaflet.providers, leafpop, lubridate, mapview, mongolite, munsell, openxlsx, plotly, plyr, png, progress, promises, proxy, randomcoloR, rapidjsonr, raster, RColorBrewer, Rcpp, readr, readxl, rematch, rio, rmarkdown, roxygen2, Rtsne, s2, sass, satellite, scales, servr, sf, sfheaders, shiny, shinycssloaders, shinydashboard, shinyWidgets, snakecase, sourcetools, sp, svglite, systemfonts, testthat, tidyr, tidyselect, tinytex, tzdb, units, usethis, uuid, V8, viridis, viridisLite, vroom, waldo, webshot, wk, xtable
#14 3.673 Installing packages into ‘/usr/local/lib/R/site-library’
#14 3.673 (as ‘lib’ is unspecified)
#14 4.464 trying URL 'https://cran.rstudio.com/src/contrib/attempt_0.3.1.tar.gz'
#14 4.871 Content type 'application/x-gzip' length 79261 bytes (77 KB)
#14 4.871 ==================================================
#14 4.897 downloaded 77 KB
#14 4.897
#14 4.899 trying URL 'https://cran.rstudio.com/src/contrib/backports_1.2.1.tar.gz'
#14 5.034 Content type 'application/x-gzip' length 21184 bytes (20 KB)
#14 5.034 ==================================================
#14 5.036 downloaded 20 KB
...
#14 10.60 ==================================================
#14 10.60 downloaded 10 KB
#14 10.60
#14 10.60 trying URL 'https://cran.rstudio.com/src/contrib/dplyr_1.0.7.tar.gz'
#14 10.91 Content type 'application/x-gzip' length 852103 bytes (832 KB)
#14 10.92 ==================================================
#14 11.06 downloaded 832 KB
#14 11.06
#14 11.06 trying URL 'https://cran.rstudio.com/src/contrib/DT_0.18.tar.gz'
#14 11.36 Content type 'application/x-gzip' length 1578749 bytes (1.5 MB)
#14 11.36 ==================================================
#14 11.56 downloaded 1.5 MB
#14 11.56
#14 11.57 trying URL 'https://cran.rstudio.com/src/contrib/e1071_1.7-8.tar.gz'
#14 11.78 Content type 'application/x-gzip' length 581347 bytes (567 KB)
#14 11.78 ========================
#14 71.61 downloaded 282 KB
#14 71.61
#14 71.61 Error in download.file(url, destfile, method, mode = "wb", ...) :
#14 71.61   (converted from warning) downloaded length 288991 != reported length 581347
#14 71.61 Error in download.packages(pkgs, destdir = tmpd, available = available,  :
#14 71.61   (converted from warning) download of package ‘e1071’ failed
#14 71.61 Calls: <Anonymous> ... with_envvar -> force -> force -> i.p -> download.packages
#14 71.66 Execution halted

As highlighted in #71, we currently hardcode the column for inferred_labels

This is because the line to find the column

does not take hidden columns into account.

inferred_label_index is set to 17, when it should actually be 5

Is there a way we can find the index after accounting for the hidden labels?

Right now, there are pipeline stages that combine the user input with the cleaned trip objects to create confirmed trips.
A sample confirmed trip object from the master/ceo_ebike_program branch looks like

{'_id': ObjectId('606b8bf0c77a1ff9e630f422'),
 'user_id': UUID('d4376620-fbcd-4aab-95bf-8c2e0ecf9adf'),
 'metadata': {'key': 'analysis/confirmed_trip',
 'platform': 'server',
 'write_ts': 1617660912.6729634,
 'time_zone': 'America/Los_Angeles',
 'write_local_dt': {'year': 2021, 'month': 4, 'day': 5, 'hour': 15, 'minute': 15, 'second': 12, 'weekday': 0, 'timezone': 'America/Los_Angeles'},
 'write_fmt_time': '2021-04-05T15:15:12.672963-07:00'},
 'data': {'source': 'DwellSegmentationTimeFilter',
 'end_ts': 1617659216.0,
 'end_local_dt': {'year': 2021, 'month': 4, 'day': 5, 'hour': 14, 'minute': 46, 'second': 56, 'weekday': 0, 'timezone': 'America/Los_Angeles'},
 'end_fmt_time': '2021-04-05T14:46:56-07:00',
 'end_loc': {'type': 'Point', 'coordinates': [-122.0867274, 37.3911479]},
 'raw_trip': ObjectId('606b8bf0c77a1ff9e630f3e7'),
 'start_ts': 1617658219.0,
 'start_local_dt': {'year': 2021, 'month': 4, 'day': 5, 'hour': 14, 'minute': 30, 'second': 19, 'weekday': 0, 'timezone': 'America/Los_Angeles'},
 'start_fmt_time': '2021-04-05T14:30:19-07:00',
 'start_loc': {'type': 'Point', 'coordinates': [-122.0870928, 37.390054]},
 'duration': 997.0,
 'distance': 2458.7832149780197,
 'start_place': ObjectId('606b8bf0c77a1ff9e630f41a'),
 'end_place': ObjectId('606b8bf0c77a1ff9e630f41b'),
 'cleaned_trip': ObjectId('606b8bf0c77a1ff9e630f3f1'),
 'user_input': {'mode_confirm': 'bike', 'purpose_confirm': 'pick_drop', 'replaced_mode': 'drove_alone'}}}

After #23, which includes setnames(gsub("user_input", "", names(.))) %>%, the mode_confirm, purpose_confirm and replaced_mode entries automagically show up in the trip table.

But now I want to have a column in the participant table with the number of unlabeled trips.
For now, I have implemented (in my fork alone)

.[is.na(mode_confirm), .(unconfirmed = .N), by = user_id]

shankari@735a93b#diff-4e87ef70cdab2756b9d4aa419fd97ffea60f2ec8d9592573ffee2e4ab33dcf53R162-R165

But not everybody is going to use the mode_confirm object, in particular, the travel survey folks will want to potentially have a more complex object.

In python, I do ct_df_confirmed = ct_df[ct_df.user_input != {}], which does not use any field names, but I don't know how to implement a similar check in R.

@asiripanich you asked me to file an issue with the details, and I did 😄

Since we merged #8, emdash has no customised parts that only work on RCITI branches. We should also update this on the README file.

I'm getting these warnings running the current version of emdash:

Warning: 'esquisserUI' is deprecated.
Use 'esquisse_ui' instead.
See help("Deprecated") and help("esquisse-deprecated").
Warning: 'filterDF_UI / filterDF' is deprecated.
Use 'datamods::filter_data_ui / datamods::filter_data_server' instead.
See help("Deprecated") and help("esquisse-deprecated").

Reported by @shankari.

We should profile our app and see where the real issue is. The easiest way to fix this is to run generate_trajectories() in the background process, so that the UI doesn't freeze.

Causes the map not to display on the dashboard, as encountered by #20 and #13 (comment).

Found today while demoing the dashboard to program administrators, see screenshots:

Participant tab, participant labels on top	Trip tab, also participant labels on top

Quick fix for #29.

Use shinymanager or shinyauthr.

The dashboard still has many components, mainly those that responsible for querying data, specific to my own e-mission-phone branches (rciti1 to rciti4).

To generalise this it would involve knowing the database structure of the master phone branch.

Multiple program managers requested this today. We just need to merge it into the trip table, and it will automatically work for the trip table, the map and the map popups as well.

Is there a way to make changes to the dashboard in RStudio without having to reinstall the package each time a change is made? I cloned emdash on my computer, and each time I want to test out my new changes, I use something like this:

detach("package:emdash", unload = TRUE)
remotes::install_local(path = 'C:/Users/mallen2/emdash',upgrade="never",force = TRUE)
library(emdash)
emdash::run_app(mongo_url = 'mongodb://localhost:27017')

Concrete example:

• flag participants who are not contributing data correctly and give hints on why

Example rules:

• Apple phone, no app communication, no data upload: Most likely have the “always” permission turned off and have “closed”/force-killed the app
• Android 11, recent app communication, but no data upload: Most likely have "always" permission turned off
• Both OSes, no recent app communication: token missing or not configured properly?
• Samsung phone acting weirdly
  • Android < 11, communication, but no data: likely aggressive power savings https://dontkillmyapp.com/

If there are no query results, then many functions break. For the most recent change (#17), these were:

normalise_uuid
summarise_server_calls
tidy_server_calls

A workaround is to add the following short-circuit code to the beginning of each function.

+  if (nrow(.data) == 0) {
+    return(.data);
+  }

The query results can return no values early in the data collection; when there have been no trips yet, for example, or the user has not synced with the server yet.

I am not an R expert, so was curious about how you plan to handle the case in which there are no query results. Instead of adding short-circuit code to each function, is there a way to create a blank data table at the query level?

from time to time, there may be additional ad-hoc tables that need to be displayed in the dashboard.

These are not critical to the core trip tracking functionality, but for additional services that might induce participants to install the app and provide data. They typically represent app state (e.g. the current ranking in the leaderboard; the current size of the bear). As such, they are not part of the core functionality, and we don't care too much about including them into a progressive system or making them be idempotent.

It seems like we should be able to easily create separate tables for them.

Since data:table automatically displays the retrieved data, we should be able to actually do this with very little code and in a generic fashion. For example, we could have an entry in the config that looks like this:

- supplementary tables
   - Checkinout
     tab name: "Bike Check in"

which could create a new tab called Bike Check in populated with data from the Checkinout collection.

we could even make the new table editable
https://www.r-bloggers.com/2018/11/editable-datatables-for-shiny-applications/

and allow people to specify the operations that they want to enable

- supplementary tables
   - Checkinout
     tab name: "Bike Check in"
     editable: D

Valid values for editable could be from CRUD (https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) or at least CUD since R is the default

@asiripanich I took a look at the CI output since I want to start with a working version and make sure that I don't break anything.

It looks like windows and OSX pass, but linux fails because

* checking package dependencies ... ERROR
Error: Packages required but not available:
  'mapview', 'mongolite', 'randomcoloR', 'sf'

I took a look at the mapview package, and it looks like it has binaries only for windows and OSX
https://cloud.r-project.org/web/packages/mapview/index.html

Windows binaries: | r-devel: mapview_2.9.0.zip, r-release: mapview_2.9.0.zip, r-oldrel: mapview_2.9.0.zip
macOS binaries: | r-release: mapview_2.9.0.tgz, r-oldrel: mapview_2.9.0.tgz

Could that be the reason why?