JSON enrollment data importer (v1) about evap

[janno42]

• Ambiguity in emails: We should probably filter emails through clean_email to have INSTITUTION_EMAIL_REPLACEMENTS applied?

yes, this logic needs to be applied here as well

• I could see conflicts with Update user data on login #2096 if SSO and CMS have different names stored. Maybe we should close it (depending on when we expect the CMS to go up)

closed

• Removal of Evaluations / Courses: Those will simply not be contained anymore in the JSON, so we would just delete all courses/evaluations of the import semester whose cms_id is not contained in the json file?

yes, but only if they have a cms_id set (so we won't delete evaluations that have been created manually)

• The logged attempted changes for state >= APPROVED should probably be grouped by course/evaluation.

yes, please

• We treat their gguid values as blobs that must bit-match to be identical? Then, for us, 0xf and 0xF would be two different values. Especially asking because I'd expect them to send RFC4122-compliant UUIDs where capitalization doesn't matter. In the worst case, we would delete a course and recreate it with a differently capitalized cms_id.

we can assume them to be standard-compliant case-insensitive UUIDs for now

• In general: We expect high-quality data (e.g. trimmed attributes). We trust the CMS to not mess up our data and we do not perform sanity checks.

yes (for now)

• An import run could possibly delete (almost) all courses/evaluations of the semester during the preparation phase.

do you mean if for some reason the CMS doesn't export them correctly anymore? courses/evaluations are only added to the JSON file when there are enrollments for them. so there should be almost no cases where they are manually deleted from that point on. then let's say for all deletions we only notify by email and expect the managers to do the deletion manually.

[richardebeling]

Thoughts / Questions / Ideas

• Ambiguity in emails: We should probably filter emails through clean_email to have INSTITUTION_EMAIL_REPLACEMENTS applied?

• Currently only implied, maybe specify more explicit: We update UserProfiles of contributors and students with the values from the json file.

  • I could see conflicts with #2096 if SSO and CMS have different names stored. Maybe we should close it (depending on when we expect the CMS to go up)

• Removal of Evaluations / Courses: Those will simply not be contained anymore in the JSON, so we would just delete all courses/evaluations of the import semester whose cms_id is not contained in the json file?

• The logged attempted changes for state >= APPROVED should probably be grouped by course/evaluation.

• We treat their gguid values as blobs that must bit-match to be identical? Then, for us, 0xf and 0xF would be two different values. Especially asking because I'd expect them to send RFC4122-compliant UUIDs where capitalization doesn't matter. In the worst case, we would delete a course and recreate it with a differently capitalized cms_id.

• In general: We expect high-quality data (e.g. trimmed attributes). We trust the CMS to not mess up our data and we do not perform sanity checks.

  • The logged actions would allow to manually undo most damage except for deletions where data will be gone irrecoverably.
  • An import run could possibly delete (almost) all courses/evaluations of the semester during the preparation phase. We accept that? Possible counter-measures could be:
    • Require manual user confirmation if some modification/deletion threshold is reached
    • Make a backup of the courses/evaluations/contributions of the semester before the import