auth0-extensions / auth0-user-import-export-extension

Extension that allows you to import/export users from your Auth0 account

Auth0 user import and export extension

This extension makes it possible to import and export users from Auth0.

Running Locally

• Be sure you have ngrok installed.
• Start ngrok on port 3000 ngrok http 3000 and take note of your ngrok HTTP URL
• Create a Client in your account with read:connections, create:users, read:users and create:passwords_checking_job access to the Auth0 Management API.
• Create a config.json file under ./server/ containing the following settings:

{
  "EXTENSION_SECRET": "any-random-value-will-do",
  "AUTH0_DOMAIN": "YOUR_DOMAIN",
  "AUTH0_CLIENT_ID": "YOUR_CLIENT_ID",
  "AUTH0_CLIENT_SECRET": "YOUR_CLIENT_SECRET",
  "WT_URL": "http://YOUR_NGROK",
  "AUTH0_RTA": "https://auth0.auth0.com"
}

• Install dependencies and run the extension under node 8:

nvm use 8
yarn install
npm run serve:dev

• To view the extension visit your ngrok URL (note that you are using the extension outside of Auth0 dashboard).
• If you get errors regarding nonce or token, try logging out of Auth0, clearing site data for the extension, logging back into Auth0, and then re-visiting the extension.

Building for Production

npm run build

Adding or Updating Dependencies

yarn add "some-dep@some-version" --ignore-engines

Release Process

Deployment is currently done using this tool: https://auth0-extensions.us8.webtask.io/extensions-deploy

First bump the version in package.json and in webtask.json

Then build the extension:

nvm use 8
yarn install
npm run build

Bundle file (auth0-user-import-export.extension.VERSION.js) is found in /dist Asset files are also found in /dist

Follow the instructions in the deployment tool. This tool will also automatically generate a PR in the auth0-extensions repo. Only after the PR is merged will the extension be available in production. Before merging the PR you can use this tool to test the upgrade: https://github.com/auth0-extensions/auth0-extension-update-tester by overriding the extensions.json file that is fetched by the dashboard.

auth0-user-import-export-extension's Issues

Composed field

Hi,
I need in my export a composed field derived from the concatenation of 2 field like for example:

// user model
{
...
    "user_metadata": {
        "first_name": "First name",
        "last_name": "Last name",
    }
}

I'd like to have in my csv a field full_name calculated chaining first_name and last_name.

Is there some way to do this with the current version?

Cheers,
Mirko

[export] Bug when clicking "Export" button while lucene query field is focused

The clicking of the button blurs the lucene query field, which triggers the query to re-run, which confuses the export tool which now displays "exporting undefined users" (where undefined should be the user count).

Unable to import user "*******@noemail.com" Forbidden attribute in appUser:

I'm trying to use this extension to upload new users to my auth0 tenant. I am see this issue when importing

Unable to import user "[email protected]":
Forbidden attribute in appUser:

It does not clear say which attribute is not allowed. I have used this https://auth0.com/docs/users/import-and-export-users/bulk-user-import-database-schema-and-examples guide to create schema for my json file.

My sample json file:

[{"blocked":false,"email":"@noemail.com","email_verified":true,"family_name":"Test","given_name": "Test","name":"Test","nickname":"Test","username":"9052316526","password_hash":"$2b***","custom_password_hash":{"algorithm":"sha512","hash":{"value":"*******************","encoding": "hex"}}}]

[Request] Make zip exports compatible with Mac's Archive Utility

Our helpdesk has downloaded an export file in a gz format. These files cannot be decompressed using Mac's Archive Utility.

We have to either teach them how to use gzip, get another program on their machines or do ping-pong with engineers to unzip them. It'd be good if the files were compatible with Archive Utility.

Exports in ndjson format, but not indicated in GUI

I'm a DSE and I'm still seeing tickets about this issue (#45). Even though it's in the docs (as seen in the last response), would it be more clear if it were indicated on the extensions page? It says "Export Format: JSON file (*.json)" but it could mention that exports are in ndjson format.

Multiple customers have tried to export and then import immediately to another tenant.

Let me know if I should continue on issue #45's thread instead.

Export is limited to 100k users

How can we export all users from a tenant if they have more than 100k users?

JSON export is not valid json

The json exported when selecting json format is not valid json as there is no enclosing array and items are not comma separated. This makes it a pain for onward processing out of system as have to go in and add the commas and enclosing braces in a text editor

{"Id":"sdgfdsg"}
{"Id":"sdgfdsg"}

should be

[
  {"Id":"sdgfdsg"},
  {"Id":"sdgfdsg"}
]

Better error handling when a connection is disabled

Steps to reproduce:

1. Create a new connection in the dashboard
2. Import a json file to the extension and pick the new connection
3. This should fail because the connection is not enabled for any client:

Oh snap! You got an error!

An error occured while uploading the file: Request failed with status code 400

4. This error is unclear. In the network trace you'll see "errorCode": "connection_is_disabled". This is the real error

So instead of showing "Request failed with status code 400" we should show the following message:

Cannot import users to a connection that is not enabled. Please (enable)(https://manage.auth0.com/#/connections/database/CONNECTION_ID_HERE/clients) at least one connection for NAME_OF_THE_CONNECTION

TSV file format

Stretching my memory here, but I believe "new line" conventions for Windows systems is \r\n, for Unices \n, while the exported TSV uses \r which seems to be at odds with any standard format.

Cannot import, exported JSON.

The exported JSON is not valid JSON.

Firstly, This is whats exported. It's not valid JSON. Im not going point out why. (values are intentionally blank)
{"Id":"","Given Name":"","Family Name":"","Nickname":""} {"Id":"","Given Name":"","Family Name":"","Nickname":""}

You cannot import it. Complains about not valid JSON, After its valid. You get an email with total users processed, all failed.

[export] UX issues

While I'm here...

The lucene search feature wasn't very intuitive. It wasn't clear whether the search was triggered automatically, or when I did manage to trigger it, it wasn't clear exactly what triggered it. My best guess as a web developer is that it is searching on 'change' or 'blur', but if that's the case, there should be a button to trigger it, since naturally you would want to hit enter or click something to tell it to "go", if auto-search is too expensive. It's unnatural to click away from the thing you're trying to use in order to get it to do something. In fact, my coworker is asking me for instructions on how to use this UI, even though I'm pretty sure he's used it before, and he's a developer, so he should be able to figure it out without instructions... I think?

Payload too large error for file sizes < 10MB

Getting a 413 Payload too large error when importing a file containing users with <10MB size (i.e.: 2MB, 5MB, etc...).

Error:

Error: request entity too large
   at makeError (/data/_verquire/body-parser/1.12.4/node_modules/raw-body/index.js:154:15)
   at readStream (/data/_verquire/body-parser/1.12.4/node_modules/raw-body/index.js:188:15)
   at getRawBody (/data/_verquire/body-parser/1.12.4/node_modules/raw-body/index.js:95:12)
   at read (/data/_verquire/body-parser/1.12.4/node_modules/body-parser/lib/read.js:64:3)
   at jsonParser (/data/_verquire/body-parser/1.12.4/node_modules/body-parser/lib/types/json.js:116:5)
   at Layer.handle [as handle_request] (/data/_verquire/express/4.12.4/node_modules/express/lib/router/layer.js:82:5)
   at trim_prefix (/data/_verquire/express/4.12.4/node_modules/express/lib/router/index.js:302:13)
   at /data/_verquire/express/4.12.4/node_modules/express/lib/router/index.js:270:7
   at Function.proto.process_params (/data/_verquire/express/4.12.4/node_modules/express/lib/router/index.js:321:12)
   at next (/data/_verquire/express/4.12.4/node_modules/express/lib/router/index.js:261:10)

Support CSV imports

Allow users to upload CSV files instead of JSON files. The structure should be as follows:

email,username,app_metadata.age
[email protected],john,20

The header should contain the exact names of the fields. After uploading the CSV, we can then convert the file to a JSON file and upload it to API v2

Export Errors are not presented on the UX

A customer was trying to add a sort field but they added user.user_id instead of just user_id. The API responded with a nice error message, but it was hidden in the developer tools. It would be nice if that error were presented on the UX instead. This also would let users know that there was an error and not just a stalled export.

When users are imported we need to see the reason of the failed imports.

Can we display the reason why some users are not imported in the extension UI.

Page freezes when trying to upload two files with the same name

Steps to reproduce (only tested on Firefox):

1. Upload file a.json, suppose it fails
2. Make a change to file a.json trying to correct the error, preserving the filename
3. Try to upload the updated file

The "Import Status" column goes blank, and trying to remove any of the entries causes the page to freeze:

Support for `upsert`

/api/v2/jobs/users-imports supports upsert to "Update the user if already exists". The extension should provide an affordance for the same functionality.

Accessing user.user_metadata[something] returning undefined

Hello!

Thanks for this tool, it is incredibly useful. I'm exporting and trying to access user.user_metadata.whatever and getting an undefined back. However I know that user.user_metadata is present because it outputs [Object object]. Why is this different from app_metadata? How do I get to user_metadata?

Shows total of users when doing the export.

If I have a total of 9 users and the query that I inserted in the search bar queries 5 users, this screenshot happens.

Export will only let me export 1,000 users

The export functionality is only letting me choose the option 'EXPORT 1000 USERS', even though I have over 10,000 users on the tenant.

Is there are a reason this is set at 1,000; is there a way that I can increase it?

Support CSV exports

Would be useful if the extension supported export to CSV.