rahulpsd18 / cognito-backup-restore

AIO Tool for backing up and restoring AWS Cognito User Pools

cognito-backup-restore

AIO Tool for backing up and restoring AWS Cognito User Pools

Amazon Cognito is awesome, but has its own set of limitations. Currently there is no backup option provided in case we need to take backup of users (to move to another service) or restore them to new Userpool.

cognito-backup-restore tries to overcome this problem by providing a way to backup users from cognito pool(s) to json file and vice-versa.

Please Note: There is no way of getting passwords of the users in cognito, so you may need to ask them to make use of ForgotPassword to recover their account.

Requirements

Requires node 6.10 or newer

Installation

cognito-backup-restore is available as a package on npm.

npm install -g cognito-backup-restore

Usage

cognito-backup-restore can be used by importing it directly or via CLI (recommended).

Imports

Make sure you have installed it locally npm install --save cognito-backup-restore. Typings are available and included.

import * as AWS from 'aws-sdk';
import {backupUsers, restoreUsers} from 'cognito-backup-restore';

const cognitoISP = new AWS.CognitoIdentityServiceProvider();

// you may use async-await too
backupUsers(cognitoISP, <USERPOOL-ID>, <directory>)
  .then(() => console.log(`Backup completed`))
  .catch(console.error)

restoreUsers(cognitoISP, <USERPOOL-ID>, <JSON-File>, <Password?>)
  .then(() => console.log(`Restore completed`))
  .catch(console.error)

This is useful incase you want to write your own wrapper or script instead of using CLI.

CLI

Run cognito-backup-restore or cbr to use it. Make use of -h for help.

cbr <command> [options]

Available options are:

--region -r: The region to use. Overrides config/env settings

--userpool --pool: The Cognito pool to use. Possible value of all is allowed in case of backup.

--profile -p: Use a specific profile from the credential file. Key and Secret can be passed instead (see below).

--aws-access-key --key: The AWS Access Key to use. Not to be passed when using --profile.

--aws-secret-key --secret: The AWS Secret Key to use. Not to be passed when using --profile.

--delay: delay in millis between alternate users batch(60) backup, to avoid rate limit error.

--use-env-vars: Use AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional) as environment variables

--use-ec2-metadata: Use credentials received from the metadata service on an EC2 instance

• Backup

  cbr backup
  cbr backup <options>

  --directory option is available to export json data to.

  

• Restore

  cbr restore
  cbr restore <options>

  --file option is available to read the json file to import from.

  --pwd option is available to set TemporaryPassword of the users. If not provided, cognito generated password will be used and email will be sent to the users with One Time Password.

  --pwdModule option is available to make use of custom logic to generate password. If not provided, cognito generated password will be used and email will be sent to the users with One Time Password, unless --pwd is used. Make sure to pass absolute path of the file. Refer this.

  

In case any of the required option is missing, a interactive command line user interface kicks in to select from.

Todo

• Fine tune the backup process
• Implement Restore
• Write detailed Readme with examples
• Convert JSON to CSV
• Implement Amazon Cognito User Pool Import Job
• AWS Cross-Region Cognito Replication

Contributors

Thanks goes to these wonderful people (emoji key):

adityamedhe-cc 📖 💻	juno-visualsquares 💻 🤔	Charlie Brown 🐛	Alvaro Del Valle 💬	Vlad Korolev 💻	ashish kumar 📖 💻	ufoo68 💻
steveizzle 📖 💻	M. Holger 💻

This project follows the all-contributors specification. Contributions of any kind welcome!

cognito-backup-restore's Issues

Allow me to suppress emails and SMS

I'm just trying to do a backup and restore to a dummy database for testing purposes and I can't have it sending emails to live users (which has already happened).

Also since it's using Cognito's own email system (i don't have SES) it fails after 50. Even after I have disabled "automatically verify users" in the UserPool.

According to the docs, setting MessageAction to "SUPPRESS" will suppress the welcome message.

Looking for collaborators!

Hi guys!

I am unable to look out and maintain this project anymore (barely doing it anyway since 2020😞)
I am looking for contributors and maintainers to keep this project alive.
Please let me know if you want to be added as a collaborator here and on the npm package.

Thanks!

cc: @jstarmx @jfstephe @mholger

External Provider/Social Sign In Accounts

It looks like accounts added from external oath sources (Google, Facebook, etc) won't keep that status on restore. As a test, I backed some of those users up, restored them, and then tried to sign in with that same account. Instead of forcing a password reset for that user, a new user was signed-up and added to the pool.

Allow login with Multi Factor Auth (MFA) on the command line

I am not able to use the command line tool because my account assumes a role and uses MFA.

It is possible to allow MFA like so

const token = '123456'
const profile = 'profile'
  const credentials = new AWS.SharedIniFileCredentials({
    profile,
    tokenCodeFn: (mfaSerial, done) => done(null, token),
  })
  AWS.config.credentials = credentials

No option to completely ommit password creation

What if I just want to restore a backup from the prod account to the dev pool and don't need to send everyone a notification with the new password? I just want to import the result of the backup command.
This should be obvious when working with this software that notifications will be sent and only need to be done when someone explicitly sets this to true, otherwise it is not clear and may confuse users when they actually have nothing to worry about.

Is there a way to customise the email?

Is there a way to specify what the contents of the email are? I was getting a generic message which can be confusing to the users as they won't know the password for what has been reset. I'd like to add more information.

High-priority vulnerabilities in the dependencies

Hi there,

It looks like npm audit is flagging inquirer-file-path as having a high-severity vulnerability in its own dependency (lodash).

export backup to S3

I don't know about the others, but I would like to use this backup solution with a scheduled lambda function. It would be sweet to have the possibility to export to a S3 bucket. And then it would also make sense to allow restoring from an export in a S3 bucket.

Missing Cognito Groups and User Group Membership

Cognito stores groups and user membership information which is critical for associating users with Roles and therefore with shared permissions. Even if the roles fell outside of the scope of Cognito User Pools (being in Identity pools, not user pools), having the group information and membership would be an important part of the functionality of Cognito that should be retained through backup, restore, and migration.

restoreUsers function not importing data to userpool

const data = await restoreUsers(cognitoProvider, 'asasasasasasa', './backups/data1.json','demooooo!@!!12121212')
I am using above code snippet to restore users to pool.Backup is done successfully but not able to restore to specific pool id.
I am using same region to import and export the users. restoreUsers function returning 200 success code but not importing data to user pool.

Not all users imported successfully and exception thrown.

There are 312 users in the exported .json file but only a handful imported successfully with this exception thrown in the console:

$ cbr restore -f 'users.json' -p <profile> --pool <pool> -r <region>
✔ Users imported successfully to <poll>

/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52
  resp.error = util.error(new Error(), error);
                          ^

LimitExceededException: Exceeded daily email limit for the operation or the account. If a higher limit is required, please configure your user pool to use your own Amazon SES configuration for sending email.
    at Request.extractError (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'LimitExceededException',
  time: 2022-09-07T02:29:15.790Z,
  requestId: '220b9b98-bc64-437b-8ba4-10df913201ac',
  statusCode: 400,
  retryable: false,
  retryDelay: 64.88455069958468
}

Node.js v18.8.0

self signed certicate in chain

Hi,
I have this "issue".
The firewall in my company inspect the SSL traffic, so in AWS CLI and other tools I have configured to use the firewall certificate (that is a self signed).
When running the tool I receive the "error": "* self signed certificate in chain" and nothing happens....I can't do the backup.

How can I solve this ?
Thanks
Dario

Issue with DOS line endings

Hi there!

I installed [email protected] using yarn global add

success Installed "[email protected]" with binaries:
      - cbr
      - cognito-backup-restore
✨  Done in 14.36s.

However, when running the executable cbr, I was greeted with this error:

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/c/n/.bin> cbr
env: node\r: No such file or directory

To fix this, I simply ran dos2unix on the executable:

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/.bin> dos2unix ../cognito-backup-restore/build/cli/cli.js
dos2unix: converting file ../cognito-backup-restore/build/cli/cli.js to Unix format...

And now it runs fine!

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/.bin> cbr
? Choose the mode (Use arrow keys)
❯ Backup
  Restore

default to region from current environment settings

when the region is set in the environment - via profile, configuration, or environment variables - it would be useful to default to those settings

AWS Version

Your AWS version uses caret "aws-sdk": "^2.259.1", and the latest version of the SDK no longer has the shared_ini.js file.

I suggest updating to the latest SDK or changing your package.json to use only that specific version of the SDK.

Import the same sub ids

After creating the backup, how can we import the same sub ids when we restore to a new pool? It generates new sub ids when you restore.

Can't restore a google account

I am doing some migrations. All was good until I hit a google_xxx account. I haven't been able to locate the error. Is it possible? Links to documentation would be great.

Note: I am raising it here because these value came out of a backup and then I tried to restore and can imagine other people having this problem. In practice what happens is that the non external accounts (ie cognito) restore but it is very hard to notice this.

My assumption is that either we need to strip values of external accounts on restore or setup the pool differently. The former seems more probable because I don't/can't change my pool without deleting it.

UPDATE: Looking at linking cognito user to federated accounts it would seem that it is not surprising we can't out of the box.

General Settings > Attributes

• has email as required.

Federation > Google > Attribute mapping

• email
• email_verified
• name
• given_name
• family_name
• sub (--> username)

Here's my script, sample json from a backup and the error message.

➜  cognito-migrate node ./node_modules/.bin/cbr restore \
>      --file to_add.json \
>     -r xxxx \
>     -p default \
>     --pool XXXXXXX
✔ Users imported successfully to XXXXXXX

[
  {
    "Username": "google_100051227669222736914",
    "Attributes": [
      {
        "Name": "sub",
        "Value": "93085109-16ac-4997-9680-0b61dbd4771d"
      },
      {
        "Name": "identities",
        "Value": "[{\"userId\":\"100051227669222736914\",\"providerName\":\"Google\",\"providerType\":\"Google\",\"issuer\":null,\"primary\":true,\"dateCreated\":1609832494628}]"
      },
      {
        "Name": "email_verified",
        "Value": "true"
      },
      {
        "Name": "name",
        "Value": "J T"
      },
      {
        "Name": "given_name",
        "Value": "J"
      },
      {
        "Name": "family_name",
        "Value": "T"
      },
      {
        "Name": "email",
        "Value": "[email protected]"
      }
    ],
    "UserCreateDate": "2021-01-05T07:41:34.635Z",
    "UserLastModifiedDate": "2021-01-05T07:41:34.635Z",
    "Enabled": true,
    "UserStatus": "EXTERNAL_PROVIDER"
  }
]

cognito-migrate node ./node_modules/.bin/cbr restore \
>      --file to_add.json \
>     -r xxxx \
>     -p default \
>     --pool xxxxxx
✔ Users imported successfully to xxxxxx

cognito-migrate/node_modules/aws-sdk/lib/protocol/json.js:52
  resp.error = util.error(new Error(), error);
                          ^

InvalidParameterException: Cannot modify the non-mutable attribute identities
    at Request.extractError (cognito-migrate/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (cognito-migrate/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (cognito-migrate/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (cognito-migrate/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at cognito-migrate/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (cognito-migrate/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (cognito-migrate/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'InvalidParameterException',
  time: 2021-06-18T06:40:27.805Z,
  requestId: '90f09f62-7b7e-412f-a3c4-eb6296e2f727',
  statusCode: 400,
  retryable: false,
  retryDelay: 64.47731523692524

Getting this error upon cognito backup restore

/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52
resp.error = util.error(new Error(), error);
^

InvalidParameterException: User pool does not have SMS configuration to send messages.
at Request.extractError (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'InvalidParameterException',
time: 2022-08-12T17:17:28.468Z,
requestId: 'b0d0cee0-e499-46a6-9afd-542f0f10bc04',
statusCode: 400,
retryable: false,
retryDelay: 40.703989640319094
}

Rate Limit exceeded error

While taking the backup from Aws cognito user pool, if the users are more than 10,000. it's giving rate limit exedeed error.
can you look into that and let use know when can you resolve that?
I Can also help if you need any help

connect ETIMEDOUT *****:443

Getting timeout while execute the command, please assist.

cbr backup --pool ***** -r us-east-1 --dir ./cognito_emg_backup

connect ETIMEDOUT *****:443

Doesn't run on MacOS

There is a bug in src/cli/cli.ts (line 1) that causes the following error on MacOS:
env: node\r: No such file or directory

I think the CRLF line ending is causing it, but perhaps the shebang line could be removed altogether as it is not widely used.

No email provided but desired delivery medium was Email

When restoring a users pool got the following error:

UnhandledPromiseRejectionWarning: InvalidParameterException: No email provided but desired delivery medium was Email
...
(node:61020) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:61020) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Both pools configured exactly the same with terraform creation.

Actually it seems doing the job one by one user and issue the error right after restoring it. It would be nice to have at least a "continue on error" option if some users are buggy

Missing phone_number and phone_number_verified fields

Looking to do a backup and restore of a user pool. Unfortunately we require the {"Name":"phone_number","Value":"[phone]"} and {"Name":"phone_number_verified","Value":"true"} fields. While I was able to manually add them to the resulting json, it would be great to have them added automatically to the export.

Thanks!

User data isn't completely flushed to disk when `backupUsers` resolves

Within my backup process, I'm calling the backupUsers process and expect that the file is complete when the promise returned from that function resolves, however the data I can immediately pull from disk is not complete (the stream has not finished writing all of the data to the file).

I see here (src/index.ts#L61) that there is no promise wrapping the callback pattern used for the stream, so there's no way to wrap this function call to ensure that the stream has finished writing all of the data to disk.

The following example:

const cbr = require('cognito-backup-restore');
const aws = require('aws-sdk');
const fs = require('fs');
const userPoolId = 'us-east-1_abc123';
const cognitoIsp = new aws.CognitoIdentityServiceProvider();
cbr.backupUsers(cognitoIsp, userPoolId, '/tmp').then(() => {
  JSON.parse(fs.readFileSync(`/tmp/${userPoolId}.json`).toString())
});

will yield the error:

(node:13604) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected end of JSON input

Because the final bracket (]) has not been written to disk

Please support Cognito authentication using roles

It seems you only accept a key/password IAM credential. Using a role is a best practice. In the aws config file this is specified as follows (example from aws cli documentation):
[profile marketingadmin]
role_arn = arn:aws:iam::123456789012:role/marketingadminrole
credential_source = Ec2InstanceMetadata