50fifty / auth-scheduled-backup

Firebase Extension to schedule backups for users in Firebase Authentication.

auth-scheduled-backup

Firebase Extension for any Firebase project that automatically backs up the Authentication users' data to a Google Cloud Storage bucket on a predefined schedule.

Author: Foo Jen Sean (www.foojensean.com)

Description: Automatically backs up Firebase Authentication users to a Google Cloud Bucket on a predefined schedule.

Overview

This extension simplifies the process of regularly backing up Firebase Authentication users' data. By automatically creating backups at a specified interval, this extension provides an extra layer of security and ensures data recoverability for your Firebase project.

Details

This extension regularly backs up user data from Firebase Authentication. It captures user details such as unique identifiers (UIDs), email addresses, custom claims, and more. These backups can be vital for disaster recovery, auditing, and analysis purposes. The backups are in JSON format and are stored in a Google Cloud Storage bucket. You can configure the extension to run at a specified interval, such as daily or weekly. The backups can be easily imported into a Firebase project using the Firebase Admin SDK.

Setup

Installation

To install this extension for your Firebase project, you can install it via the Firebase Extensions Hub.

Usage

Exporting Firebase Authentication Users

The extension runs automatically based on your configured schedule. There is no need for manual intervention after setup. Backups are created and stored in the specified Cloud Storage bucket.

Importing Firebase Authentication Users

TODO: Add instructions for importing users from backups. Possibly create a separate extension for this. This will most likely be a done via a Node.js script that uses the Firebase Admin SDK.

Billing

To use this extension, your Firebase project must be on the Blaze (pay as you go) plan due to its use of Firebase and Google Cloud Platform services. Be aware that this extension uses Cloud Storage for backups, which may incur associated charges based on usage.

Access Required

The extension requires appropriate IAM roles to access and manage resources in your project:

• Firebase Authentication admin: This role is necessary for retrieving user data for backup.
• Storage Admin: This role is necessary for creating backups in Cloud Storage.

Contributing

Contributions to this extension are welcome! Feel free to submit issues, feature requests, or pull requests to enhance its functionality. See the CONTRIBUTING guide for more information.

Testing

All tests are written with Mocha.

Unit Tests

The unit tests target the performBackup function in extension/src/index.ts.

Unit tests can be run locally. It uses fake data and mocks to simulate the Firebase Authentication and Cloud Storage services.

To run the unit tests, run the following command in the extension folder:

npm run test:unit

Integration Tests

To run the integration tests, you first need to set up a Firebase project and install the Firebase CLI. The integration tests are created to test connections to external services such as Firebase Authentication and Cloud Storage.

To run the integration tests, run the following command in the extension folder:

npm run test:integration

System Tests

To run the system tests, you first need to set up a Firebase project and install the Firebase CLI.

1. Create a .env file in extension/tests/system-tests/configs with the following contents:

   PROJECT_ID="<project id>" # Optional: Project ID of the Firebase project to test with, if you want to test with a live project
   BUCKET_NAME="<bucket name>"
   CRON_SCHEDULE="<cron schedule>"
   

   You can refer to the sample.env file in the same folder for an example.

2. Optional: Download a service account key for the Firebase project you want to test with and save it as extension/tests/configs/serviceAccountKey.json. This is required if you want to test the extension with a live Firebase project in addition to using the fakes.

3. Optional: Run the system tests with the Firebase Emulator Suite by running the below command in the auth-scheduled-backup directory. This is required if you want to run the tests that use the Firebase Emulator Suite.

   firebase emulators:exec "cd extension && npm run test:system"

License

This project is licensed under the Apache License, Version 2.0. See LICENSE for more information.

Contact

If you have any questions or need assistance, you can reach out to me at:

• Email: [email protected]

Further Resources

For more information about Firebase Authentication and its features, you can refer to the official Firebase Authentication Documentation.

auth-scheduled-backup's Issues

Enhance listAllUsers

Current implementation is to return an array of UserRecord objects. This is fine if there is relatively low amount of users in the Firebase project. However, if there are millions of users this will be an issue as millions of records will be loaded in memory.

Multithreading might be needed to read users concurrently. (to improve speed)

Some issues:

1. Projects with large amount of users might take very long to list all users. (9 minutes for v1 function, 60 minutes for v2 function)
2. Projects with large amount of users might cause cloud function to crash as all users are currently loaded in memory.
3. ListAllUsers should also be an atomic read from start to end. (query based on user creation before CRON_SCHEDULE?)

Reorganise modules and dependencies.

Backups will override each other if they more than one backup is executed each day.

This is due to the folder names being only the current date.

Add ability for extension to automatically create bucket in Google Cloud Storage

Currently, a bucket has to be set up before installing this extension. The user then selects the bucket during the installation process.

A more user-friendly way is create the bucket during the lifecycle event when the extension is first installed. The cloud function then references that bucket in its scheduled backups.

Perfomance improvement attempt.

Backup of 13,000 users took around 24 seconds, and given the time cap of 9 minutes of cloud functions, this will give a maximum backup of around 3,000,000 users. This may not be ideal for projects with large user base.

Improve storage space on Google Cloud bucket.

Utilise gzip to improve storage space.

Check if there is an alternative to adding typescript and eslint to dependencies from devDependencies.

Add icon, improve readme, preinstall and postinstall

Change name and fix links

Create eslint configuration

Seperate user backup json file into chunks

Improve markdown documentations

Setup integration testing.

Some pointers:

• Fast execution.
• Upload to Google Cloud Bucket.
• Verify data was exported.
• Delete data from Google Cloud Bucket,

Extension not using cron time set by user

Extension is not using cron time set by user due to small bug in extension.yaml

Update cloud function to 2nd generation

Setup integration testing with Github actions

Improve parameters in extension.yaml

Update Cloud Function to 2nd generation

Add ability to choose scheduled time through params

Current scheduled time is hardcoded to "0 16 * * *".

• Allow ability to choose timezone (possibly from a dropdown)
• Allow ability to enter crontab schedule (some form of validation needed)

Some good references:
Crontab regex to validate user input

Import users script