This project takes a self-serve backup generated from the Firebase Realtime Database and outputs a flattened CSV as output.
The CSV is made to be uploaded to BigQuery but can be used for any number of purposes.
The term flattened is used to represent the following transformation
{
"a": true,
"b": {
"c": true
}
}
will be converted to
a,true
b,c,true
This project was created and tested using Node v9.1.0.
Setup using: npm install
To generate a self-serve backup:
- Go to the Firebase console (
https://console.firebase.google.com/project/<YOUR_PROJECT>/database/backups
). - Click on a backup to download it from GCS. (or create a public link to it)
Run this using: node export.js --inputURL http://link/to/backup.gz --outputPath /tmp/output.gz
required
This flag indicates where the input resides.
required
This flag indicates where the output csv should reside.
optional
This flag takes no parameters. By default, it is expected that the input is gzipped. If it is not, use this flag to indicate as such.
optional
This flag takes no parameters. By default, it is expected that the output will be gzipped. If it should not, use this flag to indicate as such.
optional
This flag takes no parameters. By default, the output will contain rows of mixed-width depending on how deep the input data is. By using this flag, the output will be normalized to contain 32 columns.
- Navigate to the BigQuery console.
- Under the desired Data Set, click "Create new table".
- Use the following params:
- Source Data:
Create from source
- Location:
Google Cloud Storage
, set the path (e.g.gs://my-bucket/output.gz
) - File format:
CSV
- Table name:
your-dataset
.your-table
- Schema: (If you used
--fillWithNulls
, you can setAutomatically detect
), otherwise clickEdit as Text
and input:c1:STRING,c2:STRING,c3:STRING,c4:STRING,c5:STRING,c6:STRING,c7:STRING,c8:STRING,c9:STRING,c10:STRING,c11:STRING,c12:STRING,c13:STRING,c14:STRING,c15:STRING,c16:STRING,c17:STRING,c18:STRING,c19:STRING,c20:STRING,c21:STRING,c22:STRING,c23:STRING,c24:STRING,c25:STRING,c26:STRING,c27:STRING,c28:STRING,c29:STRING,c30:STRING,c31:STRING,c32:STRING
- Field delimiter:
Comma
- Allow jagged rows:
Check
(ONLY if you didn't use--fillWithNulls
)
- Source Data:
- Click
Create Table
.
This is not an official Google product