rocketchat / filestore-migrator

filestore-migrator

Provided as-is. Make use of backups and use at your own risk

filestore-migrator is a tool to move files uploaded to a Rocket.Chat instance between object storage providers. Currently we support as targets any object storage provider compatible with the S3 API, as well as, the local file system and Google Cloud Storage. GridFS is also compatible as a source target only.

FIX ORDER OF readPreferred!!!!

Installation

If you are installing filestore-migrator from source, make sure that you have a recent version of the go runtime installed and that you $GOPATH is in your OS system-wide PATH. Then, clone the repo to $GOPATH/src/github.com/RocketChat/filestore-migrator or use go get github.com/RocketChat/filestore-migrator. Now you only need to run the following commands on the tool source directory:

make build
filestore-migrator --help

Usage

Usage of filestore-migrator:
  -action string
    	Type of action to me performed by the tool (migrate, upload, download ) (default "download")
  -config string
    	Config File full path. Defaults to current folder
  -databaseUrl string
    	Rocket.Chat database connection string
  -destinationType string
    	Destination storage provider (s3, google, fs) (default "s3")
  -destinationUrl string
    	Destination connection string
  -detectDestination
    	Autodetect the destionation using the Rocket.Chat configuration
  -detectSource
    	Autodetect the source target using the Rocket.Chat configuration (default true)
  -skipErrors
    	Skip on error
  -sourceType string
    	Source storage provider (s3, google, gridfs, filesystem) (default "s3")
  -sourceUrl string
    	Source connection string
  -store string
    	Name of the storage to be used in the operation (default "Uploads")
  -tempLocation string
    	Temporary file location (default "/tmp/filestore-migrator")
  -verbose
    	Enable verbose logs (default true)

filestore-migrator accepts parameters either via flags or via a yaml configuration file, which is examplified in the cmd directory. Be aware that each URL type flag have specific patterns, as shown below:

• databaseUrl: Rocket.Chat database connection string. Use the official supported mongo connection string-
• sourceUrl: Source storage provider (s3, google, gridfs, filesystem)
  • gridfs: Automatically retrieved from the Rocket.Chat instance database
  • s3: http://${endpoint}/${bucket_name}?ssl=${ssl}&region=${region}&accessId=${accessId}&accessKey=${accessKey}
  • google: ${json_key}/${bucket_name}
  • filesystem: Normal OS path
• destinationUrl: Destination storage provider (s3, google, fs)
  • s3: http://${endpoint}/${bucket_name}?ssl=${ssl}&region=${region}&accessId=${accessId}&accessKey=${accessKey}
  • google: ${json_key}/${bucket_name}
  • filesystem: Normal OS path

Running with Docker

For those who prefer using filestore-migrator via docker, we provide a Dockerfile on the root of the directory. First you will need to compile the image using make docker. After the build process, execute with your parameters:

docker run filestore-migrator:latest -databaseUrl=mongodb://mongo:27017/rocketchat ...

filestore-migrator's Issues

WebDav Migration

Hi, i was wondering if it is possible to use WebDav as source and destination.

Question about migration

Hello,
I have question about migration from mongodb gridfs to filesystem scenario.
I used command:
filestore-migrator -action=migrate -databaseUrl=mongodb://mongo:27017/rocketchat -sourceType=girdfs -destinationType=fs -destinationUrl=/app/uploads.
It works fine, i have attachements in file system, and I can open it in application.
What about co mongdb database size - collection rocketchat_uploads.chunks still have migrated objects.

How to safty shrik database size and delete moved objects ?
Thank you for your work!

ensure files directory exists

[1/1320] Downloading Clipboard - 16. November 2018 15:16 from: AmazonS3
panic: open files/22nrCxQQyhTM7vgzk: no such file or directory

otherwise get this

AmazonS3 to GoogleStorage migration

Dear all, when trying to migrate AmazonS3 source data to GoogleStorage.

It is showing the following error.

2023/10/13 17:18:25 [10/13/2023 17:18:25] Source store type set to:  AmazonS3
2023/10/13 17:18:25 [10/13/2023 17:18:25] Destination store type set to:  GoogleStorage
2023/10/13 17:18:25 Beginning migration of files
2023/10/13 17:18:25 [10/13/2023 17:18:25] Store:  Uploads
2023/10/13 17:18:26 [10/13/2023 17:18:26] uniqueId {uniqueID xxxxxxxxxxxxxxxxxxxxx}
2023/10/13 17:18:26 [10/13/2023 17:18:26] rocketchat_uploads AmazonS3:Uploads
2023/10/13 17:18:45 [10/13/2023 17:18:45] Found 183441 files

2023/10/13 17:18:45 [10/13/2023 17:18:45] [1/183441] Downloading Clipboard - 20 de Outubro de 2020 às 12:32 from: AmazonS3

2023/10/13 17:18:45 [10/13/2023 17:18:45] [1/183441] Uploading to GoogleCloudStorage to: xxxxxxxxxxxxxxx/uploads/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

panic: write exception: write errors: [Updating the path 'AmazonS3' would create a conflict at 'AmazonS3']

Below is the configuration file used

Follow command

filestore-migrator -action migrate -config config.yaml -verbose true

Support for Mongo5 (and use of new Mongo driver)

Upon trying to connect to the RocketChat database, the migrator fails and errors out with a panic:

panic: server returned error on SASL authentication step: BSON field 'saslContinue.mechanism' is an unknown field.

If we search online for others with similar issues, the issue was always an outdated version of a Mongo driver for Go. Looking at the code here, it seems it's using a driver that does not support Mongo5.

There is an official Mongo driver that is well supported and can connect to Mongo5 (which is the latest recommended version of Mongo for Rocketchat).

The current driver used is also unmaintained: https://github.com/go-mgo/mgo and as such should be replaced.