GithubHelp home page GithubHelp logo

tritondatacenter / manta-archiver Goto Github PK

View Code? Open in Web Editor NEW
0.0 35.0 3.0 155 KB

CLI Utility to backup local data on Manta

License: Mozilla Public License 2.0

Java 98.03% Shell 1.97%
manta joyent backup cli backup-cli

manta-archiver's Introduction

Manta Archiver

Manta Archiver is a simple CLI tool that allows you to back up all files from a local directory to a remote Manta path. All files will be stored in Manta in a compressed state with optional encryption provided by the Manta Java SDK.

Usage

Manta Archiver is configured in the same fashion as java-manta but provides a way to interactively generate a configuration file so you don't need to learn about java-manta in order to use this project. See the following wiki articles for setup instructions:

Encryption

Oracle and Azul JVMs require installation of the Java Cryptography Extensions for Oracle JVMs and the Zulu Cryptography Extension Kit for Azul JVMs. This does not apply as of Java 8 update 161, which includes JCE by default for both Oracle and Azul. OpenJDK distributions do not need any modifications to support stronger encryption modes.

If you see the following error it means JCE is required but not installed:

Cannot support TLS_RSA_WITH_AES_256_CBC_SHA256 with currently installed providers

Commands

Universal options relevant to all commands

--log-destination: STDOUT, STDERR (default), FILE --log-level: TRACE, DEBUG, INFO, WARN (default), ERROR

generate-env

Arguments: [ bits ]

bits: Optional key strength in bits. May be omitted or one of the following: 128, 192, 256.

Interactively builds a configuration file from a template. If bits is provided the generated configuration will have encryption enabled. See the footnote about encryption strength requirements if selecting a value greater than 128.

connect-test

This command validates that the utility can connect properly to Manta with the current configuration. Additionally, it outputs the final configuration.

Example output:
$ java -jar target/manta-archiver-1.0.0-SNAPSHOT.jar connect-test
  Creating new connection object
    com.joyent.manta.client.MantaClient@6e4784bc
  Connection configuration
    BaseChainedConfigContext{mantaURL='https://us-east.manta.joyent.com', user='user', mantaKeyId='00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00', mantaKeyPath='/home/user/.ssh/id_rsa', timeout=20000, retries=3, maxConnections=24, httpBufferSize='4096', httpsProtocols='TLSv1.2', httpsCiphers='TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256', noAuth=false, disableNativeSignatures=false, tcpSocketTimeout=10000, connectionRequestTimeout=15000, verifyUploads=true, uploadBufferSize=16384, skipDirectoryDepth=null, clientEncryptionEnabled=false, permitUnencryptedDownloads=false, encryptionAuthenticationMode=Mandatory, encryptionKeyId=null, encryptionAlgorithm=AES/CTR/NoPadding, encryptionPrivateKeyPath=null, encryptionPrivateKeyBytesLength=null object}
  Attempting HEAD request to: /user
    com.joyent.manta.client.MantaObjectResponse{path='/user', contentLength=null, contentType='application/x-json-stream; type=directory', etag='null', mtime='null', type='directory', requestId='2949dbbc-0169-11e8-bc16-57a48844193f', httpHeaders=MantaHttpHeaders{wrappedHeaders={connection=keep-alive, x-response-time=18, x-server-name=a28d054b-d8e0-40ab-befc-e5728ef85650, x-request-id=2949dbbc-0169-11e8-bc16-57a48844193f, content-type=application/x-json-stream; type=directory, x-load-balancer=165.225.164.14, server=Manta, date=Thu, 25 Jan 2018 00:46:21 GMT, result-set-size=4}}, directory=true}
  Request was successful

generate-key

Arguments: <encryption-strength-bits> <path>

bits: 128, 192, 256 (only supported values)
path: path on local filesystem to save key

This command generates an a new encryption key using the specified parameters and saves it to a local path. This is your secret key and it must be proctected.

validate-key

Arguments: <path>

path: path to key on local filesystem to check

This command validates an existing encryption key to determine if it is a valid key.

upload

Arguments: <local-directory> <manta-directory>

local-directory: the directory path on the local file system to send to Manta
manta-directory: the remote directory path on Manta to upload data to

This command uploads all of the files and directories under the specified directory to Manta to Manta.

download

Arguments: <local-directory> <manta-directory>

local-directory: the directory path on the local file system to copy files to from Manta
manta-directory: the remote directory path on Manta to download data from

This command downloads all of the directories and files from Manta the specified remote Manta path.

verify-local

Arguments: [--fix] <local-directory> <manta-directory>

--fix: optional flag that indicates we upload any missing or different files to Manta
local-directory: the directory path on the local file system to copy files to from Manta
manta-directory: the remote directory path on Manta to download data from

This command verifies that the contents of a local directory (files and directories) match the contents of a remote Manta path. If the --fix flag is present, then this command will upload any missing files/directories or files that do not match the remote file. Additionally, this command compares the local file system with the files stored on Manta.

verify-remote

Arguments: <manta-directory>

This command verifies that the contents of a remote directory match the checksum stored in Manta's metadata. It does this by downloading each file and performing a checksum on the contents and comparing it to the checksum in the metadata.

manta-archiver's People

Contributors

dekobon avatar douglasatjoyent avatar tjcelaya avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

manta-archiver's Issues

JCE cannot authenticate the provider BC

Exception in thread "main" java.lang.SecurityException: JCE cannot authenticate the provider BC
	at javax.crypto.JceSecurity.getInstance(JceSecurity.java:113)
	at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:303)
	at com.joyent.manta.client.crypto.SecretKeyUtils.generate(SecretKeyUtils.java:69)
	at com.joyent.manta.archiver.MantaArchiverCLI$GenerateKey.run(MantaArchiverCLI.java:298)
	at com.joyent.manta.archiver.MantaArchiverCLI.main(MantaArchiverCLI.java:135)
Caused by: java.util.jar.JarException: file:/jar/manta-archiver-1.0.0-SNAPSHOT.jar has unsigned entries - com/joyent/manta/archiver/DirectoryUpload.class
	at javax.crypto.JarVerifier.verifySingleJar(JarVerifier.java:500)
	at javax.crypto.JarVerifier.verifyJars(JarVerifier.java:361)
	at javax.crypto.JarVerifier.verify(JarVerifier.java:289)
	at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:159)
	at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:185)
	at javax.crypto.JceSecurity.getInstance(JceSecurity.java:109)
	... 4 more

JCE cannot authenticate the provider BC - error when generating key

java -jar target/manta-archiver-1.0.0-SNAPSHOT.jar generate-key 'AES' 128 ./uber-key
Exception in thread "main" java.lang.SecurityException: JCE cannot authenticate the provider BC
	at javax.crypto.JceSecurity.getInstance(JceSecurity.java:113)
	at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:303)
	at com.joyent.manta.client.crypto.SecretKeyUtils.generate(SecretKeyUtils.java:69)
	at com.joyent.manta.archiver.MantaArchiverCLI$GenerateKey.run(MantaArchiverCLI.java:273)
	at com.joyent.manta.archiver.MantaArchiverCLI.main(MantaArchiverCLI.java:124)
Caused by: java.util.jar.JarException: file:/Users/elijah/Development/Joyent/manta-archiver/target/manta-archiver-1.0.0-SNAPSHOT.jar has unsigned entries - com/joyent/manta/archiver/FileUpload.class
	at javax.crypto.JarVerifier.verifySingleJar(JarVerifier.java:500)
	at javax.crypto.JarVerifier.verifyJars(JarVerifier.java:361)
	at javax.crypto.JarVerifier.verify(JarVerifier.java:289)
	at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:159)
	at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:185)
	at javax.crypto.JceSecurity.getInstance(JceSecurity.java:109)
	... 4 more

LoggingUncaughtExceptionHandler does not handle null root cause

Running the test suite in our Jenkins environment causes ObjectUploadQueueLoaderTest to spew the following error indefinitely (though with varying thread names):

Exception: java.lang.NullPointerException thrown from the UncaughtExceptionHandler in thread "ForkJoinPool-1-worker-2"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.