GithubHelp home page GithubHelp logo

godaddy / asherah Goto Github PK

View Code? Open in Web Editor NEW
74.0 13.0 47.0 4.35 MB

Asherah is a multi-language, cross-platform application encryption SDK

License: MIT License

C# 42.79% Shell 0.88% Java 30.04% Python 1.44% Dockerfile 0.15% Gherkin 0.06% Go 24.16% JavaScript 0.48%
security cryptography go java

asherah's Introduction

License CircleCI Codecov

Asherah

An application-layer encryption SDK that provides advanced encryption features and defense in depth against compromise.

Its goal is to provide an easy-to-use library which abstracts away internal complexity and provides rapid, frequent key rotation with enterprise scale in mind.

Table of Contents

Introduction

Asherah makes use of multiple layers of keys in conjunction with a technique known as "envelope encryption". Envelope encryption is a practice where a key used to encrypt data is itself encrypted by a higher-order key and stored alongside the encrypted data, hence forming an envelope structure. The master key used at the root of the key hierarchy is typically managed by a Hardware Security Module (HSM) or Key Management Service (KMS).

The SDK generates cryptographically strong intermediate keys in the hierarchical model and manages their storage via a pluggable backing datastore. The integration with a HSM or KMS provider for the root (master) key in the hierarchy is implemented using a similar pluggable model. This allows for supporting a wide variety of datastores and cloud providers for different architectures.

The SDK provides implementations in multiple languages using native interoperability mechanisms to securely manage and cache internally-generated keys in off-heap protected memory. The combination of secure memory management and the hierarchical key model's partitioning help minimize attack exposure in the event of compromise. Using the protected memory cache has an added benefit of reducing interactions with external resources to improve latency and minimize incurred costs.

Getting Started

The basic use of the SDK proceeds in 3 steps:

Step 1: Create a session factory

A session factory is required to generate encryption/decryption sessions. For simplicity, the session factory uses the builder pattern, specifically a step builder. This ensures all required properties are set before a factory is built.

To obtain an instance of the builder, use the static factory method newBuilder. Once you have a builder, you can use the withXXX setter methods to configure the session factory properties.

Below is an example of a session factory that uses in-memory persistence and static key management.

SessionFactory sessionFactory = SessionFactory.newBuilder("some_product", "some_service")
    .withInMemoryMetastore() // in-memory metastore
    .withNeverExpiredCryptoPolicy()
    .withStaticKeyManagementService("thisIsAStaticMasterKeyForTesting") // hard-coded/static master key
    .build());

Step 2: Create a session

Use the factory to create a session.

Session<byte[], byte[]> sessionBytes = sessionFactory.getSessionBytes("shopper123");

The scope of a session is limited to a partition id, i.e. every partition id should have its own session. Also note that a payload encrypted using some partition id, cannot be decrypted using a different one.

Step 3: Use the session to accomplish the cryptographic task

The SDK supports 2 usage patterns:

Encrypt / Decrypt

This usage style is similar to common encryption utilities where payloads are simply encrypted and decrypted, and it is completely up to the calling application for storage responsibility.

String originalPayloadString = "mysupersecretpayload";

// encrypt the payload
byte[] dataRowRecordBytes = sessionBytes.encrypt(originalPayloadString.getBytes(StandardCharsets.UTF_8));

// decrypt the payload
String decryptedPayloadString = new String(sessionBytes.decrypt(dataRowRecordBytes), StandardCharsets.UTF_8);

Store / Load

This pattern uses a key-value/document storage model. A Session can accept a Persistence implementation and hooks into its load and store calls.

Example HashMap-backed Persistence implementation:

Persistence dataPersistence = new Persistence<JSONObject>() {

  Map<String, JSONObject> mapPersistence = new HashMap<>();

  @Override
  public Optional<JSONObject> load(String key) {
    return Optional.ofNullable(mapPersistence.get(key));
  }

  @Override
  public void store(String key, JSONObject value) {
    mapPersistence.put(key, value);
  }
};

Putting it all together, an example end-to-end use of the store and load calls:

// Encrypts the payload, stores it in the dataPersistence and returns a look up key
String persistenceKey = sessionJson.store(originalPayload.toJsonObject(), dataPersistence);

// Uses the persistenceKey to look-up the payload in the dataPersistence, decrypts the payload if any and then returns it
Optional<JSONObject> payload = sessionJson.load(persistenceKey, dataPersistence);

Sample Applications

The samples directory includes sample applications that demonstrate use of Asherah SDK using various languages and platforms.

Further Reading

Supported Languages

Derived Projects

Asherah-Cobhan implementations built atop the Asherah SDK for Go.

Feature Support

  • AWS KMS Support
  • RDBMS Metastore
  • DynamoDB Metastore
  • Session caching
  • Encrypt/Decrypt pattern
  • Store/Load pattern

Contributing

All contributors and contributions are welcome! Please see our contributing docs for more information.

asherah's People

Contributors

agabeev-godaddy avatar agerard-godaddy avatar aka-bo avatar bdittmer-godaddy avatar dalibor avatar dependabot[bot] avatar eabrams007 avatar ethompson-godaddy avatar jchannell-godaddy avatar jeffchannell avatar jgowdy avatar jgowdy-godaddy avatar jpage-godaddy avatar jpaskhay avatar jpaskhay-godaddy avatar jwilhelm-godaddy avatar kperry-godaddy avatar lramsudh-godaddy avatar nikoo28 avatar nlohia-godaddy avatar ravinaik1312 avatar renovate[bot] avatar sbos-godaddy avatar sjlbos avatar smimani-godaddy avatar snyk-bot avatar sushantmimani avatar tarkatronic avatar zeroaltitude avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

asherah's Issues

ALL: Investigate improving unit testing around low level native calls

There are a few empty unit test cases that have yet to be implemented in Java and C# (LibcProtectedMemoryAllocatorTest in particular). The affected library source code is covered via other unit tests that are in place, but some of the low level native calls are difficult to unit test in the traditional sense. We could theoretically setup functional/integration tests to cover them.

Regression tests fail due to a deadlock

In case we use an external RDBMS metastore and try to run the regression tests, some of the multi-threaded tests fails.

This is because multiple threads try to execute a write query on the db. The way our store query is written (WHERE NOT EXISTS) causes a deadlock.

C#: Investigate ways to unit test OS/arch flow handling

In ProtectedMemorySecretFactoryTest.cs, there is logic around detecting whether we can support the runtime OS/architecture meta. These are currently implemented using static methods. They seem like they should be able to be instance methods as a simple fix. Wondering what it's lifecycle ends up looking like after #8.

[Golang] Memguard research

AEL-749

  • Upcoming giant PR (awnumar/memguard#81) and its potential impact
    • not using development branch. reassess when they merge it
  • Investigate how memguard is being used (looks like for crypto too?)
    • claims not being used for crypto but i see it being used for the random key generation. should verify whether this is needed โ€“ come back to this
  • Looks like they have encryption in memory as well?
    • that is in development branch
    • if so, evaluate when merged and how it compares vs openssl
  • Appears to also have guard pages adjacent to mlock/mprotect page that contains the secret. need to know impact
    • looks like will cause memory scans to crash the process when those are scanned. will want to call out in documentation for users IMO
  • Verify SM aspects
    • how it compares to our implementation from functional and security aspect (i.e. memguard and features we're using)

AC:

  • Any further research/verification of above notes are handled
  • Any changes needed as a result of above are implemented
  • Any documentation needed is in place (even MVP/minimal)

ALL: Finalize edge case memory leak potential in memory allocation

The C# ProtectedMemorySecret initialization has try/catch logic around the initial copying of the source data to handle edge case failures that could lead to memory leaks (not dealloc'ing). This seems like something that should be in the Java implementation as well. Additionally, there is an inconsistency in the aforementioned C# cleanup handling that is different than the regular cleanup/close path (not doing the pointer exchange).

We had untested changes around this that didn't make it into this public repo. We should normalize the approaches between the languages and update the Secure Memory documentation to reflect it accordingly. May be good opportunity to also verify whether the pointer exchanges are needed (if we're clearing/deallocating anyway).

[Golang] User level interface

AEL-701

AC:

  • SessionFactory: we need consistency on builder pattern. alternatively, Connor and Aji suggested using using GoLang functional options (see internal github: iharbani/go-logger/blob/master/Logger.go#L59 as example)

  • Basically, via the test app and ref app, we'll need to ensure that the user experience using the library is essentially the same as with other libraries

  • POC/evaluation of functional options vs method chaining / builder pattern

  • Apply POC to Remaining types (Metastore, KeyManagementService, CryptoPolicy)

  • Update any existing tests, samples, etc.

Java: Investigate impact of removing SafeAutoCloseable class

The SafeAutoCloseable class was in place as a work-around for avoiding some checked exception handling. If this is still desired, we can potentially just move up the removal of the throws class in the close method in the impacted classes that are using SafeAutoCloseable.

All: Consider renaming AppEncryption interface to AppEncryptionSession

Since we have AppEncryptionSessionFactory and speak in terms of sessions throughout our documentation, we should consider renaming the primary AppEncryption to AppEncryptionSession, Session, or similar. It would potentially be more self-documenting. Believe it's something we may have discussed a while back but never circled back to. Would be a fair amount of refactoring and docs cleanup.

[Golang] Refactor Go SecureMemory into separate repo

AEL-696

  • Create separate go-securememory repo in appservices AEL-766
  • Make Secret abstract/interface so it's not coupled to memguard AEL-767
  • Move any other memguard functionality we're using for SM aspects into new repo AEL-768

AC:

  • Separate AE and SM repos exist in app services (DomainBackend can fork, etc as needed)
  • Specifically, following changes should be isolated in SM repo:
    • SM abstraction of memguard usage.
      • Secret should be interface, of which memguard is one implementation of. Later OpenSSL may be a replacement one
      • Ideally abstracted by factory pattern (if possible in golang)
    • Should contain other memguard related setup, e.g. core dumps, destroy all keys cleanup

Spike research on artifact deployment

Parent: Use external CICD to build and deploy distributables

AEL-843

Need to know process for uploading to:

  • Maven repo
  • Nuget repo
  • Use other similar projects for reference on what they use for repos
  • Upcoming repos (e.g. golang, etc.)
  • How are credentials handled for this
  • Finalize a cross language versioning scheme
  • Will C# approach need to be changed (currently forcing "x.y.a-alpha" to be replaced, similar to java style snapshot)
  • Once complete, create additional issues for CD specific work.

ALL: Improve serialization of DRR

Currently the DRR serialization is essentially JSON and includes base64-encoded data. We should investigate and implement a more compact serialization strategy to save on storage the DRK/DRR take up (namely the resulting encrypted data payload).

Likely less important for the SK and IK size as they don't have data payloads, but we could investigate there as well if it makes sense.

Use external CICD to build and deploy distributables

AEL-787

AC:
From the Asherah repo, build and deploy using the chosen external CICD service

  • Spike on CircleCI Monorepo (multi language, images)
  • Spike research on other public CICD alternatives
  • Spike research on artifact deployment
  • Evaluate existing helper scripts wrt external cicd
  • Implement Java Secure Memory CI
  • Implement Java App Encryption CI
  • Implement Java Reference App CI
  • Implement Java Test App CI
  • Implement C# Logging CI
  • Implement C# Secure Memory CI
  • Implement C# App Encryption CI
  • Implement C# Reference App CI

All: Investigate better way of abstracting envelope encryption implementation

Currently EnvelopeEncryptionJsonImpl contains the core envelope encryption algorithm. We also have a wrapper EnvelopeEncryptionBytesImpl that sits as a wrapper around that class. This was done as a quick fix for allowing a different serialization mechanism for the DRR. Investigate if we can abstract this a bit more to avoid having a hard-coded dependency. May be addressed as part of #14.

Java: Improve SecureRandom Usage

The Java implementation is currently using SecureRandom a couple different ways, and we've been meaning to circle back on them. During our open sourcing effort I happened to notice tink-crypto/tink#72 and https://github.com/aws/aws-encryption-sdk-java/blob/ed30f4fd796e0315e4d08a201845bbe6aad19693/src/main/java/com/amazonaws/encryptionsdk/internal/Utils.java#L35 but forgot to create an Issue for it.

As part of this, we should also check if there are similar changes needed for the RNG being used in C# (or make specific Issue for it).

All: Consider using generic type for key in store/load

The store and load methods in the AppEncryption interface currently only operate with a single String key. Consider changing to be a generic type that would give more flexibility.

Simple enough to do for Java and C# but not sure what other languages' support for this would look like.

C#: Refactor Secret Dispose usage

Refactor Secret and ProtectedMemorySecret Dispose usage to avoid requiring implementing classes from having to suppress the finalizer. Refer to https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/implementing-dispose.

Abstract class should end up looking like:

protected abstract void Dispose(bool disposing);
public void Dispose()
{
    this.Dispose(true);
    GC.SuppressFinalize(this);
}

Will need to test if the new allocator instance is needed as in the current implementation.

ALL: Make dependencies optional where possible

All dependencies are currently required in existing language implementations (Java and C# as of this writing). For example, if someone uses the RDBMS metastore implementation, the SDK will still pull in the AWS DynamoDB SDK and all its transitive dependencies.

We can make pluggable external integrations and optional features only use optional dependencies so the user doesn't have to pull in dependency chains for unused parts.

An alternative approach we had discussed a while back was breaking out the external dependencies into separate projects/modules. The optional dependency approach seems like it will have less overhead in the SDK repo itself, and it shouldn't have much of a negative impact from an end user perspective.

Can discuss which approach seems best. Regardless of which approach is taken, we'll need to update documentation accordingly.

Onboard Eddie

AEL-684

  • Do it

AC:

Eddie can run java and C# tests
And check in code! i.e. bugs

Edit: removed internal info

C#: Refactor ILogger usage to eliminate LogManager facade

The Logging project was put in place as a work-around for the lack of static factory methods in the official C# logging extensions. It's a kluge and we should replace it with proper DI.

Make sure it fits well with configuration extension usage that would be typical in real apps. May need to do some decent refactoring to make it happen.

ALL: Clarify bzero related usage

There are some notes and TODOs around bzero usage in the Java and C# repos. Need to investigate and clarify any open questions. Apply any needed changes and/or cleanup comments.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.