GithubHelp home page GithubHelp logo

joshcrosby / aws-glue-local-etl-docker Goto Github PK

View Code? Open in Web Editor NEW

This project forked from jnshubham/aws-glue-local-etl-docker

0.0 1.0 0.0 3 KB

Contains docker image for setting up the glue libraries locally for etl developments

Dockerfile 100.00%

aws-glue-local-etl-docker's Introduction

aws-glue-etl-local-docker

Link to Docker Image

This docker image is used to run your glue etl jobs on your local environment. This helps in the development of etl jobs locally without incurring additional costs by running Glue Devendpoints or Glue jobs.

Having glue libraries locally helps in the development and making it easier for the developer to update/change the code and test it locally before commiting it to a job.

Build Image Manually

To build the image on your system, follow these steps:

  1. Clone the repository to get the latest Dockerfile
git clone https://github.com/jnshubham/aws-glue-local-etl-docker.git
  1. Start Docker
system start docker
  1. Run build command
docker build -t jnshubham/glue_etl_local .
  1. Once the build is successful, run the container using commands in Run section.

To get the prebuild image directly

Run

docker pull jnshubham/glue_etl_local:latest

Check downloaded image by running

docker images

Commands to run the container

To run the container and get into pyspark shell directly

docker run jnshubham/glue_etl_local "gluepyspark"

To get into the terminal and submit a job run

docker run -it jnshubham/glue_etl_local

gluesparksubmit script_name parameters

To checkout the image visit Docker page here

Thanks!

aws-glue-local-etl-docker's People

Contributors

jnshubham avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.