GithubHelp home page GithubHelp logo

mujoko / aws-glue-libs Goto Github PK

View Code? Open in Web Editor NEW

This project forked from awslabs/aws-glue-libs

0.0 1.0 0.0 94 KB

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.

License: Other

Shell 0.63% Python 99.37%

aws-glue-libs's Introduction

aws-glue-libs

This repository supports python libraries for local development of glue pyspark batch jobs. Glue streaming is not supported with this library.

Contents

This repository contains:

  • awsglue - the Python libary you can use to author AWS Glue ETL job. This library extends Apache Spark with additional data types and operations for ETL workflows. It's an interface for Glue ETL library in Python.
  • bin - this directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.

Python versions by Glue Version

Different Glue versions support different Python versions. The following table below is for your reference, which also includes the associated repository's branch for each glue version.

Glue Version Python 2 Version Python 3 Version aws-glue-libs branch
0.9 2.7 Not supported glue-0.9
1.0 2.7 3.6 glue-1.0
2.0 Not supported 3.7 glue-2.0
3.0 Not supported 3.7 glue-3.0
4.0 Not supported 3.10 master

You may refer to AWS Glue's official release notes for more information

Setup guide

If you haven't already, please refer to the official AWS Glue Python local development documentation for the official setup documentation. The following is a summary of the AWS documentation:

The awsglue library provides only the Python interface to the Glue Spark runtime, you need the Glue ETL jar to run it locally. The jar is now available via the maven build system in a s3 backed maven repository. Here are the steps to set up your dev environment locally.

  1. install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
  2. use copy-dependencies target in Apache Maven to download the jar from S3 to your local dev environment.
  3. download and extract the Apache Spark distribution based on the Glue version you're using:
    • Glue version 0.9: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-0.9/spark-2.2.1-bin-hadoop2.7.tgz
    • Glue version 1.0: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz1
    • Glue version 2.0: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1
    • Glue version 3.0: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz
    • Glue version 4.0: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz
  4. export the SPARK_HOME environmental variable to the extracted location of the above Spark distribution. For example:
    Glue version 0.9: export SPARK_HOME=/home/$USER/spark-2.2.1-bin-hadoop2.7
    Glue version 1.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
    Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
    Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
    Glue version 4.0: export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0
    
  5. now you can run the executables in the bin directory to start a Glue Shell or submit a Glue Spark application.
    Glue shell: ./bin/gluepyspark
    Glue submit: ./bin/gluesparksubmit
    pytest: ./bin/gluepytest
    

(The gluepytest script assumes that the pytest module is installed and available in the PATH env variable)

Licensing

The libraries in this repository licensed under the Amazon Software License (the "License"). They may not be used except in compliance with the License, a copy of which is included here in the LICENSE file.


Release Notes

August 27 2021

  • The master branch has been modified from representing Glue 0.9 to Glue 3.0, we have also created a glue-0.9 branch to reflect the former state of the master branch with Glue 0.9. To rename your local clone of the older master branch and point to the glue-0.9 branch, you may use the following commands:
git branch -m master glue-0.9
git fetch origin
git branch -u origin/glue-0.9 glue-0.9
git remote set-head origin -a

aws-glue-libs's People

Contributors

vaviliv avatar neilagupta avatar moomindani avatar jpeddicord avatar mbeacom avatar skycmoon avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.