periscopedata / redshift-udfs

SQL for many helpful Redshift UDFs, and the scripts for generating and testing those UDFs

Redshift UDF Harness

AWS Redshift recently announced support Python-based User Defined Functions. This repository contains SQL for many helpful Redshift UDFs, and the scripts for generating and testing those UDFs.

If you'd like to contribute more UDFs, please send us a pull request or contact us over at Periscope.io

Usage

The scripts require Ruby. To connect to your cluster to add or test UDFs, you'll need the the pg gem and the following vars in your env or in the config.yaml:

• UDF_CLUSTER_HOST -- host of your Redshift cluster, e.g. my-cluster.us-west-2.redshift.amazonaws.com
• UDF_CLUSTER_PORT -- cluster's db port, e.g. 5439
• UDF_CLUSTER_DB_NAME -- database name, e.g. master
• UDF_CLUSTER_USER -- database user, e.g. read-write-user
• UDF_CLUSTER_PASSWORD -- database user's password

Running the main file udf.rb without args shows how to use it:

Usage:
  ruby udf.rb <action> [udf_name]

Actions:
  load    Loads UDFs into your database
  drop    Removes UDFs from your database
  test    Runs UDF unit tests on your database
  print   Pretty-print SQL for making the UDFs

Examples:
  ruby udf.rb load
  ruby udf.rb drop harmonic_mean
  ruby udf.rb test json_array_first
  ruby udf.rb print

We look forward to bug reports, UDF requests, and general feedback :)

redshift-udfs's Issues

Project status

Looks like last commit here was in 2016. Is this project still used an maintained by Periscope?

We use this project at Simple, although we don't have to interact with it much since we rarely define new UDFs.

We may start using UDFs more now that UDFs in SQL are available and perform better than the python versions. We may start looking more into that and I'd be interested to hear if others are relying on this project and would want to work together on adding support for SQL UDFs.

Aggregate functions not supported

According to Amazon Redshift Documentation, creating aggregate functions like the below example are not publicly available features. This repository is throwing errors and fails the tests.

http://docs.aws.amazon.com/redshift/latest/dg/udf-creating-a-scalar-udf.html

Example failing construct.

create aggregate second_max (float)
(
        initfunc = agg_init_blank_varchar,
        aggfunc = agg_agg_numbers_to_list,
        finalizefunc = agg_finalize_second_max
);

Switch from stable to immutable

I don't see why we set the volatility of all UDFs to stable instead of immutable here: https://github.com/PeriscopeData/redshift-udfs/blob/master/udf_harness.rb#L108

The Redshift documentation says this:

http://docs.aws.amazon.com/fr_fr/redshift/latest/dg/r_CREATE_FUNCTION.html