popopenguin / aws_ml_crypto

A Java API that gathers historical cryptocurrency pricing data (via CryptoCompare API) & makes predictions (via AWS Machine Learning API)

AWS_ML_Crypto

AWS_ML_Crypto is a Spring-based RESTful API written in Java that gets & stores historical cryptocurrency pricing. It then uses that data to make price predictions.
AWS_ML_Crypto utilizes a number of third-party tools to accomplish this, including CrytoCompare API, AWS SDK for Java, AWS Redshift & AWS Machine Learning Real Time Predictions.

Using an hourly cronjob, this API continuously stores new data, predicts future values, and evaluates its predictions.

Please note: This API is in active development.

Getting Started

Requirements

The following must be installed on your local machine to use this program as is.

• Java 8 JDK
• Maven
• SQL Workbench/J

Clone the project to your desired location

$ git clone https://github.com/meejahnsnutshell/AWS_ML_Crypto
$ cd AWS_ML_Crypto
$ mvn clean install

AWS Setup

You will need an AWS account. Please note that some services may incur fees. Avoid unnecessary charges by disabling services after use.

• Set up your AWS Credentials.

• Launch a Redshift Cluster. In the application.properties file of the application, update the datasource url, username, and password.

• Follow these instructions to connect Workbench/J to your Redshift cluster.

Loading Historical Data

• Cryptocompare API is source for historical data of current cryptocurrency pricing. All API calls for aggregating and loading data are within the HistoController. Calls that can be made include "data",
  "backload", "backloadYear", and "coin". "coin" should be run first to populate data tables for coin list before "backloadYear" is used to backload the data for cryptocurrency wanted.

• Parameters for API call include type (type of historical call being made; default = histohour), fsym (from Symbol; default = BTC), tysm (to Symbol; default = USD), e (exchange; default = CCCAGG), extraParams (name of application), sign (If set to true, the server will sign the requests; default = false), tryConversion (If set to false, it will try to get values without using any conversion at all; default = true), aggregate (default = 1), limit (default = 50), toTs (unix Timestamp).

Required Parameters: Type, fsym, tsym, and e are the minimum parameters that should be included for all API calls.

• Ex of a URI call

.../histo/data?type=histohour&fsym=BTC&tsym=USD&limit=10&aggregate=1&e=CCCAGG

• Data API calls mirror API calls from Cryptocompare. Data received will be checked against database for new data entries between timestamps. Any new data will be inserted into the database.

• Backload API calls will check database for most current timestamp for requested cryptocurrency and will aggregate new data up to current time stamp and insert into database.

• BackloadYear API call will aggregate all data since the requested year change based on current timestamp and cryptocurrency selected. Parameters are the same as other API calls with the addition of year (default value = -1). This parameter is the change to the current year date for data requested. Any new data that does not exist in the database will be entered. Ex. URI for backloading data for previous year:

/histo/backloadYear?type=histohour&fsym=BTC&tsym=USD&e=CCCAGG&limit=500&year=-1

• Coin API calls will get all current cryptocurrency listed on Cryptocompare and will insert any new currency into the database.

Making Predictions

• Using the AWS console, ensure you have a datasource (which is data that AWS exports from your Redshift db to an S3 bucket) available for your machine learning model. Models can be created programatically or via the console - both use an existing datasource.

• To programatically create a model, set createmodel=true. Optional: modelId=xxxx&modelName=xxxx. Id and name are generated if not provided. Ex URIs:

.../predict/realtime?createmodel=true&modelId=1234&modelName=MyModel

or

.../predict/realtime?createmodel=true

Model parameters may be configured in the code: [insert screenshot]

• To use an existing model, provide id and name.

.../predict/realtime?createmodel=false&modelid=1234&modelname=MyModel

• Make a prediction HTTP call. For example, using an existing model:

.../predict/realtime?modelid=1234&modelname=model1234

Contributors

• Jialor Cheung
• Meghan Boyce

Acknowledgments

• CryptoCompare API
• AWS SDK for Java
• Machine Learning Samples by awslabs