GitHubHelp

ericbottard/spring-cloud-dataflow-acceptance-tests

★ 0Forks 0JavaGitHub ↗Compare

README

Spring Cloud Data Flow Acceptance Tests

This project bootstraps a dataflow server on a target platform, executes a series of tests by creating a series of streams and tasks and then cleans up after its done.

How to run it

The main script is called run.sh and supports a few flags:

USAGE: run.sh -p <PLATFORM> -b <BINDER> [-s -t -c]
  The default mode will setup, run tests and clean up, you can control which stage you want to
  have executed by toggling the flags (-s, -t, -c)

Flags:

[*] -p  | --platform - define the target platform to run
    -b  | --binder - define the binder (i.e. rabbit, kafka) defaults to rabbit
    -tests - comma separated list of tests to run (you can also specify expressions such as *http* for all tests with http word on it)
    -s  | --skipSetup - skip setup phase
    -t  | --skipTests - skip test phase
    -c  | --skipCleanup - skip the clean up phase
    -d  | --doNotDownload - skip the downloading of the server

[*] = Required arguments

The first option is to choose a PLATFORM, available options cloudfoundry, local and k8s (kubernetes). The scripts for each plaform are located in folders of the same name in the main directory.

By default the script will execute three main phases:

  • Setup: The setup phase will traverse each folder and call create.sh scripts. At the end of this phase you should expect to have an environment available with the Spring Cloud Data Flow server along with the services required for it to run.

  • Test: The test phase will invoke the mvn test and deploy apps into the environment and run tests.

  • Clean: The clean up phase will undeploy the server and remove any services.

Each phase can be toggled by setting the appropriate flag (-s, -t, -c)

The services created in the setup phase are mysql and redis. Depending on the binder selected, it will create a Rabbit or Kafka service.

Examples

To run the tests locally cleaning up services using Rabbit (the default binder)

./run.sh -p local

To run the tests locally cleaning up services using Kafka

./run.sh -p local -b kafka

To run the tests locally and keep the Data Flow Server, Kafka, and other services running afterwards

./run.sh -p local -b kafka -c

To run the tests on cloudfoundry, cleaning up services and using Rabbit (the default binder)

./run.sh -p cloudfoundry

To setup a Data Flow Server and services on CloudFoundry, but not run tests

./run.sh -p cloudfoundry -c -t

General configuration

Make sure you have JAVA_HOME configured correctly in your environment.

Each platform will have a file named env.properties located on init/env.properties, change those to reflect your environment. Each platform has different flags, but the global ones should be:

  • RETRIES : Number of times to test for a port when checking a service status (6 by defaut)

  • WAIT_TIME: How long to wait for another port test (5s by default)

  • SPRING_CLOUD_DATAFLOW_SERVER_DOWNLOAD_URL: Location of the dataflow jar file to be downloaded.

Required environment variables when skipping setup

If you want to point to an already running Data Flow server, set the environment variables

  • SERVER_URI - default is http://localhost:9393

  • STREAM_REGISTRATION_RESOURCE - default is maven + rabbit based apps

  • TASK_REGISTRATION_RESOURCE - deafult is maven + rabbit based tasks

To point to where your server is located and also specify which artifacts you want to register with the server.

Platform specific notes

Local

Pre-requisites

  • docker and docker-compose installed. Make sure you can connect to the docker daemon without using 'sudo', e.g. docker info works.

  • $DOCKER_SERVER environment variable properly set. Defaults to localhost, which works on unix. For MacOS 192.168.99.100 should work.

The local deployment will always try to connect to a service running on your local machine. So if you have a local redis we will use it.

If a local service is not found, the script will try to deploy using docker-compose so it’s important that you have that installed and configured properly.

When cleaning up, the script will only remove docker images, if you are using a local service like redis or mysql the script will not do anything to it

Cloudfoundry

Pre-requisites

On Cloudfoundry, make sure you have the following environment variables exported. We will not include them on any files to prevent it to be leaked into github repos with credentials.

  • SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL

  • SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN

  • SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME

  • SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD

Configuration

You can override service names and plans by either exporting or changing the following properties:

  • MYSQL_SERVICE_NAME

  • MYSQL_PLAN_NAME

  • RABBIT_SERVICE_NAME

  • RABBIT_PLAN_NAME

  • REDIS_SERVICE_NAME

  • REDIS_PLAN_NAME

Kubernetes

Pre-requisites

Google Cloud SDK installed with the kubectl component enabled.

Configuration

For Kubernetes, make sure you have the following environment variables exported.

  • GCLOUD_PROJECT

  • GCLOUD_COMPUTE_ZONE

  • GCLOUD_CONTAINER_CLUSTER

Note
 You can also set a KUBERNETES_NAMESPACE environment variable that specifies an existing namespace to use for the testing. If this is not specified the 'default' namespace will be used.

If you use a service account make sure to set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to your service account key file and to use the following to authenticate:

gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS

Code formatting guidelines

  • The directory /etc/eclipse has two files for use with code formatting, eclipse-code-formatter.xml for the majority of the code formatting rules and eclipse.importorder to order the import statements.

  • In eclipse you import these files by navigating Windows → Preferences and then the menu items Preferences > Java > Code Style > Formatter and Preferences > Java > Code Style > Organize Imports respectfully.

  • In IntelliJ, install the plugin Eclipse Code Formatter. You can find it by searching the "Browse Repositories" under the plugin option within IntelliJ (Once installed you will need to reboot Intellij for it to take effect). Then navigate to Intellij IDEA > Preferences and select the Eclipse Code Formatter. Select the eclipse-code-formatter.xml file for the field Eclipse Java Formatter config file and the file eclipse.importorder for the field Import order. Enable the Eclipse code formatter by clicking Use the Eclipse code formatter then click the OK button.

    • NOTE: If you configure the Eclipse Code Formatter from File > Other Settings > Default Settings it will set this policy across all of your Intellij projects.

Contributors

viniciusccarvalhotrisbergcppwfsmarkpollackilayaperumalgjvalkealolegz

Issues