liubodong / scaleph

Open data platform based on Flink and Kubernetes, supports web-ui click-and-drop data integration with SeaTunnel on Flink, manage flink jar job both YARN and Kubernetes. Now Scaleph is working on Flink SQL online editor

scaleph

The Scaleph project features data integration, develop, job schedule and orchestration and aims to run on cloud environment.

It trys to provide one-stop data platform for simplifying development of data application. Scaleph hopes to help peoples to aggregate and analyze data, free data internal worth and make profit from them.

Scaleph is driven by personal interest and evolves actively through faithful developer, flowerfine is open and appreciates any helps.

News

Congratulation, Scaleph has been ready for releasing 1.0.4 version and providing full data integration scene backened by SeaTunnel on 2023-04-23!

Now, community is working on preparation job such as docs, bug fix, or deploy tools. There are so many miscellaneous to handle and we are really looking forward any help from you! Just associate us on issue or wechat group and make scaleph better.

Features

• Provide Flink ecosystem to every data developer easily.
• Predefined expert Flink decisions and excellent step wizard.
• Job management.
  • Step job wizard
  • Multiple versions
  • Full job detail web
  • Configuration, dependency and resource

data integration

• Web-ui click-and-drag data integration ways backended by Apache SeaTunnel(incubating)
• Support 2.3.1 V2 out-of-the-box connectors and transforms
• DataSource management

data develop

• Multiple deployment modes and different resource providers.
  • Standalone。Session
  • YARN。Application, Per-Job, Session
  • Kubernetes。Session

Quick Start

Whenever people want to explore Scaleph system, they want a running Scaleph application, then people can interact with Scaleph through Scaleph Admin.

Luckily, deploy Scaleph locally just takes three steps.

• Make sure Docker installed on your machine.
• Clone the repository
• Use Docker Compose and Scaleph Docker image quickly install and run Scaleph.

git clone https://github.com/flowerfine/scaleph.git

cd scaleph
git checkout 1.0.4

cd scaleph/tools/docker/deploy/scaleph
docker-compose up

Once all containers have started, the UI is ready to go at http://localhost, user can login by sys_admin/123456.

Next, read documentation website quick start and try to run prefined SeaTunnel demos.

Documentation

please refer website

Build and Deployment

• develop. This doc describes how to set up local development environment of Scaleph project.

• checkstyle. Scaleph project requires clean and robust code, which can help Scaleph go further and develop better.

• build. This doc describes how to compile scaleph-api or scaleph-ui-react from source and build corresponding docker image. For more information about build from source, refer workflows and actions.

  • compile. Scaleph adopts maven as its build system.

    • local. People has to install jdk, maven, node, use mvn clean package or npm command to compile project.
    • docker. Recommend way. Just need git and docker, after clone the source code, use docker as the build tools to compile project, which is slower than local.

  • docker. As more application runs in container on cloud then bare metal machine, Scaleph provides own image.

    • docker-image-build。Choose different docker compose yaml file, people can build scalph-api, scaleph-ui-react or both.
    • docker-image-build-zh。For Chinese user.

• deploy. For different deployment purpose such as develop, test or production, Scaleph make the best effort for people deploy project on local, docker and kubernetes.

  • docker. require fluent network.
  • kubernetes. work in process.
  • binary. you can get the binary from github or build yourself.

RoadMap

features

1. data ingress and egress.
   1. Flink way data integration . Scaleph reduces complexity of integration through seatunnel, flink-cdc-connectors and other flink connectors.
   2. Friendly web-ui to newbies .
2. data develop
   1. udf + sql.
   2. support multi-layer data warehouse development.
3. job schedule and orchestrate

architectures

1. cloud native
   1. container and kubernetes development and runtime environment.
      1. flink operator
      2. seatunnel operator
      3. scaleph operator
   2. java 17, quarkus.
2. plugins. https://dubbo.apache.org/zh/docsv2.7/dev/principals/

Code of Conduct

This project adheres to the Contributor Covenant code of conduct

Contributing

For contributions, please refer CONTRIBUTING

Thanks for all people who already contributed to Scaleph!

Contact

• Bugs and Features: Issues

• Chinese user:

  

Sponsor

Thanks to JetBrains for supporting us free open source licenses.

License

Scaleph is licenced under the Apache License Version 2.0, link is here.