zhaolei-best / incubator-inlong

Apache InLong

Apache InLong

• What is Apache InLong?
• Features
• When should I use InLong?
• Building InLong
• Deploying InLong
• Join the Community
• Documentation
• License

What is Apache InLong?

Apache InLong(incubating) is a one-stop data streaming platform that provides automatic, secure, distributed, and efficient data publishing and subscription capabilities. This platform helps you easily build stream-based data applications.

InLong was originally built at Tencent, has served online businesses for more than 8 years, to support massive data (data scale of more than 40 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules: Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster greatly meets the data reporting service requirements in the business big data scenario.

For getting more information, please visit our project documentation at https://inlong.apache.org/en-us/ .

Features

Apache InLong offers a variety of features:

• Ease of Use: a SaaS-based service platform, you can easily and quickly report, transfer, and distribute data by publishing and subscribing to data based on topics.
• Stability & Reliability: derived from the actual online production environment, it delivers high-performance processing capabilities for 10 trillion-level data streams and highly reliable services for 100 billion-level data streams.
• Comprehensive Features: supports various types of data access methods and can be integrated with different types of Message Queue (MQ) services, it also provides real-time data extract, transform, and load (ETL) and sorting capabilities based on rules, allows you to plug features to extend system capabilities.
• Service Integration: provides unified system monitoring and alert services, it provides fine-grained metrics to facilitate data visualization, you can view the running status of queues and topic-based data statistics in a unified data metric platform, configure the alert service based on your business requirements so that users can be alerted when errors occur.
• Scalability: adopts a pluggable architecture that allows you to plug modules into the system based on specific protocols, so you can replace components and add features based on your business requirements

When should I use InLong?

InLong is based on MQ and aims to provide a one-stop, practice-tested module pluggable data stream access service platform，based on this system, users can easily build stream-based data applications. It is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.

InLong is only a one-stop data reporting pipeline platform. It cannot be used as a persistent data storage, nor does it support complex business logic processing on data streams.

Building InLong

More detailed instructions can be found at Quick Start section in the documentation.

# Clone a repo
$ git clone https://github.com/apache/incubator-inlong.git
$ cd incubator-inlong

# Build InLong
$ mvn clean install -DskipTests

# Deploy components
 # see Deploying InLong section

Deploying InLong

InLong integrates a complete component chain for data reporting in big data scenarios, and not support automatic installation of modules now, so we need to choose manually to install all or some modules according to actual needs. Please refer to Quick Start in our project documentation.

Join the Community

• Ask questions on Apache InLong Slack
• Join Apache InLong mailing lists

  Name	Scope
  [email protected]	Development-related discussions	Subscribe	Unsubscribe	Archives

• Contributing: we always welcome new contributions, whether for trivial cleanups, new features or other material rewards, more details see How to contribute.

Documentation

• Home page: https://inlong.apache.org/en-us/
• Issues: https://issues.apache.org/jira/browse/InLong

License

© Contributors Licensed under an Apache-2.0 license.