GithubHelp home page GithubHelp logo

pravega / gstreamer-pravega Goto Github PK

View Code? Open in Web Editor NEW
27.0 27.0 17.0 938 KB

GStreamer Plugins for Pravega

License: Apache License 2.0

Rust 46.07% Shell 16.98% Dockerfile 1.30% Python 14.72% Jupyter Notebook 20.08% JavaScript 0.17% HTML 0.09% Makefile 0.12% C++ 0.46%

gstreamer-pravega's Introduction

Pravega Build Status codecov License Version CII Best Practices

Pravega is an open source distributed storage service implementing Streams. It offers Stream as the main primitive for the foundation of reliable storage systems: a high-performance, durable, elastic, and unlimited append-only byte stream with strict ordering and consistency.

To learn more about Pravega, visit https://pravega.io

Prerequisites

  • Java 11+

In spite of the requirements of using JDK 11+ to build this project, client artifacts (and its dependencies) must be compatible with a Java 8 runtime. All other components are built and ran using JDK11+.

The clientJavaVersion project property determines the version used to build the client (defaults to 8).

Building Pravega

Checkout the source code:

git clone https://github.com/pravega/pravega.git
cd pravega

Build the pravega distribution:

./gradlew distribution

Install pravega jar files into the local maven repository. This is handy for running the pravega-samples locally against a custom version of pravega.

./gradlew install

Running unit tests:

./gradlew test

Setting up your IDE

Pravega uses Project Lombok so you should ensure you have your IDE setup with the required plugins. Using IntelliJ is recommended.

To import the source into IntelliJ:

  1. Import the project directory into IntelliJ IDE. It will automatically detect the gradle project and import things correctly.
  2. Enable Annotation Processing by going to Build, Execution, Deployment -> Compiler > Annotation Processors and checking 'Enable annotation processing'.
  3. Install the Lombok Plugin. This can be found in Preferences -> Plugins. Restart your IDE.
  4. Pravega should now compile properly.

For eclipse, you can generate eclipse project files by running ./gradlew eclipse.

Note: Some unit tests will create (and delete) a significant amount of files. For improved performance on Windows machines, be sure to add the appropriate 'Microsoft Defender' exclusion.

Releases

The latest pravega releases can be found on the Github Release project page.

Snapshot artifacts

All snapshot artifacts from master and release branches are available in GitHub Packages Registry

Add the following to your repositories list and import dependencies as usual.

maven {
    url "https://maven.pkg.github.com/pravega/pravega"
    credentials {
        username = "pravega-public"
        password = "\u0067\u0068\u0070\u005F\u0048\u0034\u0046\u0079\u0047\u005A\u0031\u006B\u0056\u0030\u0051\u0070\u006B\u0079\u0058\u006D\u0035\u0063\u0034\u0055\u0033\u006E\u0032\u0065\u0078\u0039\u0032\u0046\u006E\u0071\u0033\u0053\u0046\u0076\u005A\u0049"
    }
}

Note GitHub Packages requires authentication to download packages thus credentials above are required. Use the provided password as is, please do not decode it.

If you need a dedicated token to use in your repository (and GitHub Actions) please reach out to us.

As alternative option you can use JitPack (https://jitpack.io/#pravega/pravega) to get pre-release artifacts.

Quick Start

Read Getting Started page for more information, and also visit sample-apps repo for more applications.

Running Pravega

Pravega can be installed locally or in a distributed environment. The installation and deployment of pravega is covered in the Running Pravega guide.

Support

Don’t hesitate to ask! Contact the developers and community on slack (signup) if you need any help. Open an issue if you found a bug on Github Issues.

Documentation

The Pravega documentation is hosted on the website: https://pravega.io/docs/latest or in the documentation directory of the source code.

Contributing

Become one of the contributors! We thrive to build a welcoming and open community for anyone who wants to use the system or contribute to it. Here we describe how to contribute to Pravega! You can see the roadmap document here.

About

Pravega is 100% open source and community-driven. All components are available under Apache 2 License on GitHub.

gstreamer-pravega's People

Contributors

jdmaguire avatar jkhalack avatar raulgracia avatar vangork avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gstreamer-pravega's Issues

Panic on shutdown prevents flush

When running the gstreamer plugin a user encountered this:

+++ dirname ./scripts/pravega-to-file.sh
++ readlink -f ./scripts/..
+ ROOT_DIR=/home/sumit/Pravega-Idea/gstreamer-pravega
+ pushd /home/sumit/Pravega-Idea/gstreamer-pravega/gst-plugin-pravega
~/Pravega-Idea/gstreamer-pravega/gst-plugin-pravega ~/Pravega-Idea/gstreamer-pravega
+ cargo build
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
+ ls -lh /home/sumit/Pravega-Idea/gstreamer-pravega/target/debug/libgstpravega.so
-rwxrwxr-x 2 sumit sumit 325M May 16 07:56 /home/sumit/Pravega-Idea/gstreamer-pravega/target/debug/libgstpravega.so
+ export GST_PLUGIN_PATH=/home/sumit/Pravega-Idea/gstreamer-pravega/target/debug:
+ GST_PLUGIN_PATH=/home/sumit/Pravega-Idea/gstreamer-pravega/target/debug:
+ export GST_DEBUG=pravegasrc:LOG
+ GST_DEBUG=pravegasrc:LOG
+ export PRAVEGA_VIDEO_LOG=info
+ PRAVEGA_VIDEO_LOG=info
+ export RUST_BACKTRACE=1
+ RUST_BACKTRACE=1
+ PRAVEGA_CONTROLLER_URI=127.0.0.1:9090
+ PRAVEGA_SCOPE=examples
+ PRAVEGA_STREAM=split2
+ ALLOW_CREATE_SCOPE=true
+ OUTPUT_FILE=/home/sumit/split2.ts
+ gst-launch-1.0 -v pravegasrc stream=examples/split2 controller=127.0.0.1:9090 allow-create-scope=true '!' filesink location=/home/sumit/split2.ts sync=false
0:00:00.064894554  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:780:gstpravega::pravegasrc::imp:<pravegasrc0> query: BEGIN: query=Query { ptr: 0x557c06980c00, type: "caps", structure: Some(GstQueryCaps, filter=(GstCaps)"NULL", caps=(GstCaps)"NULL";) }
0:00:00.064972032  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:831:gstpravega::pravegasrc::imp:<pravegasrc0> query: END: result=true, query=Query { ptr: 0x557c06980c00, type: "caps", structure: Some(GstQueryCaps, filter=(GstCaps)"NULL", caps=(GstCaps)ANY;) }
0:00:00.065034898  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:780:gstpravega::pravegasrc::imp:<pravegasrc0> query: BEGIN: query=Query { ptr: 0x557c06980ca0, type: "caps", structure: Some(GstQueryCaps, filter=(GstCaps)"NULL", caps=(GstCaps)"NULL";) }
0:00:00.065078946  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:831:gstpravega::pravegasrc::imp:<pravegasrc0> query: END: result=true, query=Query { ptr: 0x557c06980ca0, type: "caps", structure: Some(GstQueryCaps, filter=(GstCaps)"NULL", caps=(GstCaps)ANY;) }
Setting pipeline to PAUSED ...
0:00:00.065343154  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:517:gstpravega::pravegasrc::imp:<pravegasrc0> start: BEGIN
0:00:00.065384665  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:535:gstpravega::pravegasrc::imp:<pravegasrc0> start: scope=examples, stream=split2, index_stream=split2-index
0:00:00.065426388  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:536:gstpravega::pravegasrc::imp:<pravegasrc0> start: start_mode=Earliest, end_mode=Unbounded
0:00:00.065455121  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:541:gstpravega::pravegasrc::imp:<pravegasrc0> start: controller=127.0.0.1:9090
0:00:00.065488100  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:543:gstpravega::pravegasrc::imp:<pravegasrc0> start: keycloak_file=None
0:00:00.065691583  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:547:gstpravega::pravegasrc::imp:<pravegasrc0> start: config=ClientConfig { max_connections_in_pool: 4294967295, max_controller_connections: 3, connection_type: Tokio, retry_policy: RetryWithBackoff { current: 1, base: 10, max_delay: Some(10s) }, controller_uri: PravegaNodeUri("127.0.0.1:9090"), transaction_timeout_time: 90000, mock: false, trustcert: "./ca-cert.crt", is_tls_enabled: false, credentials: Credentials { inner: Basic { method: "Basic", token: "Og==" } }, is_auth_enabled: false, reader_wrapper_buffer_size: 1048576, request_timeout: 30s }
0:00:00.065748272  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:548:gstpravega::pravegasrc::imp:<pravegasrc0> start: controller_uri=127.0.0.1:9090
0:00:00.065767384  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:549:gstpravega::pravegasrc::imp:<pravegasrc0> start: is_tls_enabled=false
0:00:00.065784903  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:550:gstpravega::pravegasrc::imp:<pravegasrc0> start: is_auth_enabled=false
0:00:00.067289934  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:557:gstpravega::pravegasrc::imp:<pravegasrc0> start: allow_create_scope=true
0:00:00.379170643  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:604:gstpravega::pravegasrc::imp:<pravegasrc0> start: Opened Pravega reader for data
0:00:00.389535994  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:612:gstpravega::pravegasrc::imp:<pravegasrc0> start: Opened Pravega reader for index
0:00:00.389593971  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:641:gstpravega::pravegasrc::imp:<pravegasrc0> start: end_offset=18446744073709551615
0:00:00.389790703  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:651:gstpravega::pravegasrc::imp:<pravegasrc0> start: Started
0:00:00.389842292  7847 0x557c06b36640 DEBUG             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:654:gstpravega::pravegasrc::imp:<pravegasrc0> start: END: result=Ok(())
0:00:00.390030387  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:680:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: BEGIN: segment=Segment { format: Time, start: "0:00:00.000000000", offset: "0:00:00.000000000", stop: "--:--:--.---------", rate: 1.0, applied_rate: 1.0, flags: (empty), time: "0:00:00.000000000", base: "0:00:00.000000000", position: "0:00:00.000000000", duration: "--:--:--.---------" }
0:00:00.390079707  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:730:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: initial_seek=true
0:00:00.390204092  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:739:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: seeking to timestamp=1970-01-01T00:00:00.000000000Z (0 ns, 0:00:00.000000000)
0:00:00.392529942  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:742:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: index_record=Err(Custom { kind: UnexpectedEof, error: "Index has no records" })
0:00:00.392783928  7847 0x557c06b36640 ERROR             pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:755:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: index is empty; segment=Segment { format: Time, start: "0:00:00.000000000", offset: "0:00:00.000000000", stop: "--:--:--.---------", rate: 1.0, applied_rate: 1.0, flags: (empty), time: "0:00:00.000000000", base: "0:00:00.000000000", position: "0:00:00.000000000", duration: "--:--:--.---------" }
0:00:00.392833127  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:775:gstpravega::pravegasrc::imp:<pravegasrc0> do_seek: END: result=false
0:00:00.393087436  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:836:gstpravega::pravegasrc::imp:<pravegasrc0> stop: BEGIN
0:00:00.393204443  7847 0x557c06b36640 INFO              pravegasrc gst-plugin-pravega/src/pravegasrc/imp.rs:848:gstpravega::pravegasrc::imp:<pravegasrc0> stop: END: result=Ok(())
ERROR: Pipeline doesn't want to pause.
thread 'tokio-runtime-worker' panicked at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /home/sumit/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.5.0/src/runtime/blocking/shutdown.rs:51Setting pipeline to NULL ...
:21
stack backtrace:
Freeing pipeline ...

Tokio panicked becaused the Runtime was blocked in an async context. This prevented the flush and resulted in no output.
Also submitted as: pravega/pravega-client-rust#267

NNStreamer Support

If you share your implementation of nnstreamer support (as discussed in https://lists.lfaidata.foundation/g/nnstreamer-technical-discuss/topic/pravega_io_nnstreamer/82135550?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,82135550 ) along with some error-logs, reproducible test cases, and such, we (nnstreamer contributors at https://github.com/nnstreamer ) might be able to help.

We may be able to include pravega for our on-device training projects as well: https://github.com/nnstreamer/nntrainer . We are working on on-device training (nntrainer) to enable neural network personalization on devices without exporting personal data to clouds. Both nnstreamer and nntrainer are standard machine learning frameworks of Tizen for TV, CE, and IoT devices.

Some issues with deepstream-pravega

  1. The following error is met when executing gstreamer-pravega/deepstream/docker-run.sh
luis@luis-precision:~/projects/gstreamer-pravega/deepstream$ ./docker-run.sh 
+++ dirname ./docker-run.sh
++ readlink -f ./..
+ ROOT_DIR=/home/luis/projects/gstreamer-pravega
+ xhost +
access control disabled, clients can connect from any host
+ docker run -it --rm --gpus all --network host --privileged --log-driver json-file --log-opt max-size=10m --log-opt max-file=2 -v /home/luis/projects/gstreamer-pravega:/root/work/gstreamer-pravega -v /tmp/.X11-unix:/tmp/.X11-unix -v /dev/log:/dev/log -e DISPLAY=:1 -e CUDA_VER=11.1 -w /root/work/gstreamer-pravega pravega/deepstream:latest
/entrypoint.sh: line 15: ENTRYPOINT: Required environment variable not set
  1. --conn-str in
    https://github.com/pravega/gstreamer-pravega/blob/master/deepstream/README.md?plain=1#L53 and https://github.com/pravega/gstreamer-pravega/blob/master/deepstream/README.md?plain=1#L53
    need to be updated. routing_key is set in the config file instead of from the uri path right now.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.