GithubHelp home page GithubHelp logo

mirakc / mirakc-arib Goto Github PK

View Code? Open in Web Editor NEW
23.0 3.0 6.0 441 KB

mirakc-tools for Japanese TV broadcast contents

License: Apache License 2.0

CMake 5.89% C++ 92.58% Shell 1.53%
mirakc arib mpeg-ts epg

mirakc-arib's Introduction

mirakc-arib

mirakc-tools for Japanese TV broadcast contents

CI coverage

How to build

Get the repository including the Git submodule:

git clone --recursive https://github.com/mirakc/mirakc-arib.git

Make sure that tools listed below has already been installed:

  • autoconf
  • automake
  • cmake
  • dos2unix
  • gcc/g++
  • libtool
  • make
  • ninja (optional)
  • patch
  • pkg-config

Then:

cmake -S . -B build -G Ninja -D CMAKE_BUILD_TYPE=Release
ninja -C build vendor
ninja -C build
build/bin/mirakc-arib -h

Cross compilation

Use a CMake toolchain file like below:

cmake -S . -B build -G Ninja -D CMAKE_BUILD_TYPE=Release \
  -D CMAKE_TOOLCHAIN_FILE=/path/to/toolchain.cmake
ninja -C build vendor
ninja -C build

Content shown below is a CMake toolchain file which can be used for a cross compilation on a Debian-based Linux distribution, targeting AArch64 like ROCK64:

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(MIRAKC_ARIB_HOST_TRIPLE aarch64-linux-gnu)

set(CMAKE_C_COMPILER ${MIRAKC_ARIB_HOST_TRIPLE}-gcc)
set(CMAKE_C_COMPILER_TARGET ${MIRAKC_ARIB_HOST_TRIPLE})

set(CMAKE_CXX_COMPILER ${MIRAKC_ARIB_HOST_TRIPLE}-g++)
set(CMAKE_CXX_COMPILER_TARGET ${MIRAKC_ARIB_HOST_TRIPLE})

Make sure that a toolchain for the cross compilation has been installed before running cmake with a CMake toolchain file.

Several CMake toolchain files are included in the toolchain.cmake.d folder.

How to test

cmake -S . -B build -G Ninja -D CMAKE_BUILD_TYPE=Debug -D MIRAKC_ARIB_TEST=ON
ninja -C build vendor
ninja -C build test

Logging

Define the MIRAKC_ARIB_LOG environment variable like below:

cat file.ts | MIRAKC_ARIB_LOG=info mirakc-arib scan-services

One of the following log levels can be specified:

  • trace
  • debug
  • info
  • warning
  • error
  • critical
  • off

Why not use tsp?

tsp creates a thread for each plug-in. This approach can work effectively when running a single tsp with multiple plug-ins.

Usages of mirakc-arib are different from tsp:

  • Multiple mirakc-arib sub-commands may be executed in parallel
    • Costs of context switching may be considerable
  • It seems not to be a good idea to construct a mirakc-arib sub-command of multiple plug-ins
    • Communication costs between plug-ins may be considerable

Dependencies

mirakc-arib is static-linked against the following libraries:

The following libraries are used for testing purposes:

TODO

  • Add more unit tests

ARIB STD Specifications

ARIB STD specifications can be freely downloaded from this page.

  • ARIB STD-B10: Additional definitions of tables and descriptors
  • ARIB STD-B24: Character encoding

Acknowledgments

mirakc-arib is implemented based on knowledge gained from the following software implementations:

License

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

mirakc-arib's People

Contributors

dependabot[bot] avatar github-actions[bot] avatar masnagam avatar masnagam-github-actions avatar mirakc-ci[bot] avatar takumin avatar vroad avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

mirakc-arib's Issues

Examination of a method to determine recording start point using PCR

Any TS stream contains PCR packets which are transmitted periodically (within several hundreds milliseconds) in order to synchronize media tracks.

The TS stream also contains TOT/TDT packets which can be used for synchronizing dates and times between the broadcaster and the local device.

Using these packets, it can be possible to pick a pair of synchronized PCR and TOT/TDT with accuracy within several hundreds milliseconds and use it as a baseline for computing a PCR value corresponding the recording start point of a TV program.

Here we assume that the PCR has no gap other than wrapping on overflow.

Replace TSDuck with LibISDB

LibISDB is more suitable than TSDuck in the following points:

  • LibISDB has all features mirakc-arib requires
    • TSDuck does not have some functions like the logo downloader LibISDB has
  • Building TSDuck takes a longer time than LibISDB

Proposal: Support offline build

First of all, thank you to everyone who made this wonderful application!

Background

I want to use buildroot to embed this application.

buildroot requires support for offline builds (or download caches).

However, ExternalProject_Add defined in CMakeLists.txt must be downloaded from the Git repository, so it cannot be included in buildroot at this time.

Proposal

Change the Git download of ExternalProject_Add in CMakeLists.txt to URL download.

Example CMakeLists.txt

It simply replaces GIT_REPOSITORY and GIT_TAG (GIT_SHALLOW is not required).

Before

  ExternalProject_Add(vendor-googletest
    PREFIX
      ${MIRAKC_ARIB_VENDOR_DIR}
    GIT_REPOSITORY
      https://github.com/google/googletest.git
    GIT_TAG
      ${MIRAKC_ARIB_GOOGLETEST_VERSION}
    GIT_SHALLOW
      YES
    CMAKE_ARGS
      -D CMAKE_BUILD_TYPE=Release
      -D CMAKE_INSTALL_PREFIX=${MIRAKC_ARIB_VENDOR_DIR}
      -D CMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}
    EXCLUDE_FROM_ALL
      YES
  )

After

  option(MIRAKC_ARIB_GOOGLETEST_URL "googletest Download URL" "")

  if(NOT MIRAKC_ARIB_GOOGLETEST_URL)
    set(
      MIRAKC_ARIB_GOOGLETEST_URL
      "https://github.com/google/googletest/archive/${MIRAKC_ARIB_GOOGLETEST_VERSION}.tar.gz"
    )
  endif()

  ExternalProject_Add(vendor-googletest
    PREFIX
      ${MIRAKC_ARIB_VENDOR_DIR}
    URL
      ${MIRAKC_ARIB_GOOGLETEST_URL}
    CMAKE_ARGS
      -D CMAKE_BUILD_TYPE=Release
      -D CMAKE_INSTALL_PREFIX=${MIRAKC_ARIB_VENDOR_DIR}
      -D CMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}
    EXCLUDE_FROM_ALL
      YES
  )

Example Buildroot

The following buildroot definitions will be available!

The ability to define MIRAKC_ARIB_EXTRA_DOWNLOADS allows you to send pull requests to the buildroot upstream package.

(The following is a conceptual code, so it will change when you actually send a pull request)

################################################################################
#
# mirakc-arib
#
################################################################################

MIRAKC_ARIB_VERSION = 0.16.3
MIRAKC_ARIB_SITE = $(call github,mirakc,mirakc-arib,$(MIRAKC_ARIB_VERSION))
MIRAKC_ARIB_LICENSE = Apache-2.0 and MIT
MIRAKC_ARIB_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT

MIRAKC_ARIB_GOOGLETEST_VERSION = release-1.10.0

MIRAKC_ARIB_EXTRA_DOWNLOADS += \
	$(call github,google,googletest,$(MIRAKC_ARIB_GOOGLETEST_VERSION))/google-googletest-$(MIRAKC_ARIB_GOOGLETEST_VERSION).tar.gz

MIRAKC_ARIB_CONF_OPTS += \
	-DMIRAKC_ARIB_GOOGLETEST_URL=$(MIRAKC_ARIB_DL_DIR)/google-googletest-$(MIRAKC_ARIB_GOOGLETEST_VERSION).tar.gz

define MIRAKC_ARIB_BUILD_CMDS
	$(TARGET_MAKE_ENV) $(MAKE_ENV) $(MAKE) $(MAKE_OPTS) -C $(@D) vendor
	$(TARGET_MAKE_ENV) $(MAKE_ENV) $(MAKE) $(MAKE_OPTS) -C $(@D)
endef

$(eval $(cmake-package))

Appendix: Reference

Alternative option

Add a Git submodule to create a tar.gz for release.
This may be useful when creating Debian packages etc.

PES packets are dropped before the target TV program is started

mirakc-arib filter --sid=<SID> --eid=<EID> drops PES packets before the target TV program is started.

On the other hands, Mirakurun holds PES packets. The maximum buffer size for holding TS packets regarding the preceding TV programs is defined by maxBufferBytesBeforeReady in server.yml. The value of maxBufferBytesBeforeReady is rounded down by the TS packet size (188) when initializing TSFilter._maxBufferBytesBeforeReady.

time_limit for collect-eits doesn't work correctly

A snippet from mirakc log:

 2020-02-09T02:39:01.750Z INFO  mirakc::tuner                 > tuner#2: Activate with GR 26
 2020-02-09T02:39:01.760Z DEBUG mirakc::tuner                 > tuner#2.68: Spawned 68: `recpt1 --device /dev/px4video6 26 - -`
 2020-02-09T02:39:01.761Z INFO  mirakc::tuner                 > tuner#2.68: Activated with GR 26
 2020-02-09T02:39:01.762Z INFO  mirakc::tuner                 > tuner#2.68.1: Subscribed: Job(eit-collector) (priority=-1)
 2020-02-09T02:39:01.762Z DEBUG mirakc::broadcaster           > tuner#2.68: Started
 2020-02-09T02:39:01.763Z INFO  mirakc::tuner                 > tuner#2.68.1: Started streaming
using device: /dev/px4video6
pid = 68
 2020-02-09T02:39:01.783Z DEBUG mirakc::command_util          > tuner#2.68.1: Spawned 69: `mirakc-arib collect-eits --sids=1032 --sids=1033 --sids=1034`
...
[2020-02-09 02:40:01.902] [collect-eits] [info] EIT: onid(7FE1) tsid(7FE1) sid(0408) tid(0059/59) sec(50:51/F8) ver(07)
[2020-02-09 02:40:03.394] [collect-eits] [info] TOT: 2020/02/09 11:40:04.000
[2020-02-09 02:40:08.372] [collect-eits] [info] TOT: 2020/02/09 11:40:09.000
[2020-02-09 02:40:13.416] [collect-eits] [info] TOT: 2020/02/09 11:40:14.000
[2020-02-09 02:40:18.392] [collect-eits] [info] TOT: 2020/02/09 11:40:19.000
[2020-02-09 02:40:23.366] [collect-eits] [info] TOT: 2020/02/09 11:40:24.000
[2020-02-09 02:40:28.413] [collect-eits] [info] TOT: 2020/02/09 11:40:29.000
[2020-02-09 02:40:33.389] [collect-eits] [info] TOT: 2020/02/09 11:40:34.000
[2020-02-09 02:40:38.433] [collect-eits] [info] TOT: 2020/02/09 11:40:39.000
...

The defualt time_limit is 30s, but the command never terminates after timeout...

`sync-clocks` for CS6 and CS16 fails

There seem to be PCR packets which have no PCR...

$ recpt1 --device /dev/px4video0 CS6 - - 2>/dev/null | MIRAKC_ARIB_LOG=debug mirakc-arib sync-clocks
[2020-02-08 15:35:40.045] [sync-clocks] [info] Read packets from STDIN...
[2020-02-08 15:35:40.045] [sync-clocks] [info] --sids SIDs:
[2020-02-08 15:35:40.045] [sync-clocks] [info] --xsids SIDs:
[2020-02-08 15:35:40.045] [sync-clocks] [info] Feed packets...
[2020-02-08 15:35:41.495] [sync-clocks] [debug] Demux SDT
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0404 for SID#0126 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0402 for SID#0144 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0406 for SID#0149 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0403 for SID#014B ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0407 for SID#0154 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0408 for SID#0155 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0405 for SID#0162 ServiceType(01)
[2020-02-08 15:35:44.294] [sync-clocks] [debug] Demux PMT#0401 for SID#016B ServiceType(01)
[2020-02-08 15:35:44.295] [sync-clocks] [debug] PCR#1004 for SID#0126
[2020-02-08 15:35:44.320] [sync-clocks] [debug] PCR#1006 for SID#0149
[2020-02-08 15:35:44.321] [sync-clocks] [debug] PCR#1008 for SID#0155
[2020-02-08 15:35:44.321] [sync-clocks] [debug] PCR#1007 for SID#0154
[2020-02-08 15:35:44.351] [sync-clocks] [debug] PCR#1002 for SID#0144
[2020-02-08 15:35:44.351] [sync-clocks] [debug] PCR#1003 for SID#014B
[2020-02-08 15:35:44.352] [sync-clocks] [debug] PCR#1001 for SID#016B
[2020-02-08 15:35:44.353] [sync-clocks] [debug] PCR#1005 for SID#0162
[2020-02-08 15:35:44.353] [sync-clocks] [debug] Demux TOT
[2020-02-08 15:35:46.488] [sync-clocks] [info] Time: 2020/02/08 15:35:46.000
[2020-02-08 15:35:46.488] [sync-clocks] [critical] Assertion failed: packet.hasPCR()

`sync-clocks` doesn't finish

2021-04-27T12:03:00.004737984+09:00 DEBUG mirakc_core::command_util: tuner#2.1692: Spawned 8816: `recpt1 --device /dev/px4video2 13 - -`
2021-04-27T12:03:00.005135517+09:00  INFO mirakc_core::tuner: tuner#2.1692: Activated with GR/13
2021-04-27T12:03:00.005279306+09:00  INFO mirakc_core::tuner: tuner#2.1692.1: Subscribed: Job(clock-synchronizer) (priority=-1)
2021-04-27T12:03:00.005428928+09:00 DEBUG mirakc_core::broadcaster: tuner#2.1692: Started
2021-04-27T12:03:00.005552883+09:00 DEBUG mirakc_core::broadcaster: Subscribe with tuner#2.1692.1
2021-04-27T12:03:00.005689672+09:00  INFO mirakc_core::tuner: tuner#2.1692.1: Started streaming
2021-04-27T12:03:00.008122704+09:00 DEBUG mirakc_core::command_util: tuner#2.1692.1: Spawned 8817: `mirakc-arib sync-clocks --sids=23656`

scan-services doesn't finish

log message from mirakc:

mirakc_core::job: update-schedules: acquiring semaphore...
mirakc_core::job: update-schedules: Already running, skip

scan-services got stuck:

$ ps
...
 6582 root     12h57 recpt1 --device /dev/px4video2 13 - -
 6583 root      3h52 mirakc-arib scan-services --sids=23656

config.yml

channels:
  - name: JCOM-TV
    type: GR
    channel: '13'
    services: [23656]

Reimplement collect-logos using LibISDB

Currently, collect-logos works only for GR services. However, BS/CS services also have logos.

There are several descriptors which are required for that but have not been implemented in tsduck-arib.

On the other hand, LibISDB has implemented all descriptors for that. So, it's better to use it for that.

It also may be possible to completely replace tsduck-arib with LibISDB, but it should be processed in another issue.

PCR resync in `record-service` didn't work

Many logs like below were shown:

record-service PCR#01FF: too large delta 7157363997+083 -> 0246959985+008, invalidate the clock for resync

Probably, PCR resync didn't work properly.

Command for the timeshift recording

Design and implement a command for the timeshift recording.

The command has to record a TS stream coming from stdin into a single ring buffer file or multiple files for each TV program. The command also has to record information about recorded TV programs on the filesystem.

The command has to output events regarding the timeshift recording to stdout in a JSON format.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.