Lots of -Warning: bad line- warnings when trying to train from vhs-decode vbi content about vhs-teletext

[ali1234]

Thanks for testing this.

The training split command has a card option in latest git: https://github.com/ali1234/vhs-teletext/blob/master/teletext/cli/training.py#L32

You can use card ddd-tbc instead of slicing the files yourself.

Short .dat files mean some patterns were never observed. This is expected if you use the wrong card type, because lines are checksummed to ensure they are always correctly decoded.

Lots of warnings are also expected because your data has blank lines and video lines in the VBI. Every one will cause a warning. This is not harmful as bad lines are always ignored.

So if you have correctly sized pattern file then it means every possible pattern was correctly observed at least once, which means the training worked as well as it can with the data supplied.

I wouldn't expect the result to be significantly better or worse than the existing patterns. Maybe a little bit better, but perfect decoding is impossible due to bandwidth limitation of VHS. Long play tapes are particularly hard to decode.

[itewreed]

That's interesting. I also made assumptions because I've traditional and vhs-decoded captures of the same VBI data. The traditional data captured with a bt8x8 card looks immaculate compared to vhs-decode capture, after decoding.

I tried the ddd-tbc and ddd-vbi options now, after pulling the latest revision of the software. I shoul've thought about this earlier. However a try on three different systems lead to the following error messages. Doesn't matter if tbc or vbi file. Also doesn't matter if GPU or CPU is used. Two systems are Ubuntu 20.04, one is 21.10.

teletext training split  -c ddd-tbc /home/itewreed/teletext/capture/ttxtrainlp2/ /home/itewreed/nfs_local/decoded/ttxtrain/ttx240lp.tbc
  0%|                                                                                                                | 1/11997936 [00:00<2619:54:32,  1.27L/s, R:0%]/usr/local/lib/python3.8/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number
  warnings.warn('creating CUBLAS context to get version number')
/usr/local/lib/python3.8/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number
  warnings.warn('creating CUBLAS context to get version number')
/usr/local/lib/python3.8/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number
  warnings.warn('creating CUBLAS context to get version number')
/usr/local/lib/python3.8/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number
  warnings.warn('creating CUBLAS context to get version number')
Process SpawnProcess-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 64, in worker
    renumerate(function(denumerate(work, control, tmp_queue), *args, **kwargs), result, tmp_queue)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 34, in renumerate
    r = [next(iterator)]
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 159, in process_training
    for l in lines:
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 157, in <genexpr>
    lines = (TrainingLine(chunk, n) for n, chunk in chunks)
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/line.py", line 77, in __init__
    resample_tmp[:self.config.line_length] = self._original
ValueError: could not broadcast input array from shape (2270,) into shape (1135,)
Process SpawnProcess-1:
Traceback (most recent call last):
Process SpawnProcess-2:
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 64, in worker
    renumerate(function(denumerate(work, control, tmp_queue), *args, **kwargs), result, tmp_queue)
Traceback (most recent call last):
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 34, in renumerate
    r = [next(iterator)]
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 159, in process_training
    for l in lines:
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 157, in <genexpr>
    lines = (TrainingLine(chunk, n) for n, chunk in chunks)
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/line.py", line 77, in __init__
    resample_tmp[:self.config.line_length] = self._original
ValueError: could not broadcast input array from shape (2270,) into shape (1135,)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 64, in worker
    renumerate(function(denumerate(work, control, tmp_queue), *args, **kwargs), result, tmp_queue)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 34, in renumerate
    r = [next(iterator)]
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 159, in process_training
    for l in lines:
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 157, in <genexpr>
    lines = (TrainingLine(chunk, n) for n, chunk in chunks)
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/line.py", line 77, in __init__
    resample_tmp[:self.config.line_length] = self._original
ValueError: could not broadcast input array from shape (2270,) into shape (1135,)
Process SpawnProcess-3:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 64, in worker
    renumerate(function(denumerate(work, control, tmp_queue), *args, **kwargs), result, tmp_queue)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 34, in renumerate
    r = [next(iterator)]
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 159, in process_training
    for l in lines:
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 157, in <genexpr>
    lines = (TrainingLine(chunk, n) for n, chunk in chunks)
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/line.py", line 77, in __init__
    resample_tmp[:self.config.line_length] = self._original
ValueError: could not broadcast input array from shape (2270,) into shape (1135,)
Traceback (most recent call last):
  File "/usr/local/bin/teletext", line 11, in <module>
    load_entry_point('teletext', 'console_scripts', 'teletext')()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/cli/clihelpers.py", line 242, in cmd
    return f(*_args, **_kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/cli/clihelpers.py", line 64, in wrapper
    return f(config=config, *args,**kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/cli/clihelpers.py", line 87, in wrapper
    return f(chunker=chunker, *args, **kwargs)
  File "/home/itewreed/git/vhs-teletext/teletext/cli/training.py", line 56, in split
    split(results, files)
  File "/home/itewreed/git/vhs-teletext/teletext/vbi/training.py", line 193, in split
    for offset, chopped in data:
  File "/home/itewreed/git/vhs-teletext/teletext/cli/training.py", line 52, in <genexpr>
    results = (r for r in results if isinstance(r, tuple))
  File "/home/itewreed/git/vhs-teletext/teletext/stats.py", line 74, in __iter__
    for l in self._lines:
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 267, in itermap
    yield from pool.apply(iterable)
  File "/home/itewreed/git/vhs-teletext/teletext/mp.py", line 153, in apply
    raise ChildProcessError('Worker exited unexpectedly.')
ChildProcessError: Worker exited unexpectedly.

[ali1234]

That's a bug... looking in to it...

[ali1234]

Fixed. Because class "Line" refered to the config through an explicit reference to itself, any configuration set on derived class TrainingLine was ignored. This caused the wrong configuration to be used during training only. This probably affected other capture cards but only caused a crash with DdD because it has 16 bit samples and therefore the array ended up totally the wrong size.

[itewreed]

Testing it right now on the 4 hour tbc file. It runs now successfully. Also only one Warning: bad line message after one hour of 4 hours processed. Will test tomorrow how the trained data will perform.

[itewreed]

The splitted dat files are all zero byte, despite the progress bar looks like it does process.

[ali1234]

If you look at the progress bar, the last item R: indicates the percentage of lines that were rejected. For some reason they are being rejected without emitting the warning. This indicates the problem is now in a different place. Training does still work with bt8x8 captures. I will continue to investigate this.

[ali1234]

Pushed some more changes. Use self instead of Line in more places, and silence the bad line warning. You can look at R: instead to see if it is working.

[itewreed]

Training was successful. The trained files also lead to a better decoding now.