ali1234 / vhs-teletext

Software to recover teletext data from VHS recordings.

This is a suite of tools for processing teletext signals recorded on VHS, as well as tools for processing teletext packet streams. The software has only been tested with bt8x8 capture hardware, but should work with any VBI capture hardware with appropriate configuration.

This is the second rewrite of the original software. The old versions are still available in the v1 and v2 branches of this repo, or from the releases page.

You can see my collection of pages recovered with this software at:

https://al.zerostem.io/~al/teletext/

And more at:

http://www.teletextarchive.com

And:

http://archive.teletextart.co.uk/

INSTALLATION

In order to use CUDA decoding you need to use the Nvidia proprietary driver.

To install with optional dependencies run:

pip3 install -e .[CUDA,spellcheck,viewer]

If CUDA or pyenchant are not available for your platform simply omit them from the install command.

In order to use OpenCL you need to install pyopencl and the appropriate opencl runtime for your card. Then run the deconvolve command with the '-O' option.

In order for the output to be rendered correctly you need to use a specific font and terminal:

sudo apt-get install tv-fonts rxvt-unicode

Then enable bitmap fonts in your X server:

cd /etc/fonts/conf.d
sudo rm 70-no-bitmaps.conf
sudo ln -s ../conf.avail/70-yes-bitmaps.conf .

After doing this you may need to rehash:

xset fp rehash

Finally open a terminal with the required font:

urxvt -fg white -bg black -fn teletext -fb teletext -geometry 41x25 +sb &

USAGE

First capture VBI from VHS:

teletext record -d /dev/vbi0 > capture.vbi

Deconvolve the recording:

teletext deconvolve capture.vbi > stream.t42

Examine the headers to find services on the tape:

teletext filter -r 0 stream.t42

Split capture into services:

teletext filter --start <N> --stop <N> stream.t42 > stream-1.t42

Display all copies of a page in a stream:

teletext filter stream.t42 -p 100

Squash duplicate subpages, which reduces errors:

teletext squash stream.t42 > output.t42

Generate HTML pages from a stream:

teletext html output/ stream.t42 

Interactively view the pages in a t42 stream:

teletext service stream.t42 | teletext interactive

In the interactive viewer you can type page numbers, or '.' for hold.

Run each command with '--help' for a complete list of options.

vhs-teletext's Issues

Implement a split command

Like the old one for v2. This should read in a t42 and split it out in to separate files for each magazine/page/subpage.

Unlike the old one, it should take a format string that describes how the output should be named/organized. For example:

out/$mag/$page/$subpage.t42

would make a directory called "out", subdirectories for each magazine, then page, and finally a t42 for each unique subpage number.

Don't import a module as deconvolution parameters

Currently the capture card settings are imported as a python module. Instead, a text file should be used.

"--card" option produces error

It seems that "--card" option is not working good. Using this option produces stacktrace:

D:\vhs-teletext-master>teletext deconvolve --card saa7131 -C D:\teletext\FULLs\БТ\full.vbi
CUDA disabled by user request.
  0%|                                                                                        | 1676/2976265 [00:00<92:34:36,  8.93L/s, R:100%, M:|        |]Process SpawnProcess-2:
Process SpawnProcess-3:
Process SpawnProcess-1:
Traceback (most recent call last):
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 36, in slave
    renumerate(function(denumerate(quit_event, work_queue, tmp_queue), *args, **kwargs), done_queue, tmp_queue)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 36, in slave
    renumerate(function(denumerate(quit_event, work_queue, tmp_queue), *args, **kwargs), done_queue, tmp_queue)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 36, in slave
    renumerate(function(denumerate(quit_event, work_queue, tmp_queue), *args, **kwargs), done_queue, tmp_queue)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 27, in renumerate
    for item in iterator:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 27, in renumerate
    for item in iterator:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 27, in renumerate
    for item in iterator:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 222, in process_lines
    yield getattr(Line(chunk, number), mode)(mags, rows)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 222, in process_lines
    yield getattr(Line(chunk, number), mode)(mags, rows)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 222, in process_lines
    yield getattr(Line(chunk, number), mode)(mags, rows)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 165, in deconvolve
    if not self.is_teletext:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 165, in deconvolve
    if not self.is_teletext:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 165, in deconvolve
    if not self.is_teletext:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 130, in is_teletext
    elif self.noisefloor > 80:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 130, in is_teletext
    elif self.noisefloor > 80:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 130, in is_teletext
    elif self.noisefloor > 80:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 107, in noisefloor
    self._noisefloor = np.max(gauss(self._original[:self.config.start_slice.start], self.config.gauss))
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 107, in noisefloor
    self._noisefloor = np.max(gauss(self._original[:self.config.start_slice.start], self.config.gauss))
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 107, in noisefloor
    self._noisefloor = np.max(gauss(self._original[:self.config.start_slice.start], self.config.gauss))
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 2505, in amax
    initial=initial)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 2505, in amax
    initial=initial)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 2505, in amax
    initial=initial)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity
ValueError: zero-size array to reduction operation maximum which has no identity
ValueError: zero-size array to reduction operation maximum which has no identity
Process SpawnProcess-4:
Traceback (most recent call last):
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 36, in slave
    renumerate(function(denumerate(quit_event, work_queue, tmp_queue), *args, **kwargs), done_queue, tmp_queue)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\mp.py", line 27, in renumerate
    for item in iterator:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 222, in process_lines
    yield getattr(Line(chunk, number), mode)(mags, rows)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 165, in deconvolve
    if not self.is_teletext:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 130, in is_teletext
    elif self.noisefloor > 80:
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\teletext-3.0.0-py3.7.egg\teletext\vbi\line.py", line 107, in noisefloor
    self._noisefloor = np.max(gauss(self._original[:self.config.start_slice.start], self.config.gauss))
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 2505, in amax
    initial=initial)
  File "C:\Users\Byteman\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy-1.16.2-py3.7-win32.egg\numpy\core\fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

OS: Windows 10
Python version: 3.7

Write unit tests

A lot of the code would benefit from this. Especially coding, elements, packet and service.

More documentation is needed for training new signals

A short tutorial on how to train your VCR would be appreciated.

Improve results when subpage receives multiple copies of the same row

Sometimes the stream will contain duplicate rows, and in some cases they can be mostly empty. This appears to be caused by incorrectly deconvolved filler packets.

Currently subpage reads packets in broadcast order and insert them into a table. At the end the table will contain the last example of each row/dc received. When there are a lot of filler packets, this means those rows nearly always end up empty.

filler-error.t42.gz

Fix-up and re-factor Line

This goes along with #13. The deconvolution code in Line could really use a general clean-up.

Write a differential decoder for live broadcast captures

The deconvolving method used for VHS recordings is not compatible with live broadcasts. They can be decoded much easier and faster with a differential decoder anyway. This could be implemented for completeness. Depends on #17.

VBI viewer crashes when trying to display some view types

This is due to missing or renamed properties. Pressing various number keys causes a crash because the property containing the data is no longer available.

Implement squashing page enhancement packets

Placeholder :)

No teletext data from deconvolve

For some reason I am not able to get any teletext data from the VBI files. I've captured the data through the vbicat /dev/vbi0 > capture.vbi then tried to run deconvolve -H -S 20 -T 4 capture.vbi > headers.txt. However, once that completes, it states that:
0:01:02 : 12001 lines, 193/s total, 0/s teletext, 100% rejected.

I assume this to mean that it wasn't able to find any teletext data in the VBI data I captured. This is probably due to an issue with my setup, but I can watch the video contents of the VHS tape fine using xawtv, so it is not a driver issue. Additionally, vbicat accepts the data from /dev/vbi0 fine, and only occasionally displays a Frame dropped? message.

I've tried several different VHS tapes, but no luck with any of them. I've uploaded a sample VBI file which is from a BBC 2 program that is definitely before 2012, in the hope that some kind person who has a known working setup can confirm if it is an issue with my hardware, or my software setup. (My video player is a cheap Matsui from the late 1990s, so perhaps I need a better player?). When I try and run deconvolve on this particular VBI file, I get the following output:

No module named pycuda.driver
CUDA init failed. Using slow CPU method instead.
No module named pycuda.driver
CUDA init failed. Using slow CPU method instead.
No module named pycuda.driver
CUDA init failed. Using slow CPU method instead.
No module named pycuda.driver
CUDA init failed. Using slow CPU method instead.
0:01:02 : 12001 lines, 193/s total, 0/s teletext, 100% rejected.

I have attached the headers.txt file that this generates, in case it is of use to someone more knowledgeable than me.

I am really excited to discover this project, and hope to begin retrieving teletext pages soon. Huge thanks to the developers of such a fantastic project.

"Cannot reshape array" error when deconvolving data from SAA7131 capture card

Hi,

I'm getting error "cannot reshape array" when doing deconvolve process on any data from SAA7131 capture card. I've tried to recover some of my old captures, but no success.

Here is tracelog:

Traceback (most recent call last):
File "/usr/local/bin/teletext", line 11, in
load_entry_point('teletext==3.1.99', 'console_scripts', 'teletext')()
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/teletext/clihelpers.py", line 235, in cmd
return f(_args, **_kwargs)
File "/usr/local/lib/python3.6/dist-packages/teletext/clihelpers.py", line 62, in wrapper
return f(config=config, *args,**kwargs)
File "/usr/local/lib/python3.6/dist-packages/teletext/clihelpers.py", line 198, in wrapper
for p in packets:
File "/usr/local/lib/python3.6/dist-packages/teletext/pipeline.py", line 81, in to_file
for p in packets:
File "/usr/local/lib/python3.6/dist-packages/teletext/stats.py", line 48, in iter
for p in self._packets:
File "/usr/local/lib/python3.6/dist-packages/teletext/cli.py", line 324, in
packets = (p for p in packets if isinstance(p, Packet))
File "/usr/local/lib/python3.6/dist-packages/teletext/stats.py", line 74, in iter
for l in self._lines:
File "/usr/local/lib/python3.6/dist-packages/teletext/mp.py", line 264, in itermap
yield from pool.apply(iterable)
File "/usr/local/lib/python3.6/dist-packages/teletext/mp.py", line 212, in apply
yield next(self._proc)
File "/usr/local/lib/python3.6/dist-packages/teletext/vbi/line.py", line 291, in process_lines
yield getattr(Line(chunk, number), mode)(mags, rows)
File "/usr/local/lib/python3.6/dist-packages/teletext/vbi/line.py", line 231, in deconvolve
bits_array = normalise(self.chop(0, 368))
File "/usr/local/lib/python3.6/dist-packages/teletext/vbi/line.py", line 126, in chop
return np.mean(self.fchop(start, stop).reshape(-1, 8), 1)
ValueError: cannot reshape array of size 2931 into shape (8)

Here is a short VBI sample where error occures.
polsat_crop.zip

Thanks,
Alexander

Fix HTML generation

This is currently broken on v3.

filter allows incompatible options

running filter with a non zero -r prevents it from matching any pages/subpages to -p and -s

corrupt page headers

There's an off-by-one error in the decoding of page header control bytes in PageHeader from_bytes

control = (values[0] >> 7) | (values[1] >> 6) | (values[2] << 3)

should be

control = (values[0] >> 7) | (values[1] >> 5) | (values[2] << 3)

Missing functions

Hi Alistair

Printer.py is missing do_print()

Line.py is missing mrag_decode()

It won't run without them.

Jason

Catch exceptions when optional extras are not installed

Currently you'll just get a stack trace and ImportError if you try to eg use teletext spellcheck without installing pyenchant. These exceptions should be caught and a simple error message displayed.

Extend the subpage class to store page enhancement packets

This involves adding more rows to the subpage array. The extra row positions need to take in to account both the packet row and also the designation code for packet 27.

Blocks #34

'DisplayPacket' object has no attribute 'page_str' on running t42html

Hi Alistair,

I'm getting this on the latest commit:

Traceback (most recent call last):
File "D:\My Documents\vhs-teletext2\t42html.py", line 14, in
service = make_service(sciter(reader(open(sys.argv[1]))))
File "D:\My Documents\vhs-teletext2\teletext\t42\pipeline.py", line 85, in make_service
for s in paginate(packet_iter, pages=pages, yield_func=subpages):
File "D:\My Documents\vhs-teletext2\teletext\t42\pipeline.py", line 52, in paginate
if magbuffers[mag][0].page_str() in pages:
AttributeError: 'DisplayPacket' object has no attribute 'page_str'

Support rendering enhancement data

The enhancement data can't be displayed when writing individual packets to stdout - it would require rewinding the terminal and editing previous packets. However it should be possible to support them in interactive, html, and url commands.

No timeline for this feature. :)

• interactive
• html
• url : see https://github.com/rawles/edit.tf/wiki/Teletext-page-hashstring-format

What type of Capture Card do I need to capture raw data?

Hi I would realy love to capture some of my VHS Tapes and extraxt teletext from them but I don't have Hauppauge WinTV PCI card but I have Gigabyte GT-U8000-RH: so I wonder can I use that to capture raw vbi data?

Thanks for Anwsering and Best Regards

Implement parsing for the rest of the packets

Reading:

• X/26
• X/27
• X/28
• X/29
• X/30
• X/31

Writing:

• X/26
• X/27
• X/28
• X/29
• X/30
• X/31

Blocks #35

Limit recording time

Could you add a parameter to limit the recording time? I now manually have to stop the recording. Would be nice if this could be done automatically after a certain amount of minutes.

What to look for in headers.txt?

I'm not sure what to look for in the deconvolved headers.txt file.

The file is a binary file, which is something less warns me about. So when I see it, it's mostly gibberish.

Am I missing some step?

Add an error message when the terminal isn't big enough for interactive viewer

Instead of just crashing...

TypeError: slice indices must be integers or None or have an __index__ method

I've just bought an old PCI Hauppauge winTV card (with the 878 chip) to experiment with your software, but I'm getting the following error when I try to deconvolve my VBI capture:

./deconvolve -H -S 20 capture_trunc.vbi > headers.txt
No module named pycuda.driver
CUDA init failed. Using slow CPU method instead.
Traceback (most recent call last):
File "./deconvolve", line 102, in
for l_list in split_seq(it, args.squash):
File "./deconvolve", line 94, in split_seq
item = list(itertools.islice(it, size))
File "/home/johnnyk/Documents/dev/vhs-teletext/teletext/vbi/map.py", line 82, in raw_line_map
for l in map_func(func, raw_line_reader(filename, line_length, start, stop)):
File "/usr/lib/python2.7/multiprocessing/pool.py", line 269, in
return (item for chunk in result for item in chunk)
File "/home/johnnyk/Documents/dev/vhs-teletext/teletext/vbi/map.py", line 24, in wrap
return func(self, timeout=timeout if timeout is not None else 1e100)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
TypeError: slice indices must be integers or None or have an index method

Unfortunately I can't try the newer CUDA method, so I'm forced to use the CPU. Have you seen anything like this before? Possibly my VBI capture is just bad, but I don't know if I need to do any setup for that. I have attached a truncated excerpt of the cap. which reproduces the error.

This is how my card is set up for capture at the moment:
v4l2-ctl -d /dev/vbi0 --get-fmt-vbi
Format VBI Capture:
Sampling Rate : 35468950 Hz
Offset : 244 samples (6.87926e-06 secs after leading edge)
Samples per Line: 2048
Sample Format : GREY
Start 1st Field : 7
Count 1st Field : 16
Start 2nd Field : 320
Count 2nd Field : 16

Cheers for any insight you can give, apologies if this is user error.

capture_trunc.tar.gz

John.

Add support for extended hashstring format to the URL command

The url subcommand doesn't currently support this, which means it loses page metadata like page number and fastext links.

The format is documented here: https://github.com/rawles/edit.tf/wiki/Teletext-page-hashstring-format

List of tags which are defined:

• PN - Page Number
• PS - Page Status
• SC - Subpage Code
• X25 - Packet data for row 25
• X26 - Local enhancement data and object definitions
• X270 - Editorial linking
• X280 - Packet X/28/0 format 1 enhancement data
• X284 - Packet X/28/4 enhancement data
• zx - zxnet editor settings

We probably can't support all of these.

t42 format description

Hi first of all amazing work :)

I am interested in having a dump of Teletext page so that I could see what packets build it and them interpret it by "hand" for debug purposes.

My main goal would be to extract those pages from a ts and then examine it.

The file format .t42 seems to fit my needs but I don't know how it is to be read ? Is there any description of the format?

Thanks in advance!

Fix-up and re-factor training

One more thing that still needs to be cleaned up.

Implement checksum generation/verification

We can now decode the checksums from packet X/27. We need to be able to verify it and generate for new pages.

Javascript implementation of the CRC from @ZXGuesser:

    function calculatePageCRC(){
        /* calculate the CRC checksum of a page per ETS 300 706 9.6.1 */
        
        var checksum = 0;
        
        for (var c=8; c<32; c++){
            checksum = calculateCRCByte(setParity(renderer.level1PageArray[0][c]), checksum);
        }
        
        for (var r=1; r<25; r++){
            for (var c=0; c<40; c++){
                checksum = calculateCRCByte(setParity(renderer.level1PageArray[r][c]), checksum);
            }
        }
        
        // TODO: use a real packet 25
        for (var c=0; c<40; c++){
            checksum = calculateCRCByte(setParity(0x20), checksum);
        }
        
        return checksum;
    }
    
    function calculateCRCByte(pageByte, checksum){
        for (var b=7; b>-1; b--){
            var outBit = ((pageByte>>b)&1) ^ ((checksum>>6)&1) ^ ((checksum>>8)&1) ^ ((checksum>>11)&1) ^ ((checksum>>15)&1);
            checksum = outBit | ((checksum&0x7FFF)<<1);
        }
        
        return checksum;
    }

Note that there is a bug in the teletext spec regarding checksum calculation. It says:

During a sequence of 8 192 clockpulses bytes 14 to 37 from packet X/0 and the following character bytes (bytes 14 to 45) of packets X/1 upto X/25, in ascending address order, form the input.

"(bytes 14 to 45)" appears to be incorrect - it does not cover the displayed bytes of rows X/1-25. The javascript code above appears to be a correct implementation as far as we know.

Cannot deconvolve any data using v2 software

Hi!

I've tried to perform deconvolution of raw vbi data captured from SAA7231 based capture card. It produces 1440 samples per line without black samples on the right. I've edited config.py file by modifying saa7131 config (1440 instead of 1600 lines), also default "card" type was hardcoded because of "--card" option issue.
I get no meaningful data in the produced t42 stream, but with "v1" software I was able to decode at least something.

Could you have a look at sample data and help to determine the root cause of no data decoded, and help to "tweak" deconvolve if needed?

Capture card: Behold H8 (SAA7231), 1440 samples per line
VCR: Panasonic AG-4700 (edit mode)
Source: SECAM recording, RTR channel (1995, Russia)
Sample: https://files.byteman.by/vhs-teletext/RTR_1995_SAA7231.zip (about 40 minutes of capture)

Thanks in advance.

Implement charset support for different languages

This got skipped because it isn't really practical to implement it when printing packets one at a time to the console. However, now that there are other types of output, it should be implemented.

This line makes no sense

vhs-teletext/teletext/pipeline.py

Line 47 in 515d3b2

numbers = mode(np.stack([np.clip(sp.numbers, -100, -1) for sp in splist]), axis=0)[0][0].astype(np.int64)

Qt Teletext decoder and editor

Already in progress. Features/to do:

• Basic rendering.
• Load single page from file.
• Hide next character when double width is present.
• Hide next row when double height is present.
• Decoder mode to play back streams.
• Mouse navigation in decoder mode.
• Editing mode.
• Selection with copy/paste.
• Mouse drawing.
• Conceal
• Box

Support new Unicode 16.0 Teletext code points

This will require some refactoring to the way mosaics are handled as the new code points do not define separate glyphs for solid and separated mosaics.

Related links:

https://www.unicode.org/L2/L2019/19025-terminals-prop.pdf

https://gitlab.gnome.org/GNOME/vte/issues/189

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944609

GPU deconvolve does not work on my system

When i run deconvolve, my terminal is spamming several lines of:
"/usr/local/lib/python3.6/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number"
followed by several lines of:
"error invoking 'nvcc --version': [Errno 2] No such file or directory: 'nvcc': 'nvcc'
CUDA init failed. Using slow CPU method instead."

i have to run the deconvolve command as root. it complain about not finding pycuda when running without root.

it would be nice to get GPU deconvolving to work. CPU deconvolve works, but it takes 8+h on a 4h capture, using all cores.

when i capture, is there any capture settings i should use. like enable or disable AGC?. view the video signal while capturing for the AGC to balance the picture. i think i lost a bit of vbi data from a piece of vhs recording because i did not view the video signal while capturing. my capture setup involves two bt878 cards. one card for capturing vbi and one card for viewing the live vbi signal.

OS: linux mint 19.3
Kernel: Linux 5.3.0-40-generic x86_64
GPU: RTX 2070
NVIDIA driver: 440.64
CUDA version: 10.2

Teletext interactive not showing magazine 800

Teletext interactive works for pages 100-799, but once you select pages 800-899, they cannot be found (while they are present in the stream).

Bring back the old squashing algorithm from V1

In V1, subpages would be compared and only squashed if some percentage of characters (not including whitespace) were a match.

This code was removed when testing showed that the deconvolution algorithm had become good enough to nearly always get the correct subpage code. However, it turns out that often the subpage codes are not broadcast correctly - either every subpage is "0001", or it can be used for another purpose.

The upshot of this is that those broadcasts won't squash correctly in V3 because completely different subpages get squashed together, which ruins them.

So this issue is about forward porting the old algorithm, with an option to turn it on/off.

Inconsistent numpy imports

Some places use import numpy and others use import numpy as np. These should all be changed to the latter.

Implement index command

This should list all pages and optionally subpages present in a t42 file.

WinTV USB support?

Hello, I'm trying to capture teletext data from VHS through a WinTV USB device, model 40214; I'm doing that with GraphEdit under 32 bit Windows (Linux's usbvision driver doesn't create the /dev/vbiX device) and the result is the file attached here. The device produces 704 samples per line, and with vbiview I can see that the teletext data is definitely there, but somehow "off-centered", and I can't deconvolve anything.
I also own a WinTV PCI card, and using the same Windows software for capture I can get good results (though I had to set "line_length" to 2000 and "field_lines" to 18 in vbi/config.py) but I would like to use the USB one because it gets a cleaner video signal, and has S-Video input. Could you please have a look to my data and help me to fix the USB capture?
Thanks in advance.

Sample from USB: https://drive.google.com/file/d/14c7Rz65DmjCCjJxFcpMaysbRKn8DlbYZ/view
Sample from PCI: https://drive.google.com/file/d/1MeGCUrUARYyrA9FGw7SpCVvD98N0uK3e/view

Provide files for Betamax and Video 2000

I know vhs-teletext comes with patterns for VHS recorders, but it would be soooo cool for teletext fans like me to actually be able to get the signals from Betamax and Video 2000 tapes as well. I know vhs-teletext can create training files but not all of us have the technical equipment needed for this. I suppose a switch could be added to vhs-teletext to indicate what kind of tape we want to read, instead of overwriting teletext/vbi/data each time? Who is willing to share their training files?

Get rid of CUBLAS warnings

Is it possible to remove the confusing warning 'creating CUBLAS context to get version number'? There is no need to show that message.

--step seems to be broken

eg if step is 1, instead of 1, 3, 5, 7... you get 1, 2, 3, 4 up to the half-way point in the file.

Refactor multiprocess deconvolution pool

This should be refactored to be a class, allowing work to be submitted at any time, with results fetched asynchronously.

itermap() should then be refactored as a light wrapper around this class.

Blocks #32

Documentation is out of date

README.md is not too bad but TRAINING.md and HOW_IT_WORKS could really use a re-write.

Implement extra stats for tqdm progress bars

The old speed monitor code measured rejects. This should be re-implemented for the new tqdm progress bars. Also it would be nice to optionally see magazine and row stats.

Training creates hamming files that are the wrong length for PatternCUDA

Hi Alistair,

I piped the output from training -g to raspi-teletext, recorded it on Betamax, then captured it using vbicat. I then did a 'training -t' on the vbi file, then on the output of that did --parity, --hamming and --full (not sure if 'full' corresponds to debruijn, but that's a different matter).

The hamming file isn't a multiple of 1024 so PatternCUDA won't accept it. Am I doing something wrong or is there an issue with the training script?

Thanks

Jason

New Qt VBI Viewer

Things to do:

• Raw VBI display
• Teletext display
• Timeline widget
• Line cache
• Metadata
• VBI Export

Fix finders

Currently finders don't work at all. It seems to not make a big difference in results, but it would be nice to fix them and extend them to extract useful information from the data.