Non-Ascii characters about rtv

[michael-lazar]

I think this is a fair point, I don't have anything against adding unicode. I went with ASCII because I was worried about compatibility with terminal emulators and different system configurations lacking unicode support.

[michael-lazar]

#33

[michael-lazar]

Fixed pull request in 47ad49a.

However, there still appear to be some issues with python3. For example,

$ python2 -m rtv -l http://www.reddit.com/r/LearnJapanese/comments/2ylsz9/request_japanese_audio_textbook_audiobook_or/

loads correctly, but

$ python3 -m rtv -l http://www.reddit.com/r/LearnJapanese/comments/2ylsz9/request_japanese_audio_textbook_audiobook_or/

crashes with the following error

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/michael/Projects/rtv_project/rtv/__main__.py", line 7, in <module>
    main()
  File "/home/michael/Projects/rtv_project/rtv/main.py", line 84, in main
    page.loop()
  File "/home/michael/Projects/rtv_project/rtv/submission.py", line 28, in loop
    self.draw()
  File "/home/michael/Projects/rtv_project/rtv/page.py", line 143, in draw
    self._draw_content()
  File "/home/michael/Projects/rtv_project/rtv/page.py", line 194, in _draw_content
    attr = self.draw_item(subwindow, data, inverted)
  File "/home/michael/Projects/rtv_project/rtv/submission.py", line 95, in draw_item
    return self.draw_comment(win, data, inverted=inverted)
  File "/home/michael/Projects/rtv_project/rtv/submission.py", line 122, in draw_comment
    win.addnstr(row, 1, text, n_cols-1)
_curses.error: addnwstr() returned ERR

Passing --force-ascii fixes the crash, but I would like to figure out why some unicode characters appear to be messing with curses in python3.4.

[0xr0bert]

That doesn't make a lot of sense because this works http://www.reddit.com/r/LearnJapanese/comments/2yonx8/is_there_any_difference_between_%E9%87%8D%E3%81%9F%E3%81%84_and_%E9%87%8D%E3%81%84/ and that has japanese characters in it. However I get the same issue on my computer

[michael-lazar]

I did some investigating into this! It looks like the problem is with how python3 curses.addnstr() calculates the n. The expected behavior is that curses.addnstr(y, x, text, n) will print the first n characters of the_text_ string onto the screen. Here's what's happening.

import curses
import locale
locale.setlocale(locale.LC_ALL, '')

def func(stdscr):

    stdscr.clear()

    text = 'a' * 5
    stdscr.addnstr(0, 0, text, 6)
    stdscr.addnstr(1, 0, text, 5)
    stdscr.addnstr(2, 0, text, 4)

    text = 'あ' * 5
    stdscr.addnstr(4, 0, text, 6)
    stdscr.addnstr(5, 0, text, 5)
    stdscr.addnstr(6, 0, text, 4)

    text = ('あ' * 5).encode('utf-8')
    stdscr.addnstr(8, 0, text, 6)
    stdscr.addnstr(9, 0, text, 5)
    stdscr.addnstr(10, 0, text, 4)


    stdscr.refresh()
    stdscr.getch()
    stdscr.getch()

if __name__ == '__main__':
    curses.wrapper(func)

1. Passing unicode characters within ordinal range(128) works as expected, each character takes up one column of the terminal and up to n characters are printed.
2. Passing unicode characters outside of the ordinal range(128). Up to n characters are printed, but each character takes up two columns in the terminal. So unless we know exactly how much space each character takes, passing in a value for n becomes useless for any practical sense.
3. Passing utf-8 encoded bytes. Now this one is odd, it looks like n is counting the number of bytes passed in. The 'あ' character is 3-bytes long in utf-8. Setting n to 6 allows for 6/3=2 full characters to be printed. Setting n below six cuts off the final character. It is unclear what happens to the partial character bytes. This mode is also practically useless because it doesn't account for the width of each character on the screen.

[michael-lazar]

I don't know if if there is a good way to handle this.

I didn't realize that some unicode characters take up more space than others. This will break how I am currently using textwrap to format paragraphs to fit on the page.

[yskmt]

Yes. I observed too that sometimes a unicode character breaks at the line-change.

[michael-lazar]

Heads up, I changed the default encoding to ascii until this is resolved. 7db8c2f

[michael-lazar]

Interesting reading on the subject http://stackoverflow.com/questions/3634627
and https://pypi.python.org/pypi/wcwidth/0.1.4

[michael-lazar]

I've refactored the code to follow the "unicode sandwich" design method described here. This should make the problem easier to address in the future while considering both py2 and py3.

[firecat53]

We used unicodedata.east_asian_width to help calculate column widths for my curses CSV viewer.

[michael-lazar]

Hey guys! I just merged a large update to the master branch that will hopefully smooth out all of the unicode issues in the codebase. We are now using the kitchen python package, which provides a bunch of unicode-aware text formatting functions. I also set the default program mode to enable unicode, with the option to disable it with the --ascii flag.

@yskmt do you think you could help me test this out, or point me to some non-english subreddits? I would really appreciate it.

[yskmt]

@michael-lazar Sure. Have you implemented it? Is it on master branch?

[michael-lazar]

Yes it is currently on master. Unicode mode should be turned on by default, so all you have to do is checkout master and run it.

[michael-lazar]

I haven't heard any objections so I'm closing this for now. If anybody discovers a problem, please open a new issue.