aligrudi / neatvi Goto Github PK

languages such as shell or python or sh most likely always have a #!/bin/sh or similar at the first line of the file, used by the kernel to execute.

Hello Bro.
as an Arabic user I have to use some special Arabic symbols like ( ْ ِ ٍ ٌ ُ ً َ ), for some reason in Neatvi It's rendered as (_).
Example:
دَعوني أُوَفّي السَيفَ في الحَربِ حَقَّهُ وَأَشرَبُ مِن كاسِ المَنِيَّةِ صافِيا
وَمَن قالَ إِنّي سَيِّدٌ وَاِبنُ سَيِّدٍ فَسَيفي وَهَذا الرُمحُ عَمّي وَخالِيا
In Neatvi It's:

or:
ـﻪــﻘـﺣ ـبﺮـﺤﻟا ﻲﻓ ـﻒﻴـﺴﻟا ﻲـﻓـوـأ ﻲﻧﻮﻋـد ﺎﻴـﻓﺎﺻ ـﺔــﻴـﻨـﻤﻟا ـسﺎﻛ ﻦـﻣ ـبـﺮﺷـأـو
ـﺪــﻴـﺳ ـﻦﺑـاـو ـﺪــﻴـﺳ ﻲـﻧـإ ـلﺎﻗ ﻦـﻣـو ﺎﻴـﻟﺎﺧـو ﻲـﻤـﻋ ـﺢﻣـﺮﻟا اﺬـﻫـو ﻲﻔﻴـﺴـﻓ
Sadly, I don't know C, so I can't figure out what is the problem

Hello Ali, is it possible to somehow get the functionality of a negated char set but for more than 1 character?

For example see my current expression here I am trying to solve:
https://github.com/kyx0r/neatvi/blob/master/conf.c#L58

What it does is look for */ in the string and colors everything that goes backward from the */ until it finds the closing / or it doesn't
find it. This is used for C multiline comment implementation and generally it works fine in %90 of the cases. But my problem is that I only match one character as delimiter which is / but to be a 100% accuracy it needs to match /* which is 2 characters. But the problem with posix regex is that it does not support look behind and look ahead, but the negated char can only be 1 character.

Is there any way to construct a regex in such a way that it does what I need & is properly delimited?

Hi. and thank you so much for this great repository.
I compiled neatvi with default settings. But i realized that farsi characters shaping only works in st terminal emulator:

I've tested with tilda, tilix, xfce4-terminal. But shaping characters failed in all of them.

Hello @aligrudi I was looking at commit history and I find that this commit 0d6850f
has a bug. The problem is that function conf_placeholder can modify the wid variable without the character actually being a placeholder, so for example if the character is 'a' and last placeholder in the placeholders array is width of 5, the 'a' character will be treatead as width of 5, which is obviously wrong. The only saving grace from having this bug appear is that currently there isn't any placeholder in array to be wider than 1.

Also see my code for proper implementation.
https://github.com/kyx0r/neatvi/blob/fee6b5a1acccc4bec8ed14eff61f5a362b1b8d75/ren.c#L176

To be honest, I am quite skeptical of this change in the first place, as those extra checks actually hold an astunding performance implication. What is your take on the issue?

Take a look at my regex.c https://github.com/kyx0r/neatvi/blob/master/regex.c
I can't think of any more ways to make it faster now, (unless rewrite completely from scatch using a diffrent approach described in many research papers.) I was able to cut down cpu usage by twice which is very substantial already. Overall the net patch will probably cost you +25 more lines on code, but it will definitely be worth it. Also you will have way less active memory usage and allocations, and thus faster compile times despite the fact that I precompute the bracket expressions. It's just so much better, also the problem is solved ireratively, but it still behaves like recursion would, but zero overhead of stackframe allocations and useless copy of data. Also the marks array for subs is reset to -1 for every character, well that was a huge bottleneck, so instead of memsetting the whole array each time I only reset a max portion that was used and it causes a massive speed up!

I am reading the code cause I would like to know if neatvi supports hidden buffers that are in a modified state. It seems that the answer is no, but I prefer asking the question to be sure.

Nvi allows this when using bg and fg commands. I am trying to understand how buffers manipulation works in neatvi. I see there is an autowrite setting, but wondering if a hidden buffer should obviously be saved on disk?

Thanks!

In MacOS
with neatvi latest pull.

In C file:

upper window = neatvi
lower window = vim

Salam (Hi),

thanks for neatvi. Is it possible to add the main feature of neatvi, i.e. BiDi support, to Vim? Vim's present support for Persian language is limited and flawed. Neatvi is light and neat, however, my Vim is beefed up with a few packages that neatvi (or vi) doesn't support them.

thanks

Using gUw on the last word of a line, does not show the change until Ctrl+l is pressed. It also puts the cursor onto the first char of the next line (probably related to #10).

Hi,

As neatvi has neither a command history nor a cedit (like in nvi) to edit command history, how is your workflow to re-enter previous (or slightly modified previous) commands?

I can see it has a special buffer : that contains the previous ex command, and it also has ^p to paste the default buffer in normal mode. So I am able to paste the default buffer in an ex command with ^p, but I cannot paste from other buffers (such as :).

Do you have a workaround to accomplish such a thing?

Thanks

FYI neatvi does not support less's v command because of the missing plus option.
You have to define LESSEDIT="%E %g" as a workaround.

May not really be an issue for everyone, however if your terminal happen to support pasting like st does, you will find that doing so in neatvi will totally mess up the indenting, and the bigger the paste, the more shifted to the right text will become. I am not a big fan of that, so if you want to disable this just set
xai = 0
in ex.c

Though since technically pasting is the same as just dumping text character by character to get the right output vi has to be in insert mode. One thing I don't like though is that it does preserve the formatting, it seems like the terminal copies blanks as spaces and so the code will end up formatted with spaces instead of tabs. I will patch this out at some point in my personal version of neatvi.

Hello,

From what I can see, neatvi does not provide file completion such as what is done for example in nvi.

I guess it might be too complex and so that is why it is not implemented?

And if that is the case, how do you deal with filepath? Do you type them blindly in ex commands? Or do have tricks that make use of gf to open files under the cursor, or maybe you only navigate using tags?

Thanks

Given the text foo bar baz and the cursor being on the 'b' of "bar", pressing cw and entering quxEsc leaves the text as foo quxbaz. That behaviour makes sense for dw but is awkward for cw.

When changing or deleting the last word of a line, the newline gets swallowed resulting in joining with the next line.

There is a BiDi/RTL standardization effort. And it is already supported by VTE library (not yet released though), see https://terminal-wg.pages.freedesktop.org/bidi/implementations.html#vte

See also https://gist.github.com/XVilka/a0e49e1c65370ba11c17 for tracking other terminals and programs.

@aligrudi xcol is not updated because mod is set to 0, but xoff is being changed.

The following commit in my fork fixes the bug.
kyx0r/nextvi@a408170

Also take a note that the function vi_scrollforeward() has a typo, (should be vi_scrollforward) I addressed that in future commits.

I'm currently window shopping for an editor because my current linux distro does not package nvi and building it is quite a pain in the butt. There also seems to be a need to heavily patch it (most linux distros do). I'm not going to touch that myself...

Now, neatvi ticks almost all the boxes. What I'm also missing is ex-mode history and file auto-completion but I can live without those.

A bigger issue for me is the lack of the ability to filter ranges through an external command. Being able to do that is a rather crucial feature in my opinion, as it gives a lot of power. Is there some kind of roadmap for this project or a general list of features you want to implement and things you will never do (like split windows for instance)?

Hello @aligrudi, Japanese characters are rendered as placeholders. Does this indicate that only a certain range of UTF-8 is supported, anything that is above 3 bytes except the Arabic symbols. How do I implement a complete UTF support so that characters are not placeholders and their lengths are correctly interpreted by vi?

Currently if one does any successful %s// replacement and then uses u to undo it, the screen always jumps to line 0 of the open file. I don't think this makes sense and is sort of annoying. What it should do instead is jump to the first change that has been done by %s// in lbuf and of course undo the result, OR it should just stay in the same place, no jump. @aligrudi

Is neatvi can read Hebrew files? how to set this?

cmd.c line 111,112 may have -1 passed in which is invalid. Add if statements to prevent this.

On my hardware neat-vi hits major performance issue when faced with huge lines, about
6000 characters is enough to make it unusable and slow even with syntax highlight disabled.
Moving around is really slow and fills the cpu core to 100%.

@aligrudi I took some time to investigate the cause and it was happening because
the function dir_reorder() is extremely slow in such cases. The algorithm is O(n)
where n is the number of characters in the line. But what is worse is that any motion key
has to go through dir_reorder(). But that wasn't the only cause for bad performance,
with it disabled things still did not look good and it was easy to feel the lag. The syn_highlight
is also extremely inefficient and does a O(n) regex search on entire line for no reason. Finally
a lot of the code has redundant computations, un_slen() does not scale well either.

I managed to solve most of these problems in my fork.
First I run dir_fix() only on the visible lines so instead of it being
O(n) it becomes O(c) where c is the number of terminal cells.
Secondly, the same strategy is used for syntax highlight
but regex is done only from cell 0 to number of terminal cells.
So as you can see the kind of panning essentially reduces
number of operations required to a constant. There is no
need to process things that will not get displayed on the
terminal anyway.
Finally, I have cleaned up the redundant mallocs and uc_slen(). (Not all of them yet)
After solving all these, the scrolling works fine no matter how
huge the line gets, and it's performance moved to a reasonable level.

Ali, what do you think about these changes? Maybe you can take a look
what else can be improved, and maybe would be reasonable to add these
changes into your base version?

Regards and Happy New Year!

@aligrudi Hello Ali, I found a weird corner case when lbuf_pair does not work.

Consider the following text:

'☺' ) {
}

paste it, and try using % while cursor is on { character, it will not work. Oddly enough it works if you do it from } character, or
if you remove the smiley face or if you remove the ) character

Hello,

When building the project with make, it leaves a lot of build-related files untracked in git.

This would benefit from having a .gitignore to avoid listing them in git.

Would you be accepting a PR for that?

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        cmd.o
        conf.o
        dir.o
        ex.o
        lbuf.o
        led.o
        mot.o
        reg.o
        regex.o
        ren.o
        rset.o
        rstr.o
        sbuf.o
        syn.o
        tag.o
        term.o
        uc.o
        vi
        vi.o

Thanks

:5|10|15 should result in line 15.
kyx0r/nextvi@d200131

Hi!
The next line is affected, this behavior doesn't seem standard to me.

The read ex command fails to read from pipes. I tested r !seq 10 and it does not insert anything into the buffer.

If you enter vi_prompt() via pressing ':' and press ^t (or ctrl+t) will result in segfault. Might be low priority to fix, but if you ever press that accidentally you might loose unsaved data.

I know neatvi aims to be small, but why did you choose ‘int’ variables for buffer sizes and cursor positions, instead of ‘size_t’ ? Even if neatvi is small, using ‘size_t’ variables would make it possible to open huge files.

BTW, can cursor positions (or sizes) become negative? If they can never be negative, maybe it’s straightforward to just replace the proper ‘int’ variables by ‘size_t’ ones.

Hello @aligrudi re_groupcount() is incorrectly computing the number of groups in the regex if the bracket expression has brackets inside of it. Regex engine does not support escaped brackets (and it should not because it's useless either way). Though that function didn't even try to check for escapes. But anyway, just delete that code it is redundant and causes problems.

The following patch simplifies the function and fixes the bug.

From ecdfce5414ba1c471a48f70f0ac691d7ea9953e5 Mon Sep 17 00:00:00 2001
From: Kyryl Melekhin <[email protected]>
Date: Sun, 12 Sep 2021 10:09:02 +0000
Subject: [PATCH] fix re_groupcount() to only count groups

---
 rset.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/rset.c b/rset.c
index 553b4fd..88672ba 100644
--- a/rset.c
+++ b/rset.c
@@ -15,25 +15,10 @@ struct rset {
 
 static int re_groupcount(char *s)
 {
-	int n = 0;
-	while (*s) {
-		if (s[0] == '(')
+	int n = *s == '(' ? 1 : 0;
+	while (*s++)
+		if (s[0] == '(' && s[-1] != '\\')
 			n++;
-		if (s[0] == '[') {
-			int dep = 0;
-			s += s[1] == '^' ? 3 : 2;
-			while (s[0] && (s[0] != ']' || dep)) {
-				if (s[0] == '[')
-					dep++;
-				if (s[0] == ']')
-					dep--;
-				s++;
-			}
-		}
-		if (s[0] == '\\' && s[1])
-			s++;
-		s++;
-	}
 	return n;
 }
 
-- 
2.33.0

Kind regards,
Kyryl.

Hello Ali, this question has been on my mind for a very long time parlially because I can't understand why it happens.
To be fair this likely isn't a problem for the general user, that being a rare edge case where the text rendered by led_printparts() gets printed on the next line if there is some kind of special CR2L character in the line. The cursor drops down a line.
I can't understand what is causing this, maybe the -1 row parameter to led_print() does not do what's expected if there is some kind of
special character in the line?

#define LNPREF does not help, and it makes sense cause I don't use terminal like that

To reproduce goto conf.h, find the line that says CR2L, enter insert at the 0 col, insert a bunch of tabs until the line goes over the screen.

The result should look like this:

https://kyryl.tk/neatvi_led_print.png
https://kyryl.tk/nextvi_led_print.png
https://kyryl.tk/if_keep_inserting_tabs.png

Nextvi does string rendering in many ways differently (as it's more efficient) than neatvi but as you can see it also happens there.
It probably doesn't need a fix as this edge case seems to only be caused by that CR2L string only, I just want to know why it happens
and If can be easily fixed?

Hello, entering insert mode using 'o' or 'i' and then hitting linefeed shifts every line and ends up not drawing the line above the cursor until you exit the insert mode. This is really annoying when editing and not something I would expect to happen. Let me know if this is an interesting issue to you as well. My terminal of choice is st, yet I have confirmed that this bug does not occur in tty nor xterm. The culprit is patch https://st.suckless.org/patches/anysize/st-anysize-0.8.1.diff which changes st window height to not be rounded to the equivalent rows * charheight, etc. Is there a way to patch out neatvi to be independent of this terminal size limitation? I'm thinking redraw stuff midway in insert mode may do the trick.

Hello Ali,
Hope all is well.
Yesterday I've taken a look at some other optimization option like rstr.
I've successfully implemented rstr as an optional patch for nextvi.
I even made a multi threaded version too.
[1] https://github.com/kyx0r/nextvi/blob/patches/rstr.patch

As I was testing, I found a bug. Searches like ^int$ do not work correctly.
I have fixed the problem in the patch above, never mind the macro craziness though
(Yes I am that hardcore and determined to optimize constant if statements out the
hot loops)

Basically my implementation fixes the bug by running 2 separate loops one for ^ and last one
for $, if ^ loop matched and $ is set the execution is handed to the $ loop for another round
of verification.
This solution may not be in realm of neatvi's standards but I haven't thought of any other ways.
I just made it work and efficiently.

Offtopic:
I have made another cool patch for my needs, this one is like grep implementation in nextvi
[2] https://github.com/kyx0r/nextvi/blob/patches/grep.patch

From our last discussion I have came up with some even better solutions and optimizations
to led_render() function, basically this is a good as it will ever get. I guess the truly genius
part was how I reassign the hl attributes after the led_bounds() call.
Now I can actually give a proper answer as to why this problem is so damn hard,
and it's hard because any character can be at any arbitrary screen position out
of sequence, this is the definition of pure chaos that no algorithm can really
solve efficiently and with the same results as naive approach.
I even tried some other genius idea I had, that was to try min - max the pointers
of chrs array that will get placed on the screen, then try to run a bound from
min to max pointer. This does work for the most part except when characters
get reordered in such a way that the computed character at chrs[0] and chrs[n]
both can meet on the same screen. When something like this happens obviously
the bound will not be a bound anymore, it will be exact copy of entire string. It will
be impossible to highlight a line that gets reordered like that without processing
entire string, which sadly is utterly slow. That is why current solution in led.c
relies on a guaranteed string, but it's not going to be exactly the same when
text is in reverse. Still it's way better than anything I ever had before and very
well defined behavior.

Kind Regards,
Kyryl.

Hello Ali,

I fixed this long time ago in my fork, but there is a problem of not being able to escape "|" in prompt.

Because you can't escape it, ex_line will treat it like a pipe for commands.
The implication is that it is impossible to run substitution commands that have "|" character in them as the searchable text.

This commit fixed it ex.c kyx0r/nextvi@c435fd1

Hello, this issue might also apply for neatlibc but REG_NEWLINE is not implemented in regex.c. Are there any plans in doing so? It would be nice to have syntax highlighting for C multi-line comments for example.

If anybody is interested in customizing this editor and adding extra features, I have uploaded my personal version here: https://github.com/kyx0r/nextvi .
I did quite a few changes that I felt like was lacking, changes listed in top readme file. Also I might add more stuff as times goes on.

Hello Ali, lbuf_replace is utterly slow.

I haven't really looked into what's causing this but when I fire up a 20MB book file and do a simple word substitution
it takes a (few hours ? maybe even infinite time) to complete.
The whole book is 385 000 lines, trimmed to 80 char limit.

I think those memmove() inside lbuf_replace() have exponential complexity, which sucks.

We need to speed this up, this is on my todo list in the near future.
Any ideas?

Hello @aligrudi I have found a way to break vi by looking at files with all kinds of weird encodings and characters, and basically somehow it is possible for the function uc_code(s); in many places to read past the malloc buffer. The good news is though that it's only a off by 1 error from what I saw on valgrind. Valgrind did not find any bad writes, so it may never crash, but the screen surely does show random artifacts. Overall I don't think it's that worth the hassle ( I cound not solve this myself) but maybe you will have a quick way to place a check somewhere? Generally I think uc_chop may be the root cause but I digress.

==31038== Invalid read of size 1
==31038==    at 0x11BF4C: uc_code (in /usr/bin/vi)
==31038==    by 0x11C5C2: uc_shape (in /usr/bin/vi)
==31038==    by 0x11AD30: led_render (in /usr/bin/vi)
==31038==    by 0x11B7F6: led_print (in /usr/bin/vi)
==31038==    by 0x112A78: vi_drawrow (in /usr/bin/vi)
==31038==    by 0x112BF8: vi_drawagain (in /usr/bin/vi)
==31038==    by 0x110546: main (in /usr/bin/vi)
==31038==  Address 0x49b1268 is 0 bytes after a block of size 104 alloc'd
==31038==    at 0x48D080F: malloc (vg_replace_malloc.c:307)
==31038==    by 0x11772F: lbuf_replace (in /usr/bin/vi)
==31038==    by 0x117DFF: lbuf_rd (in /usr/bin/vi)
==31038==    by 0x116CB6: ec_edit (in /usr/bin/vi)
==31038==    by 0x117382: ex_init (in /usr/bin/vi)
==31038==    by 0x110403: main (in /usr/bin/vi)
==31038==
==31038== Invalid read of size 1
==31038==    at 0x11BF4C: uc_code (in /usr/bin/vi)
==31038==    by 0x11C5C2: uc_shape (in /usr/bin/vi)
==31038==    by 0x11AD4A: led_render (in /usr/bin/vi)
==31038==    by 0x11B7F6: led_print (in /usr/bin/vi)
==31038==    by 0x112A78: vi_drawrow (in /usr/bin/vi)
==31038==    by 0x112BF8: vi_drawagain (in /usr/bin/vi)
==31038==    by 0x110546: main (in /usr/bin/vi)
==31038==  Address 0x49b1268 is 0 bytes after a block of size 104 alloc'd
==31038==    at 0x48D080F: malloc (vg_replace_malloc.c:307)
==31038==    by 0x11772F: lbuf_replace (in /usr/bin/vi)
==31038==    by 0x117DFF: lbuf_rd (in /usr/bin/vi)
==31038==    by 0x116CB6: ec_edit (in /usr/bin/vi)

This kind of stuff but not only for uc_shape.

vi_test.zip

Here is the offending file to reproduce the bug. I had to zip because github is dum dum.
Actually, this file is from mpv build source, it has some python at the top but the offending
part is at the bottom of the file, which I believe is a binary blob, but still I don't think vi should
just give up on that.

Hello @aligrudi when using dw to delete a word if the word is touching the \n the \n is deleted also. Is that a bug or a feature? I know that busybox vi does not delete the \n

Hello @aligrudi, and sorry for spamming your mailbox so much as of late, but vi intentionally does not render the last row where the vi_prompt goes. What was the original design decision for doing so? The way I see it, it's better to render that row also, but erase it whenever the prompt or message has to be displayed. That way you maximize the amount of useful info on the screen. Also if you know where I can change the way it works, I would be very grateful, and save some of my time.

Also, is github okay method of communication for you, or it's more preferred to send you an email with questions like this? I just do this on github so that anybody else can see if they have similar problems. But I don't think there is a mailing list for your software projects, right?

Arrow keys should work the same as hjkl when in command mode. Instead, the arrow keys behaves like the arrow keys of insert mode (output ABCD) when in command mode.

In nvi, arrow keys behave the same as in command and insert mode (move around the text like hjkl).

Even with the patch recommended in issue #1, NeatVI renders Hebrew diacritical marks as �, so

קֵן לַצִּפּוֹר

renders in NeatVI as

ק�ן ל�צ��פ�ו�ר

The terminal font (Cousine) can render these diacritical marks perfectly. I've also tested this in various terminals (Kitty, Alacritty, Terminal.app), and they all exhibit this issue. (I'm testing with the bicon test file.)

Thanks for the awesome work!

Hello @aligrudi . Could you explain how you use the @ key? In my fork I have updated the readme:

@ in normal mode: will execute the sequence of normal commands placed in
yank buffer. Specify yank buffer character or hit @ again for 0.

This is from what I understood by reading the source. But it feels like it's hard to imagine a proper use case. I think it's
kinda trying to be what vim macro recording is, but is lacking proper implementation.

I've done some googling and it seems like this is standard vi behavior, but the solution given is to run :set backspace=indent,eol,start which fails in neatvi with unknown option. what do i do?