Implement more efficient `uniq` aggregators about pash

[nvasilakis]

I am not sure we can gain much by re-implementing a custom aggregator for uniq. This is because the partial inputs to the aggregator have already unique, so the only comparison we can gain is from comparisons at n input boundaries — which I don't see yielding serious improvements over simply re-running uniq.

[nvasilakis]

There is a first version of a custom aggregator for uniq -c. (Not ready for prime time yet, but getting there.)

[angelhof]

I am not sure we can gain much by re-implementing a custom aggregator for uniq. This is because the partial inputs to the aggregator have already unique, so the only comparison we can gain is from comparisons at n input boundaries — which I don't see yielding serious improvements over simply re-running uniq.

I was thinking of adding auxiliary map outputs (the end line) so that we don't even have to traverse the whole file, because this would allow us to just copy the file without ever touching it. However, I think you might be right, this doesn't sound like a big performance win. Let's make it low priority.

[nvasilakis]

That's actually a great idea, I didn't think of that. El jue, 10 de dic. de 2020 a la(s) 08:26, Konstantinos Kallas ( [email protected]) escribió:

…

I am not sure we can gain much by re-implementing a custom aggregator for uniq. This is because the partial inputs to the aggregator have already unique, so the only comparison we can gain is from comparisons at *n* input boundaries — which I don't see yielding serious improvements over simply re-running uniq. I was thinking of adding auxiliary map outputs (the end line) so that we don't even have to traverse the whole file, because this would allow us to just copy the file without ever touching it. However, I think you might be right, this doesn't sound like a big performance win. Let's make it low priority. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADL55ELN4NGNLPNUMH44ULSUDD63ANCNFSM4T7T4KGQ> .

[angelhof]

This can also be applied for

tr -s ' ' '\n'

or other similar commands that remove duplicates etc

[angelhof]

Closing because this issue will have to do with the annotation library rather than PaSh itself now