alpha-unito / pico Goto Github PK

https://cliutils.gitlab.io/modern-cmake/chapters/basics/structure.html

• [filter] read -> pass if even -> write
• [expander] read -> tokenize -> write

Special Cases

• kernel: never adding to collector

read -> add-key -> sum-by-key -> write

Special Cases

• one item for each key
• all items with same key

logs

stdin -> windowing sum -> stdout

Special Cases

• less items than window width

root@DESKTOP-HUPC6JQ:~/repositories/PiCo/examples/word-count# make
make: Für das Ziel „all“ ist nichts zu tun.
root@DESKTOP-HUPC6JQ:~/repositories/PiCo/examples/word-count# echo "hello world" > test
root@DESKTOP-HUPC6JQ:~/repositories/PiCo/examples/word-count# ./pico_wc test test_out
=== Semantic Graph
[SEMGRAPH] adjacency [operator]=>[operators]:
        ac83ec0[0xc83ec0] => ac84270[0xc84270]
        ac84270[0xc84270] => ac84290[0xc84290]
        ac84290[0xc84290] => ac85080[0xc85080]
        ac85080[0xc85080] =>
[SEMGRAPH] first operator: ac83ec0
[SEMGRAPH] last operator: ac85080

pico_wc: ../../pico/Operators/InOut/../../ff_implementation/OperatorsFFNodes/InOut/ReadFromFileFFNode.hpp:275: virtual void ReadFromFileFFNode_par::Partitioner::begin_callback(): Assertion `rend > rbegin' failed.
Abgebrochen (Speicherabzug geschrieben)

Am I doing it wrong?

read -> pair(read, map, filter) -> write

• both read operators read the same file (containing pairs)
• map applies some processing to input pairs
• filter duplicates some pairs and filters out the others

should be optimized as in flatmap + reduce-by-key

problem resides in compiling the optimized farm (map + reduce-by-key) regardless the kernel type

Check all files are licensed.

read -> merge(-, read) -> write

https://github.com/catchorg/Catch2/blob/master/docs/cmake-integration.md#installing-catch2-from-git-repository

https://cliutils.gitlab.io/modern-cmake/chapters/testing/catch.html

Add a variant to the test for flatmap-join-by-key with an additional reduce-by-key stage.

it seems some bytes at the end of the stream are ignored, probably due to ignoring the result of the last call to read, when the buffer is filled only partially.

word-count

• read -> +1 -> write

Special Cases

• empty input
• void kernel function

• function naming (Camel vs underscores etc.)
• file naming
• match include guards with paths

Design some mechanism (e.g., exception-based?) to report errors of different kinds:

• typing (data/structure) errors
• generic errors (e.g., non-existing file)
• ...

ideas:

• implementation choice as template parameter (similar to grppi)

Signature should be:

template<typename K, typename V>
class ReduceByKey(std::function<V(V &, V &)>);

@cmisale @FalkenRambaldi do you agree?

(related to #16)

• variant A: read | iterate(flatmap) | write
• variant B: read | iterate(flatmap-join(empty, read)) | write

sketch: flatmap could filter out some items and produce modified duplicates of other items

• read -> iterate(random filter) -> write
• [page-rank kernel] read -> iterate(pair(-, read, join-by-key+map(sum))) -> write

Special Cases

• 0 iterations
• more iterations than input elements (or more, until all items filtered)

• read -> to(write, write)
• read -> to(write, +1) -> write
• read -> to(+1, +2) -> write

read -> write

http://www.di.unito.it/~drocco/pico-pvs/

32a920c introduced a deadlock in first unit test for the iteration.

• export pico as target
• import Catch as target
• replace variables with targets

Guidelines:

• https://cliutils.gitlab.io/modern-cmake/chapters/basics.html
• https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html

socket -> add key -> windowing sum -> stdout

Special Cases

• for some key, less items (with that key) than window width

• clang-format
• cmake
• travis on Mac

not working with some input files (e.g., input_output_file.cpp)

• read -> sum -> write

Corner Cases

• single input item

Miss stock pricing input file

type problem with 'to' and 'add'