Memory limit about columnar

[sanikolaev]

Hi. We use docker for that:

    container_name: "manticore_columnar_engine"
    build:
      context: ./tests/${test}/manticore_columnar
      args:
        - commit
        - buffer
    mem_limit: ${mem}m
    cpuset: "6,7"
    privileged: true
    mem_swappiness: 0
    memswap_limit: 0
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "127.0.0.1:9307:9306"
      - "127.0.0.1:9309:9308"
    volumes:
      - ./tests/${test}/manticore_columnar/idx:/var/lib/manticore
      - ./tests/${test}/manticore_columnar/logs:/var/log/manticore
      - ./tests/${test}/data:/input
      - ./tests/${test}/manticore_columnar/manticore.conf:/etc/manticoresearch/manticore.conf

We do have tests on a larger dataset and we'll publish them as soon as we make performance optimizations in the library. Note the library is in its early stage and is being actively developed. But the clear thing is that the columnar library can work on e.g. 1.8B documents (NYC taxi rides) with 60GB mem limit, but Manticore Search w/o columnar can't since there's just not enough memory critically (410GB required vs 60GB limit).

[sangensong]

Ok, Thanks for your relpy.

[sangensong]

I had went to check the NYC taxi rides data set. I see

- 2.8 billion total trips
     - 1.7 billion taxi
     - 1.1 billion for-hire vehicle

Edit error？

[sanikolaev]

We tested on 1.7 billion taxi

[sangensong]

ok