spurious data in result when msiGetMoreRows returns CAT_NO_ROWS_FOUND about irods_rule_engine_plugin_python

[trel]

The fix for irods/irods#4157 will take care of this issue.

[tsmeele]

Just to confirm that we experience production issues as we use the python rule engine for gen queries.
Any query that would return a multitude of 256 rows crash the agent process!!

May I request warm attention to this bug pls.

[d-w-moore]

Ton,

This is my next bug to fix, I'll be going after it today.

Meanwhile, our 4-2-stable and master development branches of the Python Rule Engine plugin offer a GenQuery iterator which in tests has proven stable even for results of size 256,512,etc. It will be out officially in 4.2.5 of iRODS / Python RE , but I'd welcome you to try it out now if you like.

It's also fairly self-contained, being just a python module that sits in /etc/irods/genquery.py , so there is every reason to believe it can be ported into any vsn 4.2.0+ iRODS system and work fine.

You can find it at:
https://github.com/d-w-moore/irods_rule_engine_plugin_python/tree/4-2-stable#genquerypy

I'd be glad to answer any questions - if you end up using it - or, of course, any problems you might discover.

Thanks!

• Dan

[dworkin]

Here are two scripts to help you reproduce the problem.

setup18.r

# create 256 AVUs for /tempZone/home/rods
setupIssue18 {
    msiString2KeyValPair("", *kvp);
    *kvp.ISSUE18 = "0";
    msiSetKeyValuePairsToObj(*kvp, *path, "-C");
    for (*i = 1; *i < 256; *i = *i + 1) {
        *kvp.ISSUE18 = "*i";
        msiAssociateKeyValuePairsToObj(*kvp, *path, "-C");
    }
}

input *path="/tempZone/home/rods"
output ruleExecOut

issue18.py

import json

# return list of 256 values
def getAVUs(callback):
    avus = []
    # setup query
    ret_val = callback.msiMakeGenQuery(
        "META_COLL_ATTR_VALUE",
        "COLL_NAME = '/tempZone/home/rods' AND META_COLL_ATTR_NAME = 'ISSUE18'",
        irods_types.GenQueryInp())
    query = ret_val["arguments"][2]

    # perform query
    ret_val = callback.msiExecGenQuery(query, irods_types.GenQueryOut())
    while True:
        result = ret_val["arguments"][1]

        # process results
        for row in range(result.rowCnt):
            value = result.sqlResult[0].row(row)
            avus.append(value)

        # Continue with this query.
        if result.continueInx == 0:
            break
        ret_val = callback.msiGetMoreRows(query, result, 0)
    callback.msiCloseGenQuery(query, result)

    return avus

def issue18(rule_args, callback, rei):
    avus = getAVUs(callback)

    # Convert to json string and write to stdout.
    callback.writeLine("stdout", json.dumps(avus))