irods / irods_rule_engine_plugin_python Goto Github PK

License: BSD 3-Clause "New" or "Revised" License

CMake 4.65% Python 10.94% C++ 69.85% R 14.57%

irods_rule_engine_plugin_python's Issues

PluginContext objects' map() method crashes in some PEPs

Some PEPs allow access to the fields of a rule argument of type irods_types.PluginContext via the object's map( ) method, whereas in other PEP's calling this method crashes the agent .

In pep_resource_create_post, this works without apparent flaws:
With this rule in core.py:

def  ( rule_args, callback, rei ):
    print ('map',repr(rule_args[1].map()))

an iput will print context fields from the returned python dict object:

('map', "{'proxy_auth_info_auth_scheme': '', 'auth_scheme': 'native', 'logical_path': '/tempZone/home/rods/dotdot.py', 'physical_path': '/var/lib/irods/Vault/home/rods/dotdot.py', 'proxy_auth_info_host': '', 'proxy_auth_info_ppid': '0', 'proxy_auth_info_auth_str': '', 'flags_kw': '3', 'proxy_user_other_info_user_modify': '', 'file_size': '0', 'proxy_auth_info_auth_flag': '5', 'proxy_rods_zone': 'tempZone', 'user_sys_uid': '0', 'in_pdmo': '', 'user_user_other_info_user_comments': '', 'proxy_user_other_info_user_info': '', 'user_user_name': 'rods', 'user_user_other_info_user_create': '', 'mode_kw': '384', 'user_auth_info_ppid': '0', 'user_auth_info_host': '', 'l1_desc_idx': '-1', 'user_auth_info_auth_scheme': '', 'proxy_user_name': 'rods', 'proxy_user_other_info_user_create': '', 'user_user_other_info_user_modify': '', 'proxy_user_type': '', 'file_descriptor': '10', 'dataType': '', 'user_auth_info_flag': '0', 'proxy_sys_uid': '0', 'user_user_other_info_user_info': '', 'user_rods_zone': 'tempZone', 'user_user_type': '', 'dataId': '0', 'user_auth_info_auth_flag': '5', 'resc_hier': 'demoResc', 'proxy_user_other_info_user_comments': '', 'client_addr': '127.0.0.1', 'proxy_auth_info_flag': '0', 'repl_requested': '-1', 'user_auth_info_auth_str': ''}")

But in, for example, pep_api_data_obj_put_post,

def pep_api_data_obj_put_post ( rule_args, callback, rei ):
    print ('map',repr(rule_args[1].map()))

The same iput statement triggers a crash with this stacktrace in the logs:

Caught signal [11]. Dumping stacktrace and exiting

Dumping stack trace
<0>     Offset:         Address: 0x7f243aa2f123 /usr/lib/libirods_server.so.4.2.8(+0xd64123) [0x7f243aa2f123]
<1>     Offset:         Address: 0x7f2437173390 /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f2437173390]
<2>     Offset:         Address: 0x7f24358c3840 /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x4ae840) [0x7f24358c3840]
<3>     Offset: 0x5f    Address: 0x7f2435d4669f boost::python::objects::caller_py_function_impl<boost::python::detail::caller<boost::python::dict (*)(irods::plugin_context*), boost::python::default_call_policies, boost::mpl::vector2<boost::python::dict, irods::plugin_context*> > >::operator()(_object*, _object*)
<4>     Offset: 0x2fb   Address: 0x7f24351ec8eb boost::python::objects::function::call(_object*, _object*) const
<5>     Offset:         Address: 0x7f24351eeb65 /opt/irods-externals/boost1.67.0-0/lib/libboost_python27.so.1.67.0(+0x25b65) [0x7f24351eeb65]
<6>     Offset: 0x47    Address: 0x7f24351f5f17 boost::python::handle_exception_impl(boost::function0<void>)
<7>     Offset:         Address: 0x7f24351ee7f6 /opt/irods-externals/boost1.67.0-0/lib/libboost_python27.so.1.67.0(+0x257f6) [0x7f24351ee7f6]
<8>     Offset:         Address: 0x7f2434b3f2b3 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f2434b3f2b3]
<9>     Offset:         Address: 0x7f2434add39c /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x543c) [0x7f2434add39c]
<10>    Offset:         Address: 0x7f2434c1611c /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x85c) [0x7f2434c1611c]
<11>    Offset:         Address: 0x7f2434b6c3b0 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13e3b0) [0x7f2434b6c3b0]
<12>    Offset:         Address: 0x7f2434b3f2b3 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f2434b3f2b3]
<13>    Offset:         Address: 0x7f2434c15547 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f2434c15547]
<14>    Offset:         Address: 0x7f2434b814e7 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallFunction+0xb7) [0x7f2434b814e7]
<15>    Offset:         Address: 0x7f24356e6ae1 /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x2d1ae1) [0x7f24356e6ae1]
<16>    Offset: 0x33a   Address: 0x7f24356e542a exec_rule(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<17>    Offset: 0x2e    Address: 0x7f243572bf7e std::__1::__function::__func<irods::error (*)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback&&)
<18>    Offset: 0xf6    Address: 0x7f243a8bcd76 irods::error irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&, irods::callback)
<19>    Offset: 0x85    Address: 0x7f243a8bcc05 std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)
<20>    Offset: 0x197   Address: 0x7f243a8bd7c7 irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'()::operator()() const
<21>    Offset: 0xce    Address: 0x7f243a8bc89e irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)
<22>    Offset: 0x103   Address: 0x7f243a8bc473 irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&) const
<23>    Offset: 0x100   Address: 0x7f243a8bc0f0 irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&), std::__1::tuple<>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*, BytesBuf*, portalOprOut**>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&&, irods::plugin_context&&&, DataObjInp*&&, BytesBuf*&&, portalOprOut**&&)
<24>    Offset: 0x629   Address: 0x7f243a8bbb89 irods::error irods::api_entry::invoke_policy_enforcement_point<DataObjInp*, BytesBuf*, portalOprOut**>(irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>, irods::plugin_context, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DataObjInp*, BytesBuf*, portalOprOut**)
<25>    Offset: 0xe67   Address: 0x7f243a7b0257 int irods::api_entry::call_handler<DataObjInp*, BytesBuf*, portalOprOut**>(rsComm_t*, DataObjInp*, BytesBuf*, portalOprOut**)
<26>    Offset: 0x73a   Address: 0x7f243aa6177a rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<27>    Offset: 0xa4b   Address: 0x7f243aa6381b readAndProcClientMsg(rsComm_t*, int)
<28>    Offset: 0xd08   Address: 0x7f243aa54738 agentMain(rsComm_t*)
<29>    Offset: 0x1fa5  Address: 0x7f243aa52985 runIrodsAgentFactory(sockaddr_un)
<30>    Offset:         Address: 0x42706d       /usr/sbin/irodsServer(main+0x4cd) [0x42706d]
<31>    Offset:         Address: 0x7f2436db8830 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f2436db8830]
<32>    Offset:         Address: 0x426ac9       /usr/sbin/irodsServer(_start+0x29) [0x426ac9]

Jul 27 13:16:46 pid:1340  ERROR: Agent process [17146] exited with status [11]

Exiting with code -808000 does not actually stop code execution

Tested in 4.2.6 and 4.2.11

When you use 'msiExit' to try and exit out of python rule engine execution, the execution is not stopped.

Example:

@make(inputs=[], outputs=[], handler=Output.STORE)
def test_rule(ctx):
    # -808000 CAT_NO_ROWS_FOUND
    ctx.callback.msiExit("-808000", "Rows not found")
    ctx.callback.writeLine("stdout", "This should not appear")

def main(rule_args, callback, rei):
    output = callback.test_rule()


INPUT null
OUTPUT ruleExecOut

irods@irods:/rules$ irule -r irods_rule_engine_plugin-python-instance -F /rules/tests/run_test_rule.r
This should not appear
Level 0: Rows not found

irods@irods:/rules$

This issue only seems to appear (as far as we know) with code "-808000". When using "-808001" it does actually exit out.
Is this behavior intentional?

move to four digit versioning based on irods-dev package

get plugin building and tests working on main branch

Using irods_development_environment and irods_testing_environment

refactor boost::python to avoid long compilation times

separate into different translation units so that irods_types doesn't need to completely recompile every time.

Need symbolic access to irods error and state tables

master
4-2-stable

Python Rule Engine Plugin currently must use error codes copied from rodsErrorTable.h or irods_state_table.h ; it would be better to enable programmatic, symbolic access to these constants whether in a module or microservice.
Compare irods/irods#5043

Python plugin won't build against iRODS runtime 4.3.0

need to create a stable branch as 4-2-stable, still pointing at 4.2.5
on master branch, update find_package to use only IRODS
remember to change master to declare 4.3.0 EXACT REQUIRED before release

FindPythonInterp.cmake and FindPythonLibs.cmake deprecated in CMake 3.12.

We should switch to FindPython.cmake, because the Development component provides a complete Python::Python target and does not require the interpreter component to define the version variables.

Debian package dependency is missing "python-dev"

When building a deb package, it results in PyExc_SystemError when parsing the core.py. (first line, import datetime)
It appears the package "python-dev" is missing.
In the Cmake file the following line needs to have the package listed:
set(CPACK_DEBIAN_PACKAGE_DEPENDS "${IRODS_PACKAGE_DEPENDENCIES_STRING}, irods-runtime (= ${IRODS_VERSION}), libc6")

But it may also be needed for other packages, so perhaps IRODS_PACKAGE_DEPENDENCIES_STRING should have it.

PREP does not support continuations

master
4-2-stable

Feature Request

exec_rule, exec_rule_text, and exec_rule_expression do not support anything other than returning zero on success. To support continuation, these functions need to allow rules to pass different codes back to the REPF.

irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx

Line 487 in f9ef6e4

return SUCCESS();

irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx

Line 614 in f9ef6e4

return SUCCESS();

irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx

Line 684 in f9ef6e4

return SUCCESS();

These lines need to be changed to CODE(RULE_ENGINE_CONTINUE). That would give other REPs a chance to handle the PEP.

Workaround

Keep all of a rule's functionality under a single REP. Do not split it across REPs (e.g. part of the rule is in the native rule engine while the rest of the rule is in the python rule engine).

pluginContext object not fully populated and/or accessible

All PEPs should populate and make available the full context of a running operation.

This is currently not being handled consistently.

def pep_resource_create_pre(rule_args, callback, rei):
    callback.writeLine("serverLog", str(rule_args)) 
    print(dir(rule_args[1]))

produces on an iput:

Sep 16 16:26:11 pid:23542 NOTICE: writeLine: inString = ['demoResc', <irods_types.PluginContext object at 0x7f7636cfac20>, '']
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'rule_results']

The <irods_types.PluginContext object at 0x7f7636cfac20> should allow introspection and be holding the information, similar to the iRODS Rule Language *CONTEXT.

Use "default" in acSetNumThreads in core.py.template (for multithreaded operation tests)

master
4-2-stable

test_multithreaded_[irepl|iphymv|icp]__issue_5478 uses acSetNumThreads to determine how many threads to use in a parallel transfer. This needs to be implemented in core.py in order to work in the PREP test suite.

a BytesBuf has a "void*buf" and "int len", but no means to set content

4-2-stable
main

The iRODS primitive data type bytesBuf_t, may be instantiated in the Python RE plugin as irods_types.BytesBuf( ), but it is impractical as an input buffer parameter for the microservice msiDataObjWrite (which accepts it based on its type) because its content and length may not be set. (Strings and bytestrings are also impractical for the purpose, as the write is curtailed at the first NUL character in the string's content, regardless of the length parameter specified to msiDataObjWrite).

I propose adding a method, setbuf(const char*c, int len) , whereby this can be done.

It remains to be seen how boost-python deals with the destruction of such objects when they go away. Possibly all that would be needed is to add a "__del__ method".

Install dependencies for building the plugin

Maybe interesting for documentation or other people:
On EL7, I first had to install these dependencies before i was able to build the plugin:
irods-externals* irods-devel gcc-c++ python-devel openssl-devel rpm-build

I also had to use the cmake of irods-externals-cmake, not the distro cmake, so I did run /opt/irods-externals/cmake3.5.2-0/bin/cmake ../ instead of just cmake ../

Python needs to be able to add entries to keyval pairs

Build fails with non-constant-expression cannot be narrowed from type 'int' to 'char'

Hello
I am trying to deploy irods python rule engine for iRODS 4.2.5.
when I send make package command I get the following error:

/home/ubuntu/irods_rule_engine_plugin_python/irods_types.cpp:572:30: note: in instantiation of function template specialization 'make_init_function<rodsEnv, char [64], char [64], int, char [1088], char [1088], char [64], char [64], char
[64], int, char [256], char [64], char [256], char [256], int, int, int, char [128], char [64], char [64], char [1088], char [1088], char [1088], char [1088], char [1088], char [1088], char [1088], int, int, char [128], int, int,
int, char [1088]>' requested here
.def("init", make_init_function(
^
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: error: non-constant-expression cannot be narrowed from type 'int' to 'char' in initializer list [-Wc++11-narrowing]
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ubuntu/irods_rule_engine_plugin_python/init_struct.hpp:60:20: note: insert an explicit cast to silence this issue
new T{ populate_helper(args, index, iter, index_into_array)... }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
static_cast( )
4 warnings and 9 errors generated.
CMakeFiles/irods_rule_engine_plugin-python.dir/build.make:86: recipe for target 'CMakeFiles/irods_rule_engine_plugin-python.dir/irods_types.cpp.o' failed
make[2]: *** [CMakeFiles/irods_rule_engine_plugin-python.dir/irods_types.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/irods_rule_engine_plugin-python.dir/all' failed
make[1]: *** [CMakeFiles/irods_rule_engine_plugin-python.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

Best regards
Mohamad

Change to python 3

Python 2 has end-of-life scheduled for the end of 2019, so we should stop depending on it and move to python 3 sometime in the next year.
Need to remember to remove the removal of register around boost::python:

#define register
...
#undef register

allow rodsuser to run a rule string

The command:

irule -r irods_rule_engine_plugin-python-instance python_rule null  null

should succeed if executed by a rodsuser.
In this context, the command line argument python_rule should be taken as the identifier of a python function (a rule) defined in the global namespace of /etc/irods/core.py.
Currently this command only succeeds for a rodsadmin. A rodsuser gets the errorSYS_NO_API_PRIV.

wrap irods::query to provide specific query in Python rules

Partially done - see this branch

sample core.py in comments of the plugin main .cxx source file
now testing with these specific queries active

----
sq1
select group_user_id, user_name from R_USER_GROUP ug inner join R_USER_MAIN u on ug.group_user_id = u.user_id where user_type_name = 'rodsgroup' and ug.user_id = (select user_id from R_USER_MAIN where user_name = 'dan' and user_type_name != 'rodsgroup')
----
sq2arg
select group_user_id, user_name from R_USER_GROUP ug inner join R_USER_MAIN u on ug.group_user_id = u.user_id where user_type_name = 'rodsgroup' and ug.user_id = (select user_id from R_USER_MAIN where user_name = ? and user_type_name != ?)

replication from compound using python

Bug Report

Irods version 4.2.11, centos7

I have the following resource hierarchy (OldResource being an old resource that I want to migrate data from)

DiskResource:unixfilesystem
OldResource:compound
├── OldArchiveResource:univmss
└── OldCacheResource:unixfilesystem

Data is only in the archive, the cache is empty.

I've written a custom rule that manages replication since it is a lengthy operation that I need to manage myself

migrateOneObj.r content:

main {
    testReplicationFromCompound();
}
INPUT null
OUTPUT ruleExecOut

core.re content:

testReplicationFromCompound {
    msiDataObjRepl("/zone/file.zip",
        "rescName=OldResource++++backupRescName=DiskResource", *Status);
}

When I run irule -F migrateOneObj.r under the user that owns file.zip, it works correctly.

However, if I move the function testReplicationFromCompound to python (core.py):

def testReplicationFromCompound(rule_args, callback, rei):
    callback.msiDataObjRepl("/zone/file.zip",
        "rescName=OldResource++++backupRescName=DiskResource", 0)

and run irule -F migrateOneObj.r (under the user that owns file.zip) the following happens:

the operation blocks on retrieving data from archive and never finishes
there are 2 irodsServer processes, that started when irule was started. strace says that
- first is blocked on read from a pipe
- second (child of the previous one) is blocked on futex(0x7f85e89d8ec8, FUTEX_WAIT_PRIVATE, 2, NULL

rodsLog's last line is

Jan 17 12:52:03 pid:88 NOTICE: execCmd:../../var/lib/irods/msiExecCmd_bin/migration- 
interface.sh argv:stageToCache '/data/archive/dev/file.zip' '/data/cache/dev/file.zip

univmss driver (shell script migration-interface.sh) is never run (otherwise it would create an entry in a custom log file)
there is an empty file created in the cache resource vault
in the sql db there are 2 rows:
- first one corresponds to the archive replica (data_is_dirty = 4)
- second one corresponds to the cache replica (data_is_dirty = 2)

test_irepl_multithreaded fails when python is the default rule engine instance

 --- IrodsSession: icommand executed by [otherrods] [iput largefile.txt] --- 
Assert Command: iput largefile.txt
Expecting EMPTY: ['']
  stdout:
    | 
  stderr:
    | remote addresses: 127.0.0.1 ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: rcvTranHeader: toread = 24, read = 0
    |  ERROR: [-]	/home/rskarbez/software/irods/lib/core/src/procApiRequest.cpp:176:int sendApiRequest(rcComm_t *, int, void *, bytesBuf_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message []
    | 	[-]	/home/rskarbez/software/irods/lib/core/src/sockComm.cpp:1258:irods::error sendRodsMsg(irods::network_object_ptr, const char *, bytesBuf_t *, bytesBuf_t *, bytesBuf_t *, int, irodsProt_t) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [failed to call 'write body']
    | 		[-]	/home/rskarbez/software/irods/plugins/network/tcp/libtcp.cpp:355:irods::error tcp_send_rods_msg(irods::plugin_context &, const char *, bytesBuf_t *, bytesBuf_t *, bytesBuf_t *, int, irodsProt_t) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [writeMsgHeader failed]
    | 			[-]	/home/rskarbez/software/irods/lib/core/src/sockComm.cpp:475:irods::error writeMsgHeader(irods::network_object_ptr, msgHeader_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message []
    | 				[-]	/home/rskarbez/software/irods/plugins/network/tcp/libtcp.cpp:293:irods::error tcp_write_msg_header(irods::plugin_context &, bytesBuf_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [wrote 0 expected 139]
    | 
    |  ERROR: putUtil: put error for /tempZone/home/otherrods/2016-11-22Z22:49:49--irods-testing-rteRDg/largefile.txt, status = -27000 status = -27000 SYS_COPY_LEN_ERR
Unexpected output on stderr

FAILED TESTING ASSERTION

plugin generates macro-redefined warnings during compile when pyconfig.h includes feature test macros

master
4-2-stable

Both pyconfig.h and feature.h can include definitions for _POSIX_C_SOURCE and _XOPEN_SOURCE. While feature.h contains logic for handling cases in which these defines already exist, pyconfig.h does not.
At present, both of these headers are in the include hierarchy, with feature.h coming in after pyconfig.h, causing the following warnings to be generated:

In file included from /irods_rule_engine_plugin_python/irods_types.cpp:12:
In file included from /irods_rule_engine_plugin_python/init_struct.hpp:5:
In file included from /irods_rule_engine_plugin_python/raw_constructor.hpp:9:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python.hpp:11:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/args.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:50:
/usr/include/python2.7/pyconfig.h:1223:9: warning: '_POSIX_C_SOURCE' macro redefined [-Wmacro-redefined]
#define _POSIX_C_SOURCE 200112L
        ^
/usr/include/features.h:281:10: note: previous definition is here
# define _POSIX_C_SOURCE        200809L
         ^
In file included from /irods_rule_engine_plugin_python/irods_types.cpp:12:
In file included from /irods_rule_engine_plugin_python/init_struct.hpp:5:
In file included from /irods_rule_engine_plugin_python/raw_constructor.hpp:9:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python.hpp:11:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/args.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:50:
/usr/include/python2.7/pyconfig.h:1245:9: warning: '_XOPEN_SOURCE' macro redefined [-Wmacro-redefined]
#define _XOPEN_SOURCE 600
        ^
/usr/include/features.h:207:10: note: previous definition is here
# define _XOPEN_SOURCE  700
         ^
In file included from /irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx:24:
In file included from /irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.hpp:16:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python.hpp:11:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/args.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:50:
/usr/include/python2.7/pyconfig.h:1223:9: warning: '_POSIX_C_SOURCE' macro redefined [-Wmacro-redefined]
#define _POSIX_C_SOURCE 200112L
        ^
/usr/include/features.h:281:10: note: previous definition is here
# define _POSIX_C_SOURCE        200809L
         ^
In file included from /irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx:24:
In file included from /irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.hpp:16:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python.hpp:11:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/args.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:50:
/usr/include/python2.7/pyconfig.h:1245:9: warning: '_XOPEN_SOURCE' macro redefined [-Wmacro-redefined]
#define _XOPEN_SOURCE 600
        ^
/usr/include/features.h:207:10: note: previous definition is here
# define _XOPEN_SOURCE  700

(note: the location of pyconfig.h and the values defined for the macros can vary)

I'll have a pull request for this shortly.

genquery.py double-executes & fetches completed queries

Current genquery.py row_iterator always performs a single extra msiGetMoreRows after all rows are consumed (continue index is already <= 0) (https://github.com/irods/irods_rule_engine_plugin_python/blob/master/genquery.py#L81). This appears to cause the query to be re-executed.

The auto-closing code (https://github.com/irods/irods_rule_engine_plugin_python/blob/master/genquery.py#L101) then has to fetch all query results again, since we are always left with an open query.

The issue does not occur when a query is aborted (i.e. a break or exception occurs before all rows have been yielded), since the query is never re-opened then.

Based on testing against genquery.py of #34 (because I was very curious why there was a performance difference), I see a 1.94x performance difference for a query with 9308 result rows, when all rows are consumed (test rule, data).

Boost.Python 1.67 needs version-agnostic symbolic link

This repository presently offers two versions of the Boost header libraries, 1.60 and 1.67.

Unlike version 1.60 however, Boost.Python 1.67 does not include the symbolic link "sans version" in the lib subdirectory for e.g. plugins that might want to link against it using the option -lboost_python. This recently helped to cause a delay in building/testing of Python RE Plugin against iRODS master at 4.3.0, which was attained in the end but required two corrections:

forge the required sym-link in the /opt/irods-externals/boost1.67.0-0/lib directory
CMakeLists.txt mods, see #19 in Python RE Plugin

Specifically the issue is that whereas in 1.60 we have:

libboost_python.so -> libboost_python.so.1.60.0

.... but, in 1.67 , we're missing the corresponding symbolic link. This causes failures in building and testing the Python Rule Engine against the iRODS 4.3.0 runtime at the bench as well as presumably also in Continuous Integration

spurious data in result when msiGetMoreRows returns CAT_NO_ROWS_FOUND

In RuleCallWrapper::call (irods_rule_engine_plugin-python.cxx:253), an error code of CAT_NO_ROWS_FOUND is ignored. When this occurs with msiGetMoreRows, which is called with an output parameter that contains the previous batch, it will be copied in the result without modification. This will lead to the same batch being reprocessed, followed by a crash on the next call to msiGetMoreRows.

callback object exists for delayExec() but not remoteExec()

On a machine with hostname daniel-ThinkPad-W540 , we are observing two scenarios in the use of the python rule engine plugin during calls to delayExec() and remoteExec() . Scenario (1) acts as expected, whereas scenario (2) displays the problem behavior.
Note incidentally that in each scenario, the remote and local computers are the self-same machine, with the implication that rule engine resolution order should be the same remotely as it is locally.

scenario (1):

With the irods rule language plugin preceding the python rule language in the server config's resolution order, and a rule file py_re_testexec.r containing:

def danRule(rule_args, callback, rei):
  callback.delayExec ("<PLUSET>1s</PLUSET>", "writeLine('serverLog','delaytest')","")
  callback.remoteExec ("daniel-ThinkPad-W540","","writeLine('serverLog','remotetest')","")

INPUT null
OUTPUT ruleExecOut

and executed via irule -r irods_rule_engine_plugin-python-instance -F py_re_testexec.r , we arrive at this (expected) output in the serverLog:

May  4 15:50:45 pid:495 NOTICE: writeLine: inString = remotetest
May  4 15:51:13 pid:502 NOTICE: writeLine: inString = delaytest

scenario (2):

Reversing the order of resolution then -- so that the python rule engine plugin comes first and irods rule language second -- and this time using a rule file py_py_testexec.r containing the following (note the callback object is referred to in the rule code because the python rule plugin will be answering):

def danRule(rule_args, callback, rei):
 callback.delayExec ("<PLUSET>1s</PLUSET>", "callback.writeLine('serverLog','delaytest')","")
 callback.remoteExec ("daniel-ThinkPad-W540","","callback.writeLine('serverLog','remotetest')","")

INPUT null
OUTPUT ruleExecOut

After this time invoking via irule -r irods_rule_engine_plugin-python-instance -F py_py_testexec.r we see the following in the server log (after waiting for the output from the delayExec to appear):

May  4 15:52:46 pid:584 remote addresses: 127.0.0.1, 152.54.9.29 ERROR: [exec_rule_text:532] Improperly formatted rule text in Python rule engine plugin
May  4 15:52:46 pid:584 remote addresses: 127.0.0.1, 152.54.9.29 ERROR: rsExecMyRule : -1828000, [-]	/tmp/tmpC8lQD0/irods_rule_engine_plugin-python.cxx:533:irods::error exec_rule_text(const irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) :  status [RULE_ENGINE_ERROR]  errno [] -- message [Improperly formatted rule_text]

May  4 15:52:46 pid:584 DEBUG: 
May  4 15:52:46 pid:584 remote addresses: 127.0.0.1, 152.54.9.29 ERROR: caught python exception: Traceback (most recent call last):
  File "<string>", line 12, in danRule
RuntimeError: [iRods__Error__Code:-1828000] [-]	/tmp/tmpLGq9_F/server/re/include/irods_re_plugin.hpp:321:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [RULE_ENGINE_ERROR]  errno [] -- message [applyRuleUpdateParams failed for rule remoteExec]
	[-]	/tmp/tmpLGq9_F/plugins/rule_engines/irods_rule_engine_plugin-irods_rule_language/libirods_rule_engine_plugin-irods_rule_language.cpp:348:irods::error exec_rule(irods::default_re_ctx &, const std::string &, std::list<boost::any> &, irods::callback) :  status [RULE_ENGINE_ERROR]  errno [] -- message [applyRuleUpdateParams failed for rule remoteExec]


May  4 15:52:46 pid:584 remote addresses: 127.0.0.1, 152.54.9.29 ERROR: rsExecMyRule : -1828000, [-]	/tmp/tmpC8lQD0/irods_rule_engine_plugin-python.cxx:539:irods::error exec_rule_text(const irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) :  status [RULE_ENGINE_ERROR]  errno [] -- message [irods_rule_engine_plugin_python::irods::error exec_rule_text(const irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) Caught Python exception.
Traceback (most recent call last):
  File "<string>", line 12, in danRule
RuntimeError: [iRods__Error__Code:-1828000] [-]	/tmp/tmpLGq9_F/server/re/include/irods_re_plugin.hpp:321:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [RULE_ENGINE_ERROR]  errno [] -- message [applyRuleUpdateParams failed for rule remoteExec]
	[-]	/tmp/tmpLGq9_F/plugins/rule_engines/irods_rule_engine_plugin-irods_rule_language/libirods_rule_engine_plugin-irods_rule_language.cpp:348:irods::error exec_rule(irods::default_re_ctx &, const std::string &, std::list<boost::any> &, irods::callback) :  status [RULE_ENGINE_ERROR]  errno [] -- message [applyRuleUpdateParams failed for rule remoteExec]


]

May  4 15:53:14 pid:590 NOTICE: writeLine: inString = delaytest

This shows that although the python rule engine is "picking up the phone" for scenario (2) -- as one would predict -- the existence of the callback object, while recognized during the evaluation of the delayExec() call, is NOT recognized during the remoteExec() call.

packaging should require irods-server

msiExecCmd not usable from Python Rules

msiExecCmd should be callable from rules written for the PREP (Python rule engine plugin), similarly as from rules in iRODS's native rule language.

Currently, however, the following rule will abort on the call to the msiExecCmd microservice when run from /etc/irods/core.py :

import os ,  irods_types
def myTestRule(rule_args, callback, rei):
    #source_file = global_vars['*SourceFile'][1:-1]
    source_file = '/tempZone/home/rods/VERSION.json'
    import os
    (coll, filen) = os.path.split(source_file)
    ret_val = callback.msiObjStat(source_file, irods_types.RodsObjStat())
    source_file_stat = ret_val['arguments'][1]
    ret_val = callback.msiExecCmd( "hello", "a b", "null", "null", "null", irods_types.ExecCmdOut() );
    ret_val = callback.msiObjStat(coll, irods_types.RodsObjStat())
    coll_stat = ret_val['arguments'][1]
    callback.writeLine('serverLog', 'Type of object is written into a RodsObjStat_PI structure')

Note the iRODS data object /tempZone/home/rods/VERSION.json should exist before attempting to call myTestRule ; and the call can be done from a ".r" file with no arguments.
The attempted execution of the above rule function produces this error output in the rodsLog :

Jul 12 15:30:36 pid:1481 NOTICE: writeLine: inString = Type of object is written into a RodsObjStat_PI structure
Jul 12 15:57:26 pid:3742 remote addresses: 127.0.0.1, 152.54.8.225 ERROR: caught python exception: Traceback (most recent call last):
  File "/etc/irods/core.py", line 15, in myTestRule
    ret_val = callback.msiExecCmd( "hello", "a b", "null", "null", "null", irods_types.ExecCmdOut() );
RuntimeError: iRODS Exception:
    file: /home/dmoore/github/irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.hpp
    function: msParam_t (anonymous namespace)::msParam_from_object_impl(boost::python::object &)
    line: 117
    code: -66000
    message:
        Attempted to extract a boost::python::object containing a non-conforming type
<< stack trace follows >>

The presence of the msiExecCmd in the above Python rule is just for demonstration, to show contrast with an analogous call to a different microservice msiObjStat, which is similar in pattern and superficial purpose to the failed microservice call.

If the offending line of code is reduced to contain only the python expression irods_types.ExecCmdOut() , or if it is commented out entirely, the rule will complete its execution, showing that it is the actual call to msiExecCmd that triggers the error in question, rather than the act of instantiating the python class irods_types.ExecCmdOut . One will then see the expected output in the rodsLog :

Jul 12 15:30:36 pid:1481 NOTICE: writeLine: inString = Type of object is written into a RodsObjStat_PI structure

Note also, when we run the native rule-engine version of the msiExecCmd call that failed above, with identical (or as closely corresponding as possible) arguments:

myTestNativeRule
{
    msiExecCmd( "hello", "a b", "null", "null", "null", *OUT );
}

we get the expected behavior of:

Jul 12 16:00:51 pid:4060 NOTICE: execCmd:../../var/lib/irods/msiExecCmd_bin/hello argv:a b

irods_types.hpp needs Python.h for Python 3

irods_types.hpp contains references to PyObject but does not #include anything to define it.

housekeeping (versions, whitespace, spelling)

Update for iRODS 4.3

Deadlock in msiDataObjRepl & msiDataObjCopy when called from Python

Tested in 4.2.6 and 4.2.7

When these microservices are called from a PREP rule for data objects larger than
parallel transfer thresholds[1], they do not terminate.

Calling the microservices from the native rule language works without issues.

Example:

def A(rule_args, callback, rei):
    import irods_types
    callback.msiDataObjCopy('/tempZone/home/rods/x.bin',
                            '/tempZone/home/rods/y.bin',
                            'forceFlag=', irods_types.BytesBuf())

$ head -c40M /dev/zero > x.bin
$ iput -f x.bin
$ irule A null null
... (hung)

ps output indicates the responsible agent is waiting on a futex:

$ ps -lq 4995
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
1 S   997  4995  4164  0  80   0 - 259245 futex_ ?       00:00:00 irodsServer

Killing the agent produces an interesting stacktrace[2], which points to a
spawned data transfer/copy thread sameHostPartialCopy(), created by
sameHostCopy(), that does not terminate (join in the main agent thread
hangs forever).

Apparently, in #1, a recursive mutex was introduced to prevent concurrent PREP
access across parallel transfer threads.
This change allowed multiple threads to safely trigger Python PEP rules
sequentially, but, if I understand correctly, it did not take into account that
the PREP may already be held hostage by a Python rule on a different thread
(i.e. the rule that triggered the transfer by calling an msi).

So we now have a deadlock: The spawned sameHostPartialCopy thread wants to
execute a PEP rule and is waiting for access to the PREP, while the thread
holding the PREP mutex is waiting for that spawned thread to finish (in
sameHostCopy).

As expected in this case, creating dummy implementations[3] of all static and
dynamic PEPs in the native rule language (which must then also be specified
before the prep in the server config) bypasses this issue, as the PREP no
longer needs to be consulted for PEPs during parallel transfer threads.
This is not really a usable workaround.

[1]

>= transfer_buffer_size_for_parallel_transfer_in_megabytes =4M (?) for msiDataObjRepl
> maximum_size_for_single_buffer_in_megabytes =32M for msiDataObjCopy

[2]

irodsServer: /tmp/irods_externals6CFhcy/clang3.8-0_src/llvm/projects/libcxx/src/mutex.cpp:86: std::__1::recursive_mutex::~recursive_mutex(): Assertion `e == 0' failed.
Caught signal [6]. Dumping stacktrace and exiting

Dumping stack trace
<0>	Offset:      	Address: 0x7f8d4218da0c	/lib/libirods_server.so.4.2.7(+0xb90a0c) [0x7f8d4218da0c]
<1>	Offset:      	Address: 0x7f8d3f84d5f0	/lib64/libpthread.so.0(+0xf5f0) [0x7f8d3f84d5f0]
<2>	Offset:      	Address: 0x7f8d3e55f337	/lib64/libc.so.6(gsignal+0x37) [0x7f8d3e55f337]
<3>	Offset:      	Address: 0x7f8d3e560a28	/lib64/libc.so.6(abort+0x148) [0x7f8d3e560a28]
<4>	Offset:      	Address: 0x7f8d3e558156	/lib64/libc.so.6(+0x2f156) [0x7f8d3e558156]
<5>	Offset:      	Address: 0x7f8d3e558202	/lib64/libc.so.6(+0x2f202) [0x7f8d3e558202]
<6>	Offset:      	Address: 0x7f8d3eb5c8e1	/opt/irods-externals/clang-runtime3.8-0/lib/libc++.so.1(+0x4f8e1) [0x7f8d3eb5c8e1]
<7>	Offset:      	Address: 0x7f8d3e562c99	/lib64/libc.so.6(+0x39c99) [0x7f8d3e562c99]
<8>	Offset:      	Address: 0x7f8d3e562ce7	/lib64/libc.so.6(+0x39ce7) [0x7f8d3e562ce7]
<9>	Offset:      	Address: 0x7f8d421aab77	/lib/libirods_server.so.4.2.7(+0xbadb77) [0x7f8d421aab77]
<10>	Offset:      	Address: 0x7f8d3e55f3b0	/lib64/libc.so.6(+0x363b0) [0x7f8d3e55f3b0]
<11>	Offset:      	Address: 0x7f8d3f8499f3	/lib64/libpthread.so.0(pthread_cond_wait+0xc3) [0x7f8d3f8499f3]
<12>	Offset: 0x3b 	Address: 0x42988b	boost::condition_variable::wait(boost::unique_lock<boost::mutex>&)
<13>	Offset: 0x66 	Address: 0x7f8d40ce8586	boost::thread::join_noexcept()
<14>	Offset: 0x502	Address: 0x7f8d42196d02	sameHostCopy(rsComm_t*, DataCopyInp*)
<15>	Offset: 0x5b 	Address: 0x7f8d41eb9d9b	_rsDataCopy(rsComm_t*, DataCopyInp*)
<16>	Offset: 0x3ac	Address: 0x7f8d41ed132c	dataObjCopy(rsComm_t*, int)
<17>	Offset: 0x222	Address: 0x7f8d41ec11f2	_rsDataObjCopy(rsComm_t*, int, int, transferStat_t**)
<18>	Offset: 0x250	Address: 0x7f8d41ec0960	rsDataObjCopy(rsComm_t*, DataObjCopyInp*, transferStat_t**)
<19>	Offset: 0xca 	Address: 0x7f8d4231933a	msiDataObjCopy(MsParam*, MsParam*, MsParam*, MsParam*, RuleExecInfo*)
<20>	Offset: 0x19b	Address: 0x7f8d422d4aab	int irods::ms_table_entry::call_handler<MsParam*, MsParam*, MsParam*, MsParam*, RuleExecInfo*>(MsParam*, MsParam*, MsParam*, MsParam*, RuleExecInfo*)
<21>	Offset: 0x274	Address: 0x7f8d422db9c4	irods::default_microservice_manager<RuleExecInfo*>::exec_microservice_adapter(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, RuleExecInfo*, std::__1::list<boost::any, std::__1::allocator<boost::any> >&)
<22>	Offset: 0x74 	Address: 0x7f8d422e4194	irods::error irods::default_microservice_manager<RuleExecInfo*>::exec_microservice<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, RuleExecInfo*, irods::unpack&&)
<23>	Offset: 0x45 	Address: 0x7f8d422e40a5	std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)
<24>	Offset: 0xfe 	Address: 0x7f8d422e447e	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, RuleExecInfo*&&&, irods::unpack&&)::'lambda'()::operator()() const
<25>	Offset: 0x9f 	Address: 0x7f8d422e3e3f	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, RuleExecInfo*&&&, irods::unpack&&)
<26>	Offset: 0x103	Address: 0x7f8d422e2e93	irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const
<27>	Offset: 0x23c	Address: 0x7f8d422e2b1c	irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::tuple<>, irods::unpack>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)
<28>	Offset: 0x1d4	Address: 0x7f8d3b915ba4	irods::error irods::callback::operator()<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)
<29>	Offset:      	Address: 0x7f8d3b90f54a	/usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x2b454a) [0x7f8d3b90f54a]
<30>	Offset: 0x3a 	Address: 0x7f8d3b9155ba	boost::python::detail::raw_dispatcher<boost::python::dict (*)(boost::python::tuple const&, boost::python::dict const&)>::operator()(_object*, _object*)
<31>	Offset: 0x37a	Address: 0x7f8d3b43218a	boost::python::objects::function::call(_object*, _object*) const
<32>	Offset:      	Address: 0x7f8d3b4343c5	/opt/irods-externals/boost1.60.0-0/lib/libboost_python.so.1.60.0(+0x293c5) [0x7f8d3b4343c5]
<33>	Offset: 0x4d 	Address: 0x7f8d3b43a59d	boost::python::handle_exception_impl(boost::function0<void>)
<34>	Offset:      	Address: 0x7f8d3b43406a	/opt/irods-externals/boost1.60.0-0/lib/libboost_python.so.1.60.0(+0x2906a) [0x7f8d3b43406a]
<35>	Offset:      	Address: 0x7f8d3ae7dab3	/lib64/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8d3ae7dab3]
<36>	Offset:      	Address: 0x7f8d3ae8caa5	/lib64/libpython2.7.so.1.0(+0x5aaa5) [0x7f8d3ae8caa5]
<37>	Offset:      	Address: 0x7f8d3ae7dab3	/lib64/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8d3ae7dab3]
<38>	Offset:      	Address: 0x7f8d3aed4e77	/lib64/libpython2.7.so.1.0(+0xa2e77) [0x7f8d3aed4e77]
<39>	Offset:      	Address: 0x7f8d3ae7dab3	/lib64/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8d3ae7dab3]
<40>	Offset:      	Address: 0x7f8d3af12286	/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2336) [0x7f8d3af12286]
<41>	Offset:      	Address: 0x7f8d3af1908d	/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed) [0x7f8d3af1908d]
<42>	Offset:      	Address: 0x7f8d3aea29c8	/lib64/libpython2.7.so.1.0(+0x709c8) [0x7f8d3aea29c8]
<43>	Offset:      	Address: 0x7f8d3ae7dab3	/lib64/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8d3ae7dab3]
<44>	Offset:      	Address: 0x7f8d3af0f947	/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f8d3af0f947]
<45>	Offset:      	Address: 0x7f8d3af2fed7	/lib64/libpython2.7.so.1.0(PyEval_CallFunction+0xb7) [0x7f8d3af2fed7]
<46>	Offset:      	Address: 0x7f8d3b90aaa3	/usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x2afaa3) [0x7f8d3b90aaa3]
<47>	Offset: 0x385	Address: 0x7f8d3b909a15	exec_rule(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<48>	Offset: 0x2e 	Address: 0x7f8d3b94a31e	std::__1::__function::__func<irods::error (*)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback&&)

[3] https://gist.github.com/cjsmeele/dfb029048e7a31643acd04d8eb200e96
(assumes core.re already contains all static PEP rules)

Split irods_types.cpp class_ and enum_ definitions into multiple files

irods_types.cpp takes a disgusting amount of time to compile. To fix this, we should follow Boost's guidance and split it up into multiple files:
https://www.boost.org/doc/libs/1_76_0/libs/python/doc/html/tutorial/tutorial/techniques.html#tutorial.techniques.reducing_compiling_time

core.py unable to import some Python Standard Library modules

On Ubuntu 18, with PREP 4.3.0.0 installed and configured:
Create an /etc/irods/core.py with one line:

import csv

Any client command such as ils will then cause a crash in the agent, and an error in the /var/log/irods/irods.log to the effect that _Py_NoneStruct cannot be resolved.

exec_rule_text, exec_rule_expression not importing core

bp::object core_module = bp::import("core");

Means that e.g. delay rules can't immediately use functions in core.py. Haven't tried putting import core in the rule, that may be a workaround.

Perhaps this is desired? I'm not sure what the consequences of using an import statement within a rule are. Would a module imported in one call to exec_rule be available in a later call to exec_rule_text within the same agent?

Implement types in boost python to offer more pythonic interface

need a capability to build python rule engine package from a local externals directory or repo

4-2-stable
master

Test Failure: `test_all_rules.Test_AllRules.test_rulemsiDataObjUnlink_r`

The replNum option has been removed from unlink. Please update the example rule to reflect the changes.

make condInput iterable

current required syntax to access the keys and values in a conditional input data structure

def pep_api_data_obj_trim_post(rule_args, callback, rei):
    for i in range(rule_args[2].condInput.len):
        print(str(rule_args[2].condInput.key[i]),str(rule_args[2].condInput.value[i]))

a cleaner syntax might be...

def pep_api_data_obj_trim_post(rule_args, callback, rei):
    for k,v in rule_args[2].condInput.items():
        print(k,v)

should also support iteritems() and viewitems()?

Port icommands/test/rules into Python to support Python REP in CI

Python 2.7 libraries use incompatible 'register' keyword

C++17 removes the register keyword, and the headers for python 2.7 make use of that keyword. Recommended solution is to #define register before the include and #undef register afterwards. This is sort of a hack, but there is no other option without shipping our own edited python headers and it should be safe since register is an ignored keyword in C, and the python headers are written in C. This is also a stopgap measure until we start shipping with python 3, given that python 2 end-of-life is scheduled for late 2019.

Build fails, find_package(IRODS 4.3.0 REQUIRED) in CMakeList.txt

The build fails, because no iRODS 4.3.0 can be found.

CMake Error at CMakeLists.txt (find_package):", " Could not find a configuration file for package \"IRODS\" that is compatible", " with requested version \"4.3.0\".", "", " The following configuration files were considered but not accepted:", "", " /usr/lib/irods/cmake/IRODSConfig.cmake, version: 4.2.0"

compiling inside the irods_development_environment (centos7) hangs

I have built the docker environemnt for building the RPMS. It is working well for the base packages, storage_tiering and s3. But it is not building the python rule engine. it seems to hang at the following section:

[ 66%] Building CXX object CMakeFiles/irods_rule_engine_plugin-python.dir/irods_types.cpp.o
/irods_rule_engine_plugin_python/irods_types.cpp:3:9: warning: 'BOOST_SYSTEM_NO_DEPRECATED' macro redefined [-Wmacro-redefined]
#define BOOST_SYSTEM_NO_DEPRECATED BOOST_PYTHON_MAX_ARITY 45
^
:1:9: note: previous definition is here
#define BOOST_SYSTEM_NO_DEPRECATED 1
^
In file included from /irods_rule_engine_plugin_python/irods_types.cpp:7:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/module_init.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:50:
In file included from /usr/include/python2.7/pyconfig.h:6:
/usr/include/python2.7/pyconfig-64.h:1191:9: warning: '_POSIX_C_SOURCE' macro redefined [-Wmacro-redefined]
#define _POSIX_C_SOURCE 200112L
^
/usr/include/features.h:168:10: note: previous definition is here

define _POSIX_C_SOURCE 200809L

define _XOPEN_SOURCE 700

In file included from /irods_rule_engine_plugin_python/irods_types.cpp:7:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/module_init.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:151:
In file included from /usr/include/python2.7/Python.h:85:
/usr/include/python2.7/unicodeobject.h:534:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register PyObject obj, / Object */
^~~~~~~~~
/usr/include/python2.7/unicodeobject.h:553:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register PyObject obj / Object */
^~~~~~~~~
/usr/include/python2.7/unicodeobject.h:575:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register const wchar_t w, / wchar_t buffer */
^~~~~~~~~
/usr/include/python2.7/unicodeobject.h:593:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register wchar_t w, / wchar_t buffer */
^~~~~~~~~
In file included from /irods_rule_engine_plugin_python/irods_types.cpp:7:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/module_init.hpp:8:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/prefix.hpp:13:
In file included from /opt/irods-externals/boost1.67.0-0/include/boost/python/detail/wrap_python.hpp:151:
In file included from /usr/include/python2.7/Python.h:94:
/usr/include/python2.7/stringobject.h:173:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register PyObject obj, / string or Unicode object */
^~~~~~~~~
/usr/include/python2.7/stringobject.h:174:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register char *s, / pointer to buffer variable */
^~~~~~~~~
/usr/include/python2.7/stringobject.h:175:5: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
register Py_ssize_t len / pointer to length variable or NULL
^~~~~~~~~

IRODS_ERROR_PREFIX could be better

Change from const std::string to std::string_view or const char*
Remove leading [
Use fmt::format for formatting messages

DYNAMIC_PEP_RULE_REGEX is missing `except` and `finally`

Add except|finally to the regex

Missing <mutex> include

irods_rule_engine_plugin-python.cxx uses std::recursive_mutex and std::lock_guard, but does not #include <mutex>. This can cause errors to be generated during compilation.

Invalid pointer free when error occurs inside of callback.msiSetKeyValuePairsToObj

main
4-2-stable

Bug Report

iRODS Version, OS and Version

4.2.11 server, 4.2.11.0 plugin

What did you try to do?

Execute a dynamic PEP which calls a microservice from a configured rulebase in PREP (e.g. core.py). The rule looks like this:

def pep_api_touch_pre(rule_args, callback, rei):
    out = callback.msiString2KeyValPair('touch_pre_fired=YES', 0)
    if not out['status']:
        callback.writeLine('stdout', 'ERROR: ' + str(out['code']))
    kvp = out['arguments'][1]
    callback.msiSetKeyValuePairsToObj(kvp, '{0}', '-d') # this should produce an error

This can be triggered by simply executing itouch:

$ itouch foo

Expected behavior

An error code and python stacktrace in the log, like this:

Dec 21 17:00:21 pid:3215 NOTICE: checkAndGetObjectId cmlExecuteNoAnswerSql(rollback) succeeded
Dec 21 17:00:21 pid:3215 NOTICE: rsModAVUMetadata: rcModAVUMetadata failed
Dec 21 17:00:21 pid:3215 remote addresses: 192.168.16.2, 192.168.16.3 ERROR: caught python exception: Traceback (most recent call last):
  File "/etc/irods/core.py", line 337, in pep_api_touch_pre
    callback.msiSetKeyValuePairsToObj(kvp, '{0}', '-d')
RuntimeError: [iRods__Error__Code:-817000] [-]  /irods_source/server/re/include/irods_re_plugin.hpp:326:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, RuleExecInfo *&, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, RuleExecInfo *&, irods::unpack>] :  status [CAT_UNKNOWN_FILE]  errno [] -- message [exec_microservice_adapter failed]
        [-]     /irods_source/server/re/src/irods_re_plugin.cpp:132:irods::error irods::default_microservice_manager<RuleExecInfo *>::exec_microservice_adapter(std::string, irods::default_ms_ctx, std::list<boost::any> &) :  status [CAT_UNKNOWN_FILE]  errno [] -- message [exec_microservice_adapter failed]

Observed behavior (including steps to reproduce, if applicable)

An invalid pointer free kills the connected agent. A sample of the output in the log:

Dec 21 17:29:21 pid:3419 NOTICE: rsModAVUMetadata: rcModAVUMetadata failed
free(): invalid pointer
Caught signal [6]. Dumping stacktrace and exiting

Dumping stack trace
<0>     Offset:         Address: 0x7fe183b8c9b8 /usr/lib/libirods_server.so.4.2.11(+0x15149b8) [0x7fe183b8c9b8]
<1>     Offset:         Address: 0x7fe18095a980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fe18095a980]
<2>     Offset:         Address: 0x7fe180595fb7 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fe180595fb7]
<3>     Offset:         Address: 0x7fe180597921 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fe180597921]
<4>     Offset:         Address: 0x7fe1805e0967 /lib/x86_64-linux-gnu/libc.so.6(+0x89967) [0x7fe1805e0967]
<5>     Offset:         Address: 0x7fe1805e79da /lib/x86_64-linux-gnu/libc.so.6(+0x909da) [0x7fe1805e79da]
<6>     Offset:         Address: 0x7fe1805eef0c /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4dc) [0x7fe1805eef0c]
<7>     Offset:         Address: 0x7fe184f82222 /usr/lib/libirods_common.so.4.2.11(clearMsParam+0x92) [0x7fe184f82222]
<8>     Offset:         Address: 0x7fe17ce163b4 /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x75f3b4) [0x7fe17ce163b4]
<9>     Offset: 0x76    Address: 0x7fe17ce2c206 boost::python::detail::raw_dispatcher<boost::python::dict (*)(boost::python::tuple const&, boost::python::dict const&)>::operator()(_object*, _object*)
<10>    Offset: 0x34    Address: 0x7fe17ce2c104 boost::python::objects::full_py_function_impl<boost::python::detail::raw_dispatcher<boost::python::dict (*)(boost::python::tuple const&, boost::python::dict const&)>, boost::mpl::vector1<_object*> >::operator()(_object*, _object*)
<11>    Offset: 0x2fb   Address: 0x7fe17c48f85b boost::python::objects::function::call(_object*, _object*) const
<12>    Offset:         Address: 0x7fe17c491ad5 /opt/irods-externals/boost1.67.0-0/lib/libboost_python27.so.1.67.0(+0x25ad5) [0x7fe17c491ad5]
<13>    Offset: 0x47    Address: 0x7fe17c498e87 boost::python::handle_exception_impl(boost::function0<void>)
<14>    Offset:         Address: 0x7fe17c491766 /opt/irods-externals/boost1.67.0-0/lib/libboost_python27.so.1.67.0(+0x25766) [0x7fe17c491766]
<15>    Offset:         Address: 0x7fe17c063913 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fe17c063913]
<16>    Offset:         Address: 0x7fe17c090eec /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a1eec) [0x7fe17c090eec]
<17>    Offset:         Address: 0x7fe17c063913 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fe17c063913]
<18>    Offset:         Address: 0x7fe17c00f31a /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12031a) [0x7fe17c00f31a]
<19>    Offset:         Address: 0x7fe17c063913 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fe17c063913]
<20>    Offset:         Address: 0x7fe17bf8d951 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x41e1) [0x7fe17bf8d951]
<21>    Offset:         Address: 0x7fe17c0ce908 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fe17c0ce908]
<22>    Offset:         Address: 0x7fe17c069649 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17a649) [0x7fe17c069649]
<23>    Offset:         Address: 0x7fe17c063913 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fe17c063913]
<24>    Offset:         Address: 0x7fe17c0cde37 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fe17c0cde37]
<25>    Offset:         Address: 0x7fe17c03c381 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallFunction+0xa1) [0x7fe17c03c381]
<26>    Offset:         Address: 0x7fe17ce1caf4 /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x765af4) [0x7fe17ce1caf4]
<27>    Offset:         Address: 0x7fe17ce0e1ca /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-python.so(+0x7571ca) [0x7fe17ce0e1ca]
<28>    Offset: 0x834   Address: 0x7fe17ce0bb14 exec_rule(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<29>    Offset: 0x141   Address: 0x7fe17ced1b21 irods::error std::__1::__invoke_void_return_wrapper<irods::error>::__call<irods::error (*&)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback>(irods::error (*&&&)(std::__1::tuple<> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::tuple<>&&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&&&, irods::callback&&)

<snip>

See the attached log for the full output: prep_invalid_pointer_free.log.txt

Further investigation will be needed, but it is definitely related to fixes introduced for leaks occurring which were discovered in irods/irods#4649.

In particular, the line causing the invalid pointer free - seen in frame <7> - is here, added in 4.2.11.0:

irods_rule_engine_plugin_python/irods_rule_engine_plugin-python.cxx

Line 278 in 785b279

clearMsParam(&msParams.front(), 1);

The issue seems to only manifest itself when a python exception is being thrown (notice frame <13>). When a proper data object path is given to the microservice, an error is not generated and the crash does not occur.

Add missing static PEP acPostProcForDataCopyReceived to core.py.template

In plugin package installations, irods-owned paths may become root-owned

4-2-stable
master

Bug Report

iRODS Version, OS and Version

iRODS 4.2.9 + Python REP 4.2.9.0 (just pre-release) on Centos 7
BUT this may apply to many other plugins, for all we know....

What did you try to do?

Install an iRODS plugin ( in this case the Python rule engine plugin ) RPM that writes files to /etc/irods or /var/lib/irods ...

Expected behavior

These directories should remain owned by irods service account user.

Observed behavior (including steps to reproduce, if applicable)

The directories become root-owned, preventing the irods user from being able to create new files within them. (As hinted, this was first discovered while running Python REP tests on Centos 7).

Explanation:

Because of a snag in CMake RPM defaults, whenever files are installed from a package, the paths "above" them are also marked as install targets.

master failing because it needs more json and currently relies on jansson

irods/irods#4234

irods / irods_rule_engine_plugin_python Goto Github PK

irods_rule_engine_plugin_python's Issues

Feature Request

Workaround

Bug Report

scenario (1):

scenario (2):

define _POSIX_C_SOURCE 200809L

define _XOPEN_SOURCE 700

Bug Report

iRODS Version, OS and Version

What did you try to do?

Expected behavior

Observed behavior (including steps to reproduce, if applicable)

Bug Report

iRODS Version, OS and Version

What did you try to do?

Expected behavior

Observed behavior (including steps to reproduce, if applicable)

Explanation:

Recommend Projects

Recommend Topics

Recommend Org

Jobs