Bogus method wrappers about eman2

[sludtke42]

Hi John, When EMAN2 was first written >15 years ago, there was an automatic converter which would process the C++ code to produce templates like this, and this level of detail was, indeed required. Our general policy for this type of large scale change is, if it ain't broke, don't fix it, as we have (very) limited manpower, and this sort of refactoring generally does lead to bugs which also need to then be fixed. Meaning, we only fix it, if it impacts functionality. As to the first case, it looks like there are several overloading functions up higher in the code which are actually taking precedence over this line, which is correct, but is calling the incorrect function (I imagine someone copied this from another line and made an editing error. I'll take care of this one. There are actually quite a few places where the same function is defined both using the automatic wrapper syntax as well as explicit overloading functions. These were due to the extremely poor Boost python documentation at the time, which led one programmer to believe that both were necessary in many cases. It wasn't until many years later that things became clearer, and we understood that there were some redundancies. As they don't impact functionality, they haven't been remediated. Regarding the Vec4f, ... Yes, you will find quite a few cases in the code with const mismatches. These normally will go in the permissible direction, ie - passing a non-const into a function which takes a const. Yes, these produce warning messages, but are harmless, and as I said, we simply don't have the manpower available for code remediation at that level when there is no functional impact. The level of programming skill on people contributing code varies pretty widely, and const mismatches are probably not the radar of quite a lot of students. While I welcome cleanup of issues like this, please just fix it and do a PR. EMAN2 has ~1/2 million lines of code, and if we get a reported Issue for every C++ warning we won't be able to use the Issue system for problems with functional impact any more. I will take a look at the translate_newbasis issue, as there does seem to be a missing C++ function for one of the overloads... cheers

…

-------------------------------------------------------------------------------------- Steven Ludtke, Ph.D. <[email protected] <mailto:[email protected]>> Baylor College of Medicine Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem <http://www.bcm.edu/biochem>) Academic Director, CryoEM Core (cryoem.bcm.edu <http://cryoem.bcm.edu/>) Co-Director CIBR Center (www.bcm.edu/research/cibr <http://www.bcm.edu/research/cibr>)

On Apr 10, 2020, at 12:36 PM, John Bollinger ***@***.***> wrote: It seems to be standard practice in this codebase to insert cast operators into the declarations of Boost.Python method wrappers, such as in this example from line 612 of libpyEM/libpyAligner2.cpp: .def("compute_1d_fromimage", (std::vector<float,std::allocator<float> > (EMAN::EMAN2Ctf::*)(int, float, EMAN::EMData*) )&EMAN::EMAN2Ctf::compute_1d, (std::vector<float,std::allocator<float> > (EMAN_EMAN2Ctf_Wrapper::*)(int, float, EMAN::EMData*))&EMAN_EMAN2Ctf_Wrapper::default_compute_1d_fromimage) I'm not seeing the point of such casts (current Boost.Python does not seem to require them), but they shouldn't be harmful, provided that the target type of the cast is compatible with the type of the function. If it isn't compatible, however, as in the case of the above example, then undefined behavior ensues when the function is called via the wrapper. In practice, it looks like the mismatches of this type appearing in the codebase are likely to produce genuine misbehavior. The issue with the above example is that EMAN2Ctf::compute_1d takes four arguments (one with a default value), not three, and the type of the third is EMAN::Ctf::CtfType, not EMAN::EMData *`. Several other occurrences of such issues appear in libpyEM/libpyTransform2.cpp: at line 228 (Vec4f.set_value) at line 254 (Vec3f.set_value) at line 305 (Vec3i.set_value) at line 305 (Vec3i.set_value) at line 352 (Vec2f.set_value) at line 578(Transform.translate_newbasis overload) The problem in all the vector cases is that the relevant set_value() methods in the vector templates declare their arguments to be const references, whereas that cast specifies that they ardinary ordinary passed-by-value built-in types. The problem in the Transform case appears to be that the wrong method is being wrapped -- from the signature and a comparison to the order of method declarations in libEM/transform.h, it appears that the intention was to wrap the fourth form of Transform.translate(), not a second form of Transform.translate_newBasis() (which does not exist). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#461>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPKF2VGCN2YN4WATP4FV5DRL5KKZANCNFSM4MFSZIZQ>.

[jcbollinger]

The main problem with the Vector wrappers is not the constness mismatch, which I agree does not likely present an issue in practice, but rather the reference vs value mismatch, which is hard to believe yields a result that works in any way correctly, unless by the grace of a lucky compiler optimization.

Changing the cast to match the function does eliminate the warning, and, I expect, correct any misbehavior. But inasmuch as the vector templates appear to be used only for built-in numeric types, a bigger win might be achieved by changing the functions to match the cast -- that is, to accept arguments by value instead of by reference. I would have gone ahead and submitted the PR if I were confident which of those alternatives would be preferred.

[sludtke42]

Take a look at the first and last line here from libpyTransform: typedef void (EMAN::Vec4f::*vec4f_set_value_float)(const float&, const float&, const float&, const fl oat&); typedef float (EMAN::Vec4f::*vec4f_at_float)(int) const; class_< EMAN::Vec4f >("Vec4f", "typedef Vec4<float> Vec4f;", init< >()) .def(init< float, float, float, float >()) .def(init< const std::vector<float,std::allocator<float> >& >()) .def(init< const EMAN::Vec4i& >()) .def(init< const EMAN::Vec4f& >()) .def("normalize", &EMAN::Vec4f::normalize) .def("length", &EMAN::Vec4f::length) .def("set_value", (void (EMAN::Vec4f::*)(const std::vector<float,std::allocator<float> >&) )& EMAN::Vec4f::set_value) .def("set_value_at", vec4f_set_value_at_float(&EMAN::Vec4f::set_value_at)) .def("set_value", (void (EMAN::Vec4f::*)(float, float, float, float) )vec4f_set_value_float(& EMAN::Vec4f::set_value)) This certainly isn't the way I would have written this, and may have come out of the old automatic template writing code (which Boost no longer even has), but you can see the pass by value to pass by reference is handled here if rather obliquely.

…

-------------------------------------------------------------------------------------- Steven Ludtke, Ph.D. <[email protected] <mailto:[email protected]>> Baylor College of Medicine Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem <http://www.bcm.edu/biochem>) Academic Director, CryoEM Core (cryoem.bcm.edu <http://cryoem.bcm.edu/>) Co-Director CIBR Center (www.bcm.edu/research/cibr <http://www.bcm.edu/research/cibr>)

On Apr 10, 2020, at 5:28 PM, John Bollinger ***@***.***> wrote: The main problem with the Vector wrappers is not the constness mismatch, which I agree would not likely present an issue in practice, but rather the reference vs value mismatch, which is hard to believe yields a result that works in any way correctly, unless by the grace of a lucky compiler optimization. Changing the cast to match the function does eliminate the warning, and, I expect, correct any misbehavior. But inasmuch as the vector templates appear to be used only for built-in numeric types, a bigger win might be achieved by changing the functions to match the cast -- that is, to accept arguments by value instead of by reference. I would have gone ahead and submitted the PR if I were confident which of those alternatives would be preferred. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#461 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPKF2TSRRLSW2PJ4VSESPLRL6MP5ANCNFSM4MFSZIZQ>.

[sludtke42]

I'd also add that I have no idea why these are being passed by reference in the first place. I'd agree that it doesn't seem to make a lot of sense here, particularly since the reference is larger than the value itself. However, changing this would require someone to filter through all of the code and make sure that this really isn't used anywhere else in the whole system.

…

-------------------------------------------------------------------------------------- Steven Ludtke, Ph.D. <[email protected] <mailto:[email protected]>> Baylor College of Medicine Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem <http://www.bcm.edu/biochem>) Academic Director, CryoEM Core (cryoem.bcm.edu <http://cryoem.bcm.edu/>) Co-Director CIBR Center (www.bcm.edu/research/cibr <http://www.bcm.edu/research/cibr>)

On Apr 10, 2020, at 5:54 PM, Steve Ludtke ***@***.***> wrote: Take a look at the first and last line here from libpyTransform: typedef void (EMAN::Vec4f::*vec4f_set_value_float)(const float&, const float&, const float&, const fl oat&); typedef float (EMAN::Vec4f::*vec4f_at_float)(int) const; class_< EMAN::Vec4f >("Vec4f", "typedef Vec4<float> Vec4f;", init< >()) .def(init< float, float, float, float >()) .def(init< const std::vector<float,std::allocator<float> >& >()) .def(init< const EMAN::Vec4i& >()) .def(init< const EMAN::Vec4f& >()) .def("normalize", &EMAN::Vec4f::normalize) .def("length", &EMAN::Vec4f::length) .def("set_value", (void (EMAN::Vec4f::*)(const std::vector<float,std::allocator<float> >&) )& EMAN::Vec4f::set_value) .def("set_value_at", vec4f_set_value_at_float(&EMAN::Vec4f::set_value_at)) .def("set_value", (void (EMAN::Vec4f::*)(float, float, float, float) )vec4f_set_value_float(& EMAN::Vec4f::set_value)) This certainly isn't the way I would have written this, and may have come out of the old automatic template writing code (which Boost no longer even has), but you can see the pass by value to pass by reference is handled here if rather obliquely. -------------------------------------------------------------------------------------- Steven Ludtke, Ph.D. ***@***.*** ***@***.***>> Baylor College of Medicine Charles C. Bell Jr., Professor of Structural Biology Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem <http://www.bcm.edu/biochem>) Academic Director, CryoEM Core (cryoem.bcm.edu <http://cryoem.bcm.edu/>) Co-Director CIBR Center (www.bcm.edu/research/cibr <http://www.bcm.edu/research/cibr>) > On Apr 10, 2020, at 5:28 PM, John Bollinger ***@***.*** ***@***.***>> wrote: > > > The main problem with the Vector wrappers is not the constness mismatch, which I agree would not likely present an issue in practice, but rather the reference vs value mismatch, which is hard to believe yields a result that works in any way correctly, unless by the grace of a lucky compiler optimization. > > Changing the cast to match the function does eliminate the warning, and, I expect, correct any misbehavior. But inasmuch as the vector templates appear to be used only for built-in numeric types, a bigger win might be achieved by changing the functions to match the cast -- that is, to accept arguments by value instead of by reference. I would have gone ahead and submitted the PR if I were confident which of those alternatives would be preferred. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub <#461 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPKF2TSRRLSW2PJ4VSESPLRL6MP5ANCNFSM4MFSZIZQ>. >

[jcbollinger]

I apologize. It appears that I would have saved us both some time by preparing the following demo earlier. It segfaults when I run it against a binary installation of EMAN2 v2.31 (the official 64-bit Linux build, that is, not one of my own):

from EMAN2 import Vec3f

v = new Vec3f()
# prints "Vec3f(0.00,0.00,0.00)":
print v

# causes a segfault in the official build:
v.set_value(1.0, 1.0, 1.0)

# prints garbage in those of my own builds that reach this point:
print v

The behavior of the above example shows that the typedefs you point out and their use to interpose references to function pointers into the wrappers do not resolve the issue. I see no reason to think that they ever did. My best guess is that they were effective at disguising the problem from the contributor's compiler, so that it no longer warned, leading the contributor to suppose erroneously that the issue was in fact resolved.

The Python code in the current codebase does not exercise these particular wrappers, which mitigates the issue, and may explain why it has not been fixed. Of course, they could be unused in part because the issue has not been fixed. In any event, the issue stands as a trap for project contributors who try to use those wrappers in the future.

I will shortly submit a PR that actually does fix the problem for each of the wrapped vector classes. Although I think that the best solution would be to change the vector templates so that these particular functions accept their arguments by value (as their history shows they once did), and although my analysis of the current codebase is that doing so would have only positive effects any and everywhere within, I understand your reluctance to apply changes with broad scope. Accordingly, the PR will resolve the issue simply by fixing the wrappers to match the classes and removing the needless, otherwise-unused typedefs.