Possible regression in enum handling with pyside2 about freecad

[wwmayer]

when calling the "Involute Gear" command from PartDesign

Does this only happen with this command or with others too that load a task dialog (e.g. Pad, Pocket, ...)?

[wwmayer]

I wonder whether the two toEnum() functions really have to be that complicated and that we can't simply write:

qsizetype PythonWrapper::toEnum(PyObject* pyPtr)
{
    try {
        Py::Long longObj(PyNumber_Long(pyPtr), true);
        return longObj.as_long();
    }
    catch (Py::Exception&) {
        Base::PyException e;
        e.ReportException();
        return 0;
    }
}

qsizetype PythonWrapper::toEnum(const Py::Object& pyobject)
{
    return toEnum(pyobject.ptr());
}

This will have the same effect like the casts to an int on Python side:
return int(QtGui.QDialogButtonBox.Ok | QtGui.QDialogButtonBox.Cancel)

In #13611 it was claimed that this cast caused various problems but when looking at the mentioned issues all of them had other reasons and thus I doubt that Shiboken::Enum::getValue is needed at all.

[wwmayer]

I haven't figured out who really caused this. It could be libshiboken

For some reason the passed value to Shiboken::Enum::getValue is not of the expected type in your case and that's why check() causes the assert to fail:

long Shiboken::Enum::getValue(PyObject *enumItem)
{
    assert(Shiboken::Enum::check(enumItem));
...
}

[NomAnor]

when calling the "Involute Gear" command from PartDesign

Does this only happen with this command or with others too that load a task dialog (e.g. Pad, Pocket, ...)?

From the code I think this will happen with any dialog that is created in python, but I haven't checked.

Using PyNumber_Long will fixed it. There are multiple helper functions to convert from python to c++. I a sense, using toEnum in getStandardButtons is just wrong because it returns a QFlags not an enum so it should call a toFlags function. On the other hand toEnum returns an int not an enum which should probably a template function with a static_cast<T>() in it.

I haven't figured out who really caused this. It could be libshiboken

For some reason the passed value to Shiboken::Enum::getValue is not of the expected type in your case and that's why check() causes the assert to fail:

That was clear, but it worked before and I supect some weired interactions with package updates. But It could also be just the PR that changed the int casts.

I can test with your simplified function, I'm sure that will work.

[NomAnor]

@wwmayer I tested the simplified functions you posted and the crash is gone.