Inconsistent shape of instance_defaults.value between ObjectiveMechanism and its function_object about psyneulink

[kmantel]

I'm pretty sure this would be "fixed" by removing these lines, but the problem is that there are a lot of widespread assumptions about the value of mechanisms always being 2d, so it's not quite that simple. If it would help, we could cast the default value for a function assigned to a mechanism to 2d, until we have more time to work on the main issue.

index d7ad5a524..43974c6eb 100644
--- a/psyneulink/components/mechanisms/mechanism.py
+++ b/psyneulink/components/mechanisms/mechanism.py
@@ -2219,26 +2219,7 @@ class Mechanism_Base(Mechanism):
                                               context=context,
                 )
 
-                # IMPLEMENTATION NOTE:  THIS IS HERE BECAUSE IF return_value IS A LIST, AND THE LENGTH OF ALL OF ITS
-                #                       ELEMENTS ALONG ALL DIMENSIONS ARE EQUAL (E.G., A 2X2 MATRIX PAIRED WITH AN
-                #                       ARRAY OF LENGTH 2), np.array (AS WELL AS np.atleast_2d) GENERATES A ValueError
-                if (isinstance(return_value, list) and
-                    (all(isinstance(item, np.ndarray) for item in return_value) and
-                        all(
-                                all(item.shape[i]==return_value[0].shape[0]
-                                    for i in range(len(item.shape)))
-                                for item in return_value))):
-
-                        return return_value
-                else:
-                    converted_to_2d = np.atleast_2d(return_value)
-                # If return_value is a list of heterogenous elements, return as is
-                #     (satisfies requirement that return_value be an array of possibly multidimensional values)
-                if converted_to_2d.dtype == object:
-                    return return_value
-                # Otherwise, return value converted to 2d np.array
-                else:
-                    return converted_to_2d
+                return return_value
 
             # Call only subclass' function during initialization (not its full _execute method nor rest of this method)
             elif self.initMethod is INIT_FUNCTION_METHOD_ONLY:
@@ -2291,27 +2272,6 @@ class Mechanism_Base(Mechanism):
             context=context
         )
 
-        # IMPLEMENTATION NOTE:  THIS IS HERE BECAUSE IF return_value IS A LIST, AND THE LENGTH OF ALL OF ITS
-        #                       ELEMENTS ALONG ALL DIMENSIONS ARE EQUAL (E.G., A 2X2 MATRIX PAIRED WITH AN
-        #                       ARRAY OF LENGTH 2), np.array (AS WELL AS np.atleast_2d) GENERATES A ValueError
-        if (isinstance(value, list) and
-            (all(isinstance(item, np.ndarray) for item in value) and
-                all(
-                        all(item.shape[i]==value[0].shape[0]
-                            for i in range(len(item.shape)))
-                        for item in value))):
-                pass
-        else:
-            converted_to_2d = np.atleast_2d(value)
-            # If return_value is a list of heterogenous elements, return as is
-            #     (satisfies requirement that return_value be an array of possibly multidimensional values)
-            if converted_to_2d.dtype == object:
-                pass
-            # Otherwise, return value converted to 2d np.array
-            else:
-                # return converted_to_2d
-                value = converted_to_2d
-
         # Set status based on whether self.value has changed
         self.status = value
 

[jvesely]

If "mechanism value is 2d" and "function_object value matches mechanism value" should both be preserved than casting the value looks like a solution to me.

Current compilation code relies on the latter. It's not difficult to adapt if such a guarantee is too cumbersome to maintain, but it simplifies the code.

[kmantel]

I'm about to merge in a commit that would do this casting, though not for Reinforcement and BackPropagation functions because of what they return. Some casual testing hasn't found any mismatches for these functions, but if you find any, reopen this issue/let me know.

[jvesely]

The commit (c040883) turns LCControlMechanism output into 3d structure:

>>> import psyneulink as pnl
>>> A = pnl.LCControlMechanism(default_variable=10.0)
>>> print(A.instance_defaults.value)
(array([[ 0.5]]), array([[ 0.]]), array([[ 0.]]), array([ 0.]))

The double arrays look redundant. Was the change in format intentional?
If it was, would vector values expand in the first or second dimension of the 2d array?

[kmantel]

It seems that my first attempt maybe went too far. Several Functions return tuples, and in these cases I'm not sure what the right conversion is or if it even makes sense to convert. (FHNIntegrator used by LCControlMechanism is this example). Also in DDM, matching the function value and mechanism value seems not easy to do in a general way at the Function level considering how the mechanism value is constructed.

Is it helpful to keep the casting for most normal functions if you still have to workaround a few cases like above?

[jvesely]

This report was the last remaining inconsistency that I ran into. I'm not sure what the correct solution for PsyNeuLink is. The more general solution the better so I don't have to special case situations (I think the python codepaths benefit from this as well).

On a high level, I expect that the result for the last function in a mechanism (is that always self.function_object?) can be passed to output state parsing function (as specified by parse_variable_spec function).

I think the LCControlMechanism value format matched the function value format (otherwise I'd need a workaround for that as well), so my take is that the functions that return multiple values don't need conversion.

[kmantel]

Ok. I think making a fully correct and general solution would take significant effort and design, and probably should be considered when/if the 2d restriction on mechanism values is lifted. So, if the latest changes in 17b09bf will work for you without workarounds then I'll leave it at that for now.

[jvesely]

Thank you. with current devel and 17b09bf I can remove the last output shape workaround and pass all the tests.

[jvesely]

I could remove the workaround in db76513.
Thank you.