Removal of identifier values in toMany() function about doctrine-laminas-hydrator

[michanismus]

@driehle Yes, I will create a PR the coming days.

[driehle]

If the hydrator doesn't work with custom ID generators, I think we should consider this a bug.

However, I have honestly no idea why this line is there. It seems like it has been added in 5a7bf9d by @phpboyscout. That commit follows up to e729e14, which says to update "toMany() to allow complex hydration" and refers to zfcampus/zf-apigility-doctrine#183.

You could try to remove this line and check if unit tests still all pass. Maybe also @TomHAnderson knows something about this?

[driehle]

Related:

• doctrine/DoctrineModule#527
• doctrine/DoctrineModule#535

[TomHAnderson]

This is my first trip down this rabbit hole. I would only suspect the line, mentioned above, removes the reference to the primary key in a case where an entity has a compound primary key with at least one generated value. A read edge case, to be certain, and I don't think this should cater towards that edge case, if I'm correct.

[michanismus]

Ok, I think this is a special case.

@driehle The tests fail with my adjustments.

In my case I also need a simple cache for objects with an ID set and nested relations (find function).
We can close the issue if you think so...

My classes (if you'd like to see my requirements):

<?php declare(strict_types=1);

namespace Mic\Core\Framework\DataAbstractionLayer\Doctrine\Hydrator;

use Doctrine\Laminas\Hydrator\DoctrineObject;
use Doctrine\Laminas\Hydrator\Strategy;
use Doctrine\Persistence\ObjectManager;
use Laminas\Stdlib\ArrayUtils;
use Mic\Core\Framework\DataAbstractionLayer\EntityInterface;
use RuntimeException;
use Traversable;

class Hydrator extends DoctrineObject
{
    protected $defaultByValueStrategy = FrameworkStrategy::class;

    protected array $objects = [];
    protected bool $processing = false;

    public function __construct(ObjectManager $objectManager)
    {
        parent::__construct($objectManager, true);
    }

    public function setDefaultByValueStrategy($defaultByValueStrategy)
    {
        throw new RuntimeException('Framework hydrator strategy cannot be changed.');
    }

    /**
     * {@inheritDoc}
     */
    public function hydrate(array $data, object $object)
    {
        if (!$object instanceof EntityInterface) {
            throw new RuntimeException('EntityInterface expected.');
        }

        $reset = false;
        if (false === $this->processing) {
            $this->processing = true;
            $reset = true;
        }

        $this->prepare($object);

        $this->hydrateByValue($data, $object);

        if ($object->getId() && !$this->objectManager->contains($object)) {
            $this->objects[$object->getId()] = $object;
        }

        if ($reset) {
            $this->objects = [];
            $this->processing = false;
        }

        return $object;
    }

    /**
     * {@inheritDoc}
     */
    protected function find($identifiers, $targetClass)
    {
        if ($identifiers instanceof $targetClass) {
            return $identifiers;
        }

        if ($this->isNullIdentifier($identifiers)) {
            return null;
        }

        if (is_array($identifiers) && isset($identifiers['id'])) {
            $id = (string) $identifiers['id'];
        }

        if (isset($id) && array_key_exists($id, $this->objects)) {
            return $this->objects[$id];
        }

        $object = $this->objectManager->find($targetClass, $identifiers);

        if (isset($id) && null !== $object) {
            $this->objects[$id] = $object;
        }

        return $object;
    }

    /**
     * {@inheritDoc}
     */
    protected function toMany($object, $collectionName, $target, $values)
    {
        $metadata   = $this->objectManager->getClassMetadata($target);
        $identifier = $metadata->getIdentifier();

        if (! is_array($values) && ! $values instanceof Traversable) {
            $values = (array) $values;
        } elseif ($values instanceof Traversable) {
            $values = ArrayUtils::iteratorToArray($values);
        }

        $collection = [];

        // If the collection contains identifiers, fetch the objects from database
        foreach ($values as $value) {
            if ($value instanceof $target) {
                // assumes modifications have already taken place in object
                $collection[] = $value;
                continue;
            }

            if (empty($value)) {
                // assumes no id and retrieves new $target
                $collection[] = $this->find($value, $target);
                continue;
            }

            $find = [];
            if (is_array($identifier)) {
                foreach ($identifier as $field) {
                    switch (gettype($value)) {
                        case 'object':
                            $getter = 'get' . $this->inflector->classify($field);

                            if (is_callable([$value, $getter])) {
                                $find[$field] = $value->$getter();
                            } elseif (property_exists($value, $field)) {
                                $find[$field] = $value->$field;
                            }

                            break;
                        case 'array':
                            if (array_key_exists($field, $value) && $value[$field] !== null) {
                                $find[$field] = $value[$field];
                                // unset($value[$field]); // removed identifier from persistable data
                            }

                            break;
                        default:
                            $find[$field] = $value;
                            break;
                    }
                }
            }

            if (! empty($find)) {
                $found = $this->find($find, $target);
                if ($found) {
                    $collection[] = is_array($value) ? $this->hydrate($value, $found) : $found;
                    continue;
                }
            }

            $collection[] = is_array($value) ? $this->hydrate($value, new $target()) : new $target();
        }

        $collection = array_filter(
            $collection,
            static function ($item) {
                return $item !== null;
            }
        );

        // Set the object so that the strategy can extract the Collection from it

        $collectionStrategy = $this->getStrategy($collectionName);
        assert($collectionStrategy instanceof Strategy\AbstractCollectionStrategy);
        $collectionStrategy->setObject($object);

        // We could directly call hydrate method from the strategy, but if people want to override
        // hydrateValue function, they can do it and do their own stuff
        $this->hydrateValue($collectionName, $collection, $values);
    }

    /**
     * {@inheritDoc}
     */
    private function isNullIdentifier($identifier)
    {
        if ($identifier === null) {
            return true;
        }

        if ($identifier instanceof Traversable || is_array($identifier)) {
            // Psalm infers iterable as a union of array|Traversable, but
            // ArrayUtils::iteratorToArray() doesn't accept iterable, so this
            // needs to be overwritten manually here.
            // See https://github.com/vimeo/psalm/issues/6682
            /** @psalm-var array|Traversable $identifier */
            $nonNullIdentifiers = array_filter(
                ArrayUtils::iteratorToArray($identifier),
                static function ($value) {
                    return $value !== null;
                }
            );

            return empty($nonNullIdentifiers);
        }

        return false;
    }
}

<?php declare(strict_types=1);

namespace Mic\Core\Framework\DataAbstractionLayer\Doctrine\Hydrator;

use Doctrine\Laminas\Hydrator\Strategy\AllowRemoveByValue;
use Doctrine\Common\Collections\ArrayCollection;
use Doctrine\Common\Collections\Collection;
use LogicException;

use function array_udiff;
use function get_class;
use function method_exists;
use function sprintf;

class FrameworkStrategy extends AllowRemoveByValue
{
    /**
     * Converts the given value so that it can be hydrated by the hydrator.
     *
     * @param  mixed      $value The original value.
     * @param  array|null $data  The original data for context.
     *
     * @return mixed      Returns the value that should be hydrated.
     */
    public function hydrate($value, ?array $data)
    {
        $setter = 'set' . $this->inflector->classify($this->collectionName);

        if (!method_exists($this->object, $setter)) {
            throw new LogicException(
                sprintf(
                    'Framework strategy for DoctrineModule hydrator requires setter %s to be defined in %s
                     entity domain code, but seem to be missing',
                    $setter,
                    get_class($this->object)
                )
            );
        }

        $this->object->$setter($value);

        return $value;
    }
}

[driehle]

I analysed this further and looked into whats happens when commenting out the unset($value[$field]); line. The result is the following:

1) DoctrineTest\Laminas\Hydrator\DoctrineObjectTest::testHydrateOneToManyAssociationByReferenceUsingIdentifiersArrayForRelations
Error: Cannot access protected property DoctrineTest\Laminas\Hydrator\Assets\ByValueDifferentiatorEntity::$id

/home/driehle/PhpstormProjects/doctrine-laminas-hydrator/src/DoctrineObject.php:432
[...]

Now this is certainly weird, because $reflProperty->setAccessible(true); is called before, so in line 432 there really cannot be such an error. I therefore added a check if the object and the used reflection class match, as such:

    protected function hydrateByReference(array $data, $object)
    {
        $tryObject = $this->tryConvertArrayToObject($data, $object);
        $metadata  = $this->metadata;
        $refl      = $metadata->getReflectionClass();

        if (is_object($tryObject)) {
            $object = $tryObject;
        }

        if (get_class($object) !== $refl->getName()) {  // ADDITIONAL DEBUG CHECK START
            echo "\nObject:     " . get_class($object) . "\n";
            echo "Reflection: " . $refl->getName() . "\n";
        }  // ADDITIONAL DEBUG CHECK END

        foreach ($data as $field => $value) {
            $field = $this->computeHydrateFieldName($field);

            // Ignore unknown fields
            if (! $refl->hasProperty($field)) {
                continue;
            }

            $reflProperty = $refl->getProperty($field);
            $reflProperty->setAccessible(true);

            if ($metadata->hasAssociation($field)) {
                $target = $metadata->getAssociationTargetClass($field);
                assert($target !== null);

                if ($metadata->isSingleValuedAssociation($field)) {
                    $value = $this->toOne($target, $this->hydrateValue($field, $value, $data));
                    $reflProperty->setValue($object, $value);
                } elseif ($metadata->isCollectionValuedAssociation($field)) {
                    $this->toMany($object, $field, $target, $value);
                }
            } else {
                $reflProperty->setValue($object, $this->hydrateValue($field, $value, $data)); // LINE 432
            }

            $this->metadata = $metadata;
        }

        return $object;
    }

And the output of that is:

Object:     DoctrineTest\Laminas\Hydrator\Assets\ByValueDifferentiatorEntity
Reflection: DoctrineTest\Laminas\Hydrator\Assets\OneToManyEntity

So there is a bug in the unit tests. The test testHydrateOneToManyAssociationByReferenceUsingIdentifiersArrayForRelations is not correctly mocked, as it results the wrong reflection class from ClassMetadata. We are mocking ClassMetadata, but the unit test there did not consider, that class metadata needs to return different reflection classes for different entities. And in the mentioned unit tests, there are the entities OneToManyEntity and ByValueDifferentiatorEntity involved. I have created #45 to solve this issue.

Now here comes the interesting thing: Once that bug is fixed, it doesn't make any difference wheter we add or remove the line with unset($value[$field]);.

[michanismus]

Thanks for your work @driehle.

That means, once the changes for the test are reviewed and merged, we could remove the line unset($value[$field]); and everything should work as expected?

Otherwise an option would be to pass a kind of context, array or object, to the hydrator and for example line unset($value[$field]); will be executed or not, based on context options!?

Another thing I discovered in my case. I pass an ID in the array to hydrate and the hydrator tries to fetch the existing object via object manager. If it does not exist the object is a "new" one and will by hydrated to a new object. Now, when the same ID is used for associations and it is still not persisted a new object will be created twice or even multiple times and the relation is broken.

It is not related to this issue directly. But maybe it should be possible to add a kind of "find strategy" !?
I hope it is clear what I mean. Maybe have a look at my current implementation #44 (comment)

Edit:

The only modification to achieve that behavior (in my case) would be the following change of the hydrate() function:

    /**
     * Hydrate $object with the provided $data.
     *
     * {@inheritDoc}
     */
    public function hydrate(array $data, object $object)
    {
        $this->prepare($object);

        if ($this->byValue) {
            $object = $this->hydrateByValue($data, $object);
        } else {
            $object = $this->hydrateByReference($data, $object);
        }

        if ($object->getId() && !$this->objectManager->contains($object)) {
            $this->objectManager->persist($object);
        }

        return $object;
    }

[driehle]

@michanismus

That means, once the changes for the test are reviewed and merged, we could remove the line unset($value[$field]); and everything should work as expected?

From the perspective of what we cover in our unit tests, yes, we can remove that line. I might add a few more unit tests to double check this, but I don't expect any breaking changes.

Otherwise an option would be to pass a kind of context, array or object, to the hydrator and for example line unset($value[$field]); will be executed or not, based on context options!?

I strongly dislike this option for making the code unnecessarily complex and harder to maintain.

Another thing I discovered in my case. I pass an ID in the array to hydrate and the hydrator tries to fetch the existing object via object manager. If it does not exist the object is a "new" one and will by hydrated to a new object. Now, when the same ID is used for associations and it is still not persisted a new object will be created twice or even multiple times and the relation is broken.

This will only happen if you create the "same" new object twice within one hydration. This is not supported by the hydrator and I think the current architecture doens't allow doing so. It is very common to have IDs generated upon persistence, i.e. let MySQL/PostgreSQL assign an ID. In such a case, there is no possibility to identify duplicate objects during persistence, simply because they don't have an ID at that point.

The issue only arises when using custom ID generators, like you do. That's the only possibility to actually enter a new object twice.

It is not related to this issue directly. But maybe it should be possible to add a kind of "find strategy" !? I hope it is clear what I mean. Maybe have a look at my current implementation #44 (comment)

That could be a possible solution. Please provide a pull request, then we can see if we can get this feature in one of the next releases.

The only modification to achieve that behavior (in my case) would be the following change of the hydrate() function:

    /**
     * Hydrate $object with the provided $data.
     *
     * {@inheritDoc}
     */
    public function hydrate(array $data, object $object)
    {
        $this->prepare($object);

        if ($this->byValue) {
            $object = $this->hydrateByValue($data, $object);
        } else {
            $object = $this->hydrateByReference($data, $object);
        }

        if ($object->getId() && !$this->objectManager->contains($object)) {
            $this->objectManager->persist($object);
        }

        return $object;
    }

This may work in your case, but it cannot be transferred to other cases. First of all, DoctrineORM supports composite primary keys, i.e. the primary key may be more than one column. Furthermore, it doesn't have to be named id / getId() (though one could retrieve the actual name of the identity columns from ClassMetadata). And last but not least, when IDs are not assigned using a custom generator, objects do not have an ID before they are persisted.

[driehle]

#45 is now merged, would you mind providing a PR?