EAV support about masquerade

[peterjaap]

Here's how another tool does it; https://github.com/mpchadwick/dbanon#magento-2

[erikhansen]

As a workaround for now, I am running a manual query to remove the one sensitive EAV attribute I found (added by a custom extension):

UPDATE customer_entity_varchar
    SET `value` = "[phone]"
    WHERE attribute_id =
    (
        SELECT attribute_id FROM eav_attribute
            WHERE attribute_code = "telephone"
            AND entity_type_id = (
                SELECT entity_type_id
                FROM eav_entity_type
                WHERE entity_type_code = "customer"
            )
    );

I found this attribute by reviewing the attribute codes returned from this query:

-- Find all attribute codes that are stored in the EAV tables and aren't category-related
SELECT `eav_attribute`.`attribute_id`,
	`eav_entity_type`.`entity_type_code`,
	`eav_attribute`.`attribute_code`,
	`eav_attribute`.`backend_type`
FROM `eav_attribute` 
LEFT JOIN `eav_entity_type` ON `eav_attribute`.`entity_type_id` = `eav_entity_type`.`entity_type_id`
WHERE 
	`eav_attribute`.`backend_type` != 'static' 
	AND `eav_entity_type`.`entity_type_code` NOT IN ('catalog_product', 'catalog_category')
ORDER BY `eav_entity_type`.`entity_type_code`, `eav_attribute`.`attribute_code`;

On a native Magento Open Source 2.3.2 install (with sample data), the above query won't return any data. However custom extensions frequently add EAV attributes, so you'll want to review this list to ensure data added by custom extensions are properly anonymized.

The above query does return a few attributes on a Magento 2.2.x Commerce installation. For example:

attribute_id	entity_type_code	attribute_code	backend_type
180	customer	reward_update_notification	int
181	customer	reward_warning_notification	int
178	order	reward_points_balance_refunded	integer
179	order	reward_salesrule_points	integer
191	rma_item	condition	int
192	rma_item	reason	int
193	rma_item	reason_other	varchar
190	rma_item	resolution	int

[johnorourke]

Just considering this... looking at the two linked approaches:

• both the Divante and dbanon tools have built in logic to parse the Magento EAV structure
• other systems have different structures (Drupal for example has something similar to EAV)

It seems to be there's a more general solution we could use, for example:

• on each table, allow a custom query to select the records (eg. select * from catalog_product_entity_varchar where attribute_id=57)
• allow queries which return a list of the primary keys to work on (eg. select value_id from catalog_product_entity_decimal inner join eav_attribute using (attribute_id) where eav_attribute.attribute_code like '%price%')
• allow "query providers", classes which can return a SQL query or even return an Illuminate query object - you could have a class for Magento EAV, a class for Wordpress post metadata, a class for Drupal field values, etc.

The 'query provider' idea could even return query objects for other tables - eg. you supply 'catalog_product_entity' and ask it to anonymise the 'cost_price' field, and it could return a query object for the 'catalog_product_entity_decimal' filtered to only records for that attribute.

catalog:
  catalog_product_entity:
    provider:
      - name: magento_eav
    columns:
      ...etc...

OR

catalog:
  custom_table:
    provider:
      - name: custom_query
        query: 'select id from custom_table where field1 = "value1"'

[johnorourke]

Thinking further on this, it would be nice to set locale differently for different sets of values, too:

• Magento example: you might want to set the Faker locale based on the default locale of the website which a customer is attached to
• Drupal example: you might want to set the Faker locale based on the 'language' field in the EAV-style table of field values

[johnorourke]

Working for me now with Magento2. I've also drafted an example of another table provider (not tested yet) - for Wordpress 'meta' tables: getjohn@5581edb

[peterjaap]

Thanks to @johnorourke we now have EAV support! 🎉 🥳

[johnorourke]

@peterjaap awesome, thanks so much for testing and merging. Glad I could help :)