Sensitive Data Detection about data-describe

[terrytangyuan]

+1 to this idea and incorporating into this project as we discussed earlier today.

From maturity level, it seems like Presidio is a good option. It makes more sense to reuse the existing solution since a lot of domain knowledge and often business-specific.

Regarding anonymization, Presidio seems to also support some level of customizable anonymization. Do we want to leverage that? It looks like there isn't much popularity for Trumania. Perhaps we can just use Faker and build something suitable for our project?

[brianray]

GCP DLP does this but is cloud native. I like the API https://cloud.google.com/dlp/docs/apis. One thing I thought of was some sort of abstract data scheme maybe also with an architecture behind (like Apache Arrow) that enforced end-to-end handling of PII/PHI/Financial data. Can we:

• Detect where unknown
• Hash and encrypt where possible
• keep the general Info Types in tact (ref https://cloud.google.com/dlp/docs/infotypes-reference)

[terrytangyuan]

GCP DLP does this but is cloud native. I like the API https://cloud.google.com/dlp/docs/apis. One thing I thought of was some sort of abstract data scheme maybe also with an architecture behind (like Apache Arrow) that enforced end-to-end handling of PII/PHI/Financial data.

Could you elaborate a bit on this? What do you mean by "abstract data scheme"?

Can we:

* [ ]   Detect where unknown

* [ ]  Hash and encrypt where possible

* [ ]  keep the general Info Types in tact (ref https://cloud.google.com/dlp/docs/infotypes-reference)

These sound great to me.

[brianray]

"abstract data scheme", maybe something like what's found in PEP438 where we define a typed schema, ie:

from typing import NewType, TypedDict

EMAIL_ADDRESS = NewType('EMAIL_ADDRESS', string)

class Person(TypedDict):
    first: int
    last: int
    email: EMAIL_ADDRESS

And then when writing to the Person dict, we are real careful to secure the data by whatever means is decided acceptable. DLP has a number of ways to do this.

[terrytangyuan]

I see. That is a good idea.

[terrytangyuan]

I just created a new issue #34 for adding a design doc for this and assigned to you @truongc2. Please follow the template I put together in #33. It's easier to review designs on pull requests since we can comment in-line.

[truongc2]

Did you want to merge the template into master? Or was it intended for something else?

[terrytangyuan]

Yes, but needs at least one approval before merging. I just requested reviews.