data-dog / datadogauditbundle Goto Github PK
View Code? Open in Web Editor NEWStores all database changes for doctrine ORM
License: MIT License
Stores all database changes for doctrine ORM
License: MIT License
Is it possible to have this bundle work for multiple connections?
I have the Users table in a separate database, and I need to keep an audit log of what the users are changing in different databases.
I see that the "assocations" table only keeps string values so it should be possible but I installed it, configured it in each connection in doctrine.yaml, but it does not store audit for cross-connection changes.
Currently unable to install this bundle with Symfony 5.
Currently this bundle isn't compatible with Symfony 6.
Currently every log entry gets its own timestamp. This is the same except when the second changes while the log is written (rounding to seconds).
This difference in the timestamp is an irrelevant detail. (Writing the entity has happened before anyway.) And it makes reading more complicated. Therefore please write the same timestamp while in one flush call.
Hi,
is there any plan to support setting blamed user
from impersonated user
(where applicable)?
Are PR accepted?
Cheers
Hi,
We're currently building a new large project with a relatively large database (around 200 tables).
We need to keep track of changes done on mostly all tables and be able to show diffs through the UI.
This bundle seems to fit our needs but, if I'm not mistaken, it stores changes in only two tables. So, with a lot of users, I fear that we'll quickly have two gigantic tables with billions of records which will make them slow to lookup or to query (to show diff in the UI) and as a consequence will be harder to backup (and restore).
So, is it possible to split audit logs into more tables (one audit table per audited table for example as simplethings/entity-audit-bundle does) ?
If no, do you have some feedback about similar use cases?
I installed the bundle, I checked that it registers in database, but I do not know how to show the view with logs.
Is there any possibility to retrieve the ID of the entity which is created/updated/deleted? In the audit_logs table I only see the fields that are inserted in the 'diff'-field but not the ID's such that I can retrieve the particular entity.
@l3pp4rd Do you have an idea if there are issues with making this compatible with Symfony 3?
I can't get any soft-delete event logged. I'm using DoctrineExtensions for soft-delete but changes doesn't get logged as remove and neither as update.
When logging in as user Yoda, I get
You have requested a non-existent service "security.context".
Hi,
Is it possible to forward the logging to queue system (for example rabbitmq)? I'm afraid that flushing the logs on the same transaction will slow it down. By forwarding it to queue system on post flush, it will fasten the process while still have the logging feature.
Thanks.
Hi!
I'm looking to use this bundle, but found OneToMany associations are not recorded on inversed side (mappedBy) of a bi-directional relation.
This makes it hard to see what relations an entity had on creation.
You can see it from owning side, but this isn't really enough for a good audit.
For example If you have Project with OneToMany Tickets, Ticket->Project is recorded when viewing the diff for a ticket, but Project->Tickets is not recorded when viewing the Project, and it's very hard to find the tickets for a project.
I've had a quick look at the code. We could approach it two ways.
AuditSubscriber::associate
method for One-To-Many relations (at the moment it only does Many-To-Many collection changes).Possible option 1 might be nicer, as dissociate
could also be used to record removals, for example if a Ticket was deleted, a Project would be dissociated
from the ticket.
Would you be willing to accept a PR?
Do you have a preference on approach?
Thanks.
There is no documentation how various relations are stored.
Please write documentation about this.
Or even better write the OneToMany associations handier. (Doing database queries for a json string is unhandy and not precise. Even finding the fk requires this hack.)
What I found out:
ManyToMany association, owning side:
AuditLog (of owning side entity):
action: 'associate' #
source:
# -> points to Association
id: # internal id of this association
typ: # type (similar to bundle+class) of associated entity
tbl: # table where the entity of this association is stored
label: # value of associated entity, as string
fk: # id of the associated entity
class: # class of associated entity
target:
# -> points to Association
class: class of related association
tbl: table of ...
fk: id of related association
diff: []
# empty array
ManyToMany association, other side:
#nothing
OneToMany association, owning side:
AuditLog (of owning side entity):
action: 'update' # or insert
source:
# -> points to Association
fk: id of this audit log
target:
null
diff: # as json
...
associationProperyName:
old:
typ: # see in ManyToMany
tbl: # see in ManyToMany
label: # see in ManyToMany
fk: # id of the associated entity
class: # see in ManyToMany
new:
typ: # see in ManyToMany
...
col: # column name
I see the setLabeler function in the audit subscriber but I don't know where to set it. Could you provide an example of how to use the labeler in the example?
Hello,
I think about possibility not to log specified fields on audited entity. We could use for each variable representing column (in Entity) such annotation: @DataDogAudit\Unaudited . These information can be used to filter $uow->getEntityChangeSet output (unset unaudited elements in array). Due to that change, we can for example avoid logging critical private data or columns changed automatically by application. If I would develop this functionality, would it be included in audit bundle? (changes in AuditSubscriber, probably new file in DBAL folder) .
Add indexes for columns that are likely going to be used for searching for data.
Meaning keep track of modification and revert them with a version number ?
(I am testing the loggable tool to do so, but I am interesting to use your tool as well to track every changes)
Thanks for your work !
Add a repository for fetching entries.
PHP Fatal error: Uncaught TypeError: Argument 1 passed to DataDog\AuditBundle\EventSubscriber\AuditSubscriber::__construct() must be an instance of Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorage, instance of Symfony\Component\Security\Core\Authentication\Token\Storage\UsageTrackingTokenStorage given, called in /tmp/intima_admin/cache/ContainerKRKwjFo/srcApp_KernelDevDebugContainer.php on line 373 and defined in /home/vagrant/code/intima_admin/vendor/data-dog/audit-bundle/src/DataDog/AuditBundle/EventSubscriber/AuditSubscriber.php:49
I want to log all changes of ORM entity into a Mongo database. Is this possible?
I've recently run into a problem, we need to do a database update which requires some data migrations. I've written a symfony command to do the data migrations but it creates an AuditLog with a null
blame id. I was able to get around the problem my hacking around it and simulating a login like this: https://stackoverflow.com/questions/22606444/symfony-2-simulate-login-and-security-context-in-console-command
But it would be great if we could programatically set the user that the bundle will use, just like the doctrine extensions blameable: https://github.com/Atlantic18/DoctrineExtensions/blob/v2.4.x/lib/Gedmo/Blameable/BlameableListener.php#L60
I'm happy to contribute this myself if you would like?
Add automated tests to ensure compatibility with supported package versions.
I don't think it should create multiple rows with same data whenever object is used?
Looks like it could be an unintended consequence of the mapping changes in previous commits, and may only affect postgres, I'm not using this bundle with any other DB platforms at the moment so unable to confirm if it happens on Mysql, SQLite etc ...
(https://user-images.githubusercontent.com/4586381/157653353-4d1724d1-9249-46e6-aa12-ce9234b9ba86.png) exception
Originally posted by @acornforth in #93 (comment)
Hello,
This leads to
Fatal error: Uncaught Symfony\Component\Debug\Exception\FatalThrowableError: Maximum function nesting level of '256' reached, aborting! in /srv/vendor/monolog/monolog/src/Monolog/Processor/PsrLogMessageProcessor.php on line 31
Symfony\Component\Debug\Exception\FatalThrowableError: Maximum function nesting level of '256' reached, aborting! in /srv/vendor/monolog/monolog/src/Monolog/Processor/PsrLogMessageProcessor.php on line 31
when a one is flushing more than 256 times in a script, while using this bundle. One can see in the stack trace a long chain of LoggerChain.php
instances, which at some point trips the php's configured max recursion level.
There are use cases when one needs to flush a lot in a script.
Fyi.
Is it possible to change the name of the tables and fields that are created by the package?
https://www.doctrine-project.org/projects/doctrine-orm/en/2.6/tutorials/embeddables.html
There should be support for it or some ability to skip it, because now it will fail to log whole entity.
Hi,
I am using your auditing bundle and I seem to have encountered a problem...
Here is the situation
I have a doctrine entity with a inverse one to one relationship.
When I try to modify it, auditing triggers and I get the following error
'MyProject\MyBundle\Entity\EntityA#entityB' is not an association with a single join column.
The error is triggered on line 377 of 'DataDog\AuditBundle\EventSubscriber\AuditSubscriber' :
private function diff(EntityManager $em, $entity, array $ch)
{
$uow = $em->getUnitOfWork();
$meta = $em->getClassMetadata(get_class($entity));
$diff = [];
foreach ($ch as $fieldName => list($old, $new)) {
if ($meta->hasField($fieldName)) {
$mapping = $meta->fieldMappings[$fieldName];
$diff[$fieldName] = [
'old' => $this->value($em, Type::getType($mapping['type']), $old),
'new' => $this->value($em, Type::getType($mapping['type']), $new),
'col' => $mapping['columnName'],
];
} elseif ($meta->hasAssociation($fieldName) && $meta->isSingleValuedAssociation($fieldName)) {
$mapping = $meta->associationMappings[$fieldName];
$colName = $meta->getSingleAssociationJoinColumnName($fieldName);
$assocMeta = $em->getClassMetadata($mapping['targetEntity']);
$diff[$fieldName] = [
'old' => $this->assoc($em, $old),
'new' => $this->assoc($em, $new),
'col' => $colName,
];
}
}
return $diff;
}
The else if block checks for $meta->isSingleValuedAssociation($fieldName)
before calling $meta->getSingleAssociationJoinColumnName($fieldName);
but in the Doctrine ClassMetadataInfo class we see :
public function getSingleAssociationJoinColumnName($fieldName)
{
if ( ! $this->isAssociationWithSingleJoinColumn($fieldName)) {
throw MappingException::noSingleAssociationJoinColumnFound($this->name, $fieldName);
}
return $this->associationMappings[$fieldName]['joinColumns'][0]['name'];
}
The function calls isAssociationWithSingleJoinColumn
(which is not checked for in the bundle code) and throws an exception.
I am using doctrine version 2.4.8.
Thanks!
don't work with composite key ?
Hi There,
I've looked through the documentation, and I can't see any configuration or similar in regards to the archival or rotation of entries.
The reason being that the audit tables get pretty large quite quickly on one of our instances.
Is there anything baked-in in terms of cleanup?
Thanks
Drop support for unsupported versions of Symfony and Doctrine. This will reduce complexity with backwards compatibility.
You have declared the AuditLog's field "diff" as "json_array" type, for which doctrine handles the conversion automatically.
In AuditSubscriber you convert to json the diff data (here and here) and the result in the database is that we have data encoded twice (in my case the second conversion applies a \ before every ") and when we try to retrieve the data using getDiff() we've the data as a json and not as array.
We are doing something wrong?
We're using Symfony 3.4 with doctrine/dbal 2.5
Update
We've seen pull request #51, and we think that it might've been implemented by mistake (probably it was an encoding issue on Mopster's side, since doctrine's json and json_array type require UTF-8 encoding on the DB's column).
Currently a blame entry is written to the audit_association table for every changed table. (Example: when one table and two associated ones are changed, the blaming user is written 3 times with the same info.) All of this blame entries contain the same data. (Because DataDogAudit is called at the very end, the user entity can not change while writing the log.)
Writing the same data multiple times bloats the database unnecessary.
This should not break backward compatibility (reading is the same as before).
ToDo:
As in subject.
Example on defining labeler (as from docs):
datadog.event_subscriber.audit:
class: 'DataDog\AuditBundle\EventSubscriber\AuditSubscriber'
arguments: ["@security.token_storage"]
tags:
- { name: doctrine.event_subscriber, connection: default }
calls:
- ['setLabeler', [['\AppBundle\Labeler\AuditLabeler', 'getLabel']]]
debug:config sees config okay
dump inside isEntityUnaudited proves the array of unaudited entities is empty.
Has anyone done any work on purging data for a specific entity and all its relations.
ie, we store data for a fixed period of time, after which we purge the "case" and need to remove all its audit records including a fairly deep set of nested relations.
not everything is stored for the same period, so it's not just a case of deleting old entries - the cut off is defined by a specific action on each case which is extremely variable
I'm running into an exception thrown in the DataDog\AuditBundle\EventSubscriber\AuditSubscriber class, line 320 when the subscriber tries to record changes to an entity with a composite key.
For now, I'm just adding it to unaudited entities. I'm willing to do modifications to help allow composite key entities to be audit-able. I'm thinking there may some discussion needed on strategy for handling things. I'll have to look into it more thoroughly when I find a space of time.
This is an awesome bundle! Only issue I'm having is that my project bundle is called AppBundle
and so is the one in the example. Which means that my Project
and User
entities get namespace conflicts with the example.
It doesn't actually cause any errors, but it does make PhpStorm complain that there are multiple definities for the classes.
Is there a way I can install the bundle without the example?
Hello
I can't run the demo
Because there are a lot of issues when trying make
When extending AuditSubscriber.php
in order to override the blame()
method, static analysis tools (phpStan) complain about unreachable code since it thinks that $blameUser
cannot be null
.
Typehint annotation for the $blameUser
prop, should be /** @var UserInterface|null $blameUser */
better yet, since support for php <7.4 is generally being dropped all over the place, maybe it's time to start using some proper PHP native types ?
In the audit_associations
table, Doctrine proxies are being used for the class
and typ
columns instead of the actual entity.
Expected:
class |
typ |
---|---|
App\Entity\User |
app.entity.user |
Result:
class |
typ |
---|---|
Proxies\__CG__\App\Entity\User |
proxies.__cg__.app.entity.user |
We have a bunch of entities with title fields that relate to the title of the user (Mr, Mrs, Miss)
Your switch statement on the subscriber prioritises a title field, which means the title gets saved as Mr rather then our __toString or name fields.
If you swap __toString to be the upmost priority (as toString is the defacto string representation of the object not any other field) and then check for other fields if its not implemented it would make it more user friendly. Adding a config in the yml for selecting a label field would also be a better implementation.
I have overridden the subscriber, created a new subscriber, inherited and overridden the label function myself for our project, but I think it would be beneficial to swap the order in the repo. I can make a pull request for this if you don't have time.
Other then that its a great bundle. Thanks.
Hello, this is a nice bundle and I want to implement it inside my project.
I'm using Symfony 3.3. I just include your src bundle and it's saving the data. But I want it to show the data and I don't know how to.
Another thing is that I think you´re not saving the user that's make the changes, is that right?
Thanks a lot for this bundle and I hope you can help me with it.
Best
The class 'DataDog\AuditBundle\Entity\AuditLog' was not found in the chain configured namespaces
facing this issue
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.