trixterfilm / trackteroid Goto Github PK
View Code? Open in Web Editor NEWDeclarative, object-oriented wrapper for Ftrack queries. Powerful functional-style interactions with resulting collections.
License: BSD 3-Clause "New" or "Revised" License
Declarative, object-oriented wrapper for Ftrack queries. Powerful functional-style interactions with resulting collections.
License: BSD 3-Clause "New" or "Revised" License
Get rid of sourcing entity classes like this getattr(importlib.import_module("..entities", __name__), entity_type.__name__)
.
We have more and more use cases where we need to retrieve the actual entity class and this is an ugly approach.
We could potentially expose a proper type map on the session instance that allows convenient access to entity classes.
When accessing a collection attribute before it's been queried, that attribute becomes inaccessible. We have tracked it down to a bug in the Ftrack API, but because trackteroid internally uses the Ftrack API, this issue is also affecting it. This issue is to come up with a workaround for it while Ftrack gets it fixed.
Even if the initial collection entity has no data, after querying that field it should be available.
Provide any relevant background information or context that may help in understanding the issue.
The pure ftrack reproducible code:
import ftrack_api
session = ftrack_api.Session(auto_populate=False)
# We query an Asset on ftrack that has at least one version
asset = session.query("Asset where versions.id like '%'").first()
# Print the list of versions. Because we haven't included the "versions" projection, this will
# expectedly print an empty list. This line is what triggers the bug: When we access asset["versions"],
# in AbstractCollectionAttribute.get_value:L437 there's a piece of code that checks whether the value is
# NOT_SET and in that case it sets it's local value to None, which here translates to an empty collection.
# The rationale of this is stated on a comment on said piece of the source code, the users should be able
# to use the same collection API whether there's any data or not, this desired behaviour is understandable,
# but unfortunately we have unknowingly set the local value of the entity's ftrack attribute storage to an
# empty collection without recording the operation, meaning that now the local value is favored over the
# remote value and will always be returned even if later on we populate the remote value
print(list(asset["versions"]))
# Now we query that same asset but this time we want to fetch the "versions" field.
asset = session.query(f"select versions from Asset where id is {asset['id']}").one()
# As mentioned before, this will now perform a lookup of the ftrack attribute storage dict, find a local value
# and think that that local value is the correct one, not being able to distinguish between an actual user created
# local value and the placeholder collection the API has set automatically. While the versions are properly contained
# in the remote attribute storage key, the empty collection from the local will be returned here.
print(list(asset["versions"]))
# Now to properly visualize the bug, we can create a new session, devoid of any local overrides and run the same
# query again. This time, because there's no local value already set, and we are actually retrieving that data we
# can see how the asset now returns its actual list of versions.
session = ftrack_api.Session(auto_populate=False)
asset = session.query(f"select versions from Asset where id is {asset['id']}").one()
print(list(asset["versions"]))
And by extension, this also happens on trackteroid:
from trackteroid import Query, Asset, AssetVersion, SESSION
asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first()
print(asset_collection.versions.id)
asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first(projections=["versions"])
print(asset_collection.versions.id)
SESSION.reconnect()
asset_collection = Query(Asset).by_id(AssetVersion, "%").get_first(projections=["versions"])
print(asset_collection.versions.id)
Using the logic implemented in Ftrack's __getitem__
method within the entities triggers the issue, but individually getting the local or remote values of the internal does not. Therefore a good workaround would be to ensure that when accessing the internal Ftrack entities, instead of using entity[attribute]
or entity.get(attribute)
we do something like this:
attribute = entity.attributes.get(item)
local_value = attribute.get_local_value(entity)
remote_value = attribute.get_remote_value(entity)
if local_value is not ftrack_api.symbol.NOT_SET:
return local_value
return remote_value
Updating used_in_versions
should be reflected in uses_versions
of the assigned collection and vise versa.
Currently, the operation is recorded properly and can be updated on the server when committing but it is not reflected within the local cache.
This code can be used to reproduce the issue:
from trackteroid import *
collection1 = Query(AssetVersion).get_all(limit=1, projections=["used_in_versions", "uses_versions"])
collection2 = Query(AssetVersion).get_all(limit=1, offset=1, projections=["used_in_versions", "uses_versions"])
print(
f"collection1 id: {collection1.id[0]}\n"
f"collection2 id: {collection2.id[0]}\n"
f"collection1 uses ids: {collection1.uses_versions.id}\n"
f"collection2 used in ids: {collection2.used_in_versions.id}\n"
)
# output:
# collection1 id: 029a25ce-8b60-11eb-bdb7-c2ffbce28b68
# collection2 id: 082158fc-6591-11ed-a73a-92ba0fc0dc3d
# collection1 uses ids: ['1fe0e8ae-6596-11ed-a73a-92ba0fc0dc3d']
# collection2 used in ids: EmptyCollection[AssetVersion]
collection1.uses_versions = collection2
print(
f"collection1 id: {collection1.id[0]}\n"
f"collection2 id: {collection2.id[0]}\n"
f"collection1 uses ids: {collection1.uses_versions.id}\n"
f"collection2 used in ids: {collection2.used_in_versions.id}\n"
)
# output:
# collection1 id: 029a25ce-8b60-11eb-bdb7-c2ffbce28b68
# collection2 id: 082158fc-6591-11ed-a73a-92ba0fc0dc3d
# collection1 uses ids: ['082158fc-6591-11ed-a73a-92ba0fc0dc3d']
# collection2 used in ids: EmptyCollection[AssetVersion]
To work with properly reflected data committing and re-querying should reflect the attribute changes.
When accessing an attribute that holds a KeyValueMappedCollectionProxy
type currently the value of the metadata
is being returned.
The actual attribute value should be returned and not necessarily metadata.
from trackteroid import *
print(Query(Sequence).get_all(projections=["custom_attributes", "metadata"]).custom_attributes)
Accessing the actual data via wrapped ftrack entities.
This code resolves the projections incorrectly and fails:
from trackteroid import *
empty_collection = Query(Asset).by_name("ThisDoesntExist123").get_first()
entity_collection = Query(Asset).get_first(projections=[ComponentLocation.resource_identifier])
for collection in [empty_collection, entity_collection]:
result = collection.ComponentLocation.resource_identifier
if result:
print("Result: ", result)
else:
print("NO Result", result)
result = collection.versions.get(ComponentLocation).resource_identifier
if result:
print("Result: ", result)
else:
print("NO Result", result)
This code shouldn't fail and correctly resolve the underlying relationship.
Provide any relevant background information or context that may help in understanding the issue.
The problem is that the underlying _get_relatives
function still refers to the original collection entity type which is Asset
instead of referring to AssetVersion
retrieved via the versions
attribute.
EntityCollection.map(predicate)
produces a generator and not a list anymore.
EntityCollection.map(predicate)
should produce a list and not a generator object, as this was the intended behavior and correlates to the current behavior of the group_and_map
method.
Provide any relevant background information or context that may help in understanding the issue.
collection.map(lambda _: True)
.Add tests.
The proprietary predecessor of Trackteroid had good test coverage. We have to identify if those tests can be ported and potentially rework them, but obviously, we need tests.
We want to consider going fully with pytest instead of unittest
.
Trackteroid should validate if the returned value of the RELATIONSHIPS_RESOLVER
follows the expected schema.
This should cover
The idea is to make the configuration process of the API less error-prone and easier to set up.
When parsing relationships from the schema it is not always considering the shortest path.
Instead of going through a redirection of fields resulting in the same collection it should consider the shortest possible path. Additional logic might be required to decide on one relationship path incase the pathlength is identical for multiple possible relationships.
Provide any relevant background information or context that may help in understanding the issue.
This should reveal the problem:
from trackteroid import *
Query(Shot).get_first()
print(Shot.relationship[Note])
print(Query(TypedContext).by_name(ObjectType, "Shot"))
print(Query(TypedContext).by_name(Project, "name"))
A different relationship can be provided within the configurable RELATIONSHIP_RESOLVER
value that would override the relationship inferred from the schema.
two of the tests are failing, namely
test_session.py -> test_get_cached_collections
test_authoring.py -> test_create_note
As a user, I want to be able to directly import the SCHEMA
object from Trackteroid as I can do with Query
and SESSION
.
Example:
from trackteroid import (
AssetVersion,
Query,
Sequence,
SCHEMA
)
Query(AssetVersion, schema=SCHEMA.custom).by_name(Sequence, "foobar").get_all()
This ensures consistency with the imports of all other relevant classes and objects.
Setting an attribute like this fails if the attribute that needs to be updated produced an EmptyCollection
:
from trackteroid import *
collection1 = Query(AssetVersion).get_all(limit=1)
collection2 = Query(AssetVersion).get_all(limit=1, offset=1)
collection1.uses_versions = collection2
print(
f"collection1 id: {collection1.id[0]}\n"
f"collection2 id: {collection2.id[0]}\n"
f"collection1 uses ids: {collection1.uses_versions}\n"
f"collection2 used in ids: {collection2.used_in_versions}\n"
)
This example should work.
Provide any relevant background information or context that may help in understanding the issue.
See the example which leads to the following error:
... line 577, in __setattr__
key, collection = attribute_value._source
ValueError: not enough values to unpack (expected 2, got 1)
Setup the "Publish Python Package" Github action.
When doing a new release via Github we should automatically publish it as a Python package to PyPI.
When retrieving collections using children
, parent
, descendants
, and ancestors
, the resulting collection should be a TypedContext
collection, and the type coercion should be performed automatically.
Currently, the type of the collection is retrieved by inspecting the first element in the collection. As with the mentioned attributes, there s no guarantee it will produce a collection of the same type we have to coerce it to its actual parent type which is TypedContext
.
This is breaking change and requires filtering all the time.
fix the vulnerability detected in requirements.txt
https://github.com/TrixterFilm/trackteroid/security/dependabot
ย
Setting attributes won't work if the attribute is a relationship shortcut. This results in a KeyError.
This code will work and the task will be set accordingly
from trackteroid import (
AssetVersion,
Task,
Query
)
av_collection = Query(AssetVersion).get_first(projections=[Task])
task_collection = Query(Task).get_first()
av_collection.Task = task_collection
Provide any relevant background information or context that may help in understanding the issue.
See expected behavior.
Use the full relationship attribute access.
Accessing the children
attribute is producing a query to update children.object_type.name
. This is a leftover from the past and should be removed.
Accessing children
should not perform queries as this might be redundant and brings performance costs.
Provide any relevant background information or context that may help in understanding the issue.
According to the docs, two files have to be created - trackteroid_user_config.py
and trackteroid_relationships.json
, but it does not state where to put them and where to import/call them. The RELATIONSHIPS_RESOLVER
is being called in two places - trackteroid.query.__init__
and trackteroid.query.query`, which is confusing - one of them should be removed.
With a bit of digging, all this information can be concluded from the code but that is time consuming and an inconvenience for the user.
To avoid the user having to find where to place the user files and how to hook them in without diving into the code, the user files could be included in an empty form / containing the default RELATIONSHIPS_RESOLVER
from configuration.py (where it can be removed). The RELATIONSHIPS_RESOLVER
is then created in one place only and is called only once. The docs can then point to these files directly and all the user has to do is copy/paste the example and all is working.
Certain operations should be prevented between collections if those are using different session objects and/or schemas.
As each session maintains its own cache we need to prevent the creation of collections that are based on different sessions. The same is true for different user-provided schemas, as these are driving the attribute resolution.
The following methods need to consider a check:
__setattr__
(also used by apply
)union
difference
symmetric_difference
intersection
Sourcing a user configuration doesn't work at the moment as importlib.load_source
doesn't exist anymore in Python >=3.7
As user configuration should be sourced and the provided overrides be applied.
Provide any relevant background information or context that may help in understanding the issue.
Provide a user configuration file via TRACKTEROID_CONFIGURATION
envvar and import trackteroid.
No workaround.
Add a apply(predicate, attribute_name=None)
method on EntityCollection
. This method should be responsible for assigning a generated value to a given attribute on all collection items or the collection items of the caller directly.
Single-entity collections can not be assigned to a target collection that contains multiple entities. The same is true for primitive data types. This leads to the situation that if you want to assign a single value or single entity collection to multiple receivers you'd have to loop as we don't want to change this explicit concept.
Nevertheless, the current expected user approach:
for single_collection in multiple_collection:
single_collection.some_attr = single_collection.another_attr[0] + "_edited"
could be streamlined towards:
multiple_collection.apply(lambda c: c.another_attr[0] + "_edited", "some_attr")
This new method should also be able to put a generated value directly onto the caller collection when no attribute is provided.
assetversion_collection.Task.Status.apply(lambda avc: status_collection)
While linking properly creates Link objects and committing to the database works fine the changes aren't reflected on the local attributes incoming_links
, and outgoing_links
.
Extending and removing links should be reflected in the beforementioned attributes on the collection.
Provide any relevant background information or context that may help in understanding the issue.
from trackteroid import (
Query,
AssetBuild,
SESSION
)
assetbuild_collection = Query(AssetBuild).get_all(
limit=3,
projections=[
"name",
"incoming_links.from",
"incoming_links.from.name",
"outgoing_links.to",
"outgoing_links.to.name"
]
)
print(f"Link to add: {assetbuild_collection[0].name}")
print(f"Currently linked: {getattr(assetbuild_collection[1].incoming_links, 'from').name}")
print(f"Local attribute holds: {getattr(assetbuild_collection[1].link_inputs(assetbuild_collection[0]).incoming_links, 'from').name}")
Retrieving the newly created link objects can be done by committing the changes first and querying again.
In order to use projections like this Query(AssetVersion).get_first(projections=[SequenceComponent.name])
we need to have them implemented as actual entity subclass.
Currently, there is a difference between a ForwardDeclare
and an Entity
subclass as they don't provide the same features. Similar to what we are doing for TypedContext
subclasses already we should automatically create Entity
subclasses for all available declarations automatically.
An alternative approach to consider would be to introduce a factory system to provide custom types or to extend existing types. But this doesn't exist currently.
There are no obvious risks.
Initialize a session when actually needed and not on module import.
Currently, the SESSION
singleton will be created when importing trackteroid
which is not ideal as it results in an ImportError
if a connection can not be established. Instead, the connection attempt should be made when actually needed.
As our Session
class already acts as a delegate for the Ftrack Python API Session we might be able to track and establish a connection on any self._session
attribute access.
Make default projections configurable.
Currently, entities provide a list of default projections, hardcoded for each implemented entity. This is not necessarily reflecting a user's requirements or expectations, so it should be configurable.
It could be considered to use Ftrack's factory mechanism to override default projections and not provide any additional default projections within Trackteroid.
If the API is in use and default projections would change for codebases that already expect and rely on certain of them this might be a breaking change and code can break.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.