VQPy is an object-oriented language, designed to address two major issues in video analytics: (1) it is hard to express complex query logic with a pure declarative language (like SQL) and perform actions (e.g., send a message) when certain events are detected (e.g., a senior person falls) and (2) it is hard to ``grow'' queries in similar ways to how an object-oriented language (such as Java) allows its programs to build on top of each other (check out Guy Steel's OOPSLA'98 keynote on growing a language).
To solve these problems, VQPy embraces the four pillars of object-orientation: Inheritance, Polymorphism, Encapsulation, and Abstraction, with syntax designed specifically for analyzing video frames. VQPy allows a complex query to be expressed with a very small number of lines of code. VQPy supports query sharing and composition---finding a red car can build on an existing query that finds a general car; monitoring traffic for a city can build on car monitoring queries built for individual districts and intersections, thereby significantly simplifying development and deployment. Please check out our examples below for details.
VQPy is still under active development. VQPy compiler, which generates a query plan with optimized performance for users' video analytics applications, is working in progress. With VQPy compiler, users can simply focus on the declaration of video queries for their own applications, and multiple optimizations defined in the compiler will be transparently applied to the user’s video analytics pipeline.
The development of VQPy was initiated by Harry Xu's group at UCLA, and has evovled over the time into a community effort, involving folks from both academia and industry.
Show installation details
VQPy is developed and tested on Linux. Therefore we highly recomand you to try VQPy on Linux, to avoid encountering some unknown errors.
You can follow the steps below to install VQPy.
We recommend using conda to prepare the Python environment as follows:
conda create -n vqpy python=3.8 # "vqpy" is conda environment name, you can use any name you like.
conda activate vqpyWe haven't publish vqpy to pypi yet. You can use the commands below to install VQPy from Github.
pip install torch torchvision numpy==1.23.5 cython
pip install 'vqpy @ git+https://github.com/vqpy/vqpy.git'You can test whether vqpy has been successfully installed with
import vqpy
from vqpy import queryBelow is an architecture graph which describes the VQPy framework.
In the frontend, we provide interfaces for users to declare their query, i.e VObj and Query.
In the backend, vqpy automatically select the best plan to execute the query, within user specified budget (e.g. accuracy, inference time, .etc).
VQPy also provides a library containing rich models and property functions that help save users' efforts to build queries.
Note that users can also register their own models and functions to vqpy library, to enable reusability between different queries and customize their own end to end video query pipeline.
In order to declare a video query with VQPy, users need to extend two classes defined in VQPy, namely Query and VObj. VObj defines the objects of interest (e.g., cars, humans, animals, etc.) in one or more video streams, and Query defines the video query.
Users can define their own objects of interest, as well as the property in the objects they hope to query on, with a VObj class.
To define a VObj class, users are required to inherit the vqpy.VObjBase class. User can also define properties in the VObj that can be used in query (like "license plate" of a "Vehicle"). To define a property, the definition should start either with a @vqpy.property decorator or a @vqpy.cross_object_property decorator, where @vqpy.property indicates that the calculation of the property only based on the attributes of the VObj instance, and @vqpy.cross_vobj_property indicates that the calculation of the property requires the arributes of other Vobj instances on the same frame.
For example, if we are interested in the vehicle object in the video, and want to query the license plate. We can define a Vehicle class as below.
class Vehicle(vqpy.VObjBase):
@vqpy.property()
def license_plate(self):
# infer license plate with vqpy built-in openalpr model
return self.infer('license_plate', {'license_plate': 'openalpr'})And if we want to query the owner of a baggage object where the baggage's owner is a person object who is closest to the baggage. We can define our interested VObjs as below. Note that the owner property of Baggage VObj should be decorcated with @vqpy.cross_vobj_property.
class Person(vqpy.VObjBase):
pass
class Baggage(vqpy.VObjBase):
@vqpy.cross_vobj_property(vobj_type=Person, vobj_num="ALL", vobj_input_fields=("track_id", "tlbr"))
def owner(self, person_id_tlbrs):
passTo retrieve the attribute values in property calculation, you can use self.getv("property_name", index={history_index}) or self.infer. For more information about how to retrieve attribute values in VObj, you can refer to document here.
You can find more details about VObj as well as the decorators in our VObj API document and VObj decorator API document (Currently is the docstring in our source code).
Users can express their queries through SQL-like constraints with VObjConstraint, which is a return value of the setting method in their Query class. In VObjConstraint, users can specify query constraints on the interested VObj with filter_cons, and select_cons gives the projection of the properties the query shall return.
Note that the keys for both filter_cons and select_cons should be strings of property names, where the property name can either be a vqpy built-in property name or a user declared property name of the interested VObj. The value of filter_cons dictionary should be a boolean function, the VObj instances which satisfy all the boolean functions in filter_cons will be selected as query results. The value of select_cons dictionary is the postprocessing function applied on the property for query output. If you want to directly output the property value without any postprocessing functions, you can just set the value to None.
The code below demonstrates a query that selects all the Vehicle objects whose velocity is greater than 0.1, and chooses the two properties of track_id and license_plate as outputs.
class ListMovingVehicle(vqpy.QueryBase):
@staticmethod
def setting() -> vqpy.VObjConstraint:
filter_cons = {'__class__': lambda x: x == Vehicle,
'velocity': lambda x: x >= 0.1}
select_cons = {'track_id': None,
'license_plate': None}
return vqpy.VObjConstraint(filter_cons=filter_cons,
select_cons=select_cons)We also provide boolean functions that can be used in query, like vqpy.query.continuing, which checks whether a condition function continues to be true for a certain duration. For the detailed usage, please refer to our People Loitering example.
After declaring VObj and Query, users should call the vqpy.launch function to deploy the video query, below is an example of this interface. Users should specify the detection class they expect, and map the object detection result to the defined VObj types.
vqpy.launch(cls_name=vqpy.COCO_CLASSES, # detection class
cls_type={"car": Vehicle, "truck": Vehicle}, # mappings from detection class to VObj
tasks=[ListMovingVehicle()], # a list of Queries to apply
video_path=args.path, # the path of the queried video
save_folder=args.save_folder # result of query will be saved as a json file in this folder
)Under the hood, VQPy will automatically select an object detection model that outputs the specified cls_name, and decide the best predicates order to execute the query. Multiple video optimizations will be conducted transparently to improve the end-to-end video query performance.
We also support customizing components (e.g. detector) to build the end to end query pipeline. For more details on customization, please refer to the document here.
We have included several real-world video analytics examples for demonstrating VQPy.
- Fall Detection: Recognize fallen people in a video.
- List red moving vehicle: show the license plates of red moving vehicles.
- People Loitering: detect suspicious activities of person loitering around.
- People Counting: count the number of people heading different directions.
- Unattended Baggage Detection: detect unattended baggages.
- Use VQPy's slack channel to ask questions and share ideas!
- Create a github issue.
We are grateful to the generous support from:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent