paretogroup / omni-reports

OmniReports is a client to request, normalize and consolidate reports from digital marketing platforms like Google Ads and Facebook Ads

Omni Reports

Omni Reports is a client to request, normalize and consolidate reports from several digital marketing platforms using a simple, declarative and concise request structure. Behind the scenes, the Omni Report Client will convert the report definition into platform-specific report requests.

The Omni Report Definition is a json-like data structure based on Google Ads Report Definition, accepting several elements to query and segment a report.

Installation

To install Omni Reports, use pip:

pip install omni-reports

Usage

Create a ReportTypeResolver to resolve report types of each platform (like Google Ads). Then, create a ReportClient and execute your report definition.

from omni_reports.client import ReportClient, ReportTypeResolverBuilder
from omni_reports.google_reports import GoogleAdsReportTypeResolver

resolver = ReportTypeResolverBuilder() \
    .extend(GoogleAdsReportTypeResolver) \
    .build()

client = ReportClient(resolver)
result = client.execute_report({
    'report_type': 'GOOGLE_ADS_ACCOUNT_PERFORMANCE_REPORT',
    'report_name': 'my_report',
    'selector': {
        'fields': ['account_id', 'cost', 'conversions', 'cost_per_conversion'],
        'predicates': [
            {
                'field': 'cost',
                'operator': 'greater_than',
                'values': ['0']
            }
        ]
    }
})

print(result)

Additionally, you can pass a context to ReportClient so that a ReportTypeResolver can be configured.

from omni_reports.client import ReportClient
from omni_reports.google_reports import GoogleAdsReportTypeResolver

client = ReportClient(GoogleAdsReportTypeResolver, {
    'GOOGLE_ADS_CLIENT_ID': 'MY_CLIENT_ID',
    'GOOGLE_ADS_CLIENT_SECRET': 'MY_CLIENT_SECRET',
})

# or via `execute_report`
report_definition = {...}
client.execute_report(report_definition, {
    'GOOGLE_ADS_CUSTOMER_ID': 'MY_CUSTOMER_ID',
})

Creating a ReportType

Create a ReportType in Omni Reports is simple. Just create a new class extending ReportType, declare the fields of your report type and implement the resolve method. The ReportClient will validate all definitions of your report type and call the resolver.

To create your report type, you must declare all fields of report with its behaviors and metadata. There is three types of behaviors: attribute, metric and segment.

• Attribute: The attribute fields must always reflect the current state of your data, ignoring the timespan of the report;
• Metric: The metric fields reflect the data over the timespan of the report;
• Segment: The segment field contains dimension data that is used to group metrics. Including a metric field into your report definition may split a single row into multiple rows. The value of metric fields reflect the data over the timespan of the report.

After the identification of the behavior for each report type field, you can start declaring the report type class:

from omni_reports.client import ReportClient, ReportTypeResolverBuilder
from omni_reports.client.types import ReportType
from omni_reports.client.fields import AttributeReportField, MetricReportField, SegmentReportField


class MyAdReportType(ReportType):
    """
    Example of ReportType to query perfomance of all accounts in platform
    """

    account_id = SegmentReportField()
    campaign_id = SegmentReportField()
    ad_group_id = SegmentReportField()
    ad_id = SegmentReportField()
    
    cost = MetricReportField()
    conversions = MetricReportField()
    cost_per_conversion = MetricReportField()

    campaign_status = AttributeReportField()
    ad_group_status = AttributeReportField()
    ad_status = AttributeReportField()

    def resolve(self, fields, predicates, report_definition, context, client):
        # logic to apply predicates and filter fields
        return []

resolver = ReportTypeResolverBuilder() \
    .add_type("MY_AD_REPORT", MyAdReportType()) \
    .build()

client = ReportClient(resolver)
client.execute_report({
    'report_type': 'MY_AD_REPORT',
    'report_name': 'report_name_here',
    'selector': {
        'fields': ['ad_group_id', 'cost', 'conversions', 'cost_per_conversion'],
        'predicates': [
            {
                'field': 'campaign_id',
                'operator': 'equals',
                'values': ['1234']
            }
        ]
    }
})

Contributing

To contribute, see the CONTRIBUTING guides.

License

MIT

omni-reports's Issues

Improve the usage of asyncio event loop and client sessions

relates to #8

Rename google_reports to contrib.googleads and facebook_reports to contrib.facebookads

We have to move files relative to specific reports and extensions to contrib directory. We also have to use more specific names like googleads and facebookads to decrease the possibility of conflicts (ex. googleanalytics).

Poc - Teste Changelog

Marshmallow post_load returns UnmarshalResult

When Marshmallow Schema deserializes, the decorator @post_load returns a model with value. We use this model to access some attributes directly:

result = schema.loads()
print(result.attr)

Sometimes, when marshmallow lib version is incompatible with the project marshmallow version, the @post_load returns an UnmarshallResult object instead of the model value.

marshmallow-code/marshmallow#830

To fix this, we need to use result.data.attr instead of access directly the model attributes. This is a workaround, but the ideal scenario is @post_load always returns the Model instance.

Create an alias for report fields in report definition

The report definition does not provide a way to declare field aliases. It will help to normalize reports and to keep the conformance with platform reports documentation.

result = client.execute_report({
    'report_type': 'GOOGLE_ADS_ACCOUNT_PERFORMANCE_REPORT',
    'report_name': 'my_report',
    'selector': {
        'fields': ['account_id', 'cost', 'conversions', 'cost_per_conversion'],
        'field_aliases': {
            'cost': 'daily_cost'
        },
        'predicates': [
            {
                'field': 'cost',
                'operator': 'greater_than',
                'values': ['0']
            }
        ]
    }
})

Analisar o download e processamento dos relatórios que contém um volume grande de dados

Proposta:

• Enviar o CSV, ou arquivo baixado, para o S3, ou serviço similar;
• Usar uma solução como o Spark, ou serviço similar, para resolver o processamento dos arquivos.

Preview:

class ReportDownload:
	# Loads the report on memory and serializes to json.
	pass

class BigFileReportDownload:
	# Send the report to S3 and uses the Spark to filter and process the big report.
	pass

class GoogleAdsReportType:
	report_downloader: BigFileReportDownload

Add support for Facebook Report

Add async support for Google Reports

Implement tests for core lib and contrib reports

• Configure test suite
• Configure github workflow to run tests
• Implement tests for core lib (Achieve 80% or more in coverage)
• Implement tests for contrib reports (Achieve 80% or more in coverage)

Rename repository name to omni-reports in codebase

We need to rename in our codebase all references to the old name pareto_reports.

Places to verify:

• In setup.py
• In readme.md
• In import declarations

Missing Things

• Melhorar a passagem do event loop e do client session;
• Criação de testes;
• Facebook:
  • Aceitar inner fields no predicado;

Add currency type inside googlereports type

We need to add currency field inside googlereports/types.py.

currency = AttributeReportField(target_name="AccountCurrencyCode", display_name="Currency")