Disclaimer: This is not an officially supported Google Product.
B-Con (short for Billing Reconciliation) is a solution that makes it easier to reconcile invoices for Google Ad Products. This is useful for ad agencies as well as any team that has to deal with a large number of invoices every month from Google which may take a lot of time to reconcile with both the numbers reported in the product UI as well as the company's own internal purchase order/media order system.
Assume you are an Ad Agency and on a monthly basis, your finance team receives 1000 invoices. The finance team needs to verify the amounts in each of these invoices and reconcile them before they can request payment from your clients and pay Google on their behalf. The steps involved typically are as follows:
- Receive invoices in PDF format via email.
- Copy the data into an excel sheet for easier calculation.
- Reach out to Adops to generate reports from the Product UI for the invoice period and compare them against the invoice.
- Reach out again for discrepancies and any clarifications or re-running the report.
- Compare against the purchase order for the specific client campaign and see how much has been delivered already.
- Create an invoice for the client and send it to them.
- Make the payment to Google against the invoice.
- Repeat this every month.
Steps 1-5 above take the longest time and it is not unusual for teams of individuals spending 2-3 weeks reconciling the numbers. B-Con helps by automating the most time consuming steps.
-
B-Con currently supports the following Google Ad Products. To support additional products, you B-Con can be modified by adding more API handlers.
-
Display & Video 360 (DV360)
-
Campaign Manager (CM)
-
-
Data is uploaded to BigQuery so it can be made available for Data Studio or other reporting dashboards.
-
Data access and permissions on dashboard are governed by actual permissions on CM and DV360. i.e. users can only see what they have access to in the product UI.
-
Configurable admin users who have access to all the data irrespective of UI permissions (required for Finance team)
-
Config sheet in Google Sheets which makes it easier to manage
-
Email addresses are one-way hashed with SHA256 so PII is not stored in database.
-
Multi-region and Multi-market support to keep the data and access separate.
Prerequisites for B-Con are:
- Access to Google Payments - The finance team (or whoever will use B-Con) will need to have read access to Google Payments. This allows them to bulk download all the invoices in CSV format.
- Internal billing code in campaign name/insertion order name - To reconcile each invoice entry accurately with the internal purchase order/media order system, the internal billing code needs to be included in the campaign name (for CM and DV360) or insertion orders (DV360 only)
- The Adops team includes the internal purchase order number in the advertiser name, campaign name or insertion order name. This has to be done when the campaign/insertion order is created. The reason for doing this is that invoices have one or more of advertiser name, campaign name or insertion order name listed in them and this is used to match the amount to the internal billing system.
- When the invoices are available, the finance team downloads them from Payments Center and uploads it to Sharepoint/Drive that B-Con is configured for.
- B-Con performs the following steps:
- Process invoices
- Runs reports using product APIs
- Uploads all the data to BigQuery which feeds into Data Studio.
- Google Cloud Project - B-Con uses the following features of Google Cloud:
- Compute Engine - To run the backend load jobs
- BigQuery - Where the data is eventually stored in.
- Service Accounts - B-Con uses service accounts to access the API.
- Data Studio - Reporting dashboard to show the data from BigQuery
-
Create a Google Cloud Project.
-
Enable billing or link to an existing billing account. This should be already taken care of if you are using your company's google cloud account.
-
Create a new VM instance with the following configuration:
- Machine type: n1-standard-1 (1vCPU, 3.75 GB memory)
- Under
Identify and API access
select the optionAllow full access to all Cloud APIs
. This is needed to download the service account key (explained later). Once the key is downloaded, you can modify this toAllow default access
.
Leave everything else as default and wait for the instance to get ready.
-
Open an SSH window by clicking on the ssh button.
-
Run update:
sudo apt update && sleep 2 && sudo apt upgrade -y
-
Check that the python version is at least 3.8. If it is 3.8 or later, then you can skip the next step.
python3 --version
-
If you don't have Python 3.8 or higher, you will need to compile Python from source. Follow the steps below to do this:
-
Install the necessary tools and dependencies for Python:
sudo apt install libsqlite3-dev libbz2-dev liblzma-dev uuid-dev libssl-dev zlib1g-dev libncurses5-dev tk-dev libgdbm-dev libreadline-dev libffi-dev git -y
-
Download and compile python (
3.8.5
is latest at the time of writing)mkdir ~/python_versions cd ~/python_versions wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz tar -zxf Python-3.8.5.tgz cd Python-3.8.5 ./configure --enable-loadable-sqlite-extensions --enable-optimizations --with-ensurepip=install make
Wait for the compilation to complete.
-
While still in the same folder, add it to the path and make sure it's accessible when needed:
ln -s python python3 echo "export PATH=\"`pwd`:\${PATH}\"" >> ~/.bash_aliases source ~/.bash_aliases
-
Check that the latest version is now at least 3.8.
python --version && python3 --version
-
Go back to the home folder:
cd
-
-
Setup the code:
-
Clone the b-con repository:
git clone https://github.com/google/b-con.git
This creates the folder
b-con
. -
Create a python virtual environment:
cd b-con python3 -m venv v
-
Activate the virtual environment:
source v/bin/activate
-
Install the python requirements:
pip install wheel pip install -r requirements.txt
-
-
Enable APIs on the Cloud account - These are all the APIs that B-Con will be accessing.
- Go to APIs and Services in Google Cloud and click on "Enable APIs and Services"
- Search for and enable the following APIs:
- DoubleClick Bid Manager API
- Display & Video 360 API
- DCM/DFA Reporting And Trafficking API
- Google Sheets API
- Google Drive API
- BigQuery API
-
Create a service account which will be used to access all the APIs:
-
Go to
APIs & Services > Credentials
in Google Cloud. -
Select
Create Credentials
and selectService Account
. -
Call it
bcon-robot
. And then clickCreate
. -
In permissions give it the following roles:
- BigQuery Data Editor
- BigQuery Job User
Hit
Continue
. -
Click
Done
. -
Copy the email address of the service account which should look like this:
bcon-robot@<project-id>.iam.gserviceaccount.com
. This will be used in the next step.
-
-
Back in compute engine ssh window, create a new key for the service account you created and download the file to VM:
cd ~/b-con gcloud iam service-accounts keys create bcon_robot.json --iam-account bcon-robot@<project-id>.iam.gserviceaccount.com
This will download the private keys into the VM.
The private key in the json file will allow anyone who possesses the file to access anything that service account has access to, even if the users themselves don't have access to (Ex: Bigquery, DV360, CM etc). So take precaution to give access to only the relevant people to the VM and the private key file.
-
Grant access permissions to the service account to DV360 and CM
- DV360 - Create a new user for the service account email. It needs to be given "Reporting Only" access to all the partners that you want to use it with. Refer to this page for more details.
- CM - Create a new user profile for the service account. It needs to be given "Advanced Agency Admin" access to all the campaign manager accounts. Refer to this page for more details.
-
BigQuery setup - Open the BigQuery page and do the following:
- Create a new dataset called
bcon
within the project. - In the dataset, create empty tables (no columns) with the following names:
- dash_ui
- dash_invoice_report
- reports
- invoices
- invoice_entries
- admin_users
- user_perms
- cm_dash_ui
- cm_dash_invoice_report
- cm_reports
- cm_user_perms
- cm_advertiser_accounts
- cm_user_perms_advertiser
- Create a new dataset called
-
Configuration - The following steps will allow you to set up the configuration.
-
Google Sheets:
- Make a copy of this sheet.
- Update either the Sharepoint tab or the Drive tab depending on which file storage system you are using.
- Add the DV360 partners to the Partners tab.
- Add the admin users to the Admin users tab. Note that admin users are the same for DV360 and for CM.
- Grant read access to the email address of the service account created earlier.
-
Code config:
-
Back in the ssh window, make a copy of
config_template.yaml
and call itconfig.yaml
:cp config_template.yaml config.yaml
-
Update the details as per the documentation in the file. Note that you only need either sharepoint or drive and not both.
-
If you are using b-con for multiple markets and wish to keep them separate, you can include more than one config sheet in
config.yaml
.
-
-
-
Running B-Con - All the scripts needed for running B-Con are in the scripts folder. You need to be in the root folder of B-Con to run them as follows (for example):
cd ~/b-con scripts/run_dv360.sh
-
Business Logic - The python code only fetches the data and uploads it to BigQuery. All the business logic is in the sql scripts. This is visible in the folder
sql
. To run the sql scripts, you can runscripts/run_sql_load.sh
. -
Cron setup - You can set up cron jobs for all the scripts so that they run regularly. Note that since we're using a virtual environment for python, you will need to give the commands like the example below. This runs the specific script
run_drive.sh
and then runsrun_sql_load.sh
.# Run every 60 minutes. */60 * * * * cd /home/<user>/b-con;source v/bin/activate;scripts/run_drive.sh > /tmp/run_drive.log 2>&1;scripts/run_sql_load.sh > /tmp/run_sql_load.log 2>&1;deactivate
There are many resources available online on setting up cron on a linux machine, so we will not go further into this here.
-
User permissions scripts - B-Con syncs the user access permissions from DV360 and Campaign Manager to ensure that data is visible only to the relevant users. This is done by the scripts
scripts/run_user_perms.sh
andscripts/run_cm_user_perms.sh
. Care should be taken that these scripts are set up in cron to run regularly so that the access permissions are propagated. -
User permissions views
views/v_admin_users.sql
is a convenience view for admin users which can be set up on BigQuery and used in data studio.views/v_user_perms.sql
is a convenience view for DV360 users which can also be set up in BigQuery. It lists the advertiser id that each user has access to.
Both these views can be set up and used to filter data for users as shown below.
-
Data Studio setup - It is recommended to use Google Data Studio for accessing and sharing the information since it integrates well with BigQuery. More info available here.
-
Data Studio user permissions setup - Data studio allows you to set up the dashboard in a way that only the users who have access to a particular advertiser in CM or DV360 will see the data. B-Con makes the user permissions available at the advertiser level in the tables
cm_user_perms_advertiser
anduser_perms
. The email address in these tables is hashed with SHA256 and then encoded to Base64. This can be used to manage per user access to the data in Data studio as follows. This requires the@DS_USER_EMAIL
parameter enabled in data studio as explained here.Here is an example script:
SELECT DISTINCT d.* FROM `<project-id>.bcon.dash_invoice_report` AS d LEFT JOIN `<project-id>.bcon.users` AS u ON CAST(d.advertiser_id AS INT64) = u.advertiser_id WHERE TO_BASE64(SHA256(LOWER(@DS_USER_EMAIL))) = u.user OR TO_BASE64(SHA256(LOWER(@DS_USER_EMAIL))) IN ( SELECT user FROM `<project-id>.bcon.v_admin_users` ) -- Admin