Datastudio dashboard is not showing my data about fourkeys

[dinagraves]

I have a fix for the setup script, but if you don't want to run the whole script over again, you can just do the part of scheduling your queries. CD into the queries directory and run:

  export FOURKEYS_PROJECT={your_project_id}
  token=$(gcloud auth print-access-token)

  python3 schedule.py --query_file=changes.sql --table=changes --access_token=${token}
  python3 schedule.py --query_file=deployments.sql --table=deployments  --access_token=${token}
  python3 schedule.py --query_file=incidents.sql --table=incidents --access_token=${token}

[adilnaimi]

I'm having exactly the same issue.

[dinagraves]

Thanks! I'm working on replicating this error.

[dinagraves]

Hi Jo,

Thank you for reporting this issue! Could you share what the connection screen looks like? It sounds like you're seeing the public data in the public template. Did your connection flow look like this? https://youtu.be/VLVqRVX6Cg4?t=76

I'll be adding the code for the connector to the repo shortly, so you can see how it's manipulating the data.

-Dina

[jgrumboe]

Hi Dina,

Thanks for your quick response. This is our connection detail.
I compared to your video and it seems to be quite identical. What I haven't done is clicking the "create report" button.

Looking forward to inspecting the connector to know what's happening behind the curtain 😉
Jo

[jgrumboe]

Now I clicked the "Create Report" button and get a copy of your template.

But the counts are low, although I created mock data and as you see one graph is even throwing an error. 😢

[dinagraves]

Found the issue in the setup:

details = "BigQuery Data Transfer service account is not found, this could be fixed by re-enabling BigQuery Data Transfer service API for project fourkeys-016361. Please ask the project owner to re-enable the API at https://console.cloud.google.com/apis/library/bigquerydatatransfer.googleapis.com."

The dashboard is empty b/c the BQ transfers are not successful. Adding in a check to the setup script to make sure the API is enabled properly.

[dinagraves]

One more note: You can check the state of your data by going into the bigquery console. I would double check that there is data in the events_raw table. This is where the mock data will have gone and it's what the scheduled queries read off of. If you need to generate data, follow the instructions here: https://github.com/GoogleCloudPlatform/fourkeys#generating-mock-data

[dinagraves]

Ah found one more issue! The data generator wasn't updated to take into account this definition change: #34.

Updating and adding tests!

[jgrumboe]

Hi @dinagraves
I took your PR and rerun the setup script.
I also stumbled upon the BQ Datatransfer API thing, so this is already enabled.
But I get an error while mocking data.

...
Would you like to create a separate new project to test deployments for the four key metrics? (y/n):n
Would you like to generate mock data? (y/n):y
Updated property [core/project].
Creating mock data...
+ python3 /Users/jgrumboeck/git/fourkeys/setup/../data_generator/github_data.py
Traceback (most recent call last):
  File "/Users/jgrumboeck/git/fourkeys/setup/../data_generator/github_data.py", line 128, in <module>
    num_success += generate_data()
  File "/Users/jgrumboeck/git/fourkeys/setup/../data_generator/github_data.py", line 108, in generate_data
    num_success += send_mock_github_events("push", curr_change)
  File "/Users/jgrumboeck/git/fourkeys/setup/../data_generator/github_data.py", line 79, in send_mock_github_events
    response = urlopen(request)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Users/jgrumboeck/.pyenv/versions/3.7.9/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error
+ set +x
Check BigQueryDataTransfer is enabled
WARNING: You are using pip version 20.1.1; however, version 20.3 is available.
You should consider upgrading via the '/Users/jgrumboeck/.pyenv/versions/3.7.9/bin/python3.7 -m pip install --upgrade pip' command.
Creating BigQuery scheduled queries for derived tables..
+ python3 schedule.py --query_file=changes.sql --table=changes --access_token=ya29.a0AfH6SMDA6r-14KqNkEVmbQOjQLocNu_4aOSJQlINTKwR0vWsZ1UvaxMR5OQxHKQQFMLOORWL4EqocDzhR7A0wkxTNXJQhhOCR-lddBYk72haQ-Xl3UEwQmBZ8aVQUCRk3AOV4hASMdB3AOIfffWcT9Y34OiUY4J_21f258Vzz2JqiGEaUgw
Updated scheduled query 'projects/266035869840/locations/us/transferConfigs/5ff4be8a-0000-22eb-88ea-3c286d431a7e'
+ python3 schedule.py --query_file=deployments.sql --table=deployments --access_token=ya29.a0AfH6SMDA6r-14KqNkEVmbQOjQLocNu_4aOSJQlINTKwR0vWsZ1UvaxMR5OQxHKQQFMLOORWL4EqocDzhR7A0wkxTNXJQhhOCR-lddBYk72haQ-Xl3UEwQmBZ8aVQUCRk3AOV4hASMdB3AOIfffWcT9Y34OiUY4J_21f258Vzz2JqiGEaUgw
Updated scheduled query 'projects/266035869840/locations/us/transferConfigs/5fe33f69-0000-2acd-875d-f403043816d8'
+ python3 schedule.py --query_file=incidents.sql --table=incidents --access_token=ya29.a0AfH6SMDA6r-14KqNkEVmbQOjQLocNu_4aOSJQlINTKwR0vWsZ1UvaxMR5OQxHKQQFMLOORWL4EqocDzhR7A0wkxTNXJQhhOCR-lddBYk72haQ-Xl3UEwQmBZ8aVQUCRk3AOV4hASMdB3AOIfffWcT9Y34OiUY4J_21f258Vzz2JqiGEaUgw
Updated scheduled query 'projects/266035869840/locations/us/transferConfigs/5fee3798-0000-2151-b9eb-f403043ab0ac'
+ set +x

Waiting for BigQuery jobs to complete...
BigQuery jobs done!
...

Hmmmm ... mocking data did run successful last time 😕

Within the event-handler logs I found this error message:

Exception: Unverified Signature
at index (/app/event_handler.py:50)
at dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1935)
at full_dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1949)
at reraise (/usr/local/lib/python3.7/site-packages/flask/_compat.py:39)
at handle_user_exception (/usr/local/lib/python3.7/site-packages/flask/app.py:1820)
at full_dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1951)
at wsgi_app (/usr/local/lib/python3.7/site-packages/flask/app.py:2446)

Found that one and created a new issues #40

[jgrumboe]

I also check BQ console and I do have mock data in events_raw:

And I also checked the derived tables.
Tables changes looks fine.

Table incidents looks fine.

However deployments is empty .. maybe because I generated with a script older than #39.

Dashboard still looks like in my above comment (#37 (comment)) 😢

[jgrumboe]

Latest update:

After rerunning mockdata with the correct event-handler secret and the updated version from PR #39 I scheduled a backfill of all the scheduled queries.
Now I have data in the deployments table and the dashboard looks fine - as far as I can interpret this 😀

So I really think that #39 closes this issues.
Thanks very much!