Comments (6)
Succeeded in 37 minutes!
from bigquery-etl.
I'm currently testing a process where we first produce a list of document_ids with the number of occurrences, then we select all the records where occurrences = 1
into the stable table, then we do the window query on just the rows where occurrences > 1
. The vast majority of rows have no duplicates in the live table, so this may make the window function tenable. It may also be possible to express this entire operation as a single query without the need for a temp table.
First attempt failed, and now I'm going to try breaking into pieces.
from bigquery-etl.
On the bright side, the other copy_deduplicate job (which handles populating all the stable tables besides main) finished in just under 10 minutes, so that looks to be working well.
from bigquery-etl.
I was able to successfully create a deduped version of main_v4
via a query broken into three parts; the process took just over 30 minutes of runtime. I've now unified those parts together into a single query and am waiting to see if that completes:
CREATE TABLE
tmp.klukas_main_deduped2
PARTITION BY
DATE(submission_timestamp)
CLUSTER BY
sample_id AS
WITH
base AS (
SELECT
*
FROM
`moz-fx-data-shared-prod.telemetry_live.main_v4`
WHERE
DATE(submission_timestamp) = '2019-08-22' ),
--
duped_docids AS (
SELECT
document_id,
COUNT(document_id) AS occurrences
FROM
base
GROUP BY
document_id
HAVING
occurrences > 1),
--
nonduped AS (
SELECT
base.*
FROM
base
LEFT JOIN
duped_docids
USING
(document_id)
WHERE
duped_docids.document_id IS NULL),
--
numbered_duplicates AS (
SELECT
base.*,
ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) AS _n
FROM
base
JOIN
duped_docids
USING
(document_id) ),
--
deduped AS (
SELECT
* EXCEPT (_n)
FROM
numbered_duplicates
WHERE
_n = 1 )
SELECT
*
FROM
nonduped
UNION ALL
SELECT
*
FROM
deduped
from bigquery-etl.
The above single query succeeded in 42 minutes. I'm going to PR this change to bigquery-etl.
from bigquery-etl.
The new docker image has been built and published, so I kicked off the airflow job to run again. We should see that succeed in ~40 minutes.
from bigquery-etl.
Related Issues (20)
- Add custom ping into Shredder HOT 4
- Change broken_site_report view to only use historical tables
- Add a separate view for live user_reports
- Improve dryrun skip logic for stage deploys
- Encode Derived Funnel Dimensions into UDF HOT 3
- Adding the ability to override scheduling parameters on backfill
- Schema updates for tables with not associated query.sql HOT 3
- Support publishing only authorized views
- Use schema.yaml for static tables instead of schema.json HOT 1
- [Data Checks Bug]: Apple CPUs are represented as empty string HOT 1
- Python 3.12 bootstrap - ModuleNotFoundError: No module named 'distutils'
- Create new derived query for Firefox-CI worker costs
- Support secrets in DAG generation
- event_monitoring_live generation is skipping apps that only have events defined in dependencies HOT 2
- materialized views don't get updated after deploy HOT 2
- Recreating materialized views may cause occasional dry run failures on main HOT 1
- Union default_table_workgroup_access with table-level workgroup_access in metadata generation logic
- `event` column from `events_stream` inaccessible in Looker HOT 2
- CI may be running stale tests in some cases HOT 1
- Monitor plus Attribution: Update ETL to store the channel for relevant Bigquery Tables and Looker explore HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigquery-etl.