googlechromelabs / discovery

Discoveries on Sustainable Loading research

discovery

Discoveries learned exploring Sustainable Loading for the web.

What is the Sustainable Loading project?

Modern Web applications load code and data. Loading and processing (parsing, evaluating etc.) this data is not free, and negatively impacts the RAIL of the application. Further, as the apps become more complex, the code/data loading demands increase, and so does their impact. The impact becomes critical on mobile devices, leading to terrible user experience, unhappy users and developers.

Sustainable Loading is an initiative in Chrome to solve the problem of scalably loading complex applications with minimal impact on RAIL by giving developers the right tools, primitives, and narrative.

discovery's Issues

Data-set: Sites representative of popular JavaScript loading patterns

Requirements:

• Ordered by Alexa ranking
• Optionally use Webpack (check Webpack JSON signature)
• Optionally look at RequireJS usage. Is code bundled with it still being deployed?
• Optionally use Browserify (separate run)
• Have first-party JS where they're shipping more than 1 x JS file (separate run)

Who wanted this?
V8

Data-set: Highly ranked Progressive Web Apps

Why?

This may be useful to help understand whether the presence of a Service Worker helps improve code caching in V8 given the heuristics are far easier to trigger when SW is present and scripts are stored from Cache Storage.

Just need to generate the list of sites and V8 can take on benchmarking from there.

Production Web Apps Performance Study Q4/16 - Q1/17

Goals

• Understand the cost of JS Parse/Compile/FunctionCall times on apps
• Discover how quickly apps become interactive on average mobile hardware
• Learn how much JavaScript apps are shipping down on desktop + mobile

Sample information
6000+ production sites using one of React, Angular 1.0, Ember, Polymer, jQuery or Vue.js. Site URLs were obtained from a combination of Libscore.io, BigQuery, BuiltWith-style sites, framework wikis. Sample sets were 10% eye-balled to verify usage of frameworks. Sets not reliable were discarded from the final study.

URLs: https://docs.google.com/a/google.com/spreadsheets/d/1_gqtaEwjoJGbekgeEaYLbUyR4kcp5E7uZuMHYgLJjGY/edit?usp=sharing

Trivia: All in all, 85,000 WebPageTest results were generated as part of this study. Yipes.

Tools used in study
WebPageTest.org (with enhancements such a JS cost, TTI, aggregated V8 statistics added thanks to Pat Meenan as the project progressed), Catapult (internal Google tool), Chrome Tracing.

Summary observations

This data may be useful to developers as it shows:

• Real, production apps using their favorite stacks can be much more expensive on mobile than they might think.
• Closer attention to lower parse times and time-to-interactive points is likely required if you're choosing something off the shelf.
• Some, but not all, apps are shipping larger bundles. Where this is the case invest in code-splitting & reducing how much JS is used.

Where are the medians and aggregates?

The primary goals of this study were to highlight trends looking at the different data-sets available to me as a whole. Initially, I focused on summarizing this data at a per-framework level (e.g React apps in set 1 exhibited characteristic A). After reviewing this with the Chrome team, we decided presenting per-framework breakdowns were more susceptible to the takeaway being "oh, so I should just use framework X over Y because it is 2% better" instead of the important takeaways about parse/compile being a problem we all face.

To that end, the below charts are generated locally by fetching each of the WebPageTest reports for data-sets, iterating over a particular dimension (e.g Time-to-interactive, JS parse time) and getting the medians for different sets that are then plumbed into either Google Sheets or Numbers for charting. If you wish to recreate that setup yourself, you can grab the CSVs from the below reports.

Raw WebPageTest runs - Round 2 (January, 2017)

• Ember production sites, Moto G4, 3G: https://www.webpagetest.org/result/170122_XX_TQD/ with runtime call stats
• Ember production sites, desktop, cable: https://www.webpagetest.org/result/170122_RM_3GW/ with runtime call stats
• Sites from the React Wiki, Moto G4, 3G: https://www.webpagetest.org/result/161218_05_c8320bc70144d48df218f230701ec2b8/
• Sites from the React Wiki, Desktop, cable: https://www.webpagetest.org/result/170113_XJ_17TN/ (with runtime call stats)
• Sites from the React Wiki, Nexus 5: https://www.webpagetest.org/result/170116_Z6_9GS/
• Sites from the React Wiki, iPhone 6, cable: https://www.webpagetest.org/result/170116_MV_8VQ/
• Libscore React apps, Moto G4, 3G: https://www.webpagetest.org/result/161218_GD_297def2a6c8e5bcf355566577d4ed7b9/
• Libscore React apps, Desktop, cable: https://www.webpagetest.org/result/170114_TD_V4Z/
• React + Webpack usage survey sites, Moto G4, 3G: https://www.webpagetest.org/result/170115_HM_EYT/
• React + Webpack usage survey sites, Desktop, cable: https://www.webpagetest.org/result/170114_WN_V01/
• Subset of Libscore Angular 1.0 apps, G4, 3G: https://www.webpagetest.org/result/170115_SZ_Z5N/
• Subset of Libscore Angular 1.0 apps, desktop, cable, runtime call stats: https://www.webpagetest.org/result/170114_25_1DC/
• Libscore jQuery apps, G4, 3G: https://www.webpagetest.org/result/161218_A4_20acbeb009bd0a44527d3e4d93f1de2a/
• Vue.js production apps, G4, 3G: https://www.webpagetest.org/result/170115_HE_10EN/
• Vue.js production apps, desktop, cable: https://www.webpagetest.org/result/170115_KH_10F5/ (note: the Vue sets are based on less complex apps than the other sets appear to contain, but this is a personal observation. <3 Vue otherwise)
• Polymer apps in production, G4, 3G: https://www.webpagetest.org/result/170120_59_40GD/
• Polymer apps in production, desktop, cable: https://www.webpagetest.org/result/170120_W1_40PP/
• Polymer (based on custom list they provided) Moto G4, 3G: https://www.webpagetest.org/result/170125_GX_G9YF/
• Mobile Top 10 Moto G4, 3G: https://www.webpagetest.org/result/161218_TD_2fcc131f975da7e19ddb750f4991421d/
• Mobile Top 10 Desktop, Cable (with V8 runtime call stats in traces): https://www.webpagetest.org/result/170114_VE_TSK/
• Production Progressive Web Apps, Moto G4, 3G: https://www.webpagetest.org/result/161218_DW_b3e87329f9a5029d63b67759054fb0ce/
• Production Progressive Web Apps, Desktop, Cable: https://www.webpagetest.org/result/170114_CE_TY5/
• Chrome Loading team top sites, Moto G4, 3G: https://www.webpagetest.org/result/161218_RS_4be952cee9b8bede5ea16c26594ac38a/
• Chrome Loading team top sites, Desktop, cable: https://www.webpagetest.org/result/170114_4R_TT6/
• V8 Team Top 25, Moto G4, 3G: https://www.webpagetest.org/result/161218_Q6_b3202df8ed80da21ec73d9afcedead76/
• V8 Team Top 25, Desktop, Cable: https://www.webpagetest.org/result/170114_Y9_TW6/

Raw WebPageTest runs - Round 1 - Older study (December, 2016)

I put together this graphic when internally sharing the first version of this study. I decided to redo it as at least the network throttling setup from this study wasn't the same between the 2-3 web perf tooling systems used. This meant that while overall weight, time in script (parse/eval), FMP and load time were fine, the TTI numbers could not be concretely confirmed as 100% accurate. Instead, I redid the study once we added support for TTI to WebPageTest and I'd trust the numbers there (Round 2) a lot more.

• Summary insights + further data from this study: https://docs.google.com/a/google.com/spreadsheets/d/1QRdYgGVdDYB7kgfFKg0-0ViUonpvt8SXxcvv0gXKsEg/edit?usp=sharing
• React prod sites from their wiki - Moto G - https://www.webpagetest.org/result/161210_WK_aa40594011795974a4e2d26a24e0c3b6/
• React prod sites from their wiki - iPhone - https://www.webpagetest.org/result/161210_M1_7d32c93c8dcfaaa46cef42125aec75ba/
• Ember production sites - iPhone - https://www.webpagetest.org/result/161211_56_a8275acd05ef23932e85e1d83ceb8787/
• Ember production sites - Moto G - https://www.webpagetest.org/result/161211_PJ_d46293b5026e8c3ca7fa85d680cf2b0d/
• Loading team top sites - Moto G - https://www.webpagetest.org/result/161210_PC_82eaf67c4d57a8dbc533ad9c4a1ee08c/
• Loading team top sites - iPhone - https://www.webpagetest.org/result/161210_B7_9b36e27c2eaa0cec9cd5f8353a64acaa/
• Global top 10 mobile sites - iPhone - https://www.webpagetest.org/result/161210_93_ec315ea70330ecf03041396f21f5098f/
• Global top 10 mobile sites - Moto G - https://www.webpagetest.org/result/161210_WD_e2f65275bd6b3950dd2b367195e4d395/
• V8 top sites - Moto G - https://www.webpagetest.org/result/161210_MB_36ccea8a66638d08c5ea2855a04ca023/
• V8 top sites - iPhone - https://www.webpagetest.org/result/161210_37_6bca043df3316fb6e55efdba67d4b7bf/
• Speedometer 2 alpha - Moto G - https://www.webpagetest.org/result/161208_ET_6f0c253c8b2a92a74c8989b7fbdb8c9a/
• Catapult tool - Libscore.io React apps - Nexus 5 - https://docs.google.com/spreadsheets/d/1C4mO62NjVSVOVTsezlPwyY_pMIDV5b0401arE9GNviA/edit?usp=sharing
• Catapult tool - BuiltWith Angular - Nexus 5 - https://docs.google.com/spreadsheets/d/12Wb7fb15Kq9XeoAqKneGx12oD8dshe8MfCAf46lG1qQ/edit?usp=sharing
• BigQuery - angular.module() usage - https://docs.google.com/spreadsheets/d/1xdOr1cV9JBPyqlsCcATZI-cToUh7AC3Zhp1KP1YZzdk/edit?usp=sharing
• Catapult tool - Ember apps - Nexus 5 - https://docs.google.com/spreadsheets/d/1RwjhNnUQRkUXZXcnqj3zHISQFuiQnqhbH3qUc6owpHg/edit?usp=sharing
• Catapult tool - Libscore jQuery apps - Nexus 5- https://docs.google.com/spreadsheets/d/10vHRVUF7WoFubki_wKCFKXvWILxva8gGbzomC-8-CqQ/edit?usp=sharing
• Catapult tool - BigQuery - Sites using mod_pagespeed - Nexus 5 - https://docs.google.com/spreadsheets/d/1XpTMwpOkhrnpNLVfQGETqVK3TWVxIzrnGq5i1uqitb4/edit?usp=sharing
• WebPageTest - Libscore - Angular (2000 apps): https://docs.google.com/spreadsheets/d/1OBvcDSyoKHTjdH2CALui88orDF_n-0Yi92J93LDpUjQ/edit#gid=646387798&vpid=AD1 (Median insights ~ compile 445ms, eval 1784ms, FMP 7.2s, SI: 9s. Most apps spent 4.6s in JS on startup)
• WebPageTest - Libscore - Ember (695 apps): https://docs.google.com/spreadsheets/d/1m9eFna_Xg70c_MPG4tU9D4fOdJT9SZQ2u5fQCxEO--g/edit#gid=1536554622&vpid=AG649 (Median insights ~ compile 374ms, eval 2.7s, SI: 10s, FMP: 9s, load: 13.8s, most apps spent 6s in JS on startup)
• WebPageTest - Libscore - React (2000 apps): https://docs.google.com/spreadsheets/d/1HliA4jUMfqd4vFLDkYoTZQRv1oM1rM_8ahBAsXYfuAE/edit#gid=1982534624&vpid=A1 (Median insights ~ compile 481ms, eval 2609ms, load: 15.5s, FMP 6.5s, SI: 8s. Most apps spent 4.5s in script on startup)

Other data sets generated (Dec, 2016)

Note: many of the below data sets were generated before we installed Moto G4s in WebPageTest and had to use the Moto G1 instead. Some of the data sets will also be using earlier versions of the time-to-interactive metric and should not be directly compared in most cases to the latest data from 2017. This is historical data that's interesting and may be worth reexploring where particular data-sets didn't end up making it to the final study results.

• React production sites from their wiki - moto G1, iPhone 5c
• WebPageTest > BigQuery > Webpack built apps - limited to the first 2K of them.
• JS Framework benchmark modified to run automatically and deployed on https://v8-parse-eval-study.firebaseapp.com/. WebPageTest results over a Moto G, 3G.
• V8's Top 25 sites tested on a Moto G and iPhone 5
• Global mobile top 10 sites tested on a Moto G and iPhone 5
• Sites using mod_pagespeed in the wild - data from Catapult tests - TTI
• BuiltWithAngular apps - Catapult tests - TTI - Nexus 5
• Apps built using jQuery - Catapult - TTI. I decided not to heavily use this set too much. jQuery could be getting pulled in as a transitive dependency for ads.
• Libscore > React apps - Catapult - TTI - Nexus 5X
• Chrome's loading team top sites on a Moto G vs on an iPhone
• WebPageTest > BigQuery > angular.module() usage (I decided this set was too noisy)
• Libscore Angular 1.0 apps

What is the delta between First Interactive and Consistently Interactive?

This answers the question: Do we need two different TTI metrics?

@brendankenny started work on calculating the distribution of differences and might have some data on this one.

What is the 1st-party vs 3rd-party breakdown of sites with worst TTCI?

For sites with the worst time to consistently interactive, what percentage of these sites are loading JS from a first party origin ("site/app code") vs third party (maybe ads, widgets)?

This one may not be possible to compute without a change in Lighthouse that includes the 1st vs 3p breakdown. It may also be something we can calculate using WPT if we can figure out how to get the joins right.

Sites with the highest time-to-consistently interactive

Sample query from Rick:

SELECT
  JSON_EXTRACT(report, "$.audits.consistently-interactive.rawValue") AS ttci,
  JSON_EXTRACT(report, "$.audits.first-interactive.rawValue") AS ttfi,
  JSON_EXTRACT(report, "$.audits.first-meaningful-paint.rawValue") AS fmp
FROM
  [httparchive:har.latest_lighthouse_mobile]

Introduce README with details and logo

Low-effort. Already prepared but just need to upstream to this repo.

How stable are the TTFI and TTCI metrics?

This may provide data on overall confidence of the metrics.

Stability can be aggregate changes from run to run. We could measure the variance with the idea that if two runs were close in time together we would hope variance is low. Some sites will change. Some will have low variance.