Comments (6)
@derekperkins I'm sorry if I left you with the impression that query planning is just string parsing.
What I meant was more that PGO could probably help a lot of things that are going to be common across most use cases - such as string parsing that will not much be different between query sets.
query planning is a super difficult and very hard, rocket-science level task 😅 just so my boss knows how necessary and complex it really is! ;)
FWIW - I think we should do this. It would surprise me very much if our performance didn't improve noticeably.
from vitess.
It does sound promising indeed, I wonder what the numbers would be with Vitess. We would have to try this out before implementing it in our release builds.
Regarding the collection of profiles, to keep the entire build secure and "inside a box" we could automatically generate the profiles in our release build workflow that way we avoid having a dynamic parameter in our build.
from vitess.
More details about their implementation. TL;DR: collect profiles during benchmarking.
https://dolthub.com/blog/2024-04-19-golang-pgo-builds-using-github-actions/
This should fit pretty nicely into our existing infrastructure, given all the work that has gone into benchmark tooling.
from vitess.
This was just discussed in the monthly meeting. The question was asked if arewefastyet benchmarks would be representative enough, and if we risk making things slower
- paraphrased from @systay, "query planning is just string parsing, so I expect it to be beneficial"
- Should be close to risk-free, per the PGO docs
“While a profile that is not representative of production behavior will result in optimizations in cold parts of the application, it should not make hot parts of the application slower” https://go.dev/doc/pgo
The first steps will be to collect any profile information and submit a PR with a default.pgo
files next to vtgate
and/or vttablet
main.go
, which should automatically be incorporated into the build.
The standard approach to building is to store a pprof CPU profile with filename default.pgo in the main package directory of the profiled binary. By default, go build will detect default.pgo files automatically and enable PGO.
Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source.
from vitess.
@systay I wasn't implying anything about the complexity of query planning, just agreeing with your comment that to PGO, it's lots of string parsing ;)
from vitess.
Cloudflare is reporting ~3.5% improvements across their fleet
https://blog.cloudflare.com/reclaiming-cpu-for-free-with-pgo
from vitess.
Related Issues (20)
- Bug Report: MySQL 8.4 default configuration contains removed configuration variable `default_authentication_plugin`
- Feature Request: Add more flexibility backing up/restoring from different backup engines. HOT 1
- Feature Request: allow current/before `PRIMARY` to be specified in PRS HOT 3
- Bug Report: Uncorrelated subquery with a aggregation is not working HOT 1
- Optimize `schemadiff` inter-dependent diff computation path HOT 1
- Bug Report: Premature buffering stop during concurrent external reparenting can lead to query failures HOT 1
- Feature Request: Speed up disk read for backups with fadvise(2) HOT 1
- Release of `v18.0.6` HOT 1
- Release of `v19.0.5` HOT 1
- Release of `v20.0.1` HOT 1
- Feature Request: Allow cross cell promotion in PRS HOT 2
- RFC: Add upstream MySQL 9.0 vector type to Vitess
- Feature Request: MySQL 8.4 LTS support
- Bug Report: Can't cancel Materialize workflow on non-existent table HOT 1
- BUG: Outer join conditions for sharded queries
- Feature Request: Binlog Timestamp Watermarking for VStream HOT 11
- VStream can panic with incorrectly specified TableLastPKs HOT 1
- Feature Request: Generalize result-level statistics aggregation
- Bug Report: incorrect logging from restoring a backup
- Bug Report: Column name is the value of bindvar when using SELECT 1 with aggregation HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vitess.