Comments (2)
stas00
commented on June 1, 2024
1
Thank you for catching my mistake, Valentin - super appreciating noticing the reversal!
Fixed here: #44
from ml-engineering.
stas00
commented on June 1, 2024
oh, bf16 regime just means that you do the math in bf16 - either through AMP or no AMP where model weights are in bf16.
At least in the LM the inputs are IDs, not floats. but further forwards' inputs are floats.
If you feel that more commentary is needed in that section please kindly suggest what to add.
from ml-engineering.
Related Issues (18)
- Parallel training hangs HOT 10
- Daisy chain batch jobs HOT 1
- Improve folder structure HOT 3
- Convert to bfloat16 failing HOT 2
- pip install -r build/requirements.txt fails due to github_md_utils HOT 3
- Clarification for gradient memory in mixed precision training HOT 3
- Quarto Site HOT 3
- Conflicting opinions about streaming data from cloud storage? HOT 2
- discuss the solutions to Not fully recovering spikes HOT 7
- ML
- TPU v4 has 1,200GB/s of mem bandwidth and not 2,400, right? HOT 1
- Question about the right hidden dim when using SwiGLU HOT 3
- Missing `hparams` section HOT 2
- Adding another logbook (kinda) HOT 1
- convert markdown to pdf HOT 10
- Minor Typo in emulate multi node HOT 4
- GPU requirements and cost estimation. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ml-engineering.