Comments (3)
Hello! It may be related to the scale of your dataset partition. If the test/val dataset is too small, then the loss will be unstable.
On the other hand, the evaluation accuracy only depends on one exact value, which is parsed from the generated text, but the val/test loss is calculated among all the tokens the model generates.
We also find that the validation loss may not be a reliable indicator of the generalization performance. For more details, please refer to our paper.
Best regards,
from federatedscope.
Hello! It may be related to the scale of your dataset partition. If the test/val dataset is too small, then the loss will be unstable. On the other hand, the evaluation accuracy only depends on one exact value, which is parsed from the generated text, but the val/test loss is calculated among all the tokens the model generates. We also find that the validation loss may not be a reliable indicator of the generalization performance. For more details, please refer to our paper. Best regards,
I wonder the phenomenon discussed in your paper is just in low-fidelity scenario or in general FL?
from federatedscope.
Hello! It may be related to the scale of your dataset partition. If the test/val dataset is too small, then the loss will be unstable. On the other hand, the evaluation accuracy only depends on one exact value, which is parsed from the generated text, but the val/test loss is calculated among all the tokens the model generates. We also find that the validation loss may not be a reliable indicator of the generalization performance. For more details, please refer to our paper. Best regards,
I wonder the phenomenon discussed in your paper is just in low-fidelity scenario or in general FL?
In the paper, what we observe is in a low-fidelity scenario, but finetuning LLM in general FL, it may be interesting to investigate the relationship between val/test loss and the final evaluation accuracy. I'm not sure there's been a study on this。
from federatedscope.
Related Issues (20)
- How to use multi GPU to finetune Llama2 HOT 2
- Unable to run demo in hyperparameter optimization HOT 2
- Server global evaluation total number HOT 1
- Error with 4 bit quantized LLM HOT 3
- TypeError: call_file_data() missing 1 required positional argument: 'client_cfgs' HOT 6
- Some questions about Backdoor Bench HOT 2
- 训练得到的total_flops是负数 HOT 2
- LDA splitter:ValueError: too many values to unpack (expected 2) HOT 2
- how can i use my saving model HOT 1
- how can i use my saving model HOT 3
- How can i get the output result? HOT 1
- Soft prompt Tuning
- one bug in federatedscope/gfl/fedsageplus/trainer.py
- 关于图学习中链接任务的样例使用错误 HOT 1
- Issue with federate.method set to global HOT 1
- 运行代码FederatedScope/tree/FSreal,得不到想要的结果,请问可能是什么原因?
- GPU Memory Issue HOT 10
- too many values to unpack (expected 2) in model_builder.py HOT 1
- WARNING: Skip the batch due to the loss is NaN, it may be caused by exceeding the precision or invalid labels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from federatedscope.