Comments (3)
[@Brofessional is interested in re-examining this issue, so here's some more info.]
Detecting systematic incast delays (or other problems) would be facilitated by the new-ish options that support logging of detailed timing-data.
-
configure syslog[-ng] to forward from servers (and clients, assuming we're not sure what issues might be involved). Consider your configuration of NFS vs RDMA DAL. You probably want syslog[-ng] over TCP (rather than UDP) so you don't miss any lines.
-
Add timing flags to your favorite namespace and/or repo, in MARFSCONFIGRC. These flags are OR'ed together, so conceivably you could have some on the repo and some on the NS, but I prefer to leave the repo alone, and simply have a duplicate NS (same repo as the one I would normally use), but with timing flags installed. That way, I can choose to use this NS when I want to collect timing info. For example:
# everything the same, but with timing_flags (and mnt_path)
<namespace>
<name>mc.timing</name>
<mnt_path>/mc.timing</mnt_path>
<alias>...</alias>
<bperms>RM,WM,RD,WD,TD,UD</bperms>
<iperms>RM,WM,RD,WD,TD,UD</iperms>
<iwrite_repo_name>...</iwrite_repo_name>
<range>
<min_size>0</min_size>
<max_size>-1</max_size>
<repo_name>mc3+1</repo_name>
</range>
<md_path>.../mdfs</md_path>
<trash_md_path>.../mc-trash</trash_md_path>
<fsinfo_path>.../fsinfo</fsinfo_path>
<quota_space>-1</quota_space>
<quota_names>-1</quota_names>
<timing_flags> OPEN,RW,CLOSE,RENAME,ERASURE,THREAD,HANDLE </timing_flags>
</namespace>
-
invoke pftool with '-l'
-
dig through syslog. It may be useful to look for the '*_h' histogram lines in the TIMING_INFO diagnostics, in syslog. If you want to separate out threads, use the "separate_log_threads" utility to extract per-thread logs out of syslog.
from marfs.
We dug into this some time ago, which lead to a discovery of what looked like "incast" problems. For the cctest and ODSU test-beds, we got big improvements in performance and reliability by tuning the network parameters. tcp_sack=1 seemed to be key to resolving the incast behavior as observed at the client, but I'm still concerned there could be incast at some/all of the servers, as they e.g. make many concurrent requests of other servers. This is tricky to measure, but one approach is to do brief tcpdump captures during activity, and then analyze for systematic delays.
Meanwhile, there has been a lot of new development that might affect throughput (e.g. writing recovery-info). One way to take measurements is to instrument pftool for gprof/valgrind, and then collect and analyze the profile output-files after a run. I'll do this again, when I get back to this issue.
from marfs.
This has always been
from marfs.
Related Issues (20)
- return-codes from str_2_post(), etc, should distinguish version-mismatch from other errors
- marfs_statvfs() should reflect quota-limitations HOT 1
- Installation checker HOT 1
- sync_ and close_data() failures are masked HOT 2
- clean-up: always build with DAL/MDAL HOT 1
- Multiple Simultaneous Writers HOT 6
- rename / unlink bug HOT 1
- strip out DAL/MDAL build-options HOT 1
- misconfigured RDMA auth-file gets ENODATA from fuse
- MarFS trash semantics - HOT 2
- the system calls getxattr("security.capability") on every FUSE write()! HOT 1
- marfs_rename() should do trash_truncate on the destination HOT 9
- Pftool Libxml Include Requires Subdir HOT 1
- GC Rewrite: Per-Namespace Trash
- Quota Tool Modification: Per-Namespace Trash
- MarFS should do more in the setup process HOT 1
- Create a single node MarFS example
- Move documentation for MarFS online HOT 1
- Line wrap in install doc doesn't work as expected HOT 1
- Change multi line commands in tutorial to be one command HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marfs.