Comments (8)
@sambacha Okay so I tried this and while it does work , its painfully slow. Like you'll be waiting over 15 minutes (and this was for a snap synced dir, so I imagine archive one will be much worse) to do initial dir load (as there's a dir scan that happens) and sshfs will do a copy to in memory of each file. Chaindata itself will have many hundred of files so it's just not feasible.
from evm-dev-station.
yes sshfs , and this was on local network too so I imagine remote on different network will be even worse
sshfs -o debug,sshfs_debug,loglevel=debug,allow_other,kill_on_unmount,reconnect,allow_other,direct_io,auto_cache thelio-archive:/eth-archive/goerli temp-mount-2
where sshfs running on my MacBook and thelio-archive is my local network linux machine.
from evm-dev-station.
Agree - this is a good idea. I think as a fastest solution , I could do like what emacs tramps kind of does. that is, is you provide the path to the db (and I assume you know you can't use this on a running chain, just a snapshot hence I realize why you said replications service aka you can temp stop it while using the db)
you provide the path to the db starting with ssh, so like ssh:///home/repl1/eth-mainnet/geth/chaindata
and then I from then on will do ssh reads for whenever the golang code needs a file read operation . I think that's the fastest, easiest way cause I don't want a second binary/daemon/websocket. Everyone already has ssh access to their servers
from evm-dev-station.
also are you mainly using pebble backed or leveldb backed machines? I assume probably leveldb?
from evm-dev-station.
actually I just looked at this again https://github.com/libfuse/sshfs - did you try it, maybe it should work already as is?
from evm-dev-station.
@sambacha Okay so I tried this and while it does work , its painfully slow. Like you'll be waiting over 15 minutes (and this was for a snap synced dir, so I imagine archive one will be much worse) to do initial dir load (as there's a dir scan that happens) and sshfs will do a copy to in memory of each file. Chaindata itself will have many hundred of files so it's just not feasible.
What are you using for remote FS, FUSE?
from evm-dev-station.
OK OK, how about something similar to this?
This PR is allows users to export their chain into an an archive format called Era1. It is formulated similarly to the Era1 format, which is optimized for reading and distribution CL data. The Era and Era1 format are stricter subsets of a simple type-length-value scheme called e2store2, both developed by the Nimbus team.
from evm-dev-station.
Interesting - I didn't know about that will look up
from evm-dev-station.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from evm-dev-station.