s0nik42 / lotus-farcaster Goto Github PK
View Code? Open in Web Editor NEWPrometheus exporter for Filecoin lotus node
License: MIT License
Prometheus exporter for Filecoin lotus node
License: MIT License
=> CACHED [ 8/10] RUN chmod 0775 /usr/local/bin/docker_run_script.sh 0.0s
=> ERROR [ 9/10] RUN ln -sv /usr/local/bin/python3.9 /usr/bin/python3 0.2s
------
> [ 9/10] RUN ln -sv /usr/local/bin/python3.9 /usr/bin/python3:
#11 0.225 ln: failed to create symbolic link '/usr/bin/python3': File exists
------
executor failed running [/bin/sh -c ln -sv /usr/local/bin/python3.9 /usr/bin/python3]: exit code: 1
Ubuntu 20.04,
Grafana is installed from installation guide
Prometheus from package
farcaster installation logs:
sudo ./install.sh _<user>_
[sudo] password for :
Install required debian packages (python3-toml python3-aiohttp python3-pip prometheus-node-exporter) ? Yes [ OK ]
Verify local prometheus node exporter connectivity [ OK ]
Configure prometheus node exporter folder [ OK ]
Install python modules ? Yes [ OK ]
Install farcaster binary ? Yes [ OK ]
Install cron job [ OK ]
Configure farcaster [ OK ]
Launch configuration assistant ? Yes [ -> ]
Configure Miner API connection string [ -> ]
Use info found in ~/.lotusminer path [eyJhb...XqG24:/ip4/0.0.0.0/tcp/2345/http] ? Yes [ OK ]
Configure Daemon API connection string [ -> ]
Use info found in ~/.lotus path [eyJhb...6XfAc:/ip4/127.0.0.1/tcp/1234/http] ? Yes [ OK ]
Does the miner has a markets node (if unsure answer no) ? No [SKIP]
The script is now checking that everything is working fine [ -> ]
Check if _<user>_ exist : [ OK ]
Check if lotus-exporter-farcaster.py run properly : ERROR: list index out of range
[ KO ] error encountered :
# HELP lotus_scrape_execution_succeed return 1 if lotus-farcaster execution was successfully
# TYPE lotus_scrape_execution_succeed gauge
lotus_scrape_execution_succeed { } 0
TODO : run manually : sudo -u _<user>_ /usr/local/bin/lotus-exporter-farcaster.py
next:
sudo -u `<user>` /usr/local/bin/lotus-exporter-farcaster.py
# HELP lotus_scrape_execution_succeed return 1 if lotus-farcaster execution was successfully
# TYPE lotus_scrape_execution_succeed gauge
lotus_scrape_execution_succeed { } 0
ERROR: list index out of range
ERROR http://10.10.0.1:14200/rpc/v0 returned no result
DEBUG: method StateMarketStorageDeal / params [1213665, []]
DEBUG: {'jsonrpc': '2.0', 'id': 3, 'error': {'code': 1, 'message': 'deal 1213665 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed'}}
ERROR http://10.10.0.1:14200/rpc/v0 returned no result
DEBUG: method StateMarketStorageDeal / params [1213664, []]
DEBUG: {'jsonrpc': '2.0', 'id': 3, 'error': {'code': 1, 'message': 'deal 1213664 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed'}}
get spammed with mails!
Hi, what is the LOTUS_USER_USERNAME??? How does it mean??
when installing and using the user "filecoin" the cron job is installed with "filecoin" as user - even if that user has no write access to var/lib/prometheus/node-exporter
debian 10
you see the typo. and the info is wrong - the miner is below 10TB due to post failure see: #11
lotus-miner version 1.16.1
Ubuntu 20.04
i can not install lotus-farcaster. Install script runs through, but then this happens. I have split market node, no boost
lotus-exporter-farcaster.py --debug
_# HELP lotus_scrape_execution_succeed return 1 if lotus-farcaster execution was successfully
_# TYPE lotus_scrape_execution_succeed gauge
lotus_scrape_execution_succeed { } 0
ERROR: Traceback (most recent call last):
File "/usr/local/bin/lotus-exporter-farcaster.py", line 1615, in main
run(args, output=sys.stdout)
File "/usr/local/bin/lotus-exporter-farcaster.py", line 1582, in run
collect(daemon, miner, markets, metrics, addresses_config)
File "/usr/local/bin/lotus-exporter-farcaster.py", line 1435, in collect
pending_publish_deals = markets.get_pending_publish_deals()["dealPublish"]
TypeError: 'NoneType' object is not subscriptable
This wasn't an issue for me until I crossed the mining threshold, lol -- but instead of scouring logs for "did I miss a block", it would be great to see some historic data about blocks mined along with details on the calculation time variance, misses, etc. I'm not sure if that info is available any way other than grepping logs though, so it might be impractical.
I had finished the installation, when I test it with curl -s -o - http://localhost:9100/metrics -s |grep "lotus_info"
, it output nothing.
When I run python3 /usr/local/bin/lotus-exporter-farcaster.py
, the output became normal with lots of information of miner and lotus node, but it reports error at last:
Traceback (most recent call last): File "/usr/local/bin/lotus-exporter-farcaster.py", line 656, in <module> main() File "/usr/local/bin/lotus-exporter-farcaster.py", line 406, in main gpus = len(info["Resources"]["GPUs"]) TypeError: object of type 'NoneType' has no len()
Would be great to see the following graph:
Daily changes on the sectors status / power changes: new sectors added since last winpost, or show this numbers under the existing "Miner Power" graph, or "Other-Miner Power" graph.
Burn fees are quite a concern for miners. Would be great to show a graph for the burn fees, and weekly or monthly total accumulated fees if possible.
Can I run it with normal user? I don't have the root permission.
My miner works in Ubuntu 20.04 Server version (no graphic user interface). After installation,
curl -s -o - http://localhost:9100/metrics -s |grep "lotus_info" output nothing while
/usr/local/bin/lotus-exporter-farcaster.py output correctly without mistakes. I guess it's due to ubuntu server version. My question is how to deploy and configure on miner ubuntu server version.
"timeFrom": null, "timeShift": null, "title": "时空证明", "transformations": [ { "id": "seriesToColumns", "options": { "byField": "index" } }, { "id": "organize", "options": { "excludeByName": { "Time": true, "Time 1": true, "Time 2": true, "Time 3": true, "Time 4": true, "Time 5": true, "__name__": true, "__name__ 1": true, "__name__ 2": true, "__name__ 3": true, "__name__ 4": true, "instance": true, "instance 1": true, "instance 2": true, "instance 3": true, "instance 4": true, "instance 5": true, "job": true, "job 1": true, "job 2": true, "job 3": true, "job 4": true, "job 5": true, "miner_id": true, "miner_id 1": true, "miner_id 2": true, "miner_id 3": true, "miner_id 4": true, "miner_id 5": true }, "indexByName": { "Time": 1, "Value #A": 6, "Value #B": 9, "Value #C": 10, "Value #D": 11, "Value #E": 7, "Value #F": 8, "__name__": 2, "index": 0, "instance": 3, "job": 4, "miner_id": 5 }, "renameByName": { "Value #A": "开始时间", "Value #B": "全部扇区", "Value #C": "恢复扇区", "Value #D": "失败扇区", "Value #E": "有效扇区", "Value #F": "Proven Partition", "index": "Id", "miner_id": "" } } } ]
I am trying to change the title to Chinese, the time will become a numerical value, is it because my translation method is incorrect? Can you provide a complete translation solution?
As a miner, I would like to see my multiple lotus processes running within a single worker machine appearing in the "Worker" section of farcaster dashboard so that I can monitor the status on each of them.
Background:
I was able to mange to run two C2 processes in a single worker machine, each of them using different port, and different GPU. However, farcaster only shows one.
Challenges:
when using lotus-miner sealing workers, these two C2 processes showing up as separate Id and entry but both of them are using the same host name (because they are running in the same machine). It would be good for forcaster to list them separately under the Worker section.
Issue Description
try to install lotus-farcaster by running the install.sh. It displayed the following error:
ERROR: user filecoin doesn't seems to be a lotus user /home/filecoin/.lotus/config.toml doesn't exist
Expected Results
Seems the farcaster tries to find the lotus on a default installation path. Lotus-farcaster should use the LOTUS_MINER_PATH or command line parameters like --miner-repo to find the correct lotus installation.
Hello,
The CRON was working at the beginning, but it stop working after some issue on my miner restart..
Any idea how I can get this fixed? The dashboard are awesome..
My miner ID are f0400103
# TYPE lotus_power gauge
lotus_power { miner_id="f0400103", scope="miner", power_type="RawBytePower" } 0
lotus_power { miner_id="f0400103", scope="miner", power_type="QualityAdjPower" } 0
lotus_power { miner_id="f0400103", scope="network", power_type="RawBytePower" } 3900315704791924736
lotus_power { miner_id="f0400103", scope="network", power_type="QualityAdjPower" } 3901392404504870912
# HELP lotus_power_mining_eligibility return miner mining eligibility
# TYPE lotus_power_mining_eligibility gauge
ERROR MinerGetBaseInfo return no result
KNOWN_REASON your miner reports wrong info to the chain and thats pretty bad (not just for the dashboard)
SOLUTION restart your miner and node
lotus_scrape_execution_succeed { } 0
ERROR http://127.0.0.1:1234/rpc/v0 returned no result :
DEBUG : {'jsonrpc': '2.0', 'id': 3, 'error': {'code': 1, 'message': 'deal 1246079 not found'}}
Finished the installation and got to the testing part:
FARCASTER INSTALLATION COMPLETED
********************************************************************************
TESTING :
curl -s -o - http://localhost:9100/metrics -s |grep "lotus_info"
I can get other metrics, but there is no lotus_info metric in there.
From the syslog I found:
Dec 3 12:29:01 CRON[2448262]: (root) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var
/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/promet
heus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi )
need a go version of lotus-farcaster
I have two miner on the same node, how can I monitor then within one place?
After installing based on instructions (On Ubuntu 20.04.1), the cronjob gives this error every minute:
Traceback (most recent call last):
File "/usr/local/bin/lotus-exporter-farcaster.py", line 616, in <module>
main()
File "/usr/local/bin/lotus-exporter-farcaster.py", line 212, in main
miner_control0 = daemon_stats["result"]["ControlAddresses"][0]
TypeError: 'NoneType' object is not subscriptable
?
I ran my local lotus node with my root account.
When I run ./install.sh root
, it returns user root doesn't seems to be a lotus user /root/.lotus/config.toml doesn't exist
or when I run the command with my only user ./install.sh user
, it returns the same error.
What could I do?
I'm a daily user of farcaster and basically run most of my storage provider operation from info generated on the platform. As filecoin is moving from phase 1 to phase 2, and deals get more and more important, its very obvious that lotus cli tools has no meaningful visualization tools to monitor and handle incoming deals. lotus-miner storage-deals list
is becoming useless as I write...
I see a new version of farcaster including a seperate dashboard (so running two screens). One screen showing the sealing, storage and chain data (as we know farcaster today), and one screen delivering the deal related insights - here are some ideas for a second dash:
There is probably plenty more. My hope is that it can be collected it a one-screen super dense view, like we know from the current dashboard.
KNOWN_REASON your miner needs to have a power >0 for Farcaster to work. Its linked to a Lotus API bug)
SOLUTION restart your miner and node
we are running multiple workers on a single host.
the dashboard is only showing one worker.
the idefix 1/2 hosts have 2 workers each. the total worker count is 9+1 miner --> 10
the workers use one GPU each, so there is information missing about one GPU used on these hosts (i wouldn't care otherwise how many workers are shown/host in that section)
with version of lotus v1.17.2-rc2 when I run Farcaster I have to comment out :
# # GENERATE PENDINGDEALS
# pending_publish_deals = markets.get_pending_publish_deals()["dealPublish"]
#
# if pending_publish_deals and len(pending_publish_deals["Deals"]) > 0:
# # Remove microseconds because not managed by python then convert to epoch
# publish_start = pending_publish_deals["Start"]
# plus_position = publish_start.find("+")
# publish_start = publish_start[:plus_position-3] + publish_start[plus_position:]
# publish_start = datetime.datetime.strptime(publish_start, '%Y-%m-%dT%H:%M:%S.%f%z').strftime('%s')
#
# # Clean PublishINseconds
# publish_in_seconds = pending_publish_deals["Period"] / 1000000000
#
# for deal in pending_publish_deals["Deals"]:
# deal_size = int(deal["PieceSize"]["n"])
# client = daemon.address_lookup(deal["ClientAddress"])
# duration = int(deal["EndEpoch"]["n"])-int(deal["StartEpoch"]["n"])
# provider_collateral = int(deal["ProviderCollateral"]["n"])
# metrics.add("miner_pending_publish_deal", value=1, miner_id=miner_id, deal_size=deal_size, client=client, duration=duration, provider_collateral=provider_collateral, publish_start=publish_start, publish_in_seconds=publish_in_seconds)
And then it works, just an FYI. Don't know how to fix it else (I'm not running Boost on this node, just the Market node)
Hello team! I am wondering if farcaster currently supports Venus implementation of filecoin?
Tips there is no power, but actually have 32Gi power
root@filecoin:/usr/src/lotus-farcaster/lotus-exporter-farcaster# ./check.sh root
Check if root exist : [ OK ]
Check if root is a lotus user : [ OK ]
Check if lotus-exporter-farcaster.py run properly : ERROR MinerGetBaseInfo return no result
KNOWN_REASON your miner needs to have a power >0 for Farcaster to work. Its linked to a Lotus API bug)
SOLUTION restart your miner and node
[ KO ] error encountered :
TODO : run manually : sudo -u root /usr/local/bin/lotus-exporter-farcaster.py
root@filecoin:/usr/src/lotus-farcaster/lotus-exporter-farcaster# lotus-miner info
Enabled subsystems (from miner API): [Mining Sealing SectorStorage Markets]
Enabled subsystems (from markets API): [Mining Sealing SectorStorage Markets]
Chain: [sync ok] [basefee 139.396 pFIL]
Miner: f01241194 (32 GiB sectors)
Power: 32 Gi / 10.9 Ei (0.0000%)
Raw: 32 GiB / 10.85 EiB (0.0000%)
Committed: 32 GiB
Proving: 32 GiB
Below minimum power threshold, no blocks will be won
Miner Balance: 236.608 mFIL
PreCommit: 54.483 mFIL
Pledge: 158.955 mFIL
Vesting: 0
Available: 23.17 mFIL
Market Balance: 0
Locked: 0
Available: 0
Worker Balance: 133.403 mFIL
Control: 156.035 mFIL
Total Spendable: 312.608 mFIL
Sectors:
Total: 28
Proving: 1
Removed: 27
Storage Deals: 0, 0 B
Retrieval Deals (complete): 0, 0 B
How to install farcaster for lower versions of ubuntu, such as 18.04
Progress track:
since 1.2.2 the base gas fee is shown with lotus-miner info
it would be nice to have it in the dashboard
lotus-worker info shows what tasks they are allowed to to
if there a way to get that info through the miner?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.