beefyfinance / beefy-cowllector

the one that harvest all the beefy vaults

Archived repository

This repository has been archived and is no longer maintained.

Please use https://github.com/beefyfinance/beefy-cowllector-v2 instead

🧑‍🌾 Cowllector 🌾

The path to your next Open Source cowtribution

🐮 About Cowllector

This is the bot that harvest all strategies from all Beefy's chains, also do another things like:

• Notify the BIFI rewards pool
• Harvest the Beefy fee batch from all Beefy´s chains to distribute it.

Cowllectors does NOT focus in to be profitable, so don't expect it to win some profit when it harvests. Cowllector harvest script was create it to bring resilience and consistency in all active vaults strategies, giving at least ONE harvest every 24 hour to every strat in every chain

After every run of harvests in every chain, you can find the harvest report in our discord channel #Harvester

📖 Prerequisites

In order to run the project you need node>=16.13, yarn>=1.22 installed on our development machines

🖥️ Local development

To install the application:

yarn

To start harvest all chains in parallel:

yarn harvest

To start harvest one chain:

node ./scripts/harvest_child.js <chain id>

🔑 Enviroment

See enviroment variable example file for ENV required to run cowllector

🤝 Contributing

We encourage you to contribute to Cowllector!

We recommend to use this commit convention that helps you write your commits in a way that is easy to understand and process by others.

In case you want to contribute, please follow next steps:

• fork this repo
• create a new branch and named using conventional commit reference
• commit your changes using conventional commit
• push your change in your forked repo
• createa a PR from your new branch directly to our master branch

🍕 Community

• Got Questions? Join the conversation in our Discord.
• Want to up to date with Beefy? Follow us in Twitter.

beefy-cowllector's Issues

Cronos harvesting isn't working properly

I've tried multiple RPC nodes, they all return Error: processing response error (body="{\"jsonrpc\":\"2.0\",\"id\":55,\"error\":{\"code\":-32000,\"message\":\"internal\"}}\n", error={"code":-32000}, requestBody="{\"method\":\"eth_sendRawTransaction\",\"params\":[\"0xf86c82017586048c27395000833d09009417f499f009bb796132b493f4be12f8650a4f076880844641257d55a066a14d39214e921502490b56632d071026663c46b968dd5e9f20aba8105e80fea03f55f3570254412c6d62116efc9818e408e626c0ff3baa60faa622d98f0416b1\"],\"id\":55,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="https://cronosrpc-1.xstaking.sg/", code=SERVER_ERROR, version=web/5.0.9)

Make all the cowllector use the new unified LP endpoint

There's a new endpoint to use as LP oracle at https://api.beefy.finance/lps

The cowllector currently uses a bunch of different custom endpoints, like "/pancake/lps" and "jetfuel/lps". We should move everything to use the single endpoint so that the other ones can be deleted from the API.

Add redundancy to RPC selection when harvesting.

Description

We are looking to improve the reliability of our harvesting bot. Its objective is to make sure that we call the harvest() function at least every x hours. Different strategies have different harvest intervals configured.

The script that harvests can be found at scripts/harvest.js. The current implementation is not very resilient. If it has troubles connecting to a particular chain's RPC, or for some reason the RPC fails it just keeps going to the next strategy.

The problem is that RPCs in some chains are pretty flaky. The idea would be to add 3-5 RPCs as options for each chain. The script would try to harvest using one of them. If it fails for an RPC issue it would select another one, until it succeeds.

Tasks

• If a strategy harvest fails due to an RPC error, try with a different one from 3-5 options.

The cowllector won't harvest a strat without at least one manual harvest

The current isNewHarvestPeriod check always returns false if a strategy has never been harvested before. This is because the StratHarvested() event has never been fired.

The correct action would be to return true to pass validation and harvest in this case

Add yarn script to fill out data about a new vault automatically

It could save time and avoid errors to have a simple yarn command that you can give it a new vault address and it:

• Fills out the strategy data for strats.json
• Fills out the vault data for defistation.json

Defistation script is broken

It seems like we are reporting 0 TVL on Defistation https://www.defistation.io/. The relevant script is updateDefistation.js

We should fix this to show up in their leaderboard again.

Add bot to cowllector that harvests profitably

Description

Beefy strategies have a harvest() function that generates yield. It should be called at least once per day to make sure we are compounding rewards earned by Beefy users. The harvest function is public and there's a small reward in place to incentivize people running bots that maintain our vaults.

We run one of these bots to help harvests happen. There are times where a vault is small, or not very profitable and other bots don't harvest them. We run a bot to make sure that at least a minimum level of harvesting happens. Around 1-2 times per day should be enough.

Existing

We currently have a script that runs every hour on Heroku. It uses the Heroku scheduler add-on. The script that runs can be found at scripts/harvest.js.

It is a fairly simple solution. It doesn't care about profitability of harvests. It is concerned with guaranteeing a minimum level of service. It will go through each strategy on the strats.json and check if more than the configured interval has passed since the last harvest. If enough time passed without a harvest, it will go ahead and harvest it.

Required

We initially had two bots running. One that provided this minimum level of service, even if at a loss. The other bot would seek to harvest profitably with a certain profit margin, for example 20%. The profitable bot can harvest popular vaults more frequently which helps pay for the other bot.

Tasks

• Add bot that harvest based on profitability.
• Configurable target profit margin.

Stop harvesting old vaults with little TVL

We have a bot running that guarantees at least a minimum level of service for vaults. We have discussed about managing costs and decided to stop harvesting vaults that have less than $10k in TVL and are older than 2 months. This would cut down our costs considerably.

Try again when 'replacement fee is too low'

There are sometimes that a transaction from the cowllector fails because there's a pending tx with a higher gas price with the same nonce.

When this happens, we should repeat the tx with slighly higher gas price. We should keep increasing the gas price (within reason) until the tx passes.

Make notifyRewards more reliable

Context

RIght now the code to notifyRewards just tries to notify once and doesn't do anything if it fails.

Problem

If notifyRewards fails for example because it has a problem reaching the RPC node, no rewards are notified. This means that the BIFI Maxi vault harvest fails. This eventually creates a big harvest that loses a lot of revenue from slippage.

Solution

Making the call try to notify until it succeeds could help resolve the recurrent issue with BIFI Maxi failed harvests.

Harvest all chains in parallel

Context

The current harvest.js script is pretty much the same that we had when we were in 1-2 chains and with less than 100 vaults.

It harvests vaults one by one without any type of parallelization. This has become a problem as the script takes hours to run.
There are multiple improvements we could do to this structure. The easiest one which would decrease execution time by a lot is to do all chains in parallel. There's no reason why Fantom vaults should wait while BSC vaults get harvested.

It shouldn't be too hard to adapt the script to run all chains at the same time.

Add script to check if all strats and vaults are present

I just reviewed the strats.json file and the defistation file and:

1. Added quite a bit of missing vaults.
2. Corrected a bunch of addresses in strats.json, where the vault contract address was added instead of the strat address.

We could use a simple script that checks if all the addresses on strats.json and defistation.json are correct. And if some vaults are missing. We could use the pools.js file from the app to validate.

Add missing vaults/strats into defistation.json and strats.json

There are some mentions that some vaults seem to be missing from the harvest schedule, and from Defistation.

1. We should review the list of vaults at defistation.json and add the missing vaults. The vault contract address goes here.

2. We should review the list of strats at strats.json and add the missing strats. The strategy contract address goes here.

Create bot to optimize callFee on strategies

Context

Strategies have a public harvest() function that is responsible for earning yield. This function should be called at least once per day to provide decent value for vault users.

There is an incentive in place for people to call the function. It is a percent of the harvested rewards and it's determined by the callFee variable. callFee starts at 50 which means that the person calling the function keeps 0.5% of the harvested rewards.

The incentive for Beefy is to keep that number as low as possible while also getting the required service. There are currently many instances where we are overpaying for the service.

Most of the value of compounding comes from doing it 1-3 times per day. But if harvesting is profitable then bots will do it more often, there are times where harvests happen every 6-9 seconds. This is unnecessary and signals that the callFee is too high for that vault at that given moment. Every $ that we can save on harvesting fees is $ that goes to the BIFI holders or the treasury instead.

The Beefy strategy contracts have a setWithdrawalFee() function that can decrease/increase the callFee. We are not doing much with this at the moment.

Solution

Create a bot that we can run as part of the cowllector, or a standalone. It should seek to optimize the callFee of vaults, lower it when we're paying too much and increasing it if it's too low.

There are many ways to do this, for example checking the harvest frequency of the past 5-10 harvests and if it's happening too often (more than once per 4h period for example) then it should decrease the callFee slightly.

Create abi encoder helper function

Many of our scripts consume encoded transaction data before executing. For example the data fields in all the timelock scripts.

At the moment we are encoding these using https://abi.hashex.org/. It would be great to have function that is easy to use and we can use from within the scripts. Without having to manually encode using another website.

Remove duplicated sources for chain data.

We first had some helpers to get chain data at utils/getChainData.js. We are now storing everything in a json at data/chains.js

It seems like we should unify those two and move the last two data pieces from getChainData.jsinto `chains.js. Refactoring the chains json array into an object might make it easier to use as well.