filecoin-station / core Goto Github PK

File:

logs/activity.log

Example:

2023-02-16T22:59:47.385+0100 Saturn Node is online and connected to 9 peers

We currently store logs in format

[$DATE] $LINE

• simple to read for humans
• can be handled with bash tools
• is not too hard to parse in Node.js.

We could change it to NDJSON:

{"date":date, ...line}

• structured data

• simple to read if you use tools like pino-pretty
• can still be handled with bash tools (albeit more)
• is trivial to parse

Default value:7834 (npx named-port Filecoin Station Core)

We have recently reworked how rewards are paid to Station operators to use the model "push once a month". Station Desktop no longer shows "total earnings" but instead displays "scheduled rewards", see filecoin-station/desktop#1082

We need to make similar changes in Station Core and possibly in the underlying modules (Zinnia, Bacalhau) too.

Keywords to search for in the codebase: totalEarnings, total_earnings.

Slack threads started by our users:

• https://filecoinproject.slack.com/archives/C03S6LXSRB8/p1699528485011289
• https://filecoinproject.slack.com/files/U02N6KCL19S/F0652MPRRBN/image.png
• https://filecoinproject.slack.com/archives/C04N76Y689M/p1699512083801239

$STATION_ROOT = ~/.station/

I am currently unable to access ghcr.io due to network restrictions, which is preventing me from using the necessary images for my tasks. As a workaround, I kindly request that a Docker Hub mirror be made available to allow for uninterrupted use of the images required. Thank you for your prompt attention to this matter.

I ran this on a VM with 20 gigs and it filled and then failed due to running out of space

[SATURN]  Recoding downloaded data 32768
[SATURN] 2023-04-01T12:09:42.819-0400   WARN    dagstore/upgrader       mount/upgrader.go:209   failed to refetch       {"shard": "122020daa0b6f298b7e6927867e15bea735f009a79c46278636c883e2df91c255cc3", "error": "failed to fetch and copy underlying mount to transient file: failed to download: failed to write: write /<snip>/.cache/filecoin-station-core/modules/saturn-L2-node/transients/transient-122020daa0b6f298b7e6927867e15bea735f009a79c46278636c883e2df91c255cc3.partial: no space left on device"}
[SATURN] 2023-04-01T12:09:42.819-0400   ERROR   car-store       logs/logs.go:46 download failed as failed to register/acquire shard     {"id": "4f14c43a-da16-85ed-152d-72466e3c1b9e", "err": "failed to acquire shard: failed to acquire shard: failed to acquire reader of mount on initialization: mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to download: failed to write: write /<snip>/.cache/filecoin-station-core/modules/saturn-L2-node/transients/transient-122020daa0b6f298b7e6927867e15bea735f009a79c46278636c883e2df91c255cc3.partial: no space left on device"}
[SATURN] 2023-04-01T12:09:42.819-0400   INFO    car-store       logs/logs.go:35 released cache miss semaphore   {"id": "4f14c43a-da16-85ed-152d-72466e3c1b9e"}
[SATURN] 2023-04-01T12:09:48.606-0400   INFO    l1-interop      l1interop/l1sseclient.go:184    received request from L1        {"l1": "145.239.92.38", "json": "{\"requestId\":\"4444e53f21e4f3c0b364b98b5d778c3e\",\"root\":\"QmbFmReFr32th9ZrC722KEkFiMBVwStmP5wr1UTuSqxtWj\"}"}
[SATURN] 2023-04-01T12:09:48.606-0400   INFO    saturn-l2       logs/logs.go:35 parsed CAR transfer request received from L1    {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e", "l1": "145.239.92.38", "req": {"RequestId":"4444e53f-21e4-f3c0-b364-b98b5d778c3e","Root":{"/":"QmbFmReFr32th9ZrC722KEkFiMBVwStmP5wr1UTuSqxtWj"},"Selector":{}}}
[SATURN] 2023-04-01T12:09:48.606-0400   INFO    car-store       logs/logs.go:35 got CAR request {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e", "root": "QmbFmReFr32th9ZrC722KEkFiMBVwStmP5wr1UTuSqxtWj"}
[SATURN] 2023-04-01T12:09:48.606-0400   INFO    car-store       logs/logs.go:35 queued to cache miss buffer     {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e"}
[SATURN] 2023-04-01T12:09:48.610-0400   INFO    car-store       logs/logs.go:35 dequeued request from cache miss buffer {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e"}
[SATURN] 2023-04-01T12:09:48.611-0400   INFO    car-store       logs/logs.go:35 more than one cache miss for given root, downloading and caching it     {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e"}
[SATURN] 2023-04-01T12:09:48.611-0400   INFO    car-store       logs/logs.go:35 acquired cache miss semaphore   {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e"}
[SATURN] 2023-04-01T12:09:48.611-0400   INFO    saturn-l2       logs/logs.go:35 not serving CAR as CAR not found        {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e", "err": "CAR not found"}
[SATURN] 2023-04-01T12:09:48.612-0400   INFO    saturn-l2       logs/logs.go:35 not sending CAR over POST       {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e", "err": "failed to send http post request with CAR to L1;url=https://145.239.92.38/data/QmbFmReFr32th9ZrC722KEkFiMBVwStmP5wr1UTuSqxtWj?requestId=4444e53f-21e4-f3c0-b364-b98b5d778c3e, err=Post \"https://145.239.92.38/data/QmbFmReFr32th9ZrC722KEkFiMBVwStmP5wr1UTuSqxtWj?requestId=4444e53f-21e4-f3c0-b364-b98b5d778c3e\": CAR not found", "l1": "145.239.92.38"}
[SATURN]
[SATURN]  Recoding downloaded data 32768
[SATURN] 2023-04-01T12:09:49.742-0400   WARN    dagstore/upgrader       mount/upgrader.go:209   failed to refetch       {"shard": "1220bfe79eb6450a504b9f080d95c3f0cbbe36a3fb7a48591644a5392278603b7820", "error": "failed to fetch and copy underlying mount to transient file: failed to download: failed to write: write /<snip>/.cache/filecoin-station-core/modules/saturn-L2-node/transients/transient-1220bfe79eb6450a504b9f080d95c3f0cbbe36a3fb7a48591644a5392278603b7820.partial: no space left on device"}
[SATURN] 2023-04-01T12:09:49.742-0400   ERROR   car-store       logs/logs.go:46 download failed as failed to register/acquire shard     {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e", "err": "failed to acquire shard: failed to acquire shard: failed to acquire reader of mount on initialization: mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to download: failed to write: write /<snip>/.cache/filecoin-station-core/modules/saturn-L2-node/transients/transient-1220bfe79eb6450a504b9f080d95c3f0cbbe36a3fb7a48591644a5392278603b7820.partial: no space left on device"}
[SATURN] 2023-04-01T12:09:49.742-0400   INFO    car-store       logs/logs.go:35 released cache miss semaphore   {"id": "4444e53f-21e4-f3c0-b364-b98b5d778c3e"}
Error: ENOSPC: no space left on device, write

While fixing our docker builds, I noticed that Station Core running in a docker container is printing deprecation warnings about file handles; see below. This happened when the Station could not start Saturn L2 Node (the executable was not found). Now that I fixed the problem, the warnings are no longer printed. Maybe our error handling for Saturn startup is not as robust as it should be.

(node:1) Warning: Closing file descriptor 20 on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1) [DEP0137] DeprecationWarning: Closing a FileHandle object on garbage collection is deprecated. Please close FileHandle objects explicitly using FileHandle.prototype.close(). In the future, an error will be thrown if a file descriptor is closed during garbage collection.
(node:1) Warning: Closing file descriptor 28 on garbage collection
(node:1) Warning: Closing file descriptor 27 on garbage collection
(node:1) Warning: Closing file descriptor 19 on garbage collection
(node:1) Warning: Closing file descriptor 29 on garbage collection
(node:1) Warning: Closing file descriptor 30 on garbage collection
(node:1) Warning: Closing file descriptor 26 on garbage collection
(node:1) Warning: Closing file descriptor 25 on garbage collection

Per the discussion in #75

• Add disclaimer to docs

Station Core tells modules to store state files in ~/Library/Application Support:

According to Apple File System Conventions, these files are backed up by iTunes and iCloud.

Use this directory to store all app data files except those associated with the user’s documents. For example, you might use this directory to store app-created data files, configuration files, templates, or other fixed or modifiable resources that are managed by the app. An app might use this directory to store a modifiable copy of resources contained initially in the app’s bundle. A game might use this directory to store new levels purchased by the user and downloaded from a server.
All content in this directory should be placed in a custom subdirectory whose name is that of your app’s bundle identifier or your company.
In iOS, the contents of this directory are backed up by iTunes and iCloud.

I am proposing to change Station Core to use the cache path instead, the same way as Station Desktop does.

https://github.com/filecoin-station/desktop/blob/7ab56a4479ec1fe96bceed1c3c787cc365d9ca6f/main/consts.js#L30-L31

    case 'darwin': // macOS
      return path.join(app.getPath('home'), 'Library', 'Caches', app.name)

Quoting from File System Conventions:

Use this directory to write any app-specific support files that your app can re-create easily. Your app is generally responsible for managing the contents of this directory and for adding and deleting files as needed.
In iOS 2.2 and later, the contents of this directory are not backed up by iTunes or iCloud. In addition, the system removes files in this directory during a full restoration of the device.
In iOS 5.0 and later, the system may delete the Caches directory on rare occasions when the system is very low on disk space. This will never occur while an app is running. However, be aware that restoring from backup is not necessarily the only condition under which the Caches directory can be erased.

See also macOS Library Directory Details:

Application Support - Contains all app-specific data and support files. These are the files that your app creates and manages on behalf of the user and can include files that contain user data.
Caches - Contains cached data that can be regenerated as needed. Apps should never rely on the existence of cache files. Cache files should be placed in a directory whose name matches the bundle identifier of the app.

Docker image builds appear to have been failing since v5.0:
https://github.com/filecoin-station/core/actions/workflows/docker.yml

From looking at the logs, it appears that the npm ci RUN command is failing as it depends on the scripts/post-install.js file which is not copied prior to this step.

(A fix may be to copy the scripts directory prior to this step in the Dockerfile, but I am opening as an issue as I'm not sure if that dependency is desirable or unintended.)

filecoin-station/desktop#1074 (comment)

See https://github.blog/changelog/2023-04-13-dependabot-now-supports-fetching-release-notes-and-changelogs-for-docker-images/

You can now fetch release notes, changelogs and commit history for Docker update pull requests with Dependabot. This will allow you to quickly evaluate the stability risk of the dependency upgrade. To enable support, add the org.opencontainers.image.source label to the Dockerfile with the URL of the source repository. Additionally, the repository should be tagged with the same tags as the published Docker images.

Let's add the org.opencontainers.image.source label to our Dockerfile to enable this feature for people building on top of Station Core docker images.

Note: This is a very low priority for us. I am filling this issue to see if anybody from our community is interested in this feature.

• File: $STATION_ROOT/logs/modules/${MODULE_NAME}.log
• Example: 2023-02-16T22:59:47.385+0100 INFO: Saturn Node is online and connected to 9 peers

• Publish as @filecoin-station/core
• Add install instructions to README

ghcr.io is blocked by china's firewall

• Create docker container
• Publish to docker registry
• Document usage

docker run filecoin-station-core --name station --detach

• https://nodejs.org/en/docs/guides/nodejs-docker-webapp/
• https://www.docker.com/blog/how-to-use-the-alpine-docker-official-image/
• https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#non-root-user)

• metrics

We are running a Station Core instance on Fly.io. In the past, we observed out-of-memory errors when running with 512MB and 768MB available RAM. We solved this problem by bumping up the RAM size to 1GB.

If Station Core does indeed require at least 1GB of RAM, then running Station Desktop (which includes Station Core) on low-end consumer computers and Raspberry PIs will become unpleasant.

We should investigate our memory usage and reduce it to a more reasonable size.

Periodically submit this to Influx (bucket with 30d retention), on a low frequency

In order to better know how much resources the network is able to offer, capture:

• CPU
  • core count #411
  • core clock #411
  • core brand #411
  • core model?
• Storage: free/total #411
• Memory: free/total
• OS filecoin-station/desktop#582 #49

• File: $STATION_ROOT/logs/modules/${MODULE_NAME}.err.log
• Example: 2023-02-16T22:59:47.385+0100 Dang!

See https://github.blog/changelog/2023-04-19-npm-provenance-public-beta/

npm packages built on a cloud CI/CD system (like GitHub Actions) can now publish with provenance, meaning the package has verifiable links back to its source code and build instructions.

It would be great to setup our release publishing process to get the provenance badge on the npm registry website.

• Add disclaimer to docs

• File: $STATION_ROOT/logs/metrics.log
• Example: 2023-02-16T22:59:47.385+0100 {"totalJobsCompleted":123}

• docs

• When operators supply their Station Desktop address as FIL_WALLET_ADDRESS, they can use Station Desktop to withdraw (sponsored). This is the suggested flow.

• Operators can withdraw manually by direct contract interaction or using the unsponsored transfer web UI (coming later, see below)