bacalhau-project / bacalhau Goto Github PK
View Code? Open in Web Editor NEWCompute over Data framework for public, transparent, and optionally verifiable computation
Home Page: https://docs.bacalhau.org
License: Apache License 2.0
Compute over Data framework for public, transparent, and optionally verifiable computation
Home Page: https://docs.bacalhau.org
License: Apache License 2.0
P0 - some way to run the command through the bacalhau CLI and execute against local for detecting syntax errors
P1 - some form of syntax checker
As we move towards moving the project towards production, have tests that run regularly.
https://github.com/filecoin-project/bacalhau/tree/main/examples
export file_path=./high_quality_picture.jpg
cid=$(IPFS_PATH=/tmp/bacalhau-ipfs3163549147 ipfs add -q $file_path)
sudo apt-get -y update && sudo apt-get install -y graphicsmagick && \
gm convert high_quality_picture.jpg -quality 20% -colorspace Gray gray_scale.jpg
Unfortunately right now, logging interleaves various servers/nodes together in output.
Eg
Starting: /home/user/go/bin/dlv dap --check-go-version=false --listen=127.0.0.1:43559 --log-dest=3 from /home/user/code/bacalhau/cmd/bacalhau
DAP server listening at: 127.0.0.1:43559
2022-03-12T20:53:44Z error layer=debugger can't find build-id note on binary
Type 'dlv help' for list of commands.
20h53m45.48 DEBUG runtime/proc.go:6498 Log level from LOG_LEVEL_ENV_VAR: debug
Zap log level: debug
=== RUN TestCommands
20h53m45.76 DEBUG system/utils.go:16 IPFS Command: sudo [pkill ipfs]
20h53m46.78 DEBUG internal/dev_stack.go:37 [---------------------]
20h53m46.78 INFO internal/dev_stack.go:38 Creating Node #0
20h53m46.78 DEBUG internal/dev_stack.go:39 [---------------------]
20h53m47.35 DEBUG libp2p/libp2p.go:108 [Waiting for bacalhau libp2p context to finish.
]
20h53m47.36 DEBUG internal/dev_stack.go:68 bacalhau multiaddress: /ip4/172.31.35.25/tcp/33843/p2p/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T
20h53m47.36 DEBUG system/utils.go:16 IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs718364386]
20h53m47.36 DEBUG internal/jsonrpc_server.go:67 [Waiting for json rpc context to finish.]
20h53m47.37 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs init
20h53m47.37 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [init]
20h53m47.97 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs bootstrap rm --all
20h53m47.97 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m48.20 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/37257
20h53m48.20 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/37257]
20h53m48.44 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.API /ip4/0.0.0.0/tcp/44999
20h53m48.44 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/44999]
20h53m48.75 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/40349"]
20h53m48.75 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/40349"]]
20h53m49.07 DEBUG ipfs/utils.go:80 Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs daemon
20h53m49.07 DEBUG ipfs/utils.go:100 [waiting for ipfs context done
]
20h53m50.07 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs id
20h53m50.07 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [id]
20h53m50.32 DEBUG internal/dev_stack.go:104 bacalhau multiaddress: /ip4/172.31.35.25/tcp/33843/p2p/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T
20h53m50.32 DEBUG internal/dev_stack.go:105 ipfs multiaddress: /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2
20h53m50.32 DEBUG internal/dev_stack.go:106 ipfs repo: /tmp/bacalhau-ipfs718364386
20h53m50.32 DEBUG internal/dev_stack.go:116 [==== Complete]
20h53m50.32 DEBUG internal/dev_stack.go:37 [---------------------]
20h53m50.32 INFO internal/dev_stack.go:38 Creating Node #1
20h53m50.32 DEBUG internal/dev_stack.go:39 [---------------------]
20h53m50.95 DEBUG internal/dev_stack.go:68 bacalhau multiaddress: /ip4/172.31.35.25/tcp/43707/p2p/QmVwQH415Bi4XGtVkLyaeTJ8LWMEXBoSYChL7r9vz8YmSF
20h53m50.95 DEBUG libp2p/libp2p.go:108 [Waiting for bacalhau libp2p context to finish.
]
20h53m50.99 DEBUG internal/jsonrpc_server.go:67 [Waiting for json rpc context to finish.]
20h53m50.99 DEBUG system/utils.go:16 IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs150944631]
20h53m50.99 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs init
20h53m50.99 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [init]
20h53m51.70 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs bootstrap rm --all
20h53m51.70 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m51.85 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2
20h53m51.85 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h53m52.16 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/39687
20h53m52.16 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/39687]
20h53m52.29 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.API /ip4/0.0.0.0/tcp/45093
20h53m52.29 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/45093]
20h53m52.42 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/38221"]
20h53m52.42 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/38221"]]
20h53m52.73 DEBUG ipfs/utils.go:80 Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs daemon
20h53m52.73 DEBUG ipfs/utils.go:100 [waiting for ipfs context done
]
20h53m53.73 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs id
20h53m53.73 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [id]
20h53m53.84 DEBUG internal/dev_stack.go:104 bacalhau multiaddress: /ip4/172.31.35.25/tcp/43707/p2p/QmVwQH415Bi4XGtVkLyaeTJ8LWMEXBoSYChL7r9vz8YmSF
20h53m53.84 DEBUG internal/dev_stack.go:105 ipfs multiaddress: /ip4/127.0.0.1/tcp/38221/p2p/12D3KooWPGdapcvdoeVd6MGvf3aWWKGHKUUgctQJ952QQ9fzVo2A
20h53m53.84 DEBUG internal/dev_stack.go:106 ipfs repo: /tmp/bacalhau-ipfs150944631
20h53m53.84 DEBUG internal/dev_stack.go:116 [==== Complete]
20h53m53.84 DEBUG internal/dev_stack.go:37 [---------------------]
20h53m53.84 INFO internal/dev_stack.go:38 Creating Node #2
20h53m53.84 DEBUG internal/dev_stack.go:39 [---------------------]
20h53m54.62 DEBUG libp2p/libp2p.go:108 [Waiting for bacalhau libp2p context to finish.
]
20h53m54.63 DEBUG internal/dev_stack.go:68 bacalhau multiaddress: /ip4/172.31.35.25/tcp/45821/p2p/QmRqsfviqXhQ6Kj6zoHEdQ4BSA9PvDEgxnkSHtFmKCVXNK
20h53m54.64 DEBUG system/utils.go:16 IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs4068152898]
20h53m54.64 DEBUG internal/jsonrpc_server.go:67 [Waiting for json rpc context to finish.]
20h53m54.65 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs init
20h53m54.65 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [init]
20h53m54.98 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs bootstrap rm --all
20h53m54.98 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m55.12 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2
20h53m55.12 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h53m55.27 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/45503
20h53m55.27 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/45503]
20h53m55.39 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.API /ip4/0.0.0.0/tcp/45955
20h53m55.39 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/45955]
20h53m55.50 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/43017"]
20h53m55.50 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/43017"]]
20h53m55.63 DEBUG ipfs/utils.go:80 Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs daemon
20h53m55.63 DEBUG ipfs/utils.go:100 [waiting for ipfs context done
]
20h53m56.63 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs id
20h53m56.63 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [id]
20h53m56.76 DEBUG internal/dev_stack.go:104 bacalhau multiaddress: /ip4/172.31.35.25/tcp/45821/p2p/QmRqsfviqXhQ6Kj6zoHEdQ4BSA9PvDEgxnkSHtFmKCVXNK
20h53m56.77 DEBUG internal/dev_stack.go:105 ipfs multiaddress: /ip4/127.0.0.1/tcp/43017/p2p/12D3KooWMQQjoKrjwx8Ar5dGuXPsVhgVcekmGBpbdgpfvKbpbUsy
20h53m56.77 DEBUG internal/dev_stack.go:106 ipfs repo: /tmp/bacalhau-ipfs4068152898
20h53m56.77 DEBUG internal/dev_stack.go:116 [==== Complete]
20h53m56.77 DEBUG internal/dev_stack.go:123 [Finished provisioning nodes.]
=== RUN TestCommands/grep
20h53m58.77 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -Q ../../testdata/grep_file.txt
20h53m58.77 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/grep_file.txt]
20h53m58.88 DEBUG bacalhau/devstack_test.go:261 About to submit job:
cmd: grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
20h53m58.89 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"job_created","JobSpec":{"Engine":"","Commands":["grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
20h53m58.89 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m58.89 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
20h53m58.89 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h53m58.89 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs718364386}
20h53m58.89 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs refs local
20h53m58.89 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.89 INFO bacalhau/submit.go:130 Submitted Job Id: e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2
20h53m58.89 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m58.89 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
20h53m58.89 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h53m58.89 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs150944631}
20h53m58.89 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m58.89 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
20h53m58.89 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h53m58.89 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs4068152898}
20h53m58.89 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs refs local
20h53m58.89 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs refs local
20h53m58.89 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.89 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.90 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h53m58.90 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h53m59.00 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h53m59.00 DEBUG ipfs/utils.go:208 Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.00 DEBUG ipfs/utils.go:210 Total number of local refs: 23
20h53m59.00 DEBUG ipfs/utils.go:216 CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
20h53m59.00 INFO internal/compute_node.go:128 No matching CIDs found on this server. Passing on job
20h53m59.00 DEBUG internal/compute_node.go:67 We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m59.01 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h53m59.01 DEBUG ipfs/utils.go:208 Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.01 DEBUG ipfs/utils.go:210 Total number of local refs: 23
20h53m59.01 DEBUG ipfs/utils.go:216 CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
20h53m59.01 INFO internal/compute_node.go:128 No matching CIDs found on this server. Passing on job
20h53m59.01 DEBUG internal/compute_node.go:67 We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m59.09 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h53m59.09 DEBUG ipfs/utils.go:208 Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.09 DEBUG ipfs/utils.go:210 Total number of local refs: 25
20h53m59.09 DEBUG ipfs/utils.go:216 CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: true
20h53m59.09 INFO internal/compute_node.go:123 CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
20h53m59.09 DEBUG internal/compute_node.go:62 We are bidding on a job because the data is local!
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m59.09 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
20h53m59.10 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T"]},"JobState":{"State":"running","Status":"","Outputs":null}}
20h53m59.11 DEBUG system/utils.go:16 IPFS Command: mkdir [-p /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h53m59.11 DEBUG system/utils.go:54 Running system command: sudo [docker run --privileged -d --rm --name bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
20h53m59.70 DEBUG runtime/runtime.go:141 Script to run for job: sleep 2
grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
sleep 2
20h53m59.70 DEBUG system/utils.go:54 Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.3212996715.sh bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/job.sh]
20h53m59.86 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs init]
20h53m59.90 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m00.43 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs bootstrap rm --all]
20h54m00.69 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h54m00.95 DEBUG runtime/runtime.go:201 Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs daemon --mount]
20h54m01.90 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m04.91 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m05.95 DEBUG system/utils.go:28 Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
kiwi is delicious
20h54m08.91 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
Starting up command 'bash /job.sh' and attaching to process
Process finished (4.00 seconds)
/usr/local/lib/python3.8/dist-packages/psrecord/main.py:219: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
ax.set_ylim(0., max(log['cpu']) * 1.2)
20h54m11.90 DEBUG runtime/runtime.go:249 Writing stdout to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/stdout.log
20h54m11.90 DEBUG runtime/runtime.go:255 Writing stderr to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/stderr.log
20h54m11.90 INFO runtime/runtime.go:261 Finished writing results of job (Id: f7a1a9e3-82cf-4689-b65d-49517876a426) to results folder (/home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T).
20h54m11.90 DEBUG runtime/runtime.go:270 Copying files - Writing metrics.log to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.log
20h54m11.90 DEBUG system/utils.go:16 IPFS Command: sudo [docker cp bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/tmp/metrics.log /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.log]
20h54m12.07 DEBUG runtime/runtime.go:270 Copying files - Writing metrics.png to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.png
20h54m12.07 DEBUG system/utils.go:16 IPFS Command: sudo [docker cp bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/tmp/metrics.png /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.png]
20h54m12.21 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -rq /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T
20h54m12.21 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [add -rq /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h54m12.39 DEBUG system/utils.go:54 Running system command: sudo [docker rm -f bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426]
20h54m12.49 DEBUG runtime/runtime.go:214 Failed to run : exit status 137
20h54m12.74 INFO internal/compute_node.go:86 Completed the job - results cid: QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8
&{Id:e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2 Owner:QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T Spec:0xc00017e070 Deal:0xc00161b4a0 State:map[QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T:0xc0029d8200]}
20h54m12.75 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"results","JobSpec":null,"JobDeal":null,"JobState":{"State":"complete","Status":"Got job results cid: QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8","Outputs":[{"Engine":"ipfs","Cid":"QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8"}]}}
=== RUN TestCommands/sed
20h54m41.73 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -Q ../../testdata/sed_file.txt
20h54m41.73 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/sed_file.txt]
20h54m41.86 DEBUG bacalhau/devstack_test.go:261 About to submit job:
cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
20h54m41.86 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"job_created","JobSpec":{"Engine":"","Commands":["sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
20h54m41.87 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m41.87 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
20h54m41.87 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h54m41.87 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs718364386}
20h54m41.87 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs refs local
20h54m41.87 INFO bacalhau/submit.go:130 Submitted Job Id: 172c0fc0-a270-4937-965d-ae1f53d8107a
20h54m41.87 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m41.87 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
20h54m41.87 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h54m41.87 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs4068152898}
20h54m41.87 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs refs local
20h54m41.87 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87 DEBUG internal/compute_node.go:54 Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m41.87 DEBUG internal/compute_node.go:112 Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
20h54m41.87 DEBUG ipfs/utils.go:197 [Beginning to collect all refs in IPFS Repo.]
20h54m41.87 DEBUG ipfs/utils.go:198 RepoPath {/tmp/bacalhau-ipfs150944631}
20h54m41.87 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h54m41.87 DEBUG ipfs/utils.go:24 ipfs command --> IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs refs local
20h54m41.87 DEBUG system/utils.go:61 Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h54m41.99 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h54m41.99 DEBUG ipfs/utils.go:208 Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m41.99 DEBUG ipfs/utils.go:210 Total number of local refs: 23
20h54m41.99 DEBUG ipfs/utils.go:216 CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
20h54m41.99 INFO internal/compute_node.go:128 No matching CIDs found on this server. Passing on job
20h54m41.99 DEBUG internal/compute_node.go:67 We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m42.00 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h54m42.00 DEBUG ipfs/utils.go:208 Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m42.00 DEBUG ipfs/utils.go:210 Total number of local refs: 34
20h54m42.00 DEBUG ipfs/utils.go:216 CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
20h54m42.00 INFO internal/compute_node.go:123 CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
20h54m42.00 DEBUG internal/compute_node.go:62 We are bidding on a job because the data is local!
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m42.00 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
20h54m42.01 DEBUG libp2p/libp2p.go:329 Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T"]},"JobState":{"State":"running","Status":"","Outputs":null}}
20h54m42.01 DEBUG system/utils.go:16 IPFS Command: mkdir [-p /home/user/.bacalhau/results/172c0fc0/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h54m42.01 DEBUG system/utils.go:54 Running system command: sudo [docker run --privileged -d --rm --name bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
20h54m42.02 DEBUG ipfs/utils.go:204 [Finished collecting refs in IPFS Repo.]
20h54m42.02 DEBUG ipfs/utils.go:208 Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m42.02 DEBUG ipfs/utils.go:210 Total number of local refs: 23
20h54m42.02 DEBUG ipfs/utils.go:216 CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
20h54m42.02 INFO internal/compute_node.go:128 No matching CIDs found on this server. Passing on job
20h54m42.02 DEBUG internal/compute_node.go:67 We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m42.63 DEBUG runtime/runtime.go:141 Script to run for job: sleep 2
sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
20h54m42.63 DEBUG system/utils.go:54 Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.4225559538.sh bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c:/job.sh]
20h54m42.87 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs init]
20h54m42.87 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m43.51 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs bootstrap rm --all]
20h54m43.79 DEBUG system/utils.go:54 Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h54m44.11 DEBUG runtime/runtime.go:201 Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs daemon --mount]
20h54m44.88 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m47.88 DEBUG system/utils.go:43 Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m49.11 DEBUG system/utils.go:28 Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
2021-01-01 05:54:00,LISBON,38.7305,-9.1502,11.0983,2.7888
2021-01-01 06:33:00,LISBON,38.7384,-9.1600,11.4359,3.9713
2021-01-01 12:54:00,LISBON,38.7482,-9.1376,14.3912,4.4965
2021-01-01 16:14:00,LISBON,38.7444,-9.1408,12.3883,4.0889
Detaching and terminating target process
To simplify debugging, we should understand/figure out how to group server information/output together.
// TODO: This is stupid (for now) but need to add the %s at the end because we don't have an elegant way to run without a cid (yet). Will fix later.
Lots of times there's a syntax error before submitting. We need to check the bash script before submission.
When I execute the following command from the CLI, it works fine:
sed -n '/38.7[2-4]..,-9.1[3-7]../p' examples/gps-substitution/temperature_sensor_data.csv
However, whene I execute the following, it doesn't work:
bin/bacalhau submit --cids=$cid --commands="sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/$cid" --jsonrpc-port=35203
With the following error:
sed: -e expression #1, char 12: unterminated address regex
Ideas?
Not sure what's going on, but when I checkout master on the DO node, and do make test
it runs forever and then times out.
That said, it runs fine in Github Actions.
Anything I can do to debug?
No response on execution - not clear anything happened.
attempting to execute the following:
go run . --jsonrpc-port=39359 submit --commands "docker run hello-world"
Does not work with the following:
we are updating a job!:
&{JobId:d5d620a9-aed5-47c2-a807-c12dcaebe517 NodeId:QmeSWYJnbTYks3xqBnE9x7TTekmTnYKEX87bBdwcjXMq4c State:error Status:Error running the job: exit status 1
Output:}
I'm sure it's some permissions thing - but we should make sure to have a clear(er) error output.
Particularly for testing, but generally important
https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360
We should detect this and fail.
kubectl –wait
Submit:
awk -F',' '{x=38.7077507-; y=-9.1365919-; if(x^2+y^2<0.3^2) print}' /ipfs/$cid
Appears as:
awk -F' ' '{x=38.7077507-; y=-9.1365919-; if(x^2+y^2<0.3^2) print}' /ipfs/QmPVy9XiWRBrfvtiSBcCiWZSwSBZwVrG6oJo6eiar1qycP
Notice the lack of ,
sed
! (and we can see the result in the docker logs)At least with docker mode, build an arm64 image.
Deliverable:
Repro:
Mac M1, fresh from main
Execute:
make test
Result:
===========================
[2207] INF bacalhau/submit.go:147 > Submitted Job Id: 06b7779c-87fe-4c0c-b4a0-6e399f92e58e
[2207] INF internal/compute_node.go:138 > CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
[2207] INF internal/compute_node.go:192 > Results directory for job id (06b7779c-87fe-4c0c-b4a0-6e399f92e58e) exists: /Users/daaronch/.bacalhau/results/06b7779c/QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx
[2207] ERR internal/compute_node.go:95 > ERROR running the job: Error connecting host ipfs into container:
Error: exit status 1
Output: Error: connect 12D3KooWDTMfZuHVaYRGRbHmgS6AMeoDKWCiMDDekcBBPSVbWuBx failure: failed to dial 12D3KooWDTMfZuHVaYRGRbHmgS6AMeoDKWCiMDDekcBBPSVbWuBx:
* [/ip4/172.17.0.3/tcp/4001] dial tcp4 172.17.0.3:4001: connect: connection refused
&{Id:06b7779c-87fe-4c0c-b4a0-6e399f92e58e Owner:QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx Spec:0x1400299b7a0 Deal:0x1400306b2c0 State:map[QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx:0x14000a528c0]}
devstack_test.go:233:
Error Trace: devstack_test.go:233
Error: Received unexpected error:
open /Users/daaronch/.bacalhau/results/06b7779c/QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx/stdout.log: no such file or directory
Test: TestCommands/grep
devstack_test.go:235:
Error Trace: devstack_test.go:235
Error: Should be true
Test: TestCommands/grep
devstack_test.go:237:
Error Trace: devstack_test.go:237
Error: Not equal:
expected: 1
actual : 4
Test: TestCommands/grep
Messages: Count mismatch:
Expected: 4
Actual: 1
The IPFS instance inside the Ignite VMs is told to connect to:
added /ip4/127.0.0.1/tcp/35787/p2p/12D3KooWFwnmuDniA3VY18BdyJMLhJKfbE4ntDswNmRjuvFxbQXE
However, 127.0.0.1 on that port doesn't route to the private IPFS network outside the VM, however it does work because of some announcement trick that IPFS does. It's nice that it works by chance, but make it more robust by giving it not the 0'th address but instead, maybe all of them? Or the one that looks like an IP of the host that the VMs can route to.
Used a URL for peering with the following - this is a correct address, shouldn't fail.
Mar 19 11:28:23 ip-10-0-1-15 bacalhau[1611]: failed to parse multiaddr "/ip4/ip-10-0-1-78.eu-west-1.compute.internal/tcp/0/p2p/Qmdpeuf4BpyP51cm3W3ezazgLrMGy8rrJt9Bf3stbvByZ6": invalid value "ip-10-0-1-78.eu-west-1.compute.internal" for protocol ip4: failed to parse ip4 addr: ip-10-0-1-78.eu-west-1.compute.internal
2022/03/08 01:32:34 Starting ipfs daemon --mount inside the runtime failed with: exit status 137
13ff6ac3-85cd-4be7-9a20-5fe4216b71aa03887500-6c97-4228-a3d3-e36209cd0b31
Concern?
https://github.com/filecoin-project/bacalhau/tree/main/examples
Ran submit job:
go run . submit --cids=$cid --commands="grep -o 'W' /ipfs/$cid | wc -l" --jsonrpc-port=41923
Everything ran normally up to this point:
was selected because jobs CID are local:
[Qmdd3GdLhQJENPLs6VyGByaZWytDCcjJezGUcWonEybrzu]
Output:}
INFO[0001] Created VM with ID "8af9ab483799ac66" and name "d978308c-95d2-4193-99b6-a1c1f9fbd189a9fa74f4-e62f-46b8-87ae-28676ba25fd4"
INFO[0001] Networking is handled by "cni"
INFO[0001] Started Firecracker VM "8af9ab483799ac66" in a container with ID "ignite-8af9ab483799ac66"
INFO[0001] Waiting for the ssh daemon within the VM to start...
generating ED25519 keypair...done
peer identity: 12D3KooWLaE9mxuP57Pn4wC3JrKTnpmeqKWQJmYwB7RwjSbiYaqK
initializing IPFS node at /root/.ipfs
to get started, enter:
ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme
2022-03-01T17:22:42.502Z ERROR provider.queue queue/queue.go:124 Failed to enqueue cid: leveldb: closed
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt
removed /ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
removed /ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
added /ip4/127.0.0.1/tcp/36815/p2p/12D3KooWPszx6ZjoexB2nMnWWxPCtxbnSv6S4tck61jPCDLGid5x
It APPEARS that ignite is taking over and re-creating a new ipfs stack?
When I execute the following command:
LOG_LEVEL=debug go test -run ^TestCommands$ github.com/filecoin-project/bacalhau/cmd/bacalhau/
The first (and only the first) test passes - though non-deterministically - e.g. could be any of the three randomly chosen.
Here's the log output:
[2207] DBG internal/dev_stack.go:116 > ==== Complete
[2207] DBG internal/dev_stack.go:123 > Finished provisioning nodes.
[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
name: awk
cmd: awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/%s
file: ../../testdata/awk_file.txt
========================================
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/awk_file.txt]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2\u003c0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] INF bacalhau/submit.go:130 > Submitted Job Id: 5b5a108a-3c17-4cef-a6be-277c42761788
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 25
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}
[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: 5b5a108a-3c17-4cef-a6be-277c42761788)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (5b5a108a-3c17-4cef-a6be-277c42761788) exists: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.1711282195.sh bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562:/job.sh]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs init]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs daemon --mount R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
2021-01-01 00:00:00,LISBON,38.5940,-8.9303,5.9366,21.9316
2021-01-01 16:39:00,LISBON,38.6990,-8.9610,10.9692,15.2670
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
/usr/local/lib/python3.8/dist-packages/psrecord/main.py:219: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
ax.set_ylim(0., max(log['cpu']) * 1.2)
Starting up command 'bash /job.sh' and attaching to process
Process finished (4.02 seconds)
[2207] DBG runtime/runtime.go:256 > Writing stdout to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:260 > Directory found: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:266 > Expected folder /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua exists?: true R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:273 > Writing stderr to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stderr.log
R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] INF runtime/runtime.go:279 > Finished writing results of job (Id: 265f0663-6204-4da1-bc29-722bdb83c562) to results folder (/home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua). R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:288 > Copying files - Writing metrics.log to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/metrics.log
R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:288 > Copying files - Writing metrics.png to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/metrics.png
R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -rq /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker rm -f bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562]
[2207] DBG runtime/runtime.go:220 > Failed to run : exit status 137 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] INF internal/compute_node.go:95 > Completed the job - results cid: QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF
&{Id:5b5a108a-3c17-4cef-a6be-277c42761788 Owner:QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua Spec:0xc002e60460 Deal:0xc0001f1e00 State:map[QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua:0xc0001f2980]}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"results","JobSpec":null,"JobDeal":null,"JobState":{"State":"complete","Status":"Got job results cid: QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF","Outputs":[{"Engine":"ipfs","Cid":"QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF"}]}}
[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
name: grep
cmd: grep kiwi /ipfs/%s
file: ../../testdata/grep_file.txt
========================================
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/grep_file.txt]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] INF bacalhau/submit.go:130 > Submitted Job Id: d61f7f7f-4ab8-4488-ab3d-f25ffe43a159
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
name: sed
cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/%s
file: ../../testdata/sed_file.txt
========================================
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/sed_file.txt]
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 34
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: true
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: d61f7f7f-4ab8-4488-ab3d-f25ffe43a159)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (d61f7f7f-4ab8-4488-ab3d-f25ffe43a159) exists: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] INF bacalhau/submit.go:130 > Submitted Job Id: 73e5c514-62f3-42c1-8257-52039dcbe73d
[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 34
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: 73e5c514-62f3-42c1-8257-52039dcbe73d)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (73e5c514-62f3-42c1-8257-52039dcbe73d) exists: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
sleep 2
R:D61F7F7F-4AB8-4488-AB3D-F25FFE43A159
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.2044548828.sh bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3:/job.sh]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
R:73E5C514-62F3-42C1-8257-52039DCBE73D
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.578337918.sh bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c:/job.sh]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs init]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs init]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs daemon --mount R:D61F7F7F-4AB8-4488-AB3D-F25FFE43A159
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs daemon --mount R:73E5C514-62F3-42C1-8257-52039DCBE73D
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon
[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon
[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon
[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon
[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon
[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon
[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon
[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon
[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon
[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
--- FAIL: TestCommands (30.88s)
--- FAIL: TestCommands/grep (0.11s)
devstack_test.go:235:
Error Trace: devstack_test.go:235
Error: Received unexpected error:
open /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log: no such file or directory
Test: TestCommands/grep
devstack_test.go:237:
Error Trace: devstack_test.go:237
Error: Should be true
Test: TestCommands/grep
devstack_test.go:239:
Error Trace: devstack_test.go:239
Error: Not equal:
expected: 1
actual : 4
Test: TestCommands/grep
Messages: Count mismatch:
Expected: 4
Actual: 1
--- FAIL: TestCommands/sed (6.15s)
devstack_test.go:235:
Error Trace: devstack_test.go:235
Error: Received unexpected error:
open /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log: no such file or directory
Test: TestCommands/sed
devstack_test.go:237:
Error Trace: devstack_test.go:237
Error: Should be true
Test: TestCommands/sed
devstack_test.go:239:
Error Trace: devstack_test.go:239
Error: Not equal:
expected: 1
actual : 7
Test: TestCommands/sed
Messages: Count mismatch:
Expected: 7
Actual: 1
I SUSPECT this is something about after the job finishing, the system closes down the nodes.
For example, midway through the log, you can see:
[2207] INF internal/compute_node.go:180 > Results directory for job id (73e5c514-62f3-42c1-8257-52039dcbe73d) exists: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
Which is confirmed here - https://github.com/filecoin-project/bacalhau/blob/reformatting_logs_for_multi_threading/internal/compute_node.go#L176
However, by the time it gets to here: https://github.com/filecoin-project/bacalhau/blob/d0aef41ca20deb14741a614eced28d92e3ed0f78/internal/runtime/runtime.go#L245
The directory is gone. A bit stumped - my suspicion:
(I think the last one is the most likely, since it's the newest)
vm
is being run.We just got bitten by trying to write directly to the os.Stdout & std.Stderr streams from go-routines and this also just blocked the tests
The solution is to just return a buffer of the commands stdout & stderr and print that buffer to the console when the job has finished
It would be nice to have streaming output though - let's do this by reading the buffer from a go-routine whilst the command is running
// TODO: We should bail out if we do not fit the execution profile of this machine. E.g., the below:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.