cannot retrieve logs from faasd about faasd

[LucasRoesler]

@fundef1 thank you for the report, unfortunately, I have been working on this already for the last couple of weeks. I have made some progress in #93, but not enough to actually close this.

One of the issues seems to be an unfortunate behavior where faasd code would try to read logs from the journalctl output before journalctl had written any output. This is probably what you are seeing now

The code (once cleaned up) in #93 addresses this issue.

However, there seems to be another issue also happening.

In some personal branches, i have added additional logging in faasd and the gateway code. I have determined that

1. the faasd log provider does correctly tail the logs,
2. that it correctly sends the logs to the faasd log handler
3. the log handler correctly sends the logs to the gateway and
4. the gateway log handler correctly writes the logs to the ResponseWriter as they are received.

Basically, everything inside the openfaas authored code is working (or can at least log the expected log line at each moment)

It seems like there might be some kind of buffering on the response writer. For example, seems to hold the last line of the logs for some time and I can not determine the exact way to force it to actually flush to the client. Note that we already use the Flusher interface. I have been able to get it to "tail" the logs by just spamming the function with invocations. Eventually it does send logs, but it takes 15 -- 20s for each batch.

To me, the fact that the Gateway sees and is able to log every expected function log line suggests that is some bug (a missing config or header) in the Gateway or with how faasd configures the provider server.

[alexellis]

@LucasRoesler is right, there's something we're looking into that is making the CLI's logs feature fail in some circumstances.

Here's a workaround that you can use to access logs:

export NAME="figlet"
journalctl -t openfaas-fn:$NAME

You'll need to run it over SSH or SSH into the faasd host to execute it.

[fundef1]

Thanks for the update. Good to know it's not something on my end.

[LucasRoesler]

@alexellis this is how I was debugging the issue up to the gateway

Update the gateway image to use docker.io/theaxer/gateway:508b17b6, it is based on this branch

sudo vim /var/lib/faasd/docker-compose.yaml

Deploy the branch version of faasd

make local \
    && sudo pkill -9 faasd \
    && sudo cp bin/faasd /usr/local/bin

Stop the faasd components and restart

sudo systemctl stop faasd
sudo -i
cd /var/lib/faasd

pkill faasd && faasd up

and

sudo -i
cd /var/lib/faasd-provider

faasd provider

Redeploy the function to avoid any networking issues

faas-cli deploy --image=functions/nodeinfo-http:latest --name=nodeinfo
faas-cli invoke nodeinfo <<< "" && date
faas-cli logs nodeinfo

Get the raw logs response using curl

curl  -u "$(sudo cat /var/lib/faasd/secrets/basic-auth-user):$(sudo cat /var/lib/faasd/secrets/basic-auth-password)" -v -H "Connection: keep-alive" "http://localhost:8080/system/logs?follow=true&name=nodeinfo&tail=-1"

[alexellis]

I think this can be closed now. @fundef1 please can you retest? I'll reopen the issue if it's still a problem in the latest version