Comments (13)
For the record, I managed to have log insights working in a Kubernetes cluster, with what you suggested: Run the pganalyze-collector in a Docker container. Have the Postgres container write the log output into a file that sits on a shared volume that the pganalyze-collector container has access to. Then use the LOG_LOCATION setting to specify that log file.
All good, thanks !
from collector.
Is there any follow-up on this?
I still think that it would be very convenient to have a way for kubernetes-managed log collectors like fluentd or vector push logs into pganalyze (whether to the collector or directly to the centralized ingest endpoint).
from collector.
See #503 for an implementation that should work in the typical Kubernetes environment using fluentbit for log routing (tested with CloudNativePG, need to do a bit more testing with other operators).
from collector.
I have the same question: how does log collections work under Kubernetes ?
from collector.
@JordanP We don't support log collection in Kubernetes today - though looking to add this in the future.
For @sebasmagri's initial question - Sebastian, were you looking to monitor Kubernetes pods, or were you targeting something external, like a cloud provider database?
(generally log collection is enabled by default in recent collector releases - if supported for the current platform, that is)
from collector.
@lfittl it's for running the collector on ECS and monitoring RDS. We had to go with a custom docker image, but it's still pretty annoying that now the server names are forced to be the rds instance ID, since we create a lot of RDS instances dynamically and discard the old ones, being really a single server instance all the time.
from collector.
@sebasmagri Ah - makes sense. You can use the following settings to override the system identifiers:
PGA_API_SYSTEM_ID
PGA_API_SYSTEM_TYPE
PGA_API_SYSTEM_SCOPE
You can see the defaults that get assigned here: https://github.com/pganalyze/collector/blob/master/config/identify_system.go#L15 (note you could e.g. choose to override PGA_API_SYSTEM_ID in your setup to have a stable server record)
from collector.
Thanks for the pointer @lfittl !
We'd still like to be able to use the docker image as provided though, any chance that logging collection can be enabled by default for this use case?
from collector.
@sebasmagri In the case where you are not using the instance ID in the hostname, utilizing the AWS_INSTANCE_ID
variable should work (that, or the hostname, is what the collector checks against to determine which code path to take)
from collector.
@lfittl beside Kubernetes, does the Collector supports monitoring a PG instance running in a container ?
from collector.
@JordanP There are two ways to do that, assuming you are referring to Postgres running inside a Docker container on a VM that you manage:
-
Run the pganalyze-collector on the Docker host, and use the
db_log_docker_tail
setting - this will directly fetch the logs from the Docker engine. Note that this setting is mostly intended for our own development purposes, and is considered experimental. -
Run the pganalyze-collector in a Docker container. Have the Postgres container write the log output into a file that sits on a shared volume that the pganalyze-collector container has access to. Then use the
LOG_LOCATION
setting to specify that log file.
from collector.
Excellent ! Option 2 seems doable for us, nice !
from collector.
@lfittl How realistic would it be to allow for pganalyze-collector
to stream logs from a web socket? We're in a similar situation here, and our logs end up in Loki. We read these logs in other areas of our application via web socket.
from collector.
Related Issues (20)
- Unable to DB_USE_IAM_AUTH with aws RDS instance HOT 4
- Error calling pg_stat_statements_reset() as requested intermittently HOT 2
- Regular collector oom-kill at midnight UTC on Sundays HOT 1
- Publish Docker image to ECR public registry HOT 1
- Error to send snapshots HOT 2
- Upgrade go.uuid module HOT 3
- Missing image for v0.50.1 HOT 2
- Additional security settings for pganalyze collector HOT 3
- Add rudimentary test suite for Helm chart
- Debian build uses EOL version HOT 1
- Error: Failed to reload collector HOT 1
- OTEL connector - can it support Datadog APM traces for PostgreSQL HOT 2
- OTEL connector - can it support Datadog?
- Feature request: Add DB_PASSWORD_FILE or similar HOT 1
- Regarding SSO login HOT 1
- Configuring Log Insights for pganalyze runnning on kubernetes pod HOT 1
- Can't ctrl-c pganalyze-collector HOT 1
- panic: runtime error: index out of range [0] with length 0 HOT 5
- Inefficient usage of AWS DescribeDBInstances and DescribeDBClusters API HOT 1
- OTEL Collector: allow plain json payloads HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from collector.