serdnam / pino-cloudwatch-transport

@serdnam/pino-cloudwatch-transport

Pino v7+ transport for sending logs to AWS CloudWatch Logs, built using pino-abstract-transport and the AWS SDK for JavaScript v3.

Install

npm i @serdnam/pino-cloudwatch-transport

Configuration

This transport expects the following options:

export interface PinoCloudwatchTransportOptions { 
  logGroupName: string,
  logStreamName: string,
  awsRegion?: string,
  awsAccessKeyId?: string,
  awsSecretAccessKey?: string,
  interval?: number
}

You may optionally pass in the awsRegion, awsAccessKeyId and awsSecretAccessKey options, otherwise, the AWS SDK client will pick up those parameters from the environment (as described in the AWS SDK for Javascript documentation).

The transport, upon initialization, will create the LogGroup and LogStream with the names logGroupName and logStreamName, respectively, if they don't already exist.

After initializing the transport will start receiving logs from Pino formatting them in the format expected by CloudWatch:

{
    timestamp: 1645381110905
    message: '{"level":30,"time":1645381110905,"pid":320325,"hostname":"andres-latitudee7440","msg":"Hello, CloudWatch Logs!"}'
}

The transport will use the time property of the received log as the timestamp if it finds it, otherwise, it will use Date.now().

The transport will store received logs in a buffer, and will flush them out to CloudWatch using a PutLogEvents call when one of the following conditions is met:

• The buffer has reached the size limit described in the AWS CloudWatch documentation.

• The number of logs in the buffer has reached the size limit of 10,000 logs as described in the AWS CloudWatch documentation.

• The transport has just received a log, and the last time a log has been stored before this one was longer than interval miliseconds.

Usage

After installing, you can use the transport as follows:

import 'dotenv/config';
import pino from "pino";

const transport = pino.transport({
    target: '@serdnam/pino-cloudwatch-transport',
    options: {
        logGroupName: 'pino-cloudwatch-test',
        logStreamName: 'pino-cloudwatch-test-stream',
        awsRegion: process.env.AWS_REGION,
        awsAccessKeyId: process.env.AWS_ACCESS_KEY_ID,
        awsSecretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        interval: 1_000, // this is the default
    }
});

const logger = pino(transport);

logger.info('Hello, CloudWatch Logs!');

With Fastify:

import Fastify from 'fastify';
import { request } from 'undici';
import pino from 'pino';

const transport = pino.transport({
    target: '@serdnam/pino-cloudwatch-transport',
    options: {
        logGroupName: 'pino-cloudwatch-test',
        logStreamName: 'pino-cloudwatch-test-stream',
        awsRegion: process.env.AWS_REGION,
        awsAccessKeyId: process.env.AWS_ACCESS_KEY_ID,
        awsSecretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
    }
});

const logger = pino(transport);

const server = Fastify({ logger });

server.get('/', async function(req, reply){
    req.log.info('Hello, CloudWatch Logs!');
    return { message: 'OK' };
});

await server.listen(8888);

await request('http://localhost:8888/');

await server.close();

License

MIT

pino-cloudwatch-transport's Issues

Error: Cannot find module 'tslib' when omitting devDependencies

I'll try to revisit this in the coming days, but encountered this and lost an hour of my weekend :(

1. On a project with @serdnam/pino-cloudwatch-transport as a dependency, install only dependencies with NODE_ENV=production npm ci.
2. Run your fastify app, and observe the transport stream die and crash application

   Error: Cannot find module 'tslib' (followed by stack trace)
   Error: the worker has exited (followed by stack trace)

Can I use this as a stream?

We recently had a corporate proxy set up, so I had to make some modification to our app to support that, and I can no longer get this cloudwatch transport to work.

Error: unable to determine transport target for "@serdnam/pino-cloudwatch-transport"

I am also using pino-pretty for local logging, and I had to convert that to a stream for it to work correctly. Any ideas?

Does not work with timestamp formatting options

Pino allows to set the format of the timestamp by specifying e.g.

const logger = pino(
    {
        level: ...,
        timestamp: pino.stdTimeFunctions.isoTime, // <-- makes pino-cloudwatch-transport fail
    },
    {
        targets: [{
            target: '@serdnam/pino-cloudwatch-transport',
            options: { ... }
        }]
    }
);

However, it seems that pino-cloudwatch-transport assumes the time property to be a number, because it does not send any message to AWS Cloudwatch if the above time format property is present.

Does this only work with pino or also pino-http?

Hey thanks for the package. I've tried using it with pino-http but my logs don't seem to be getting transported to cloudwatch. Interestingly, the log group and log stream gets created by your package, but no logs get saved to that log stream...

Idea: option to disable log group/stream creation

We manage our AWS infra with AWS CDK as Infrastructure-as-Code (IaC), and prefer our CDK stack creates the log group and stream. Would love to see an option to disable auto-creation of these from this library.

It does't work inside AWS Nodejs lambda

Hi, we spent tons of time to make it work inside AWS lambda.
But with out success.

Can you jump into this case ?
AWS lambda Nodejs 14

Rotating stream name

CloudWatch is quite slow pulling up one big log stream. I'm interested in a rotating stream name the way Lambda does it. Because the stream name for pino-cloudwatch-transport is part of the constructor, I don't see a way to switch streams without restarting my service.

Is there a way to switch streams, or at least replace the transport without restarting?

Thanks!

No option to set retention policy

Please add an option to set rentention policy.