GithubHelp home page GithubHelp logo

rodrigogs / mysql-events Goto Github PK

View Code? Open in Web Editor NEW

This project forked from involvestecnologia/mysql-events

136.0 12.0 52.0 246 KB

A node package that watches a MySQL database and runs callbacks on matched events.

License: BSD 3-Clause "New" or "Revised" License

JavaScript 97.33% Shell 2.67%
mysql database reactive nodejs binlog events event-sourcing

mysql-events's Introduction

mysql-events

CircleCI Code Climate Test Coverage

A node.js package that watches a MySQL database and runs callbacks on matched events.

This package is based on the original ZongJi and the original mysql-events modules. Please make sure that you meet the requirements described at ZongJi, like MySQL binlog etc.

Check @kuroski's mysql-events-ui for a mysql-events UI implementation.

Install

npm install @rodrigogs/mysql-events

Quick Start

const mysql = require('mysql');
const MySQLEvents = require('@rodrigogs/mysql-events');

const program = async () => {
  const connection = mysql.createConnection({
    host: 'localhost',
    user: 'root',
    password: 'root',
  });

  const instance = new MySQLEvents(connection, {
    startAtEnd: true,
    excludedSchemas: {
      mysql: true,
    },
  });

  await instance.start();

  instance.addTrigger({
    name: 'TEST',
    expression: '*',
    statement: MySQLEvents.STATEMENTS.ALL,
    onEvent: (event) => { // You will receive the events here
      console.log(event);
    },
  });
  
  instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, console.error);
  instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);
};

program()
  .then(() => console.log('Waiting for database events...'))
  .catch(console.error);

Check the examples

Usage

#constructor(connection, options)

  • Instantiate and create a database connection using a DSN

    const dsn = {
      host: 'localhost',
      user: 'username',
      password: 'password',
    };
    
    const myInstance = new MySQLEvents(dsn, { /* ZongJi options */ });
  • Instantiate and create a database connection using a preexisting connection

    const connection = mysql.createConnection({
      host: 'localhost',
      user: 'username',
      password: 'password',
    });
    
    const myInstance = new MySQLEvents(connection, { /* ZongJi options */ });
  • Options(the second argument) is for ZongJi options

    const myInstance = new MySQLEvents({ /* connection */ }, {
      serverId: 3,
      startAtEnd: true,
    });

    See more about ZongJi options

#start()

  • start function ensures that MySQL is connected and ZongJi is running before resolving its promise
    myInstance.start()
      .then(() => console.log('I\'m running!'))
      .catch(err => console.error('Something bad happened', err));

#stop()

  • stop function terminates MySQL connection and stops ZongJi before resolving its promise
    myInstance.stop()
      .then(() => console.log('I\'m stopped!'))
      .catch(err => console.error('Something bad happened', err));

#pause()

  • pause function pauses MySQL connection until #resume() is called, this it useful when you're receiving more data than you can handle at the time
    myInstance.pause();

#resume()

  • resume function resumes a paused MySQL connection, so it starts to generate binlog events again
    myInstance.resume();

#addTrigger({ name, expression, statement, onEvent })

  • Adds a trigger for the given expression/statement and calls the onEvent function when the event happens
    instance.addTrigger({
      name: 'MY_TRIGGER',
      expression: 'MY_SCHEMA.MY_TABLE.MY_COLUMN',
      statement: MySQLEvents.STATEMENTS.INSERT,
      onEvent: async (event) => {
        // Here you will get the events for the given expression/statement.
        // This could be an async function.
        await doSomething(event);
      },
    });
  • The name argument must be unique for each expression/statement, it will be user later if you want to remove a trigger
    instance.addTrigger({
      name: 'MY_TRIGGER',
      expression: 'MY_SCHEMA.*',
      statement: MySQLEvents.STATEMENTS.ALL,
      ...
    });
    
    instance.removeTrigger({
      name: 'MY_TRIGGER',
      expression: 'MY_SCHEMA.*',
      statement: MySQLEvents.STATEMENTS.ALL,
    });
  • The expression argument is very dynamic, you can replace any step by * to make it wait for any schema, table or column events
    instance.addTrigger({
      name: 'Name updates from table USERS at SCHEMA2',
      expression: 'SCHEMA2.USERS.name',
      ...
    });
    instance.addTrigger({
      name: 'All database events',
      expression: '*',
      ...
    });
    instance.addTrigger({
      name: 'All events from SCHEMA2',
      expression: 'SCHEMA2.*',
      ...
    });
    instance.addTrigger({
      name: 'All database events for table USERS',
      expression: '*.USERS',
      ...
    });
  • The statement argument indicates in which database operation an event should be triggered
    instance.addTrigger({
      ...
      statement: MySQLEvents.STATEMENTS.ALL,
      ...
    });
    Allowed statements
  • The onEvent argument is a function where the trigger events should be threated
    instance.addTrigger({
      ...
      onEvent: (event) => {
        console.log(event); // { type, schema, table, affectedRows: [], affectedColumns: [], timestamp, }
      },
      ...
    });

#removeTrigger({ name, expression, statement })

  • Removes a trigger from the current instance
    instance.removeTrigger({
      name: 'My previous created trigger',
      expression: '',
      statement: MySQLEvents.STATEMENTS.INSERT,
    });

Instance events

  • MySQLEvents class emits some events related to its MySQL connection and ZongJi instance
    instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, (err) => console.log('Connection error', err));
    instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, (err) => console.log('ZongJi error', err));

Available events

Tigger event object

It has the following structure:

{
  type: 'INSERT | UPDATE | DELETE',
  schema: 'SCHEMA_NAME',
  table: 'TABLE_NAME',
  affectedRows: [{
    before: {
      column1: 'A',
      column2: 'B',
      column3: 'C',
      ...
    },
    after: {
      column1: 'D',
      column2: 'E',
      column3: 'F',
      ...
    },
  }],
  affectedColumns: [
    'column1',
    'column2',
    'column3',
  ],
  timestamp: 1530645380029,
  nextPosition: 1343,
  binlogName: 'bin.001',
}

Make sure the database user has the privilege to read the binlog on database that you want to watch on.

LICENSE

BSD-3-Clause © Rodrigo Gomes da Silva

mysql-events's People

Contributors

datatraders avatar jfbueno avatar jonathanargentiero avatar pschuster avatar rodrigogs avatar silenzium avatar sirhanshafahath avatar spencerlambert avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mysql-events's Issues

support mariadb

I'm wondering if this package supports mariadb as well or not. Any idea?

why binary logs are required?

I tested this library with success in my mysql local installation and worked out of the box.
But when I deploy to server prod, it raises and error about:
code: ‘ER_NO_BINARY_LOGGING’, errno: 1381, sqlMessage: ‘You are not using binary logging’, sqlState: ‘HY000’, index: 0, sql: ‘SHOW BINARY LOGS’

we do not want to activate binary logging at server because it compromises performance.

TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: utf8mb3

TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: utf8mb3
at stringSlice (buffer.js:630:9)
at Uint8Array.toString (buffer.js:666:10)
at Parser.parseString (/var/www/api/kameazy/realtime/node_modules/mysql/lib/protocol/Parser.js:281:28)
at Object.exports.readMysqlValue (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/common.js:446:25)
at readRow (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/rows_event.js:111:16)
at UpdateRows._fetchOneRow (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/rows_event.js:143:13)
at UpdateRows.RowsEvent (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/rows_event.js:62:27)
at new UpdateRows (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/rows_event.js:135:13)
at BinlogHeader.parse (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/lib/packet/binlog_header.js:47:23)
at Protocol._parsePacket (/var/www/api/kameazy/realtime/node_modules/mysql/lib/protocol/Protocol.js:272:12)
{ Error: Packets out of order. Got: 178 Expected: 7
at Parser._tryReadPacketHeader (/var/www/api/kameazy/realtime/node_modules/mysql/lib/protocol/Parser.js:470:15)
at Parser.write (/var/www/api/kameazy/realtime/node_modules/mysql/lib/protocol/Parser.js:33:29)
at process._tickCallback (internal/process/next_tick.js:61:11)
--------------------
at Protocol._enqueue (/var/www/api/kameazy/realtime/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at Immediate._start (/var/www/api/kameazy/realtime/node_modules/@rodrigogs/zongji/index.js:240:31)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5) code: 'PROTOCOL_PACKETS_OUT_OF_ORDER', fatal: true }

Problem event update or delete multiple effected rows

I've a problem.

Sorry for my bad english

When I was testing with one row update/delete, event trigger worked properly, but when with multiple rows update/delete, event trigger only running for couple rows not all, for example
UPDATE employee SET emp_status = 'N' WHERE emp_id = 1 --> working well.
UPDATE employee SET emp_status ='N' --> not working well, event trigger not fired for all rows, only couple rows (random).

Could you help me for this situation ? Thanks a lot.

Column names are changed

Great package! Thanks Rodrigo!

Maybe I am doing something wrong... but I get the columns names changed.
It's not aleatory, so when I want to use Id I know I can use ev.affectedRows[0].after.addition

But It would be nice to use ev.affectedRows[0].after.id
My table has these columns:

'id'
'alert_begin'
'alert_sent'
'alert_answered'
'text'
'addition'
'updated'
'alert_type_id'
'call_rule_id'
'user_attended_id'

And when I print the results are changed:

console.log("Id:" + ev.affectedRows[0].after.addition); 

I get the Id printing ev.affectedRows[0].after.addition and so on ...

console.log("updated:" + ev.affectedRows[0].after.id);
console.log("alert_type_id:" + ev.affectedRows[0].after.text); 
console.log("alert_begin:" + ev.affectedRows[0].after.alert_answered);
console.log("alert_answered:" + ev.affectedRows[0].after.alert_begin); /
console.log("alert_sent:" + ev.affectedRows[0].after.alert_sent); 
console.log("text:" + ev.affectedRows[0].after.alert_type_id);
console.log("addition:" + ev.affectedRows[0].after.call_rule_id);
console.log("call_rule_id:" + ev.affectedRows[0].after.updated); 

Only this field is the same
console.log("user_attended_id:" + ev.affectedRows[0].after.user_attended_id);

console.log("AF:" + ev.affectedColumns); - This is also changed

MySQL 5.1

Hello Friend,
I saw in the README that you tested from the most recent version up to version 5.5 on MySQL.

I am trying to implement on a MySQL 5.1 but it really is not working.
Do you know any solution for this?

The connection is made normally, but the triggers do not work.

Grateful for the attention.

Custom ports wont work.

const mysql = require('mysql');
const MySQLEvents = require('@rodrigogs/mysql-events');

const program = async () => {
  const connection = mysql.createConnection({
    host: 'localhost',
    user: 'root',
    password: 'root',
    port: 1337
  });

  const instance = new MySQLEvents(connection, {
    startAtEnd: true,
    excludedSchemas: {
      mysql: true,
    },
    port: '1337'
  });

  await instance.start();

  /*instance.addTrigger({
    name: 'TEST',
    expression: '*',
    statement: MySQLEvents.STATEMENTS.ALL,
    onEvent: (event) => { // You will receive the events here
      console.log(event);
    },
  });
  
  instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, console.error);
  instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);
  */
};

program()
  .then(() => console.log('Waiting for database events...'))
  .catch(console.error);

Produces

Error: connect ECONNREFUSED 127.0.0.1:3306 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16) -------------------- at Protocol._enqueue (C:\Users\admin\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\protocol\Protocol.js:144:48) at Protocol.handshake (C:\Users\admin\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\protocol\Protocol.js:51:23) at Connection.connect (C:\Users\admin\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\Connection.js:116:18) at C:\Users\admin\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:6:75 at new Promise (<anonymous>) at connect (C:\Users\admin\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:6:31) at connectionHandler (C:\Users\admin\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:42:11) at MySQLEvents.start (C:\Users\admin\node_modules\@rodrigogs\mysql-events\lib\MySQLEvents.js:88:29) at program (C:\Users\admin\Documents\GitHub\mhkdbmirror\test.js:21:18) at Object.<anonymous> (C:\Users\admin\Documents\GitHub\mhkdbmirror\test.js:37:1) { errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 3306, fatal: true }

It tries to connect to the Port 3306, even though I said that it should use port 1337.
Have found no documentation here or at ZongJi, I am slowly starting to lose my mind :D
(1337 as a string or integer does not work either, I tried everything thats possible. Entering the port parameter into the new MySQLEvents which is passed to ZongJi does not work either.)

Doesn't work for Sequel Pro changes?

It works great for ORM changes and direct queries, but doesn't seem to trigger for changes made directly using database mangers like Sequel Pro. Any thoughts?

Sails

have you try mysql-events with sails i cant make this work this is how my mysqlEvents object looks like

image

and my log file with docker is this
sudo docker exec dockercontainers_db_1 cat /var/log/mysql/mysql-bin.000002

image

Error: A slave with the same server_uuid/server_id as this slave has connected to the master;

Hi @rodrigogs,

Has been working great!
But looking at the log error I saw a new one:

'A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000089' at 43535345, the last event read from '/mysql/binlog/mysql-bin.000089' at 56994447, the last byte read from '/mysql/binlog/mysql-bin.000089' at 56994447.',
sqlState: 'HY000' }

I have 2 databases (Production, Development )
And have 2 listeners that gives this error

My Code Dev

const mysql = require('mysql');
var MySQLEvents = require("@rodrigogs/mysql-events");
 const schemaName ='nms';

My Code Prod

const mysql = require('mysql');
var MySQLEvents = require("@rodrigogs/mysql-events");
 const schemaName ='nmsProd';
Same code for both
const db_config = { host: '35.234.131.94', user:XXX', password: XXX', database: schemaName, };

let connectDB = mysql.createConnection(db_config);
let instance = new MySQLEvents(connectDB, {
    startAtEnd: true,
    excludedSchemas: {
        mysql: true
    },
});
startTrigers();

function handleDisconnect() {
    console.log('handleDisconnect');
    connectDB.destroy();
    connectDB = mysql.createConnection(db_config);
    connectDB.connect(function (err) {
        if (err) {
            console.log(' Error when connecting to db  (DBERR001):', err);
            setTimeout(handleDisconnect, 1000);
        }
        else{
            // startTrigers();
        }
    });
}


instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, (err) => {
    console.log('Connection is asleep (time to wake it up): ', err);
    setTimeout(handleDisconnect, 1000);
    handleDisconnect();
});

instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);

function startTrigers() {

    instance.start();

    instance.addTrigger({
        name: 'ALERT_INSERTED',
        expression: schemaName + '.nms_alert',
        statement: MySQLEvents.STATEMENTS.INSERT,
        onEvent: (event) => {
            console.log("", new Date());
            console.log("==== ALERT INSERTED  ===");
            sendAlert(event);
        },
    });

Do I have to change something?

Unable to start from last position

I'm using this module and I have to say that so far it worked like a charm.
The only issue I have is when trying to restart from last known position:

const mysqlInstance = new MySQLEvents(connection, { binlogName: 'bin.000015', nextPosition: 18540004 });

This simply start from the beginning. 'bin.000001'. It's completely ignored.

MySQL 8.0.23 Support

I have tried the package with MySQL 8.0.23 and its not working properly, like not showing the detailed events as its supposed or not showing the events at all even if returned a successfull message saying that its connected.

¿Any thoughts regarding of this mysql version?

Monitoring type: ALTER, CREATE

hi its possible for monitoring DDL Data Definition Language

CREATE - to create a database and its objects like (table, index, views, store procedure, function, and triggers)
ALTER - alters the structure of the existing database
DROP - delete objects from the database
TRUNCATE - remove all records from a table, including all spaces allocated for the records are removed
COMMENT - add comments to the data dictionary
RENAME - rename an object

Sequelize support

var Sequelize = require('sequelize');
var sequelize = new Sequelize(config.database, config.username, config.password, config);

Does it work with an ORM like sequelize? What would be the syntax?

Does not show any event... No error appears.

Hello, first of all, thank you for making the code available.
I tried, I've already tested both on localhost and directly on the almalinux server... It doesn't give any errors, but no event appears either. No matter how much I touch the database... Nothing happens. It doesn't come out of this, as shown in the attached print.

What can it be?

Server MySql:
Server: Localhost via UNIX socket
Server type: MySQL
Server connection: SSL is not being used Documentation
Server version: 5.7.44-log - MySQL Community Server (GPL)
Protocol version: 10
User: netsexyrede@localhost
Server charset: cp1252 West European (latin1)

MySql from LocalHost (my pc):
Server: 127.0.0.1 via TCP/IP
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.4.32-MariaDB-log - mariadb.org binary distribution
Protocol version: 10
User: root@localhost
Server charset: UTF-8 Unicode (utf8mb4)

see attached
mysl-events

onEvent argument should be a function

I install your plugin and use your example, but show me error:

Error: onEvent argument should be a function
    at MySQLEvents.addTrigger (C:\wamp64\www\upport54\services\call_popup\node_modules\@rodrigogs\mysql-events\lib\MySQLEvents.js:166:46)
    at program (C:\wamp64\www\upport54\services\call_popup\file.js:14:14)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Error specific expression

hi rodrigo,

I have been trying to monitor only one table using expression, but it seems it doesn't work, from the log I can see that it is monitoring all the tables.
Could you please check?

Thanks

ER_SPECIFIC_ACCESS_DENIED_ERROR but does have all the privileges needed

My user definitely has the REPLICATION CLIENT privilege (as well as REPLICATION SLAVE) but I still get the error below. I have even tried the script with the SUPER privilege. When I read the binlog with only using ZongJi I get no errors and can read the binlog.

code: 'ER_SPECIFIC_ACCESS_DENIED_ERROR',
  errno: 1227,
  sqlMessage:
   'Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation',
  sqlState: '42000',
  index: 0,
  sql: 'SHOW BINARY LOGS'

Connection error when using mysql-events

try using example from readme.

const mysql = require('mysql');
const MySQLEvents = require('@rodrigogs/mysql-events');

const program = async () => {
  const connection = mysql.createConnection({
    host: 'localhost',
    user: 'root',
    password: 'root',
  });

  const instance = new MySQLEvents(connection, {
    startAtEnd: true,
    excludedSchemas: {
      mysql: true,
    },
  });

  await instance.start();

  instance.addTrigger({
    name: 'TEST',
    expression: '*',
    statement: MySQLEvents.STATEMENTS.ALL,
    onEvent: (event) => { // You will receive the events here
      console.log(event);
    },
  });
  
  instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, console.error);
  instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);
};

program()
  .then(() => console.log('Waiting for database events...'))
  .catch(console.error);

so get error

 Error: connect ETIMEDOUT
     at Connection._handleConnectTimeout (/var/www/html/node_modules/mysql/lib/Connection.js:409:13)
     at Object.onceWrapper (node:events:627:28)
     at Socket.emit (node:events:513:28)
     at Socket._onTimeout (node:net:553:8)
     at listOnTimeout (node:internal/timers:564:17)
     at process.processTimers (node:internal/timers:507:7)
     --------------------
     at Protocol._enqueue (/var/www/html/node_modules/mysql/lib/protocol/Protocol.js:144:48)
     at Protocol.handshake (/var/www/html/node_modules/mysql/lib/protocol/Protocol.js:51:23)
     at Connection.connect (/var/www/html/node_modules/mysql/lib/Connection.js:116:18)
     at /var/www/html/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:6:75
     at new Promise (<anonymous>)
     at connect (/var/www/html/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:6:31)
     at connectionHandler (/var/www/html/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:42:11)
     at MySQLEvents.start (/var/www/html/node_modules/@rodrigogs/mysql-events/lib/MySQLEvents.js:88:29)
     at program (/var/www/html/index.js:12:18)
     at Object.<anonymous> (/var/www/html/index.js:27:1) {
   errorno: 'ETIMEDOUT',
   code: 'ETIMEDOUT',
   syscall: 'connect',
   fatal: true
 }

t

Start with a position and real time updates

When use with startAtEnd:true This fires events near realtime. But when if the application crashed or turned off we can not get the events happen between downtime.

If I make startAtEnd:false events doesn't get fired real time it takes very noticeable delay to events fire and get some very old event. can we use position and startAtEnd to have both these cases satisfied?

Connection not being recognised

Seeing an intermittent error.

Passing a Mysql Lib (same version) Connection into EventHandler but when I start all the config has been dropped and it's trying to connect to localhost.

    this.mysqlEvents = mysqlEvents;
    this.mysqlconnection = mysql.createConnection({
      host: connection.host,
      user: connection.user,
      password: connection.password,
    });

    this.eventWatcher = new this.mysqlEvents(this.mysqlconnection, {
      startAtEnd: true,
      excludedSchemas: {
        mysql: true,
      },
    })
    this.eventWatcher.start();

Connection Error

2019-08-14T14:21:21.188574900Z 
(node:25) UnhandledPromiseRejectionWarning: Error: connect ECONNREFUSED 127.0.0.1:3306
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
    --------------------
    at Protocol._enqueue (/src/node_modules/@rodrigogs/mysql-events/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/src/node_modules/@rodrigogs/mysql-events/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/src/node_modules/@rodrigogs/mysql-events/node_modules/mysql/lib/Connection.js:119:18)
    at Promise (/src/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:6:75)
    at new Promise (<anonymous>)
    at connect (/src/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:6:31)
    at connectionHandler (/src/node_modules/@rodrigogs/mysql-events/lib/connectionHandler.js:42:11)
    at MySQLEvents.start (/src/node_modules/@rodrigogs/mysql-events/lib/MySQLEvents.js:88:29)

This is because it's not recognizing the Connection as an instance of a connection but instead just as an object so the config is being defaulted again

Now weirdly when it prints out the connection as an object it looks like this

creating connection from object: Connection {
  _events: [Object: null prototype] {},
  _eventsCount: 0,
  _maxListeners: undefined,
  config:
   ConnectionConfig {
     host: 'redacted',
     port: 3306,
     localAddress: undefined,
     socketPath: undefined,
     user: 'root',
     password: 'ourrootpassword',
     database: undefined,
     connectTimeout: 10000,
     insecureAuth: false,
     supportBigNumbers: false,
     bigNumberStrings: false,
     dateStrings: false,
     debug: undefined,
     trace: true,
     stringifyObjects: false,
     timezone: 'local',
     flags: '',
     queryFormat: undefined,
     pool: undefined,
     ssl: false,
     multipleStatements: false,
     typeCast: true,
     maxPacketSize: 0,
     charsetNumber: 33,
     clientFlags: 455631 },
  _socket: undefined,
  _protocol:
   Protocol {
     _events: [Object: null prototype] {},
     _eventsCount: 0,
     _maxListeners: undefined,
     readable: true,
     writable: true,
     _config:
      ConnectionConfig {
        host: 'redacted',
        port: 3306,
        localAddress: undefined,
        socketPath: undefined,
        user: 'root',
        password: 'ourrootpassword',
        database: undefined,
        connectTimeout: 10000,
        insecureAuth: false,
        supportBigNumbers: false,
        bigNumberStrings: false,
        dateStrings: false,
        debug: undefined,
        trace: true,
        stringifyObjects: false,
        timezone: 'local',
        flags: '',
        queryFormat: undefined,
        pool: undefined,
        ssl: false,
        multipleStatements: false,
        typeCast: true,
        maxPacketSize: 0,
        charsetNumber: 33,
        clientFlags: 455631 },
     _connection: [Circular],
     _callback: null,
     _fatalError: null,
     _quitSequence: null,
     _handshake: false,
     _handshaked: false,
     _ended: false,
     _destroyed: false,
     _queue: [],
     _handshakeInitializationPacket: null,
     _parser:
      Parser {
        _supportBigNumbers: false,
        _buffer: <Buffer >,
        _nextBuffers: [BufferList],
        _longPacketBuffers: [BufferList],
        _offset: 0,
        _packetEnd: null,
        _packetHeader: null,
        _packetOffset: null,
        _onError: [Function: bound handleParserError],
        _onPacket: [Function: bound ],
        _nextPacketNumber: 0,
        _encoding: 'utf-8',
        _paused: false } },
  _connectCalled: false,
  state: 'disconnected',
  threadId: null }

So not sure why it doesn't think it's an instance of connection. I had thought initially mysql lib versions meaning different interface but seem to both be the same "mysql": "^2.17.1",

In my code if I try to the same instanceof Connection check it passes but in the mysqlevents it does not.

Apologies if already seen.

Thanks for the library as well it's been great. I have a fix for this but not sure it's ideal. Will send a PR in later. Been unable to willfully recreate this as well but it has happened a few times. It's very odd.

PROTOCOL_CONNECTION_LOST

Hi @rodrigogs

After around 2 days it loses the connection with the server.

Suggestion:
pooled MySQL connections like here
https://fearby.com/article/how-to-setup-pooled-mysql-connections-in-node-js-that-dont-disconnect/

Or Change to mysql2 ?

(I am using pm2 to manage the script)

All the error detail:
{ Error: Cannot enqueue Query after fatal error.
at Protocol._validateEnqueue (/home/timmeh55/nmsSocket/node_modules/mysql/lib/protocol/Protocol.js:200:16)
at Protocol._enqueue (/home/timmeh55/nmsSocket/node_modules/mysql/lib/protocol/Protocol.js:138:13)
at Connection.query (/.../nmsSocket/node_modules/mysql/lib/Connection.js:200:25)
at ZongJi._fetchTableInfo (/home/timmeh55/nmsSocket/node_modules/@rodrigogs/zongji/index.js:203:23)
at Binlog._callback (/../nmsSocket/node_modules/@rodrigogs/zongji/index.js:251:18)
at Binlog.BinlogHeader (/home/timmeh55/nmsSocket/node_modules/@rodrigogs/zongji/lib/sequence/binlog.js:43:22)
at Protocol._parsePacket (/home/timmeh55/nmsSocket/node_modules/mysql/lib/protocol/Protocol.js:278:23)
at Parser.write (/.../nmsSocket/node_modules/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/.../nmsSocket/node_modules/mysql/lib/protocol/Protocol.js:38:16)
at Socket. (/.../nmsSocket/node_modules/mysql/lib/Connection.js:91:28) code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR', fatal: false }
{ Error: Connection lost: The server closed the connection.
at Protocol.end (/home/timmeh55/nmsSocket/node_modules/mysql/lib/protocol/Protocol.js:112:13)
at Socket. (/.../nmsSocket/node_modules/mysql/lib/Connection.js:97:28)
at Socket. (/.../nmsSocket/node_modules/mysql/lib/Connection.js:502:10)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1094:12)
at args.(anonymous function) (/usr/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)
at process._tickCallback (internal/process/next_tick.js:63:19) fatal: true, code: 'PROTOCOL_CONNECTION_LOST' }

No DELETE events on mariadb:10.7

I'm testing this module non mariadb server on Docker. Binlog enabled via command line such as

command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --log-bin

It apparently works flawlessly with UPDATE statements, but DELETE is completely discarded.

Any ideas?

Is there any way to get the raw SQl statement that was executed during an event?

I know that there's a new setting available as of MySQL 5.6.2 called binlog_rows_query_log_events which logs the raw SQL to the binlog.

Is there a way to get that value?

The reason why I need this is because the existing package lets me know that some event occurred. Such as INSERT, UPDATE or DELETE. However, it doesn't tell me what the 'WHERE' condition that was used in cases of UPDATE or DELETE.

It'd be good to get the raw sql in that case to know exactly which query was executed.

old row new row, urgent

i would like to get old row and new row like https://www.npmjs.com/package/mysql-events. right now all i got is IntVar {
timestamp: 1604743109000,
nextPosition: 4795,
size: 9,
type: 2,
value: 17,
binlogName: 'mysql-bin.000001'
} basically i want more detail. can you help me ?
and it doesnt detect update too. here is my code
const mysql = require('mysql');
const MySQLEvents = require('@rodrigogs/mysql-events');

const program = async () => {
const connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: '',
database: 'sociominer'
});

const instance = new MySQLEvents(connection, {
startAtEnd: true,
excludedSchemas: {
mysql: true,
},
});

await instance.start();

instance.addTrigger({
name: 'TEST',
expression: '*',
statement: MySQLEvents.STATEMENTS.ALL,
onEvent: (event) => {
console.log(event); // { type, schema, table, affectedRows: [], affectedColumns: [], timestamp, }
},
});

instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, console.error);
instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);
};

program()
.then(() => console.log('Waiting for database events...'))
.catch(console.error);

and db
image

i just want to see if there are insert update delete event from test table ( only test table ) and coresponding old row and new row
and what about if i want to listen specificly to certain table only cause currently i listen to al table. already try expression: '*.test', but no event comming instead ( no event )
note : i never and dont want to use schema if possible
my aim is really similar to this https://github.com/kuroski/mysql-events-ui/blob/master/server.js except i need to know which table and row inserted and updated so i know which client shoud i emitted to. especially on insert and update

Support for MariaDB

Hello everybody.

Did anybody manage to get this software going with MariaDB?

RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to access memory outside buffer bounds

Hi,

When i start my application, and i do a UPDATE from my database, i get this error:

RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to access memory outside buffer b                                                                                                                                                             ounds
    at new NodeError (node:internal/errors:329:5)
    at boundsError (node:internal/buffer:84:11)
    at Buffer.readUInt8 (node:internal/buffer:252:5)
    at parseBinaryBuffer (/projects/whatsapp/complimaatje/node_modules/@rodrigog                                                                                                                                                             s/zongji/lib/json_decode.js:55:24)
    at module.exports (/projects/whatsapp/complimaatje/node_modules/@rodrigogs/z                                                                                                                                                             ongji/lib/json_decode.js:28:25)
    at Object.exports.readMysqlValue (/projects/whatsapp/complimaatje/node_modul                                                                                                                                                             es/@rodrigogs/zongji/lib/common.js:468:16)
    at readRow (/projects/whatsapp/complimaatje/node_modules/@rodrigogs/zongji/l                                                                                                                                                             ib/rows_event.js:111:16)
    at UpdateRows._fetchOneRow (/projects/whatsapp/complimaatje/node_modules/@ro                                                                                                                                                             drigogs/zongji/lib/rows_event.js:143:13)
    at UpdateRows.RowsEvent (/projects/whatsapp/complimaatje/node_modules/@rodri                                                                                                                                                             gogs/zongji/lib/rows_event.js:62:27)
    at new UpdateRows (/projects/whatsapp/complimaatje/node_modules/@rodrigogs/z                                                                                                                                                             ongji/lib/rows_event.js:135:13) {
  code: 'ERR_BUFFER_OUT_OF_BOUNDS'
}
Error: Packets out of order. Got: 28 Expected: 7
    at Parser._tryReadPacketHeader (/projects/whatsapp/complimaatje/node_modules                                                                                                                                                             /mysql/lib/protocol/Parser.js:470:15)
    at Parser.write (/projects/whatsapp/complimaatje/node_modules/mysql/lib/prot                                                                                                                                                             ocol/Parser.js:33:29)
    at processTicksAndRejections (node:internal/process/task_queues:76:11)
    --------------------
    at Protocol._enqueue (/projects/whatsapp/complimaatje/node_modules/mysql/lib                                                                                                                                                             /protocol/Protocol.js:144:48)
    at Immediate._start (/projects/whatsapp/complimaatje/node_modules/@rodrigogs                                                                                                                                                             /zongji/index.js:240:31)
    at processImmediate (node:internal/timers:464:21) {
  code: 'PROTOCOL_PACKETS_OUT_OF_ORDER',
  fatal: true
}

It happends sometimes, but not always. How can i fix this?

Bulk create support

I am not getting any event for the following query.

INSERT INTO `ticks` (`instrument_token`,`last_price`,`open`,`high`,`low`,`close`,`change`,`volume_traded`,`createdAt`,`updatedAt`) VALUES (1071105,242.4,242,244.1,237.55,240.45,0.8109794135995081,9931,'2022-12-28 13:17:25','2022-12-28 13:17:25'),(940033,431.55,432.9,438.95,430.1,430.05,0.34879665155214507,67414,'2022-12-28 13:17:25','2022-12-28 13:17:25') ON DUPLICATE KEY UPDATE `instrument_token`=VALUES(`instrument_token`),`last_price`=VALUES(`last_price`),`open`=VALUES(`open`),`high`=VALUES(`high`),`low`=VALUES(`low`),`close`=VALUES(`close`),`change`=VALUES(`change`),`volume_traded`=VALUES(`volume_traded`);

AWS MySql RSD not working

Hi

I'm trying to monitor my MySQL rds DB. I get the log message "Waiting for database vents" bu nothing happens when I change values in the DB tables

I'm using this example code
`

const MySQLEvents = require('@rodrigogs/mysql-events');

const program = async () => {
    const instance = new MySQLEvents({
        host: DB_HOST,
        user: DB_USER,
        password: DB_PASSWORD,
     }, {
         startAtEnd: true,
    });

    await instance.start();

    instance.addTrigger({
        name: 'Whole database instance',
        expression: '*',
        statement: MySQLEvents.STATEMENTS.ALL,
        onEvent: (event) => {
            console.log(event);
        },
    });

     instance.on(MySQLEvents.EVENTS.CONNECTION_ERROR, console.error);
     instance.on(MySQLEvents.EVENTS.ZONGJI_ERROR, console.error);
};

program()
    .then(() => console.log('Waiting for database vents...'))
    .catch(console.error);

`

Any ideas?

Can't connect to any database.

I tried to use my credentials and even on the runkit It gives the same error:

const connection = mysql.createConnection({
   host:     "secret.rds.amazonaws.com",
   user:     "secret",
   password: "secret",
   port: 3306
 })
{ Error: connect ECONNREFUSED 127.0.0.1:3306
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
    --------------------
    at Protocol._enqueue (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\protocol\Protocol.js:144:48)
    at Protocol.handshake (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\protocol\Protocol.js:51:23)
    at Connection.connect (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\node_modules\mysql\lib\Connection.js:118:18)
    at Promise (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:5:75)
    at new Promise (<anonymous>)
    at connect (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:5:31)
    at connectionHandler (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\lib\connectionHandler.js:30:11)
    at MySQLEvents.start (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\node_modules\@rodrigogs\mysql-events\lib\MySQLEvents.js:88:29)
    at program (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\src\events\message.js:35:18)
    at Object.<anonymous> (C:\Users\Nathan\Desktop\Rocket-Elevators-AI\src\events\message.js:50:1)
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 3306,
  fatal: true }

For some reason It keeps using localhost.

TypeError [ERR_UNKNOWN_ENCODING]

TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: utf32
at stringSlice (buffer.js:631:9)
at Uint8Array.toString (buffer.js:667:10)

Reconnection after connection lost

Hi, congratulations for this module, I use it in production on my platform.

I use pm2 as a process manager.

Sometimes I get the following error:

{ Error: Connection lost: The server closed the connection.
    at Protocol.end (/home/imroot/node_modules/mysql/lib/protocol/Protocol.js:112:13)
    at Socket.<anonymous> (/home/imroot/node_modules/mysql/lib/Connection.js:94:28)
    at Socket.<anonymous> (/home/imroot/node_modules/mysql/lib/Connection.js:526:10)
    at Socket.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19) fatal: true, code: 'PROTOCOL_CONNECTION_LOST' }

I think it's because my app occasionally disconnects from the database.

I found this and much more, but I'm not sure how to adapt it to your model with this different syntax, could you help me out? it could be useful for many.

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.