What is this? This module / helper is used for interfacing with mysql. It helps with common tasks like loading files to tables, manipulating tables, etc.
This library is promise based. Most or all commands will return a promise.
Npm install
$ npm install nodeetl-mysql
Require Library
const MySQL = require('nodeetl-mysql');
Instantiate a mysql instance by passing the connection string
let mysql = new MySQL('user:pass@mysqlurl:port/database');
(Optional) Enable debugging. You can pass winston to have messages log through your existing winston instance.
const logger = requrie('winston');
// ... later
if(process.env.LOGGING_LEVEL === 'debug') mysql.setDebug(logger);
SQL Functions
Table Functions
File Functions
When you create a new mysql
instance, a pool is automatically generated. You can manually acquire a connection do run multiple sql commands if needed.
Note This is primary used as an internal method. You usually won't need to use this.
Example
mysql.getConnection()
.then(function(connection){
connection.query('something');
})
Returns (Promise | Object)
A promise with the connection
from the pool. You can release the connection back to the pool using connection.release()
.
You can run a query through the mysql
instance. This will automatically acquire a connection from the pool, run the command and release the connection for you. Any valid query can be used.
Parameters
command
(string) - The sql command to run...variables
- Loose list of variables to pass into the sequal query. They replace any?
or??
in yourcommand
parameter. See node mysql docs for more.
Example
mysql.query(`CREATE TABLE copy LIKE original`)
.then(function(results){
console.log(results);
}).catch(function(err){
if(err) console.error(err);
})
Returns (Promise | Object)
A Promise with the results
object from MySQL
You can check if a table exists by passing it's name to this method.
Parameters
table
(sting) - name of table to look for
Examples
mysql.tableExists('test')
.then(function(){
// Table exists
})
.catch(function(){
// Table does not exist
})
/*
* You can also pass as fallback to first .then
*/
mysql.tableExists('test')
.then(
function(){
// Table exists
},
function(){
// Table does not exist
}
)
Returns (Promise)
A promise - resolving if table is found, rejecting if table is not found.
This is mostly an internal method used by the class when importing files. But you can use it to create a staging copy of an existing table.
Parameters
- table (string) - The table to make a copy of
Examples
mysql.createStagingTable('table')
.then(function(newTable){
console.log(newTable);
// 'table_staging'
})
Returns (Promise | String)
A promise with the name of the new table - always the name of the table passed + _staging
like "table_staging"
Parameters (Object)
table
(string) - The name of the table to createheaders
(array) - An array of the header names. All headers are automatically set to VARCHAR(10000).index
(string) - If you want to set an index on the table after it's created, pass the string of the header you want the index to be create for. If you want to useoverwrite
and notindex
, passnull
forindex
.prependHeaders
(bool | Defaultfalse
) - If the table name should be prepended to each field. Useful for merging tables together.overwrite
(bool | Defaultfalse
) - If an existing table already exists with thename
, you can overwrite with by passingtrue
.
Examples
mysql.createNewTable({table: 'newTable', headers: ['email', 'first', 'last']})
.then(function(newTable){
// returns the name of the new table if successful
})
Returns (Promise | string)
Returns a promise containing the name of the new table created
You can index a field or fields in a table using this method. It accepts one or more fields and indexes them on the table.
Parameters
table
(string | required) - name of table to add index to.index
(string or array | required) - Name of field or fields to index on the table.
Examples
mysql.addIndex('test', ['email', 'customerid'])
.then(function(){
// Indexes added
})
.catch(function(err){
// Error adding indexes
});
Returns (Promise)
A promise - resolving if indexes are successfully created, rejecting if error occurs.
Retrieve the fields on an existing table.
Parameters
table
(sting) - name of table to look for
Examples
mysql.tableExists('test')
.then(function(fields){
// array of fields
})
.catch(function(err){
// error
})
Returns (Promise | Array)
A promise - resolving with an array of fields, rejecting if error.
This is mostly an internal method used by the class for loading files, but can be invoked manually. This will find the matching production and staging version of the table
paramater, drop the production version and rename the staging version.
This is used to reduce the amount of time a table or rows in a table are locked during data load. For example, this will be called after a file is loaded into table_staging
. While the data is being loaded, table
and all its rows will still be available for queries. Once the load is complete, table
will be dropped and table_staging
will be renamed to table
.
In the event that table
doesn't exist and only table_staging
is found, then it simply renames table_staging
to table
.
Parameters
table
(string) - Name of table to swap/rename.
Examples
mysql.swaptables('table')
.then(function(){
// successfully swapped
})
.catch(function(err){
// failed to swap / rename or couldn't find table_staging
})
Returns (Promise)
Resolves if the swap occurs successfully, rejects if there was an issue or table_staging wasn't found.
You can check if a table exists by passing it's name to this method.
Parameters
table
(sting | array) - name of table(s) to look for. Can pass a single table as a string, or multipel tables as an array.
Examples
mysql.dropTable('test')
.then(function(){
// Table dropped
})
.catch(function(){
// Error dropping table
});
Returns (Promise)
A promise - resolving if table is or tables are dropped, rejecting if error occurs.
This is a utility for getting the headers of a text file
Parameters
filepath
(string) - Location of the filedelimiter
(optional | default "," | string) - Delimiter separating the fields.
Examples
mysql.getHeaders('./tmp/myfile.csv ')
.then(function(headers){
// headers array
})
/**
* Or you can pass a delimiter for non CSV files
*/
mysql.getHeaders('./tmp/myfile.tsv', '\t')
.then(function(headers){
// headers array
})
Returns (Promise Array)
A Promise passing an Array
of the headers from the file.
This method will import a file to an existing table. It's most useful when loading RDS data, where you know the structure of the table and the file will match one another and the table can be copied for import.
This function takes a table, makes a copy of it, loads all the file contents into it, and then swaps the copy with the live version of the table using mysql.swapTables
.
This uses the Class method createStagingTable and MySQL's LOAD DATA INFILE
command, which is documented here.
Parameters (object)
filepath
(string | required) - path to the file to loadtable
(string | optional) - name of the table to load the file into. Fallsback to the file name if not provided.headers
(array | optional) - If you only want to import certain fields from the file, you can pass the fields you want using this array.delimiter
(string | optional | default =","
) - The delimiter of the file you're importing. Defaults to comma.quotes
(string | optional | default =''
) - String enclosing each field of the file you're importing. Pass a single example of the character. For example, for quotes, passquotes: '"'
. Defaults to none.newline
(string | optional | default ="\n"
) - Line terminator in your file. Defaults to\n
.
Example
mysql.loadFileToTable({
filepath: './tmp/example.csv',
table: 'rds'
headers: ['customerid', 'session date', 'wishlist id'],
quotes: '"',
})
.then(function(rowsAffected){
// rowsAffected = count of records imported
})
Returns (Promise | Int)
Returns a promise containing the number of rows affected. This is the number of records imported.
This method will create a new table and import a file into that table. It's most useful when you want to use a file to define a table, and quickly import it's data. This function is used by the merge method to import each file into a table before merging.
This function can create a new table with headers you pass or by reading the headers in the file. It can also overwrite an existing table if one exists, but you must pass the overwrite
option as True.
If you're not going to use overwrite
but plan to run this function on a recurring basis, make sure you drop the table after you're done with it so the next occurrence doesn't throw an error.
This uses the Class Method createNewTable and MySQL's LOAD DATA INFILE
command, which is documented here.
Parameters (object)
filepath
(string | required) - path to the file to loadtable
(string | optional) - name of the table to load the file into. Fallsback to the file name if not provided.overwrite
(bool | default =false
) - If an existing table with thetable
name exists, overwrite it. Defaults to false, throwing an error if the table already exists.headers
(array | optional) - If you only want to import certain fields from the file, you can pass the fields you want using this array.delimiter
(string | optional | default =","
) - The delimiter of the file you're importing. Defaults to comma.quotes
(string | optional | default =''
) - String enclosing each field of the file you're importing. Pass a single example of the character. For example, for quotes, passquotes: '"'
. Defaults to none.newline
(string | optional | default ="\n"
) - Line terminator in your file. Defaults to\n
.
Example
mysql.importFileAndCreateTable({
filepath: './tmp/example.csv',
quotes: '"',
})
.then(function(rowsAffected){
// rowsAffected = count of records imported
})
/**
* Create a table with only 3 headers
* Only import the matching headers from the file
*/
mysql.importFileAndCreateTable({
filepath: './tmp/example.csv',
headers: ['contactId', 'firstname', 'lastname']
quotes: '"',
})
.then(function(rowsAffected){
// rowsAffected = count of records imported
})
Returns (Promise | Int)
Returns a promise containing the number of rows affected. This is the number of records imported.
This file will export all data from a table to a flat file. It's handy when you need to quickly extract data out of MySQL into a csv or other delimited file.
It uses the same options object as the file import methods, but the options are defining what the output file should look like.
NOTE - You'll need to pass __dirname
at the start of your filepath
to be extra specific as to where the file should be exported to. Since MySQL is running this command, you must tell it that you want the file on your server, not on the MySQL server.
This uses MySQL's SELECT INTO OUTFILE
command, which is documented here.
Parameters (object)
filepath
(string | required) - path to where the file should export. You should pass__dirname
at the start of yourfilepath
.table
(string | optional) - name of the table to export the dadta from. Fallsback to the file name if not provided.headers
(array | optional) - If you only want to export certain fields from the table, you can pass the fields you want using this array.delimiter
(string | optional | default =","
) - The delimiter you want in your output file. Defaults to comma.quotes
(string | optional | default =''
) - Optionally wrap each field value in quotes in the output file. Pass a single example of the character. For example, for quotes, passquotes: '"'
. Defaults to none.newline
(string | optional | default ="\n"
) - Line terminator in your file. Defaults to\n
.
Example
mysql.exportFileFromTable({
filepath: __dirname + '/tmp/example.csv',
quotes: '"',
})
.then(function(rowsAffected){
// rowsAffected = count of records exported
})
Returns (Promise | Int)
Returns a promise containing the number of rows affected. This is the number of records exported.
This method will take flat files and merge the columns together. You pass a merge object to describe how the tables link to one another and it matches records based on that definition.
Will output a final output file.
Parameters (object)
files
(array of objects | required) - Collection of objects describing input Filesfilepath
(string | required) - path to the file to loadtable
(string | optional) - name of the table to load the file into. Fallsback to the file name if not provided.overwrite
(bool | default =false
) - If an existing table with thetable
name exists, overwrite it. Defaults to false, throwing an error if the table already exists.headers
(array | optional) - If you only want to import certain fields from the file, you can pass the fields you want using this array.index
(array | string | required) - Provide the header that should be indexed - this is the field that will be used to join to other files / tables. This makes joining faster.delimiter
(string | optional | default =","
) - The delimiter of the file you're importing. Defaults to comma.quotes
(string | optional | default =''
) - String enclosing each field of the file you're importing. Pass a single example of the character. For example, for quotes, passquotes: '"'
. Defaults to none.newline
(string | optional | default ="\n"
) - Line terminator in your file. Defaults to\n
.
merge
(object | required) - Object describing how files map Togethertable1.table1_field: table2.table2_field
- key should be a table and it's index field. Value should be another table and it's indexed field. Do this for each file. Please note, you'll need to reference table. The fields will also be prepended with the table name.
output
(string | require) - Filepath to where the file should save. Just like with exportFileFromTable you'll need to be very specific where this file should go.
Example
let files = {
filepath: './test/data.csv',
table: 'datamerge1',
index: 'email',
quotes: '"'
},
{
filepath: './test/data2.csv',
table: 'datamerge2',
index: 'email',
quotes: '"',
};
let merge = {
'datamerge1.datamerge1_email': 'datamerge2.datamerge2_email'
};
mysql.mergeFiles(files, merge, __dirname + '/tmp/example.csv')
.then(function(rowsAffected){
// rowsAffected = count of records exported
})
Returns (Promise | Int)
Returns a promise containing the number of rows affected. This is the number of records exported. Will also export file to disk at output location.
To make changes to this class, ensure you document your methods or changes to any functionality. Additionally, make sure all unit testing passes. If you're adding a new method, write new tests in the /test/test.js file.
You can run unit tests with:
$ npm test
v1.0.0 - Complete methods and documentation v0.0.1 - Utility functions, tests, and readme