jeremydaly / data-api-client Goto Github PK

How can I connect to local mysql db for unit testing using data-api-client?

Currently, I don't want to connect to AWS for each unit test.

In my table, I have a column "duration" of type numeric(7,3). The Data API client is returning this as a string:

  {
    id: 365,
    file_uuid: 'fdc4c6ed-d72e-4621-aaa6-be2efe6ca5a1',
    name: "At World's Edge",
    owner_id: '9dbb70d7-3d17-4215-8966-49815e461dee',
    duration: '285.082',
    // ...
  }

1. Is there a way to run multiple SQL Statements and get multiple result sets within a single call?
   ExecuteSql supported this https://docs.aws.amazon.com/rdsdataservice/latest/APIReference/API_ExecuteSql.html

2. Is there a way to get multiple result sets when calling a stored procedure that returns them?

Would be great to have a similar wrapper for the newly announced Redshift Data API https://aws.amazon.com/about-aws/whats-new/2020/09/announcing-data-api-for-amazon-redshift/

Hello Jeremy, How are you doing ?

Do you know if in this API exist some resource or alternative to query something like we do on Oracle Cursor Expressions in order to receive a result as a sub level on JSON ? (there we do something like this select 1, 2, 3, cursor(select 4,5,6 from dual) from dual and then we have a set with a subset of records).
As I'm using MySql Aurora Serverless and the current version does not have JSON features already present on other versions for MySql I was wondering if something like that already exists in the API.
If not, would you possibly recommend something ?

Thank You in Advance !

Carlos

Not a very important issue, but I thought you'd like to know... :)

The github link for the Data API Client here:

https://www.jeremydaly.com/current-projects/#data-api-client

points to the Serverless MySQL repo. Took me a second to realize I was in the wrong place.

The Data API supports postgres arrays by returning {"arrayValue": {"<valueType>": [<value1>, <value2>]}}, but this package is returning {"<valueType>": [<value1>, <value2> ]}, meaning it's not only transforming the first level of the object. Expected result would be [<value1>, <value2>].

So far I bypassed this issue by transforming the field manually, but i'd be nice if this package supported this out of the box.

Updating these values works as expected, but fetching them does not.

Hello,

I get this error on about 1% of executions...

Looks like there are no more available connections, but isn't aurora serverless supposed to autoscale automatically?

{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "BadRequestException: Too many connections",
    "reason": {
        "errorType": "BadRequestException",
        "errorMessage": "Too many connections",
        "code": "BadRequestException",
        "message": "Too many connections",
        "time": "2020-02-04T18:31:38.387Z",
        "requestId": "c0cdad58-cefe-42a8-b3f6-acf1f4bffb07",
        "statusCode": 400,
        "retryable": false,
        "retryDelay": 99.64592804784691,
        "stack": [
            "BadRequestException: Too many connections",
            "    at Object.extractError (/var/task/node_modules/aws-sdk/lib/protocol/json.js:51:27)",
            "    at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)",
            "    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
            "    at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
            "    at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)",
            "    at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)",
            "    at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)",
            "    at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10",
            "    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)",
            "    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:685:12)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: BadRequestException: Too many connections",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:210:5)",
        "    at process.EventEmitter.emit (domain.js:476:20)",
        "    at processPromiseRejections (internal/process/promises.js:201:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:94:32)"
    ]
}

Hi,
Am using data-api-client for AWS lambda function in order to get the data to my UI. Now the question is how do I write my unit test cases. Here the below example code,

My lambda function consists with the below code,

/Lambda function code/
const dataApi = require("data-api-client")({
secretArn: process.env.secretArn,
resourceArn: process.env.clusterArn,
database: process.env.database,
options: {
httpOptions: {
proxy: process.env.env == 'local' ? 'http://host.docker.internal:3128' : null
}
}
});
const cognitoGroup='xyz';
const tenantStatus='Active';
exports.handler = async (event) => {
const tenantDetail = await findTenantIDForCognitoUserGroup(cognitoGroup, tenantStatus);
console.log('tenantDetail ':+tenantDetail );
};
function findTenantIDForCognitoUserGroup(cognitoGroup, tenantStatus) {

return dataApi.query(
SELECT tenant_id, tenant_code, tenant_glue_database_name FROM tenants WHERE tenant_cognito_user_group = :cognitoGroup and deleted is NULL and tenant_status = :tenantStatus,
{ cognitoGroup: cognitoGroup,tenantStatus: tenantStatus }

);
}

So, here my lambda function queries for data and then returns and prints.
Can some one help me how can I write unit and Integration test cases for the same. I've explored and came to know using jest we can do, but am a newbie and am not sure how to write the same.
Can some one guide me line by line to perform unit and interagtion test cases.

Thanks

It would be great to include a type declaration file.

I'm currently using this for now, but I'm seeing TypeError: config.RDS.executeStatement is not a function when executing query().

declare module 'data-api-client' {
  import RDSDataService from 'aws-sdk/clients/rdsdataservice';
  import {ConfigBase as Config} from 'aws-sdk/lib/config';

  export interface iParams {
    secretArn: string;
    resourceArn: string;
    database: string;
    keepAlive?: boolean;
    hydrateColumnNames?: boolean;
    options?: Config & RDSDataService.Types.ClientConfiguration;
  }

  export interface iDataAPIClient {
    query(...x: any[]): iDataAPIQueryResult;
  }

  export interface iDataAPIQueryResult {
    records: Array<any>;
  }

  export default function (params: iParams): iDataAPIClient
}

I'm trying to use batch query update but I cannot make it update all the items from the parameters array:

If I have machine (id 7501) with name: 'machine7501' and machine (id 7502) with name: 'machine7502' and run the following query

await this.dataApiClient.query(
  `UPDATE machine SET ${Object.keys(machine).map(key => `${key}=:${key}`)}
  WHERE id=:id`,
  [
    [{
      id: 7501, name: 'new_machine7501',
    }],
    [{
      id: 7502, name: 'new_machine7502',
    }],
  ],
);

I expect to both names to be updated (new_machine7501 and new_machine7502) respectively but the query only update the machine with 7501.

This might be an issue with my directory structure, but when i test the api i get an error saying it can't find the data-api-client module. I ran the npm install and my package.json has the data-api-client as a dependency. Any recommendations?

I've created a simple CLI tool that is supposed to create a new table using the Data API:

// tslint:disable: no-console

import dataApiClient from 'data-api-client'

const aSecretArn = process.env.AURORA_SECRET_ARN
const aClusterArn = process.env.AURORA_CLUSTER_ARN
const aDbName = process.env.AURORA_DATABASE_NAME

if (
  aSecretArn === undefined ||
  aClusterArn === undefined ||
  aDbName === undefined
) {
  throw new Error('one or more env vars are undefined')
}

const data = dataApiClient({
  database: aDbName,
  resourceArn: aClusterArn,
  secretArn: aSecretArn,
  options: {
    // aurora serverless data API is only available in us-east-1 for now
    // see https://read.acloud.guru/getting-started-with-the-amazon-aurora-serverless-data-api-6b84e466b109
    region: 'us-east-1'
  }
})

;(async () => {
  try {
    const result = await data.query(`
      create table reminders
      (
        id varchar(36) not null,
        PRIMARY KEY (id)
      )
    `)

    console.log('query result')
    console.dir(result)
  }
  catch (e) {
    console.log('query error')
    console.dir(e)
  }
})()

The issue is that when I run this about 9 out of 10 times I get the following error:

query error
{ BadRequestException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at Object.extractError (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.extractError (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
    at Request.callListeners (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/tommedema/projects/prem/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/tommedema/projects/prem/node_modules/aws-sdk/lib/request.js:685:12)
  message: 'Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.',
  code: 'BadRequestException',
  time: 2019-08-05T02:46:16.940Z,
  requestId: '8cc3fcb3-6928-4b52-ab24-8ac9023fcd84',
  statusCode: 400,
  retryable: false,
  retryDelay: 25.77670642672414 }

Only sometimes (after many retries) I get a valid response (in this case "table already exists").

Using data-api-client version 1.0.0-beta

My Aurora Serverless cluster was made in cloudformation:

    RDSAuroraServerlessCluster:
      Type: AWS::RDS::DBCluster
      Properties:
        MasterUsername: ${{env:AURORA_MASTER_USERNAME}}
        MasterUserPassword: ${{env:AURORA_MASTER_PASSWORD}}
        DatabaseName: ${{env:AURORA_DATABASE_NAME}}
        Engine: aurora
        EngineMode: serverless
        ScalingConfiguration:
          AutoPause: true
          MaxCapacity: 4
          MinCapacity: 1
          SecondsUntilAutoPause: 500

    RDSAuroraClusterMasterSecret:
      Type: AWS::SecretsManager::Secret
      Properties:
        Description: This contains the RDS Master user credentials for RDS Aurora Serverless Cluster
        SecretString:
          !Sub |
            {
              "username": "${{env:AURORA_MASTER_USERNAME}}",
              "password": "${{env:AURORA_MASTER_PASSWORD}}"
            }

And I enabled the data API manually:

aws rds modify-db-cluster --db-cluster-identifier ARN --enable-http-endpoint

Note that when it does give a valid response, it seems to keep working for a while. After a while it stops working again and gives me the BadRequestException issue for many subsequent tries. This makes me believe that the issue is related to the cold starts of Aurora Serverless. How did you take care of this?

Note that increasing the connectTimeout option does not seem to help:

options: {
    maxRetries: 10,
    httpOptions: {
      connectTimeout: 30000
    }
  }

Hi Jeremy !

Thanks so much for this really great wrapper.

I also read your article: https://www.jeremydaly.com/serverless-mysql-at-scale/, where you mention another tool of yours (https://github.com/jeremydaly/serverless-mysqL) because serverless raises a lot of issues such as max_connections...

I'm using the DATa API today but I wonder: does data-api-client and the DATA-API itself solve the issues you raised in your article or should I use both your DATA-API -CLIENT with at the same time SERVERLESS-MYSQL ?

Maybe the aws DATA-API actually solves the issues you raised in https://www.jeremydaly.com/serverless-mysql-at-scale/ but it's not toally crystal clear for me.

I researched on why cloudformation didn't accept over 200mb but that is their rule. So I figured out why the project has over 200 mb and found out that the data-api-client has very high file size. How could you deploy without getting the file oversize issue? I couldnt delete the node_module because of data-api-client. Please help.

sorry really puzzled by some weird phenomenons with the data-api-client but not managing to draft for the moment my question and bug in precise terms

Hi Jeremy,

Have been using your wrapper for the past weeks, and it makes working with Aurora serverless a breeze (much easier than native use) ! thanks so much

However, I read the docs and the numerous examples, where you stress the fact some things must be "done" in a certain way (for example the ordering of the parameters), and i could not find a clear idea on the following issue:

• what if one of the parameter is "handled" by a pure SQL command and not the usual =, > ...
  example: check out in my example below the following lines :
  author_email IS NOT NULL
article_publication_date >= CURDATE() - INTERVAL 180 DAY

  I mean I can't put these "pure sql" words outside the sql statement, so is it OK to leave it like my example ?

• as a consequence of the point above : what about the impact on the rigidity of the "ordering of parameters" you warn about ?
  Indeed now, as you can see on the example below I end up having 5 constraints/lines inside the WHERE ...but only 2 in the parameters block { } so is this "imbalance" (5 vs 2) important? will it break things?
  To be as safe as possible I decided to order the 5 constraints inside the WHERE clause like this : start by writing with the 2 constraints where you have something matching inside the parameters block { } and only after this add all the pure sql stuff (IS NULL, CURDATE() - INTERVAL 180 DAY...) .

  What do you think ? Will it enable me to dodge the "ordering issue" you mention here ?

EXAMPLE

let existing_author_from_db_query = await data.query(     
      `
        SELECT
          author_email,
          author_full_name,
          author_first_name,
          author_last_name
        FROM 
          results
        WHERE
          status = :status AND
          url_domain = :url_domain AND
          author_email IS NOT NULL AND
          author_full_name IS NOT NULL AND
          article_publication_date >= CURDATE() - INTERVAL 180 DAY
      `,
      {
        status: process.env.CONTEXT === "production" ? "prod" : "dev",         
        url_domain: article_domain,
      }          
    );

What would you recommend in terms of how to use your wrapper in such a case ?

Thanks so much for any help!

Hi,
is it possible to use continueAfterTimeout with transactions?

An original implementation of this was supplied in #20, but it needs some more thought.

Handling date input should be straightforward. If a native JavaScript Date object is passed as a parameter, we could automatically cast it to a stringValue (assuming MySQL and Postgres could handle it).

Parsing it back into a Date object might be a little more tricky. First, this would need to be optional. So we would likely add another config parameter like parseDates. This could be passed on instantiation or as a query configuration parameter (like database) and override the global setting. Second, we'd need to inspect the columnMetaData and whitelist data types that can be coerced.

Looking for feedback on this feature.

Hello,

First of all, thanks for this library. It is so annoying having to use AWS SDK for this...

Anyway, I detected what I think is a bug.

My project uses Amplify, AppSync (GraphQL), Lambdas and Aurora Serverless.

One of the types I have in my graphql schema is this one:

type User {
  userId: ID!
  dob: AWSDate!
  ….
}

NOTE: dob is date of birth, so I only care about the date, not about time.

The SQL command I have in my lambda function is something like:
SELECT userId, dob …..FROM User WHERE userId='$id'

When using AWS framework
var dbResponse = await RDS.executeStatement(params).promise();
I receive something like this:

{
    "numberOfRecordsUpdated": 0,
    "records": [
        [
            {
                "stringValue": "Ricardo"
            },
            {
                "stringValue": "1956-03-12"
            }
        ]
    ]
}

And everything works fine.

However, if I use this library in my lambda function:
let dbResponse = await dataApiClient.query('SELECT userId, dob FROM User WHERE userId =:userId;', { userId: userId });
Then I receive this:

{
    "records": [
        {
            "userId": "01B452F74BC941E58642766EE9D2C473",
            "dob": "1977-09-19T00:00:00.000Z",
            ....
        },

And because of that, graphql mapping is failing with this error:

▿ Can't serialize value (/getUsersForGame[0]/user/dob) : Unable to serialize `1977-09-19T00:00:00.000Z` as a valid date.
  ▿ object : 3 elements
    ▿ 0 : 2 elements
      - key : "message"
      - value : Can't serialize value (/getUsersForGame[0]/user/dob) : Unable to serialize `1977-09-19T00:00:00.000Z` as a valid date.
    ▿ 1 : 2 elements
      - key : "path"
      ▿ value : 4 elements
        - 0 : getUsersForGame
        - 1 : 0
        - 2 : user
        - 3 : dob
    ▿ 2 : 2 elements
      - key : "locations"
      - value : <null>

BTW, when I use AWS console and I use the query editor, I also see "1956-03-12", without time.

About my RDS database, the type is this one:
dob date NOT NULL

So, for some reason, seems this library is not processing the date type properly.
Do you know what's going on?
Thanks a lot.

EDIT:
I found a workaround. Not ideal, but better than having to use AWS framework:
SELECT DATE_FORMAT(dob, GET_FORMAT(DATE,'ISO')) as dob from User;

I'm not sure why but this
"Resource": "arn:aws:secretsmanager:{REGION}:{ACCOUNT-ID}:secret:{PATH-TO-SECRET}/*" didn't work while this
"Resource": "arn:aws:secretsmanager:{REGION}:{ACCOUNT-ID}:secret:{PATH-TO-SECRET}" did (no slash + asterisk at the end).

Hello,

After updating to version 1.0.0 (I was using beta) this error occurs when I try to make a transaction:

No 'sql' statement provided.
at error (/var/task/build/node_modules/data-api-client/index.js:54:35)
at parseSQL (/var/task/build/node_modules/data-api-client/index.js:60:5)
at Object.query (/var/task/build/node_modules/data-api-client/index.js:303:15)
at commit (/var/task/build/node_modules/data-api-client/index.js:421:30)
at process._tickCallback (internal/process/next_tick.js:68:7)

transaction.query(
      `INSERT INTO ${TABLE_NAME} (${keys}) VALUES(${params})`,
      objToInsert
    );

I'm curious if this library handles partial objects in the parameter sets.

This would happen when a subset of records/objects are modified at run time.

For example

// Update with named parameters
let update = await data.query(
  `UPDATE myTable SET age = :age, name = :name WHERE id = :id`,
  { age: 13, id: 5 }
  { age: 20, id: 6, name: 'Pierre' }
)

How would this library handle only one of the records containing the name parameter?

Hello,

I am getting this error when I run below code:

const data = require('data-api-client')({
  secretArn: process.env.AWS_SECRET_STORE_ARN,
  resourceArn: process.env.DB_CLUSTER_ARN,
  database: process.env.DB_NAME
});

describe('postConfirmation tests', () => {
  test('usp_get_invitation', async () => {
    expect.assertions(1);
    const email = '[email protected]';
    try {
      let invitations = await data.query(`call usp_get_invitation(:email)`, { email });
      console.log(JSON.stringify(invitations));

      expect(invitations).toEqual(expect.anything());
    } catch (error) {
      console.error(error);
    }
  });
});

TypeError: config.RDS.executeStatement is not a function
          at query (/home/ubuntu/Recruiter-backend/node_modules/data-api-client/index.js:321:22)
          at Object.query (/home/ubuntu/Recruiter-backend/node_modules/data-api-client/index.js:471:22)
          at Object.test (/home/ubuntu/Recruiter-backend/__tests__/handlers/handler.test.js:22:36)
          at Object.asyncFn (/home/ubuntu/Recruiter-backend/node_modules/jest-jasmine2/build/jasmine_async.js:82:37)
          at resolve (/home/ubuntu/Recruiter-backend/node_modules/jest-jasmine2/build/queue_runner.js:52:12)
          at new Promise (<anonymous>)
          at mapper (/home/ubuntu/Recruiter-backend/node_modules/jest-jasmine2/build/queue_runner.js:39:19)
          at promise.then (/home/ubuntu/Recruiter-backend/node_modules/jest-jasmine2/build/queue_runner.js:73:82)
          at <anonymous>

Looking forward your help.

I tried to the best of my ability find documentation regarding schema migrations, however, couldn't find any information regarding it.

Does data api client support schema migrations? Is there any documentation?

I want to load a csv file to aurora serverless database.

My sql command is
LOAD DATA LOCAL INFILE '/Users/AAA/book.csv' REPLACE INTO TABLE book fields terminated by ',' lines terminated by '\n' ignore 1 lines (book_id,source_id,@YR,@mo,@da) set year_of_pub = IF(@YR = '',null, @YR), month_of_pub = IF(@mo = '',null, @mo), day_of_pub = IF(@da = '',null, @da);"

This SQL command worked when I uploaded the file to EC2 and loaded data there.
But when I tried to do it on my own laptop with Data API ExecuteSql operation (awscli or boto3)
I got error like:
An error occurred (BadRequestException) when calling the ExecuteSql operation: Unable to open file '/Users/AAA/book.csv'for 'LOAD DATA LOCAL INFILE' command.Due to underlying IOException:

** BEGIN NESTED EXCEPTION **

java.io.FileNotFoundException
MESSAGE: /Users/AAA/book.csv (No such file or directory)

Please help!

Some compatibility issues were brought up in #25 that show conflicts with the PostgreSQL version (likely because of sqlstring). There are also some additional data types (like structValue) that are only supported by the PostgreSQL version.

Overall, I think the two engines are rather interchangeable (thanks for that AWS). We likely need to add an engine config option or something similar since I don't think the library can "auto detect" the underlying version.

I've created a new development branch (https://github.com/jeremydaly/data-api-client/tree/postgres-support) that should be the target of any pull requests regarding this. Please feel free to add comments to this issue with ideas or compatibility observations.

It's stated in the README that using array values is not currently supported. However, I'm fairly sure the AWS Data API does indeed support it now (it may not have until recently though, I'm not sure). The syntax for it is a little different than the other types as the SqlParameter type would look something like (as per the related example in the README):

{ name: 'id', value: { arrayValue: { longValues: [1,2,3,4,5] } } }

This does seem like a major limitation for using this library, and I'm not sure if this has been tried prior to this point.

One thing that would need to be considered is that there may be a limit to how big this array can be in a query depending on the DBMS. I am not sure if this library would be the best place to implement some of that logic and break apart the arrays into chunks with limited capacity, or if that should be left to the consumer.

Just wondering if there is any way to get at out params from stored procedure calls?

My INSERT is unfortunately hidden in a sproc - sp_create_thingy(blah, out newId). In the standard node MySQL client it's clunky but you can pass something like this "CALL sp_create_thingy(blah, @foo); select @foo;" (as long as the multiple-statements option is turned on).

I'm not sure how to achieve a similar effect with this library? Is this possible?

Amazing library!

As you know, there is a 1 MB limit for Data API results. I'm just working through how to deal with this generically for my queries (I'm implementing a GraphQL API, so I'm constructing SQL queries generically from GraphQL queries). I need to somehow split up any query given to the Data API into chunks <= 1 MB in size.

My question is, how are people dealing with this? And would it be appropriate to somehow build this chunking into the data-api-client? Not having to deal with this limitation directly would be an amazing feat of abstraction for this library.

Thanks!

sslEnabled: false for local dev fails with the following error 'Protocol "http:" not supported. Expected "https:"'.
To work around this issue you must also disable keepAlive to prevent the custom httpsAgent being configured

const dataClient = require('data-api-client')({
  resourceArn: '*******',
  secretArn: '*******',
  database: '*******',
  sslEnabled: false,
  keepAlive: false, // <------- work around
  options: {
    endpoint: 'localhost:8080',
  },
});

In Postgres, an insert that has jsonb data type gives the error:
column [column name] is of type jsonb but expression is of type character varying

Possibly related to #40, which was closed.

I'd like to be able to specify a timeout on the client, so that I can detect slow responses (usually due to the DB resuming after auto-pause) before my Lambda function times out, and handle them appropriately.

I've tried passing through RDSDataService timeout properties via options but they have no effect.

require('data-api-client')({
...
options: {
    httpOptions: {
        // Low values for test purposes
        connectTimeout: 10,
        timeout: 10
    }
})}

This isn't surprising as the socket connection will be made promptly, even if the database query is slow.

Is there anything else that I can try? Appreciate that this is probably a limitation of the underlying client/service rather than your (extremely useful) wrapper.

Dear data-api-client supporters,

I have a simple question: How to handle null / undefined values when inserting / updating data rows? No matter if I leave parameters as undefined or explicitly set them as null, I always receive an error message either saying 'field_x' is an invalid type (when passing the parameter as undefined) or Unknown column 'field_x' in 'field list' (when passing the parameter as null). How can I explicitly set a field as NULL in the database?

Thanks a lot and best regards
The Smart Home Maker

AWS Lambda Runtimes provide by default the aws-sdk in the AWS environment, so it's only necessary to npm install it in the devDependencies.

Assuming this scenario - where we use the aws-sdk provided by the AWS environment - the following occurs when running with node 8.10 runtime:

module initialization error: TypeError
at module.exports (/var/task/node_modules/data-api-client/index.js:464:10)
at Object.<anonymous> (/var/task/testJS.js:5:17)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)

which is this line here - that accesses the RDSDataService.

And when we run in the 10.x runtime, a more specific error is shown (tested based on this issue):

{
    "errorType": "TypeError",
    "errorMessage": "config.RDS.executeStatement is not a function",
    "stack": [
        "TypeError: config.RDS.executeStatement is not a function",
        "    at query (/var/task/node_modules/data-api-client/index.js:321:22)",
        "    at Object.query (/var/task/node_modules/data-api-client/index.js:471:22)",
        "    at Runtime.module.exports.main [as handler] (/var/task/testJS.js:21:34)",
        "    at Runtime.handleOnce (/var/runtime/Runtime.js:63:25)",
        "    at process._tickCallback (internal/process/next_tick.js:68:7)"
    ]
}

My guess is that this is due to the fact that AWS provides the following versions in the respective runtimes:

Name	Identifier	Node.js Version	AWS SDK for JavaScript	Operating System
Node.js 10	nodejs10.x	10.15	2.437.0	Amazon Linux 2
Node.js 8.10	nodejs8.10	8.10	2.290.0	Amazon Linux

If we analyze the aws-sdk changelog, the errors make sense now. Probably because the 2.290.0 does not include RDSDataService at all, and 2.437.0 does not have executeStatement (#3) (released in 2.466.0.

My question is: is there a way to change the version of the aws-sdk (the one that's default in each runtime)? Or we have to package it with our lambdas - thus increasing the size of the package?

Nonetheless, I suggest updating the README with this information (if agreed, I'm up to contributing on this particular issue).

Is there a way to retrieve r.InsertID of multiple queries inside a following query? The following code will demonstrate my use case:

db.transaction()
        .query(insertContactSQL, formData.sender)
        .query(insertContactSQL, formData.receiver)
        .query(insertContactSQL, formData.customer)
        .query(insertOrderSQL, {
            senderID: INSERT_ID_OF_FIRST_QUERY,
            receiverID: INSERT_ID_OF_SECOND_QUERY,
            customerID: INSERT_ID_OF_THIRD_QUERY,
        });

I am trying to get all records where one column called 'job_status' is EITHER equal to 'submitted' or to 'in_process'. Should be easy but it doe snot work.

let submitted_jobs_db_query_results = await data.query(
    `
      SELECT 
        job_id
      FROM
        aws_async_comprehend_jobs
      WHERE
        type = :type AND
        job_status = :job_status
      ORDER BY
        aws_async_comprehend_job_id
      LIMIT 5
    `,
    { 
      type: "text-analysis",
      job_status: "SUBMITTED" || "IN_PROGRESS"
    }

I thought of using the "OR" concept directly inside the SQL query (instead of inside the {} parameters block) but I failed at that too (I think you actually say in the Docs it does not work : https://github.com/jeremydaly/data-api-client#you-cant-send-in-an-array-of-values)

So how to do this "rather basic" query (saying if "value is A or B") with data-api-client ?

Thanks a lot
M.

I'm trying to run the following query for dynamic sorting.

SELECT * FROM tenant ORDER BY ::column DESC
{ column: 'industry' }

I get the following error though.

BadRequestException: ERROR: operator does not exist: ` character varying
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.

Hello,

I'm wondering something: if I let the default keepAlive=true, does that mean all lambdas (concurrently executin js code which contain data-api-client query to my table) will share the same "session" ?

If yes, is it an issues as regards "row locks" ?

Indeed I read here that :

Using transactions with InnoDB (auto-commit turned off), a SELECT ... FOR UPDATE allows one session to temporarily lock down a particular record (or records) so that no other session can update it.

So if SQL statement execution by lambda 1 sets a row lock but subsequent lambdas execution trying to read/writing theses rows SHARE the same session, will they be, as I wish, prevented from doing it thanks to the row lock?

If they're not prevent to read, that is to say if there's no row lock => is it possible to set up a row lock on aurora serverless MySQL while allowing improved performance thanks to keep-alive ?

Thanks for any help

I tried to create a policy based on the one in the README:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "rds-data:ExecuteSql",
          "rds-data:ExecuteStatement",
          "rds-data:BatchExecuteStatement",
          "rds-data:BeginTransaction",
          "rds-data:RollbackTransaction",
          "rds-data:CommitTransaction"
        ],
        "Resource": "arn:aws:rds:us-east-1:XXXXXXXX:cluster:YYYYYYYYY"
      },
      {
        "Effect": "Allow",
        "Action": [ "secretsmanager:GetSecretValue" ],
        "Resource": "arn:aws:secretsmanager:us-east-1:XXXXXXXX:secret:ZZZZZZZZZ"
      }
    ]
}

The AWS policy UI showed this error:

Going through with the policy, the visual editor shows a better error:

I found this in the docs:

Amazon RDS Data API does not support specifying a resource ARN in the Resource element of an IAM policy statement. To allow access to Amazon RDS Data API, specify “Resource”: “*” in your policy.

https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazonrdsdataapi.html#amazonrdsdataapi-resources-for-iam-policies

I use a lot of data-api-client queries inside my lambdas (js/node) with no issue but I stumbled into a weird issue.

This code in my lambda is working:

`SELECT             
        qr.query_result_id,
        qr.url,       
        MAX(coe.email_nb) as max_email_nb_sent_by_this_customer_to_this_author_on_this_query
      FROM
        query_results qr
      LEFT JOIN
        customers_outreach_emails coe
      ON
        qr.author_email = coe.author_email AND
        coe.customer_id = ${customerId}         
      WHERE              
        qr.status = :status AND 
        qr.query_id = :query_id AND
        qr.author_email IS NOT NULL AND
        qr.article_publication_day >= CURDATE() - INTERVAL ${oldest_article_associated_with_outreached_person} DAY AND
        qr.query_result_id NOT IN (
          SELECT query_result_id
          FROM customers_outreach_emails
          WHERE            
            customer_id = 2 AND 
            query_id = 2
        ) AND        
        qr.ready_for_being_used_for_emailing IS TRUE
      GROUP BY
        qr.author_email
      ORDER BY 
        qr.query_result_id ASC
      LIMIT 
        20
`,
 { 
      status: process.env.CONTEXT === "production" ? "prod" : "dev",
      query_id: queryId,
      customer_id: customerId,
      query_id2: queryId,
      customer_id2: customerId,
      customer_id3: customerId,
    }    

But as soon as I change customer_id = 2 to customer_id = :customer_id, like this:

 WHERE            
          customer_id = :customer_id AND 
          query_id = 2

... it starts to fail ! So the query below fail... this one little change makes everything crumble :(

`SELECT             
        qr.query_result_id,
        qr.url,       
        MAX(coe.email_nb) as max_email_nb_sent_by_this_customer_to_this_author_on_this_query
      FROM
        query_results qr
      LEFT JOIN
        customers_outreach_emails coe
      ON
        qr.author_email = coe.author_email AND
        coe.customer_id = ${customerId}         
      WHERE              
        qr.status = :status AND 
        qr.query_id = :query_id AND
        qr.author_email IS NOT NULL AND
        qr.article_publication_day >= CURDATE() - INTERVAL ${oldest_article_associated_with_outreached_person} DAY AND
        qr.query_result_id NOT IN (
          SELECT query_result_id
          FROM customers_outreach_emails
          WHERE            
            customer_id = :customer_id AND 
            query_id = 2
        ) AND 
        qr.ready_for_being_used_for_emailing IS TRUE
      GROUP BY
        qr.author_email
      ORDER BY 
        qr.query_result_id ASC
      LIMIT 
        20
`,
 { 
      status: process.env.CONTEXT === "production" ? "prod" : "dev",
      query_id: queryId,
      customer_id: customerId,
      query_id2: queryId,
      customer_id2: customerId,
      customer_id3: customerId,
    }  

The error I get on Cloudwatch is:

{
    "errorMessage": "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':customer_id AND \n            query_id = 2\n        ) AND\n        qr.author_email' at line 34",
    "errorType": "BadRequestException",
    "stackTrace": [
        "            query_id = 2",
        "        ) AND",
        "        qr.author_email' at line 34",
        "Object.extractError (webpack:///./node_modules/aws-sdk/lib/protocol/json.js?:51:27)",
        "Request.extractError (webpack:///./node_modules/aws-sdk/lib/protocol/rest_json.js?:55:8)",
        "Request.callListeners (webpack:///./node_modules/aws-sdk/lib/sequential_executor.js?:106:20)",
        "Request.emit (webpack:///./node_modules/aws-sdk/lib/sequential_executor.js?:78:10)",
        "Request.emit (webpack:///./node_modules/aws-sdk/lib/request.js?:683:14)",
        "Request.transition (webpack:///./node_modules/aws-sdk/lib/request.js?:22:10)",
        "AcceptorStateMachine.runTo (webpack:///./node_modules/aws-sdk/lib/state_machine.js?:14:12)",
        "eval (webpack:///./node_modules/aws-sdk/lib/state_machine.js?:26:10)",
        "Request.eval (webpack:///./node_modules/aws-sdk/lib/request.js?:38:9)",
        "Request.eval (webpack:///./node_modules/aws-sdk/lib/request.js?:685:12)"
    ]
}

My intuition which might be wrong: because you say Named parameters MUST be sent in order (https://github.com/jeremydaly/data-api-client#named-parameters-must-be-sent-in-order), it means when it's a nested WHERE, it's not because it comes as the third parameter (if you read the code from up to bottom) that's it's actually the third that SQL "sees", so my order of parameters in the parameters block is wrong.

How can I make this work? In the "worst" case where dat-api-client does not work inside "nested" WHERE, then can I use the string literal ${ } as it's a JavaScript file and it might be interpreted (and of course remove it from the parameters block) ?

 WHERE            
          customer_id = ${customer_id} AND 
          query_id = 2

I tried and it seems replacing everywhere i have :named_parameters with literals ${ } work. If the case I can use ${ } whenever in the sql query I used to have :named_parameter, what is the value-added of named parameters, I mean is there a case where I can't use ${ } and should only use your named parameters ?

Thanks

ive been trying to track down an answer to my issue all day. when running a simple query (looking up a user by id for instance) i am getting this error:

i just looked at the aws js sdk docs and noticed there is no param type that directly correlates to uuid

there is also this note on the aws data api general docs

so my question is can this really not handle querying with a uuid?

I think it would be wonderful if v2 of this lib could use sql-template-strings because it offers a really nice dev experience with clean syntax.

// book and author are variables in scope
await query(SQL`SELECT author FROM books WHERE name = ${book} AND author = ${author}`)

I'd like to hear your thoughts on this enhancement request, @jeremydaly . Thank you for the wonderful lib!

If I want to run a query like:

SHOW DATABASES

I get this error:

"errorMessage": "No 'database' provided."

Even though I'm able to run a command like this on no database and get results:

aws rds-data execute-statement --resource-arn $resourceArn --secret-arn $secretArn --sql "SHOW DATABASES"

Would it be possible to allow queries that aren't tied to a specific database?

I am wondering if this library should provide support for highload interaction with RDS Data API?

RDS-serverless is great AWS service, but it has a tons caveats within highload interoperation. For example, DML query result can be received as error, although it had been successfully fulfilled, or transaction commit can be reported as successful, although it had been rollbacked.

There are minimum four known highload RDS Data API issues: https://redd.it/i3gnpz/ , https://redd.it/g0h4jq/ , https://redd.it/i4qcz1 , https://redd.it/f6ag4c/

Some of that issues can be wrapped around in library code by introducing auto-retries on AWS temporary errors and performing additional queries on ambiguous results, which had been caused by RDS-service errors.

At least following RDS services wrappers are mandatory: https://git.io/JUS3K and https://git.io/JUS36 , including inspecting error message name, type, code, message and stack, depend on current error.

CC @nitesmeh

Hi,

How can we make a Select query with contains or like?

Great npm!
Thanks

Seems like this client could work nicely with serverless-offline as an interface to a local mysql instance.

How would you recommend using data api locally? Do you have any plans on supporting serverless-offline?

Hi,
This client is very useful for using Data API with Aurora Serverless, however, it 's working only in local (serverless offline). After deploying, it returned Timeout issue.
I raised the issue in Stackoverflow. Please refer to the post

Any suggestion is appriciated.

Hello, thanks for writing this great library!

One issue I am having is running batch inserts. The documentation states that you can pass in an array to insert multiple rows, however, when I try that it only inserts the first row.

let insert = await data.query(
"INSERT INTO users(username, firstName, lastName) VALUES(:username,:firstName,:lastName)",
[
{
"username": "Username test 1",
"firstName": "first name 0.6054065569727269",
"lastName": "last name 0.5419870975135661"
},
{
"username": "Username test 2",
"firstName": "first name 0.6424165900774252",
"lastName": "last name 0.8571923498518657"
},
{
"username": "Username test 3",
"firstName": "first name 0.13976832378006354",
"lastName": "last name 0.8039286127593184"
}
]
);

I did some digging through the source code, batch processing is enabled if the processed param's first child is an array:

const isBatch = processedParams.length > 0
&& Array.isArray(processedParams[0]) ? true : false

Passing in parameters as an array of arrays triggers the batch insert:

let insert = await data.query({
sql: "INSERT INTO users(username, firstName, lastName) VALUES(:username,:firstName,:lastName)",
parameters:
[
[
{
"username": "Username test 1",
"firstName": "first name 0.6054065569727269",
"lastName": "last name 0.5419870975135661"
}
],
[
{
"username": "Username test 2",
"firstName": "first name 0.6424165900774252",
"lastName": "last name 0.8571923498518657"
}
],
[
{
"username": "Username test 3",
"firstName": "first name 0.13976832378006354",
"lastName": "last name 0.8039286127593184"
}
]
]
});

It would be great if one could pass in a one-dimensional array instead of a 2D one to insert multiple rows.

Thanks!