downloaded large log file fail about rds-pgbadger

[jellis]

This is indeed an issue with the AWS API.

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/RESTReference.html#RESTReference.DownloadCompleteDBLogFile.CLIversion

Currently, the only method to download complete log files. I've built a wrapper for it outside of the PHP SDK (my chosen language), which appears to stream the files down okay.

Having had a conversation with AWS support, they're aware the AWS CLI/API interface for downloadRDSLogPortion is buggy - hence they're leaving the deprecated rds-cli tool in place until they've figured it out.

Attempting to download and process RDS logs in a meaningful way is proving to be a HUGE headache.

[fpietka]

Since I'm not sure how to reproduce, I may need additional informations:

• What size are the truncated files?
• Do you happen to get all the logs in multiple files?
• Are you able to download the whole file through the AWS console?

[fpietka]

It may be related to aws/aws-cli#1504

[fpietka]

Thanks for your feedback on this issue!

I'll try to take a look at how to wrap this process then.

[jellis]

Just as a POC - I threw this together yesterday using some info from the AWS docs, modified to suit the REST endpoint.

# AWS Version 4 signing example

# IAM API (CreateUser)

# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a GET request and passes request parameters
# and authorization information in the query string
import sys, os, base64, datetime, hashlib, hmac, urllib
import requests # pip install requests

# ************* REQUEST VALUES *************
method = 'GET'
service = 'rds'
host = 'rds.ap-southeast-2.amazonaws.com'
region = 'ap-southeast-2'
logfile = 'error/postgresql.log.2017-11-22-03'
endpoint = 'https://rds.ap-southeast-2.amazonaws.com/v13/downloadCompleteLogFile/name-of-instance/' + logfile
contenttype = 'application/json'

# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning

# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print 'No access key is available.'
    sys.exit()

# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ') # Format date as YYYYMMDD'T'HHMMSS'Z'
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope


# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

# Because almost all information is being passed in the query string,
# the order of these steps is slightly different than examples that
# use an authorization header.

# Step 1: Define the verb (GET, POST, etc.)--already done.

# Step 2: Create canonical URI--the part of the URI from domain to query 
# string (use '/' if no path)
canonical_uri = '/v13/downloadCompleteLogFile/data-dev-cda/' + logfile

# Step 3: Create the canonical headers and signed headers. Header names
# must be trimmed and lowercase, and sorted in code point order from
# low to high. Note trailing \n in canonical_headers.
# signed_headers is the list of headers that are being included
# as part of the signing process. For requests that use query strings,
# only "host" is included in the signed headers.
canonical_headers = 'content-type:' + contenttype + '\n' 'host:' + host + '\n' + 'x-amz-date:' + amzdate + '\n'
signed_headers = 'content-type;host;x-amz-date'

# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'

# Step 4: Create the canonical query string. In this example, request
# parameters are in the query string. Query string values must
# be URL-encoded (space=%20). The parameters must be sorted by name.
# canonical_querystring = 'Action=CreateUser&UserName=NewUser&Version=2010-05-08'
# canonical_querystring = '&X-Amz-Algorithm=AWS4-HMAC-SHA256'
# canonical_querystring += '&X-Amz-Credential=' + urllib.quote_plus(access_key + '/' + credential_scope)
# canonical_querystring += '&X-Amz-Date=' + amzdate
# canonical_querystring += '&X-Amz-Expires=30'
# canonical_querystring += '&X-Amz-SignedHeaders=' + signed_headers
canonical_querystring = ''


# Step 5: Create payload hash. For GET requests, the payload is an
# empty string ("").
payload_hash = hashlib.sha256('').hexdigest()

# Step 6: Combine elements to create create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash


# ************* TASK 2: CREATE THE STRING TO SIGN*************
string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request).hexdigest()


# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key
signing_key = getSignatureKey(secret_key, datestamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode("utf-8"), hashlib.sha256).hexdigest()


# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# The auth information can be either in a query string
# value or in a header named Authorization. This code shows how to put
# everything into a query string.
#canonical_querystring += '&X-Amz-Signature=' + signature
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
headers = {'content-type': contenttype, 'x-amz-date':amzdate, 'Authorization':authorization_header}

# ************* SEND THE REQUEST *************
# The 'host' header is added automatically by the Python 'request' lib. But it
# must exist as a header in the request.
request_url = endpoint

r = requests.get(request_url, headers=headers, stream=True)
local_filename = logfile.split('/')[-1]
with open(local_filename, 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024): 
        if chunk: # filter out keep-alive new chunks
            f.write(chunk)
f.close()

print "success"

[cbandy]

I concur with @jellis about using the REST endpoint. I pasted some code that uses boto3 to sign requests in #10 (comment).

[fpietka]

We'll see how it goes from here, since #10 should really improve that issue.
I'll keep that issue in mind in case it ends up not satisfactory enough.