juliacloud / awscore.jl Goto Github PK

View Code? Open in Web Editor NEW

48.0 48.0 26.0 5.84 MB

Amazon Web Services Core Functions and Types

License: Other

Julia 99.91% Makefile 0.09%

awscore.jl's People

Contributors

Stargazers

Watchers

awscore.jl's Issues

localhost_is_ec2() Test on EC2

Currently we are not testing localhost_is_ec2() on EC2 instances, this should be added in.

precompile() occuring after module rather than before

Reading the documentation (by typing ?__precompile__ at the REPL), it seems that "Typically, __precompile__() should occur before the module declaration in the file...".

I've no idea how important this actually is, however today I had some problems with julia appearing to hang (or at least take a very long time in precompilation) and moving the __precompile__() out of the module definition in AWSCore.jl seemed to help. I haven't actually gotten to the bottom of what was going on, so it could have just been coincidence.

Adjust for clock skew

Several other SDKs implement this: https://altereos.com/how-to-correct-clock-skew-in-aws/

I'm going to try it for AWSCore.

AWS4AuthLayer - Specified Profile only from Envvars

As I am in the process of moving the AWS4AuthLayer into this repo I'm making this issue here.

Currently when making an HTTP.request() using the AWS4AuthLayer you can only specify the profile through the environment variables AWS_PROFILE and AWS_DEFAULT_PROFILE.

We should probably pass in a kwarg with the profile name and look for these in the various methods when making AWSCredentials.

Use of http protocol by default

This system of packages seem to use http as the default, going by :

AWSCore.jl/src/names.jl

Line 27 in df12166

protocol = "http"

This causes issues in certain cases with S3:

ERROR: LoadError: InvalidArgument -- Requests specifying Server Side Encryption with AWS KMS managed keys must be made over a secure connection.

HTTP 400 -- GET http://my-bucket-name.s3.amazonaws.com/some/path/some_file.csv
 -- <?xml version="1.0" encoding="UTF-8"?>

<Error><Code>InvalidArgument</Code>
<Message>Requests specifying Server Side Encryption with AWS KMS managed keys must be made over a secure connection.</Message>
<ArgumentName>x-amz-server-side-encryption</ArgumentName>
<ArgumentValue>null</ArgumentValue>
<RequestId>ABCD1234</RequestId><HostId>ABCDEF12345678=</HostId>
</Error>

Isn't it safer to default all communication over https? Is there are workaround for this on the current codebase?

The provided 'x-amz-content-sha256' header does not match what was computed

I got an error with putting object to S3, any ideas why does this happen?
a google search indicates that this is an error with "Content-Length" header. I tried to set this header, but still got the same error.

ERROR: XAmzContentSHA256Mismatch -- The provided 'x-amz-content-sha256' header does not match what was
 computed.                                                                                            
HTTP.ExceptionRequest.StatusError(400, HTTP.Messages.Response:                                        
"""                                                                                                   
HTTP/1.1 400 Bad Request                                                                              
x-amz-request-id: 0AA8879080750357                                                                    
x-amz-id-2: 2sRfJ1BJoh2qxm9U4jUFMq9DlcRNaZCZD3H2hVCrzBzdmKhkaLNY4njRdbMAx03189CMt9Na81w=              
Content-Type: application/xml                                                                         
Transfer-Encoding: chunked                                                                            
Date: Tue, 02 Oct 2018 21:50:51 GMT                                                                   
Connection: close                                                                                     
Server: AmazonS3                                                                                      
                                                                                                      
<?xml version="1.0" encoding="UTF-8"?>                                                                
<Error><Code>XAmzContentSHA256Mismatch</Code><Message>The provided 'x-amz-content-sha256' header does 
not match what was computed.</Message><ClientComputedContentSHA256>7f08f6ff9d11494a990bed7efb105e97259
a35f1f7ed72d9fb83cec44a62e467</ClientComputedContentSHA256><S3ComputedContentSHA256>47e0eabb9907757cfe
1da1c6ea284f8ec52589e3f9243cff9850a0eb9cc511aa</S3ComputedContentSHA256><RequestId>0AA8879080750357</R
equestId><HostId>2sRfJ1BJoh2qxm9U4jUFMq9DlcRNaZCZD3H2hVCrzBzdmKhkaLNY4njRdbMAx03189CMt9Na81w=</HostId>
</Error>""")

localhost_is_ec2() == false on ECS running in AWS Batch

I am trying to get credentials working in jobs submitted by AWS Batch, which uses ECS under the hood. It seems like the intention of AWSCore.jl is to get creds from aws_config() or directly from ecs_instance_credentials(). However, these fail because localhost_is_ec2() assertions fail:

ERROR: LoadError: AssertionError: localhost_is_ec2()
Stacktrace:
[1] ecs_instance_credentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:214
[2] include_from_node1(::String) at ./loading.jl:576
[3] include(::String) at ./sysimg.jl:14
[4] process_options(::Base.JLOptions) at ./client.jl:305
[5] _start() at ./client.jl:371
while loading /testcreds.jl, in expression starting on line 4

Mocking tests for instance credentials

We should add Mocking.jl as a test dependency once that package switches to using Cassette. This will be helpful in testing credential searching logic (particularly instance credentials), as noted in #65.

dynamodb support

The AWSCore documentation gives an example of using DynamoDB

r = dynamodb(aws, "ListTables", "{}")
println(r)

but I can't see that you have published a package which provides this functionality. I do note however your example in dls/DynamoDB.jl#4. Does this code relate to that example, or do you plan to release a module for DynamoDB?

setting of timeout?

the default timeout in Requests.jl is inf, which seems not reasonable.

I was using AWSS3 and got ECONNRESET error occasionally. This ECONNRESET error was not caught and could not be handled. If it is timeout error, should be easy to catch and handle.

ERROR (unhandled task failure): readcb: connection reset by peer (ECONNRESET)                       │················
 in yieldto(::Task, ::ANY) at ./event.jl:136                                                        │················
 in wait() at ./event.jl:169                                                                        │················
 in wait(::Condition) at ./event.jl:27                                                              │················
 in wait_readnb(::TCPSocket, ::Int64) at ./stream.jl:303                                            │················
 in eof(::TCPSocket) at ./stream.jl:58                                                              │················
 in macro expansion at /root/.julia/v0.5/Requests/src/streaming.jl:66 [inlined]                     │················
 in (::Requests.##6#8{Requests.ResponseStream{TCPSocket},Requests.ResponseParser,Channel{Symbol}})()│················
 at ./event.jl:68

Read default region from config file

Following from: #34 (comment)

... it doesn't seem quite right to duplicate the region env lookup and default (the other place is here: https://github.com/JuliaCloud/AWSCore.jl/blob/master/src/AWSCore.jl#L109)

There is a slight problem in that struct AWSCredentials does not have a region field so the region from the .config or .credentials file can't currently be accessed by the aws_config function. I propose the following:

Add a default_region::String field to struct AWSCredentials.

Set default_region=get(ENV, "AWS_DEFAULT_REGION", "us-east-1") in the default constructor

In dot_aws_credentials, set default_region if specified by the config files.

In the aws_config function, change the region= kw parameter to region=creds.default_region.

I think this should preserve existing behaviour, keep the default in one place, and add support for reading the region from the config files.

Improve kw-arg wrapper codegen

See: #28 where a stack overflow resulted from ambiguity in kw arg processing.

The logs(a...; b...) = logs(a..., b) needs to be replaced by something more predictable.

AWSCore.jl/src/Services.jl

Lines 1108 to 1123 in 8a74ec7

 function logs(aws::AWSConfig, operation, args=[]) 

 AWSCore.service_json( 

 aws; 

 service = "logs", 

 version = "2014-03-28", 

 json_version = "1.1", 

 target = "Logs_20140328", 

 operation = operation, 

 args = args) 

 end 

 logs(operation, args=[]) = 

 logs(default_aws_config(), operation, args) 

 logs(a...; b...) = logs(a..., b)

Automatically run tests for downstream packages

As mentioned the current workflow is to test any changes made to AWSCore.jl with the tests from AWSS3.jl and AWSSQS.jl. It would be very useful if we automated the running of these tests in the .travis.yml for AWSCore.jl to validate any changes made here do not break these packages.

Aside: Ideally AWSCore.jl will contain enough tests to avoid breakage without having to rely on the tests from other packages but first we need to take some baby steps.

Fix CI to allow tests to run on branches

Currently if you make a branch off master, the CI tests will fail due to credential issues as seen here https://travis-ci.com/invenia/AWSCore.jl/jobs/168995448.

I'll note that the failing branch is identical to the current master.

Incorrect Dateformat for AWSCredential expiration

A colleague of mine is getting

ERROR: LoadError: MethodError: no method matching *(::Int64, ::LazyJSON.String{String})
Closest candidates are:
*(::Any, ::Any, !Matched::Any, !Matched::Any...) at operators.jl:502
*(!Matched::Missing, ::AbstractString) at missing.jl:139
*(!Matched::LightGraphs.LinAlg.Noop, ::Any) at /root/.julia/packages/LightGraphs/WBY2S/src/linalg/graphmatrices.jl:226
...
Stacktrace:
[1] unix2datetime(::LazyJSON.String{String}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/conversions.jl:49
[2] ecs_instance_credentials() at /root/.julia/packages/AWSCore/GtDFO/src/AWSCredentials.jl:298
[3] instance_credentials() at /root/.julia/packages/AWSCore/GtDFO/src/AWSCredentials.jl:245...

while testing with the most recent changes to AWSCore.

Evidently, I failed to check the date format for the retrieved expiration dates for ECS and EC2 instances were formatted similarly to that returned from an assumed role. They are formatted as dates with a trailing 'Z' (eg: "2019-02-13T22:33:28Z") rather than unix.

PutLogEvents produces StackOverflowError (fixed by `;` before kw args)

julia> logs(aws, "PutLogEvents", logGroupName="ADVISE/ps", logStreamName="sachs-test", logEvents=[["timestamp" => Dates.datetime2unix(now()), "message" => "try 3"]], "sequenceToken" => "49577007586856094356269051243319161428607555192580365458")

end of output is

 [12097] logs(::Dict{Symbol,Any}, ::String, ::Pair{String,String}, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12098] #logs#67(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12099] logs(::Dict{Symbol,Any}, ::String, ::Pair{String,String}, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12100] #logs#67(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12101] logs(::Dict{Symbol,Any}, ::String, ::Pair{String,String}, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12102] #logs#67(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [12103] (::AWSCore.Services.#kw##logs)(::Array{Any,1}, ::AWSCore.Services.#logs, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at ./<missing>:0

Move automatic documentation generation into a separate repo

As its name implies, AWSCore is core AWS functionality. However, there is a significant portion of code in src/AWSAPI.jl that automatically generates documentation and other packages, which (at least to me) seems out of scope. I would propose that this be done separately in some kind of "meta" repository.

General pagination support

The AWS API provides multiple operations that return incomplete results and require several requests to fetch the entire set of results. It would be very nice if there were an interface included in AWSCore which can handle the pagination logic for end-users instead of having users have to roll there own solution.

It does seem possible to make a common interface for this as it does exist in boto3.

Error on gzipped JSON response

AWSCore.jl automatically decodes certain response MIME types, but doesn't pay attention to the Content-Encoding header, so it fails when the data is compressed. Example:

using AWSS3

bucket = "SOMETHING-YOU-OWN"
s = """
    {"foo":"bar"}
    """
gz = open(`gzip`, "r+")
@async begin
    write(gz, s)
    close(gz.in)
end
zipped = read(gz)

s3_put(bucket, "test.json.gz", zipped, "application/json", "gzip")
s3_get(bucket, "test.json.gz")

Here, s3_get throws an error as JSON.jl attempts to parse the gzipped data. This happens in practice with AWS CloudTrail logs, for example.

JuliaWeb/HTTP.jl#256 suggests having the HTTP client handle this transparently, but I don't know whether that's the right solution or not.

Move to EzXML.jl

XMLDict.jl has not been updated since 2018-09-20, we should probably move to a more popular package such as EzXML.jl

Assuming a role from the config

When using AWS CLI you can specify in the configuration file a role_arn and a source_profile which allow you to have a profile that automatically assumes a role:

[profile marketingadmin]
role_arn = arn:aws:iam::123456789012:role/marketingadmin
source_profile = default

Trying to use this with AWSCore results in the credentials being not found.

julia> using AWSCore

julia> ENV["AWS_PROFILE"] = "marketingadmin"
"marketingadmin"

julia> AWSCore.aws_config()
Dict{Symbol,Any} with 2 entries:
  :creds  => (notfound, not...)…
  :region => "us-east-1"

Make it easier to pass return_raw=true to service endpoints

Currently to get raw output you need to call service_query/service_json/etc. directly and pass an additional keyword argument. It would be nice to have this available from the service endpoint, if possible.

I needed this to avoid using XMLDict, which returns inconsistently-structured results when a list of possibly-many items contains 1 item vs many items.

UndefVarError: hostname not defined

Hello,

I have started to get this error, which seems to be due to some inconsistency between HTTP.jl and AWSCore.jl is there some simple fix for this?

ERROR: LoadError: LoadError: UndefVarError: hostname not defined
Stacktrace:
[1] http_get at /root/.julia/v0.6/AWSCore/src/http.jl:59 [inlined]
[2] ecs_instance_credentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:219
[3] AWSCore.AWSCredentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:77
[4] include_from_node1(::String) at ./loading.jl:576

thanks a lot for help.
Tomas

CI Failing - Unable to reference AWSException

Problem

https://travis-ci.org/JuliaCloud/AWSCore.jl/builds/580418604

The current master branch is failing on Travis CI for the following line:

https://github.com/JuliaCloud/AWSCore.jl/blob/master/test/runtests.jl#L569

With the exception:

Exception: Error During Test at /Users/mattbr/.julia/dev/AWSCore.jl/test/runtests.jl:557
  Got exception outside of a @test
  UndefRefError: access to undefined reference

ReceiveMessage produces MethodError

The queue in question is a FIFO queue, if that is relevant. The message does end up being in flight.

julia> m = sqs(aws, "ReceiveMessage"; QueueUrl=queue_url)
ERROR: MethodError: no method matching Base.AbstractIOBuffer{Array{UInt8,1}}(; maxsize=380)
Stacktrace:
 [1] parse_string(::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/specialized.jl:17
 [2] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:162
 [3] parse_object(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:223
 [4] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:166
 [5] parse_array(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:196
 [6] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:168
 [7] parse_object(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:223
 [8] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:166
 [9] parse_object(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:223
 [10] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:166
 [11] parse_object(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:223
 [12] parse_value(::JSON.Parser.ParserContext{DataStructures.OrderedDict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:166
 [13] #parse#1(::Type{DataStructures.OrderedDict}, ::Type{Int64}, ::Function, ::String) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:393
 [14] (::JSON.Parser.#kw##parse)(::Array{Any,1}, ::JSON.Parser.#parse, ::String) at ./<missing>:0
 [15] do_request(::Dict{Symbol,Any}) at /home/sachs/.julia/v0.6/AWSCore/src/AWSCore.jl:479
 [16] #service_query#7(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}) at /home/sachs/.julia/v0.6/AWSCore/src/AWSCore.jl:241
 [17] (::AWSCore.#kw##service_query)(::Array{Any,1}, ::AWSCore.#service_query, ::Dict{Symbol,Any}) at ./<missing>:0
 [18] sqs(::Dict{Symbol,Any}, ::String, ::Array{Any,1}) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1574
 [19] #sqs#95(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1585
 [20] (::AWSCore.Services.#kw##sqs)(::Array{Any,1}, ::AWSCore.Services.#sqs, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at ./<missing>:0

Segmentation fault

Hi,

from time to time (if I submit larger number of jobs into AWS Batch it is more frequent), I get segmentation fault, which occurs at following line of code

const aws = AWSCore.aws_config(region = "eu-west-1", creds = AWSCore.AWSCredentials())

the error is long and I do not know, if it comprehensible. I used Julia 0.6.1 and the latest tagged versions of HTTP, AWSCore and AWSS3.

Thanks for help,
Tomas

signal (11): Segmentation fault
while loading /opt/src/ReadData/src/ReadData.jl, in expression starting on line 23
unknown function (ip: 0x7f925c6a5c7f)
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3396 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4131
emit_jlcall at /buildworker/worker/package_linux64/build/src/codegen.cpp:3241
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3415 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4131
emit_assignment at /buildworker/worker/package_linux64/build/src/codegen.cpp:3755 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4155
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4060 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6244
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1256
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3396 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4131
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6097
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1256
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3396 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4131
emit_assignment at /buildworker/worker/package_linux64/build/src/codegen.cpp:3755 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4155
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4060 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6244
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1256
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3396 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4131
emit_assignment at /buildworker/worker/package_linux64/build/src/codegen.cpp:3849 [inlined]
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4155
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4060 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6244
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1256
jl_compile_for_dispatch at /buildworker/worker/package_linux64/build/src/gf.c:1668
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:307 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:354 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:75
eval at /buildworker/worker/package_linux64/build/src/interpreter.c:242
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:74
eval at /buildworker/worker/package_linux64/build/src/interpreter.c:242
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:74
eval at /buildworker/worker/package_linux64/build/src/interpreter.c:242
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:543
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:692
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:592
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:205
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:480
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:873
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:616
include_from_node1 at ./loading.jl:576
unknown function (ip: 0x7f92501575d2)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:75
eval at /buildworker/worker/package_linux64/build/src/interpreter.c:242
jl_interpret_toplevel_expr at /buildworker/worker/package_linux64/build/src/interpreter.c:34
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:577
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:496
eval at ./boot.jl:235
unknown function (ip: 0x7f9258ed1acf)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
_require at ./loading.jl:479
require at ./loading.jl:405
unknown function (ip: 0x7f92501660f2)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1424 [inlined]
eval_import_path_ at /buildworker/worker/package_linux64/build/src/toplevel.c:403
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:430 [inlined]
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:495
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:873
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:616
include_from_node1 at ./loading.jl:576
unknown function (ip: 0x7f9258f1518b)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
include at ./sysimg.jl:14
unknown function (ip: 0x7f9258dad63b)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
process_options at ./client.jl:305
_start at ./client.jl:371
unknown function (ip: 0x7f9258f23b28)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:1926
jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1424 [inlined]
true_main at /buildworker/worker/package_linux64/build/ui/repl.c:127
main at /buildworker/worker/package_linux64/build/ui/repl.c:264
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4016bc)
Allocations: 23042560 (Pool: 23038249; Big: 4311); GC: 44
--: line 1: 213 Segmentation fault /opt/julia-0.6.1/bin/julia --depwarn=no -p 8 -L setpath.jl ${*}

Auto-Generate API Wrappers

Overview

We want to automatically generate both low and high level API wrappers for each AWS Service.

One thing to note, when generating a service. If its protocol is one we have not seen yet an alarm should be triggered.

InvalidSignatureException not retrying

Recently I have been getting:

[warn | CloudWatchLogs]: InvalidSignatureException -- AWSException
HTTP.ExceptionRequest.StatusError(400, HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
x-amzn-RequestId: 86af99a0-5c68-11e9-9ca9-b5781024a1e7
Content-Type: application/x-amz-json-1.1
Content-Length: 151
Date: Thu, 11 Apr 2019 14:46:01 GMT
Connection: close

{
    "__type": "InvalidSignatureException",
    "message": "Signature expired: 20190411T144043Z is now earlier than 20190411T144101Z (20190411T144601Z - 5 min.)"
}
""")

This error should be caught and retried in do_request, but there seems to be a failure parsing the StatusError.

Use Min.Io to test S3

As part of #99 we've introduced a dependency on a 3rd party GCS Bucket for testing S3 capabilities outside of AWS.

To do this properly we should use Min.IO(or something similar) to remove this dependency and keep everything centralized in this package.

We should install Min.IO on the TravisCI runner, setup and configure it and then update the tests here to use that instead of the 3rd party bucket.

Define AWSConfig as a struct?

Considering the existence of aws_config(), it seems to me that it would make more sense to define AWSConfig as a struct rather than a SymbolDict.

I'm curious as to why this design decision was made and whether people think this change may or may not be a good idea.

error during exception handling: tried to access `cause` field of `ErrorException`

An error can occur during exception handling wherein the (non-existant) cause field of an ErrorExcept (or other error) can be accessed. This occurs on line 446 of AWSCore.jl.

PR incoming.

Tag a new release

The assume role changes in #34 are really useful and unfortunately those changes only exist on master. Could you tag and register a new release?

duplicate of HTTP.jl ?

@samoconnor You have added a vendor folder and copied the HTTP.jl staff, isn't that a duplication?

WARNING: Method definition (::Type{String})(Base.SubArray{UInt8, 1, P, I, L} where L where I where P) in module HTTP at /usr/people/jingpeng/.julia/v0.6/AWSCore/src/vendor/HTTP.jl/compat.jl:37 overwritten in module HTTP at /usr/people/jingpeng/.julia/v0.6/HTTP/src/compat.jl:42.

add support for the AWS_PROFILE environment variable

For choosing among multiple profiles in credentials file, described in this AWS Security Blog post.

Support IMDSv2 in credentials

https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Getting HTTP 400 from AWSCredentials on AWS EC2 instance

Hi,
I am getting HTTP 400 error on an AWS EC2 instance with a EC2 role assigned, when trying to obtain AWSCredentials:

ERROR: LoadError: LoadError: HTTP.StatusError: received a '400 - Bad Request' status in response
Stacktrace:
 [1] request(::HTTP.Client, ::HTTP.Request, ::HTTP.RequestOptions, ::Bool, ::Array{HTTP.Response,1}, ::Int64, ::Bool) at /root/.julia/v0.6/HTTP/src/client.jl:347
 [2] #request#41(::HTTP.RequestOptions, ::Bool, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client, ::HTTP.Request) at /root/.julia/v0.6/HTTP/src/client.jl:360
 [3] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client, ::HTTP.Request) at ./<missing>:0
 [4] #request#42(::Dict{String,String}, ::HTTP.FIFOBuffers.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client, ::String, ::HTTP.URIs.URI) at /root/.julia/v0.6/HTTP/src/client.jl:380
 [5] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client, ::String, ::HTTP.URIs.URI) at ./<missing>:0
 [6] macro expansion at /root/.julia/v0.6/AWSCore/src/http.jl:22 [inlined]
 [7] macro expansion at /root/.julia/v0.6/Retry/src/repeat_try.jl:198 [inlined]
 [8] http_request(::Dict{Symbol,Any}) at /root/.julia/v0.6/AWSCore/src/http.jl:20
 [9] http_request(::String, ::String) at /root/.julia/v0.6/AWSCore/src/http.jl:48
 [10] ec2_metadata(::String) at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:167
 [11] ec2_instance_credentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:184
 [12] AWSCore.AWSCredentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:73

The problem seems to be in function ec2_metadata, where is http_request("http://169.254.169.254", ..., but in http_request, the host is put into headers: headers = Dict("Host" => host). The host should probably not contain the http prefix.

AWS Account Number not returned properly

https://github.com/JuliaCloud/AWSCore.jl/blob/v0.6.2/src/AWSCredentials.jl#L220

When the creds.account_number from the AWSConfig is an empty string, we are instead returning the user ARN.

This doesn't seem correct.

ERROR: Unexpected end of input

The log stream was actually created. Same error with DeleteLogStream.

julia> using AWSCore.Services.logs

julia> aws = AWSCore.aws_config();

julia> logs(aws, "CreateLogStream", logGroupName="ADVISE/ps", logStreamName="sachs-test")
ERROR: Unexpected end of input
Line: 0
Around: ......
           ^

Stacktrace:
 [1] _error(::String, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:140
 [2] byteat at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:50 [inlined]
 [3] parse_value(::JSON.Parser.ParserContext{Dict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:160
 [4] #parse#1(::Type{Dict{String,Any}}, ::Type{Int64}, ::Function, ::String) at /home/sachs/.julia/v0.6/JSON/src/Parser.jl:393
 [5] do_request(::Dict{Symbol,Any}) at /home/sachs/.julia/v0.6/AWSCore/src/AWSCore.jl:465
 [6] #service_json#8(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}) at /home/sachs/.julia/v0.6/AWSCore/src/AWSCore.jl:262
 [7] (::AWSCore.#kw##service_json)(::Array{Any,1}, ::AWSCore.#service_json, ::Dict{Symbol,Any}) at ./<missing>:0
 [8] logs(::Dict{Symbol,Any}, ::String, ::Array{Any,1}) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1110
 [9] #logs#67(::Array{Any,1}, ::Function, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at /home/sachs/.julia/v0.6/AWSCore/src/Services.jl:1123
 [10] (::AWSCore.Services.#kw##logs)(::Array{Any,1}, ::AWSCore.Services.#logs, ::Dict{Symbol,Any}, ::Vararg{Any,N} where N) at ./<missing>:0

Anonymous data access?

Currently the user is required to provide valid credentials in order to generate and aws_config. However, if a user only wants to access public buckets, this should not be necessary. For example in the python package s3fs https://github.com/dask/s3fs/blob/master/s3fs/core.py#L107 there is an option to connect to an S3 bucket without explicit authentication by the user. I think it is using boto3 for the connection, which explicitly allows to set aws_access_key_id=None, aws_secret_access_key=None https://github.com/boto/boto3/blob/02bdc5fa3fb6818f548a73d5e42dad84a946b52b/boto3/session.py#L48

Would anonymous access to public AWS S3 buckets be possible with this package as well or is there some fundamental design decision that makes this impossible?

NoAuthRequest Test Fix

https://github.com/JuliaCloud/AWSCore.jl/pull/88/files#diff-cd01adffb4cad6890105649ea6992e2aR40

Not to get too far out-of-scope with #88 but these tests run deterministically of the files existing or not. It seems that the intended behaviour was if these files exist, ensure the proper response was given back. If an error is thrown make sure it's either a AccessDenied (the file is not private) or EntityDoesNotExist (the file was removed).

These tests should generate their own public files, and ensure that it can access them.

Tag a new release

Implement refreshing credentials

Currently it's not possible to refresh credentials unless you know how they're created, and AWSCredentials doesn't store that information.

This is how botocore handles things: https://github.com/boto/botocore/blob/879f8440a4e9ace5d3cf145ce8b3d5e5ffb892ef/botocore/credentials.py#L269

Storing a refresh time and refreshing function seems like the easiest way to implement.

Consider setting HTTP.jl's idle_timeout option

idle_timeout stops pooled connections from being reused when they've been sitting idle for too long (in which case the server has probably already dropped the other end of the connection).

It seems that there isn't a one-size-fits-all idle_timeout for AWS services. IAM seems to process requests on a connection that has been idle for up to about 50 seconds, but S3 connections seem to become unusable after about 5 seconds.

bytestring not defined in AWSCredentials

When running AWSS3 and AWSCore on an EC2 instance, it fails inside ec2_metadata function:

ERROR: LoadError: LoadError: UndefVarError: bytestring not defined
Stacktrace:
[1] ec2_metadata(::String) at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:164
[2] ec2_instance_credentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:180
[3] AWSCore.AWSCredentials() at /root/.julia/v0.6/AWSCore/src/AWSCredentials.jl:70

Any plans for supporting SES ?

Thanks!

Use HTTP.jl's retry layer instead of Retry in http_request

The check there seems pretty similar to the RetryLayer in HTTP.jl, but the RetryLayer handles more of HTTP.jl's failure cases. This would require JuliaWeb/HTTP.jl#259 in order to handle the AWSCore-specific failure cases (e.g., ParseError).

Improve exception when no credentials are found

If you attempt to use aws_config() where there are no credentials setup you get a confusing error message:

ERROR: LoadError: LoadError: IOError(Base.IOError("connect: connection timed out (ETIMEDOUT)", -110) during request(http://169.254.169.254/latest/meta-data/iam/info))

Stacktrace:
 [1] wait_connected(::Sockets.TCPSocket) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.3/Sockets/src/Sockets.jl:520
 [2] connect at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.3/Sockets/src/Sockets.jl:555 [inlined]
 [3] connect at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.3/Sockets/src/Sockets.jl:541 [inlined]
 [4] #getconnection#14(::Bool, ::Int32, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:require_ssl_verification, :iofunction, :verbose),Tuple{Bool,Nothing,Int32}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{Sockets.TCPSocket}, ::SubString{String}, ::SubString{String}) at /home/omus/.julia/packages/HTTP/lZVI1/src/ConnectionPool.jl:568
 [5] #getconnection at ./none:0 [inlined]
 [6] #getconnection#11(::Int32, ::Int32, ::Int32, ::Int32, ::Bool, ::Base.Iterators.Pairs{Symbol,Union{Nothing, Int32},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :verbose),Tuple{Nothing,Int32}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}, ::SubString{String}, ::SubString{String}) at /home/omus/.julia/packages/HTTP/lZVI1/src/ConnectionPool.jl:517
 [7] (::HTTP.ConnectionPool.var"#kw##getconnection")(::NamedTuple{(:reuse_limit, :iofunction, :verbose, :require_ssl_verification),Tuple{Int32,Nothing,Int32,Bool}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}, ::SubString{String}, ::SubString{String}) at ./none:0
 [8] #request#1(::Nothing, ::Type, ::Int32, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int32,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at /home/omus/.julia/packages/HTTP/lZVI1/src/ConnectionRequest.jl:41
 [9] (::HTTP.var"#kw##request")(::NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int32,Bool}}, ::typeof(HTTP.request), ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at ./none:0
 [10] #request#1(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int32,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at /home/omus/.julia/packages/HTTP/lZVI1/src/ExceptionRequest.jl:19
 [11] #request at ./none:0 [inlined]
 [12] #request#1(::VersionNumber, ::String, ::Nothing, ::Nothing, ::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :require_ssl_verification),Tuple{Int32,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at /home/omus/.julia/packages/HTTP/lZVI1/src/MessageRequest.jl:51
 [13] (::HTTP.var"#kw##request")(::NamedTuple{(:verbose, :require_ssl_verification),Tuple{Int32,Bool}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at ./none:0
 [14] http_request(::Dict{Symbol,Any}) at /home/omus/.julia/packages/AWSCore/nRYAp/src/http.jl:42
 [15] http_get(::String) at /home/omus/.julia/packages/AWSCore/nRYAp/src/http.jl:80
 [16] macro expansion at /home/omus/.julia/packages/Mocking/hFQQ4/src/mock.jl:29 [inlined]
 [17] _ec2_metadata(::String) at /home/omus/.julia/packages/AWSCore/nRYAp/src/AWSCredentials.jl:263
 [18] ec2_instance_credentials() at /home/omus/.julia/packages/AWSCore/nRYAp/src/AWSCredentials.jl:282
 [19] #AWSCredentials#3(::Nothing, ::Type{AWSCore.AWSCredentials}) at /home/omus/.julia/packages/AWSCore/nRYAp/src/AWSCredentials.jl:146
 [20] Type at ./none:0 [inlined]
 [21] aws_config() at /home/omus/.julia/packages/AWSCore/nRYAp/src/AWSCore.jl:115
...

This seems not very user friendly.

Detect clock skew

A precursor to #45

When we get "signature exprired" we can check the time that AWS thinks it is (in the message) and compare that with the now time and the signature expiry time. If the difference between AWS time and the now time accounts for the difference in the AWS time and the signature expiry time, we can warn the user about clock skew.

Support passing profile to `aws_config`

We have a function now that does this (simplified):

function _AWSConfig(;
    profile=nothing,
)
    if profile === nothing
        profile = get(ENV, "AWS_PROFILE", nothing)
    end

    withenv("AWS_PROFILE" => profile) do
        AWSConfig(
            :creds => AWSCredentials(),
        )
    end
end

It would be nice if aws_config supported a profile argument so we didn't need to rely on environment variables.

Route53 endpoint no longer working

Original Issue: JuliaCloud/AWSSDK.jl#13

The AWSCLI submits requests to: route53.amazonaws.com, while AWSCore.jl will try to submit to route-53-${Region}.amazonaws.com which ends up failing.

The problem stems from this bit of code, here which needs to be changed to special case route53 (and possibly other AWS Services).

	function logs(aws::AWSConfig, operation, args=[])

	AWSCore.service_json(
	aws;
	service = "logs",
	version = "2014-03-28",
	json_version = "1.1",
	target = "Logs_20140328",
	operation = operation,
	args = args)
	end

	logs(operation, args=[]) =
	logs(default_aws_config(), operation, args)

	logs(a...; b...) = logs(a..., b)

juliacloud / awscore.jl Goto Github PK

awscore.jl's People

Contributors

Stargazers

Watchers

Forkers

awscore.jl's Issues

Problem

Overview

Recommend Projects

Recommend Topics

Recommend Org

Jobs