Failing to deploy about slam

[miguelgrinberg]

Could you be having permission issues? Are all the files in the .lambda_uploader_temp directory owned by root?

[seanmckaybeck]

Yes, everything is owned by root.

[miguelgrinberg]

Does the command

cat /root/infant/.lambda_uploader_temp/lambda_package/.venv/lib/python3.5/types.py

work from your terminal?

[seanmckaybeck]

It does not which is odd since it appears in the output of find.

$ ll /root/infant/.lambda_uploader_temp/lambda_package/.venv/lib/python3.5/types.py
lrwxrwxrwx 1 root root 64 Jun 18 04:12 /root/infant/.lambda_uploader_temp/lambda_package/.venv/lib/python3.5/types.py -> /usr/lib/python3./.lambda_uploader_temp/lambd
a_package5/types.py

[miguelgrinberg]

Yeah, totally weird. Does the error repeat if you trash .lambda_uploader_temp? That's a temporary directory that a dependency of this project creates while it builds the lambda zip file.

[seanmckaybeck]

I wiped the folder out and it still happens.

[seanmckaybeck]

It does seem odd that the link is -> /usr/lib/python3./.lambda_uploader_temp/lambd a_package5/types.py. It's missing the 5 after python3.

[miguelgrinberg]

Do you have anything else on this droplet? I would make a new one if possible.

[seanmckaybeck]

Yeah this droplet is pretty loaded. I'm new to DynamoDB so I wanted to test everything locally, but java appears to be non-functional on PythonAnywhere (where I was working on the project) so I shifted to the droplet.

[miguelgrinberg]

What happens if you create a 3.5 virtualenv by hand? Is types.py correct in that case?

[seanmckaybeck]

Well I tried on a different system and got further along in the process, but still ended up with the same problem, just a different file.

$ slam deploy
Building lambda package...
Unexpected error: [Errno 2] No such file or directory: '/home/thaweatherman/infant-tracker/.lambda_uploader_temp/lambda_package/.venv/lib/python3.5/imp.py' (FileNotFo
undError)
See file slam_error.log for additional details.

[miguelgrinberg]

What system is this? I have been using this tool mostly on Mac, but have tested it also on Ubuntu 16.04 without problems. There is clearly something odd in the way your virtual environments get created, but don't know exactly what. Do you get bad links in your virtualenvs if you create them by hand?

[seanmckaybeck]

I haven't created one by hand before since the tool has always just worked. PythonAnywhere is running Ubuntu 14.04 and my droplet is running 12.04.

[seanmckaybeck]

Is there some odd requirement that the environment I'm using slam in is using 3.6? I have a 3.5.2 venv that I'm using.

[miguelgrinberg]

Lambda does not support 3.5, only 3.6 and 2.7. Unless you are using something that was compiled specifically for 3.5 I doubt this would be a problem, a venv built for 3.5 should be okay as a source to build a lambda package for 3.6. But if you want to be sure, there is only one way to find out, just install 3.6 and give that a try.

[seanmckaybeck]

I compiled 3.6.1 from source for my Ubuntu 12.04 and ended up with the same error from slam.

[miguelgrinberg]

Okay. Let's try something else. Based on your comments above, after you get the error you have three virtualenvs in different locations, correct? The one in .venv is probably yours, the one in .slam/venv is created by this tool, and then one in .lambda_uploader_temp/lambda_package/.venv is created by the lambda_uploader package while it builds the lambda zip file.

Can you check if the types.py or whatever other files gives you the error is viewable/editable in all these three? I'm going to bet it is not okay on the lambda uploader venv, and fine on the other two, which suggests the lambda uploader is breaking links when it builds its temporary venv.

[seanmckaybeck]

It is viewable/editable in .venv and .slam/venv. The one under .lambda... shows a bad link.

lrwxrwxrwx 1 root root 70 Jun 22 02:44 ./.lambda_uploader_temp/lambda_package/.venv/lib/python3.6/types.py -> /usr/local/lib/python3./.lambda_uploader_temp/lambda_pac
kage6/types.py

/usr/local/lib/python3./.lam... seems incorrect. I just created a 16.04 droplet and am going to try it on there now.

[miguelgrinberg]

What version of lambda_uploader do you have in your main venv?

[seanmckaybeck]

1.2.0

[seanmckaybeck]

I set up a 3.6 environment on 16.04 and got the following:

(.venv) root@temp-ubuntu-16:~/infant-tracker# slam deploy
Building lambda package...
Unexpected error: [Errno 2] No such file or directory: '/root/infant-tracker/.lambda_uploader_temp/lambda_package/.venv/bin/python3' (FileNotFoundError)
See file slam_error.log for additional details.

Doesn't complain about types.py, but still the same general problem.

[miguelgrinberg]

Hmm. I'm really out of ideas on this. I now tested on an ubuntu 16.04 box, running Python 3.5.2, and lambda_uploader 1.2.0. The only difference is that I do not have root logins for security. I tried on a regular account, and also with sudo, and I'm getting my package built.

Is there anything unusual about your file system? Anything that can make symbolic links do weird stuff? I'm testing on a digital ocean droplet with default storage options.

[seanmckaybeck]

I'm just using a default droplet as well. Nothing should be special about the file system.

Could my project layout be causing an issue?

app
    resources
        all the resource py files for flask-restful endpoints
    __init__.py - creates Flask() and Api()
    models
        the dynamodb models 
    utils.py

[seanmckaybeck]

I made an app.py file that includes only the following:

from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
    return 'Hello'

It seems to get past the prior problem (I'm getting access denied from S3 despite setting up my credentials, but that's a separate issue).

As an aside, what permissions does my IAM user need to have to work with slam? I added AWSLambdaFullAccess, AmazonS3FullAccess, AmazonAPIGatewayInvokeFullAccess, AmazonAPIGatewayPushToCloudWatchLogs, AmazonAPIGatewayAdministrator, and AWSLambdaExecute.

[seanmckaybeck]

It seems to build the project from the advanced tutorial just fine, I just keep getting hung up on required permissions. If you had a list of the required permissions for this in the docs that would be helpful.

[miguelgrinberg]

I haven't tested a restricted set of permissions, I'm using an account that has full access, but I think you should be fine if you have full access to Lambda, S3, API Gateway and Cloudformation. What permissions does the app complain about?

[seanmckaybeck]

It was complaining about S3 despite granting full S3 access, but that seems to have resolved itself. Currently it is complaining about cloudformation, which I had to create a special policy for that seems to have solved it, but everything is deployed in a half state so I can't properly deploy or delete. It seems a root key would be much simpler, but I might as well figure it out.

My personal project is in a private repo. I can give you access if you'd like to try deploying it in your environment. Clearly something is wonky in my setup.

[miguelgrinberg]

Well, this is what happens when I try your project here:

(venv) infant-tracker $ slam build
Building lambda package...
lambda_package.20170623_164955.zip has been built successfully.

[seanmckaybeck]

So you're able to get it to build. Is this Ubuntu 16.04, Python 3.5.2?

[miguelgrinberg]

Yes. I tested it on my mac (3.6.1) and on an Ubuntu 16.04 droplet with the stock Python 3.5.2.

[seanmckaybeck]

I think I have figured it out. Using the same setup as before but with the virtual env named venv instead of .venv it worked. It didn't deploy because it didn't like my dynamo table names, but it builds! Thanks for going through this all with me.

Side question: I'm new to dynamodb and have an operation I'm not sure on how to perform. Two of my models have a datetime field and I want to be able to get the entry with the most recent datetime. I assume I'd need to build an index, but none of the conditional operators match up with what I want. Am I stuck grabbing everything and iterating to find the most recent?

[miguelgrinberg]

@ThaWeatherman very weird. FWIW, I am able to build the package using .venv as the main virtual environment.

The date problem is tricky. As you know the key is composed of a hash portion and a sort portion. If you need to get the most recent item out of the entire table, I think you can build a global secondary index that uses the same hash key for every single item in your table. Then you can run a query with ScanIndexForward=False and a limit of 1 to get the most recent date.

[seanmckaybeck]

With the datetime as the sort key, right? Thanks.

I'll try again with .venv and see what happens. That's the only thing I changed in my setup from scratch. Quite odd.

[miguelgrinberg]

With the datetime as the sort key, right? Thanks.

Yes, the datetime would be the sort key. I honestly never used the native datetime type from DynamoDB, what I do is write the time as a string to a text field, but this is just because in the past there was no native datetime type.

[seanmckaybeck]

I was getting an error when deploying complaining about the bucket name. I found some docs that say uppercase characters aren't allowed and the generated bucket name in slam.yaml had uppercase in it.

I've gotten past that now in the deployment process but am still fighting with deployment errors as a result of not using root creds.

[miguelgrinberg]

Good catch on the upper case bucket names. I'll fix that, thanks.

[seanmckaybeck]

I keep getting this during the deployment process:

2017-06-27 01:15:58,527 - slam - ERROR - [merry] Exception caught
Traceback (most recent call last):
  File "/root/infant/venv/lib/python3.5/site-packages/merry.py", line 26, in wrapper
    ret = f(*args, **kwargs)
  File "/root/infant/venv/lib/python3.5/site-packages/climax.py", line 194, in wrapper
    ctx = func(**filtered_args)
  File "/root/infant/venv/lib/python3.5/site-packages/slam/cli.py", line 411, in deploy
    waiter.wait(StackName=config['name'])
  File "/root/infant/venv/lib/python3.5/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/root/infant/venv/lib/python3.5/site-packages/botocore/waiter.py", line 321, in wait
    last_response=response,
botocore.exceptions.WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state

Any ideas? CloudFormation is in a ROLLBACK_COMPLETE state.

[miguelgrinberg]

@ThaWeatherman you should be able to see a more specific error in the cloudformation console. That will also show the progress on creating the stack, and which resource had the failure.

[seanmckaybeck]

I ended up deleting the stack and redeploying using root keys and it all worked. Clearly some other permissions are missing. If only boto told you what they are...

Thanks for sticking with me on this ticket. Now that it's deployed I can get to work on testing.

[seanmckaybeck]

Do I need to add my model attributes to slam.yaml? I only see the hash keys in there. Also, where should I look for error logs in AWS? I'm getting a 502 and don't know why.

[miguelgrinberg]

DynamoDb is a schema-less database. The only attributes that need to be declared are those that are in primary or secondary indexes.

What service is returning 502? Is this your application or one of the AWS services?

[seanmckaybeck]

I made a POST request to the /api/user endpoint on the dev stage and got a 502 back. I've tested that one locally and it worked, so I assume it's something on the AWS end.

[miguelgrinberg]

It's probably API Gateway. What does the body of the 502 response say?

I think you are going to find something about this error in the output from slam logs, also.

[seanmckaybeck]

{'message': 'Internal server error'}

slam logs isn't outputting anything.

[miguelgrinberg]

Look in CloudWatch to see if there is anything.

One possibility is that the URL that you are using is incorrect. Make sure you are using the complete URL for the stage (including the /dev at the end) before the portion of the path that belongs to your service.

[seanmckaybeck]

I'm using the full URL. CloudWatch gave me the following: (e7330f2d-5adb-11e7-92f0-993ba046a67d) Execution failed due to configuration error: Malformed Lambda proxy response. So I checked the Lambda logs and got:

No module named 'click': ModuleNotFoundError
Traceback (most recent call last):
File "/var/task/handler.py", line 39, in lambda_handler
from infant_api import app as app # noqa
File "/var/task/infant_api.py", line 1, in <module>
import click
ModuleNotFoundError: No module named 'click'

So I guess it isn't properly handling my dependencies? My requirements file doesn't list click which I figured was fine since it gets installed by flask. I dug through the slam source a few days ago and saw that it does a pip install on everything in the requirements file so I assumed that not listing dependencies like click was fine.

[miguelgrinberg]

Run a slam build, then look in the resulting zip file what folders you have in the top level. The flask package should be there, along with click and all the other dependencies. The source of these packages is the virtualenv stored in .slam/venv, so you can check if click got installed there as well.

[seanmckaybeck]

It looks like everything is in the venv but the zip files doesn't include the venv. It also is picking up my .git directory. Is there an ignore option?

[miguelgrinberg]

The venv is not going to be in the zip file. Only the packages installed in the venv will be included, at the root level. So you should see a flask directory, a click directory and so on, at the same level as your application.

[seanmckaybeck]

Those aren't ending up in the zip file

Archive:  lambda_package.20170629_000828.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      25  Defl:N       27  -8% 2017-06-29 00:08 a61dec36  .gitignore
      79  Defl:N       70  11% 2017-06-29 00:08 43175dc4  requirements.txt
     924  Defl:N      309  67% 2017-06-29 00:08 49ce76e5  infant_api.py
       0  Defl:N        2   0% 2017-06-29 00:08 00000000  slam_error.log
    4688  Defl:N     1737  63% 2017-06-29 00:08 7c7c2180  handler.py
   65201  Defl:N    51736  21% 2017-06-29 00:08 e4090d82  lambda_package.20170627_025812.zip
    3024  Defl:N     1007  67% 2017-06-29 00:08 5aa13846  slam.yaml
      94  Defl:N       83  12% 2017-06-29 00:08 637ce7e6  app/config.py
     682  Defl:N      336  51% 2017-06-29 00:08 dcc15ac9  app/utils.py
     913  Defl:N      324  65% 2017-06-29 00:08 4a2300dc  app/__init__.py
    6835  Defl:N     1175  83% 2017-06-29 00:08 9daa293c  app/resources/diaper.py
    3103  Defl:N      784  75% 2017-06-29 00:08 2cd2836e  app/resources/child.py
    1385  Defl:N      491  65% 2017-06-29 00:08 b77a11e3  app/resources/user.py
     699  Defl:N      259  63% 2017-06-29 00:08 3e83e955  app/resources/__init__.py
    7237  Defl:N     1218  83% 2017-06-29 00:08 8389d982  app/resources/feeding.py
    1395  Defl:N      574  59% 2017-06-29 00:08 5994960d  app/models/diaper.py
    1303  Defl:N      571  56% 2017-06-29 00:08 51525822  app/models/child.py
    1674  Defl:N      680  59% 2017-06-29 00:08 253c5aff  app/models/user.py
     165  Defl:N       99  40% 2017-06-29 00:08 6f69c3b6  app/models/__init__.py
    1494  Defl:N      598  60% 2017-06-29 00:08 1606109b  app/models/feeding.py
    4688  Defl:N     1737  63% 2017-06-29 00:08 7c7c2180  .slam/handler.py
      73  Defl:N       63  14% 2017-06-29 00:08 1f078b37  .git/description

The rest of the files are all from .git. This is the result of slam build --rebuild-deps