Comments (8)
Hi Jesse!
Modify the S3 cache backend code so that if the user does not have credentials, it will connect as an anonymous user rather than erroring with "No handler was ready to authenticate... Check your credentials" and disabling the S3CacheBackend.
Solutions I can think of:
- Detect whether credentials are available. But how do you suggest that pip-accel detects whether the user has credentials? The S3 credentials can be set in environment variables, they can be set in a Boto configuration file, they can come from the AWS EC2 metadata service, etc. So far I was able to hide this complexity from pip-accel. The risk is that if I re implement all these checks I may get them wrong. Or a new release of Boto adds yet another type of providing credentials and pip-accel doesn't know about it and so forces anonymous connections.
- Maybe pip-accel can catch the No handler was ready to authenticate exception and retry with
anon=True
when that happens. But then pip-accel will always need an HTTP response to find out that no credentials are available, and only then can it respond by issuing anonymous requests. That feels somewhat sub optimal :-). - We can add another pip-accel setting which explicitly enables anonymous connections to S3.
from pip-accel.
Hey Peter!
Yeah, I agree:
- We definitely don't want to replicate the code paths.
- This one is sub-optimal but was actually my first inclination because
- This one would be sub-optimal for my use case. I want to enable pip-accel on an OpenSource repo for which I allow anybody to use my S3 buckets (read-only). And I want us to be able to use the same contents in the pip-accel.conf. I.e. forks will be using the file from my master, and they should be able to run on travis without changing the config in this file, or setting up a AWS user and configuring boto. I could override with environment variables for pip-accel configuration but I'd rather not have to do that.
Perhaps another option might be to:
4. Use the boto Config object (or better - use another downstream method, I just looked really quickly) to determine if credentials exist.
>>> from boto import Config
>>> Config().dump()
This shows my credentials if they are configured and shows no credentials if not.
from pip-accel.
Hi Jesse,
I believe pip-accel version 0.22 resolves your problem.
I went with option two in the end because I realized this past weekend that if I want to allow people to use Boto's "automagic" support for the EC2 metadata service there is no way to avoid the HTTP request so I should just accept it and move on. Anyway, the EC2 metadata service uses the IP address 169.254.169.254
which is a link-local IPv4 address range so I don't think this will cause problems in practice (e.g. hanging on a timeout).
Can you confirm whether pip-accel 0.22 resolves your problem? Thanks for the feature request!
from pip-accel.
Peter - I think it seems to work, but with a few caveats.
- boto "Caught exception reading instance data" error messages get output when trying to read the metadata. See this travis build where I turned on verbose mode so you can see the flow (but the errors get printed out regardless of verbose mode).
- If you do not set
s3-readonly = yes
then when it fails uploading a file, the S3 cache backend is turned off. See this travis build
We would probably catch these and any other corner case issues or bugs by writing unit tests for the S3CacheBackend. I could totally help do that if you'd like.
from pip-accel.
boto "Caught exception reading instance data" error messages get output when trying to read the metadata. See this travis build where I turned on verbose mode so you can see the flow (but the errors get printed out regardless of verbose mode).
This is now fixed: I silenced the Boto logger. There is an escape hatch in case someone ever does need to debug the usage of Boto inside pip-accel (for details refer to ec426ea).
If you do not set s3-readonly = yes then when it fails uploading a file, the S3 cache backend is turned off. See this travis build
I'll see if I can improve on this as well.
We would probably catch these and any other corner case issues or bugs by writing unit tests for the S3CacheBackend. I could totally help do that if you'd like.
There are unit tests for the S3 backend, or rather the test suite runs twice for each supported version of Python: Once with the S3 cache backend disabled and once with the S3 cache backend enabled. This uses FakeS3 so I don't need to create and pay for an actual Amazon S3 bucket.
from pip-accel.
If you do not set s3-readonly = yes then when it fails uploading a file, the S3 cache backend is turned off.
I believe this is now fixed as well. As far as I can tell FakeS3 doesn't support simulating read only bucket access so I had to figure out how to create a read only IAM policy in Amazon S3 in order to test this manually. For previous manual testing I had used the following IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
For this testing I switched to the following read only policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:Get*", "s3:List*"],
"Resource": "*"
}
]
}
In my tests the latest version correctly falls back to read only Amazon S3 access when a PUT
fails. Note that this last "feature" is pretty specific to your scenario @jzoldak, so while I didn't mind implementing it, do note that it's rather suboptimal... In my testing Boto would retry the PUT
quite a few times before giving up, so by not setting the bucket to read only explicitly some time is wasted by having Boto and pip-accel figure this out for themselves.
Can you confirm whether these changes resolve the caveats you mentioned?
from pip-accel.
I believe the issue you reported is fixed and the caveats you mentioned are also resolved, so I'm going to go ahead and close this issue now - I'm trying to keep an overview of active/relevant issues on GitHub and the long list isn't helping :-). If you believe there are still issues with the current implementation, feel free to reopen this issue or report a new one. Thanks for suggesting the feature!
from pip-accel.
Yes, I'm all set with this now. Thanks for all your work on this really great project.
from pip-accel.
Related Issues (20)
- pip-accel not updating package HOT 3
- Doesn't work with pip 7+ HOT 2
- Add option for terse logging. HOT 5
- Windows / Appveyor - same version of cached file - invalidated and built again HOT 5
- is_wheel fails when source_directory is None HOT 6
- Not able to use pip-accel both globally and inside a virtual environment HOT 4
- Support custom pip command HOT 7
- Upgrade to pip 8 HOT 4
- Disabling S3CacheBackend because it failed HOT 2
- pip-accel does not support running as a module HOT 6
- Is there any way to use pip-accel without installing to virtualenv?
- pip-accel is not compatible with `-e` editable installs of packages that use `setuptools_scm` HOT 2
- Why does `pip-accel` make an sdist (?) archive when I `pip-accel install -e .`? HOT 3
- pip-accel doesn't support the newest pip? HOT 4
- Not compatible with alternative install locations, e.g. `--user`? HOT 2
- [need advice] Do not cache development versions
- Failed to install some packages
- pip-accel should honor settings in pip.conf
- Is this project still alive? HOT 2
- Trying to install pip-accel, it insists on downgrading pip to 2.7. How can I get it to respect the current version, 21.1.2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pip-accel.