waderobson / s3-auth Goto Github PK
View Code? Open in Web Editor NEWGenerate s3 authentication headers for munki
Generate s3 authentication headers for munki
For the life of me I can't seem to work out out to set the link timeout to be greater than the 15 minute default.
I'm doing my best to wrap my head around the way this middleware and AWS4 authentication header work, but I'm just not there.
Is there a simple way to inject Amz-Expires=XXX
into the link request? It seems like it's defaulting to a 900 second timeout, but I'm having issues pulling down large updates like an OS install.
I'll keep poking a stick at it, but I'm sure there have to be few folks that have hit this limit.
Thanks for the good work.
Hello,
First of all thanks for your work on s3-auth, i have a private repo on S3 bucket and i have follow your process to make it work with munki, but it's not working, below you will find the error i'm getting with munki:
XeroxBrain:munki admin$ sudo /usr/local/munki/managedsoftwareupdate
Managed Software Update Tool
Copyright 2010-2020 The Munki Project
https://github.com/munki/munki
Starting...
Checking for available updates...
Retrieving list of software for this machine...
ERROR: Could not retrieve manifest Packages from the server: HTTP result 403: forbidden
ERROR: Could not retrieve managed install primary manifest.
Finishing...
My bucket URL is working as i have try it by making the bucket repo public.
My Access key ID and Secret access key are also good.
I have copy middleware_s3.py in /usr/local/munki/ folder
I have also set
sudo defaults write /Library/Preferences/ManagedInstalls AccessKey 'xxxxxxx'
sudo defaults write /Library/Preferences/ManagedInstalls 'xxxxxx'
sudo defaults write /Library/Preferences/ManagedInstalls Region 'eu-west-1’
sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL ‘xxxxxxxxx'
Thanks for you help.
hey man -
love this - was just walking through a few things and found out the aws s3 sync command is finicky about the exclude. Please consider changing the following line in your wiki.
From:
aws s3 sync /path/to/munki/ s3://<S3_BUCKET_GOES_HERE>/ --exclude '*.git/*' --exclude '.DS_Store' --delete
To:
aws s3 sync /path/to/munki/ s3://<S3_BUCKET_GOES_HERE>/ --exclude "*.git/*" --exclude "*.DS_Store*" --delete
Thanks!
There seems to be an issue with redirections depending on the URL of the S3 bucket.
For example, if a bucket has a name blah.s3.amazonaws.com, S3 may return a HTTP 307 when requested. This can happen if your AWS region is anything but us-east.
If you are using the ap-southeast-2 region, and request blah.s3.amazonaws.com, you are redirected to blah.s3-ap-southeast-2.amazonaws.com
As a result, you end up with this error upon running managedsoftwareupdate:
Retrieving list of software for this machine...
ERROR: Could not retrieve manifest YOUR_MANIFEST from the server: HTTP result 307: temporarily redirected
ERROR: Could not retrieve managed install primary manifest.
Finishing.
As a workaround, you can modify the following function to look for a different string. Instead of if 's3.amazonaws.com' in options['url']:
you could substitute region specific information such as if 's3-ap-southeast-2.amazonaws.com' in options['url']:
def process_request_options(options):
"""Make changes to options dict and return it.
This is the fuction that munki calls."""
if 's3.amazonaws.com' in options['url']:
headers = s3_auth_headers(options['url'])
options['additional_headers'].update(headers)
return options
So, two options:
The error I was receiving looked like this:
iMac:~ admin$ sudo managedsoftwareupdate -v
Managed Software Update Tool
Copyright 2010-2017 The Munki Project
https://github.com/munki/munkiStarting...
Checking for available updates...
Getting manifest ms/ms...
ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
File "/usr/local/munki/managedsoftwareupdate", line 1055, in
main()
File "/usr/local/munki/managedsoftwareupdate", line 781, in main
client_id=options.id.decode('UTF-8'))
File "/usr/local/munki/munkilib/updatecheck/core.py", line 76, in check
mainmanifestpath = manifestutils.get_primary_manifest(client_id)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 156, in get_primary_manifest
manifest = get_manifest(clientidentifier)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 114, in get_manifest
manifesturl, manifestpath, message=message)
File "/usr/local/munki/munkilib/fetch.py", line 425, in munki_resource
verify=verify)
File "/usr/local/munki/munkilib/fetch.py", line 378, in getResourceIfChangedAtomically
message=message, resume=resume, follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 510, in getHTTPfileIfChangedAtomically
follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 236, in get_url
options = middleware.process_request_options(options)
File "/usr/local/munki/middleware_s3.py", line 114, in process_request_options
headers = s3_auth_headers(options['url'])
File "/usr/local/munki/middleware_s3.py", line 94, in s3_auth_headers
signing_key = get_signature_key(SECRET_KEY, datestamp, REGION, SERVICE)
File "/usr/local/munki/middleware_s3.py", line 45, in get_signature_key
kdate = sign(('AWS4' + key).encode('utf-8'), datestamp)
TypeError: cannot concatenate 'str' and 'NoneType' objects
It turns out the latest version of s3-auth changes the location where it reads the preferences. The old location for the settings was /Library/Preferences/com.github.waderobson.s3-auth
and the new location is /Library/Preferences/ManagedInstalls
. This makes sense as the rest of the munki preference keys are stored there too.
Thanks for producing and maintaining this :)
I was getting 403 access denied errors when I first tried using this.
After some digging around I found post on MacAdmins slack recommending adding this preference:
sudo defaults write /Library/Preferences/ManagedInstalls S3Endpoint 's3.your.s3.domain'
After adding the preference the 403 error went way and everything worked :) So I was wondering if it's worth adding to the wiki?
Even though I'm seeing commits for Python 3 compatibility, it seems to just bunk out. Anything obvious I'm missing?
sudo managedsoftwareupdate -vvv
Password:
Managed Software Update Tool
Copyright 2010-2019 The Munki Project
https://github.com/munki/munki
Starting...
No CA cert info provided, so nothing to add to System keychain.
No client cert info provided, so no client keychain will be created.
Checking for available updates...
No client id specified. Requesting SERIAL...
Manifest base URL is: https://URLDETAILS/munki_repo/manifests/
Getting manifest SERIAL...
Options: {'url': 'https://URLDETAILS/munki_repo/manifests/SERIAL', 'file': '/Library/Managed Installs/manifests/SERIAL.download', 'follow_redirects': 'none', 'ignore_system_proxy': False, 'can_resume': False, 'additional_headers': {'User-Agent': 'managedsoftwareupdate/4.0.0.3881 Darwin/19.0.0'}, 'download_only_if_changed': True, 'cache_data': {
etag = "\"2171375bed808ff2289b6e8e2f3178de\"";
"last-modified" = "Fri, 06 Dec 2019 20:55:21 GMT";
}, 'logging_function': <function display_debug2 at 0x112d06f80>}
Processing options through middleware
ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
File "/usr/local/munki/managedsoftwareupdate", line 1141, in <module>
main()
File "/usr/local/munki/managedsoftwareupdate", line 862, in main
client_id=unicode_or_str(options.id))
File "/usr/local/munki/munkilib/updatecheck/core.py", line 82, in check
mainmanifestpath = manifestutils.get_primary_manifest(client_id)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 175, in get_primary_manifest
manifest = get_manifest(clientidentifier, suppress_errors=True)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 122, in get_manifest
manifesturl, manifestpath, message=message)
File "/usr/local/munki/munkilib/fetch.py", line 440, in munki_resource
verify=verify)
File "/usr/local/munki/munkilib/fetch.py", line 393, in getResourceIfChangedAtomically
message=message, resume=resume, follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 525, in getHTTPfileIfChangedAtomically
follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 246, in get_url
options = middleware.process_request_options(options)
File "/usr/local/munki/middleware_s3.py", line 117, in process_request_options
headers = s3_auth_headers(options['url'])
File "/usr/local/munki/middleware_s3.py", line 97, in s3_auth_headers
signing_key = get_signature_key(SECRET_KEY, datestamp, REGION, SERVICE)
File "/usr/local/munki/middleware_s3.py", line 49, in get_signature_key
kdate = sign(('AWS4' + key).encode('utf-8'), datestamp)
TypeError: can only concatenate str (not "NoneType") to str
Hi,
I am getting this error running the middleware with munki4.
\nAuthorizationHeaderMalformed
The authorization header is malformed; the authorization header requires three components: Credential, SignedHeaders, and Signature.
I get error with munki3 as well.
ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
File "/usr/local/munki/managedsoftwareupdate", line 1140, in
main()
File "/usr/local/munki/managedsoftwareupdate", line 861, in main
client_id=options.id.decode('UTF-8'))
File "/usr/local/munki/munkilib/updatecheck/core.py", line 81, in check
mainmanifestpath = manifestutils.get_primary_manifest(client_id)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 166, in get_primary_manifest
manifest = get_manifest(clientidentifier, suppress_errors=True)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 114, in get_manifest
manifesturl, manifestpath, message=message)
File "/usr/local/munki/munkilib/fetch.py", line 428, in munki_resource
verify=verify)
File "/usr/local/munki/munkilib/fetch.py", line 381, in getResourceIfChangedAtomically
message=message, resume=resume, follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 513, in getHTTPfileIfChangedAtomically
follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 235, in get_url
options = middleware.process_request_options(options)
File "/usr/local/munki/middleware_s3.py", line 117, in process_request_options
headers = s3_auth_headers(options['url'])
File "/usr/local/munki/middleware_s3.py", line 90, in s3_auth_headers
credential_scope = '{}/{}/{}/aws4_request'.format(datestamp, REGION, SERVICE)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.