homebysix / recipe-robot Goto Github PK
View Code? Open in Web Editor NEWA kick ass tool for creating AutoPkg recipes.
License: Apache License 2.0
A kick ass tool for creating AutoPkg recipes.
License: Apache License 2.0
When somebody chooses Help > Recipe Robot Help, they should arrive at this URL in their default browser.
Word on the street is that urllib2 isn't great at downloading large files. Our inspect_download_url()
function is almost exclusively going to be handling large files, so is this something we need to consider?
One example of a "chunking" method: https://gist.github.com/gourneau/1430932
No need to wrap a flat package in a dmg before importing to Munki.
After the Recipe Robot script finishes, it should output a report-plist to /tmp or the Cache folder, so that the planned Mac app can read the status of the last run, any errors, and other information. Once this is done we should be able to go full speed ahead on the Mac app.
Without lots of passing facts around, robo_print can't add messages to the appropriate facts lists (previously "report"). Also, I don't feel like a print function should be tasked with storing lists of messages for later consumption, or for exiting the program.
SO, here's what I'm going to do:
Some of the setup stuff may need its own exception handling written in, since it will be outside of the main recipe generation try/except/finally.
This will ease my OCD over single-responsibility-principle violations!
Many recipes based on download URLs benefit from a filename
argument when including the URLDownloader
processor:
Others are just fine as-is:
How can Recipe Robot determine which is which and apply the filename
argument intelligently?
Currently, the recipes are displayed in this order:
[ ] 0. filewave - Imports a fileset into your FileWave server.
[ ] 1. munki - Imports into your Munki repository.
[ ] 2. install - Installs the app on the computer running AutoPkg.
[ ] 3. jss - Imports into your Casper JSS and creates necessary groups, policies, etc.
[ ] 4. download - Downloads an app in whatever format the developer provides.
[ ] 5. absolute - Imports into your Absolute Manage server.
[ ] 6. ds - Imports into your DeployStudio Packages folder.
[ ] 7. sccm - Creates a cmmac package for deploying via Microsoft SCCM.
[ ] 8. pkg - Creates a standard pkg installer file.
I think they should be displayed in this order, based in part on the ParentRecipe chain and in part on popularity.
[ ] 0. download - Downloads an app in whatever format the developer provides.
[ ] 1. pkg - Creates a standard pkg installer file.
[ ] 2. munki - Imports into your Munki repository.
[ ] 3. install - Installs the app on the computer running AutoPkg.
[ ] 4. jss - Imports into your Casper JSS and creates necessary groups, policies, etc.
[ ] 5. ds - Imports into your DeployStudio Packages folder.
[ ] 6. absolute - Imports into your Absolute Manage server.
[ ] 7. filewave - Imports a fileset into your FileWave server.
[ ] 8. sccm - Creates a cmmac package for deploying via Microsoft SCCM.
As of right now, the master branch is no longer producing AppStoreApp recipes. Here's what happens when I attempt to run DisplayMenu:
recipe-robot --verbose /Applications/Display\ Menu.app
-----------------------------------
| Welcome to Recipe Robot v0.2.5. |
-----------------------------------
\ _[]_
\ [oo]
d-||-b
||
_/ \_
Processing /Applications/Display Menu.app ...
Input path looks like an app.
Validating app...
App seems valid
Getting app name...
App name is: Display Menu
Getting bundle identifier...
Bundle idenfitier is: de.milchimgemuesefach.Display-Menu
Checking for a Sparkle feed...
No Sparkle feed
Determining whether app was downloaded from the Mac App Store...
App came from the App Store
Looking for version key...
Version key is: CFBundleShortVersionString (2.2.2)
Looking for app icon...
App icon is: /Applications/Display Menu.app/Contents/Resources/DisplayMenu
Getting app description from MacUpdate...
Description: Get your display menu back for OS X mavericks.
Gathering code signature information...
Code signature verification requirements recorded
3 authority names recorded
Searching for existing AutoPkg recipes for "Display Menu"...
No results
Searching for existing AutoPkg recipes for "DisplayMenu"...
No results
Generating munki recipe...
[REMINDER] I've created at least one AppStoreApp override for you. Be sure to add the nmcspadden-recipes repo and install pyasn1, if you haven't already. (More information: https://github.com/autopkg/nmcspadden-recipes#appstoreapp-recipe)
/Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/MAS-Display Menu.munki.recipe
[WARNING] Skipping install recipe, because this app was downloaded from the App Store.
/Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/Display Menu.install.recipe
[WARNING] Skipping download recipe, because this app was downloaded from the App Store.
/Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/Display Menu.download.recipe
Generating pkg recipe...
[REMINDER] I've created at least one AppStoreApp override for you. Be sure to add the nmcspadden-recipes repo and install pyasn1, if you haven't already. (More information: https://github.com/autopkg/nmcspadden-recipes#appstoreapp-recipe)
/Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/MAS-Display Menu.pkg.recipe
No recipes are actually being created. The output folder is created, but no files exist in that folder.
In order to release our public beta, I think we need these features:
--ignore-existing
from the preferences. Preferably the checkbox should appear only while holding Option, or another such minor deterrent. (Right now there's no way to access this option via the UI.)When a png icon file or a recipe already exists in the output directory, should we warn about that? I'm leaning towards no, since I often run Recipe Robot multiple times in a row while making small tweaks, and overwriting existing files is the desired behavior.
If we don't prompt for overwriting, we may want to choose a different default output path than ~/Library/AutoPkg/RecipeOverrides, in order to prevent overwriting existing overrides.
Right now Recipe Robot looks for the files:extra-info
tag marked as data
in the RSS feed, which seems to work for a subset of SourceForge-hosted files, but not all of them.
For example, here are the filenames and extra-info
tags for each file in a GrandPerspective release:
Filename | extra-info |
---|---|
README.txt | English text |
GrandPerspective-1_5_1.dmg | data |
GrandPerspective-1_5_1-src.tgz | POSIX tar archive (GNU) (gzip compressed data) |
The one in bold is the actual binary we want to download.
As a counterexample, the latest release of VLC has an extra-info
tag x86 boot sector (bzip2 compressed data)
.
Trying to find the unifying factor that determines "yes, this is the binary you want to download."
How should we handle apps with spaces in their names? Current behavior is as follows:
a. Recipe identifier does not contain spaces.
b. Recipe filenames don't contain spaces, except for JSS recipes, which do.
c. Recipe %NAME%
input variable does contain spaces.
d. Packages, downloads, etc based on %NAME%
do contain spaces.
(a) and (c) are logical and well accepted. I'm not as sure about (b) and (d).
Right now Recipe Robot reads all the sparkle:shortVersionString
and sparkle:version
information for every item in the feed, and does a basic string comparison to determine which item is the "latest" version. However, this fails easily: "999" > "1000"
A better approach would be to get the most recent pubDate
.
Even if a valid output folder is specified, sometimes clicking Reveal Recipes will produce an error.
Going back to the preferences and selecting the same output folder again seems to fix this problem.
Most autopkg repos group recipes into folders by developer name or by product name. Should Recipe Robot do that, or leave it up to the user?
Are there keys other than SUFeedURL
and SUOriginalFeedURL
that would help us download an app, given its Info.plist?
I'm guessing the answer is no.
It looks like Recipe Robot can get confused about which recipes it's building if the previous app's cache remains in the cache folder (either by using --keep-cache
or by interrupting a run).
Steps to reproduce:
Start this, but interrupt it when it gets to the unzipping part:
recipe-robot --verbose http://www.coconut-flavour.com/downloads/coconutBattery_3_2_1.zip
Run this fully:
recipe-robot --include-existing --verbose http://delicious-monster.com/downloads/DeliciousLibrary3.zip
Observe that the resulting recipes are for coconutBattery, not Delicious Library.
Only two hard things in computer science, they say: cache invalidation, naming things, and off-by-one errors.
Thanks so much for this tool. I was excited to see what it would come up with for some apps that I've already written manual recipes for. Alas, this generates a fatal error.
It would be nice to compare my old recipes with what RR comes up with. Would it be possible to tell the nice robot to ignore an existing recipe and continue anyway?
Thanks
For example, this AutoCasperNBI run failed because there's a single curly quote in the repo's description.
Processing https://github.com/macmule/AutoCasperNBI ...
Input path looks like a GitHub URL.
Getting GitHub repo...
GitHub repo is: macmule/AutoCasperNBI
Getting app name...
App name is: AutoCasperNBI
Getting GitHub description...
GitHub description is: AutoCasperNBI is an app that automates the creation of NetBoot Images (read: NBI’s) for use with Casper Imaging.
[ERROR] Recipe Robot exploded with unexpected error:
Traceback (most recent call last):
File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe-robot", line 103, in main
process_input_path(facts)
File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/inspect.py", line 127, in process_input_path
facts = inspect_func(input_path, args, facts)
File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/inspect.py", line 998, in inspect_github_url
facts["description"] = description
File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/facts.py", line 79, in __setitem__
val = NotifyingString(self.default_suffix, val)
File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/facts.py", line 141, in __new__
instance = super(NotifyingString, cls).__new__(cls, text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 80: ordinal not in range(128)
Recipe Robot was originally intended to utilize existing recipes as a foundation to build additional recipes. However, this seems like an incredibly complex thing to undertake:
ParentRecipe
existed, and therefore handle the entire download process too. Do we duplicate the download components into a new download recipe, then build pkg and other recipes on top of that?Given this, and given our initial time constraints, I'm starting to consider dropping this feature from the 1.0 release. Instead, Recipe Robot would only generate full recipe chains.
I still want to urge people not to build recipes that already exist and upload them to GitHub, in order to reduce confusion caused by recipe duplication in autopkg search
results. Maybe this would mean that if any recipe exists, Recipe Robot will politely decline to build more (unless forced to build a full set).
The function defined here is used to define the user identifier for the autopkg recipes: https://github.com/homebysix/recipe-robot/blob/master/scripts/recipe-robot#L372
It defaults to: com.github.your_name_here
It would be interesting if it could default to: com.github.(configured git user)
Typically this is contained in the file: ~/.gitconfig
or ~/.config/git/config
or /etc/gitconfig
The format is like an INI file, with sections, keys, values:
[user]
name = jgstew
Or else, it would default to: com.github.( os.getusername() )
com.github.your_name_here could be a fall back value.
Because Recipe Robot produces entire recipe chains, you shouldn't be able to select a specific recipe type in preferences unless its parent recipe type is also selected.
Here are the dependencies:
Right now %NAME%.app
is our sole blocking application when creating Munki recipes. However, if we encountered other apps in our travels (specifically while running inspect_pkg()
) I think those apps should also be included by default.
This will handle situations in which the name of the app doesn't match the filename of the app bundle.
If the app we want to use as the basis for recipe-building isn't on the root level of the dmg or zip, do we want to dig much farther, or should we error-out?
Two easy examples, both of which require the user to drag a folder to their /Applications folder, rather than an app:
Thinking about this might be out of scope for Recipe Robot, but just wanted to put it out there.
Might not be possible, but I want to see if we can find a way around SSLv3 handshake errors when trying to access various URLs. Here's an example:
$ recipe-robot -v http://www.macroplant.com/latest-binaries/adapter-mac.dmg
-----------------------------------
| Welcome to Recipe Robot v0.0.4. |
-----------------------------------
\ _[]_
\ [oo]
d-||-b
||
_/ \_
Processing http://www.macroplant.com/latest-binaries/adapter-mac.dmg ...
Input path looks like a download URL.
Download URL is: http://www.macroplant.com/latest-binaries/adapter-mac.dmg
Downloading file for further inspection...
[WARNING] I got an SSLv3 handshake error, and I don't yet know what to do with that. (<urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)>)
[ERROR] I wasn't able to determine the name of this app, so I can't make any recipes.
If it's a standard "drag this to your PreferencePanes" or "drag this to your Internet Plugins" dmg, we should be able to detect and build recipes for that. Likewise if there's a single .prefPane or .plugin file enclosed in a zip.
Not essential for 1.0, though, since there are so few preference panes and internet plugins that don't already have recipes.
When creating absolute, sccm, and filewave recipes, certain repos need to be added in order for the resulting recipe to run successfully. Remind people of this, if the repo isn't already present.
At the moment, Recipe Robot doesn't build a pkg recipe if there's no bundle_id in facts. However, I don't think we've considered that there are other possible ways to determine the bundle identifier even if we weren't able to collect it during inspect_app()
.
SparkleUpdateInfoProvider
grabs.AppDmgVersioner
can be used to determine a bundle identifier. As long as we include that before the PkgCreator
processor, we should be good to go.Right now we're doing what AutoPkg does, which is this:
try:
from recipe_robot_lib import FoundationPlist
except:
print '[WARNING] importing plistlib as FoundationPlist'
import plistlib as FoundationPlist
@sheagcraig mentioned that we might use neither FoundationPlist nor plistlib, and instead "use the preferences system correctly." I don't yet know what this means, but I want to make an issue here for reference.
Right now an interactive prompt appears when the Recipe Robot script is first run. The user is prompted for their preferred identifier, recipe save location, and preferred recipe types.
During this prompt, it would be excellent to "refresh" the screen while retaining scrollback. Right now the act of toggling recipe types on/off causes the entire list of recipes to appear again.
I believe the right way to do this is using the curses
module, although I'm not familiar with it yet.
This will not affect the Mac app, which will have a separate interface for prompting for preferences.
What is the goal of timing the recipe generation?
Is it just for fun?
I think it's fun ;)
But if so, do we need to include timing for the argparsing, welcome message, etc?
I often run Recipe Robot multiple times in a row while making small tweaks. The end result is only a handful of recipes, but the cumulative count increases on every run.
Might be more accurate not to increment the count if an existing recipe is overwritten (see #32).
Would be useful for the Popen()
command that implements #25.
SourceForge requires automated programs to limit themselves to one hit per 30 minutes per feed. In the scenario where somebody is repeatedly trying to create a SourceForge-based recipe, Recipe Robot could exceed this limit.
Potential options:
a. When the same RSS feed is tried a second time, issue an error and say "try again in X minutes."
b. Keep a local cache file of the RSS feed, and refer to that instead of the live version on subsequent runs.
Both rather complex.
Right now Recipe Robot requires you to press "y" to agree to the SLA on certain disk images.
[WARNING] Please type Y and press Enter to accept the software license agreement.
Agree Y/N? y
In order to link this up with the Mac app, we'll need to make this non-interactive.
When creating a lot of recipes in a short time, it's possible that users could run up against GitHub's API rate limits. It would be excellent if we could leverage API tokens (as AutoPkg does) to prevent this limitation.
We can't count on SourceForgeURLProvider.py to remain at the same place in jessepeterson-recipes forever, so we should have a backup plan if urlopen()
fails to retrieve it. I vote we use the copy in recipe_robot_lib as a backup option.
Any value set here:
$ defaults read com.elliotjordan.recipe-robot RecipeIdentifierPrefix
com.github.homebysix
Should be used as the default value on next --config
run:
$ recipe-robot --config
-----------------------------------
| Welcome to Recipe Robot v0.2.5. |
-----------------------------------
\ _[]_
\ [oo]
d-||-b
||
_/ \_
Showing configuration options...
Recipe identifier prefix
This is your default identifier, in reverse-domain notation.
(If you have a GitHub account, it's customary to use com.github.<your GitHub username>.)
[com.github.elliot]:
If for example we don't have the bundle identifier when we arrive at generate_pkg_recipe()
, it issues an early return
back to the parent generate_recipes()
function. However, at the end of the generate_recipes()
function it still happily writes a non-functional recipe to disk because it doesn't know any better.
I have an idea of how to fix this:
if len(recipe["keys"]["Process"]) > 0:
FoundationPlist.writePlist(recipe["keys"], dest_path)
Would that work?
When set to True, this will strictly follow the style here. This will be useful for people who are creating recipes they wish to submit to jss-recipes.
The only difference I can think of between official style and Recipe Robot's existing style is which Identifier
is used (com.github.homebysix.jss.Foo vs com.github.jss-recipes.jss.Foo), but it might be useful to have this preference setting anyway for future reference.
Is there such a thing as a Sparkle feed that doesn't use http://www.andymatuschak.org/xml-namespaces/sparkle
as its namespace? If so, Recipe Robot will probably choke on it.
Here's an example of how this might be resolved:
https://github.com/autopkg/autopkg/blob/master/Code/autopkglib/SparkleUpdateInfoProvider.py#L115-L119
Running ../recipe-robot/scripts/recipe-robot --config
Results in "too few arguments" error
Simply appending any arbitrary path to the end fixes the issue.
Not sure where this is exactly but probably want to set the input_path required flag to false when running --config
Right now Recipe Robot doesn't know how to create filewave recipes based on pkg downloads.
Here's an example download URL we can use for testing: http://zoom.us/client/latest/ZoomRooms.pkg
Right now the generate_recipes()
function and various generate_x_recipe()
functions individually determine whether they can proceed, given the info in the facts
dict. It might be more efficient to have a single fact_validator()
function that centrally determines which recipes can be built and which should be skipped.
Once #18 is built out, we should prompt users to create a GitHub token if they don't already have one. I don't think we should actually handle creation of the tokens, but at least a [WARNING] that tells them it's a good idea would be nice.
Here are the facts we need to gather from the PackageInfo and Payload files inside of an expanded flat package:
inspect_app()
.This is not working 100% reliably at the moment.
And what if the payload contains multiple apps? Do we go with the one with the shortest path, as would be the case below?
$ gunzip -c /Users/elliot/Library/Caches/Recipe\ Robot/expanded/ChronoSync.pkg/Payload | p
ax | egrep "\.app$"
./Applications/ChronoSync.app
./Applications/ChronoSync.app/Contents/Library/LoginItems/ChronoSync Scheduler.app
./Library/Application Support/ChronoSync/ChronoAgentLocal.app
Or should we use the one with the largest file size? (How can we determine file size without extracting first?)
Or should we use the one with a Sparkle feed?
Right now Recipe Robot works really well with apps that support CodeSignatureVerifier, but less so with pkgs that support CodeSignatureVerifier.
Here's what we're collecting to verify an app:
Here's what we need to collect to verify a pkg:
We may have to separate codesign_status
and codesign_reqs
into app-specific and pkg-specific keys in the facts
dict.
Do we want to have Recipe Robot actually run the recipe and make sure the exitcode is zero? I'm leaning towards no, because of the risks of running newly-created .munki, .install, or .jss recipes on a production Mac.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.