Comments (41)
Hereby volunteering as a builder on the Linux platform.
from library.
I've got Linux building nicely on Travis CI and Windows building nicely on AppVeyor. Travis CI can also build Mac stuff, but I didn't bother with that because I have a lovely Mac right in front of me.
I have not tried to run either the Linux or Windows stuff built on those services.
Two days ago on Twitter, Travis CI informed me that they are "hoping" to add Windows, but "It's not immediately planned."
Travis CI does not store the artifacts it builds, but it can deploy them to GitHub Releases, AWS S3, and more than a dozen other places. AppVeyor does store artifacts itself, which is nice, and can also deploy to more or less the same places as Travis CI. And with each service, you also can deploy using whatever custom scripts you know how to write.
Each service is free for open source projects. They are quite similar, but have some minor differences in build lifecycle and configuration. Learning one will give you the basics of the other. And of course they differ significantly in their development environments.
If you want to go the Travis/AppVeyor route, I can help to some extent. My entire experience with those services is an intense three days of focus, so though I've got the basics figured out, I'm by no means an expert.
I'm happy to write/talk about how I configured those CI services and what I've learned.
from library.
IMO those CI services are an overcomplicated mess that don't actually solve any problems easier than if I set up my own build system from scratch. I'm confident I can reduce the entire job of the Package team to a single Makefile command that you run on your PC, so there's no point in automating anything further. Additionally, builds need to be tested briefly, which CI services can't do.
from library.
@cschol: from VCVRack/Rack#686
The same problems arise from commercial plugins sold through the VCV Store, but I have proof of identification for my participating third-party developers, so intentional malware will virtually never happen.
Andrew will take responsibility for verifying authors of commercial plugins, it seems.
from library.
Here's the plan for Rack 0.6, which is scheduled for "late February" which might be the 28th or a couple days later.
We need a Repair team first, in order to make PRs and open issues about incompatibilities in the new API. See https://github.com/VCVRack/Rack/issues/258#issuecomment-346222777
Would anyone be interesting in joining the team, and would anyone be interested in leading? The goal of this team (for now) is to prepare plugins for the 0.6 release so that thousands of users don't need to wait on the developers to make a ~15 line change to their plugins.
Then, right after the release, we'll set up the Build team, so I'll have more info later.
from library.
I'm in for the Rack Plugin Repair Task Force (#RPRTF). :)
from library.
As of now, the Rack 0.6 API is stable. So the Repair Team has been launched with details at #269! We have 2-3 weeks until Rack 0.6 is launched, so it would be fantastic if we could use most of our plugins on the new version with VST/AU Bridge support, redesigned Audio and MIDI Interfaces, etc.
from library.
Updated https://github.com/VCVRack/community#for-plugin-developers with the complete instructions for plugin developers to add/update their plugins. VCV community members should read this as well.
from library.
To get a bit more technical, here are the proposed roles for members. I'll keep this post updated as we add more guidelines.
Each team will have one leader and any number of regular volunteers.
Library team
- Keep plugin metadata correct and up-to-date (e.g. slugs, versions, names, URLs, etc)
- Handle issues opened by plugin developers who want to add/update their plugin.
- Occasionally seek new plugins/updates when developers don't notify us
- Change status of new plugins/updates to "needs_review"
Review team
- Read source code of packages in the plugin submodule repo to find potential security flaws in source code, intentional or not.
- Check that the Makefile is valid, i.e. looks like this. Technically, Makefiles can be malicious too, but the main goal is to make sure it's ready to be packaged.
- Change status of successfully reviewed plugins to "needs_package", otherwise "needs_repair", etc.
To be determined: I will likely need a government-issued ID and/or professional reference on record, due to security concerns.
Build team
- Build all git submodules (i.e. external plugin repos embedded in this repo) to produce ZIP packages for each repo
- When all OSes are built for, update the manifest to include the new builds.
- Test builds manually. I highly recommend to load up Rack and add each module in the plugin to make sure it doesn't crash. At the bare minimum, ensure that Rack doesn't crash on boot when a new plugin version is built.
To be determined: I will likely need a government-issued ID and/or professional reference on record, due to security concerns.
Repair team
- Fix broken plugins, in particular caused by Rack API updates, by opening issues, sending PRs, or even forking and adopting plugins
- Change status of repaired plugins to "needs_review"
from library.
Workflow
I imagine the cogs will turn like this.
- A plugin developer opens an issue in this repository, and the Library team of a new plugin or update, someone on the team creates a JSON manifest for them with correct information and sets the "status" property to "needs_review" or something.
- The Security team will skim the code for a few minutes and if safe, change the status to "needs_package".
- The Package team will attempt to build it. If it builds, the ZIPs are uploaded (to some particular server), and download links are added to the manifest. If not, status is set to "needs_fix".
- The Repair team gets a notification and tries to build it themselves. They may fix the issue themselves and send a PR to the developer, open an issue, or fork the project and change the repo location in the manifest. Once everything is fixed and ready, they set status to "needs_package".
When a developer notifies the Library team of an update, the old version and download links are not changed, only the status in the manifest and the submodule commit hash is updated.
By "get a notification", it could probably be done by someone running a Python script at their leisure which queries for manifests of a certain type.
from library.
Cool, keep posted for more information, and make sure you're subscribed on the sidebar. ----->
from library.
I am also keen to help here in any role that doesn't require pro C++ skills. (So building, packaging, admin etc). I am also on Linux only.
from library.
My threat model would be more along the lines of the evil version of me compromising Strom or someone by having him run community-builds-from-source.sh
and then backdooring his plugin build.
I don't think we should be relying on the security and configuration integrity of a bunch of different developers machines. Andrew is running his own version of Arch; I'm running a VirtualBox/Vagrant Debian image; my mac has a particular version of XCode because of my 9-5 job, etc.
A solution to mitigate this would be reproducible builds and build automation. The zip file hashes in the community repository should match the hash created by building a plugin under appveyor and travis-ci. The build artifacts (i.e. the zip files) from the build should originate from the CI environment.
By "get a notification", it could probably be done by someone running a Python script at their leisure which queries for manifests of a certain type.
I'd suggest we use pull request model and write a Danger configuration to assist us with the code review process etc.
(Obviously, I'd like to help with this, but more on the process/automation side; I can privately provide a personal reference that should assure you of my integrity.)
from library.
If you think it's that important, I can set up three build VMs myself (guys, this is supposed to take time off my hands) that will all have >50% uptime per day and will automatically build plugins with a certain status (e.g. "needs_package"). The role of the Package team will then be to test and approve those builds. I don't want to use the build servers as a CI or a "hey, let's see if this builds. Nope, let's try again" server.
from library.
I believe @dhemery has Appveyor and Travis producing build artifacts already.
from library.
Okay honestly, trust in the Package team is not an issue with me. If you want to lead it, you can do whatever you want. I'll just require ID and professional reference for those in control of the computers producing the builds.
from library.
Moving on to the next issue...
Regarding step 1 in the workflow above, I imagine the best method of notification is for plugin developers to open an issue at this repository. It should also be a task of the Library team to occassionally seek new plugins and updates themselves in case plugin developers don't notify us for some reason.
from library.
count me in
a few other module developers and I are on freenode irc #VCVRack
we are already informally doing some of the things on the list.
there are python scripts floating around that can automate building multiple plugins.
from library.
I'm interested in providing OS X builds - but I only have my work laptop for a few more months. So I will probably have to drop out of the build team sometime later this year.
from library.
so i guess we'll need a list of who's in and the make up of the various teams
from library.
"If you think it's that important, I can set up three build VMs myself (guys, this is supposed to take time off my hands) that will all have >50% uptime per day and will automatically build plugins with a certain status (e.g. "needs_package")."
I think you should delegate that to someone you trust, you've got too much on already.
from library.
Updated team roles (my second post). I will be working on the Workflow section later. It is essentially a state machine or assembly line, to deligate small but important chunks of work to people skilled in that area. Teams won't need to know everything about how this detailed system works, just check or build a plugin and then set a status flag to pass on to the next team.
from library.
I've created a "v2" branch of this repo. https://github.com/VCVRack/community/tree/v2
I'll be documenting the workflow in the viewpoint of each team in the README, making the process as simple as possible. For fun, you can try building all plugins in the repo/
folder against the latest Rack master source.
from library.
@AndrewBelt when you have a minute and if you don't mind, would please comment on tasks/chunks of support time that you feel you need to get off your plate? Your list will probably inform some of the workflow discussions. Thanks!
from library.
Not sure I understand exactly your question, but my goal is to not manage any open-source third-party plugins at all, yet be sure the Plugin Manager content meets good quality standards, since 10k+ people use this category of plugins. I'd like to still be informed of issues with plugins, but not by experiencing them directly with plugin developers, but instead by reading about them from Issues created by the volunteers.
from library.
Thanks, @AndrewBelt, that is helpful to understand.
from library.
Sign me up for helping. Right now, limited to Windows for actual work, but I have enough Linux experience also.
from library.
A few questions:
- Does this process cover open source and commercial plugins?
- Are there any licensing issues to be considered in the reviews? I am talking specifically software licenses.
- What about Intellectual Property considerations?
from library.
- Just open-source plugins. I handle commercial plugins. Commercial plugins that aren't distributed through the Plugin Manager can still be listed, but no automatic downloads will be provided. Users will have to go to the vendors' websites and downloads and unzip the packages on their own.
- I don't think so, can you think of any?
- Same as (2)
from library.
- Just wondering about implications of
GPL
code used in plugins. Are there any recommendations that should be made to developers of plugins? TheAudibleInstruments
repo has comments at the end of the repo that say "GPL, will not port", other 3rd party (free) plugins are licenses asGPL
(e.g. Stellare-Modular Link). If there are licensing considerations from Rack's point of view, they need to be made clear to plugin developers. If there aren't and it is just the developer's choice, all the better. - I am thinking of the recent "Floats" example. Let's say it is submitted as a community plugin (free), someone reviews, and notice the obvious similarities with MN "Maths". Does the reviewer reject the plugin? Or require proof that the IP was properly licensed?
from library.
@cschol re: 2
FSF opinion: Linking non-free code against GPL'd code without a linking exception for plugins is a GPL violation. The revised BSD license that Rack has is compatible with linking GPL code; we're ok to include GPL code in our plugins.
There may be some ambiguity around this topic were someone to distribute a commercial fork of Rack (as they are entitled to do under the BSD license). I don't think it's reasonable that suddenly existing GPL plugins could be thought be in violation because someone made a commercial fork of Rack.
Part of the ambiguity is wether dynamic linking (as in Rack) produces a derived work. Here's a summary of positions on Wikipedia.
With respect to not including GPL'd code in AudibleInstruments: my guess is that Andrew wanted the entire base and plugins to be BSD licensed. It was thoughtful to release Rack under a license that is easily able to link with more restrictive licenses.
from library.
@cschol Oh, I see what you were asking now.
GPL plugins are fine. I've actually asked FSF about this (i.e. if GPL code being loaded by BSD application while commercial plugins are loaded is a GPL violation, assuming all are distributed separately) multiple times and they either don't answer the core question or don't understand it. But I think this is totally fine, since the question of who violates the GPL seems to be "no one". Obviously, if I make a GPL plugin for Photoshop, I can't then immediately sue Adobe for violating the GPL, that would be ridiculous and no law system makes that a valid case.
- Okay, I understand what you were asking. The bar is low for being added to Plugin Manager: The plugin must not be malicious (harm your computer or privacy), and it must not be an IP violation. So if you see an IP violation in a plugin submission, don't bother handling it, just reject it.
from library.
-
Yeah, I was specifically wondering about the mix of commercial and GPL plugins. Your reasoning makes sense to me.
-
Also makes sense.
from library.
@AndrewBelt, I'd help out with that as well if needed.
from library.
I am available if needed, particularly point 3 auditing and testing. I have a golden ear!
from library.
I'm going to try using the new GitHub Team feature, so bear with me. I've never tried it.
Edit: Never mind, I think GitHub issues is the most appropriate way to communicate.
from library.
The Library team issue has been created at #352
from library.
Okay, we need to way to orchestrate the following process between the teams.
- Plugin developer posts update as an issue
- Library team updates the manifest, except for
lastestVersion
andstatus
since those actually change the behavior of the plugin manager API. - If a source update or addition is needed, Review team updates the new source and changes some flag after accepting (or raises an alert if not. I don't think we need a workflow for handling bad plugins, we just figure it out at the time.)
I'm happy to add whatever properties we need to the manifest format.
Probably the most obvious workflow is an updatedVersion
and latestVersion
property. (It would make more sense to have buildVersion
and latestVersion
, but unfortunately latestVersion
is hard-coded in Rack 0.6.0 as the version to check to download, so it's there to stay.) The Library team changes updatedVersion
. If nonequal, the Review team will git pull whatever
the repo submodule for that plugin and ONLY push to master if they have reviewed it and everything looks okay.
The Build team (currently just me) will build the module, set the latestVersion
, commit, and upload the package(s) to the distribution server at api.vcvrack.com.
- How should the Build team be alerted that a build is ready? Upon accepting a PR? Where should they notify a build failure on a particular platform?
I can also give commit rights to people if that's the best way to go about things.
from library.
I have adopted the above workflow into the Review Team #354 and Library Team #352 threads. Those two teams are now "launched" and ready for PRs from anyone who reads the posts.
from library.
i feel this needs some more promotion. also, the opening posts could be updated...
from library.
This issue has served its purpose and is now mostly out of date. See #269, #352, and #354 for individual instructions for joining VCV Community teams.
from library.
Related Issues (20)
- october HOT 9
- HarmonicAnomalies HOT 4
- minimal-friction HOT 4
- MUS-X HOT 9
- sn HOT 6
- STS-Multiplier HOT 3
- astrokkidd HOT 6
- STS-Splitter
- STS-Sine-VCO
- STS-Saw-VCO
- STS-Bundle HOT 2
- SmarTAZZStudio-Free HOT 10
- kCathedral2 reverb preset crashes Rack HOT 1
- SSE HOT 6
- pachde-hc-one HOT 4
- MML HOT 5
- CVfunk HOT 17
- eightfold HOT 3
- SanguineMutants HOT 4
- Semeru HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from library.