azure / flamenco-manager-azure

Provides integration of Flamenco Render Farm Manager with Azure Batch

Flamenco Manager on Azure

Flamenco Manager and Workers can now be easily deployed on Microsoft Azure.

Quickstart - Azure Shell

• Login to Azure Cloud Shell
• Select Bash environment
• Run bash <(wget -qO- https://raw.githubusercontent.com/Azure/flamenco-manager-azure/master/install)
• Follow the instructions

Quickstart - Localhost

As an alternative to Azure Shell, it's possible to run Flamenco Manager Azure on a local machine. In that case:

• Install Azure CLI and Azure Batch Explorer.
• Run az login and log in via your browser.
• Make sure you have an SSH keypair available. The private key should be loaded into the SSH Agent (run ssh-add -L to check) or it should be an unencrypted key available in $HOME/.ssh/id_rsa. The public key is read from $HOME/.ssh/id_rsa.pub.

Deploying Flamenco on Azure

First run the above preparation. The first time you run flamenco-manager-azure you may be asked the following information:

• Subscription ID: If you have a single Azure subscription, it's used automatically. If you have multiple, you'll have to choose which one to use.
• Physical location: You'll get a list of locations to choose from.
• Resource Group: All the resources (virtual machine, storage and batch accounts, virtual network components, etc.) created during the deployment will be contained in this group.
• Storage account name: This name must be unique for the location of your choosing.
• Batch account name: This name must be unique within the resource group.
• Virtual Machine name (for Flamenco Manager): This name is also used for the domain name assigned to the public IP address of the virtual machine, and as such must be unique for the location of your choosing.

After each prompt, your answer is stored in azure_config.yaml, and will be used in subsequent runs of flamenco-manager-azure. If you want to change your answer, just delete the corresponding part of azure_config.yaml and re-run flamenco-manager-azure.

The deployment takes approximately 10 minutes.

After deployment

When deployment is done, Flamenco Manager is ready to be configured. The setup URL is logged at the end of deployment, and will be https://{VM name}.{location}.cloudapp.azure.com/setup.

The Azure Batch pool can be resized using Azure Batch Explorer.

To get the IP address of the virtual machine without re-running the deployment application, use:

az network public-ip list --query [].ipAddress

Blender Cloud Add-on configuration

The Blender Cloud Add-on should be configured to use the following settings:

• Job Storage: shaman://{VM name}.{location}.cloudapp.azure.com/. This is the same URL as the Manager, except replacing https:// with shaman://.
• Job Output: /mnt/flamenco-output/render

SSH Access

The Flamenco Manager VM can be reached via SSH using ssh flamencoadmin@{VM name}.{location}.cloudapp.azure.com. The account's password is randomised and cannot be retrieved. Access is granted only using your private key.

Get going with this Go code

Run:

az login
make
./flamenco-manager-azure -help

Publishing a new release

Before creating a release, create a Personal Access Token with API access, then save it to the file .gitHubAccessToken.

Run:

git tag v1.1  # replace with the release version
git push --tags
make release

If you get an error "Ref is not specified" from GitHub, you aren't building exactly the same tag as is available on GitHub (so either your working directory is in a different Git revision, or you didn't push the tag you're trying to build).

Some more technical details

The Azure Batch API Basics document is a nice place to start reading about Azure Batch. This document is also called "Develop large-scale parallel compute solutions with Batch" and "Developer features".

• The files are in /mnt/batch/tasks:
  • /mnt/batch/tasks/applications: zipped and unzipped application packages. Note that these are suffixed with a datetime (I'm guessing node startup), so use the environment variables (see below) to refer to them.
  • /mnt/batch/tasks/startup/std{out,err}.txt: stdout and stderr output of the startup task.
  • /mnt/batch/tasks/startup/wd: default work directory of the startup task.
  • /mnt/batch/tasks/shared: this is not shared between VMs, but shared between tasks on that VM.
  • For more info see Files and Directories.
• /var/lib/waagent contains info from the Azure Agent, like the assigned hostname, configuration settings, TLS certificates, etc.
• If the pool is configured to run the startup task as NonAdmin, it uses uid=1001(_azbatchtask_start) gid=1000(_azbatchgrp) groups=1000(_azbatchgrp).

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

flamenco-manager-azure's Issues

Ensure resource group uniqueness

The storage account name has to be unique across all regions. I picked "flamenco", which had been taken – script gives an error and exits.
Can the script capture the error and allow the user to pick another name without exiting?

Manager UI needs to allow pool create and delete

Users will want to try and use different VM sizes for differing projects or to see what works best for their project. They will therefore need the ability to delete existing pools and create new pools, specifying the VM sizes (as well as other info).

This needs to be done in manager; they can't redeploy manager each time to configure new pools.

Update to latest Flamenco Worker version

Worker 2.3.1 is specified in flamenco-manager-setup-vm.sh. Latest worker version is 2.4, which presumably should be used?

Update instructions

• Ensure deployment instructions and clear for both Linux and Windows users:
  • For Windows users, Azure Cloud Shell will be the simplest method.
  • Linux users can just use their client of also use Azure Cloud Shell.
• Have list of potential follow-on steps; e.g. user should check Batch account quota and will likely need to request an increase in quota to do anything useful.

Enable tile rendering

Cater for complex images by allowing them to be tiled, with one task per tile, then the tiles stitched together after all tiles have been rendered.

Required to speed up renders of very complex images, where whole image takes multiple hours to render.

Give user control over number of tiles.

Depends on capabilities of Flamenco and Blender Cloud plug-in.

Support GPU VMs

https://docs.microsoft.com/en-us/azure/batch/batch-pool-compute-intensive-sizes

flamenco-manager-setup-vm.sh - curl for Blender download failing

The Blender downloads are no longer .bz2 and the naming convention has changed. The curl on line 10 therefore fails. Script needs to be updated with new naming format and tar options to uncompress Blender need to change for .xz format.

Script needs to be updated to report an error if the curl fails.

Cater for customers with multiple Azure subscriptions

If multiple subscriptions, then list subscriptions and allow them to pick what they want to use.

az account set -s <sub name or id>

Use storage blob(s) for applications, not the Azure File Share

There are almost 7000 files and 1GB of files on the ‘flamenco-resources’ share – running the applications from the share is going to really stress the Azure Files share and lead to poor performance/scale, especially for Standard Files offering.

I think the most performant and efficient way to deal with the applications is to have one or a few compressed files in Azure blob storage, download onto each worker VM using resource files of application packages, and un-compress as part of start task.

Support for storage tier selection

Update deployment instructions and maybe script to point user to https://docs.microsoft.com/en-us/azure/storage/files/storage-files-planning#onboard-to-larger-file-shares-standard-tier, so they can get better performance in regions where supported (until is generally available).

• For premium storage, use one share for everything and allow user to pick file size (which also determine performance).
• For standard storage, use one share each for input and output.

Allow existing Batch account to be used for deployment

Customer quota is tied to their Batch account and creating a new Batch account every time means that customers start on default low quota.

Please allow an existing Batch account to be used when deploying the Azure environment.

Thanks

Update release tool to push releases to GitHub

Manager UI should show more pool config info

Allow user to determine VM size used for the pool. Could be shown directly in the UI or displayed in the tooltip, alongside the number of nodes.

Enable large file shares when creating standard storage account

Larger standard files shares are now generally available; when configured there is no extra cost, but they support up to 10x the previous IOPS. The script to create the storage account should configure large file shares.

To configure a storage account - https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/create#largefilesharesstate
Announcement for large file shares - https://azure.microsoft.com/en-us/blog/announcing-the-general-availability-of-larger-more-powerful-standard-file-shares-for-azure-files/

Update some of the prompt strings in the deployment script

• “Desired batch node VM size” -> “Batch pool VM size for Flamenco workers”
• “Target dedicated node count” -> “Number of dedicated Flamenco worker VMs”
• “Target low-priority node count” -> “Number of low-priority Flamenco worker VMs”
• “Desired name for new VM” -> “Flamenco manager VM name”

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Allow VM size of Manager to be specified