azure / moodle Goto Github PK

The current load testing scripts were developed before MS SQL option, and MS SQL SKUs are quite different from MySQL/PostgreSQL. So we need to update the load testing scripts to support MS SQL choice and SKUs.

According to the diagram in the readme the database is exposed on a public subnet, why is this? Wouldn't it be better to have it on a private network?

Hello,

Could you please explain me how to get the moodle password after an "automatic installation" with the button "Deploy on Azure" ?

I tried "az group deployment show --resource-group NameOfMyRsgGroup --name MasterDeploy --out json --query *.outputs"

Many thanks for your help,

Steve

The /moodle directory is shared out to frontend VMs through the replicated Gluster cluster for high availability as well. However, we are hearing that the Gluster performance is not very good and asked about other options, like copying the static content over to the local file system. I think we should at least try to allow the choice of NFS over Gluster in the template. We might use one dedicated VM for the NFS host, but also could consider utilizing the jumpbox VM. Of course we should also consider the cron job scalability on the jumbbox as well, in that case.

Some of the parameter names are sub-optimal, for example serviceObjective, skuCapacityDTU and more.

We should do a complete review of all parameter names and make them more intuitive.

This will have an impact on the documentation of these parameters (#36)

Currently the template installs the ElasticSearch plugin and 3 VMs for that. It would be good to allow choice of not installing the plugin and deploying the 3 VMs, mainly for customers who don't want the ElasticSearch (or at least load testing scenarios where elastic search is not used/needed).

It'd be interesting how to seamlessly add the elastic search VMs and configs after the initial cluster deployment without the feature selected.

Not sure if the deployment would benefit from this or if the Basic SKU is good enough: https://azure.microsoft.com/en-us/blog/the-new-azure-load-balancer-10x-scale-increase/

Currently ObjectFS is configured by default here.

It's late so I don't want to do this now incase I'm missing something, but it looks to me like we can remove some parameters from azuredeploy.json. @hosungsmsft please review the below:

skufamily appears only in documentation and has meaningless allowed values

$ grep -rnw '.' --include '*.json' -e 'skuFamily'
./azuredeploy.json:257:        "skuFamily": {
./azuredeploy.json:729:            "skuFamily": "[parameters('skuFamily')]",
./nested/mssql.json:50:        "documentation6": " skuFamily                   - Mssql sku family",
./nested/postgres.json:43:                "family": "[parameters('moodleCommon').skuFamily]",
./nested/postgres.json:58:        "documentation6": " skuFamily                   - Postgresql sku family",
./nested/mysql.json:43:                "family": "[parameters('moodleCommon').skuFamily]",
./nested/mysql.json:58:        "documentation6": " skuFamily                   - Mysql sku family",

The only use that looks potentially relevant is ./nested/mysql.json:43 but given that this is not a parameter in the REST API I believe it is not required - https://docs.microsoft.com/en-us/rest/api/mysql/servers/createorupdate#serverpropertiesfordefaultcreate

Moodle has released version 3.4.1, we should add it to the templates.

We need to document the process for releasing from this repo into the Azure Quickstarts repo. Here's an initial set of ideas:

We aim to release every 2nd and 4th Wednesday

Version numbering

Version numbers are x.m.i, where x = major version number, m = month number, i = iteration number (e.g. 1.1.2 is the version from the 4th Wednesday of the month). If interim releases are needed (e.g. security patch) they will be indicated with an addition digit, e.g. x.m.i.y, where y is an incremental number

Release Preparation (1st and 3rd Wednesday)

1. validate all issues with milestone of next release. Move out issues that are going to slip.
2. Merge dev branch into releasecandidate branch
3. Increment version numbers in dev branch
4. test and validate releasecandidate branch

Release Process (2nd and 4th Wednesday)

4. Update URLs for deploy button and script locations
5. Update version numbers
6. cp code to azure/quickstarts fork
7. issue pull request
8. validate against best practice
9. merge into quickstarts

docs/SslCert.md needs to be rewritten to allow execution in SimDem

"commandToExecute" contains secrets (passwords) and shouldn't be in the "settings" (plaintext), but placed in the "protectedSettings". See this (the Note at the bottom).

In #72 we are adding a test infrastructure to this repo that will allow us to validate contributions to this repo. This solution will do some syntax and best practice checking of the contributions and it will test that a deployment of the template works from an ARM perspective. However, it does not test that the Moodle is working and available.

Once this initiual work (#72) is complete we need to extend it to allow a deeper level of testing. For example:

1. deploy a configuration of Moodle
2. tests the correct installation of selected features in that configuration
3. runs the load testing against it and report
   [repeat for x different configurations]

Lets start with a focus on 1)

Our documentation is written to allow it to be used as a test script. Installing SimDem and running simdem -p . test Deploy.md in the docs folder will test the deployment of Moodle (item 1 above).

There is an example of using SimDem in Circle CI at https://github.com/Azure/acs-demos/blob/master/.circleci/config.yml

This is a good starting point, but happy to discuss further here once we start looking at this in detail.

If we add appropriate tags to the ARM templates this will allow Azure to track use of these templates and therefore empower Moodle partners to offer paid support to customers who want it.

Here is the link on GitHub https://github.com/catalyst/moodle-search_azure and it is also in the Moodle plugin directory here: https://moodle.org/plugins/search_azure

Here is the ARM Quickstart for having an Azure Search Service: https://azure.microsoft.com/en-us/resources/templates/101-azure-search-create/

#14 addresses the SSL cert issue, but it also talks about auto-rotating the deployed cert when it's near expiry or revoked. This requires a full integration of Azure MSI, which is currently available as a VM extension that's being replaced by IMDS. Therefore, we better wait for the IMDS MSI support before implementing the cert auto-rotation. Besides, currently the Key Vault needs to be configured (to allow access from VM) after the VM is deployed, so the auto-rotation must be configured after the deployment, which makes this task separate from the template improvement. Therefore, setting up a separate issue here.

PR #10 added the ability to choose Apache or NGinx as the server, In the PR comments the following two ToDo items were noted:

• Redirect direct outside Apache http connection (to port 81) to https (443)
• Log correct IP address instead of 127.0.0.1 for Apache http (nginx http does that already)

Hi,

Just reporting that this template needs a min of 9 Cores.
So as it stands it will not run successfully from a Free Trial account unless you set:

• fileServerType: nfs (instead of gluster)
• autoScaleVmSku: Standard_DS1_v2 (instead of DS2)
  This will reduce the core requirements to under the max of 4 for a Free Trial account

This repo contains guides on how to deploy and manage a scalable Moodle cluster on Azure. You can use an Azure free account to get started.

We can provide a bunch of pre-commit checks to catch common errors and, in some cases, provide scripts to help.

• Build a script to check the deploy, visualize, nested template and script URLs - reject commit if they don't match the git branch
  • This also needs to be run on pre-merge via the web UI, otherwise we run the risk of master not having the correct values
• Build a script to automatically update the deploy, visualize, nested template and script URLs - reject commit if they don't match the git branch

Architecture diagram should be updated to show latest deployment configurations (MS SQL, Azure Search etc.).

The current template uses the vanilla Azure Load Balancer with nginx https proxy to terminate https connections. It'd be good to allow choice of Azure App Gateway, over the current default setting. The SSL cert deployment on the Azure App Gateway might remain a small issue?

We should have a CI/CD process that validates every PR for best practices and template deployment correctness. A CI/CD process for our repo should also handle the pre-commit checks mentioned in #7. It'll probably closely follow what Azure/azure-quickstart-templates is doing, especially with their Travis CI process. With the latest azuredeploy.parameters.json and added options (MS SQL, optional redis, optional elastic search, ...) a test deployment should be possible within reasonable amount of time (about 30 minutes).

For an example CI/CD build on the Azure/azure-quickstart-template repo, see this (a PR from myself to integrate our changes in Azure/Moodle to Azure/azure-quickstart-templates repo) and the actual Travis CI build logs (can be accessed from the PR link by clicking the "Show All Checks" link and then clicking the "Details" link for continuous-integration/travis-ci/pr item.

Derived form comments on #77

A full CI is expensive to run for every commit/push in any branch, but it is interesting to still run other tests (linting/validation).

Currently install_moodle.sh and setup_webserver.sh require a bunch of command line arguments for a single site, which grow more and more and won't be sustainable sooner than later. Ideally those command line parameters should be packed in a config file and passed down to the VM deployment template. It might require using cloud-init as mentioned in #33. I'm raising another issue, because this will also help supporting multiple Moodle sites with a single deployment (#78). Eventually we'd want a config file (for multiple Moodle sites) created/shared/updated on /moodle and let each VMSS VM instance configure its html directories, moodledata directories, and Apache virtual hosts as stated in the config file. The controller VM will need to create that config file based on the deployment-time custom data (for cloud-init) that should replace the install_moodle.sh cmdline parameters, and probably the script itself with something more flexible like a Python script (to facilitate processing the custom data and config file generation better).

Azure MySQL/Postgres require some specific database sizes (e.g., 51200, 102400, 128000, ...), but our current template accepts arbitrary numeric values, which are rejected only after the deployment is started. We should add the "allowedValues" property to that param, to fail fast.

For a certain VM sku (e.g., Standard_DS2_v2, which is the default) for the web frontend VMSS, it's observed that the frontend VM's CPU utilization didn't hit the default scaling-out threshold (80%) even with very high latency (~20s) with high concurrent connection count (~200). The best scaling-out logic might be to use the latency as the trigger, but that may not be immediately feasible. It might be good to find a better scaling-out threshold so that latency wouldn't suffer. The threshold might differ from one VM sku to another, and that'll require some very extensive performance test experiments.

Unoconv is a command line program that is used to convert between different office document file formats. It uses an instance of LibreOffice to do the conversion and is used by the Assignment activity to convert documents to pdf so that they can be annotated. If unoconv is not installed - the only impact is that the assignment activities will only allow annotations when students upload a pdf document.

How widely used is this? What is the impact if we install it? Can we make it an optional install?

Would it be better for cron to run in parallel using auto-scaling web tier?

Moodle supports cron running in parallel (there is built in locking) for micro tasks. It is recommended that cron runs at every 5 minutes but some tasks could take hours depending on situation so shouldn't cron also scale?

It's also worth noting that providing a bigger controller VM might not help since Moodle's Cron reportedly does not use more than one CPU core.

In #29 and #32 we added the ability to configure the deployment of the elasticsearch VMs and O365 Plugins, There are also a number of other configuration options not currently exposed via the default azuredeploy.parameters.json file.

We need to document these options clearly and provide some example "t-shirt size" configurations to guide users. That is, what configuration should be used for "5000 students", "10000 students", "25000 students' etc.

Currently for file content search capability (a.k.a. file indexing support) with the Moodle Azure Search plugin, we need a separate Apache Tika instance. Our latest templates after integrating the plugin still does not create/configure for a Tika instance, so the file indexing support is not possible. We should extend the templates so that a Tika instance can be created/configured for the Moodle global search file indexing support.

The current templates (optionally) deploys an Elastic Search cluster. This adds 3 VMs to the overall cost of the template. By adding Azure Search we should be able to provide better functionality with a lower maintenance overhead and lower cost.

It is important that we ensure contributors know how to contribute. To this end we should create a CONTRIBUTE.md file that describes our governance model and how newcomers can become engaged.

A PR from a different branch or from a different fork shouldn't update the base URLs in azuredeploy.json and README.md (the ones with https://raw.githubusercontent.com/Azure/Moodle/master). We should add a step in CI to make sure these values are not changed and reject the PR if any of those are changed.

It would be useful to be able to host multiple user sites within a single Moodle deployment. This would not be a truly multi-tenant solution as there would be shared database server etc. However, different front ends to the system would provide different Moodle sites.

In the readme can we add one of the Next Steps for Azure Managed Application be How to Submit your Moodle Managed Application to the Azure Marketplace and link to the following documentation

And also Formally submit your Application to the Azure Marketplace and link to this nomination page

The existing Managed Application documentation has a dual focus - including using it to create IT Service Catalog and offerings - and does not strongly reference the Azure Marketplace and that being a natural use case for Managed Applications.

PR #61 introduced the /moodle/html local copy feature (primarily targeted for the gluster option), and this needs some documentation, as @rgardler suggested and copied below:

In general I think this is OK as an optional feature. I do think it needs documenting though, nothing too fancy just the outline steps in a new file at docs/moodle_file_sync.md:

• Update files in /moodle
• Update last modified file
• wait 1 minute

Azure Managed Applications enables partners to offer their Moodle based solutions to internal and external customers. The ARM templates in this repository provide the core infrastructure for such a solution, while the terms for ongoing management of the solution, along with specific customizations can be provided by the partner.

In order to help people get started with Moodle based Managed Applications on Azure we should provide an example configuration with documentation for upload to the marketplace.

Using cloud-init to do boot time configuration of our VMs might provide a level of increased flexibility during out deployment. Should we move from customer script extension to cloud-init?

Intro blog: https://azure.microsoft.com/en-us/blog/custom-data-and-cloud-init-on-windows-azure/
Overview documentation: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/using-cloud-init
Quickstart: https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-customdata

Currently the template use NGinx as the web server. However, many people prefer Apache Web Server. It would be good to have an option to deploy with either configuration.

hey,
I pressed the deploy button and got this error , any idea what to do?

Now that we have run a series of load tests against the new "local copy" feature for PHP files should we turn it on by default?

We'd like to containerize at least the frontend web VM scale set, if not the entire cluster. Kubernetes is considered a viable orchestrator, but it's uncertain how app containers may be deployed using ARM templates, so need investigation on that.

We also learned recently that Catalyst IT EU presented their Moodle containerization experiences in MoodleMoot UK/Ireland 2018 (search for "containerisation" after clicking the link), and it'd be great if we can learn and improve from their experiences.

Initially setting the milestone as v1.5.1 for planning, but it's well understood that this work might take longer, in which case the milestone will be adjusted accordingly.

In the EU it is likely that admins will want to install the Moodle GDPR plugins - Policy Plugin
and Data Privacy Plugin. We should provide a single parameter to enable the automatic deployment of these plugins.

As an example see the O365 flag implemented in PR #32

We started with Gluster, but it turns out that its performance degrades quickly with medium to high workloads. So we switched our default file share option to NFS, but it's not highly available.

We should provide a highly available and yet performant file share alternative. NFS HA (this and this) seems to be a possible option, but it feels too hairy a solution. lsyncd seems a promising alternative, just for the /moodle/html directory, while still keeping the /moodle/moodledata directory in gluster.

The Trigger Plugin (https://github.com/catalyst/moodle-tool_trigger) and the Restful Web Services Plugin (https://github.com/catalyst/moodle-webservice_restful) should be installed optionally so that the site can be configured to use the Custom Azure Logic Apps Connector for Moodle (https://github.com/catalyst/azure-connector_moodle). Not sure if we can automate the configuration of the Custom Logic App Connector - but perhaps we just have an "Enable Azure Logic Apps with Moodle" in the web UI.

At present the deployment uses self-signed SSL certs. In order to update those certs one has to upload them to the server after deployment. It would be better to allow the cert to be provided/created at deployment if possible.

Does the MS SQL implementation work with Azure SQL Database Managed Instance? https://docs.microsoft.com/en-us/azure/sql-database/sql-database-managed-instance

The Create UI definition for Managed Apps allows more flexibility in the way the UI is displayed. For example, instead of a cryptic sku name we can create an illustrative name, see https://docs.microsoft.com/en-us/azure/managed-applications/microsoft-common-dropdown

We can also use logical functions to hide elements, see https://docs.microsoft.com/en-us/azure/managed-applications/create-uidefinition-functions#logical-functions

So we can, for example, create three groups of SKU dropdowns for postgres, mysql and mssql and only have them display when the appropriate DB is selected.

It is currently possible use an existing vNet address space. However, the feature is not well tested, nor is it documented.

• Test deployment into an existing VNet
• Document deployment into an existing VNet