Running on Azure Batch

Azure Batch is a cloud-based service that allows you to run large-scale parallel and high-performance computing applications. This section provides an overview of how to set up and run SCHISM on Azure Batch.

Setup Azure

Download Azure Utilities

Azure is a paid service separate from SCHISM, BayDeltaSCHISM, and CA-DWR. Once you have set up your Azure batch and storage accounts within your Azure resource group, you’ll need to download a few things:

• Azure CLI

  • download and install on your machine to pass jobs (simulations) to Azure batch

• AzCopy

  • download and install on your machine to upload content to storage container

• Azure Batch Explorer (desktop app)

  • this isn’t technically necessary, but is quite helpful in managing ongoing simulations

• Azure Storage Explorer (desktop app)

  • again, not technically necessary, but quite useful

Setup can be done via az commands. Here we setup a batch account with associated storage.

Login

Login with your Azure credentials. First go to https://microsoft.com/devicelogin and copy your login code. Then after typing:

az login --use-device-code

Copy in the code to verify your credentials.

Create a Resource Group

See the Azure docs for details. To use the commands below, enter your values (replacing the <angle brackets and values>)

az group create --name <resource_group_name> --location <location_name>

az storage account create --resource-group <resource_group_name> --name <storage_account_name> --location <location_name> --sku Standard_LRS

az batch account create --name <batch_account_name> --storage-account <storage_account_name> --resource-group <resource_group_name> --location <location_name>

You can also create the batch account and associated account as explained here.

Download azure_dms_batch

Use the environment.yml with conda to create an environment called azure

conda env create -f environment.yml

or

pip install -r requirements.txt

Git clone *this* project

git clone https://github.com/CADWRDeltaModeling/azure_dms_batch.git

Change directory to the location of the azure_dms_batch project and then install using:

pip install --no-deps -e .

For more information on the azure_dms_batch package, see the README.md file.

References

Python SDK Setup BlobStorage Python Example Azure Batch Python API Azure Batch Python Samples Azure Batch Shipyard

MPI specific Azure Batch MPI Cluster configuration options

Intel MPI Azure settings for Intel MPI Intel MPI Pre-requisites