Step-by-step: Bay-Delta SCHISM on Azure

This guide will walk you through the process of setting up and running the Bay-Delta SCHISM model on Azure Batch.

Note

This assumes you have already set up your Azure account and have access to the Azure Batch service. If you haven’t done this yet, please refer to the Azure documentation for instructions on how to create an account and set up Batch.

Initial Setup

First, complete the Azure setup steps in Setup Azure. You should have:

  • azure_cli and azcopy command line utilities installed and configured

    • meaning you can run azcopy and az commands from the command line

    • you should be logged into your azure account

  • download and install azure_dms_batch and its conda environment

Uploading Model Input Files

Your simulation input files (e.g., mesh, forcing files, etc.) should be uploaded to the Azure Batch storage account. This is typically done using the azcopy command line utility. For more detailed info see the Microsoft documentation.

azcopy commands

The basic syntax of azcopy to copy local to Azure is:

azcopy copy "<local_directory>" "<azure_storage_account/blob_container>/?<sas_link>"
and for Azure to local:
azcopy copy "<azure_storage_account/blob_container>/?<sas_link>" "<local_directory>"

But at the Delta Modeling Section we most often use something like:

export AZLINK="https://<storage_account>.blob.core.windows.net/<blob_storage_container/"
export sas="<sas_link>"

azcopy copy "<local_directory>" "${AZLINK}<blob_storage_container>/?${sas}" --exclude-regex="outputs/.\*nc" --recursive --preserve-symlinks --dry-run

where:

  • local_directory

    • whatever local path to your simulation directory you’re uploading

  • storage_account

    • name of your Storage Account through Azure (not the same as the Batch Account name)

  • blob_storage_container

    • folder path to your blob storage container

    • this will look like a folder path (eg: project_name/simulations/)

  • sas_link

    • SAS permissions key (generated each day for security)

    • you can generate and copy this key by navigating to your storage account in the Azure Portal > going to “Containers” > find storage container > right clicking > Generate SAS > Change “Permissions”: click all boxes > Click “Generate SAS token and URL”

    • copy the “Blob SAS token option”

azcopy options

These are some of the most frequently used azcopy flag options:

  • –dry-run

    • this is useful to test your command before running

    • this flag prints a list of which files will be copied where without actually uploading/downloading anything

  • –recursive

    • this will copy all files in all subdirectories

  • –preserve-symlinks

    • any symbolic links will be preserved in the upload to the blob container

  • –include-regex

    • use a Regular Expression to limit which files are included in the upload

    • ex: –include-regex=”suisun_(2|3|7)/.*;baseline_6/.*"
      • this would upload all folder contents of

        • suisun_2/

        • suisun_3/

        • suisun_7/

        • baseline_6/

      • The .* string signifies “all contents”

  • –exclude-regex

    • use a Regular Expression to determine which files are excluded in the upload

    • this is particularly useful for things like outputs *.nc files and sflux *.nc files which are very large and costly to upload

    • ex: –exclude-regex=”outputs.*/.*nc;sflux/.*nc”

      • this would exclude any files that end in “nc” that are found in the sflux, outputs, or outputs* folders

Running Model Simulation

You can use the azure_dms_batch package’s command line utilities to submit a job to Azure.

Much of the setup for the virtual machines, compute nodes, etc. are determined by which template you specify in your run *.yml file.

Simple Run Config

Essentially, to fire a run you just need to submit the following command in a console which you have azure_cli, azcopy, and azure_dms_batch activated and which you’re logged into:

dmsbatch schism submit-job --file <path/to/sample_schism.yml>

An example of a SCHISM yaml file is here. The example yml file has many comments on each input with explanations.

Some important things to note:

The run_file.yml overrides anything in your default_config.yml (ex: alma87_mvapich2_20241018/default_config.yml)

Spot Pricing Config

The above example uses “Dedicated Nodes”. To save money (but perhaps take a bit longer time), you can use “Low Priority Nodes” or “spot pricing”. This works by sending your job to Spot VMs which run until there is another process that preempts your job.

This means your job needs the capacity to automatically restart from the last hotstart produced by SCHISM.

This example schism.yml file uses spot pricing <https://github.com/CADWRDeltaModeling/azure_dms_batch/blob/main/sample_configs/sample_schism_spot_pricing.yml>.