Salt Orchestration: The orchestrate Runner for Multi-Step, Multi-Server Workflows

SaltStack 28 min min read Updated: Nov 17, 2026 Advanced
Salt Orchestration: The orchestrate Runner for Multi-Step, Multi-Server Workflows
Advanced Topic 3 of 3

Why Orchestration?

A highstate runs states on one minion at a time, independently. Real deployments need an order across machines: 'back up the database, then upgrade the database, then upgrade web servers in small groups, then run smoke tests'. Orchestration runs on the master, controls multiple minions and steps, and can wait for each step to finish.

HighstateOrchestration
Runs onEach minion for itselfThe master controlling many minions
Order between machinesNoneYes, with requisites
Good forMaking one machine correctDeployments, upgrades, cluster changes
Commandsalt '*' state.applysalt-run state.orchestrate orch.deploy

Your First Orchestration

yaml # /srv/salt/orch/deploy.sls backup_database: salt.state: - tgt: 'db1' - sls: - postgres.backup upgrade_database: salt.state: - tgt: 'db1' - sls: - postgres.upgrade - require: - salt: backup_database upgrade_web: salt.state: - tgt: 'web*' - sls: - app.deploy - batch: 25% # a quarter of the web servers at a time - require: - salt: upgrade_database smoke_test: salt.function: - name: http.query - tgt: 'web1' - arg: - http://localhost/health - require: - salt: upgrade_web
bash sudo salt-run state.orchestrate orch.deploy sudo salt-run state.orchestrate orch.deploy test=True sudo salt-run state.orchestrate orch.deploy pillar='{"version": "1.4.0"}' sudo salt-run state.orchestrate orch.deploy saltenv=feature_x
Orchestration functionDoes
salt.stateRun state SLS files on targeted minions (like state.apply)
salt.functionRun an execution function on minions (like salt 'x' fun args)
salt.runnerRun a runner function on the master
salt.wheelRun a wheel function (keys, config files)
salt.parallel_runners, salt.parallel (newer)Run steps at the same time
salt.wait_for_eventWait until an event arrives
salt.runner + state.eventEvent-driven flows
Names are arguments: In orchestration the step ID is arbitrary. The important parts are the function (salt.state), tgt, and what to run (sls, name, or fun). require: - salt: step_id links steps.

Target Types and Batches

yaml upgrade_group: salt.state: - tgt: 'G@roles:web and not G@env:dev' - tgt_type: compound - sls: app.deploy - batch: 2 - failhard: True

Passing Pillar and Data Between Steps

yaml # Pass pillar in deploy_app: salt.state: - tgt: 'web*' - sls: app.deploy - pillar: app_version: {{ pillar.get('version', '1.4.0') }} # Use a return value in a later step (Jinja in orchestration runs on the master) {% set ips = salt['saltutil.runner']('cache.grains', tgt='web*').values() %} {% for g in ips %} # ... loop over data to make steps {% endfor %}
yaml # Use data returned by a runner at render time {% set up = salt['saltutil.runner']('manage.up') %} {% if 'web1' not in up %} fail_if_web1_down: test.fail_without_changes: - name: web1 is not connected - failhard: True {% endif %}

Rolling Deployment With a Health Check

yaml # /srv/salt/orch/rolling_deploy.sls {% set version = pillar.get('version', 'latest') %} {% for minion in salt['saltutil.runner']('manage.up', tgt='web*') | sort %} drain_{{ minion }}: salt.function: - name: cmd.run - tgt: lb1 - arg: - /usr/local/bin/lb-drain.sh {{ minion }} {% if not loop.first %} - require: - salt: health_{{ loop.previtem }} {% endif %} deploy_{{ minion }}: salt.state: - tgt: {{ minion }} - sls: app.deploy - pillar: version: {{ version }} - require: - salt: drain_{{ minion }} - failhard: True health_{{ minion }}: salt.function: - name: http.query - tgt: {{ minion }} - arg: - http://localhost:8080/health - kwarg: status: True - retry: attempts: 10 interval: 6 - require: - salt: deploy_{{ minion }} enable_{{ minion }}: salt.function: - name: cmd.run - tgt: lb1 - arg: - /usr/local/bin/lb-enable.sh {{ minion }} - require: - salt: health_{{ minion }} {% endfor %}
Jinja loops with requisites: The rolling example is a pattern. Check the loop variables (loop.previtem) in your Jinja version, test with test=True on a lab, and keep the number of generated steps reasonable. For very large fleets, use batch instead of one step per minion.

Handle Failure

yaml deploy_app: salt.state: - tgt: 'web*' - sls: app.deploy - batch: 25% rollback_app: salt.state: - tgt: 'web*' - sls: app.rollback - onfail: - salt: deploy_app notify_failure: salt.runner: - name: salt.cmd - fun: slack.post_message - channel: '#ops' - message: 'Deployment failed, rollback ran' - onfail: - salt: deploy_app

Run Orchestration From Events and APIs

bash # From the reactor: runner.state.orchestrate (see the reactor lesson) # From salt-api: curl -sS -k https://salt-master:8000/run -H 'Content-type: application/json' -d '[{ "client": "runner", "fun": "state.orchestrate", "mods": "orch.deploy", "pillar": {"version": "1.4.0"}, "eauth": "pam", "username": "deployer", "password": "***" }]'

Orchestration vs Other Tools

NeedChoice
Order steps across servers, inside SaltOrchestrate runner
Quick batch of one commandsalt -b 2 ...
Respond to an eventReactor (which can start an orchestration)
Big pipelines with approvalsCI/CD tool (GitHub Actions, GitLab, Jenkins) that calls salt-api or salt-run
Cloud infrastructureTerraform (then Salt configures the machines)

Practice Tasks

  1. Write an orchestration with a database step followed by a web step and run it with test=True
  2. Add batch: 1 and failhard: True and observe the behaviour with a failing state
  3. Add an onfail step that writes a message to a file
  4. Pass version through pillar and use it in the deploy state
  5. Trigger the orchestration from a custom event with a reactor file

Conclusion

Orchestration runs on the master and controls order across machines with salt.state, salt.function and requisites. Use batches, failhard, health checks and onfail steps to build safe deployments, and trigger them from events or CI.

Quick Summary: state.orchestrate; salt.state/function/runner/wheel; require: - salt: id; batch; onfail rollback; run from reactor or API.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators