Salt Orchestration: The orchestrate Runner for Multi-Step, Multi-Server Workflows
Why Orchestration?
A highstate runs states on one minion at a time, independently. Real deployments need an order across machines: 'back up the database, then upgrade the database, then upgrade web servers in small groups, then run smoke tests'. Orchestration runs on the master, controls multiple minions and steps, and can wait for each step to finish.
| Highstate | Orchestration |
| Runs on | Each minion for itself | The master controlling many minions |
| Order between machines | None | Yes, with requisites |
| Good for | Making one machine correct | Deployments, upgrades, cluster changes |
| Command | salt '*' state.apply | salt-run state.orchestrate orch.deploy |
Your First Orchestration
yaml
# /srv/salt/orch/deploy.sls
backup_database:
salt.state:
- tgt: 'db1'
- sls:
- postgres.backup
upgrade_database:
salt.state:
- tgt: 'db1'
- sls:
- postgres.upgrade
- require:
- salt: backup_database
upgrade_web:
salt.state:
- tgt: 'web*'
- sls:
- app.deploy
- batch: 25% # a quarter of the web servers at a time
- require:
- salt: upgrade_database
smoke_test:
salt.function:
- name: http.query
- tgt: 'web1'
- arg:
- http://localhost/health
- require:
- salt: upgrade_web
bash
sudo salt-run state.orchestrate orch.deploy
sudo salt-run state.orchestrate orch.deploy test=True
sudo salt-run state.orchestrate orch.deploy pillar='{"version": "1.4.0"}'
sudo salt-run state.orchestrate orch.deploy saltenv=feature_x
| Orchestration function | Does |
salt.state | Run state SLS files on targeted minions (like state.apply) |
salt.function | Run an execution function on minions (like salt 'x' fun args) |
salt.runner | Run a runner function on the master |
salt.wheel | Run a wheel function (keys, config files) |
salt.parallel_runners, salt.parallel (newer) | Run steps at the same time |
salt.wait_for_event | Wait until an event arrives |
salt.runner + state.event | Event-driven flows |
Names are arguments: In orchestration the step ID is arbitrary. The important parts are the function (salt.state), tgt, and what to run (sls, name, or fun). require: - salt: step_id links steps.
Target Types and Batches
yaml
upgrade_group:
salt.state:
- tgt: 'G@roles:web and not G@env:dev'
- tgt_type: compound
- sls: app.deploy
- batch: 2
- failhard: True
Passing Pillar and Data Between Steps
yaml
# Pass pillar in
deploy_app:
salt.state:
- tgt: 'web*'
- sls: app.deploy
- pillar:
app_version: {{ pillar.get('version', '1.4.0') }}
# Use a return value in a later step (Jinja in orchestration runs on the master)
{% set ips = salt['saltutil.runner']('cache.grains', tgt='web*').values() %}
{% for g in ips %}
# ... loop over data to make steps
{% endfor %}
yaml
# Use data returned by a runner at render time
{% set up = salt['saltutil.runner']('manage.up') %}
{% if 'web1' not in up %}
fail_if_web1_down:
test.fail_without_changes:
- name: web1 is not connected
- failhard: True
{% endif %}
Rolling Deployment With a Health Check
yaml
# /srv/salt/orch/rolling_deploy.sls
{% set version = pillar.get('version', 'latest') %}
{% for minion in salt['saltutil.runner']('manage.up', tgt='web*') | sort %}
drain_{{ minion }}:
salt.function:
- name: cmd.run
- tgt: lb1
- arg:
- /usr/local/bin/lb-drain.sh {{ minion }}
{% if not loop.first %}
- require:
- salt: health_{{ loop.previtem }}
{% endif %}
deploy_{{ minion }}:
salt.state:
- tgt: {{ minion }}
- sls: app.deploy
- pillar:
version: {{ version }}
- require:
- salt: drain_{{ minion }}
- failhard: True
health_{{ minion }}:
salt.function:
- name: http.query
- tgt: {{ minion }}
- arg:
- http://localhost:8080/health
- kwarg:
status: True
- retry:
attempts: 10
interval: 6
- require:
- salt: deploy_{{ minion }}
enable_{{ minion }}:
salt.function:
- name: cmd.run
- tgt: lb1
- arg:
- /usr/local/bin/lb-enable.sh {{ minion }}
- require:
- salt: health_{{ minion }}
{% endfor %}
Jinja loops with requisites: The rolling example is a pattern. Check the loop variables (loop.previtem) in your Jinja version, test with test=True on a lab, and keep the number of generated steps reasonable. For very large fleets, use batch instead of one step per minion.
Handle Failure
yaml
deploy_app:
salt.state:
- tgt: 'web*'
- sls: app.deploy
- batch: 25%
rollback_app:
salt.state:
- tgt: 'web*'
- sls: app.rollback
- onfail:
- salt: deploy_app
notify_failure:
salt.runner:
- name: salt.cmd
- fun: slack.post_message
- channel: '#ops'
- message: 'Deployment failed, rollback ran'
- onfail:
- salt: deploy_app
Run Orchestration From Events and APIs
bash
# From the reactor: runner.state.orchestrate (see the reactor lesson)
# From salt-api:
curl -sS -k https://salt-master:8000/run -H 'Content-type: application/json' -d '[{
"client": "runner", "fun": "state.orchestrate", "mods": "orch.deploy",
"pillar": {"version": "1.4.0"},
"eauth": "pam", "username": "deployer", "password": "***"
}]'
Orchestration vs Other Tools
| Need | Choice |
| Order steps across servers, inside Salt | Orchestrate runner |
| Quick batch of one command | salt -b 2 ... |
| Respond to an event | Reactor (which can start an orchestration) |
| Big pipelines with approvals | CI/CD tool (GitHub Actions, GitLab, Jenkins) that calls salt-api or salt-run |
| Cloud infrastructure | Terraform (then Salt configures the machines) |
Practice Tasks
- Write an orchestration with a database step followed by a web step and run it with
test=True
- Add
batch: 1 and failhard: True and observe the behaviour with a failing state
- Add an
onfail step that writes a message to a file
- Pass
version through pillar and use it in the deploy state
- Trigger the orchestration from a custom event with a reactor file
Conclusion
Orchestration runs on the master and controls order across machines with salt.state, salt.function and requisites. Use batches, failhard, health checks and onfail steps to build safe deployments, and trigger them from events or CI.
Quick Summary: state.orchestrate; salt.state/function/runner/wheel; require: - salt: id; batch; onfail rollback; run from reactor or API.