Infrastructure as Code Needs a Workflow
Salt files are code. Treat them like code: every change goes through Git, is reviewed, tested, and promoted step by step. This prevents the classic accident: one bad state edited on the master and applied to everything.
text
developer branch ──► Pull Request ──► CI: lint + render + container tests ──► review ──► merge to main
│
gitfs (dev env) ◄─ automatic ◄───────┘
gitfs (staging) ◄─ tag / promote
gitfs (prod) ◄─ tag + approval, canary then batches
Mapping Git Refs to Environments
With gitfs, each branch or tag can be a Salt environment (saltenv). A common simple model is below.
| Git ref | saltenv | Used by |
Branch main | base | Dev or lab servers |
Tag staging | staging | Staging servers |
Tag prod | prod | Production servers (moved after approval) |
| Feature branches | feature-xyz (optional) | Testing one change on one server |
yaml
# /etc/salt/master.d/gitfs.conf
fileserver_backend:
- gitfs
gitfs_provider: pygit2
gitfs_remotes:
- git@github.com:example/salt-states.git:
- base: main
- root: salt
- saltenv:
- staging:
- ref: staging
- prod:
- ref: prod
gitfs_update_interval: 60
gitfs_saltenv_whitelist: [base, staging, prod]
yaml
# Each minion is placed in its environment (the minion config or a grain)
# /etc/salt/minion.d/env.conf
saltenv: prod # states come from the prod tag
pillarenv: prod
Branching Rules
main is protected. No direct pushes. Pull request and at least one review are required
- CI must pass before merge
- After merge, dev servers get the change automatically (gitfs update)
- A person (or a pipeline job with an approval) moves the
staging and prod tags forward
- A rollback is a
git revert or moving the tag back to the previous commit
bash
# Promote main to staging
git fetch origin
git tag -f staging origin/main
git push -f origin staging
# Promote after validation to production (use a protected workflow in real life)
git tag -f prod staging
git push -f origin prod
# Roll back production to the previous known good commit
git tag -f prod <old-commit-sha>
git push -f origin prod
Why tags? Moving a tag is a clear, auditable action: "production now points to commit X". Many teams use release tags such as prod-2026-11-17 and move a branch to it. Pick one rule and keep it.
A Pipeline: Test, Deploy, Verify
yaml
# .github/workflows/salt.yml
name: salt-pipeline
on:
pull_request:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install yamllint salt-lint pytest pytest-testinfra docker
- run: make lint
- run: make render
- run: pytest -v tests/
deploy-staging:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: staging # can require approval
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- run: |
git tag -f staging ${{ github.sha }}
git push -f origin staging
- name: Dry run on staging through salt-api
env:
SALT_API: ${{ secrets.SALT_API_URL }}
SALT_USER: ${{ secrets.SALT_USER }}
SALT_PASS: ${{ secrets.SALT_PASS }}
run: |
TOKEN=$(curl -sS "$SALT_API/login" -H 'Accept: application/json' \
-d username="$SALT_USER" -d password="$SALT_PASS" -d eauth=pam | jq -r '.return[0].token')
curl -sS "$SALT_API/" -H "X-Auth-Token: $TOKEN" -H 'Accept: application/json' \
-d client=local -d tgt='G@env:staging' -d tgt_type=compound \
-d fun=state.apply -d kwarg='{"test": true, "saltenv": "staging"}' | tee dryrun.json
- uses: actions/upload-artifact@v4
with: { name: dryrun, path: dryrun.json }
API credentials: Use a CI-only user with narrow external_auth rights (only state.apply on staging targets). Store the password in your CI secret store. Prefer a network path that is private or allow-listed.
Safe Production Rollout
Never apply a new change to all production servers at once. Use canary and batches.
bash
# 1) Preview on everything (no change)
salt -G 'env:prod' state.apply test=True saltenv=prod --state-output=changes
# 2) Canary: one server first
salt 'web-prod-01' state.apply saltenv=prod
salt 'web-prod-01' state.apply saltenv=prod # second run should show no change
# 3) Check health (HTTP, metrics, logs), then a small batch
salt -G 'env:prod and roles:web' -b 10% state.apply saltenv=prod
# 4) Everyone else
salt -G 'env:prod' -b 20% state.apply saltenv=prod
yaml
# The same flow as an orchestration file: /srv/salt/orch/prod_rollout.sls
canary:
salt.state:
- tgt: 'web-prod-01'
- sls: highstate_marker
- saltenv: prod
health_check:
salt.function:
- name: http.query
- tgt: 'web-prod-01'
- arg: ['http://localhost/health']
- kwarg: {status: True}
- require:
- salt: canary
rollout_batches:
salt.state:
- tgt: 'G@env:prod and G@roles:web'
- tgt_type: compound
- highstate: True
- saltenv: prod
- batch: 20%
- failhard: True
- require:
- salt: health_check
Drift Reports and Scheduled Highstates
- Run
state.apply test=True every night with the minion schedule or a CI scheduled job, and send the results to a report. A server with drift gets investigated
- Decide if highstate runs automatically on a schedule (self-correcting) or only on demand (safer, but drift grows). Many teams enable it for baseline states and keep app deploys manual
- Keep
startup_states or a schedule off for risky formulas, or protect them with pillar flags
Secrets in Pipelines
- Pipeline secrets: only in the CI secret store, scoped per environment
- Salt secrets: encrypted pillar, Vault or SDB, not in the Git repo as plain text
- Never print secrets in logs. Mask them in CI
- Use short-lived credentials where possible (OIDC to Vault or the cloud)
Practice Tasks
- Create a repository with
salt/ and pillar/, and add a pull request check that runs make check
- Point
gitfs at your repository with base, staging and prod environments
- Promote a change to staging by moving a tag, then dry-run on staging minions
- Write a canary and batch rollout for production with
batch and failhard
- Practise a rollback by moving the
prod tag to the previous commit
Conclusion
Run Salt as software delivery: every change goes through Git, review, lint and container tests, then moves through environments with gitfs tags. Use dry runs, canaries and batches for production, and roll back by moving a tag.
Quick Summary: Git PR + CI; gitfs refs = saltenv; tags staging/prod; dry run via salt-api; canary then batches; rollback = move tag; secrets in CI store.