Beacons, the Schedule, Engines and Self-Healing: Watch Machines and Run Jobs on Time
What Are Beacons?
A beacon runs on the minion and watches something: a service, disk space, a file, a log, the load. When a condition is met, the beacon sends an event to the master. The master's reactor can then act. Together they give you monitoring and self-healing without polling from the master.
text
minion: beacon watches /var (disk) ββeventβββΊ master event bus βββΊ reactor βββΊ clean-up job on that minio
Configure Beacons
yaml
# /etc/salt/minion.d/beacons.conf (or set through pillar: beacons:)
beacons:
service:
- services:
nginx:
onchangeonly: True # only send an event when the state changes
delay: 5
sshd:
onchangeonly: True
- interval: 30
diskusage:
- /: 85% # fire if / is more than 85% used
- /var: 90%
- interval: 120
load:
- averages:
1m: [0.0, 4.0] # fire if the 1 minute load is outside 0 to 4
5m: [0.0, 3.0]
- interval: 60
inotify:
- files:
/etc/ssh/sshd_config:
mask:
- modify
- delete
- disable_during_state_run: True # ignore changes caused by Salt itself
- interval: 5
bash
sudo systemctl restart salt-minion
salt 'web1' beacons.list # beacons configured on the minion
salt 'web1' beacons.add service '[{"services": {"nginx": {"onchangeonly": true}}}]'
salt 'web1' beacons.disable_beacon service
salt 'web1' beacons.enable
salt 'web1' beacons.save # write current beacons to config
salt-run state.event pretty=True tagmatch='salt/beacon/*' # see them fire
| Beacon | Watches |
service | Service state (running or stopped) |
diskusage | Percent of space used on a mount point |
load, memusage, status | System load, memory, status information |
inotify | File and directory changes (needs the pyinotify Python library in the onedir) |
log, journald | Patterns in log files or the journal |
network_info, network_settings | Traffic counters and network settings |
proxy_example, salt_monitor | Custom checks or running any Salt function and sending the result |
pkg, sshd, adb, btmp, wtmp, haproxy, smartos_vmadm | Other special-purpose beacons |
List them all: salt 'web1' sys.list_modules | grep -i beacon does not list beacons. Look in the Salt docs under Beacon modules, or read salt.beacons in the installed package. Each beacon has a sample configuration in its docstring.
Self-Healing: Beacon + Reactor
yaml
# master reactor.conf
reactor:
- 'salt/beacon/*/service/*':
- /srv/reactor/heal_service.sls
yaml
# /srv/reactor/heal_service.sls
{# Event data example: {'id': 'web1', 'nginx': {'running': False}, 'service_name': 'nginx'} #}
{% if not data.get(data.get('service_name', ''), {}).get('running', True) %}
restart_service:
local.service.start:
- tgt: {{ data['id'] }}
- arg:
- {{ data['service_name'] }}
tell_the_team:
runner.salt.cmd:
- args:
- fun: slack.post_message
- channel: '#ops'
- message: 'Salt restarted {{ data["service_name"] }} on {{ data["id"] }}'
{% endif %}
Prevent restart loops: If a service keeps crashing, endless restarts hide the real problem. Add counters or a delay, and alert humans after N attempts (for example, store a counter with the cache module or in an external system).
The Minion Scheduler
The schedule runs Salt functions on the minion at times you choose, even if the master is offline. It is built into the minion.
yaml
# /etc/salt/minion.d/schedule.conf (or in pillar under 'schedule:')
schedule:
highstate_every_hour:
function: state.highstate
minutes: 60
splay: 600 # random delay up to 10 minutes
maxrunning: 1
drift_report:
function: state.apply
args:
- common
kwargs:
test: True
cron: '0 6 * * 1-5' # weekdays at 06:00 (needs python-croniter in the onedir)
returner: syslog
clean_tmp:
function: cmd.run
args:
- find /tmp -type f -mtime +7 -delete
hours: 24
jid_include: True
one_time_task:
function: pkg.upgrade
once: '2026-12-01T02:00:00' # run once at this time
once_fmt: '%Y-%m-%dT%H:%M:%S'
work_hours_only:
function: test.ping
seconds: 300
range:
start: 9:00am
end: 5:00pm
| Option | Meaning |
seconds, minutes, hours, days | Repeat every N units |
when | A time like '5:00pm' or a list |
cron | A cron expression |
once, once_fmt | Run once at a date and time |
splay | Random delay (seconds, or {start: 10, end: 60}) |
range and skip_during_range | Allowed time window or a quiet window |
maxrunning | How many copies may run at the same time |
returner, return_job | Where results go |
run_on_start: True | Also run when the minion starts |
enabled: False | Disable without deleting |
jid_include, metadata | Add job id or custom data to the job |
Manage the Schedule From the Master
bash
salt 'web1' schedule.list
salt 'web1' schedule.add check_disk function='disk.usage' minutes=15 persist=True
salt 'web1' schedule.disable_job check_disk
salt 'web1' schedule.enable_job check_disk
salt 'web1' schedule.run_job check_disk # run now
salt 'web1' schedule.delete check_disk persist=True
salt 'web1' schedule.reload
yaml
# Better: manage the schedule with a state, so it stays correct
nightly_backup_job:
schedule.present:
- name: nightly_backup
- function: cmd.run
- job_args:
- /usr/local/bin/backup.sh
- when: 2:30am
- splay: 900
- maxrunning: 1
- persist: True
Schedule vs cron: Both work. The Salt schedule is easy to manage in states and from the master, can return results to Salt returners, and can run any Salt function (like state.apply). Normal cron is simpler for shell scripts, and survives even if Salt is broken.
Engines: Long-Running Helpers on the Master or Minion
Engines are services that run inside the Salt master or minion process. They can listen for events and talk to chat systems, receive webhooks, or forward events elsewhere.
yaml
# /etc/salt/master.d/engines.conf
engines:
- slack:
token: xoxb-use-sdb-or-vault-not-plain-text
control: True # allow commands from Slack
valid_users:
- alice
valid_commands:
- test.ping
- cmd.run
aliases:
list_jobs:
type: runner
cmd: jobs.list_jobs
- webhook:
port: 5000
address: 127.0.0.1
tag_prefix: hook/
ChatOps access control: A chat engine with control: True can run commands. Restrict users and commands, run it only over TLS, and never allow cmd.run in production chat without strong controls.
Putting It Together: A Simple Alert Flow
diskusage beacon fires on a minion when /var is above 90%
- The master reactor receives
salt/beacon/web1/diskusage/ and runs a clean-up
- The reactor posts a message to chat with the minion and result
- The
schedule runs a nightly state.apply in test mode to report drift
- Reports go to a returner for later review
Practice Tasks
- Add a
service beacon for sshd and watch the event when you stop it
- Write a reactor that restarts a stopped service and sends a log line
- Create a schedule that runs
test.ping every minute and returns to syslog
- Use
schedule.present to create a job from a state
- Describe how you prevent a restart loop in a self-healing design
Conclusion
Beacons watch machines and emit events, the reactor acts on them, and the minion scheduler runs jobs on time. Combine them for self-healing, keep actions safe, and always alert people when automation intervenes.
Quick Summary: beacons: service/diskusage/inotify; salt/beacon/* events -> reactor; schedule: minutes/cron/once/splay; schedule.present; engines for chat.