Beacons, the Schedule, Engines and Self-Healing: Watch Machines and Run Jobs on Time

SaltStack 26 min min read Updated: Nov 17, 2026 Advanced
Beacons, the Schedule, Engines and Self-Healing: Watch Machines and Run Jobs on Time
Advanced Topic 2 of 3

What Are Beacons?

A beacon runs on the minion and watches something: a service, disk space, a file, a log, the load. When a condition is met, the beacon sends an event to the master. The master's reactor can then act. Together they give you monitoring and self-healing without polling from the master.

text minion: beacon watches /var (disk) ──event──► master event bus ──► reactor ──► clean-up job on that minio

Configure Beacons

yaml # /etc/salt/minion.d/beacons.conf (or set through pillar: beacons:) beacons: service: - services: nginx: onchangeonly: True # only send an event when the state changes delay: 5 sshd: onchangeonly: True - interval: 30 diskusage: - /: 85% # fire if / is more than 85% used - /var: 90% - interval: 120 load: - averages: 1m: [0.0, 4.0] # fire if the 1 minute load is outside 0 to 4 5m: [0.0, 3.0] - interval: 60 inotify: - files: /etc/ssh/sshd_config: mask: - modify - delete - disable_during_state_run: True # ignore changes caused by Salt itself - interval: 5
bash sudo systemctl restart salt-minion salt 'web1' beacons.list # beacons configured on the minion salt 'web1' beacons.add service '[{"services": {"nginx": {"onchangeonly": true}}}]' salt 'web1' beacons.disable_beacon service salt 'web1' beacons.enable salt 'web1' beacons.save # write current beacons to config salt-run state.event pretty=True tagmatch='salt/beacon/*' # see them fire
BeaconWatches
serviceService state (running or stopped)
diskusagePercent of space used on a mount point
load, memusage, statusSystem load, memory, status information
inotifyFile and directory changes (needs the pyinotify Python library in the onedir)
log, journaldPatterns in log files or the journal
network_info, network_settingsTraffic counters and network settings
proxy_example, salt_monitorCustom checks or running any Salt function and sending the result
pkg, sshd, adb, btmp, wtmp, haproxy, smartos_vmadmOther special-purpose beacons
List them all: salt 'web1' sys.list_modules | grep -i beacon does not list beacons. Look in the Salt docs under Beacon modules, or read salt.beacons in the installed package. Each beacon has a sample configuration in its docstring.

Self-Healing: Beacon + Reactor

yaml # master reactor.conf reactor: - 'salt/beacon/*/service/*': - /srv/reactor/heal_service.sls
yaml # /srv/reactor/heal_service.sls {# Event data example: {'id': 'web1', 'nginx': {'running': False}, 'service_name': 'nginx'} #} {% if not data.get(data.get('service_name', ''), {}).get('running', True) %} restart_service: local.service.start: - tgt: {{ data['id'] }} - arg: - {{ data['service_name'] }} tell_the_team: runner.salt.cmd: - args: - fun: slack.post_message - channel: '#ops' - message: 'Salt restarted {{ data["service_name"] }} on {{ data["id"] }}' {% endif %}
Prevent restart loops: If a service keeps crashing, endless restarts hide the real problem. Add counters or a delay, and alert humans after N attempts (for example, store a counter with the cache module or in an external system).

The Minion Scheduler

The schedule runs Salt functions on the minion at times you choose, even if the master is offline. It is built into the minion.

yaml # /etc/salt/minion.d/schedule.conf (or in pillar under 'schedule:') schedule: highstate_every_hour: function: state.highstate minutes: 60 splay: 600 # random delay up to 10 minutes maxrunning: 1 drift_report: function: state.apply args: - common kwargs: test: True cron: '0 6 * * 1-5' # weekdays at 06:00 (needs python-croniter in the onedir) returner: syslog clean_tmp: function: cmd.run args: - find /tmp -type f -mtime +7 -delete hours: 24 jid_include: True one_time_task: function: pkg.upgrade once: '2026-12-01T02:00:00' # run once at this time once_fmt: '%Y-%m-%dT%H:%M:%S' work_hours_only: function: test.ping seconds: 300 range: start: 9:00am end: 5:00pm
OptionMeaning
seconds, minutes, hours, daysRepeat every N units
whenA time like '5:00pm' or a list
cronA cron expression
once, once_fmtRun once at a date and time
splayRandom delay (seconds, or {start: 10, end: 60})
range and skip_during_rangeAllowed time window or a quiet window
maxrunningHow many copies may run at the same time
returner, return_jobWhere results go
run_on_start: TrueAlso run when the minion starts
enabled: FalseDisable without deleting
jid_include, metadataAdd job id or custom data to the job

Manage the Schedule From the Master

bash salt 'web1' schedule.list salt 'web1' schedule.add check_disk function='disk.usage' minutes=15 persist=True salt 'web1' schedule.disable_job check_disk salt 'web1' schedule.enable_job check_disk salt 'web1' schedule.run_job check_disk # run now salt 'web1' schedule.delete check_disk persist=True salt 'web1' schedule.reload
yaml # Better: manage the schedule with a state, so it stays correct nightly_backup_job: schedule.present: - name: nightly_backup - function: cmd.run - job_args: - /usr/local/bin/backup.sh - when: 2:30am - splay: 900 - maxrunning: 1 - persist: True
Schedule vs cron: Both work. The Salt schedule is easy to manage in states and from the master, can return results to Salt returners, and can run any Salt function (like state.apply). Normal cron is simpler for shell scripts, and survives even if Salt is broken.

Engines: Long-Running Helpers on the Master or Minion

Engines are services that run inside the Salt master or minion process. They can listen for events and talk to chat systems, receive webhooks, or forward events elsewhere.

yaml # /etc/salt/master.d/engines.conf engines: - slack: token: xoxb-use-sdb-or-vault-not-plain-text control: True # allow commands from Slack valid_users: - alice valid_commands: - test.ping - cmd.run aliases: list_jobs: type: runner cmd: jobs.list_jobs - webhook: port: 5000 address: 127.0.0.1 tag_prefix: hook/
ChatOps access control: A chat engine with control: True can run commands. Restrict users and commands, run it only over TLS, and never allow cmd.run in production chat without strong controls.

Putting It Together: A Simple Alert Flow

  1. diskusage beacon fires on a minion when /var is above 90%
  2. The master reactor receives salt/beacon/web1/diskusage/ and runs a clean-up
  3. The reactor posts a message to chat with the minion and result
  4. The schedule runs a nightly state.apply in test mode to report drift
  5. Reports go to a returner for later review

Practice Tasks

  1. Add a service beacon for sshd and watch the event when you stop it
  2. Write a reactor that restarts a stopped service and sends a log line
  3. Create a schedule that runs test.ping every minute and returns to syslog
  4. Use schedule.present to create a job from a state
  5. Describe how you prevent a restart loop in a self-healing design

Conclusion

Beacons watch machines and emit events, the reactor acts on them, and the minion scheduler runs jobs on time. Combine them for self-healing, keep actions safe, and always alert people when automation intervenes.

Quick Summary: beacons: service/diskusage/inotify; salt/beacon/* events -> reactor; schedule: minutes/cron/once/splay; schedule.present; engines for chat.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators