How Ansible Actually Works

At 2am, somebody SSHs into web-07 and changes one value in nginx.conf. The change stops the bleeding. The incident ends. The hand edit does not.

By morning, the fleet looks healthy from ten feet away: 10 web servers and a dashboard full of green. But web-07 is now a snowflake. At this size, the fleet is too large to verify by eye and still small enough to tempt us into trying.

You’ve probably done this: SSH into the one box that’s on fire, fix it, move on. That’s not negligence, it’s triage. Drift is just triage that never got reconciled.

The problem is not that one file is wrong. The problem is that the fleet no longer has one answer to a basic question: what should this configuration be?

The bash instinct

The first response is usually a loop. It is short, legible, and dangerous in exactly the ways that matter here.

for host in $(cat hosts); do
  scp /tmp/nginx.conf "$host":/tmp/nginx.conf
  ssh "$host" 'sudo cp /tmp/nginx.conf /etc/nginx/nginx.conf && sudo systemctl reload nginx'
done

This does not find drift. It executes instructions. Every run overwrites all 10 files and reloads all 10 services, including the 9 that were already right. If web-09 times out, the loop gives us a scrollback archaeology project. If two operators run it, there is no shared account of which host changed, which failed, or which never started.

We can keep adding shell: compare checksums first, fan out safely, retain per-host exit codes, retry the unreachable machines, serialize the report. Eventually the script becomes an automation engine with unusual syntax and no test suite.

A for-loop over ssh is just Ansible with the safety removed and the reporting deleted.

The loop is not the villain. The missing state model is.

Inventory: the fleet becomes data

Ansible’s first useful idea is not YAML. It is that the fleet itself should be queryable data. web-01 through web-10 belong to the web group; connection and Python settings live beside that group instead of inside somebody’s shell history.

The source is smaller than the fleet it describes. web-[01:10] is Ansible range syntax, not a shortcut added for the post.

Now ask ansible-core what that source means. Play the real ansible-inventory --list --yaml capture and scroll. The repetition is the lesson: one range becomes 10 concrete host records before a task runs.

This capture uses ansible_connection: local and a separate /tmp directory for each logical host. That makes the 10-host experiment reproducible on one laptop. It does not fake Ansible’s control flow: inventory expansion, variables, fork scheduling, module execution, results, and the recap all run through ansible-core. On physical servers, swap the local connection plugin for SSH; the state contract stays the same.

Inventory is more than an address book. It is the boundary between intent and topology. The playbook can say hosts: web without knowing whether web means 10 static names today or 200 instances discovered from an API tomorrow.

The playbook: a contract, not a transcript

Here is the entire convergence playbook used for the capture.

Read the task literally: for every host in web, the rendered template must exist at that host’s nginx.conf path with mode 0644. There is no “copy this no matter what” and no unconditional reload. The template module owns the check-then-change behavior.

That distinction is where most Ansible explanations get mushy. YAML is not automatically declarative. A playbook full of shell tasks can throw away the same guarantees as the loop. The module is the contract: it knows how to inspect a resource, compare actual state with desired state, make the smallest necessary change, and return structured truth.

Under the hood, this run has a straightforward shape:

ansible-navigator hands the play to ansible-core, which resolves the web pattern against inventory and builds one host context per match.
The default linear strategy schedules the task across hosts, up to the configured fork count (five by default). It is parallel, but deliberately bounded.
The template action renders Jinja for each host on the control side, probes the destination, and compares content and file attributes. Matching state returns ok; drift takes the write path.
Each module invocation returns structured data. Ansible associates that result with the host that produced it, then builds the recap from those results.

That fourth step hides the best trick. Ansible doesn’t scrape a shell command’s text. It bundles the module’s code, the module_utils it depends on, and your task’s arguments into a single self-contained Python file (that’s AnsiballZ), copies that one file to a temp directory on the target, and asks the interpreter there to run it. The file prints JSON to stdout, and ansible-core reads that back as the host result below.

One thing that trips people up: you ran template, but the trace shows copy.py. The template module renders the Jinja on the control node, then hands the finished file to the copy module to place it. So copy is what you watch do the PUT and the interpreter call, at -vvv, for web-07.

This fixture uses a local connection, so PUT lands in another directory on the same laptop. Over SSH, the same payload is shipped to the host; the transport changes, not the payload-and-JSON contract. That is the practical meaning of agentless, why ansible_python_interpreter matters, and why Ansible can report changed, a checksum, a mode, and a destination instead of handing us text to scrape.

No central agent is watching the server. Ansible reconstructs the truth at run time, host by host, then goes away.

Show me the drift, but do not touch it

Convergence does not have to begin with a write. --check asks modules to predict what they would change; --diff asks file-aware modules to show the candidate patch. Here is the same drifted fleet before the live run:

Nine hosts return ok. web-07 returns changed and shows the exact worker_connections 256 to 1024 patch, but the handler is skipped and the file is not written. The SHA-256 before and after this capture was identical, and web-07 still contained 256.

Check mode is a prediction, so its honesty depends on each module’s support for it. The template module supports check and diff mode fully; an arbitrary shell command cannot make the same promise. That is another reason the module choice matters more than the YAML around it.

Convergence: 10 hosts in, one change out

Press Run Playbook. This is not generated theater: the runner replays parsed stdout from a real ansible-navigator run against the seeded fleet. Nine host results are ok. Only web-07 is changed. When its successful result lands, watch the diagram above heal from amber to green.

Ansible’s native recap is per host, not an invented fleet-total line. On the first run, web-07 ends at ok=2 changed=2; the other 9 hosts end at ok=1 changed=0. The capture-derived result strip makes the same outcome glanceable: 9 ok, 1 changed, 0 failed.

The Reload nginx handler fired on web-07 only. That is the safety the loop deleted made concrete: the loop would have reloaded all 10.

Now press Run Again.

The second capture is the small “oh” moment: 10 ok, zero changed. Ansible still connects, resolves variables, renders the candidate template, and checks each destination. It simply finds no work worth doing. Idempotency is not “the command probably won’t hurt twice.” It is a module proving that actual state already matches declared state.

The safety the loop deleted

The playbook and the shell loop can both reach all 10 hosts. That is the least interesting thing they share. Ansible keeps the evidence the loop discards:

Drift detection: each module compares before it mutates. web-07 is a fact, not a hunch.
Idempotency: a clean fleet stays untouched; repeated runs are verification, not roulette.
Per-host truth: ok, changed, failed, and unreachable remain attached to hostnames.
Bounded parallelism: forks make the run concurrent without launching 10 ungoverned SSH jobs.
An audit-shaped result: the task stream and recap say what ran and what changed. AAP can retain that event history; even stdout is already more useful than an anonymous loop.

Could we build all of that in Bash? Absolutely. Once we do, the clever one-liner has acquired a state engine, scheduler, result schema, and reporter. We have rebuilt the part of Ansible that was worth using.

That is how Ansible actually works: inventory turns machines into data, modules turn desired state into a comparison, strategies schedule that comparison across the fleet, and results preserve the truth per host. The YAML is just the meeting place.

Static today, moving target tomorrow

Static inventory is fine for 10 boxes we own. But what happens when the fleet changes between the moment we write hosts and the moment the playbook runs?

Post 2 follows that question into AAP, dynamic inventory, and an F5-backed fleet, where discovering the right targets becomes part of the automation, not a file we promise to remember to update.