← All posts

October 29, 2026

The Outage That Happened Every Day at 13:11

For a few weeks, our staging environment kept a schedule. Every day at around 13:11 Brasília time, all four services on our staging cluster went down: the exam platform, a legacy course platform, the LMS backend and our SSO. Somewhere between 15:15 and 16:00 they came back on their own. On October 9 the outage lasted about 24 hours. Nobody deployed anything at 13:11, and nothing in the services' own logs explained it.

I work at a Brazilian financial-education company. This is how I found the cause and fixed it under three constraints: nothing could affect production, the instances could not get bigger, and the job causing the problem was not mine to change.

A Clock, Not a Bug

An outage that starts at the same minute every day is a scheduled job until proven otherwise. The one I found was an AWS Systems Manager Quick Setup association that scans every instance in the account for missing patches. It is account-wide, it runs once a day (anchored at 13:10 our time), it also runs on every new instance at boot, and it has a two-hour timeout.

The scan runs dnf, and dnf needs around 500 MB to do its work. Our staging hosts are t3.micro instances: 1 GiB of memory, no swap, and already carrying the containers they exist to run. There is no room for another 500 MB, so the host doesn't crash cleanly. It thrashes, and from there the chain is mechanical:

  1. Everything on the host slows to a crawl, including the ECS agent, the SSM agent and the containers that answer health checks.
  2. Health checks fail, so ECS marks the tasks unhealthy and replaces them.
  3. Replacement capacity arrives as a fresh instance, and a fresh instance runs the patch scan at boot. It hangs the same way.
  4. The episode only ends when the association hits its two-hour timeout.

That explained the shape of the outage: it started at 13:11 and ended roughly two hours later, plus however long the replacements took to settle. The comparison that confirmed it was already running in the same account. Production uses t3.medium instances, gets the same association on the same schedule, and finishes the same scan in about a minute and a half. Same job, same clock. The only difference was headroom.

Three Constraints

The obvious fixes were all off the table:

So the question got narrower: can a staging host survive dnf asking for 500 MB it doesn't have, without changing what runs, when it runs, or on what hardware?

Following dnf Into Its cgroup

The answer came from the kernel. An OOM dump on the EC2 serial console named the control group dnf was running in: task_memcg=/system.slice/amazon-ssm-agent.service. The scan doesn't get a cgroup of its own. It runs as a child of the SSM agent, inside the agent's systemd service.

That turned a capacity problem into a blast-radius problem. If the agent's cgroup has a memory ceiling, then when dnf grows past it, the kernel kills dnf inside that cgroup within seconds and the rest of the machine never feels it. The containers, the ECS agent and the health checks keep their memory. The whole fix is a systemd drop-in:

# /etc/systemd/system/amazon-ssm-agent.service.d/90-memory-limit.conf
[Service]
MemoryMax=128M
OOMPolicy=continue

MemoryMax=128M is the ceiling. OOMPolicy=continue is the line I would have missed without the serial console. By default, when the kernel OOM-kills a process inside a service, systemd stops the whole unit. A few days earlier I had watched exactly that on the console: the agent stopped, and systemd only brought it back 90 seconds later. With OOMPolicy=continue, the kernel kills dnf and the agent keeps running, so the host stays reachable and reporting.

Timing worked in my favor. The instance's user data runs about 12 seconds into boot, and the first association reaches the agent at around 20 seconds, so writing the drop-in from user data and running systemctl daemon-reload puts the limit in place before the boot-time scan starts.

Off by Default, Proven Byte for Byte

The drop-in lives in our ECS infrastructure stack, a Pulumi program in TypeScript that renders the launch template's user data for both staging and production. The change had to exist in the code without existing in production. So it is switched on by a new, optional config key, and only the staging stack's config file sets it, with one extra line: ssmAgentMemoryMax: 128M. Simplified, the gating looks like this:

const memoryMax = config.get("ssmAgentMemoryMax");

function ssmAgentMemoryLimit(memoryMax?: string): string {
  if (!memoryMax) {
    return "";
  }

  const dir = "/etc/systemd/system/amazon-ssm-agent.service.d";
  const file = `${dir}/90-memory-limit.conf`;

  return `\nmkdir -p ${dir}` +
    `\necho '[Service]' > ${file}` +
    `\necho 'MemoryMax=${memoryMax}' >> ${file}` +
    `\necho 'OOMPolicy=continue' >> ${file}` +
    `\nsystemctl daemon-reload`;
}

const userData = pulumi.interpolate`${baseUserData}${ssmAgentMemoryLimit(memoryMax)}`;

Without the key, the function returns an empty string and the user data is what it was before. But "is what it was before" is a claim, and production was the one place where I wasn't willing to rely on a claim. So I rendered the user data with Pulumi's mocks and compared it with the real launch templates. Without the key, the output was byte-identical to both the current staging and production templates. With it, the only difference was the five appended lines.

Shipping It Without Trusting Myself

Staging deploys from its own branch; production deploys from main. Nothing went to main, and the rollout was built so each step had to prove itself before the next one started:

  1. A preview-only push. One commit carried the fix. A second switched the staging deploy workflow from up to preview with a diff. Pushing both ran the pipeline without applying anything. The gate was exactly one update, the launch template, and zero creates, replaces or deletes. It showed 1 to update and 28 unchanged. That single update carried two fields, the user data and the image ID, because the stack's AMI lookup resolves to the newest ECS-optimized image.
  2. Apply by revert. A third commit reverted the preview switch, and that push ran the real update: 1 updated, 28 unchanged. The branch's final diff against main is the fix and nothing else.
  3. Swap instances in a fixed order. A launch template only affects new instances, so both hosts had to be replaced. ECS's binpack placement would happily put our SSO on one t3.micro with two other services, roughly 490 to 560 MiB of containers, which doesn't fit even without a scan. So: scale the staging SSO to zero and terminate its emptied host; drain the other host, which brings up a canary on the new template; once the other three services were placed on the canary, its memory fully reserved, scale the SSO back up so it had to land on a fresh instance of its own. The three services moved without going down. The SSO was out for about 17 minutes, against the 15 I had planned.
  4. Gate on the canary, then on the SSO's host. The kernel's OOM report had to show the 131072 kB limit, which is 128 MiB. Kills had to be only CONSTRAINT_MEMCG, and only of dnf, yum and python3. Zero global OOM events, SSM online, the ECS agent connected and all four services healthy. Only then did the old host get terminated.

My access profile doesn't include ssm:SendCommand, so I couldn't run a command on the hosts to read memory.max directly. The serial console turned out to be enough. The kernel writes everything I needed into its OOM report, and reading the console output doesn't touch the instance:

aws ec2 get-console-output --instance-id "$INSTANCE_ID" --latest --output text \
  | grep -E 'limit 131072kB|CONSTRAINT_MEMCG|global_oom'

Then production, verified rather than assumed: its launch template still on the same version, the same production instances running, no production deploy run, CloudTrail showing new launch template versions only on the staging template, and main unchanged.

What It Looks Like Now

I expected the boot scan to fail within a few minutes. It doesn't, quite. The scan retries dnf several times, and at first the kernel kills it every one to two minutes. Those attempts cost 7 to 10% CPU and no extra network traffic, and the association stays InProgress until its two-hour timeout, just as before. The difference is that nothing else on the host notices. Patch compliance in staging still reports failed, as it already did. It just no longer takes the environment down with it.

There is one trade-off: Session Manager and Run Command sessions on staging hosts also live inside the agent's cgroup, so they get the same 128 MiB. For anything heavy, sudo systemd-run --scope -p MemoryMax=infinity <cmd> runs the command in a scope of its own. Rollback is a revert on the staging branch plus the same swap sequence.

One check is still open, and it's the one that matters most: the next day's 13:00 to 16:00 window, with the 13:11 scan contained and the exam platform's healthy host count never reaching zero. The new hosts' boot scans already behaved the way the fix intends, but as I write this, that box in my notes is still unticked.

A Scar: The Branch That Wasn't There

Early in planning I concluded that the staging branch didn't exist. I had looked at git branch -a in a clone that was shallow and single-branch, with a fetch refspec that only covered main, so of course it showed nothing else. The branch was on GitHub, and its existence changed the deploy plan. Now, before I say a remote branch exists or doesn't, I ask the remote:

git ls-remote --heads origin
git rev-parse --is-shallow-repository

A local clone is a cache of the remote, not a statement about it.

What I'm Keeping