Logging, Detection Evasion & Cloud Incident Response

After this chapter you will be able to predict what a cloud provider's audit pipeline will and won't record about an action, operate beneath that threshold, establish persistence that survives credential revocation — and then switch hats and design the telemetry that catches a cross-tenant attacker anyway.

The problem

Every major cloud provider hands its customers an audit log: a running record of who did what, when, and from where. AWS calls it CloudTrail; Azure splits it into the Activity Log and the Entra sign-in/audit logs; Google calls it Cloud Audit Logs. It is marketed as the single source of truth for an account — the thing your detection engineering, your compliance posture, and your post-incident investigation all stand on.

The problem is the gap between that promise and the wiring. The audit log is not a magical, all-seeing eye. It is a managed service like any other — code the provider runs, on the control plane, that is separate from the API it records. Between "an API handler processed a request" and "an event landed in your log" there is a multi-stage pipeline, and every stage can fail. Some actions are never recorded by design. Some are recorded only if the customer pays extra. Some are recorded against the wrong actor. And — the heart of this chapter — some are missing because the provider's own code simply does not emit the event it is documented to emit.

This is the home of the detection surface — the sixth lens from Chapter 1, the one that has surfaced in every chapter's Defender's mirror without ever getting a chapter of its own. It is also a callback to Chapter 2, where you used error-message differentials to enumerate AWS roles and users without leaving a trace in the victim's logs and were promised the general theory later. This is later: "logging-invisible enumeration," generalised into a discipline.

Why it matters / how it differs from a traditional pentest

In a traditional pentest, "the logs" belong to the target organisation. Detection evasion means disabling endpoint agents, clearing event logs, timing your actions for the quiet hours — you fight the customer's SOC, and the customer owns the gaps. Cloud detection evasion is a different game.

When a cloud audit log fails to record something, the gap usually sits on the provider's side of the shared-responsibility line. If AWS ships a service that emits no event, the customer cannot fix it — no policy, no configuration setting, nothing. That makes a logging gap a provider vulnerability with an all-customer blast radius: every tenant of that service is blind to it simultaneously, and stays blind until the provider patches. For a CSP red-teamer, finding and reporting a logging gap is as valuable as finding a cross-tenant isolation break.

The second difference: there are only two planes to record. Cloud audit logs are overwhelmingly a control-plane record — API calls that mutate or read configuration. They are not an EDR; they do not see in-guest process execution on the data plane at all. A defender who forgets this watches the control plane and declares victory while malware runs untouched inside the VMs.

The methods at a glance

Detection evasion is the craft of exploiting any one stage of that six-stage pipeline. The corpus cases featured in this chapter cluster into four techniques, plus the persistence discipline that lets an attacker stay once detection fails. Each gets a breakdown section below.

Picture these as three concentric rings of visibility: the inner ring (management events) is lit by default, the middle ring (data events) is dark unless the customer pays to light it, and the outer ring stays dark no matter what the customer does. An attacker's job is to push every action outward; a defender's job is to know which ring each action lands in and to drag as many actions inward as the budget allows. The three providers draw the rings slightly differently — worth committing to memory before the breakdowns.

Note Azure's split brain: an attacker working the identity side shows up in the Entra logs but not the Activity Log.^[12] GCP's Admin Activity stream is always on and free; its Data Access stream is off by default, the exact analogue of AWS's data-event tier.^[13]

Technique 1 · Invisible enumeration

What it is. Reconnaissance that produces zero events in the victim's audit log. The attacker learns account structure, identities, and roles — and the defender's pipeline records nothing, success or failure.

How it works. CloudTrail has a fixed coverage map: it supports a defined list of AWS services and ignores the rest. Calling a service outside that map emits no event at all. But the API still responds — and an access-denied response is a free oracle, because it often returns the caller's identity or distinguishes "exists" from "does not exist". The attacker treats the error message as a boolean and scripts against it, entirely outside the victim's Stage 1.

Case #134 was recon against your own account's identity. The companion case turns the same idea outward — against someone else's account, leaving their pipeline completely empty.

// role exists, caller not trusted:
"User: arn:... is not authorized to perform: sts:AssumeRole
 on resource: arn:aws:iam::TARGET:role/SecretRole"

// role does not exist:
"Not authorized to perform sts:AssumeRole"

The family extends. Case #135 applies the verbose-error trick to users, confirming IAM usernames at scale from an account ID and a wordlist with no victim CloudTrail.^[3]#135 Then AWS patched, normalising the STS error to a flat Access denied. Rhino routed around it: case #136 found a new oracle in trust-policy principal validation — saving a trust policy that names a principal ARN forces AWS to resolve it on the backend, and an invalid principal throws Invalid principal in policy.^[4]#136 The same pattern recurs in undocumented internal APIs: case #080's Console-only AWSIdentityManagementAdminService exposed batch IAM methods that emitted no CloudTrail events, until AWS made it emit the matching iam events.^[5]#080

Technique 2 · Invisible change

What it is. Everything in Technique 1 was read-only — the attacker learned things invisibly but changed nothing. Invisible change is the step up: altering the environment with no event generated, a true post-exploitation evasion rather than recon.

How it works. Cloud providers run non-production copies of their services — beta and gamma endpoints — for internal testing. These endpoints often accept ordinary customer credentials and execute real actions, but they sit outside the production logging route (Stage 3 of the pipeline). The hard part is finding them. The AWS Console leaks them: its Content-Security-Policy header lists every endpoint the page may talk to, including the non-production ones.

This is not a one-off. Case #152 mapped AWS's IAM-enabled non-production endpoints as an entire attack-surface class, disclosing two more CloudTrail bypasses (ce:GetCostAndUsage and route53resolver:ListFirewallConfigs) plus event-source obfuscation — one action firing multiple, misleading events.^[7]#152 The non-production endpoint is a recurring seam, and the Console header is, reliably, where you find it.

Technique 3 · Under-attribution

What it is. A softer evasion than invisibility: the action is logged, but the log names the wrong actor. The event exists, the alert never fires, because nobody is looking for "the AWS service did it."

How it works. When an action is proxied through a managed service, the audit log often attributes it to the service principal, not to the identity that triggered it. The canonical case is the AppSync confused-deputy attack from Chapter 9 (#074), where a cross-account role assumption appears as appsync.amazonaws.com and blends into ordinary service noise. The cleaner teaching example sits in this chapter's own corpus.

Technique 4 · Provider logging failures

What it is. Not an unsupported service and not a non-production endpoint — a production, fully supported service that fails to emit events it is documented to emit. Stage 2 of the pipeline, broken inside the provider's own code.

How it works. "Documented as logged" is a claim, and claims can be wrong. A service may simply not emit an event that the provider's documentation lists as audited. When the missing event is an administrative action that itself disables logging, the gap and the off-switch compose into a complete audit blackout.

Technique 5 · Persistence — operating after you are caught

What it is. Persistence is the attacker's answer to being detected anyway. It is an anti-incident-response discipline: pre-positioned access designed to survive the obvious remediation — revoking the credential the responder found.

How it works. Persistence has been used in this book without being named — Golden SAML (Chapter 3) survives a password change, backdoor roles survive key rotation. The discipline has five classes, and a serious operator layers at least two, because an IR team that finds and kills one will assume the job is done.

Three more cases round out the taxonomy. Case #174 documents Entra ID restricted-management Administrative Units: an attacker account placed inside a restricted AU with no AU-scoped admin becomes un-disable-able through any tenant admin's normal flow — a "by design" abuse, the cleanest anti-containment example in the corpus.^[14]#174 Case #173 abuses Azure Policy: a Resource Policy Contributor edits a shared policy definition, and append/modify effects can disable diagnostic logging or inject backdoor VM extensions while needing no service principal and producing no logs.^[15]#173 Case #158 is the textbook trust-class case — a role SupportAWS whose trust policy named an external attacker-owned AWS account, surviving revocation of every credential in the victim account.^[16]#158

Defender's mirror — detection from the provider's vantage

This is the chapter's second half, not its coda. Everything above told you how to evade the pipeline. Now you build the pipeline that catches the evader anyway. The reader is a web-security engineer, so the detection-engineering vocabulary gets a callout before the techniques.

Even under evasion, threat actors have a fingerprint

The defender's strongest answer to Techniques 1–5 is case #102. Unit 42 took two real adversary groups — Muddled Libra (financially motivated, overlapping with Scattered Spider) and Silk Typhoon (nation-state) — and mapped each group's cloud MITRE ATT&CK techniques to the alert rules they trigger. Across 22 industries (June 2024–June 2025), Muddled Libra's 11 techniques mapped to roughly 70 unique alert rules; Silk Typhoon's 12 to about 50 — and the two sets shared only three rules.^[18]#102 The distribution of alert types in a victim environment fingerprints which group is operating, enough to attribute an intrusion from telemetry alone. The teaching point is the chapter's defensive thesis: you cannot evade every logged action — some always land in Ring 1, and the pattern of even partial telemetry is itself a signal. MITRE ATT&CK's Cloud matrices give every technique a stable ID, which is what makes this cross-group comparison possible.^[17]

Pick the right telemetry source

Case #104 is the Azure-side example. AzureHound — the Go-based Azure component of the BloodHound suite — enumerates Entra ID and Azure through the Microsoft Graph and Azure REST (ARM) APIs. Both are externally reachable, so AzureHound does not need to run inside the victim; it has been misused by Peach Sandstorm, Void Blizzard, and the ransomware operator Storm-0501. Unit 42's key observation: the activity is loud in the Entra sign-in and audit logs — a burst of Graph plus ARM read calls from one service principal — even though it is silent in the resource-level Activity Log.^[19]#104 A defender who watches only the Activity Log misses AzureHound entirely; detection engineering starts with choosing the telemetry source that carries the signal.

How you actually hunt this

Case #165 is Datadog's hypothesis-driven threat-hunting methodology, converting the abstractions above into concrete signals. Backdoor IAM user names from real intrusions — sesadministrator, ampIify-dev (a capital-I homoglyph), ses_legion — are signature IoCs, but the durable detections are behavioural: a high volume of AccessDenied from one identity, and IAM-user creation originating from an EC2 instance (a workload role should almost never call CreateUser).^[21]#165 Case #159 shows the pivot in action: SNS GetSMSAttributes enumeration in CloudTrail led Datadog to an attacker IP, then to an entire SMS-phishing campaign.^[22]#159

Cloud Incident Response from the provider's chair

When a provider runs incident response — not a tenant — they hold a cross-tenant view of the control plane, host-agent telemetry, and the internal-API logs. But they inherit the tenant's blind spots too: if a service emitted no event, the provider's SOC is just as blind. The IR loop — detect → scope → contain → evict → recover → learn — has two cloud-specific wrinkles. First, eviction must explicitly kill persistence: revoking the leaked key from #149 does nothing about the API-Gateway-fronted Lambda that re-mints users. Second, "scope" and "recover" are corrupted when the attacker tampered with the very logs you would use — which is why telemetry-class persistence does not just hide the next action, it blinds the entire post-incident investigation.

Attacker's checklist

Defender's mirror

Chapter 13 re-analyses ChaosDB, Azurescape, and SynLapse as complete chains, and asks you to attach a detection analysis to every step: what did the provider's pipeline see, what did it miss, what telemetry would have caught it. Carry Figure 12.1 forward — it is now a chaining-analysis tool. For every link in every chain, answer the sixth lens.