A smaller scan, a shorter list
Suppose a monthly scan has been running against four subnets. To shorten the run, someone trims the target range to three. The following month, every finding on the fourth subnet stops appearing in the results. Under a pipeline that treats non-appearance as remediation, those findings are marked resolved overnight, the open count drops, and the trend line looks like progress. Nothing on the fourth subnet changed. When the scope is widened again, the same issues return as new findings, with new first-seen dates and no history.
The example is hypothetical, but the mechanism is general. A vulnerability scanner does not report what is true about an estate. It reports what it observed, from where it stood, with the access it had, at the moment it ran. A finding is the issue being managed. An observation is one piece of evidence, from one scan, that the issue was present at that time. A missing observation is not an observation of absence.
Two kinds of “gone”
Two situations produce the same symptom, a finding no longer reported, and mean opposite things.
The first is remediation. An assessment ran against the component with checks capable of detecting the issue, with sufficient access, completed successfully, and did not detect it. A patch landed, or a service was removed.
The second is non-observation. The component was not assessed in a way that could have detected the issue: the host was powered off, a firewall rule changed, the asset fell out of the target list, credentials failed so local checks never ran, or the scan engine itself was unreachable. The raw output contains no line saying “resolved,” only the lack of a line, and what that means depends on what the scan covered.
NIST SP 800-115, the 2008 technical guide to security testing, makes the point about access and vantage point: a scanner holding host credentials can examine a system from the inside, while one without them sees only what the network exposes and must contend with perimeter devices that block traffic. The guidance is old and its characterizations of scanner accuracy are not current measurements, but the general limitation has not changed: a scan whose credentials silently failed, or whose path was blocked, has a narrower field of view than the run before it.
Why automated closure on non-appearance misleads
The same effect appears when the service account used for authenticated scanning expires: local checks stop, operating system and package findings vanish, and metrics improve until the password is rotated.
A subtler variant is the first run of a new scan job. Its silence about an asset it has never assessed supports nothing. Letting a first run resolve findings reported by a different job is the same error with a different cause.
Closure must be tied to coverage
Automated closure needs evidence that the affected component was reassessed using checks capable of detecting the issue, with sufficient access and successful execution. A different assessment can provide that evidence if its coverage is comparable and its basis is recorded.
“Reassessed” has an operational meaning: the component responded and was examined, not merely listed as a target. A host in scope but reported unreachable was not assessed. A host assessed without credentials was assessed for network-visible checks and not for local ones. If the tooling cannot distinguish these cases, non-appearance cannot safely trigger automated resolution.
Resolution should also carry its evidence. A resolved finding should record which assessment supported the resolution, when it ran, and with what access. Auditors and asset owners ask “what did we resolve last month, and how do we know?”, and “it stopped showing up” is not an answer.
Consider three possible explanations for a finding disappearing from a report. In the first case, the authenticated scan logged in, executed the relevant checks, and did not detect the issue: the finding is resolved, with that run recorded as the supporting assessment. In the second case, the assessment shows authentication failed on that host, so local checks never executed: the finding stays open, with the failed authentication recorded. In the third case, the host was decommissioned and removed from inventory: the finding is retired administratively, with the decommissioning record attached. The absence from the report looks identical in all three cases; only the recorded evidence tells them apart, and only the first is verified remediation.
First seen, last seen, and scope
Three fields do most of the work, and all three degrade under naive pipelines.
First seen is the earliest moment a finding was observed on an asset, and it anchors age, SLA clocks, and exposure windows. First seen records the earliest available evidence, not necessarily the start of exposure; the issue may predate the first scan able to detect it. If findings are deleted when unreported and recreated when they reappear, first seen resets and every age metric becomes optimistic.
Last seen is the most recent moment any assessment observed the finding. An old observation should trigger reassessment or a documented administrative disposition; age alone does not establish remediation. Stale findings remain unresolved and are reported separately by evidence freshness.
Scan scope is the set of assets a job is responsible for and, per run, the subset it actually reached and with what access. Comparing the two turns “not reported” into either “resolved” or “not observed.”
Showing the unknown instead of zero
Dashboards fail quietly when they conflate “no findings” with “no data.” Zero open criticals on a subnet not successfully scanned in six weeks is a blind spot with a green badge. Coverage belongs next to the findings count: assets in scope, assets assessed, assets assessed with credentials, and assets not observed within the expected interval. CISA’s Binding Operational Directive 23-01, which addresses federal civilian executive branch agencies rather than all organizations, points the same way: enumerate vulnerabilities across all discovered assets on a fixed cadence, with privileged credentials to the maximum extent possible. The principle transfers: coverage is measured, and a gap is reported as a gap. The companion article on not fabricating data covers how to present those gaps without filling them in.
In practice
Checks to run against current tooling this week:
- Take a finding resolved automatically last quarter and confirm which assessment supported the resolution, and whether it reached the component with sufficient access. If the tool cannot answer, automated closure is unverified.
- Compare the target scope of the most recent scan against the previous one. Assets that dropped out of scope should still carry open findings, not resolved ones.
- Review per-host authentication status in the last authenticated scan. Hosts where credentials failed should not have had local findings resolved.
- Look for first-seen dates that reset. A finding open for months that suddenly shows a recent first-seen date means the pipeline is deleting and recreating rather than tracking.
- Check whether the dashboard distinguishes “zero findings” from “not assessed recently.” If not, put a coverage count beside the findings count before trusting either.
This is how Sectie’s Exposure & Vulnerability Management product treats closure: a finding is resolved only when an assessment that covered the component no longer reports it, and the assessment that supported the resolution is recorded.
