Reducing False Positives in Automated Web Vulnerability Scanning

Undergraduate thesis, COMSATS University Islamabad — team lead, three-person team (2025).

Motivation

Automated web vulnerability scanners are limited less by what they detect than by what they report. A rule-based scanner flags every anomaly matching a signature, so the analyst inherits a long queue of findings whose correctness must be established by hand. The cost of assessment is not removed by automation; it is displaced from detection to triage.

This work asked whether that triage cost can be reduced without discarding genuine findings, and without the labeled ground truth that supervised approaches assume.

Approach

The central difficulty is that no ground truth exists: labeling a scanner finding as a true or false positive requires manual security expertise and does not scale to a corpus. Rather than force the problem into supervised classification, I framed it as confidence estimation — predicting the likelihood that a given finding is spurious, and treating the decision to suppress it as a tunable threshold rather than a verdict.

• Weak supervision. Deterministic domain heuristics (e.g. header-class findings with no supporting response evidence) generated pseudo-labels to direct learning, in place of expert annotation.
• Behavioral features. The model observes scanner behavior and finding context — evidence availability, vulnerability class, deployment environment — rather than raw vulnerability content, so it generalizes across applications instead of memorizing one.
• Random Forest was selected for tolerance of noisy, weakly labeled data and for probabilistic outputs suitable for threshold-based filtering.
• Threshold-based suppression. Findings are ranked by false-positive confidence and filtered against a configurable cutoff; nothing is permanently deleted from the scan record, so the operator sets the noise/recall trade-off.

Evaluation

Standard supervised metrics are inappropriate here — with no true labels, accuracy measures agreement with the heuristics rather than correctness. The evaluation therefore reports operational effects: alert reduction rate, preservation of high-severity findings, and stability of behavior across confidence thresholds.

The corpus comprises over 100,000 vulnerability records collected from deliberately vulnerable applications and from production scans, unified across scans, deduplicated, and normalized.

Limitations

Because the pseudo-labels are generated by deterministic rules, high internal precision indicates that the model has learned those rules rather than that it has learned correctness — a distinction the results are careful not to overstate. The confidence distribution is strongly polarized, which suggests the learned decision boundary closely tracks the heuristics. Establishing genuine predictive value requires a manually validated subset as ground truth, and a threshold sensitivity sweep reported in full; both are the natural next steps for this work.

System

The evaluation runs on a modular Django scanning pipeline integrating OWASP ZAP and Nikto, with passive crawling, PostgreSQL/SQLite result storage, and the filter implemented as a post-processing stage — the classifier is decoupled from the scanners, so the approach transfers to any tool that emits structured findings.

Stack: Python · Django · scikit-learn (Random Forest) · OWASP ZAP · Nikto · PostgreSQL / SQLite · REST APIs

Resources

• GitHub Repository: github.com/hannan112/Web_App_Scanner
• Thesis Document: Download Thesis PDF (Hannan Ali, 2026)