GAGodwin Etim AkpanGeospatial & Data Engineer← All projects

Data engineering / Security analytics

Distributed Security Analytics
From traffic volume to defensible detection.

A security analytics workflow connecting Hadoop and Hive exploration with reproducible PySpark classification across 2.54 million network flows. Explicit label validation, duplicate-aware partitioning, and later-capture testing turn model scores into an inspectable assessment of detection quality.

Security analytics research · 2.54M flows · Models rerun: October 2026
2.54MFlow records validated
10 classesNamed and reconciled
4 featuresBytes and TTL baselines
Jan → FebLater-capture evaluation
Explore the engineering ↓View GitHub project ↗
Project in focus

Two evaluation designs, a fuller performance picture

Paired model metrics comparing grouped random and later-capture testing for Logistic Regression and Random Forest.
The same fixed models are evaluated on separate test cohorts. Blue circles show the grouped random split; gold diamonds show January-to-February evaluation. Differences combine changes in training data, capture period, and class proportions.View full-size figure ↗

01 / The question

What does a strong detection score actually tell us?

A large traffic table needs more than an accurate classifier. The workflow begins by validating label semantics and feature alignment, then asks how a four-feature baseline behaves under two evaluation designs. A grouped random split measures within-corpus performance; training on January captures and testing on February captures probes performance across collection periods.

My contributionBuilt the original HDFS/Hive exploration and PySpark baselines, then rebuilt the modelling workflow with validated source labels, fixed class names, controlled partitions, and per-class evaluation.

02 / System walkthrough

From input to evidence.

The main stages of the implemented analytical workflow.

  1. 01

    Validate the data contract

    Read the original 49-column flow schema, check ranges and timestamps, reconcile attack categories against the supplied binary label, and merge Backdoor aliases.

  2. 02

    Separate the evaluations

    Keep exact duplicate records together in a seeded random split. For the temporal test, train only on flows ending before February and evaluate later flows.

  3. 03

    Fit controlled baselines

    Use log-transformed source and destination bytes plus their TTL values. Fit binary Logistic Regression and a ten-class Random Forest; include a training-majority baseline.

  4. 04

    Inspect detection quality

    Export ROC-AUC, average precision, named confusion matrices, macro-F1, and per-class recall and support. Preserve full-precision results and code/input hashes.

03 / Engineering decisions

Why the implementation matters.

Keep the target authoritative

The supplied 0/1 label defines Normal versus Attack. Missing attack categories become Normal only when the binary label confirms it; disagreement stops the run.

Fit without future information

Feature scaling inside Logistic Regression uses training data. Dates, addresses, ports, and target fields are excluded from model inputs. Class indices follow a declared semantic order.

Challenge aggregate accuracy

Compare Random Forest with the majority baseline and inspect every class. Macro-F1 and recall make minority-class behaviour visible alongside overall accuracy.

04 / Results & interpretation

What the experiment showed.

Executed PySpark results. The main evaluation trains on January 2015 and tests on February 2015. A grouped 70/30 random split provides a separate within-corpus comparison; models use fixed hyperparameters.

Later capture / Logistic Regression

0.979 ROC-AUC

January-trained binary discrimination on 1,452,603 February flows. Average precision: 0.871; attack prevalence: 20.6%.

Later capture / Random Forest

0.364 macro-F1

Overall accuracy is 94.7%. The class-balanced view shows how uneven detection remains across the ten named categories.

Grouped random / Random Forest

97.5% accuracy

Macro-F1 is 0.387 on the same predictions. High aggregate accuracy alone does not establish reliable minority-attack detection.

Figure / 02

Detection quality, class by class

Random Forest recall for ten named traffic classes under both evaluation designs, with February support counts.
Recall measures the fraction of each actual class correctly identified. February support keeps rare-class results in perspective; all ten classes remain visible.View full-size figure ↗
Figure / 03

Inspect the binary threshold tradeoff

ROC and precision–recall curves with exact aggregate scores for both test cohorts.
ROC and precision–recall curves describe binary Logistic Regression. Average precision depends on attack prevalence, which differs between the two test cohorts.View full-size figure ↗

05 / Scope & limitations

Scope & limitations.

  • Four-feature baselineBytes and TTL offer a compact, interpretable starting point. They omit protocol, service, timing, and wider flow context that may help distinguish attack families.
  • Benchmark and capture scopeThis is a local extract of the laboratory-generated UNSW-NB15 corpus, using a project-specific temporal split. It is not the official benchmark partition or an external-network validation. Exact-record separation does not eliminate all correlated traffic.
  • Operational thresholds remain unvalidatedBinary decisions use a fixed 0.5 threshold. Alert cost, calibration, current network prevalence, and threshold selection require separate validation before operational use.
  • Execution scopeThe corrected models were executed in local PySpark on one machine. The original project used HDFS and Hive; this rerun does not establish multi-node scaling or production intrusion-detection performance.

06 / Next iteration

Where I would take it next.

The next technical priorities for this project.

  1. Expand the feature comparison

    Evaluate protocol, service, timing, and broader flow features using the same preserved test design.

  2. Optimise for the alerting decision

    Use training-only validation to compare class weighting, calibrate probabilities, and select thresholds against explicit false-alert and missed-detection costs.

  3. Test across environments

    Add independent traffic and repeated capture periods, then benchmark distributed execution on a declared worker topology.

07 / Code & reproducibility

Explore the implementation.

Explore the implementation, selected results, and technical documentation on GitHub.

  • Corrected PySpark source with fixed class names and strict label validation.
  • Named confusion matrices, per-class metrics, precision–recall and ROC data, and split supports.
  • Input/source hashes, pinned dependencies, and tests for input contracts, split boundaries, and metric interpretation.

Figures regenerated from aggregate model outputs. Full-precision metrics, named confusion matrices, split supports, and the run manifest are available on GitHub.

Download model metrics ↓Download per-class results ↓
Explore the GitHub project ↗Return to selected work →