Detecting Protocol or Service Impersonation via Anomalous TLS, HTTP Header, and Port Mismatch Correlation

ID: DET0470
Domains: Enterprise
Analytics: AN1294, AN1295, AN1296, AN1297
Version: 1.0
Created: 21 October 2025
Last Modified: 21 October 2025

Analytics

AN1294

Untrusted processes creating outbound TLS/HTTPS connections with malformed certificates or header fields, often mismatched with target service behavior. Detects protocol impersonation attempts via traffic metadata analysis and host process lineage.

Log Sources
Data Component Name Channel
Network Connection Creation (DC0082) WinEventLog:Sysmon EventCode=3
Network Traffic Content (DC0085) NSM:Flow SSL/TLS Handshake Analysis
Mutable Elements
Field Description
IssuerOrgFilter Organizations in certificate issuer fields to allowlist or monitor.
UserContext Restrict detection to non-system users or external-facing applications.
HeaderSignatureMatch Specific HTTP header anomalies or patterns (e.g., missing User-Agent).

AN1295

Detection of binaries spawning encrypted sessions using OpenSSL or curl to external services with mismatched ports/protocols. Identifies behavior where internal services simulate trusted cloud service traffic patterns.

Log Sources
Data Component Name Channel
Process Creation (DC0032) auditd:SYSCALL execve
Network Traffic Content (DC0085) NSM:Flow Network Capture TLS/HTTP
Mutable Elements
Field Description
ProtocolMatchConfidence Threshold for header-field mismatch against expected service behavior.
TimeWindow Correlation window between process spawn and encrypted session.

AN1296

Unsigned or suspicious applications initiating network traffic claiming to be browser, mail, or cloud clients. Detects impersonation via TLS fingerprint and User-Agent string deviation.

Log Sources
Data Component Name Channel
Network Connection Creation (DC0082) macos:unifiedlog Outbound Traffic
Process Metadata (DC0034) macos:osquery Process Execution + Hash
Network Traffic Content (DC0085) NSM:Content HTTP Header Metadata
Mutable Elements
Field Description
ParentProcessFilter Limit detections to children of suspicious binaries.
HeaderAnomalyScore Threshold for deviation from expected headers (User-Agent, Host).

AN1297

ESXi hosts initiating connections from non-standard daemons mimicking HTTP/HTTPS or SNMP traffic, but with irregular payload formats or expired/unsigned TLS certificates.

Log Sources
Data Component Name Channel
Network Connection Creation (DC0082) esxi:hostd Service-Based Network Connection
Network Traffic Content (DC0085) NSM:Content TLS Fingerprint and Certificate Analysis
Mutable Elements
Field Description
TLSFingerprintMatch Allows matching against known-good or known-bad JA3/JA3S hashes.
AllowedServicePorts Tune for expected network ports per ESXi role.