Detection Strategy for Encrypted Channel via Symmetric Cryptography across OS Platforms

ID: DET0143
Domains: Enterprise
Analytics: AN0400, AN0401, AN0402, AN0403, AN0404
Version: 1.0
Created: 21 October 2025
Last Modified: 21 October 2025

Analytics

AN0400

Processes that typically do not perform cryptographic operations loading symmetric encryption libraries (e.g., bcryptprimitives.dll, aes.dll), then initiating outbound connections with high-entropy payloads. Defender correlates process creation, DLL load, and anomalous encrypted traffic patterns.

Log Sources
Data Component Name Channel
Module Load (DC0016) WinEventLog:Sysmon EventCode=7
Network Connection Creation (DC0082) WinEventLog:Sysmon EventCode=3
Mutable Elements
Field Description
AllowedCryptoProcesses Processes normally expected to use symmetric crypto (e.g., disk encryption, secure messaging).
EntropyThreshold Minimum payload entropy score for flagging unusual encrypted sessions.
TimeWindow Correlation window between module load and encrypted connection creation.

AN0401

Unexpected processes (e.g., bash, python, custom binaries) dynamically loading libcrypto or performing AES/RC4 encryption operations, then initiating outbound sessions with abnormal byte entropy or asymmetric traffic patterns.

Log Sources
Data Component Name Channel
Process Creation (DC0032) auditd:SYSCALL execve or socket/connect system calls from processes using crypto libraries
Application Log Content (DC0038) linux:syslog System daemons initiating encrypted sessions with unexpected destinations
Module Load (DC0016) linux:osquery Process linked with libcrypto.so making external connections
Mutable Elements
Field Description
TrustedCryptoLibs Baseline expected crypto libraries to suppress false positives.
TrafficAsymmetryRatio Ratio of sent/received bytes indicating possible hidden C2.

AN0402

Launchd jobs or user processes invoking symmetric crypto APIs from the Security framework and generating outbound connections carrying randomized payloads inconsistent with normal TLS patterns.

Log Sources
Data Component Name Channel
Process Creation (DC0032) macos:unifiedlog Process using AES/RC4 routines unexpectedly
Network Traffic Content (DC0085) macos:unifiedlog Encrypted connection with anomalous payload entropy
Mutable Elements
Field Description
DoHResolvers Legitimate DNS-over-HTTPS endpoints to avoid FP.
PayloadEntropyThreshold Define entropy level at which traffic should be flagged.

AN0403

ESXi daemons (hostd, vpxa) unexpectedly using symmetric encryption routines for external connections. Defender identifies logs of service traffic with encrypted payloads inconsistent with VMware management baselines.

Log Sources
Data Component Name Channel
Application Log Content (DC0038) esxi:vpxd Symmetric crypto routines triggered for external session
Network Traffic Content (DC0085) esxcli:network Socket sessions with randomized payloads inconsistent with TLS
Mutable Elements
Field Description
AllowedMgmtHosts Baseline list of approved vCenter and update endpoints.

AN0404

Flows showing encrypted payloads with high entropy not matching TLS handshake patterns, particularly when occurring on non-standard ports. Defender observes NetFlow/IPFIX byte distribution anomalies or IDS/IPS detecting symmetric encryption patterns without associated key exchange.

Log Sources
Data Component Name Channel
Network Traffic Flow (DC0078) NSM:Flow Flow records with entropy signatures resembling symmetric encryption
Network Traffic Content (DC0085) NSM:Connections Symmetric encryption detected without TLS handshake sequence
Mutable Elements
Field Description
PortProfiles Baseline expected encryption by port/protocol.
TrafficVolumeThreshold Volume thresholds for distinguishing benign VPN traffic from hidden C2.