Detection Strategy for Encrypted Channel via Asymmetric Cryptography across OS Platforms

ID: DET0543
Domains: Enterprise
Analytics: AN1496, AN1497, AN1498, AN1499, AN1500
Version: 1.0
Created: 21 October 2025
Last Modified: 21 October 2025

Analytics

AN1496

Processes not typically associated with encryption loading asymmetric crypto libraries (e.g., rsaenh.dll, crypt32.dll) and subsequently initiating outbound TLS/SSL connections with abnormal certificate chains or handshakes. Defender correlates process creation, module load, and unusual encrypted sessions.

Log Sources
Data Component Name Channel
Module Load (DC0016) WinEventLog:Sysmon EventCode=7
Network Connection Creation (DC0082) WinEventLog:Sysmon EventCode=3
Mutable Elements
Field Description
AllowedCryptoProcesses Whitelist browsers, mail clients, or apps expected to use asymmetric crypto.
CertificateAuthorityList Baseline CA list for validating abnormal certs.
HandshakeTimeout Detection of incomplete or malformed handshakes.

AN1497

Processes (e.g., bash, python, custom binaries) dynamically linking libcrypto/libssl for RSA key exchange, then creating external connections with abnormal certificate validation or handshake anomalies. Defender observes syscall traces and outbound asymmetric key exchanges from non-SSL-native processes.

Log Sources
Data Component Name Channel
Process Creation (DC0032) auditd:SYSCALL execve or socket/connect system calls for processes using RSA handshake
Application Log Content (DC0038) linux:syslog Non-standard processes negotiating SSL/TLS key exchanges
Module Load (DC0016) linux:osquery Processes linked with libssl/libcrypto performing network activity
Mutable Elements
Field Description
ExpectedCryptoLibs Baseline libraries that normally handle asymmetric crypto.
TrafficAsymmetryRatio Threshold for client-heavy data sending vs server.

AN1498

Applications or launchd services invoking RSA or public-key routines from the Security framework, followed by outbound SSL/TLS sessions with unrecognized certs or anomalous handshakes. Defender observes unified logs of API calls and suspicious network entropy.

Log Sources
Data Component Name Channel
Process Creation (DC0032) macos:unifiedlog Process invoking SecKeyCreateRandomKey or asymmetric crypto APIs
Network Traffic Content (DC0085) macos:unifiedlog TLS connections with abnormal handshake sequence or self-signed cert
Mutable Elements
Field Description
TrustedDoHEndpoints Known legitimate DoH/SSL endpoints.
PayloadEntropyThreshold Entropy scoring for outbound payloads.

AN1499

VMware services (hostd, vpxa) unexpectedly negotiating asymmetric crypto sessions to external endpoints outside vCenter or update servers. Defender sees encrypted handshakes in logs inconsistent with baseline ESXi communication patterns.

Log Sources
Data Component Name Channel
Application Log Content (DC0038) esxi:vpxd ESXi process initiating asymmetric handshake with external host
Network Traffic Content (DC0085) esxcli:network Socket inspection showing RSA key exchange outside baseline endpoints
Mutable Elements
Field Description
BaselineMgmtHosts Expected external endpoints (vCenter, update repos).

AN1500

Encrypted sessions detected with asymmetric key exchange anomalies on non-standard ports or with invalid/malformed certs. Defender correlates NetFlow/IPFIX with IDS/IPS detecting RSA exchanges outside expected TLS flows.

Log Sources
Data Component Name Channel
Network Traffic Flow (DC0078) NSM:Flow Flow records with RSA key exchange on unexpected port
Network Traffic Content (DC0085) IDS:TLSInspection Malformed certs, incomplete asymmetric handshakes, or invalid CAs
Mutable Elements
Field Description
PortProfiles Define expected ports for asymmetric cryptography (e.g., 443, 993).
CertValidationPolicy Thresholds for rejecting untrusted/self-signed certs.