Custom Cryptographic Protocol

From enterprise
Jump to: navigation, search
Custom Cryptographic Protocol
ID T1024
Tactic Command and Control
Platform Windows Server 2003, Windows Server 2008, Windows Server 2012, Windows XP, Windows 7, Windows 8, Windows Server 2003 R2, Windows Server 2008 R2, Windows Server 2012 R2, Windows Vista, Windows 8.1, Linux, Windows 10, MacOS, OS X
Data Sources Packet capture, Netflow/Enclave netflow, Process use of network, Malware reverse engineering, Process monitoring
Requires Network Yes

Adversaries may use a custom cryptographic protocol or algorithm to hide command and control traffic. A simple scheme, such as XOR-ing the plaintext with a fixed key, will produce a very weak ciphertext.

Custom encryption schemes may vary in sophistication. Analysis and reverse engineering of malware samples may be enough to discover the algorithm and encryption key used.

Some adversaries may also attempt to implement their own version of a well-known cryptographic algorithm instead of using a known implementation library, which may lead to unintentional errors.1


  • Several Lazarus Group malware families encrypt C2 traffic using custom code that uses XOR with an ADD operation and XOR with a SUB operation.2
  • 3PARA RAT will use an 8-byte XOR key derived from the string HYF54&%9&jkMCXuiS instead if the DES decoding fails.3
  • 4H RAT obfuscates C2 communication using a 1-byte XOR with the key 0xBE.3
  • BADNEWS encrypts C2 data with a ROR by 3 and an XOR by 0x23.4
  • BBSRAT uses a custom encryption algorithm on data sent back to the C2 server over HTTP.5
  • CORESHELL C2 messages are encrypted with custom stream ciphers using six-byte or eight-byte keys.6
  • ChChes can encrypt C2 data with a custom technique using MD5, base64-encoding, and RC4.78
  • CosmicDuke contains a custom version of the RC4 algorithm that includes a programming error.1
  • Derusbi obfuscates C2 traffic with variable 4-byte XOR keys.9
  • The C2 server response to a beacon sent by a variant of Emissary contains a 36-character GUID value that is used as an encryption key for subsequent network communications. Some variants of Emissary use various XOR operations to encrypt C2 data.10
  • The original variant of FakeM encrypts C2 traffic using a custom encryption cipher that uses an XOR key of “YHCRA” and bit rotation between each XOR operation. FakeM has also included HTML code in C2 traffic in an apparent attempt to evade detection. Additionally, some variants of FakeM use modified SSL code for communications back to C2 servers, making SSL decryption ineffective.11
  • Before being appended to image files, HAMMERTOSS commands are encrypted with a key composed of both a hard-coded value and a string contained on that day's tweet. To decrypt the commands, an investigator would need access to the intended malware sample, the day's tweet, and the image file containing the command.12
  • Hikit performs XOR encryption.13
  • Lurid performs XOR encryption.14
  • RTM encrypts C2 traffic with a custom RC4 variant.15
  • Sakula encodes C2 traffic with single-byte XOR keys.16
  • Sys10 uses an XOR 0x1 loop to encrypt its C2 domain.17
  • Taidoor is known to utilize encryption within network protocols.18
  • httpclient encrypts C2 content with XOR using a single byte, 0x12.3


Network intrusion detection and prevention systems that use network signatures to identify traffic for specific adversary malware can be used to mitigate activity at the network level. Since the custom protocol used may not adhere to typical protocol standards, there may be opportunities to signature the traffic on a network level for detection. Signatures are often for unique indicators within protocols and may be based on the specific protocol used by a particular adversary or tool, and will likely be different across various malware families and versions. Adversaries will likely change tool C2 signatures over time or construct protocols in such a way as to avoid detection by common defensive tools.19


If malware uses custom encryption with symmetric keys, it may be possible to obtain the algorithm and key from samples and use them to decode network traffic to detect malware communications signatures.20

In general, analyze network data for uncommon data flows (e.g., a client sending significantly more data than it receives from a server). Processes utilizing the network that do not normally have network communication or have never been seen before are suspicious. Analyze packet contents to detect when communications do not follow the expected protocol behavior for the port that is being used.19