Custom Cryptographic Protocol

Jump to: navigation, search
Custom Cryptographic Protocol
ID T1024
Tactic Command and Control
Platform Windows Server 2003, Windows Server 2008, Windows Server 2012, Windows XP, Windows 7, Windows 8, Windows Server 2003 R2, Windows Server 2008 R2, Windows Server 2012 R2, Windows Vista, Windows 8.1, Linux
Data Sources Packet capture, Netflow/Enclave netflow, Process use of network, Malware reverse engineering, Process monitoring
Requires Network Yes

Adversaries may use a custom cryptographic protocol or algorithm to hide command and control traffic. A simple scheme, such as XOR-ing the plaintext with a fixed key, will produce a very weak ciphertext.

Custom encryption schemes may vary in sophistication. Analysis and reverse engineering of malware samples may be enough to discover the algorithm and encryption key used.

Some adversaries may also attempt to implement their own version of a well-known cryptographic algorithm instead of using a known implementation library, which may lead to unintentional errors.1


  • Several Lazarus Group malware families encrypt C2 traffic using custom code that uses XOR with an ADD operation and XOR with a SUB operation.2
  • Hikit performs XOR encryption.3
  • Lurid performs XOR encryption.4
  • Taidoor is known to utilize encryption within network protocols.5
  • Derusbi obfuscates C2 traffic with variable 4-byte XOR keys.6
  • Before being appended to image files, HAMMERTOSS commands are encrypted with a key composed of both a hard-coded value and a string contained on that day's tweet. To decrypt the commands, an investigator would need access to the intended malware sample, the day's tweet, and the image file containing the command.7
  • CosmicDuke contains a custom version of the RC4 algorithm that includes a programming error.1
  • Sys10 uses an XOR 0x1 loop to encrypt its C2 domain.8
  • 4H RAT obfuscates C2 communication using a 1-byte XOR with the key 0xBE.9
  • 3PARA RAT will use an 8-byte XOR key derived from the string HYF54&%9&jkMCXuiS instead if the DES decoding fails.9
  • httpclient encrypts C2 content with XOR using a single byte, 0x12.9
  • Sakula encodes C2 traffic with single-byte XOR keys.10
  • The original variant of FakeM encrypts C2 traffic using a custom encryption cipher that uses an XOR key of “YHCRA” and bit rotation between each XOR operation. FakeM has also included HTML code in C2 traffic in an apparent attempt to evade detection. Additionally, some variants of FakeM use modified SSL code for communications back to C2 servers, making SSL decryption ineffective.11
  • The C2 server response to a beacon sent by a variant of Emissary contains a 36-character GUID value that is used as an encryption key for subsequent network communications. Some variants of Emissary use various XOR operations to encrypt C2 data.12
  • BBSRAT uses a custom encryption algorithm on data sent back to the C2 server over HTTP.13
  • BADNEWS encrypts C2 data with a ROR by 3 and an XOR by 0x23.14
  • CORESHELL C2 messages are encrypted with custom stream ciphers using six-byte or eight-byte keys.15


Network intrusion detection and prevention systems that use network signatures to identify traffic for specific adversary malware can be used to mitigate activity at the network level. Since the custom protocol used may not adhere to typical protocol standards, there may be opportunities to signature the traffic on a network level for detection. Signatures are often for unique indicators within protocols and may be based on the specific protocol used by a particular adversary or tool, and will likely be different across various malware families and versions. Adversaries will likely change tool C2 signatures over time or construct protocols in such a way as to avoid detection by common defensive tools.16


If malware uses custom encryption with symmetric keys, it may be possible to obtain the algorithm and key from samples and use them to decode network traffic to detect malware communications signatures.17

In general, analyze network data for uncommon data flows (e.g., a client sending significantly more data than it receives from a server). Processes utilizing the network that do not normally have network communication or have never been seen before are suspicious. Analyze packet contents to detect when communications do not follow the expected protocol behavior for the port that is being used.16