Custom Cryptographic Protocol
Adversaries may use a custom cryptographic protocol or algorithm to hide command and control traffic. A simple scheme, such as XOR-ing the plaintext with a fixed key, will produce a very weak ciphertext.
Custom encryption schemes may vary in sophistication. Analysis and reverse engineering of malware samples may be enough to discover the algorithm and encryption key used.
Some adversaries may also attempt to implement their own version of a well-known cryptographic algorithm instead of using a known implementation library, which may lead to unintentional errors. 
|3PARA RAT||3PARA RAT will use an 8-byte XOR key derived from the string HYF54&%9&jkMCXuiS instead if the DES decoding fails. |
|4H RAT||4H RAT obfuscates C2 communication using a 1-byte XOR with the key 0xBE. |
|APT28||APT28 installed a Delphi backdoor that used a custom algorithm for C2 communications. |
|BADCALL||BADCALL encrypts C2 traffic using an XOR/ADD cipher and uses a FakeTLS method. |
|BADNEWS||BADNEWS encrypts C2 data with a ROR by 3 and an XOR by 0x23.  |
|BBSRAT||BBSRAT uses a custom encryption algorithm on data sent back to the C2 server over HTTP. |
|Bisonal||Bisonal variants reported on in 2014 and 2015 used a simple XOR cipher for C2. |
|BRONZE BUTLER||BRONZE BUTLER has used a tool called RarStar that encodes data with a custom XOR algorithm when posting it to a C2 server. |
|Carbanak||Carbanak uses XOR with random keys for its communications. |
|Cardinal RAT||Cardinal RAT uses a secret key with a series of XOR and addition operations to encrypt C2 traffic. |
|ChChes||ChChes can encrypt C2 data with a custom technique using MD5, base64-encoding, and RC4.  |
|CORESHELL||CORESHELL C2 messages are encrypted with custom stream ciphers using six-byte or eight-byte keys. |
|CosmicDuke||CosmicDuke contains a custom version of the RC4 algorithm that includes a programming error. |
|Derusbi||Derusbi obfuscates C2 traffic with variable 4-byte XOR keys. |
|Ebury||Ebury has encrypted C2 traffic using the client IP address, then encoded it as a hexadecimal string. |
|Emissary||The C2 server response to a beacon sent by a variant of Emissary contains a 36-character GUID value that is used as an encryption key for subsequent network communications. Some variants of Emissary use various XOR operations to encrypt C2 data. |
|FakeM||The original variant of FakeM encrypts C2 traffic using a custom encryption cipher that uses an XOR key of “YHCRA” and bit rotation between each XOR operation. FakeM has also included HTML code in C2 traffic in an apparent attempt to evade detection. Additionally, some variants of FakeM use modified SSL code for communications back to C2 servers, making SSL decryption ineffective. |
|FALLCHILL||FALLCHILL uses fake Transport Layer Security (TLS) to communicate with its C2 server, encoding data with RC4 encryption. |
|Felismus||Some Felismus samples use a custom encryption method for C2 traffic using AES, base64 encoding, and multiple keys. |
|Gazer||Gazer uses custom encryption for C2 using 3DES and RSA.  |
|HAMMERTOSS||Before being appended to image files, HAMMERTOSS commands are encrypted with a key composed of both a hard-coded value and a string contained on that day's tweet. To decrypt the commands, an investigator would need access to the intended malware sample, the day's tweet, and the image file containing the command. |
|HARDRAIN||HARDRAIN uses FakeTLS to communicate with its C2 server. |
|HiddenWasp||HiddenWasp uses an RC4-like algorithm with an already computed PRGA generated key-stream for network communication. |
|Hikit||Hikit performs XOR encryption. |
|httpclient||httpclient encrypts C2 content with XOR using a single byte, 0x12. |
|Hydraq||Hydraq C2 traffic is encrypted using bitwise NOT and XOR operations. |
|InvisiMole||InvisiMole uses variations of a simple XOR encryption routine for C&C communications. |
|KeyBoy||KeyBoy uses custom SSL libraries for C2 traffic. |
|KEYMARBLE||KEYMARBLE uses a customized XOR algorithm to encrypt C2 communications. |
|Komplex||The Komplex C2 channel uses an 11-byte XOR algorithm to hide data. |
|Lazarus Group||Several Lazarus Group malware families encrypt C2 traffic using custom code that uses XOR with an ADD operation and XOR with a SUB operation. Another Lazarus Group malware sample XORs C2 traffic. Lazarus Group malware also uses a unique form of communication encryption known as FakeTLS that mimics TLS but uses a different encryption method, evading SSL man-in-the-middle decryption attacks.    |
|Lurid||Lurid performs XOR encryption. |
|Mosquito||Mosquito uses a custom encryption algorithm, which consists of XOR and a stream that is similar to the Blum Blum Shub algorithm. |
|PLAINTEE||PLAINTEE encodes C2 beacons using XOR. |
|RTM||RTM encrypts C2 traffic with a custom RC4 variant. |
|Sakula||Sakula encodes C2 traffic with single-byte XOR keys. |
|Sys10||Sys10 uses an XOR 0x1 loop to encrypt its C2 domain. |
|Taidoor||Taidoor is known to utilize encryption within network protocols. |
|TrickBot||TrickBot uses a custom crypter leveraging Microsoft’s CryptoAPI to encrypt C2 traffic. |
|WannaCry||WannaCry uses a custom cryptographic protocol over the Tor circuit. |
|Network Intrusion Prevention||Network intrusion detection and prevention systems that use network signatures to identify traffic for specific adversary malware can be used to mitigate activity at the network level. Since the custom protocol used may not adhere to typical protocol standards, there may be opportunities to signature the traffic on a network level for detection. Signatures are often for unique indicators within protocols and may be based on the specific protocol used by a particular adversary or tool, and will likely be different across various malware families and versions. Adversaries will likely change tool C2 signatures over time or construct protocols in such a way as to avoid detection by common defensive tools.|
If malware uses custom encryption with symmetric keys, it may be possible to obtain the algorithm and key from samples and use them to decode network traffic to detect malware communications signatures. 
In general, analyze network data for uncommon data flows (e.g., a client sending significantly more data than it receives from a server). Processes utilizing the network that do not normally have network communication or have never been seen before are suspicious. Analyze packet contents to detect when communications do not follow the expected protocol behavior for the port that is being used. 
- F-Secure Labs. (2014, July). COSMICDUKE Cosmu with a twist of MiniDuke. Retrieved July 3, 2014.
- Miller-Osborn, J. and Grunzweig, J.. (2017, February 16). menuPass Returns with New Malware and New Attacks Against Japanese Academics and Organizations. Retrieved March 1, 2017.
- Nakamura, Y.. (2017, February 17). ChChes - Malware that Communicates with C&C Servers Using Cookie Headers. Retrieved March 1, 2017.
- Bennett, J., Vengerik, B. (2017, June 12). Behind the CARBANAK Backdoor. Retrieved June 11, 2018.
- Villeneuve, N., Sancho, D. (2011). THE “LURID” DOWNLOADER. Retrieved November 12, 2014.
- Crowdstrike Global Intelligence Team. (2014, June 9). CrowdStrike Intelligence Report: Putter Panda. Retrieved January 22, 2016.
- Falcone, R. and Miller-Osborn, J.. (2016, January 24). Scarlet Mimic: Years-Long Espionage Campaign Targets Minority Activists. Retrieved February 10, 2016.
- Fidelis Cybersecurity. (2016, February 29). The Turbo Campaign, Featuring Derusbi for 64-bit Linux. Retrieved March 2, 2016.
- US-CERT. (2018, February 05). Malware Analysis Report (MAR) - 10135536-F. Retrieved June 11, 2018.
- Reaves, J. (2016, October 15). TrickBot: We Missed you, Dyre. Retrieved August 2, 2018.
- Hayashi, K., Ray, V. (2018, July 31). Bisonal Malware Used in Attacks Against Russia and South Korea. Retrieved August 7, 2018.
- US-CERT. (2017, November 22). Alert (TA17-318A): HIDDEN COBRA – North Korean Remote Administration Tool: FALLCHILL. Retrieved December 7, 2017.
- Settle, A., et al. (2016, August 8). MONSOON - Analysis Of An APT Campaign. Retrieved September 22, 2016.
- Lunghi, D., et al. (2017, December). Untangling the Patchwork Cyberespionage Group. Retrieved July 10, 2018.
- Dani Creus, Tyler Halfpop, Robert Falcone. (2016, September 26). Sofacy's 'Komplex' OS X Trojan. Retrieved July 8, 2017.
- Lelli, A. (2010, January 11). Trojan.Hydraq. Retrieved February 20, 2018.
- FireEye Labs. (2015, July). HAMMERTOSS: Stealthy Tactics Define a Russian Cyber Threat Group. Retrieved September 17, 2015.
- Faou, M. and Boutin, J.. (2017, February). Read The Manual: A Guide to the RTM Banking Trojan. Retrieved March 9, 2017.
- Novetta. (n.d.). Operation SMN: Axiom Threat Actor Group Report. Retrieved November 12, 2014.
- FireEye. (2015). APT28: A WINDOW INTO RUSSIA’S CYBER ESPIONAGE OPERATIONS?. Retrieved August 19, 2015.
- Lee, B. Grunzweig, J. (2015, December 22). BBSRAT Attacks Targeting Russian Organizations Linked to Roaming Tiger. Retrieved August 19, 2016.
- Dell SecureWorks Counter Threat Unit Threat Intelligence. (2015, July 30). Sakula Malware Family. Retrieved January 26, 2016.
- ESET. (2017, August). Gazing at Gazer: Turla’s new second stage backdoor. Retrieved September 14, 2017.
- Kaspersky Lab's Global Research & Analysis Team. (2017, August 30). Introducing WhiteBear. Retrieved September 21, 2017.
- US-CERT. (2018, August 09). MAR-10135536-17 – North Korean Trojan: KEYMARBLE. Retrieved August 16, 2018.
- Ash, B., et al. (2018, June 26). RANCOR: Targeted Attacks in South East Asia Using PLAINTEE and DDKONG Malware Families. Retrieved July 2, 2018.
- US-CERT. (2018, February 06). Malware Analysis Report (MAR) - 10135536-G. Retrieved June 7, 2018.
- Grunzweig, J.. (2017, April 20). Cardinal RAT Active for Over Two Years. Retrieved December 8, 2018.
- Somerville, L. and Toro, A. (2017, March 30). Playing Cat & Mouse: Introducing the Felismus Malware. Retrieved November 16, 2017.
- Falcone, R. and Miller-Osborn, J.. (2015, December 18). Attack on French Diplomat Linked to Operation Lotus Blossom. Retrieved February 15, 2016.
- Hromcová, Z. (2018, June 07). InvisiMole: Surprisingly equipped spyware, undercover since 2013. Retrieved July 10, 2018.
- ESET, et al. (2018, January). Diplomats in Eastern Europe bitten by a Turla mosquito. Retrieved July 3, 2018.
- Baumgartner, K., Golovkin, M.. (2015, May). The MsnMM Campaigns: The Earliest Naikon APT Campaigns. Retrieved April 10, 2019.
- Trend Micro. (2012). The Taidoor Campaign. Retrieved November 12, 2014.
- Counter Threat Unit Research Team. (2017, May 18). WCry Ransomware Analysis. Retrieved March 26, 2019.
- M.Léveillé, M.. (2014, February 21). An In-depth Analysis of Linux/Ebury. Retrieved April 19, 2019.
- Parys, B. (2017, February 11). The KeyBoys are back in town. Retrieved June 13, 2019.
- Sanmillan, I. (2019, May 29). HiddenWasp Malware Stings Targeted Linux Systems. Retrieved June 24, 2019.
- Novetta Threat Research Group. (2016, February 24). Operation Blockbuster: Unraveling the Long Thread of the Sony Attack. Retrieved February 25, 2016.
- Novetta Threat Research Group. (2016, February 24). Operation Blockbuster: Destructive Malware Report. Retrieved March 2, 2016.
- Sherstobitoff, R. (2018, February 12). Lazarus Resurfaces, Targets Global Banks and Bitcoin Users. Retrieved February 19, 2018.
- Sherstobitoff, R., Malhotra, A. (2018, April 24). Analyzing Operation GhostSecret: Attack Seeks to Steal Data Worldwide. Retrieved May 16, 2018.
- Counter Threat Unit Research Team. (2017, October 12). BRONZE BUTLER Targets Japanese Enterprises. Retrieved January 4, 2018.
- ESET Research. (2019, May 22). A journey to Zebrocy land. Retrieved June 20, 2019.
- Fidelis Cybersecurity. (2015, August 4). Looking at the Sky for a DarkComet. Retrieved April 5, 2016.
- Gardiner, J., Cova, M., Nagaraja, S. (2014, February). Command & Control Understanding, Denying and Detecting. Retrieved April 20, 2016.