Rule out a standard cipher first
Most payloads that look custom are a standard algorithm with an unusual key or input format. Before treating the transform as bespoke, describe the ciphertext. Base64 with padding suggests an encoded byte string. A length that is a multiple of sixteen suggests a block cipher with padding. A length equal to the plaintext suggests a stream construction, a counter mode, or an exclusive-or layer over the bytes.
Then try the common arrangements against a captured body. A standard library combined with a key derived from a string, a fixed initialisation vector, a random vector stored in the first block, and a padding scheme accounts for a large share of real implementations. If one of those produces readable JSON, the algorithm is standard and the remaining work is key recovery.
Find the constants that identify a library
Compiled crypto leaves fingerprints. Search the APK for the first bytes of the AES substitution table, the initial hash values of SHA-256, and the constants used by MD5 and SHA-1. Their presence points to a library implementation and to a standard algorithm. Search for the class names of the platform crypto provider, BouncyCastle and Conscrypt, and for the symbols of common native libraries such as OpenSSL and mbedTLS.
A transform implemented by hand often has none of those markers. The code is a loop over the bytes that combines shifts, additions and exclusive-or operations, sometimes with a position index folded in. Those loops are short and readable once you find them, and the decompiler often renders one as a single method with a few local variables.
Classify the construction with length tests
Send two requests whose bodies differ in a controlled way and compare ciphertext lengths. A stream construction preserves length exactly, so a body one byte longer produces a ciphertext one byte longer. A block mode without a padding scheme rounds up to the next multiple of the block size. A compressed payload shows a length that does not track the input, because compression absorbs the change.
Block modes have their own tests. Build a body containing two identical runs of sixteen bytes and send it. If the two runs encrypt to the same ciphertext block, the mode is electronic codebook and identical plaintext blocks leak their equality. If they differ, a chaining or counter mode is in use.
Test whether the vector changes between requests
Send the same body twice and compare the ciphertext. Identical output on both sends means the initialisation vector or nonce is fixed, either constant or derived from the key. That condition allows a known-plaintext attack on a stream mode, because two messages under the same keystream reveal their difference. It also means your client can reproduce the exact bytes the server accepts.
Different output on identical bodies means the vector is fresh per request. Look for it in the payload, usually as a prefix block, or in a header next to the body. A fresh vector that is not transmitted is a bug that shows up as a server failure on the first request, so if the server accepts the traffic the vector is somewhere in the message.
Recover the transform with chosen input
The fastest route to a custom transform is to run it on inputs you choose. Hook the method that performs the transform, whether that is a platform cipher call, a BouncyCastle entry point, or a native function, and log the input and output for each call. Then call the same method directly with a controlled input: a block of zeros, a single set bit, a repeating pattern, and every byte value in one position.
- Feed a zero block. The output is the keystream, or the result of the key schedule on a known state, which often exposes the key directly.
- Flip one bit in the input. The position and number of changed output bits show whether the transform mixes across the whole block or works byte by byte.
- Run all byte values through one position. The resulting table is a substitution box when the transform is byte-oriented, and a table is enough to reproduce the transform without understanding the original source.
A native implementation accepts the same treatment. Calling it from a hook keeps the key handling inside the app, which matters when the key is derived from device values at runtime.
Verify by reproducing a capture
The recovered scheme is correct when it turns a captured plaintext into the captured ciphertext byte for byte, including the vector. Check it on three captures that differ in length and content, and check that the server accepts the output. A scheme that matches one capture may have the vector handling wrong, which shows up only on the second request. A Python client is the usual way to hold the recovered transform, and request signing covers the digest layer that often wraps the encrypted body.
Related work
Reviewed 28 September 2026 · SReverse research desk