Devised by Dr. Ronald C. Rivest, of RSA fame, RC6 is based on Feistel rounds; but not Feistel rounds operating between the two halves of the block. Instead, the Feistel rounds operate between pairs of quarters of the block, and they are interlocked by the exchange of some data. Circular shifts the extent of which is controlled by data, and a quadratic function applied to 32-bit integers are the nonlinear elements which provide the security of this block cipher.
The design of the block cipher is such that the number of rounds, the size of the key, and the size of the block, are all flexible. It is based on the previous RC5 block cipher, which is patented by RSA Laboratories, but instead of being RC5 expanded to a larger block size, it is designed to limit the width of registers required to handle large blocks. It will be described here only with the specific number of rounds and blocksize proposed for the AES, and I am not yet including a description of the key schedule.
The original description of RC6 is available at http://theory.lcs.mit.edu/~rivest/publications.html.
RC6 uses 44 subkeys, numbered S0 to S43, each one 32 bits long. The block to be enciphered is divided into four 32-bit integers, A, B, C, and D. The first four bytes enciphered form A, and the convention is little-endian; the first byte enciphered becomes the least significant byte of A.
RC6 begins with an initial whitening step; B is XORed with S0, and D is XORed with S1.
Each round of RC6 uses two subkeys; the first one uses S2 and S3, and successive rounds use successive subkeys.
A round of RC6 proceeds as follows:
The result of subjecting B to function f, which is defined as f(x) = x(2x+1), is rotated left 5 bits, and then XORed to A.
The result of subjecting D to function f, which is defined as f(x) = x(2x+1), is rotated left 5 bits, and then XORed to C.
The least significant 5 bits of the quantity XORed to A (function f of B circular left shifted 5 bits) specifies the extent to which C is circular left shifted.
The least significant 5 bits of the quantity XORed to C (function f of D circular left shifted 5 bits) specifies the extent to which A is circular left shifted.
Since all this leaves B and D unchanged, it is invertible.
Then the first subkey for the round is XORed to A, and the second subkey for the round is XORed to C.
Then, the four quarters of the block are rotated as follows: the value of A is placed in D, the value of B is placed in A, the value of C is placed in B, and the (original) value of D is placed in C.
After the 20th round is complete, an additional whitening step takes place: A is XORed with S42, and C is XORed with S43.
Table of Contents