I learned about the design of this machine from a page on the Crypto Museum web site.
I found that page by accident when searching for information about teletypewriter machines not related to cryptography. So at first, I just glanced at the page briefly, learning about one ingenious aspect of its design.
The paper tapes containing the random keys for the machine were six-channel tapes rather than five-channel tapes. So that the keytapes would not be put in with the wrong face up, as the feed holes were in the middle of the six holes with data, a wider margin of unpunched tape was present on one side.
The sixth hole on the tape was used to indicate that the five remaining bits, instead of being key bits, were data to be transmitted (and printed) as-is. So one could have a keytape consisting of groups of five random characters separated by either spaces so indicated or CR/LF pairs so indicated, and one's ciphertext would print out neatly without the cipher machine having to contain a mechanism to count off groups of five letters.
This was clever, but upon reflection, some problems arise.
If one performs an XOR between five bits of random data and a cipher message in 5-level code, some of the resulting ciphertext characters will be control characters.
So, the ciphertext won't actually print out neatly. Worse, the recipient might have a problem distinguishing between control characters that are intentional formatting, and accidental control characters that are genuinely part of the ciphertext, although this problem might be avoided by keeping careful count of incoming characters.
Given that the machine had the capability of not advancing the normal 5-level paper tape containing the plaintext when a formatting character was to be printed, it occurred to me that this dilemma could be solved as follows:
Select an indicator letter. It could be a fixed letter, say Z, or it could be chosen by a knob on the machine.
Then: when the XOR between a plaintext character and a key character would be a control character or the indicator letter, print and transmit the indicator letter instead, and do not advance the plaintext tape: try again with that plaintext character, applying the next key character to it with XOR.
This allows the plaintext to consist of normal 5-level code, including all control characters, and the key characters on the keytape would contain five completely random bits, having any of the 32 possible values.
I believe that the resulting cipher would be equivalent in security to a true one-time pad, with the possible advantage of also having some resistance to the bit-flipping attack. It would, however, be recognizable, in that the indicator letter would occur seven times more often in the ciphertext than any other letter.
Well, when I returned to that web page later, I saw that it did contain a description of how the actual Rockex dealt with the issue I noted. While the fashion in which it did so did involve the basic principle of attempting an XOR, and then trying again if the result was not of the desired type, used in my hypothetical reconstruction above, it was none the less quite different.
In the real Rockex, the key characters were always characters from A through Z, no other bit combinations being permitted.
When the XOR of a key character with a plaintext character is a control character, the key character is transmitted and printed instead, and the plaintext tape is not advanced, with the plaintext character being XORed again with the next key character.
Unlike the key characters on the keytape, the plaintext characters could be control characters, so the plaintext could contain spaces, figures characters, and formatting. Only the all-zeroes character could not occur in the plaintext, because that would create a case which could not be distinguished from the transmission of the key character to indicate that an attempted XOR produced an undesired result.
The web site that is my source claims that this had no negative effect on security.
(Incidentally, in addition to formatting characters, the keytape used the sixth bit to cause a five-letter keytape identifier to be passed through as a message indicator.)
I am not so certain of this: the fact that the plaintext consists of arbitrary characters taken from the thirty-one possible non-null characters, but the key can only include the twenty-six letters, seems to mean that it was no longer a true one-time system in the theoretical sense.
As with SIGNIN or the M-294, it could still have been fully secure in practice. While I do think that SIGNIN was almost certainly unbreakable, I am not yet as confident in the case of the Rockex - although, even if it does leak a slight amount of information concerning the plaintext, it may well be that there is no way that information could be put to use.
Upon further reflection, this means that, for every single letter of ciphertext, either it is a keyletter, or there are six of the thirty-one possible plaintext characters to which it absolutely may not correspond. This, at least, would permit plaintext cribs to be aligned with the ciphertext.
While a one-time system device would not be used in such conditions as would normally lead to a message on the order of "Situation peaceful, nothing to report" being sent day after day, which would allow it to be reconstructed and to the transmission of a different message being detected even if it had the same length, I would still be inclined to regard that amount of information being leaked concerning the plaintext to be unacceptable.
However, I may be too hasty in making that judgment.
The presence of characters in the ciphertext that are key characters passed through instead of being enciphered plain text does, obviously, create additional confusion. While it may seem as though this additional confusion wouldn't be enough to eliminate the leak of information due to having a plaintext alphabet of 31 characters and a key alphabet of 26 characters... the fact that the ciphertext alphabet contains exactly 26 characters as well may mean that there is an exact match, and that the method used to deal with undesired XOR results can be proved equivalent to a conversion of the plaintext into a strictly alphabetic form.
If so, however, the proof would be more difficult, and require more sophistication, than the proof of security for the alternate scheme that I came up with.
At least, that's what I think. But what is really the case?
First, let's start with the system I came up with, since I claimed that it is simpler to analyze.
Look at a character of the ciphertext.
If it is a control character, then clearly it is a formatting character from the key tape.
If it is the indicator letter, that will normally be obvious; even if the indicator letter changes with each message, it is much more common than any of the other letters, and that will be visible.
For any of the other 25 letters, since the key could have had any of the 32 possible binary values, the plaintext character could also have had any of the 32 possible binary values.
So occurrences of the indicator letter show that the key managed to match the plaintext value in such a way as to lead to an undesired output character, but again, since the key could have had any of the possible 32 values, nothing is revealed.
The actual system involves a key composed only of letters of the alphabet, and the ciphertext is also composed only of letters of the alphabet. So the entropy of the two matches. But they aren't combined by modulo-26 addition, so could some of the entropy of the key be wasted in a process involving XOR of 5-bit characters and delaying the plaintext?
Look at an individual letter of the ciphertext.
Suppose, for example, that the letter of the ciphertext is E, with the binary code 10000.
We know that this ciphertext letter did not result from an XOR of the impossible key value 00000 with 10000 in the plaintext. But this doesn't mean that the plaintext could not have contained an E at that point; perhaps an E was there, forcing a delay in plaintext encipherment, and so the next letter of the ciphertext will represent that letter.
Because the system does not permit the plaintext to contain zeroes, we know that the key letter could not have been an E unless we have had a rejected XOR result.
As the key consists only of letters of the alphabet, the values it cannot take are: 00000 (nul), 00100 (space), 00010 (CR), 01000 (LF), 11011 (FIGS), and 11111 (LTRS).
And so, if an XOR took place instead of a plaintext delay, when the ciphertext is E (10000), the plaintext, in addition to not being an E, also cannot be 10100 (S), 10010 (D), 11000 (A), 01011 (G), or 01111 (V).
But all these letters could still turn up in the plaintext at that point; the cipher machine can't prevent them. They would just be delayed until the next letter of the ciphertext.
Since there are six possible letters - and one of them the most common letter of the alphabet - leading to an E being printed when the key letter is an E, some information has been leaked about the random key. But enciphering a plaintext, as noted, doesn't impose probabilities on the plaintext characters, and so we haven't seen that we've learned anything about the plaintext.
But what about the next letter of the message?
If the previous ciphertext letter was E, then presumably that means that due to the possibility a character was bumped, for the next letter, there is an elevated probability that the plaintext is E, S, D, A, G, or V.
But that doesn't help in breaking a one-time pad any more than normal letter frequencies help.
Except for the last letter of the message, which could not be a bump, and therefore messages do need to be padded at the end for security, no ciphertext letter leaks any information about the original plaintext. That the key is more likely to match the ciphertext at any given position, and that some plaintext letters would have to be moved away from some points, is leaked.
That is a slight oversimplification. If the message happened to be EEEEE EEEEE EEEEE.... right up to the end, then indeed the letter E could not occur in the plaintext. So, indeed, it isn't an absolutely perfect one-time system. But in a normal random-looking message, that looks like EXPZT VRGJQ... then while E could not have become E, it's perfectly capable of becoming X (10111) under a key of M (00111), and so given a keystream of genuinely random letters, possible characters in the plaintext are all equally likely as far as we can determine from the ciphertext.
For each ciphertext letter, there are 25 slots of equal probability corresponding to 25 possible characters in the plaintext. Of the 31 possible plaintext characters, 25 of them go into those slots, and the other six are postponed to the next ciphertext position... in which they are likely to take six of the 25 available slots there. The odd plaintext character being delayed two or three times doesn't change the fact that once each character is enciphered, the slot into which it fell was still one with a 1/25 probability... no plaintext letter becomes more or less likely because of the sequence of ciphertext letters except in the rare degenerate case of a plaintext that would fall off the end of the message.
Basically, there should be five or ten letters of random padding at the end of the message, and the ciphertext should be cut off by some rule that allows the cut-off to take place at a position where a plaintext letter is bumped (as opposed to stopping exactly when the last character of the message, plus padding, is fully enciphered).
So this system is secure precisely because the positions at which the advance of the plaintext is paused are invisible in the ciphertext, whereas the one I had proposed allowed them to be seen without affecting security.