The primary key of the algorithm is 128bit long (an MD5 value).
This encryption scheme uses the complete set of 32bit xorshift RNG (for a
total of 648 generators). Each of them requires a 32bit seed and generates
a different sequence of 32bit integers of period 2^32-1. As you probably
already know, they are fast (3 shifts and 3 xor to generate a number) and
have good statistical properties.
The algorithm has two operating modes: standard mode and stream mode.
Standard mode requires the computation of a "footprint" of the plaintext
(which is MD5(key+MD5(plaintext))) before encrypting. This means that the
entire plaintext must be known at the moment of encryption. This
"footprint", which is 16 bytes long, is added to the output ciphertext.
Stream mode does not alter plaintext length, has no "footprint"
computation (and thus is also faster) but is weak against chosen
plaintext attacks, and should not be used.
Encryption is done by using 4 xorshift PRNGs at a time, chosen from the
complete set of 648, and changing them periodically, together with their
seeds. Their 32bit output is xored with 32bit of plaintext to produce the
ciphertext. A 32bit variable, "rng_state", is used to accumulate
information on the evolution of the PRNG seeds, and to decide which ones
to use to encrypt the current word of plaintext.

Short description of standard mode encryption:

(1) key=MD5(key)
(2) if one of key's 32bit words equals 0 --> (1)
(3) Compute "footprint"=MD5(key+MD5(input file))
(4) Initialize the 4 xorshift PRNG seeds to MD5(footprint+key)
(5) For each 256 bytes block of plaintext (sequentially read):
(5.1) Choose 4 PRNGs out of 648. Decision is based on MD5(seeds)
(5.2) Initialize variable "rng_state" according to the same MD5(seeds)
(5.3) For each 32bit word of block
(5.3.1) Repeat four times (rounds) the following
(5.3.1.1) Choose which PRNG to use (of the four): chosen = rng_state&3
(5.3.1.2) Rotate right "rng_state" by three positions
(5.3.1.3) Use the chosen PRNG to generate two numbers
(5.3.1.4) Xor the first number with current 32bit word of plaintext
(5.3.1.5) Discard the second (but update the seed)
(5.3.1.6) Use the PRNG next to the chosen one (mod 4) to generate a num
(5.3.1.7) Update rng_state to (rng_state xor num)
(5.4) Re-initialize the four seeds to MD5(current seeds)

Standard mode decryption is achieved by using the same algorithm, but
with the footprint directly read from the ciphertext file.

Stream mode differs only in the initialization part: there is no
"footprint" computation and the RNG seeds depend only on the key.

Supplied test executable usage:

  xse <act> <input file> <output file> "<key>"

<act> can be "e" for standard encryption, "d" for standard decryption or
"s" for stream mode (simmetric encryption/decryption)
In stream mode <input file> can be "-" for stdin and <output file> can
be "-" for stdout. Stream mode is weaker and should not be used.
<key> is a hex value. It can be up to 16 bytes long (32 chars). You
should enter all of them (ex:"52FF253FBA82CF14F745D1ED85DB387D").

Example:
xse e testfile.dat testfile.xse "12345"
xse d testfile.xse testfile2.dat "12345"

Provided plain/ciphertext and challenge:
All of ciphertext submitted (challenge included) are encrypted with the
same password.

plain1.dat     20kb of zeroes
plain2.dat     the test executable' source code
plain3.dat     the test executable
plain4.dat     cryptography-faq/part05
challenge.dat  ??? (hint: it's an html page...this should give you some
               known plaintext to start with)

Improvements
The footprint can probably be replaced with 16 bytes read from /dev/urandom
without affecting security (under the assumption that /dev/urandom cannot be
replaced with a fake device). This would eliminate the initial delay of
computing MD5(plaintext).

I will really appreciate any feedback...if you try some tests on xse,
please post to sci.crypt your findings and/or thoughts, whatever they
are (flames excluded! :-)

Thanks
Stephen