Help with MSRC4 key format for unix UVNC DSM plugin

[krunge]

Hello,

I'm working on a Unix helper program for my vncviewer and server projects
(ssvnc and x11vnc) that will allow them to work with UVNC programs using
the DSM encryption plugin (they already work with SSL connections for SC III
and repeater_SSL.exe).

I have it more or less working with the ARC4 and AESV2 plugins (however the
tight and zlib encodings still don't work...). The encryption/decryption
part for these was pretty easy because I use OpenSSL as do the plugins.

For the MSRC4 plugin, I am hoping I can simply use my ARC4 code as long as
I can figure out how to extract the RC4 key 'rc4.key' file (76 bytes in
the key portion of the file) and transform it into 'arc4.key' key format
(16 bytes of data, i.e. a 128 bit key)

Does anyone know where I might find out or start looking for how to convert
between the two formats?

[redge]

CryptDeriveKey Function
http://msdn.microsoft.com/en-us/library/aa379916(VS.85).aspx

RC4 Cipher Usage
http://msdn.microsoft.com/en-us/library/cc207219.aspx

MS CryptoAPI keypal
post #3373 in topic "DSMPlugin cannot be configured"

[krunge]

MS CryptoAPI keypal
post #3373 in topic "DSMPlugin cannot be configured"

Very nice! That keypal link is broken, but going to its www.jensign.com site reveals the guy there is a super-genius. He has the key file format nicely described (not explicitly for RC4, but it applies). So it is simple to extract the key in C with no extra libraries.

I would have never guessed the 16 byte RC4 key was stored in reverse order!

So now I have the SSVNC Unix viewer working with MSRC4, ARC4, and AESV2 DSM plugins!! It is working with the uvnc 105 server, still some problem with SC -plugin mode, but I will look at that tomorrow.

Thanks very much for the pointers.

[redge]

> still some problem with SC -plugin mode

be carefull, you really need at both side the same release of msrc4plugin.dsm, actually is not the case.
SC contain be default 1.1.8.0 and not compatible with the new uvnc 1.0.5 have msrc4plugin.dsm 1.2.2.0

for add you own msrc4plugin.dsm up to date, add i to your krunge.zip (exemple)
you need online compiler based SC1.0 not older.
http://sc.uvnc.com/cgi-bin/upload2.pl

[redge]

thank you for the update link of keypal

KeyPal: Key Container Utility
http://www.jensign.com/KeyPal/KeyPal.exe

is a .NET 2 CryptoAPI console utility which will be of interest to developers creating and troubleshooting crypto-enabled Windows applications. KeyPal demonstrates several techniques for accessing Windows CryptoAPI functionality and more.

more information detail below.
http://www.jensign.com/KeyPal/index.html

[krunge]

be carefull, you really need at both side the same release of msrc4plugin.dsm, actually is not the case.
SC contain be default 1.1.8.0 and not compatible with the new uvnc 1.0.5 have msrc4plugin.dsm 1.2.2.0

for add you own msrc4plugin.dsm up to date, add i to your krunge.zip (exemple)
you need online compiler based SC1.0 not older.
http://sc.uvnc.com/cgi-bin/upload2.pl

Ah, thanks for the info. Use of msrc4 1.1.8.0 explains why SC sends no random salt or initialization vector at the start (not a good practice, IMHO)

I put an option in SSVNC for a SC workaround for people who build their exe using the default creator (as I did). So SSVNC will work with both plugins if the user is alert enough to toggle the option correctly.