Update: UltraVNC 1.4.3.6 and UltraVNC SC 1.4.3.6: viewtopic.php?t=37885
Important: Please update to latest version before to create a reply, a topic or an issue: viewtopic.php?t=37864

Join us on social networks and share our announcements:
- Website: https://uvnc.com/
- GitHub: https://github.com/ultravnc
- Mastodon: https://mastodon.social/@ultravnc
- Facebook: https://www.facebook.com/ultravnc1
- X/Twitter: https://twitter.com/ultravnc1
- Reddit community: https://www.reddit.com/r/ultravnc
- OpenHub: https://openhub.net/p/ultravnc

Help with MSRC4 key format for unix UVNC DSM plugin

Should you have problems with the DSM plugin, here's the place to look for help or report issues
Post Reply
krunge
20
20
Posts: 39
Joined: 2008-09-13 22:30
Contact:

Help with MSRC4 key format for unix UVNC DSM plugin

Post by krunge »

Hello,

I'm working on a Unix helper program for my vncviewer and server projects
(ssvnc and x11vnc) that will allow them to work with UVNC programs using
the DSM encryption plugin (they already work with SSL connections for SC III
and repeater_SSL.exe).

I have it more or less working with the ARC4 and AESV2 plugins (however the
tight and zlib encodings still don't work...). The encryption/decryption
part for these was pretty easy because I use OpenSSL as do the plugins.

For the MSRC4 plugin, I am hoping I can simply use my ARC4 code as long as
I can figure out how to extract the RC4 key 'rc4.key' file (76 bytes in
the key portion of the file) and transform it into 'arc4.key' key format
(16 bytes of data, i.e. a 128 bit key)

Does anyone know where I might find out or start looking for how to convert
between the two formats?
redge
1000
1000
Posts: 6797
Joined: 2004-07-03 17:05
Location: Switzerland - Geneva

Re: Help with MSRC4 key format for unix UVNC DSM plugin

Post by redge »

CryptDeriveKey Function
http://msdn.microsoft.com/en-us/library ... S.85).aspx

RC4 Cipher Usage
http://msdn.microsoft.com/en-us/library/cc207219.aspx

MS CryptoAPI keypal
[post=3373][/post]
UltraVNC 1.0.9.6.1 (built 20110518)
OS Win: xp home + vista business + 7 home
only experienced user, not developer
krunge
20
20
Posts: 39
Joined: 2008-09-13 22:30
Contact:

Re: Help with MSRC4 key format for unix UVNC DSM plugin

Post by krunge »

redge wrote: MS CryptoAPI keypal
[post=3373][/post]
Very nice! That keypal link is broken, but going to its www.jensign.com site reveals the guy there is a super-genius. He has the key file format nicely described (not explicitly for RC4, but it applies). So it is simple to extract the key in C with no extra libraries.

I would have never guessed the 16 byte RC4 key was stored in reverse order!

So now I have the SSVNC Unix viewer working with MSRC4, ARC4, and AESV2 DSM plugins!! It is working with the uvnc 105 server, still some problem with SC -plugin mode, but I will look at that tomorrow.

Thanks very much for the pointers.
redge
1000
1000
Posts: 6797
Joined: 2004-07-03 17:05
Location: Switzerland - Geneva

Re: Help with MSRC4 key format for unix UVNC DSM plugin

Post by redge »

> still some problem with SC -plugin mode

be carefull, you really need at both side the same release of msrc4plugin.dsm, actually is not the case.
SC contain be default 1.1.8.0 and not compatible with the new uvnc 1.0.5 have msrc4plugin.dsm 1.2.2.0

for add you own msrc4plugin.dsm up to date, add i to your krunge.zip (exemple)
you need online compiler based SC1.0 not older.
http://sc.uvnc.com/cgi-bin/upload2.pl
UltraVNC 1.0.9.6.1 (built 20110518)
OS Win: xp home + vista business + 7 home
only experienced user, not developer
redge
1000
1000
Posts: 6797
Joined: 2004-07-03 17:05
Location: Switzerland - Geneva

Re: Help with MSRC4 key format for unix UVNC DSM plugin

Post by redge »

thank you for the update link of keypal :-)

KeyPal: Key Container Utility
http://www.jensign.com/KeyPal/KeyPal.exe

is a .NET 2 CryptoAPI console utility which will be of interest to developers creating and troubleshooting crypto-enabled Windows applications. KeyPal demonstrates several techniques for accessing Windows CryptoAPI functionality and more.

more information detail below.
http://www.jensign.com/KeyPal/index.html
UltraVNC 1.0.9.6.1 (built 20110518)
OS Win: xp home + vista business + 7 home
only experienced user, not developer
krunge
20
20
Posts: 39
Joined: 2008-09-13 22:30
Contact:

Re: Help with MSRC4 key format for unix UVNC DSM plugin

Post by krunge »

redge wrote: be carefull, you really need at both side the same release of msrc4plugin.dsm, actually is not the case.
SC contain be default 1.1.8.0 and not compatible with the new uvnc 1.0.5 have msrc4plugin.dsm 1.2.2.0

for add you own msrc4plugin.dsm up to date, add i to your krunge.zip (exemple)
you need online compiler based SC1.0 not older.
http://sc.uvnc.com/cgi-bin/upload2.pl
Ah, thanks for the info. Use of msrc4 1.1.8.0 explains why SC sends no random salt or initialization vector at the start (not a good practice, IMHO)

I put an option in SSVNC for a SC workaround for people who build their exe using the default creator (as I did). So SSVNC will work with both plugins if the user is alert enough to toggle the option correctly.
Post Reply