Hi

I am a bit late, but I need to test what I write. And time is precious
In general, I can recommend two readers: on securityfocus about the
windows part[1], Spenneberg about the linux part[2].

Let us not make this more complicated than it needs to be. Two hosts,
isolating setup. We use transport mode with preshared keys as a beginning.


Linux-part (192.168.1.13)


The policy (SPD) is manipulated with setkey. Let's create
in /etc/racoon/setkey.conf
Code:
#!/usr/sbin/setkey -f

flush;
spdflush;

spdadd 192.168.1.13 192.168.1.16 any -P out ipsec
        esp/transport//require
        ah/transport//require; 

spdadd 192.168.1.16 192.168.1.13 any -P in ipsec
        esp/transport//require
        ah/transport//require;
I want to enable a communication .1.13 <-> .1.16, transport,
both encryption and authentication (no replay). Start with
setkey -f /etc/racoon/setkey.conf and check with setkey -DP

Next step: racoon.

We need to specify the kind of encryption. I use pre_shared_key,
3des and md5 - for simplicity.
racoon.conf in /etc/racoon/racoon.conf

Code:
path pre_shared_key "/etc/racoon/psk.txt";
#path certificate "/etc/racoon/certs";

remote 192.168.1.16 {
        exchange_mode main;
        proposal {
                encryption_algorithm 3des;
                hash_algorithm md5;
                authentication_method pre_shared_key;
                dh_group modp1024;
        }
}
However, for phase 2, we need the SA information. Let us not be
specific and use, also for simplicity DH group 2
Code:
sainfo anonymous
{
pfs_group 2;
encryption_algorithm 3des;
authentication_algorithm hmac_md5;
compression_algorithm deflate;
}
Create a file /etc/racoon/psk.txt
Code:
192.168.1.16    PresharedKey

Start racoon with racoon -F -f /etc/racoon/racoon.conf,
use -d for debugging.


Windows-part (192.168.1.16)


MMC->SnapIn->[3]

Create a new policy (no tunnel!), make sure to use the correct
encryptions MD5 and 3DES, and take the correct group: DH2 (Tab: General).
Make a new rule (for simplicity: all traffic, all protocols, ...) between
.1.13 and .1.16. (mirrored).

There is one important point: The Filter Action should have
Session key perfect forward secrecy enabled, otherwise
there is a problem getting phase 2! I would not accept or allow anything
related to unsecured communication.

Authentication: PreSharedKey. Type the same "passphrase" as in
/etc/racoon/psk.txt

Assign. Done.



Now, this is the first step to enable a IPSec transport, which
is fine for your purpose, I think. Check the traffic with a sniffer
and eventually, turn of the PreSharedKey-mode and use a certificate.


Cheers



[1] http://online.securityfocus.com/infocus/1519
http://online.securityfocus.com/infocus/1526
http://online.securityfocus.com/infocus/1528
[2] http://www.ipsec-howto.org/ipsec-howto.pdf
[3] http://www.antionline.com/showthread...hreadid=262801