Remote Access VPN with Linux racoon and MAC-OSX

If you use a Linux based router gateway, and MAC-OSX Mountain Lion, being able to created an IPSEC VPN tunnel back to your home site can be very useful.     The MAC-OSX Lion IPSEC client will use ISAKMP over UDP port 500 to negotiate the appropriate phase one key exchange parameters in order to setup a UDP NAT-Traversal IPSEC tunnel over UDP port 4500 back to your home site.

Here I include a pre-shared key based example configuration of the Linux KAME “racoon” daemon to run as an IPSEC server, and configure the MAC-OSX native IPSEC client to connect to it.   The Linux based server system in this example is Ubuntu 12.04.1 server running on a Soekris NET6501-50.   For more information on what Soekris has to offer, visit the web URL http://www.soekris.com/.  

Under Ubuntu, you will need to install two different packages in order to get started.

# apt-get install ipsec-tools
# apt-get install racoon

For the remainder of this example, I will assume that your Ubuntu Linux based system has a public IP address of 240.9.9.9, and that your desired VPN address range is 10.222.1.0/24.   I will also assume that your router gateway is properly configured for Network Address Translation (NAT) using iptables for any address that is part of your internal network which I will consider as anything in the 10.0.0.0/8 address range.   I will also assume that you are running your own internal network DNS server at 10.1.1.1.   Proper configuration of iptables is not included in this blog entry.

Public network address: 240.9.9.9
Internal LAN Network:   10.0.0.0/8
VPN network pool:       10.222.1.0/24
DNS Server:             10.1.1.1
DNS domain:             “domain.tld”

After you have installed the “racoon” package, the configuration file should be located as the file path /etc/racoon/racoon.conf.

We will start with a fully commented racoon.conf example based on the above information in order to illustrate how to configure an IPSEC VPN.  This configuration is based on a pre-shared key rather than certificate based VPN for simplicity sake, and due to the additional complexity involved with setting up your own certifying authority, generating, signing, and importing a certificate for use.

Racoon Configuration File

# set syslog level and pre-shared key file

log notify;

path pre_shared_key "/etc/racoon/psk.txt";

listen {

  adminsock disabled;     #do not listen on the admin socket

  isakmp 240.9.9.9 [500]; #address for ISAKMP

  isakmp_natt 240.9.9.9 [4500]; #address for ISAKMP NAT-Traversal

  strict_address;         #strictly bind these addresses

}

remote anonymous {      #anonymous matches ANY ipsec client

  exchange_mode main;   #ISAKMP phase 1 exchange mode

  ph1id 16;             #phase 1 proposal identifier

  proposal_check claim; #claim our own lifetime value

  lifetime time 12 hour;#phase 1 lifetime

  mode_cfg on;          #gather network information through ISAKMP

  generate_policy on;   #generate ipsec policy from initiator SA payload

  nat_traversal on;     #enable use of NAT-Traversal extension

  dpd_delay 3600;       #enable dead peer detection and set time at 3600 secs

  proposal {                  #phase 1 proposal

    encryption_algorithm aes; #phase 1 encryption algorithm

    hash_algorithm sha1;      #phase 1 hash algorithm

    authentication_method xauth_psk_server; #use xauth pre-shared key method

    dh_group 2;               #use diffie-hellman group 2 (modp1024)

  }

}

# specific mode configuration

mode_cfg {

  auth_source system;         #user auth source (system=Unix user)

  group_source system;        #group validation source (system=Unix groups)

  conf_source local;          #user local pool information below

  network4 10.222.1.50;       #base/first address in VPN pool

  netmask4 255.255.255.0;     #VPN pool network mask

  pool_size 50;               #VPN pool size

  dns4 10.1.1.1;              #VPN pool DNS server

  default_domain "domain.tld";#optional VPN pool domain suffix

  banner "/etc/racoon/motd";  #optional VPN pool message of the day

}

# security association info

sainfo anonymous {                  #anonymous matches any/all SA

  encryption_algorithm aes;         #phase 2 encryption algorithm(s)

  authentication_algorithm hmac_sha1; #phase 2 authentication hash

  compression_algorithm deflate;    #phase 2 compression

  remoteid 16;                      #phase 2 remoteid to match phase 1

}

Linux Server Pre-Shared Key File

Although the /etc/racoon/psk.txt file would typically contain entries listing individual IP addresses, you can also have wildcard entries.   Naturally when travelling your MAC-OSX client is going to have a different public IP address depending on your location, and thus a wildcard pre-shared key file on the server end of things is the easiest solution.   A better solution, as mentioned above, would be to utilize a certificate rather than pre-shared key.

In order to generate a pre-shared key, I would suggest a relatively long random character string.   This is fairly easy to generate using a combination of “dd” and “base64” in the UNIX world, although other options exist.

$ dd if=/dev/urandom bs=1 count=18 2>/dev/null | base64
mylongrandomstring

Within your /etc/racoon/psk.txt pre-shared key file on the UNIX/Linux server, you should list one entry as follows:

# pre-shared key for IPSEC VPN clients

* mylongrandomstring

Note that the string “mylongrandomstring” would actually be random characters you generated from the above command.

MAC-OSX Mountain Lion:  Cisco IPSEC VPN Client

To setup your MAC-OSX IPSEC client, you need to open Network Preferences, click on the “Lock” to make changes, and then click on the small “+” at the bottom left of the dialog to ADD a new interface.  

Set the interface type to “VPN”, and VPN Type to “Cisco IPSec”, and then type in a descriptive service name.

Click on your new IPSEC VPN connection, and enter the appropriate address or domain name of your remote server, as well as your UNIX/Linux username that you will use to connect.

Next, click on the “Authentication Setttings” button and set the “Shared Secret” to the same long random string you used for the pre-shared key on the server.  Leave the “Group Name” blank, and click OK.

Testing The Configuration

If you use the "strict_address" configuration in the "listen" section of the racoon configuration, you can only test from outside your home network.   However, if we assume that your home Linux router gateway also has a second interface for "internal" network traffic, the entire listen section of the racoon.conf file can be commented out during testing to make racoon listen on all interfaces as follows.

#listen {

#  adminsock disabled;     #do not listen on the admin socket

#  isakmp 240.9.9.9 [500]; #address for ISAKMP

#  isakmp_natt 240.9.9.9 [4500]; #address for ISAKMP NAT-Traversal

#  strict_address;         #strictly bind these addresses

#}

For testing purposes, you should use either a console on your linux server, or ssh in from another machine, and then run racoon in debugging mode from the command line as root.

# service racoon stop
# /usr/sbin/racoon -F -d

The "-F" flag instructs racoon to log all output to stdout/screen.    The more "-d" flags you add to the command line, the more debugging output you should received.   After starting racoon on the command line, you should attempt to connect from your MAC-OSX system.

Assuming that your group pre-shared key matches, if you get through IPSEC key management negotiation phase 1, your MAC-OSX system should prompt you for a username and password.   This username has to be a UNIX/Linux based username that has been added to the server system.   If successful, you should see your "banner" message of the day displayed, and receive a VPN pool IP address in the 10.222.1.0/24 network.   You can then put racoon back into normal running mode, and you have successfully configured a remote access VPN.


# /usr/sbin/racoon -F -d

# CTRL-c

# service racoon start

Good luck, and please post comments/questions on your experience.