Wednesday, January 2, 2013

Remote Access VPN with Linux racoon and MAC-OSX

If you use a Linux based router gateway, and MAC-OSX Mountain Lion, being able to created an IPSEC VPN tunnel back to your home site can be very useful.     The MAC-OSX Lion IPSEC client will use ISAKMP over UDP port 500 to negotiate the appropriate phase one key exchange parameters in order to setup a UDP NAT-Traversal IPSEC tunnel over UDP port 4500 back to your home site.

Here I include a pre-shared key based example configuration of the Linux KAME “racoon” daemon to run as an IPSEC server, and configure the MAC-OSX native IPSEC client to connect to it.   The Linux based server system in this example is Ubuntu 12.04.1 server running on a Soekris NET6501-50.   For more information on what Soekris has to offer, visit the web URL  

Under Ubuntu, you will need to install two different packages in order to get started.

# apt-get install ipsec-tools
# apt-get install racoon

For the remainder of this example, I will assume that your Ubuntu Linux based system has a public IP address of, and that your desired VPN address range is   I will also assume that your router gateway is properly configured for Network Address Translation (NAT) using iptables for any address that is part of your internal network which I will consider as anything in the address range.   I will also assume that you are running your own internal network DNS server at   Proper configuration of iptables is not included in this blog entry.

Public network address:
Internal LAN Network:
VPN network pool:
DNS Server:   
DNS domain:             “domain.tld”

After you have installed the “racoon” package, the configuration file should be located as the file path /etc/racoon/racoon.conf.

We will start with a fully commented racoon.conf example based on the above information in order to illustrate how to configure an IPSEC VPN.  This configuration is based on a pre-shared key rather than certificate based VPN for simplicity sake, and due to the additional complexity involved with setting up your own certifying authority, generating, signing, and importing a certificate for use.

Racoon Configuration File

# set syslog level and pre-shared key file
log notify;
path pre_shared_key "/etc/racoon/psk.txt";

listen {
  adminsock disabled;     #do not listen on the admin socket
  isakmp [500]; #address for ISAKMP
  isakmp_natt [4500]; #address for ISAKMP NAT-Traversal
  strict_address;         #strictly bind these addresses

remote anonymous {      #anonymous matches ANY ipsec client
  exchange_mode main;   #ISAKMP phase 1 exchange mode
  ph1id 16;             #phase 1 proposal identifier
  proposal_check claim; #claim our own lifetime value
  lifetime time 12 hour;#phase 1 lifetime
  mode_cfg on;          #gather network information through ISAKMP
  generate_policy on;   #generate ipsec policy from initiator SA payload
  nat_traversal on;     #enable use of NAT-Traversal extension
  dpd_delay 3600;       #enable dead peer detection and set time at 3600 secs

  proposal {                  #phase 1 proposal
    encryption_algorithm aes; #phase 1 encryption algorithm
    hash_algorithm sha1;      #phase 1 hash algorithm
    authentication_method xauth_psk_server; #use xauth pre-shared key method
    dh_group 2;               #use diffie-hellman group 2 (modp1024)

# specific mode configuration
mode_cfg {
  auth_source system;         #user auth source (system=Unix user)
  group_source system;        #group validation source (system=Unix groups)
  conf_source local;          #user local pool information below
  network4;       #base/first address in VPN pool
  netmask4;     #VPN pool network mask
  pool_size 50;               #VPN pool size
  dns4;              #VPN pool DNS server
  default_domain "domain.tld";#optional VPN pool domain suffix
  banner "/etc/racoon/motd";  #optional VPN pool message of the day

# security association info
sainfo anonymous {                  #anonymous matches any/all SA
  encryption_algorithm aes;         #phase 2 encryption algorithm(s)
  authentication_algorithm hmac_sha1; #phase 2 authentication hash
  compression_algorithm deflate;    #phase 2 compression
  remoteid 16;                      #phase 2 remoteid to match phase 1

Linux Server Pre-Shared Key File

Although the /etc/racoon/psk.txt file would typically contain entries listing individual IP addresses, you can also have wildcard entries.   Naturally when travelling your MAC-OSX client is going to have a different public IP address depending on your location, and thus a wildcard pre-shared key file on the server end of things is the easiest solution.   A better solution, as mentioned above, would be to utilize a certificate rather than pre-shared key.

In order to generate a pre-shared key, I would suggest a relatively long random character string.   This is fairly easy to generate using a combination of “dd” and “base64” in the UNIX world, although other options exist.

$ dd if=/dev/urandom bs=1 count=18 2>/dev/null | base64

Within your /etc/racoon/psk.txt pre-shared key file on the UNIX/Linux server, you should list one entry as follows:

# pre-shared key for IPSEC VPN clients
* mylongrandomstring

Note that the string “mylongrandomstring” would actually be random characters you generated from the above command.

MAC-OSX Mountain Lion:  Cisco IPSEC VPN Client

To setup your MAC-OSX IPSEC client, you need to open Network Preferences, click on the “Lock” to make changes, and then click on the small “+” at the bottom left of the dialog to ADD a new interface.  

Set the interface type to “VPN”, and VPN Type to “Cisco IPSec”, and then type in a descriptive service name.

Click on your new IPSEC VPN connection, and enter the appropriate address or domain name of your remote server, as well as your UNIX/Linux username that you will use to connect.

Next, click on the “Authentication Setttings” button and set the “Shared Secret” to the same long random string you used for the pre-shared key on the server.  Leave the “Group Name” blank, and click OK.

Testing The Configuration

If you use the "strict_address" configuration in the "listen" section of the racoon configuration, you can only test from outside your home network.   However, if we assume that your home Linux router gateway also has a second interface for "internal" network traffic, the entire listen section of the racoon.conf file can be commented out during testing to make racoon listen on all interfaces as follows.

#listen {
#  adminsock disabled;     #do not listen on the admin socket
#  isakmp [500]; #address for ISAKMP
#  isakmp_natt [4500]; #address for ISAKMP NAT-Traversal
#  strict_address;         #strictly bind these addresses

For testing purposes, you should use either a console on your linux server, or ssh in from another machine, and then run racoon in debugging mode from the command line as root.

# service racoon stop
# /usr/sbin/racoon -F -d

The "-F" flag instructs racoon to log all output to stdout/screen.    The more "-d" flags you add to the command line, the more debugging output you should received.   After starting racoon on the command line, you should attempt to connect from your MAC-OSX system.

Assuming that your group pre-shared key matches, if you get through IPSEC key management negotiation phase 1, your MAC-OSX system should prompt you for a username and password.   This username has to be a UNIX/Linux based username that has been added to the server system.   If successful, you should see your "banner" message of the day displayed, and receive a VPN pool IP address in the network.   You can then put racoon back into normal running mode, and you have successfully configured a remote access VPN.

# /usr/sbin/racoon -F -d
# CTRL-c
# service racoon start

Good luck, and please post comments/questions on your experience.


sharon cullen said...

What are the settings for accessing documents from remote server?
Remote access

raza wala said...

Good post. sharing valuable information about K-Secure VPN Server guide

Muhammad Amir said...

To get access to blocked anonymity websites or blocked content by circumventing geo, IP or other blockades.

Alex Perry said...

Well I think users of Linux and Mac based operating systems will not find any complication to setup VPN service on their server as long as this post stays alive here. I'm running Windows operating system based server and installing VPN on windows is pretty easy as well. Thanks.

Jennifer Heather said...

Very instructive information provided in this blog and I really enjoyed studying this post!! To configure Linux based computer for VPN using purpose I think such instructions will be helpful. Thanks.

Jessica Milly said...

Gigi said, very much educative blog. To be honest once upon a time I used Linux ubuntu and had a lots of problem on there because I didn't know much more about that system properly. But to read out this blog now I am very much capable to use that system and also vpn service as well. So I will think to bring back that system very soon. Thanks

Jilian Moon said...

Your Configuration is so great and I really love to do that in the same way. Your stuff is so educational and have all the best ideas most of the people miss while Configuring the whole setup. The remote access to the VPN with Linux racoon and MAC-OSX is a great way and I really like to follow the same procedure.

Tony Stark said...


Tony Stark said...

Good post. sharing valuable information about K-Secure VPN Server guide To get access to blocked anonymity websites or blocked content by circumventing geo, IP or other blockades. It really worked for me. I checked the change in my IP address by visiting and had an IP lookup.

Willie Ames said...

i always failed in remote accessing by connecting expressvpn, Thanks for your guide.

James Vang said...

I am using MAC and i setup VPN on my MAC with the help of this tutorial. This tutorial is very simple as compare to which i am reading. I suggest to all VPN users who are not tech savvy can use this tutorial for you MAC.

jack lance said...

All VPNs that support OpenVPN (pretty much all of them) will run on all of these platforms. If the VPN provider doesn't have an app of their own for the platform you desire then you can just download an OpenVPN client and configure it for your VPN provider. If you want to make life easy I believe Expressvpn has an app for all of these platforms. But many others will also and there is no need to limit yourself to VPN's with official support for those platforms as long as you know how to Google.

ban gofran said...

Informative. Really informative. Though VPN is now the great innovation of the world. Please release from some fatal problem by using a VPN connection.
Thank you very much for sharing your article with us.

Roman lesnar said...

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. top 10 vpn reviews

Irene Chan said...

SaferVPN supports loads of VPN protocols including PPTP, L2TP/IPsec, SSTP and OpenVPN (UDP and TCP).

James harper said...

Availing VPN service is an important step in improving your internet security. VPN provider