Chapter 4. Configuring a VPN with IPsec

In Red Hat Enterprise Linux 9, a virtual private network (VPN) can be configured using the IPsec protocol, which is supported by the Libreswan application.

4.1. Libreswan as an IPsec VPN implementation

In RHEL, a Virtual Private Network (VPN) can be configured using the IPsec protocol, which is supported by the Libreswan application. Libreswan is a continuation of the Openswan application, and many examples from the Openswan documentation are interchangeable with Libreswan.

The IPsec protocol for a VPN is configured using the Internet Key Exchange (IKE) protocol. The terms IPsec and IKE are used interchangeably. An IPsec VPN is also called an IKE VPN, IKEv2 VPN, XAUTH VPN, Cisco VPN or IKE/IPsec VPN. A variant of an IPsec VPN that also uses the Level 2 Tunneling Protocol (L2TP) is usually called an L2TP/IPsec VPN, which requires the Optional channel xl2tpd application.

Libreswan is an open-source, user-space IKE implementation. IKE v1 and v2 are implemented as a user-level daemon. The IKE protocol is also encrypted. The IPsec protocol is implemented by the Linux kernel, and Libreswan configures the kernel to add and remove VPN tunnel configurations.

The IKE protocol uses UDP port 500 and 4500. The IPsec protocol consists of two protocols:

  • Encapsulated Security Payload (ESP), which has protocol number 50.
  • Authenticated Header (AH), which has protocol number 51.

The AH protocol is not recommended for use. Users of AH are recommended to migrate to ESP with null encryption.

The IPsec protocol provides two modes of operation:

  • Tunnel Mode (the default)
  • Transport Mode.

You can configure the kernel with IPsec without IKE. This is called Manual Keying. You can also configure manual keying using the ip xfrm commands, however, this is strongly discouraged for security reasons. Libreswan interfaces with the Linux kernel using netlink. Packet encryption and decryption happen in the Linux kernel.

Libreswan uses the Network Security Services (NSS) cryptographic library. Both Libreswan and NSS are certified for use with the Federal Information Processing Standard (FIPS) Publication 140-2.

Important

IKE/IPsec VPNs, implemented by Libreswan and the Linux kernel, is the only VPN technology recommended for use in RHEL. Do not use any other VPN technology without understanding the risks of doing so.

In RHEL, Libreswan follows system-wide cryptographic policies by default. This ensures that Libreswan uses secure settings for current threat models including IKEv2 as a default protocol. See Using system-wide crypto policies for more information.

Libreswan does not use the terms "source" and "destination" or "server" and "client" because IKE/IPsec are peer to peer protocols. Instead, it uses the terms "left" and "right" to refer to end points (the hosts). This also allows you to use the same configuration on both end points in most cases. However, administrators usually choose to always use "left" for the local host and "right" for the remote host.

4.2. Installing Libreswan

This procedure describes the steps for installing and starting the Libreswan IPsec/IKE VPN implementation.

Prerequisites

  • The AppStream repository is enabled.

Procedure

  1. Install the libreswan packages:

    # yum install libreswan
  2. If you are re-installing Libreswan, remove its old database files and create a new database:

    # systemctl stop ipsec
    # rm /var/lib/ipsec/nss/*db
    # ipsec initnss
  3. Start the ipsec service, and enable the service to be started automatically on boot:

    # systemctl enable ipsec --now
  4. Configure the firewall to allow 500 and 4500/UDP ports for the IKE, ESP, and AH protocols by adding the ipsec service:

    # firewall-cmd --add-service="ipsec"
    # firewall-cmd --runtime-to-permanent

4.3. Creating a host-to-host VPN

To configure Libreswan to create a host-to-host IPsec VPN between two hosts referred to as left and right, enter the following commands on both of the hosts:

Prerequisites

  • Libreswan is installed and the ipsec service is started on each node.

Procedure

  1. Generate an RSA key pair on each host:

    # ipsec newhostkey
  2. The previous step returned the generated key’s ckaid. Use that ckaid with the following command on left, for example:

    # ipsec showhostkey --left --ckaid 2d3ea57b61c9419dfd6cf43a1eb6cb306c0e857d

    The output of the previous command generated the leftrsasigkey= line required for the configuration. Do the same on the second host (right):

    # ipsec showhostkey --right --ckaid a9e1f6ce9ecd3608c24e8f701318383f41798f03
  3. In the /etc/ipsec.d/ directory, create a new my_host-to-host.conf file. Write the RSA host keys from the output of the ipsec showhostkey commands in the previous step to the new file. For example:

    conn mytunnel
        leftid=@west
        left=192.1.2.23
        leftrsasigkey=0sAQOrlo+hOafUZDlCQmXFrje/oZm [...] W2n417C/4urYHQkCvuIQ==
        rightid=@east
        right=192.1.2.45
        rightrsasigkey=0sAQO3fwC6nSSGgt64DWiYZzuHbc4 [...] D/v8t5YTQ==
        authby=rsasig
  4. After importing keys, restart the ipsec service:

    # systemctl restart ipsec
  5. Load the connection:

    # ipsec auto --add mytunnel
  6. Establish the tunnel:

    # ipsec auto --up mytunnel
  7. To automatically start the tunnel when the ipsec service is started, add the following line to the connection definition:

    auto=start

4.4. Configuring a site-to-site VPN

To create a site-to-site IPsec VPN, by joining two networks, an IPsec tunnel between the two hosts, is created. The hosts thus act as the end points, which are configured to permit traffic from one or more subnets to pass through. Therefore you can think of the host as gateways to the remote portion of the network.

The configuration of the site-to-site VPN only differs from the host-to-host VPN in that one or more networks or subnets must be specified in the configuration file.

Prerequisites

Procedure

  1. Copy the file with the configuration of your host-to-host VPN to a new file, for example:

    # cp /etc/ipsec.d/my_host-to-host.conf /etc/ipsec.d/my_site-to-site.conf
  2. Add the subnet configuration to the file created in the previous step, for example:

    conn mysubnet
         also=mytunnel
         leftsubnet=192.0.1.0/24
         rightsubnet=192.0.2.0/24
         auto=start
    
    conn mysubnet6
         also=mytunnel
         leftsubnet=2001:db8:0:1::/64
         rightsubnet=2001:db8:0:2::/64
         auto=start
    
    # the following part of the configuration file is the same for both host-to-host and site-to-site connections:
    
    conn mytunnel
        leftid=@west
        left=192.1.2.23
        leftrsasigkey=0sAQOrlo+hOafUZDlCQmXFrje/oZm [...] W2n417C/4urYHQkCvuIQ==
        rightid=@east
        right=192.1.2.45
        rightrsasigkey=0sAQO3fwC6nSSGgt64DWiYZzuHbc4 [...] D/v8t5YTQ==
        authby=rsasig

4.5. Configuring a remote access VPN

Road warriors are traveling users with mobile clients with a dynamically assigned IP address, such as laptops. The mobile clients authenticate using certificates.

The following example shows configuration for IKEv2, and it avoids using the IKEv1 XAUTH protocol.

On the server:

conn roadwarriors
    ikev2=insist
    # Support (roaming) MOBIKE clients (RFC 4555)
    mobike=yes
    fragmentation=yes
    left=1.2.3.4
    # if access to the LAN is given, enable this, otherwise use 0.0.0.0/0
    # leftsubnet=10.10.0.0/16
    leftsubnet=0.0.0.0/0
    leftcert=gw.example.com
    leftid=%fromcert
    leftxauthserver=yes
    leftmodecfgserver=yes
    right=%any
    # trust our own Certificate Agency
    rightca=%same
    # pick an IP address pool to assign to remote users
    # 100.64.0.0/16 prevents RFC1918 clashes when remote users are behind NAT
    rightaddresspool=100.64.13.100-100.64.13.254
    # if you want remote clients to use some local DNS zones and servers
    modecfgdns="1.2.3.4, 5.6.7.8"
    modecfgdomains="internal.company.com, corp"
    rightxauthclient=yes
    rightmodecfgclient=yes
    authby=rsasig
    # optionally, run the client X.509 ID through pam to allow/deny client
    # pam-authorize=yes
    # load connection, don't initiate
    auto=add
    # kill vanished roadwarriors
    dpddelay=1m
    dpdtimeout=5m
    dpdaction=clear

On the mobile client, the road warrior’s device, use a slight variation of the previous configuration:

conn to-vpn-server
    ikev2=insist
    # pick up our dynamic IP
    left=%defaultroute
    leftsubnet=0.0.0.0/0
    leftcert=myname.example.com
    leftid=%fromcert
    leftmodecfgclient=yes
    # right can also be a DNS hostname
    right=1.2.3.4
    # if access to the remote LAN is required, enable this, otherwise use 0.0.0.0/0
    # rightsubnet=10.10.0.0/16
    rightsubnet=0.0.0.0/0
    fragmentation=yes
    # trust our own Certificate Agency
    rightca=%same
    authby=rsasig
    # allow narrowing to the server’s suggested assigned IP and remote subnet
    narrowing=yes
    # Support (roaming) MOBIKE clients (RFC 4555)
    mobike=yes
    # Initiate connection
    auto=start

4.6. Configuring a mesh VPN

A mesh VPN network, which is also known as an any-to-any VPN, is a network where all nodes communicate using IPsec. The configuration allows for exceptions for nodes that cannot use IPsec. The mesh VPN network can be configured in two ways:

  • To require IPsec.
  • To prefer IPsec but allow a fallback to clear-text communication.

Authentication between the nodes can be based on X.509 certificates or on DNS Security Extensions (DNSSEC).

The following procedure uses X.509 certificates. These certificates can be generated using any kind of Certificate Authority (CA) management system, such as the Dogtag Certificate System. Dogtag assumes that the certificates for each node are available in the PKCS #12 format (.p12 files), which contain the private key, the node certificate, and the Root CA certificate used to validate other nodes' X.509 certificates.

Each node has an identical configuration with the exception of its X.509 certificate. This allows for adding new nodes without reconfiguring any of the existing nodes in the network. The PKCS #12 files require a "friendly name", for which we use the name "node" so that the configuration files referencing the friendly name can be identical for all nodes.

Prerequisites

  • Libreswan is installed, and the ipsec service is started on each node.

Procedure

  1. On each node, import PKCS #12 files. This step requires the password used to generate the PKCS #12 files:

    # ipsec import nodeXXX.p12
  2. Create the following three connection definitions for the IPsec required (private), IPsec optional (private-or-clear), and No IPsec (clear) profiles:

    # cat /etc/ipsec.d/mesh.conf
    conn clear
    	auto=ondemand
    	type=passthrough
    	authby=never
    	left=%defaultroute
    	right=%group
    
    conn private
    	auto=ondemand
    	type=transport
    	authby=rsasig
    	failureshunt=drop
    	negotiationshunt=drop
    	# left
    	left=%defaultroute
    	leftcert=nodeXXXX
    	leftid=%fromcert
            leftrsasigkey=%cert
    	# right
    	rightrsasigkey=%cert
    	rightid=%fromcert
    	right=%opportunisticgroup
    
    conn private-or-clear
    	auto=ondemand
    	type=transport
    	authby=rsasig
    	failureshunt=passthrough
    	negotiationshunt=passthrough
    	# left
    	left=%defaultroute
    	leftcert=nodeXXXX
    	leftid=%fromcert
            leftrsasigkey=%cert
    	# right
    	rightrsasigkey=%cert
    	rightid=%fromcert
    	right=%opportunisticgroup
  3. Add the IP address of the network in the proper category. For example, if all nodes reside in the 10.15.0.0/16 network, and all nodes should mandate IPsec encryption:

    # echo "10.15.0.0/16" >> /etc/ipsec.d/policies/private
  4. To allow certain nodes, for example, 10.15.34.0/24, to work with and without IPsec, add those nodes to the private-or-clear group using:

    # echo "10.15.34.0/24" >> /etc/ipsec.d/policies/private-or-clear
  5. To define a host, for example, 10.15.1.2, that is not capable of IPsec into the clear group, use:

    # echo "10.15.1.2/32" >> /etc/ipsec.d/policies/clear

    The files in the /etc/ipsec.d/policies directory can be created from a template for each new node, or can be provisioned using Puppet or Ansible.

    Note that every node has the same list of exceptions or different traffic flow expectations. Two nodes, therefore, might not be able to communicate because one requires IPsec and the other cannot use IPsec.

  6. Restart the node to add it to the configured mesh:

    # systemctl restart ipsec
  7. Once you finish with the addition of nodes, a ping command is sufficient to open an IPsec tunnel. To see which tunnels a node has opened:

    # ipsec trafficstatus

4.7. Methods of authentication used in Libreswan

You can use the following methods for authentication of end points:

  • Pre-Shared Keys (PSK) is the simplest authentication method. PSKs should consist of random characters and have a length of at least 20 characters. In FIPS mode, PSKs need to comply to a minimum strength requirement depending on the integrity algorithm used. It is recommended not to use PSKs shorter than 64 random characters.
  • Raw RSA keys are commonly used for static host-to-host or subnet-to-subnet IPsec configurations. The hosts are manually configured with each other’s public RSA key. This method does not scale well when dozens or more hosts all need to setup IPsec tunnels to each other.
  • X.509 certificates are commonly used for large-scale deployments where there are many hosts that need to connect to a common IPsec gateway. A central certificate authority (CA) is used to sign RSA certificates for hosts or users. This central CA is responsible for relaying trust, including the revocations of individual hosts or users.
  • NULL authentication is used to gain mesh encryption without authentication. It protects against passive attacks but does not protect against active attacks. However, since IKEv2 allows asymmetrical authentication methods, NULL authentication can also be used for internet scale opportunistic IPsec, where clients authenticate the server, but servers do not authenticate the client. This model is similar to secure websites using TLS.

Protection against quantum computers

In addition to these authentication methods, you can use the Postquantum Preshared Keys (PPK) method to protect against possible attacks by quantum computers. Individual clients or groups of clients can use their own PPK by specifying a (PPKID) that corresponds to an out-of-band configured PreShared Key.

Using IKEv1 with PreShared Keys provided protection against quantum attackers. The redesign of IKEv2 does not offer this protection natively. Libreswan offers the use of Postquantum Preshared Keys (PPK) to protect IKEv2 connections against quantum attacks.

To enable optional PPK support, add ppk=yes to the connection definition. To require PPK, add ppk=insist. Then, each client can be given a PPK ID with a secret value that is communicated out-of-band (and preferably quantum safe). The PPK’s should be very strong in randomness and not be based on dictionary words. The PPK ID and PPK data itself are stored in ipsec.secrets, for example:

@west @east : PPKS "user1" "thestringismeanttobearandomstr"

The PPKS option refers to static PPKs. An experimental function uses one-time-pad based Dynamic PPKs. Upon each connection, a new part of a one-time pad is used as the PPK. When used, that part of the dynamic PPK inside the file is overwritten with zeroes to prevent re-use. If there is no more one-time-pad material left, the connection fails. See the ipsec.secrets(5) man page for more information.

Warning

The implementation of dynamic PPKs is provided as a Technology Preview, and this functionality should be used with caution.

4.8. Deploying a FIPS-compliant IPsec VPN

Use this procedure to deploy a FIPS-compliant IPsec VPN solution based on Libreswan. The following steps also enable you to identify which cryptographic algorithms are available and which are disabled for Libreswan in FIPS mode.

Prerequisites

  • The AppStream repository is enabled.

Procedure

  1. Install the libreswan packages:

    # yum install libreswan
  2. If you are re-installing Libreswan, remove its old NSS database:

    # systemctl stop ipsec
    
    # rm /var/lib/ipsec/nss/*db
  3. Start the ipsec service, and enable the service to be started automatically on boot:

    # systemctl enable ipsec --now
  4. Configure the firewall to allow 500 and 4500/UDP ports for the IKE, ESP, and AH protocols by adding the ipsec service:

    # firewall-cmd --add-service="ipsec"
    # firewall-cmd --runtime-to-permanent
  5. Switch the system to FIPS mode:

    # fips-mode-setup --enable
  6. Restart your system to allow the kernel to switch to FIPS mode:

    # reboot

Verification

  1. To confirm Libreswan is running in FIPS mode:

    # ipsec whack --fipsstatus
    000 FIPS mode enabled
  2. Alternatively, check entries for the ipsec unit in the systemd journal:

    $ journalctl -u ipsec
    ...
    Jan 22 11:26:50 localhost.localdomain pluto[3076]: FIPS Mode: YES
  3. To see the available algorithms in FIPS mode:

    # ipsec pluto --selftest 2>&1 | head -6
    Initializing NSS using read-write database "sql:/var/lib/ipsec/nss"
    FIPS Mode: YES
    NSS crypto library initialized
    FIPS mode enabled for pluto daemon
    NSS library is running in FIPS mode
    FIPS HMAC integrity support [disabled]
  4. To query disabled algorithms in FIPS mode:

    # ipsec pluto --selftest 2>&1 | grep disabled
    Encryption algorithm CAMELLIA_CTR disabled; not FIPS compliant
    Encryption algorithm CAMELLIA_CBC disabled; not FIPS compliant
    Encryption algorithm NULL disabled; not FIPS compliant
    Encryption algorithm CHACHA20_POLY1305 disabled; not FIPS compliant
    Hash algorithm MD5 disabled; not FIPS compliant
    PRF algorithm HMAC_MD5 disabled; not FIPS compliant
    PRF algorithm AES_XCBC disabled; not FIPS compliant
    Integrity algorithm HMAC_MD5_96 disabled; not FIPS compliant
    Integrity algorithm HMAC_SHA2_256_TRUNCBUG disabled; not FIPS compliant
    Integrity algorithm AES_XCBC_96 disabled; not FIPS compliant
    DH algorithm MODP1536 disabled; not FIPS compliant
    DH algorithm DH31 disabled; not FIPS compliant
  5. To list all allowed algorithms and ciphers in FIPS mode:

    # ipsec pluto --selftest 2>&1 | grep ESP | grep FIPS | sed "s/^.*FIPS//"
    aes_ccm, aes_ccm_c
    aes_ccm_b
    aes_ccm_a
    NSS(CBC)  3des
    NSS(GCM)  aes_gcm, aes_gcm_c
    NSS(GCM)  aes_gcm_b
    NSS(GCM)  aes_gcm_a
    NSS(CTR)  aesctr
    NSS(CBC)  aes
    aes_gmac
    NSS       sha, sha1, sha1_96, hmac_sha1
    NSS       sha512, sha2_512, sha2_512_256, hmac_sha2_512
    NSS       sha384, sha2_384, sha2_384_192, hmac_sha2_384
    NSS       sha2, sha256, sha2_256, sha2_256_128, hmac_sha2_256
    aes_cmac
    null
    NSS(MODP) null, dh0
    NSS(MODP) dh14
    NSS(MODP) dh15
    NSS(MODP) dh16
    NSS(MODP) dh17
    NSS(MODP) dh18
    NSS(ECP)  ecp_256, ecp256
    NSS(ECP)  ecp_384, ecp384
    NSS(ECP)  ecp_521, ecp521

4.9. Protecting the IPsec NSS database by a password

By default, the IPsec service creates its Network Security Services (NSS) database with an empty password during the first start. Add password protection by using the following steps.

Prerequisites

  • The /var/lib/ipsec/nss/ directory contains NSS database files.

Procedure

  1. Enable password protection for the NSS database for Libreswan:

    # certutil -N -d sql:/var/lib/ipsec/nss
    Enter Password or Pin for "NSS Certificate DB":
    Enter a password which will be used to encrypt your keys.
    The password should be at least 8 characters long,
    and should contain at least one non-alphabetic character.
    
    Enter new password:
  2. Create the /etc/ipsec.d/nsspassword file containing the password you have set in the previous step, for example:

    # cat /etc/ipsec.d/nsspassword
    NSS Certificate DB:MyStrongPasswordHere

    Note that the nsspassword file use the following syntax:

    token_1_name:the_password
    token_2_name:the_password

    The default NSS software token is NSS Certificate DB. If your system is running in FIPS mode, the name of the token is NSS FIPS 140-2 Certificate DB.

  3. Depending on your scenario, either start or restart the ipsec service after you finish the nsspassword file:

    # systemctl restart ipsec

Verification

  1. Check that the ipsec service is running after you have added a non-empty password to its NSS database:

    # systemctl status ipsec
    ● ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec
       Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disable>
       Active: active (running)...
  2. Optionally, check that the Journal log contains entries confirming a successful initialization:

    # journalctl -u ipsec
    ...
    pluto[6214]: Initializing NSS using read-write database "sql:/var/lib/ipsec/nss"
    pluto[6214]: NSS Password from file "/etc/ipsec.d/nsspassword" for token "NSS Certificate DB" with length 20 passed to NSS
    pluto[6214]: NSS crypto library initialized
    ...

Additional resources

4.10. Configuring an IPsec VPN to use TCP

Libreswan supports TCP encapsulation of IKE and IPsec packets as described in RFC 8229. With this feature, you can establish IPsec VPNs on networks that prevent traffic transmitted via UDP and Encapsulating Security Payload (ESP). You can configure VPN servers and clients to use TCP either as a fallback or as the main VPN transport protocol. Because TCP encapsulation has bigger performance costs, use TCP as the main VPN protocol only if UDP is permanently blocked in your scenario.

Prerequisites

Procedure

  1. Add the following option to the /etc/ipsec.conf file in the config setup section:

    listen-tcp=yes
  2. To use TCP encapsulation as a fallback option when the first attempt over UDP fails, add the following two options to the client’s connection definition:

    enable-tcp=fallback
    tcp-remoteport=4500

    Alternatively, if you know that UDP is permanently blocked, use the following options in the client’s connection configuration:

    enable-tcp=yes
    tcp-remoteport=4500

4.11. Configuring ESP hardware offload on a bond to accelerate an IPsec connection

Offloading Encapsulating Security Payload (ESP) to the hardware accelerates IPsec connections. If you use a network bond for fail-over reasons, the requirements and the procedure to configure ESP hardware offload are different from those using a regular Ethernet device. For example, in this scenario, you enable the offload support on the bond, and the kernel applies the settings to the ports of the bond.

Prerequisites

  • All network cards in the bond support ESP hardware offload.
  • The network driver supports ESP hardware offload on a bond device. In RHEL, only the ixgbe driver supports this feature.
  • The bond is configured and works.
  • The bond uses the active-backup mode. The bonding driver does not support any other modes for this feature.
  • The IPsec connection is configured and works.

Procedure

  1. Enable ESP hardware offload support on the network bond:

    # nmcli connection modify bond0 ethtool.feature-esp-hw-offload on

    This command enables ESP hardware offload support on the bond0 connection.

  2. Reactivate the bond0 connection:

    # nmcli connection up bond0
  3. Edit the Libreswan configuration file in the /etc/ipsec.d/ directory of the connection that should use ESP hardware offload, and append the nic-offload=yes statement to the connection entry:

    conn example
        ...
        nic-offload=yes
  4. Restart the ipsec service:

    # systemctl restart ipsec

Verification

  1. Display the active port of the bond:

    # grep "Currently Active Slave" /proc/net/bonding/bond0
    Currently Active Slave: enp1s0
  2. Display the tx_ipsec and rx_ipsec counters of the active port:

    # ethtool enp1s0 | egrep "_ipsec"
         tx_ipsec: 10
         rx_ipsec: 10
  3. Send traffic through the IPsec tunnel. For example, ping a remote IP address:

    # ping -c 5 remote_ip_address
  4. Display the tx_ipsec and rx_ipsec counters of the active port again:

    # ethtool enp1s0 | egrep "_ipsec"
         tx_ipsec: 15
         rx_ipsec: 15

    If the counter values have increased, ESP hardware offload works.

Additional resources

4.12. Configuring IPsec connections that opt out of the system-wide crypto policies

Overriding system-wide crypto-policies for a connection

The RHEL system-wide cryptographic policies create a special connection called %default. This connection contains the default values for the ikev2, esp, and ike options. However, you can override the default values by specifying the mentioned option in the connection configuration file.

For example, the following configuration allows connections that use IKEv1 with AES and SHA-1 or SHA-2, and IPsec (ESP) with either AES-GCM or AES-CBC:

conn MyExample
  ...
  ikev2=never
  ike=aes-sha2,aes-sha1;modp2048
  esp=aes_gcm,aes-sha2,aes-sha1
  ...

Note that AES-GCM is available for IPsec (ESP) and for IKEv2, but not for IKEv1.

Disabling system-wide crypto policies for all connections

To disable system-wide crypto policies for all IPsec connections, comment out the following line in the /etc/ipsec.conf file:

include /etc/crypto-policies/back-ends/libreswan.config

Then add the ikev2=never option to your connection configuration file.

4.13. Troubleshooting IPsec VPN configurations

Problems related to IPsec VPN configurations most commonly occur due to several main reasons. If you are encountering such problems, you can check if the cause of the problem corresponds to any of the following scenarios, and apply the corresponding solution.

Basic connection troubleshooting

Most problems with VPN connections occur in new deployments, where administrators configured endpoints with mismatched configuration options. Also, a working configuration can suddenly stop working, often due to newly introduced incompatible values. This could be the result of an administrator changing the configuration. Alternatively, an administrator may have installed a firmware update or a package update with different default values for certain options, such as encryption algorithms.

To confirm that an IPsec VPN connection is established:

# ipsec trafficstatus
006 #8: "vpn.example.com"[1] 192.0.2.1, type=ESP, add_time=1595296930, inBytes=5999, outBytes=3231, id='@vpn.example.com', lease=100.64.13.5/32

If the output is empty or does not show an entry with the connection name, the tunnel is broken.

To check that the problem is in the connection:

  1. Reload the vpn.example.com connection:

    # ipsec auto --add vpn.example.com
    002 added connection description "vpn.example.com"
  2. Next, initiate the VPN connection:

    # ipsec auto --up vpn.example.com

Firewall-related problems

The most common problem is that a firewall on one of the IPsec endpoints or on a router between the endpoints is dropping all Internet Key Exchange (IKE) packets.

  • For IKEv2, an output similar to the following example indicates a problem with a firewall:

    # ipsec auto --up vpn.example.com
    181 "vpn.example.com"[1] 192.0.2.2 #15: initiating IKEv2 IKE SA
    181 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: sent v2I1, expected v2R1
    010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 0.5 seconds for response
    010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 1 seconds for response
    010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 2 seconds for
    ...
  • For IKEv1, the output of the initiating command looks like:

    # ipsec auto --up vpn.example.com
    002 "vpn.example.com" #9: initiating Main Mode
    102 "vpn.example.com" #9: STATE_MAIN_I1: sent MI1, expecting MR1
    010 "vpn.example.com" #9: STATE_MAIN_I1: retransmission; will wait 0.5 seconds for response
    010 "vpn.example.com" #9: STATE_MAIN_I1: retransmission; will wait 1 seconds for response
    010 "vpn.example.com" #9: STATE_MAIN_I1: retransmission; will wait 2 seconds for response
    ...

Because the IKE protocol, which is used to set up IPsec, is encrypted, you can troubleshoot only a limited subset of problems using the tcpdump tool. If a firewall is dropping IKE or IPsec packets, you can try to find the cause using the tcpdump utility. However, tcpdump cannot diagnose other problems with IPsec VPN connections.

  • To capture the negotiation of the VPN and all encrypted data on the eth0 interface:

    # tcpdump -i eth0 -n -n esp or udp port 500 or udp port 4500 or tcp port 4500

Mismatched algorithms, protocols, and policies

VPN connections require that the endpoints have matching IKE algorithms, IPsec algorithms, and IP address ranges. If a mismatch occurs, the connection fails. If you identify a mismatch by using one of the following methods, fix it by aligning algorithms, protocols, or policies.

  • If the remote endpoint is not running IKE/IPsec, you can see an ICMP packet indicating it. For example:

    # ipsec auto --up vpn.example.com
    ...
    000 "vpn.example.com"[1] 192.0.2.2 #16: ERROR: asynchronous network error report on wlp2s0 (192.0.2.2:500), complainant 198.51.100.1: Connection refused [errno 111, origin ICMP type 3 code 3 (not authenticated)]
    ...
  • Example of mismatched IKE algorithms:

    # ipsec auto --up vpn.example.com
    ...
    003 "vpn.example.com"[1] 193.110.157.148 #3: dropping unexpected IKE_SA_INIT message containing NO_PROPOSAL_CHOSEN notification; message payloads: N; missing payloads: SA,KE,Ni
  • Example of mismatched IPsec algorithms:

    # ipsec auto --up vpn.example.com
    ...
    182 "vpn.example.com"[1] 193.110.157.148 #5: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=AES_GCM_16_256 integ=n/a prf=HMAC_SHA2_256 group=MODP2048}
    002 "vpn.example.com"[1] 193.110.157.148 #6: IKE_AUTH response contained the error notification NO_PROPOSAL_CHOSEN

    A mismatched IKE version could also result in the remote endpoint dropping the request without a response. This looks identical to a firewall dropping all IKE packets.

  • Example of mismatched IP address ranges for IKEv2 (called Traffic Selectors - TS):

    # ipsec auto --up vpn.example.com
    ...
    1v2 "vpn.example.com" #1: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=AES_GCM_16_256 integ=n/a prf=HMAC_SHA2_512 group=MODP2048}
    002 "vpn.example.com" #2: IKE_AUTH response contained the error notification TS_UNACCEPTABLE
  • Example of mismatched IP address ranges for IKEv1:

    # ipsec auto --up vpn.example.com
    ...
    031 "vpn.example.com" #2: STATE_QUICK_I1: 60 second timeout exceeded after 0 retransmits.  No acceptable response to our first Quick Mode message: perhaps peer likes no proposal
  • When using PreSharedKeys (PSK) in IKEv1, if both sides do not put in the same PSK, the entire IKE message becomes unreadable:

    # ipsec auto --up vpn.example.com
    ...
    003 "vpn.example.com" #1: received Hash Payload does not match computed value
    223 "vpn.example.com" #1: sending notification INVALID_HASH_INFORMATION to 192.0.2.23:500
  • In IKEv2, the mismatched-PSK error results in an AUTHENTICATION_FAILED message:

    # ipsec auto --up vpn.example.com
    ...
    002 "vpn.example.com" #1: IKE SA authentication request rejected by peer: AUTHENTICATION_FAILED

Maximum transmission unit

Other than firewalls blocking IKE or IPsec packets, the most common cause of networking problems relates to an increased packet size of encrypted packets. Network hardware fragments packets larger than the maximum transmission unit (MTU), for example, 1500 bytes. Often, the fragments are lost and the packets fail to re-assemble. This leads to intermittent failures, when a ping test, which uses small-sized packets, works but other traffic fails. In this case, you can establish an SSH session but the terminal freezes as soon as you use it, for example, by entering the 'ls -al /usr' command on the remote host.

To work around the problem, reduce MTU size by adding the mtu=1400 option to the tunnel configuration file.

Alternatively, for TCP connections, enable an iptables rule that changes the MSS value:

# iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

If the previous command does not solve the problem in your scenario, directly specify a lower size in the set-mss parameter:

# iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380

Network address translation (NAT)

When an IPsec host also serves as a NAT router, it could accidentally remap packets. The following example configuration demonstrates the problem:

conn myvpn
    left=172.16.0.1
    leftsubnet=10.0.2.0/24
    right=172.16.0.2
    rightsubnet=192.168.0.0/16
…

The system with address 172.16.0.1 have a NAT rule:

iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE

If the system on address 10.0.2.33 sends a packet to 192.168.0.1, then the router translates the source 10.0.2.33 to 172.16.0.1 before it applies the IPsec encryption.

Then, the packet with the source address 10.0.2.33 no longer matches the conn myvpn configuration, and IPsec does not encrypt this packet.

To solve this problem, insert rules that exclude NAT for target IPsec subnet ranges on the router, in this example:

iptables -t nat -I POSTROUTING -s 10.0.2.0/24 -d 192.168.0.0/16 -j RETURN

Kernel IPsec subsystem bugs

The kernel IPsec subsystem might fail, for example, when a bug causes a desynchronizing of the IKE user space and the IPsec kernel. To check for such problems:

$ cat /proc/net/xfrm_stat
XfrmInError                 0
XfrmInBufferError           0
...

Any non-zero value in the output of the previous command indicates a problem. If you encounter this problem, open a new support case, and attach the output of the previous command along with the corresponding IKE logs.

Libreswan logs

Libreswan logs using the syslog protocol by default. You can use the journalctl command to find log entries related to IPsec. Because the corresponding entries to the log are sent by the pluto IKE daemon, search for the “pluto” keyword, for example:

$ journalctl -b | grep pluto

To show a live log for the ipsec service:

$ journalctl -f -u ipsec

If the default level of logging does not reveal your configuration problem, enable debug logs by adding the plutodebug=all option to the config setup section in the /etc/ipsec.conf file.

Note that debug logging produces a lot of entries, and it is possible that either the journald or syslogd service rate-limits the syslog messages. To ensure you have complete logs, redirect the logging to a file. Edit the /etc/ipsec.conf, and add the logfile=/var/log/pluto.log in the config setup section.

Additional resources