Traffic that uses the AH protocol cannot be masqueraded. The AH protocol incorporates a cryptographic checksum across the IP addresses that the masquerade gateway cannot correctly regenerate. Thus, all masqueraded AH traffic will be discarded as having invalid checksums.
IPsec traffic using transport-mode ESP also cannot be reliably masqueraded. Transport mode ESP essentially encrypts everything after the IP header. Since, for example, the TCP and UDP checksums include the IP source and destination addresses, and the TCP/UDP checksum is within the encrypted payload and thus cannot be recalculated after the masquerade gateway alters the IP addresses, the TCP/UDP header will fail the checksum test at the remote gateway and the packet will be discarded. Protocols that do not include information about the source or destination IP addresses may successfully use masqueraded transport mode.
Apart from these limitations, IPsec masquerade is secure and reliable when only one IPsec host is being masqueraded at a time, or when each masqueraded host is communicating with a different remote host. When more than one masqueraded host is communicating with the same remote host, a few weaknesses show up:
If two or more masqueraded hosts are using transport mode to communicate with the same remote host, and the security policy of the remote host permits multiple transport-mode sessions with the same peer, it is possible for sessions to experience collisions. This happens because the IP address of the masquerading gateway will be used to identify the sessions, and any other identifying information cannot be masqueraded because it is within the encrypted portion of the packet.
If the remote host's security policy does not permit multiple transport-mode sessions with the same peer, the situation is even worse: the more-recently-negotiated transport mode session will likely completely take over all of the traffic from the older session, causing the older session to "go dead". While the established sessions from the older transport-mode IPsec session may be quickly reset if the remote host isn't expecting to receive the traffic, at least one packet of information will be sent to the wrong host. This information will probably be discarded by the recipient, but it will still be sent.
Thus, a transport-mode collision may result in leaking of information between the two sessions or termination of one or both sessions. Using IPsec in transport mode via a masquerading gateway is not recommended if there is the possibility that other transport mode IPsec sessions will be attempted via the same masquerading gateway to the same remote IPsec host.
IPsec using tunnel mode with extruded network addressing (where the masqueraded IPsec host is assigned an IP address from the remote host's network) is not subject to these problems, as the IP addresses assigned from the remote network will be used to identify the sessions instead of using the IP address of the masquerading host.
If two or more masqueraded hosts establishing a session to the same remote host happen to select the same initiator cookie when initiating ISAKMP traffic, the masquerading gateway will route all of the ISAKMP traffic to the second host. There is a 1 in 2^64 (i.e. very small) chance of this collision happening for each host, at the time of establishing the initial ISAKMP connection.
Correcting this requires including the responder cookie in the key used to route inbound ISAKMP traffic. This modification is incorporated into IPsec masquerade for the 2.2.x kernel, and the short window between the time the masqueraded host initiates the ISAKMP exchange and the remote host responds is covered by discarding any new ISAKMP traffic that would collide with the current outstanding traffic. This modification will be backported to the 2.0.x code soon.
Two or more masqueraded IPsec hosts communicating with the same remote IPsec host may negotiate to use the same SPI value for inbound traffic. If this happens the masquerading gateway will route all of the inbound traffic to the first host to receive any inbound traffic using that SPI. The possibility of this happening is about 1 in 2^32 for each outstanding ESP session, and may occur on any rekey.
Since the SPI values refer to different SAs having different encryption keys the first host will not be able to decrypt the data intended for the other hosts, so no data leakage will occur. There is no way for the masquerading gateway to detect or prevent this collision. The only way to prevent this collision is for the remote IPsec host to check the SPI value proposed by the masqueraded host to see if that SPI value is already in use by another SA from the same IP address. It is not likely that this will be done, since it imposes more overhead on an already expensive operation (the rekey) to benefit a small percentage of users in case of a relatively rare event.
This is discussed in detail in the next section.
To avoid these problems the 2.2.x code by default prevents the establishment of multiple connections to the same remote host. If the weaknesses exposed by multiple connections to the same remote host are acceptable, you can enable "parallel sessions".
Blocking parallel sessions for security reasons can be annoying: there is no way for the IPsec masquerade code to sniff the session and see when it is terminating, so the masquerade table entries will persist for the IPsec Masq Table Lifetime even if the session terminates immediately after it is established. If parallel sessions are prevented, this means that the server will be unavailable to other clients until the masq table entry for the most recent session has timed out and been deleted. This can be up to several hours.
The portion of the ISAKMP key exchange where the ESP SPI values are communicated is encrypted, so the ESP SPI values must be determined by inspection of the actual ESP traffic. Also, the outbound ESP traffic does not contain any indication of what the inbound SPI will be. This means there is no perfectly reliable way to associate inbound ESP traffic with outbound ESP traffic.
IPsec masq attempts to associate inbound and outbound ESP traffic by serializing initial ESP traffic on a by-remote-host basis. What this means is:
There are several ways this can fail to associate traffic properly:
ip_masq_ipsec.cin 2.2.x) and increasing the INIT lifetime or the number of INIT retries permitted, at the cost of increasing the blocking (and DoS) window.
IPsec Masq Table Lifetimekernel configuration parameter is slightly longer than the rekey interval, which is the longest time any given SPI pair should be used. The problem here is that you may not know all of the rekey intervals if you're masquerading for many remote servers, or some may have their rekey intervals set to unreasonably high values, such as several hours.
The best solution is to have some way to preload the masq table with the properly associated out-SPI/in-SPI pair or some other mapping of remote_host + inbound_SPI to masqueraded_host. This cannot be done by inspecting the ISAKMP key exchange, as it is encrypted. It may be possible to use RSIP (a.k.a. Host-NAT) to communicate with the masqueraded IPsec host and request notification of SPI information once it has been negotiated. This is being investigated. If something is done to implement this it will be done no sooner than the 2.3.x series, as RSIP is a fairly complex client/server NAT protocol.
When an inbound ESP packet with a new SPI is received the masquerading firewall attempts to guess which masqueraded host(s) the unassociated inbound traffic is intended for. If the inbound ESP traffic is not matched to an established session or a pending session initialization, then the packet is sent to the masqueraded host(s) who most recently rekeyed with that remote host. The "incorrect" masqueraded hosts will discard the traffic as being improperly encrypted, and the "correct" host will get its data. When the "correct" host responds, the normal ESP init serialization process occurs.