======================================================================== A Detailed Description of Destination Network Address Translation (DNAT) ======================================================================== (c) 2002-2022 Ian! D. Allen Ottawa, Canada This is a GNU FREE document. See the Copyright info at the end. VERSION #11 September 5, 2022 - added specific info for SmartRG modem at bottom - added note about enabling default ACCEPT policy in the FORWARD chain This document is a detailed description (with examples) of how Linux Destination Network Address Translation (netfilter DNAT) works in a port-forwarding situation. It is based on hints given by Rusty Russell in the NAT-HOWTO.linuxdoc.html. If reading Rusty's explanation isn't quite enough for you, try this one. This document is intentionally repetitive. I restate some things several times, using different language, in the hope that if you didn't get it the first time, perhaps one of the other times will work for you. After all, if you could understand what DNAT was by reading the existing manuals and HOWTOs, you wouldn't be here, would you? -Ian! (Purists will note that netfilter's DNAT doesn't actually "forward" anything; it simply changes the packet address such that the kernel will route it where we want it to go. Still, the commonly used phrase is "port forwarding", and we'll stick with that.) Why port-forwarding and why DNAT? --------------------------------- Most books on security discourage running a service such as a web server (e.g. Apache) directly on an Internet-visible router/firewall/gateway machine. The preferred method is to run the service on a different machine that resides on an internal network hidden behind the router. We set up the router to accept connections made to a certain public port on the router (e.g. port 80) and port-forward (using DNAT) these connections to a port on the hidden server on the internal network. Often, a special dedicated network termed a "DMZ" (De-Militarized Zone) is used to house these internal servers, to isolate them from the other internal network(s). This leads to a "three network" configuration for our router; the router is connected to three different networks: the public Internet, the DMZ network, and the truly private internal network. The router has IP addresses on each of the three networks; however, usually only the Internet network gives the router a "public" IP address visible to the rest of the world, e.g. www.myhost.ca. We would like to arrange that when some client machine, located anywhere (on either of the two local networks or out on the Internet), makes a connection to the web port (80) of our router's public address (e.g. www.myhost.ca), they are transparently redirected to the internal web server machine running on the internal DMZ network, hidden safely behind the router. We want anyone trying to get to port 80 on the router to be redirected to port 80 on this web server machine on the internal DMZ network, and we'd like reply packets from that internal server to flow correctly back to the clients. In addition to having clients on machines on external networks (e.g. the Internet) redirected to our web server, we also want clients on all our internal network(s) to be redirected to the same internal web server, using the same web addresses that our Internet clients use; we don't want our internal clients to have to use a different address from our external clients to connect to the same web server, nor do we wish to do tricks with our DNS so that the internal web server appears to have a different address inside our own private network than it does outside. (Up until Linux kernel 2.4, this internal redirection wasn't easy; now, it is.) We can do all these redirections with netfilter DNAT (Destination Network Address Translation), plus some necessary trickery involving a bit of SNAT (Source Network Address Translation) for internal clients. Note: Some people suggest that the proper way to handle a web server on the *same* internal private network as the client (not recommended!) is to use "split-view DNS" tricks to give the server two different IP addresses, one private IP for the internal network(s) and one public IP for the external network(s). For example, for clients out on the Internet, the public address www.myhost.ca would resolve to be the router's public Internet IP address (which would be translated by the router DNAT to be the internal private IP address of the web server). For clients in the DMZ or on the local private network, the same public www.myhost.ca would (using split-DNS tricks) resolve directly to be an internal private server address, not to the public address that Internet clients would see. Split-DNS tricks are more "efficient", in that local clients on the same private network as the web server don't have to send packets via the router - the packets go directly to the local IP address of the web server. However, any content management, cacheing, logging, security, or filtering you might do on the router won't happen for internal clients, since the router isn't involved as it is with Internet clients. Don't put your public web server on your internal network, if you can avoid it. I find that split-view DNS tricks can cause problems, especially in VPN environments where your machine has cached the external IP address of a server but is now connected to the VPN and must use the *internal* IP address, and vice-versa. I think DNS queries should always return the same IP address, no matter where you are. The explanations in this document assume this - no split-DNS tricks. If you do use split DNS, your internal clients will be able to send packets directly to the web server's local internal address, ignoring the router; you don't need some of the SNAT rules I will describe below. Enable packet forwarding ------------------------ Please make sure that your router is set up to forward packets between network interfaces! The majority of the Linux kernel systems can achieve this by simply echoing a "1" into the ip_forward file under /proc: # cat /proc/sys/net/ipv4/ip_forward 0 # echo 1 > /proc/sys/net/ipv4/ip_forward You may also have to set the default policy for the iptables FORWARD chain to "ACCEPT" instead of "DROP": # iptables --policy FORWARD ACCEPT # iptables --list FORWARD | head -n 1 Chain FORWARD (policy ACCEPT) Without enabling forwarding, the rules given below won't work. Three Network Interface Addresses --------------------------------- Suppose we have a router connected to three networks: an external public network 100.0.0.0/24 on eth1, an internal DMZ network 172.16.0.0/24 on eth2, and an internal trusted network 192.168.0.0/24 on eth3. (I chose these three network numbers and interface names so that you could remember that the higher the number, the more private and more trusted the network. The actual numbers don't matter here, and yours will differ.) In reality, machines don't have IP addresses; their network cards have IP addresses. In fact, you can often assign several addresses to a single network card, so to be perfectly accurate we must say that network *interfaces* have IP addresses. Our router, being connected to three networks, actually has three network interface addresses. For this example, we will number them as follows: - a public external interface address 100.0.0.254 on eth1 (this is the public address a DNS query would return for www.myhost.ca) - an internal DMZ interface address 172.16.0.254 on eth2 (this is not a public address or network) - an internal trusted network interface address 192.168.0.254 on eth3 (this is not a public address or network) Your actual router addresses and interfaces will differ from my example. Only the public external interface 100.0.0.254 is visible on the Internet and in DNS queries for "www.myhost.ca". (We are not using split-DNS, so even inside our private networks www.myhost.ca resolves to 100.0.0.254.) Thus, "port 80 on the router" really means "port 80 on any of its three network interfaces", i.e. port 80 on 100.0.0.254, port 80 on 172.16.0.254, and port 80 on 192.168.0.254. The "public" address for the router (and thus for the web server hidden behind it) will be the address of the interface connected to the Internet: 100.0.0.254. This is the address clients will find when they look up "www.myhost.ca" in the DNS and connect to it. For this document, the location of the client doesn't matter when looking up "www.myhost.ca". I assume the DNS query will always return the one external public IP address 100.0.0.254. We are not operating a "split-view" DNS where a query might return one IP address from the Internet and a different IP address from inside our own network. SNAT or MASQUERADE the internal networks ---------------------------------------- Our three-network router is presumed to already use standard iptables SNAT to allow the two internal networks (172.16.0.0/24 [eth2], 192.168.0.0/24 [eth3]) to masquerade their packets out the single public network interface (100.0.0.0/24 [eth1]). That looks something like this: iptables -t nat -A POSTROUTING -o eth1 -j SNAT ...etc... See elsewhere for how to do standard outgoing SNAT or MASQUERADE using the iptables/netfilter tools: http://www.netfilter.org/unreliable-guides/NAT-HOWTO.txt Forwarding packets to the internal web server using DNAT -------------------------------------------------------- How do we make the router forward client packets arriving at port 80, on any one of its three network interfaces, to the internal web server, and how do we forward the reply packets from the web server back to the client? The first step to make port forwarding work is to arrange that any packets sent from any clients anywhere to "port 80 on the router" (any of the router's three addresses) get forwarded to port 80 on the internal web server. For these examples let's put the internal web server on a machine with IP address 172.16.0.1 on the DMZ network. We'll assume three clients that want to connect to the web server have addresses 123.45.6.7 (on the Internet), 172.16.0.2 (on the DMZ net), and 192.168.0.2 (on the trusted internal net). We want packets incoming from any of these clients to arrive at the router and then get transparently forwarded to the internal web server at 172.16.0.1. Here is the packet destination changing we want the router to do: A client packet arriving at port 80 on the router via the Internet: Arriving: from 123.45.6.7 (client) to 100.0.0.254 (router www.myhost.ca) Change to: from 123.45.6.7 (client) to 172.16.0.1 (webserver) A client packet arriving at port 80 on the router from the DMZ: Arriving: from 172.16.0.2 (client) to 172.16.0.254 (router) Change to: from 172.16.0.2 (client) to 172.16.0.1 (webserver) A client packet arriving at port 80 on the router from the trusted network: Arriving: from 192.168.0.2 (client) to 192.168.0.254 (router) Change to: from 192.168.0.2 (client) to 172.16.0.1 (webserver) Since the router has three network interface addresses, it takes three iptables DNAT rules to rewrite each of the three possible router destination addresses to be the internal web server address 172.16.0.1. Below are the three rules using three iptables DNAT command lines. The only differences among the three occurs in the destination (-d) address in each that makes the rule apply to only that particular router network interface: iptables -t nat -A PREROUTING -d 100.0.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 iptables -t nat -A PREROUTING -d 172.16.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 iptables -t nat -A PREROUTING -d 192.168.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 Once the router has used the above DNAT rule to rewrite the destination address of the arriving client packet, it can forward the packet on to the internal web server at that address. (The original destination address is not lost; it is saved and used to un-DNAT reply packets coming back from the internal web server. More on that un-DNAT process later.) Each DNAT rule above applies only to packets arriving on one of the three router interfaces (-d address). Each rule removes the original destination address in the packet (destined to be one of the three router network interface addresses) and replaces it with the destination address of the internal web server (--to 172.16.0.1) on the DMZ network, then it forwards the packet off to the web server. The original packet had a destination address of one of the three router interfaces; the DNAT packet has a destination of the internal web server. The source address of the forwarded packets remains unchanged. The "port" part of the packet also remains untouched in this example; only the destination IP address is changed by the DNAT rule in the router. We could also route the packet to a different port on the internal web server by modifying the --to argument in each of the the DNAT rules, like this: --to 172.16.0.1:8080 These three DNAT rules cause the router to forward to the internal web server client packets received on any of the three network interfaces: Arriving: from 123.45.6.7 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 123.45.6.7 (client) to 172.16.0.1 (webserver) Arriving: from 172.16.0.2 (client) to 172.16.0.254 (router) DNAT: from 172.16.0.2 (client) to 172.16.0.1 (webserver) Arriving: from 192.168.0.2 (client) to 192.168.0.254 (router) DNAT: from 192.168.0.2 (client) to 172.16.0.1 (webserver) The usual network routing rules apply to the DNAT packets. Since they have a destination address on the DMZ network, the router will use ARP on the DMZ network to find the web server machine on that network, and it will send the DNAT packets to the web server. What about reply packets coming back from the web server? The routing of the reply packets -------------------------------- A client sends a packet to the router's public address; the router applies DNAT and forwards that client packet to the internal web server in the DMZ network. The web server has no way of knowing that DNAT was applied to the packet by the router. The source address of the packet was untouched - it is still the source address of the client desiring the connection to the web server. All the web server knows is that it has received a packet from the client and it now has to reply to the client that sent this incoming packet. The web server will reply to the incoming packet as usual, creating a reply packet with source address being the internal web server (172.16.0.1) and destination address being the client address that sent the original incoming packet. Here are the three possible cases: 1. A DNATed packet arriving on the web server from an Internet client: Arriving: from 123.45.6.7 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 123.45.6.7 (client) 2. A DNATed packet arriving on the web server from a trusted network client: Arriving: from 192.168.0.2 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 192.168.0.2 (client) 3. A DNATed packet arriving on the web server from a DMZ client: Arriving: from 172.16.0.2 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 172.16.0.2 (client) I will assume that the router is the gateway ("default route") for both the trusted network and the DMZ network. Standard Linux routing rules apply to the above reply packets coming from the web server: A) If a packet has a destination address that is *not* on the same network as the web server (not on the DMZ network), the packet is sent via the router, which is the gateway ("default route") of the DMZ network. This applies to cases #1 and #2, above. B) If a packet has a destination address on the *same* network as the web server (on the same DMZ network), the packet goes directly to the destination machine on the same network. This means that the reply packet does *not* go back via the router through which the original packet came! This applies to case #3, above. The reply generated by the web server in the DMZ will in two cases out of three above (to client 123.45.6.7 and to client 192.168.0.2) return via the router at 172.16.0.254, because everything on the DMZ network uses the router as the default gateway. This applies to cases #1 and #2, above. In the third case, where the client that sent the original packet is on the *same* DMZ network as the internal web server (client 172.16.0.2 on the same DMZ network as the web server 172.16.0.1), the reply packet from the web server to the client does *not* go back via the router. Linux routing rule "B", above, dictates that sending to a machine on the *same* network means the packet must go straight to the machine. Unfortunately, the client isn't expecting to receive any packets directly from the web server at 172.16.0.1. The client sent a request to the public address of the web server at www.myhost.ca (100.0.0.254), and that's from where it expects a reply. The client therefore *rejects* the packet arriving directly from the web server. We will discuss how to fix this problem (client and web server on the same internal network) later. Let's examine what the router does in the first two two working cases where the web server reply packets return correctly via the router. Returning reply packets from the internal web server using un-DNAT ------------------------------------------------------------------ In two cases out of three above (excluding the third case where the client is on the same network as the web server), the inbound client packets head toward the public address of the web server at www.myhost.ca, pass through the router, and thus through DNAT translation, arrive at the web server, and generate reply packets that go back via the router. This means that, in two cases out of three, packets coming via the router have replies to clients sent back via the router; this is good. As mentioned above, the third case is a problem; more on that later. The reply packets coming back to the client from the internal web server via the router have a source address that is the internal web server. (The DNAT packets were forwarded by the router to the internal web server address; the web server replies naturally return from that address.) The router can't let these reply packets continue on to the clients as-is. The internal web server address in these reply packets means nothing to the external clients that sent the original incoming packet. The clients expect to see a reply packet coming from the public address to which it sent the original request (i.e. from www.myhost.ca), not from some internal web server source address. A machine always expects to see a reply returning from the address it was originally contacting. To fix the reply packets and make them acceptable to the client, every DNAT rule on the router automatically *remembers* the original destination address to which an incoming packet was addressed; the router does not "throw away" the original destination address when it rewrites it. In our example, the router remembers that the client originally addressed the web request to 100.0.0.254. The DNAT mechanism on the router recognizes that the returning reply packets from the internal web server are in response to the original DNAT-ed packets that it sent to the web server. With each DNAT rule there is an automatic corresponding un-DNAT mechanism that the router uses to fix the reply packets going back to the client so that they appear to come from the router (www.myhost.ca), not from the internal address of the web server that actually originated the reply packets. The router does DNAT on packets coming in, and it does un-DNAT on the reply packets before sending them back out. The un-DNAT mechanism in the router replaces in each reply packet the private internal source address of the web server with the remembered router destination address saved by DNAT, before relaying the web server packet back to the client. This makes the web server reply packets look to the client as if they came from the original router address to which it sent the incoming packet. Here is what happens in the two cases: 1. A reply packet returning via the router to the client on the Internet: Reply: from 172.16.0.1 (webserver) to 123.45.6.7 (client) un-DNAT: from 100.0.0.254 (router) to 123.45.6.7 (client) 2. A reply packet returning via the router to client on the trusted network: Reply: from 172.16.0.1 (webserver) to 192.168.0.2 (client) un-DNAT: from 192.168.0.254 (router) to 192.168.0.2 (client) The above two cases cover two of the three networks. As mentioned earlier, in the third case, the web server's reply from 172.16.0.1 to the DMZ client at 172.16.0.2 does *not* return to the router for un-DNAT; it goes directly to the DMZ client on the same network and gets rejected. We'll talk about fixing that problem later. For two of the three networks, the remembered DNAT destination address and automatic un-DNAT mechanism in the router ensure that packets returning to the client appear to come from the exact router interface that received the original incoming packet from the client. The client thinks it's sneding packets to and receiving packets from www.myhost.ca. The router does DNAT and un-DNAT magic to make this appear true. In the third case - the client on the same DMZ network as the web server - the reply from the web server to the client is sent directly from the web server to the client on the same network and never passes through the router for un-DNAT. The client rejects the reply packet as coming from the wrong machine and the connection fails. DNAT and un-DNAT must work together ----------------------------------- For inbound packets from a client, the router's DNAT rule saves and changes the *destination* address, while the corresponding un-DNAT mechanism restores and changes the corresponding *source* address in the reply packets that come back from the web server and are forwarded back to the client. The two actions are a matched pair. No matter from where the client packet originates, the router applies DNAT in the same way - the router always replaces the destination address in the client packet with the address of the internal web server and delivers the packet to the web server. DNAT makes the incoming client packet look like it was destined for the internal address of the web server, while un-DNAT makes the internal web server reply packet look like it came from the public address of the router. To make port forwarding work, the router must DNAT all the inbound packets from the client and forward them to the web server, and un-DNAT all the returning reply packets from the web server and forward them back to the client. The router must be involved in both parts of the send/reply exchange. The un-DNAT mechanism happens automatically when the internal web server sends a reply packet back through this same router that did the DNAT. The router will use the address saved by DNAT to un-DNAT the reply packet, making it appear as if the reply came from the router, not from the internal web client. If the source address of an incoming client packet appears to the internal web server to be on a different network from the web server, the web server will send the reply packet back to the client via the web server's default gateway interface (the router). This gives the router the chance to un-DNAT the web server reply packet before delivering it back to the client. Thus the router is correctly involved in the DNAT translation path of client-->router(DNAT)-->server and the un-DNAT translation of server-->router(un-DNAT)-->client, and the exchange correctly has both DNAT and un-DNAT parts to it. The client thinks it is talking to the router, not the internal web server. The router does DNAT/un-DNAT to make this possible. The external client will know nothing about the internal web server. The router uses DNAT to put the internal web server destination address into incoming client packets that go to the internal web server, and the router's un-DNAT mechanism replaces the internal web server source address with one of its own network addresses before sending the return packet from the web server back to the client. As long as the router can apply both DNAT and un-DNAT to a packet, the internal web server address never appears in packets returned to the client. The client thinks it is talking directly to a web server running on port 80 on the router. The router makes it work by transparently forwarding the entire conversation in both directions, changing destination addresses in incoming packets from the client and changing source addresses in matching outgoing reply packets to the client. (We will talk later about what happens in the case of the DMZ client at 172.16.0.2, where the router has applied DNAT to the packet; but, the reply from the web server does not return to the router for un-DNAT before being sent back to the client.) Example packet trace (Internet client) -------------------------------------- Suppose an external Internet client (source address 123.45.6.7) sends a packet to port 80 at the router using the router public interface address 100.0.0.254 (DNS name www.myhost.ca). This incoming Internet packet headed for port 80 triggers the DNAT rule in the router that applies to packets arriving on the external interface (from the Internet): iptables -t nat -A PREROUTING -d 100.0.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 The packet destination address of 100.0.0.254 (router www.myhost.ca) gets saved by DNAT and replaced by the address 172.16.0.1 (the internal web server on the DMZ network). The packet is then forwarded for delivery to the internal web server on the DMZ network: Arriving: from 123.45.6.7 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 123.45.6.7 (client) to 172.16.0.1 (webserver) Packet (with new destination address) gets sent to the DMZ network. The source address of the incoming packet remains unchanged. The original destination address 100.0.0.254 (router www.myhost.ca) in the incoming client packet is replaced and remembered by the router's DNAT rule. The remembered address will be used later by the router's un-DNAT mechanism to replace the source addresses of reply packets coming back from the internal web server to the client on the Internet. The internal web server at 172.16.0.1 receives the DNATed packet from the router and generates a reply. The DNATed packet still has an Internet source address (unchanged) of 123.45.6.7. The generated reply packet therefore has a source address of the internal web server (172.16.0.1) and a destination address of the Internet client at 123.45.6.7: Arriving: from 123.45.6.7 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 123.45.6.7 (client) Since the reply destination address 123.45.6.7 (client) is not on the same local network as the internal web server (not on the DMZ network 172.16.0.0/24), the web server is forced to hand off this reply packet destined for 123.45.6.7 to its default network gateway ("default route") for delivery. That default DMZ gateway is the router at 172.16.0.254. The router's DMZ interface (172.16.0.254) receives the reply packet from the internal web server (172.16.0.1) on the DMZ network. The reply packet has a source address of the web server (172.16.0.1) and a destination address of the Internet client (123.45.6.7). The router is expecting this reply; it recognizes the reply as being a response to a previous DNAT packet that it sent to the web server, and so the router applies un-DNAT to the web server reply packet source address before forwarding the web server reply packet to the client: Arriving: from 172.16.0.1 (webserver) to 123.45.6.7 (client) un-DNAT: from 100.0.0.254 (router) to 123.45.6.7 (client) Now the reply packet from the internal web server, destined for the client and having passed through un-DNAT, appears to have a source address of the router (100.0.0.254), not of the internal web server (172.16.0.1). The translated reply packet is forwarded to the original Internet client at 123.45.6.7 on the Internet. The original external Internet client sees the router's external address (100.0.0.254) as the source of the packet. The DNAT and un-DNAT done by the router hide from the Internet client the fact that the internal web server is on a different machine hidden on an internal DMZ network. To the Internet client, the transaction looks this simple: Send: from 123.45.6.7 (client) to 100.0.0.254 (router www.myhost.ca) Reply: from 100.0.0.254 (router) to 123.45.6.7 (client) Further exchanges between the Internet client and the internal web server repeat this sequence, using the router as the DNAT translator both ways. In every incoming client packet, the router uses DNAT to change the destination address 100.0.0.254 to be the internal web server address 172.16.0.1. The original destination address is saved by the router. For every reply packet coming back from the internal web server, the router does un-DNAT to change the internal web server source address 172.16.0.1 back to the saved router external public address 100.0.0.254: At the router: Incoming: from 123.45.6.7 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 123.45.6.7 (client) to 172.16.0.1 (webserver) At the internal web server: Incoming: from 123.45.6.7 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 123.45.6.7 (client) At the router again: Incoming: from 172.16.0.1 (webserver) to 123.45.6.7 (client) un-DNAT: from 100.0.0.254 (router) to 123.45.6.7 (client) Things work fine, as long as the client and internal server are on different networks that require the packets to pass through the router's DNAT/un-DNAT mechanism in both directions. Failed replies with client and server on the same network --------------------------------------------------------- Sometimes small network sites don't have a separate DMZ network; both the client machines and the internal servers are on the one single internal network. (Having your servers on the same network as your clients is not good security; don't do it.) A client on the same internal network as the internal web server tries to connect to the web server using its published external DNS address on the router, and that times out and fails. Or, even if you have a separate DMZ network, clients on the DMZ network find they time out when accessing any servers in the DMZ network using their published external DNS addresses (on the router's external interface). Here's why. The DNAT/un-DNAT process in the router isn't fully applied if the web client and internal web server are on the *same* internal network. The original packet from the client to the published external DNS address of the web server goes to the router, has DNAT applied correctly by the router, and returns from the router to the web server on the same network as the client. The reply packet from the internal web server to the internal client, being on the same network as the client, doesn't need to go through the router. The reply packet goes straight from the web server to the client on the same network, where it is rejected. (The client is expecting a reply from the external public address on the router, where it sent the original packet, not from the web server on the local network!) The connection times out. Here's a concrete example, using our DMZ network: Suppose a client on the internal DMZ network (e.g. a client at 172.16.0.2) tries to access the web server using the external public address on the router (DNS name www.myhost.ca, public address 100.0.0.254). Since address 100.0.0.254 is not on the DMZ network where the client is, the client sends the packet to the DMZ gateway address (the router DMZ gateway address at 172.16.0.254). The router sees that the packet incoming from the DMZ network has a source address of the client (172.16.0.2) and a destination address of the external interface (100.0.0.254). The router applies DNAT to change the packet destination to be the web server in the DMZ at 172.16.0.1. The packet from the client then goes right back out onto the DMZ network again (which is where it came from!) and arrives at the web server at 172.16.0.1. So far, so good. The web server gets a DNATed packet via the router. The web server builds a reply packet with source address 172.16.0.1 (webserver) and destination address 172.16.0.2 (the client). Since destination address 172.16.0.2 is on the same network as the web server (both on the DMZ), the reply packet goes straight from the web server 172.16.0.1 to the client at 172.16.0.2, where it is rejected. (The client is expecting to receive a reply packet from the external public address on the router [100.0.0.254], where it sent the original packet, not from the web server on the local network [172.16.0.1]!) The connection times out. We noted in an earlier section that reply packets from the internal web server are only returned to the router if the client that originated the connection is not on the same network as the web server. Clients on the same network as the web server (on the DMZ network 172.16.0.0/24) have their web server reply packets sent directly to the client, bypassing the router and bypassing the un-DNAT mechanism; this doesn't work. If un-DNAT is not applied by the router to fix the packet source addresses, the packets arriving directly from server to client originate from the internal web server address. This address is not the one to which the client originally sent a packet, and the client is not expecting to get any replies directly from the web server. The client will drop and ignore these bogus direct reply packets. The client at 172.16.0.2 will wait in vain for a reply that never comes from the public router address to which it sent the original packet (100.0.0.254). Eventually the connection will time out and fail. For un-DNAT to work, the reply packets from server to client *must* be returned via the router, even for clients on the same network as the internal web server. We will see later how to make this happen. Example failing packet trace (DMZ client) ----------------------------------------- The DNAT/un-DNAT only fails if the the client is on the same network as the internal web server, e.g. a client on the DMZ at 172.16.0.2 with the internal web server also on the DMZ at 172.16.0.1. Here's why: Let's assume we have an internal DMZ client at address 172.16.0.2 on the DMZ network and the web server is still at 172.16.0.1 on the same DMZ network. The client in the DMZ tries to connect to the web server using the DNS name www.myhost.ca. The external public address of the web server (DNS name www.myhost.ca) is 100.0.0.254 in our examples. The initial packet from the client (at 172.16.0.2) to www.myhost.ca (at 100.0.0.254) does a rather odd dance, when the client is on the same network as the web server. A DMZ client packet destined for www.myhost.ca (at 100.0.0.254) is not a local packet; it has to be sent via the default DMZ gateway (the router's DMZ gateway address is 172.16.0.254). The packet goes to the router. The client packet, incoming to the router from 172.16.0.2 on the DMZ network to the web server at port 80 on 100.0.0.254 (the router, www.myhost.ca), gets correctly rewritten by DNAT in the router to have a destination address of the web server at 172.16.0.1 on the DMZ network: Original: from 172.16.0.2 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 172.16.0.2 (client) to 172.16.0.1 (webserver in DMZ) Because in this case the rewritten (by DNAT) destination address of the DMZ web server (172.16.0.1) is on the same network as the DMZ client (172.16.0.2), the client's packet that came in from the DMZ network goes right back out the same network interface into which it came! Normally this kind of network nonsense is considered an error, and an ICMP redirect packet would be generated by the router telling the client to talk directly to the web server and not to make that silly routing mistake in future. Only since kernel 2.4.0 has this redirect packet been suppressed for DNATed packets. The packet from the client comes in on the 172.16.0.0 DMZ network and is DNAT-redirected right back out the same DMZ interface, to arrive correctly at the web server in the DMZ. So far so good. When the DMZ web server at 172.16.0.1 builds its reply to this incoming client packet from 172.16.0.2, its reply must naturally go back to the source address that sent the packet. The destination address of this reply packet becomes the DMZ client address 172.16.0.2: Incoming: from 172.16.0.2 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 172.16.0.2 (client) The web server notes that this destination address (172.16.0.2) is on the same DMZ network as its own address (172.16.0.1). Because the destination address of the reply packet is on the same network as itself, the web server knows it can send the reply directly to that address on the local network without using the default gateway (the router). The web server sends the reply packet directly to the client, without going via the router. The web server sends its reply packet directly from itself at 172.16.0.1 to the client at 172.16.0.2 without using the gateway router host at 172.16.0.254. (Machines on the same network communicate directly with each other - they don't use the gateway machine that is needed when sending packets between different networks.) The DMZ client at 172.16.0.2 is not expecting any packets to arrive directly from the web server at 172.16.0.1; the client is expecting a reply packet from port 80 on the router external address at 100.0.0.254; because, that was where it sent its initial connection request. So the client throws away the bogus unexpected packet from the internal web server at 172.16.0.1, and all other packets arriving directly from the web server. The connection eventually times out; because, the client never gets any valid replies from the router address 100.0.0.254 where it started the connection. This type of reply failure happens only when the client machine uses the external public address of the web server (DNS name www.myhost.ca), and the client is also on the same network as the internal web server, causing the router to be bypassed for reply packets. The internal web server sends its replies directly to the internal client on the same network, instead of passing the replies through the router for proper un-DNAT and return to the client. Let's see how to fix this. Making clients on the same network reply to the router ------------------------------------------------------ The trick we can do to repair this short-circuit reply error, when the client and the server are on the same network, is to ensure that packets being routed from clients in the DMZ, through the router, back to servers in the DMZ always have replies returned to the router for un-DNAT. The idea is to make sure that the DNATed packets arriving at the web server from the router will generate reply packets that get sent back to the router for un-DNAT, even if the client and server are on the same network. (If the incoming packets are from clients on a different network than the web server, the reply packets automatically return via the default gateway address of the router.) The replies must not get short-circuited and go direct from the internal web server to the internal client, bypassing the router. To fix the short-circuit reply problem, we have the router make the packets coming from the client via the router appear to *originate* from the router, instead of from the client, hiding the fact that the packets actually come from the client on the DMZ network. If the packet now originates from the router, the web server is forced to send its replies back to the router, not to the client on the same network. We can ensure that the reply packets from the server to the client always get returned to the router for un-DNAT by making the router alter the *source* address of the initial client packets that pass through the router to the web server. The router alters the client packets so that they always appear to be from the router, before passing them to the web server. The web server will then be forced to send the reply packet to the router, not directly to the client, and the router will be able to apply the un-DNAT process and then forward the fixed packet on to the client. To alter packet source addresses in the router, we use iptables netfilter Source Network Address Translation (SNAT). We create a rule so that, for clients on the same network as the web server, the router will also apply SNAT to the already DNATed client packets being sent from the router to the internal web server. The SNAT rule will replace the troublesome client source address in the packets (172.16.0.2) with the DMZ address of the router itself (172.16.0.254). Just as a DNAT rule changes the *destination* address of a packet, and its corresponding un-DNAT mechanism changes the *source* address of a reply packet, the SNAT rule changes the *source* address of a packet, and its corresponding un-SNAT mechanism changes the *destination* address of a reply packet. Both incoming packets and their reply packets have to pass via the router for this to work. The incoming packet from the client must have both DNAT and SNAT applied before being forwarded to the web server. It works like this: Original: from 172.16.0.2 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 172.16.0.2 (client) to 172.16.0.1 (webserver) SNAT: from 172.16.0.254 (router) to 172.16.0.1 (webserver) In a manner analogus to DNAT, the source address being replaced by SNAT is saved for later use as a destination address by un-SNAT when the reply packet comes back to the router from the DMZ web server. (Since the source address in the packet is now the router, the reply is guaranteed to come back to the router!) DNAT saves the original destination address, to be used by un-DNAT as the source address of reply packets; SNAT saves the original source address, to be used by un-SNAT as the destination address of reply packets. After both DNAT and SNAT, when the web server receives the incoming client packet, the source address of the packet is the router, not the client. The web server is forced to send reply packets back to the router: Incoming: from 172.16.0.254 (router) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 172.16.0.254 (router) When the reply packet from the web server arrives back at the router, the router recognizes that this packet requires both un-SNAT and un-DNAT. Since the router processed the original client packet using DNAT followed by SNAT, the router handles the reply packet using un-SNAT followed by un-DNAT. The un-SNAT process has the saved client source address (172.16.0.2) restored to the destination address in the packet. The un-DNAT process has the saved router destination (100.0.0.254) address restored to the source address in the packet. Let's see this in detail. The first thing the router does with the reply packet from the web server is to apply the un-SNAT mechanism to the *destination* address. (SNAT changes a *source* address; un-SNAT changes a *destination* address.) The un-SNAT mechanism removes the current reply destination address of 172.16.0.254 (router) and replaces it with the original source address that was saved by SNAT when it translated the source address of the original incoming client packet: 172.16.0.2 (client). (Recall that the SNAT changed the source address from 172.16.0.2 [client] to 172.16.0.254 [router] on the incoming client packet and saved the original source address.) The un-SNAT works like this: Reply: from 172.16.0.1 (webserver) to 172.16.0.254 (router) un-SNAT: from 172.16.0.1 (webserver) to 172.16.0.2 (client) Second, the router applies the un-DNAT mechanism to the *source* address in the reply packet from the web server. (DNAT changes a *destination* address; un-DNAT changes a *source* address.) The un-DNAT mechanism removes the current source address of 172.16.0.1 (webserver) and replaces it with the original destination address that was saved by DNAT when it translated the destination address of the original incoming client packet: 100.0.0.254 (router www.myhost.ca). (Recall that the DNAT changed the destination address of the original incoming client's packet from 100.0.0.254 [www.myhost.ca] to 172.16.0.1 [the webserver in DMZ] and saved the original destination address.) The un-DNAT works like this: un-SNAT: from 172.16.0.1 (webserver) to 172.16.0.2 (client) un-DNAT: from 100.0.0.254 (router) to 172.16.0.2 (client) The result, after the router applies both un-SNAT and then un-DNAT, is a reply packet that originally arrived with a source address of the web server (172.16.0.1) and a destination address of the router (172.16.0.254), now has new source address 100.0.0.254 (router) and new destination address 172.16.0.2 (client). The reply packet, now correctly un-SNATed and un-DNATed, has a destination address of the client on the DMZ network and a source address of the public external interface (www.myhost.ca), and the router forwards the reply packet to the client in the DMZ, and the client is happy to accept it. The whole sequence looks like this: Reply: from 172.16.0.1 (webserver) to 172.16.0.254 (router) un-SNAT: from 172.16.0.1 (webserver) to 172.16.0.2 (client) un-DNAT: from 100.0.0.254 (router) to 172.16.0.2 (client) After both un-SNAT and un-DNAT, the reply packet continues correctly back to the client on the DMZ network, from the router, with a source address that is just as the client expects. The client sent to external public address www.myhost.ca; the reply comes back from www.myhost.ca, as expected. The use of SNAT has forced the web server's reply packets to go via the router to the client, where un-SNAT and un-DNAT have make the reply packet correct for the client. The next exchange between the client on the DMZ network and the web server's external public address repeats the process: every packet from the client goes through the router and has both DNAT followd by SNAT applied; every reply packet from the DMZ web server returns to the router and has both un-SNAT and un-DNAT applied. The client and web server communicate via the router in both directions, even though both machines are actually located on the same DMZ network. We implement SNAT by adding a SNAT rule to the router (in addition to the above DNAT rules) that changes the source addresses of packets destined for the web server, if the source addresses of the packets are on the same network as the web server: iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -d 172.16.0.1 \ -p tcp --dport 80 -j SNAT --to 172.16.0.254 The above SNAT POSTROUTING rule is traversed *after* the DNAT PREROUTING rule that changes the destination address; so, by the time we look at the packets in the SNAT rule, the destination addresses will have already been translated by DNAT from the original router external public address 100.0.0.254 to be the DMZ web server internal address 172.16.0.1 by the DNAT rule. So the above SNAT rule matches packets that have been DNATed to go to the web server, and that have a source address anywhere on the same network as the web server (172.16.0.0/24). With the above SNAT rule, any already-DNATed client packet destined for the web server (-d 172.16.0.1) that has a source address on the same DMZ network as the web server (-s 172.16.0.0/24) will have its source address changed from whatever IP address it might have on the DMZ network (e.g. 172.16.0.2) to to be the IP address of the router network interface (--to 172.16.0.254). This guarantees that the web server will generate a reply *back through the router*, not directly to the client: Before the SNAT rule was introduced: Original: from 172.16.0.2 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 172.16.0.2 (client) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 172.16.0.2 (client) [FAILS] (The reply packet does not return via the router for un-DNAT; it goes direct to the client. The bogus reply packet is dropped by the client; the client wants a reply from 100.0.0.254. The client times out waiting for a reply from the router that never arrives.) With the above SNAT rule added to the router: Original: from 172.16.0.2 (client) to 100.0.0.254 (router www.myhost.ca) DNAT: from 172.16.0.2 (client) to 172.16.0.1 (webserver) SNAT: from 172.16.0.254 (router) to 172.16.0.1 (webserver) Reply: from 172.16.0.1 (webserver) to 172.16.0.254 (router) un-SNAT: from 172.16.0.1 (webserver) to 172.16.0.2 (client) un-DNAT: from 100.0.0.254 (router) to 172.16.0.2 (client) The incoming client packet that the web server finally sees, having been through both DNAT and SNAT at the router, will have had the destination address changed by DNAT from 100.0.0.254 (router, www.myhost.ca) to 172.16.0.1 (web server in DMZ), and the source address changed by SNAT from 172.16.0.2 (client) to 172.16.0.254 (router DMZ address). The web server naturally generates a reply to this incoming client packet using itself 172.16.0.1 (web server) as the source and 172.16.0.254 (router) as the destination. The web server reply packet goes to the router, as desired. The DMZ network client packet travels through both DNAT and SNAT to the server (client-->router(DNAT+SNAT)-->server) the server reply travels through both un-SNAT and un-DNAT back to the client (server-->router(un-SNAT+un-DNAT)-->client). The DMZ client thinks it is talking to the public address www.myhost.ca (100.0.0.254), not to the internal web server. The DMZ web server thinks packets are coming from the router (172.16.0.254), not from the actual client (172.16.0.2), and it sends reply packets via the router, not directly to the client. Two machines on the same network that would normally communicate directly have been fooled by SNAT into sending their packets via the router. Note: Because SNAT has put the router's DMZ source address (172.16.0.254) into the packets sent to the web server, the web server has no idea that the packets actually came from the client located in the same DMZ network. For every client in the DMZ network, your web logs will never show any true client addresses - all those incoming DMZ client packets will appear to come from the single router DMZ interface address (172.16.0.254). There is no way to preserve the original client source address in the incoming packet and still have the reply packet return via the router. Many clients - too much SNAT ---------------------------- If we have many different port-forwarded servers (using different ports on the external public interface) receiving packets on an internal network via DNAT, we could find ourselves writing a lot of extra SNAT rules to make the individual servers work with clients on the same networks. We can simplify all the the SNAT rules for any number of servers on the DMZ network to just one, by removing the port restriction: iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -d 172.16.0.0/24 \ -j SNAT --to 172.16.0.254 Instead of specifying the the exact destination IP address of the DMZ web server, use a network expression (-d 172.16.0.0/24) so that *all* packets passing through the router that have both source and destination address on the DMZ network will have the SNAT rule applied. The above rule ensures that any packets coming from the DMZ interface being sent back out onto the DMZ interface using DNAT will have a return (source) address of the router, no matter what their destination protocol, port, or specific IP address. This one rule works for *all* the DMZ servers. With the above single, more general SNAT rule, *all* services on the DMZ network are available to all clients on the same DMZ network, not just the single internal web server. Efficiency note: What about packets that actually *originate* from the router and go to the DMZ, e.g. suppose the router is providing DNS services and generates DNS reply packets to clients in the DMZ? The above SNAT rule also needlessly applies SNAT to packets that originate from the router itself, even though they already *have* the router's source address! We don't need to SNAT packets that already have a source address of the router. Here is a fix (courtesy of Jan Engelhardt) that limits the SNAT to apply only to packets that have already had DNAT applied: iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -d 172.16.0.0/24 \ -m conntrack --ctstate DNAT -j SNAT --to 172.16.0.254 Using SNAT on each router interface ----------------------------------- The router needs a SNAT rule for every network that handles port-forwarded packets from clients that are on the same network as their servers. The above discussion shows the SNAT rule for a port-forwarded web server in the DMZ network with clients also in the DMZ network. You may have servers in your local trusted network (though this is a security risk) that also receive port-forwarded traffic from clients in the same network; the router needs a separate SNAT rule for that network, too: iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 \ -m conntrack --ctstate DNAT -j SNAT --to 192.168.0.254 The above router SNAT rule ensures that any packets with a source address on the trusted network that also have a destination address on the trusted network will have a return (source) address set to be the router. This avoids "short-circuit" replies going from the web server directly to the client on the same network, bypassing un-DNAT. Summary ------- Here is the complete set of DNAT rules for a three-network router forwarding www.myhost.ca (100.0.0.254) to port 80 on an internal DMZ web server located at 172.16.0.1: iptables -t nat -A PREROUTING -d 100.0.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 iptables -t nat -A PREROUTING -d 172.16.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 iptables -t nat -A PREROUTING -d 192.168.0.254 \ -p tcp --dport 80 -j DNAT --to 172.16.0.1 Here are the two SNAT rules that make clients and servers on the two internal networks find each other correctly via the router: iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -d 172.16.0.0/24 \ -m conntrack --ctstate DNAT -j SNAT --to 172.16.0.254 iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 \ -m conntrack --ctstate DNAT -j SNAT --to 192.168.0.254 That's it. -Ian! D. Allen - idallen@idallen.ca with helpful comments from: Jan Engelhardt (on medozas.de email to jengelh) Appendix I: SmartRG home modem configuration -------------------------------------------- As you know, behind your router in NAT world if you try to use your real external IP address (the public address of your router's WAN port) to access a service on your local network, things don't work. Below is just the bit you need, excerpted for my single-net situation. It should work for you. For this example, I use these: - my router's WAN IP is 45.72.234.196 - my router's local IP is 192.168.2.254 - my server's IP is 192.168.2.250 - a typical client IP is 192.168.2.10 Description: A packet headed from a client machine inside your local net (e.g. from 192.168.2.10) to a server via your router's real external IP address (e.g. to 45.72.234.196) heads out your router to the WAN port and needs to have its destination IP changed by the router (using DNAT) to be the server machine inside your local net (e.g. back to 192.168.2.250). The original packet with client source 192.168.2.10 and server destination 45.72.234.196: Source: 192.168.2.10, destination: 45.72.234.196 needs the destination IP to be changed by the router (using DNAT) to be the server IP at 192.168.2.250: Source: 192.168.2.10, destination: 192.168.2.250 The packet now arrives at the server at 192.168.2.250. The server constructs a reply packet to that client packet by switching the source and destination addresses: Reply: Source: 192.168.2.250, destination: 192.168.2.10 Unfortunately, the reply packet from the server at 192.168.2.250 to the client at 192.168.2.10 does *NOT* return via the router, because the server machine doing the reply to the client is on the same network as the client; it short-circuits; the reply goes directly from the server at 192.168.2.250 to the client at 192.168.2.10 and the client rejects it. (The client sent to 45.72.234.196 and wants a reply from 45.72.234.196.) The fix is that you have to make sure that, in addition to having your router do the DNAT, the router has to use SNAT to change the *source* address of the forwarded packet so that it returns *to the router* (i.e. to 192.168.2.254) and does not short-circuit to the client at 192.168.2.10. You have to SNAT the forwarded packet from this: Source: 192.168.2.10, destination: 192.168.2.250 to be from the router 192.168.2.254 like this: Source: 192.168.2.254, destination: 192.168.2.250 Now, when the server at 192.168.2.250 replies to that packet, it replies to the router at 192.168.2.254: Reply: Source: 192.168.2.250, destination: 192.168.2.254 The router un-SNATs the destination address from 192.168.2.254 to be the original client at 192.168.2.10: Reply un-SNAT: Source: 192.168.2.250, destination: 192.168.2.10 The router also un-DNATs the source address from the server at 192.168.2.250 to the original WAN address 45.72.234.196: Reply un-SNAT un-DNAT: Source: 45.72.234.196, destination: 192.168.2.10 and now the packet returns to the client with the correct source address. Here are the PREROUTING (DNAT) and POSTROUTING (SNAT) tables from my SmartRG: Chain PREROUTING (policy ACCEPT 324K packets, 20M bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- ppp0.1 any anywhere 224.0.0.0/4 1077K 69M CA_APPS_CHAIN all -- br0 any anywhere anywhere 0 0 ACCEPT tcp -- ppp0.1 any anywhere anywhere tcp dpt:30005 0 0 ACCEPT udp -- ppp0.1 any anywhere anywhere udp dpt:30007 210 11168 DNAT all -- ppp0.1 any anywhere anywhere to:192.168.2.250 25 1980 DNAT all -- br0 any anywhere 45.72.234.196 to:192.168.2.250 Chain POSTROUTING (policy ACCEPT 91076 packets, 5296K bytes) pkts bytes target prot opt in out source destination 22876 3645K SNAT all -- any any 192.168.2.0/24 192.168.2.250 to:192.168.2.254 3218 231K MASQUERADE all -- any ppp0.1 192.168.2.0/24 anywhere Below is the code I use for the SmartRG to set the above tables. The "modem" command is an "expect" script that executes commands at the SmartRG modem CLI. I run the code below from the cron every few minutes as part of a bigger script to keep the tables updated. ########################################################################## # make sure SmartRG does DNAT and SNAT for remote IP ian.idallen.ca # - the variable $ianidallencaIP contains my router's current WAN IP # - my router's local IP is 192.168.2.254 # - my server's IP is 192.168.2.250 # if [ "${ianidallencaIP-}" != "" ] ; then prerouting=$( modem 'iptables -v -t nat -L PREROUTING' ) if [ "$prerouting" = "" ] ; then # something wrong with the modem logger -s -t "$0" "Cannot find PREROUTING in modem" else good=$( echo "$prerouting" | grep "DNAT .* br0 .* $ianidallencaIP .* to:192\.168\.2\.250" ) if [ "$good" = "" ] ; then # delete old DNAT rule echo "$prerouting" | awk '$3 == "DNAT" && $6 == "br0" && $9 ~ /^[0-9.]+$/ && $10 == "to:192.168.2.250" { print "iptables -t nat -D PREROUTING -i " $6 " -d " $9 " -j DNAT --to-destination " substr($10,4) }' \ | while read -r line ; do modem "$line" done logger -s -t "$0" "Adding $ianidallencaIP to SmartRG PREROUTING" modem "iptables -t nat -A PREROUTING -i br0 -d $ianidallencaIP -j DNAT --to-destination 192.168.2.250" fi fi postrouting=$( modem 'iptables -v -t nat -L POSTROUTING' ) if [ "$postrouting" = "" ] ; then # something wrong with the modem logger -s -t "$0" "Cannot find POSTROUTING in modem" else good=$( echo "$postrouting" | grep 'SNAT .* 192\.168\.2\.0/24 *192\.168\.2\.250 *to:192\.168\.2\.254' ) if [ "$good" = "" ] ; then logger -s -t "$0" "Adding SNAT for 192.168.2.0/24 to SmartRG POSTROUTING" modem 'iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -d 192.168.2.250 -j SNAT --to 192.168.2.254' fi # sanity check to detect build-up of large POSTROUTING tables pmax=5 num=$( echo "$postrouting" | wc -l ) if [ $num -gt $pmax ] ; then logger -s -t "$0" "SmartRG POSTROUTING count $num > $pmax" echo "$postrouting" | nl fi fi fi ########################################################################## Copyright --------- Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation. http://www.tldp.org/ -- | Ian! D. Allen, BA-Psych, MMath-CompSci idallen@idallen.ca Ottawa CANADA | Home: www.idallen.com Contact Improvisation Dance: www.contactimprov.ca | Former college professor of Free/Libre GNU+Linux @ teaching.idallen.com | Improve democracy www.fairvote.ca and defend digital freedom www.eff.org