Do you find this works reliably outside routers that preserve source ports? My understanding was that TCP punching tends to depend heavily on NAT behavior.
Hardly the case in even half of typical deployment cases.
I don’t think I’ve ever seen it done successfully and have often wondered if it’s for a lack of use cases or due to its bad success rate and complexity compared to UDP hole punching.
That said, I really wish there was a standardized way to do it. Some sort of explicit (or at least implicit but unambiguous) indicator to all firewalls that a connection from a given host/port pair is desired for the next few seconds. Basically a lightweight, in-band port mapping protocol.
It could have well been an official recommendation to facilitate TCP hole punching, but I guess it’s too late now, as firewall behaviors have had decades to evolve into different directions.
It is precisely this point that has flummoxed me when connecting my p2p wireguard config[1] with a friend that uses a pfsense router, no matter what we tried, pfsense always chooses a random source port.
But in the simple case this blog outlines, if both ends use the same source port, this method punches through 2 firewalls effortlessly:
NAT is effectively your router doing DHCP with a 17-bit suffix (16-bit port + 1 bit for UDP vs TCP) to each of your applications and then not telling you the address it gave you or how long it is good for (which is what a regular DHCP lease does). This is in addition to it, most likely, already doing regular DHCP and allocating you a IP address that it does tell you about, but which is basically worthless since routing to just that prefix without the hidden suffix goes into a black hole.
If you could just ask your router for a lease on a chunk of IP+NAT addresses that you could allocate to your applications and rotate them as they expire, you would not need this horrifying mess.
The router would just need to maintain the last-leg routing table (what a concept, a router doing routing with routing tables) just like it already does DHCP.
The applications would have short-term stable addresses that they could just tell their peers and just directly tell the router/firewall to block anybody except the desired peer short-term address.
In this era where AI is eating away at how deterministic computers are, I really appreciate reading about an elegant solution to a real problem using deterministic logic.
- can't decide on a port in the same message
- don't suffer from NAT port randomization
I'm not saying it will never happen, but the Venn diagram of this being the minimum complexity solution just doesn't seem very large?
>Many home routers try to preserve the source port in external mappings. This is a property called “equal delta mapping” – it won’t work on all routers but for our algorithm we’re sacrificing coverage for simplicity.
So to what percentage is this coverage sacrificed exactly? No idea. Not as useful if the percentage is high, as you are implying.
The “just” is doing a lot of lifting there. I’m glad the various port mapping protocols didn’t really take off and it looks like IPv6 is going to actually make it instead. Much less complexity in most parts of the stack and network.
At least with IPv6 this crap becomes a little easier because you no longer have randomized source ports (which this article just ignores because some devices indeed maintain the same source port) and the IP address contains all the routing information you need. A simple simultaneous open is all you need.
NAT Behavioural Requirements for Unicast UDP, https://datatracker.ietf.org/doc/html/rfc4787
NAT Behavioural Requirements for TCP, https://datatracker.ietf.org/doc/html/rfc5382
We could have a standard for doing that directly at the NAT box level instead of relying on a third party STUN server, it simply didn't happen (and in fairness, the benefits would be quite minimal).
NAT randomization, I don't know. Depends on your setup, I guess.
TIL, thank you! I've been looking for this for quite a while after hearing it indirectly referenced recently, but only found host-side specifications for TCP simultaneous open.
Do you happen to know if common firewalls and NATs support it? If they do, I really wonder why TCP hole punching isn't more common.
I wrote little paper on this technique in school and did some practical tests, at the time I was actually unable to find an example of consumer grade router that it didn't work on! But my resources were rather limited, they certainly do exist.
Now combining AI with deterministic tool calling brings the best of both worlds.
I am pointing out how the problem NAT “solves” is just dynamic address configuration. They have implemented a N+K bit address where the N-bit prefix is routed and allocated using IP and the low K-bits are routed and allocated like a custom fever dream.
You can just do it all the same way instead of doing it differently and worse for the low bits.
To be clear, the router should rewrite zero bits in the packet under the scheme I am describing just like how routers have no need to rewrite any bits when routing to a specific globally-routable IP address.
You get a lease for a /N+K address. /N routes to your router which routes the last K bits just like normal as if it had a /N-M to a /N route. This is a generic description of homogenous hierarchical routing.
Even in the presence of a conntrack entry created by an earlier outbound SYN,!ACK ?
Got a source?
Of course there’s still plenty of deterministic software you can run… for now.
This is a theistic statement at this point, no?
It might work better with UDP but I don't think those firewalls boxes tear down the mapping immediately on getting an RST - they wait until it times out.
This still sounds like a very bad mixing of layers, even if done in a perfectly standardized and uniform way.
> It is always a mystery how people just randomly misinterpret what I write.
If this is intended literally and not as a general complaint: My main problem of understanding your suggestion is that I don't know what you mean by "IP+NAT address". NAT is a translation scheme, not an address.
Maybe it would be clearer if you could provide an example?
Honestly ISPs really missed an opportunity to essentially provide IPv6-only as a service and add an IPv4 compatibility layer to that (IPv6 already has a mechanism built in for this but grandma’s old laptop might not fully support it so you might need a local router provided by the ISP to give you native local IPv4 that allows you to access the internet) instead of CGNAT. But they chose to go with duct tape, spit, paper clips, and hope instead of investing in the correct solution. Shame on them and too bad for us.
This hs a good diagram to understand the options
https://rajsinghtech.github.io/claude-diagrams/diagrams/netw...
You can scope it to just your IPsec service, or whatever it is your hosting, or you can enable full cone for the whole subnet.
It is not DNAT, nor is it port forwarding. If you host a SIP proxy, SBC or peer to peer gaming, it will enable these use cases as well.
https://docs.netgate.com/pfsense/en/latest/nat/outbound.html
> You get a lease for a /N+K address. /N routes to your router which routes the last K bits just like normal as if it had a /N-M to a /N route.
> This still sounds like a very bad mixing of layers, even if done in a perfectly standardized and uniform way.
No, I am describing a generalization of IP to arbitrary concatenated routing prefixs.
NAT has the same problems as if we lived in a alternate world where we decomposed IPv4 into 4 8-bit layers and then used a different protocol for each layer. That is obviously stupid because the subdivision of a /8 into /16s and a /16 into /24s is fractally similar. You can just use the same protocol 4 times. Or even better, use one protocol (i.e IP) that just handles arbitrary subdivision.
In the IPv4 (no NAT) world your application has a 49-bit address. Your router is running a DHCPv4 server and allocates your computer a /32 and your computer is “running” a DHCPvPort server that allocates a 17-bit prefix to your applications.
In the IPv4+NAT world your application has a 49-bit address. Your router is “running” a DHCPv4+Port server and allocates your applications a /49, but only tells them their /32 and then rewrites the packets because the applications do not know their address because the stupid router did not tell them.
In good world your application has a 49-bit address. Your router is “running” a DHCPv4+Port server and allocates your applications a /49 and tells them their /32 prefix and 17-bit segment. No packet rewriting is necessary.
Your router could also choose to allocate your computer a /32 subnet and leave DHCPvPort to your computer. Or it could give your computer a /31 if you have 8 interfaces. Or a /34 as a /32 subnet with 2-bit port prefix. Each node routes as much or as little routing prefix as it understands/cares about.
This is a generalization of IP that can handle arbitrary-length, arbitrarily-concatenated routing in a completely uniform manner and all the pieces are basically already there, just over-specialized.
And it's simply not true that there is no NAT in the wild with IPv6: every OPNsense installation with two uplinks and the need for anything better than an "arbitrary and uncontrollable" choice of the correct uplink for each outbound connection needs network prefix translation, as the residential dual-homing story for IPv6 is vaporware otherwise. NPT is used not for address space conservation, but to defer the decision about the correct source address to the router that has the knowledge of the correct policy.
And in this sense, IPv6 is worse than IPv4: there are too many people assuming no firewall and no NAT for IPv6, and designing their applications based on these almost-working (but de-facto now broken) premises. The correct premises are the same as for IPv4.
I don't have a convenient source though.
If you are tying to work around your firewall because it isn’t yours, that’s not a legitimate use.
TCP hole punching is a way to connect two computers behind NAT routers. The technique has a lot of requirements to work.
Both sides must know each other’s WAN IPs
They must know the right external ports to use
They must connect at exactly the same time
In practice this collapses to using STUN to lookup the WAN IP, doing NAT type enumeration and port prediction, synchronizing time with NTP, and having both sides exchange all the needed meta data (WAN IP, port predictions, future NTP punch time) via some “channel.”
That involves a whole list of infrastructure and code to work – which is complex and error-prone. What if you just wanted to test whether your punching algorithm works? You don’t care about every other part of the software. This is a way to do that.
A simple way to bypass the need for fixed infrastructure in hole punching algorithms is by using a deterministic algorithm to derive all meta data from a single parameter. Here’s how it works.
First, a starting parameter is chosen based on the unix timestamp. What we need is a number both sides can converge on without requiring any communication. However, as we know in distributed systems theory – there is no “now” in networks. So a little math is used to make the “now.”
now = timestamp() max_clock_error = 20s # How far off the timestamp can be min_run_window = 10s # Total time window to run the program for both sides window = (max_clock_error * 2) + 2 bucket = int((now - max_clock_error) // window)
We define what an accepted solution looks like. There ought to be a certain amount of time that the protocol can run in. A certain range that a timestamp must fall in to be considered valid. This information is combined and then quantized in a way that both sides converge on the same number even given variations to clock skew. This we term the “bucket.”
Now that both sides share the same bucket we can use this to derive a list of shared ports. The idea behind the port list is that the local port will equal the external port. Many home routers try to preserve the source port in external mappings. This is a property called “equal delta mapping” – it won’t work on all routers but for our algorithm we’re sacrificing coverage for simplicity.
In order to generate a shared list of ports without communication between the two sides – the bucket is used as a seed to a pseudo-random number generator. This allows for a much smoother conversion of the bucket number which might otherwise have little direct variation.
large_prime = 2654435761 stable_boundary = (bucket * large_prime) % 0xFFFFFFFF
The FF… number fixes the range for the boundary number so it doesn’t overflow what the PRNG accepts as a seed. A prime number is used for the multiplier because if the multiplier shares a common factor with the modulus (0xff..) then the space of possible numbers shrink due to shared factors. A prime number ensures that the number space contains unique items.
The port range is now computed based on using a randrange func. In my code I generate 16 ports and throw out any I can’t bind to. It is expected that there will be some port collisions with existing programs in the OS when manually selecting random ports like this.
It is helpful to review the requirements for TCP hole punching before we proceed. There are very specific socket options that must be set for it to work.
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
TCP hole punching involves aggressively reusing socket addresses. Normally with TCP, if a connection fails you close the socket. But with TCP hole punching if you do any kind of cleanup you break the protocol. Cleanup means calling close. And close means possibly sending out an RST packet that tells the remote router “bad connection, disregard.”
If you close the socket the OS will also start its own cleanup process. The socket will enter states like TIME_WAIT and will be harder to reliably reuse the same address even if you do set the right socket options. I also want to add that the only really correct model for TCP hole punching is to use non-blocking sockets. You can’t use blocking sockets because you need to be able to rapidly send SYN packets without having to wait for a response.
Async networking won’t work here either. Async networking prevents your software from being able to precisely control timing and TCP hole punching is highly sensitive to timing. If packet exchange is off by even a few miniseconds the whole protocol can fail. In my opinion: the best way to implement this is to use non-blocking sockets with select for polling. This allows you to properly handle every connection state without impacting timing.
for ... sel.register(s, selectors.EVENT_WRITE | selectors.EVENT_READ)
For punching: we simply call connect_ex on a (dest_ip, port) tuple with a 0.01 second sleep until an expiry (src_port == dest_port.) Actual punching here is not terribly elegant as you’re just spamming out SYNs. But this part needs to be aggressive enough to create a remote mapping but not so aggressive that it kills your CPU. I use a 0.01 second sleep for that.
In this algorithm multiple ports are used so many successful connections can be returned. The problem is how to choose the same connection? My approach is to start by having the sides choose a leader and follower. The leader has the greater WAN IP number. The leader sends a single character on a connection and cleanly closes the rest.
winner.send(b"$") ... loser.shutdown(socket.SHUT_RDWR) loser.close()
The follower then uses select to poll events. If it finds one it calls recv(1) to choose that connection (winner.) The reason a single char is used here is because TCP is stream-based. If the “success” delimiter was a word then the follower would have to implement a buffered-reader algorithm just to ensure they have everything (and I didn’t want that complexity.) A single char is atomic.
I present to you: a simple TCP hole punching algorithm requiring only a dest IP to use. Since the protocol is deterministic no infrastructure is required for testing and there is no need for the hosts to exchange meta data to use it. The hosts may still use NTP for time which is probably recommended though optional (older OSes in VMs can’t maintain time well.)
It is assumed another process will coordinate the running of this tool. Though you can easily run everything yourself on multiple terminals as long as the commands fall within the 10 second min_run_window. Here, punching applies for common routers using equal delta allocation.
You can run tcp_punch.py 127.0.0.1 to get a feel for the code.
> Plenty of setups block incoming SYN,!ACK packets
Nowhere close to being "plenty". It's doable, but this is extremely niche.
On the other hand, there is plenty of badly written networked software. I bet most of the networked software developers have no idea how to correctly plumb their software. They just open whatever connection, e.g. sockets, their OS provides and just run with it without care of the underlying layers. The OSI model theory in fact encourages this ignorance.
Take mobile data connections, for example: Most people don't want to pay for metered (by the byte) inbound traffic they didn't ask for that also drains their battery, but do want to be able to establish P2P connections for lower latency VoIP etc.
This is a firewall that's definitionally "not theirs", but that still also serves their interests, yet usually doesn't offer any user-accessible management interface.
So may I please traverse this firewall now, or is my use case still illegitimate?
I can't really imagine why you would do it for NAT'd v4 since you can't avoid the connection tracking overhead, but you certainly could, and I don't doubt OP has run into it in the wild. I've seen much weirder firewall rules :)
What are the obvious reasons? If you're protecting a client system, you don't want to allow in any bare SYNs. (And for that matter, if you're protecting a server, you probably want to discard ill-targeted bare SYNs without consulting conntrack anyway, just as a matter of avoiding extra CPU work.)
If you are buying firewall as a service then request a user interface or change your service provider.
What provider would you suggest somebody wanting to make VoIP calls on their smartphone switch to that allows port forwarding of the kind you describe? And which popular VoIP app would support statically forwarded ports like that?
All inbound packets are matched against existing sessions. In this case none will turn up, so the packet will go through the "new session" flow and be subject to the same filtering as a bare SYN. Look up how connection tracking works, e.g. in the Linux kernel, it's rather simple and logical.