I have a need to survive ISP outages, but am not large enough to have
real things like BGP and serious internet connections, so I am using a
telco and a cable company with a few static IPs on each.
There are various tutorials on the internet for how to cope with this,
but they seem to primarily involve using iptables MARK to stain packets
and then use the iproute2 functions to route them. I dislike conjoining
these two tools. I am using source routing to keep everything straight,
though there is a gotcha involving SNAT that needs attention when a link
- Support more than one ISP.
- Use the right source IP on each link to not trip packet spoof
- Survive a link going down and back up.
- Lots of IPv4 private address machines on the inside need to be NATed
on the way out.
- IPv6 is mandatory. Using 6rd while my ISPs recover from being
blindsided by that 20 year old RFC for IPv6.
- Some servers live outside the firewall/router, I won’t speak any
more of them, but they are there.
- Load balancing. This can be addressed with your outgoing rules, but
given the disparity in quality, there is little point in using the
U-verse link for outbound connections if the Charter one is up. The
inbound connections will still use it.
- A single point of failure router is fine with me.
- Use a VLAN switch so I don’t need a flock of switches and multiple
ethernet ports on the router and “outside the firewall” boxes.
Not required, but when you see decimal points in my ethernet
device names, those are the VLAN ids.
- Use source routing so that all packets go out the interface that
matches their source IP address.
- Use iptables SNAT to let the local machines out. Choose their SNAT
address based on the outgoing interface. Let the routing rules do
conntrack to forget cached SNAT mappings when a link goes down
or comes up. This is important!
VLAN Switch, 802.11q Is Your Friend
Go read about 802.11q if you are not familiar. With this you need only
one switch. You can have as many virtual LANs as you like and configure
on a port by port basis which LANs appear on that port. If you have gear
that doesn’t do 802.11q you can set a single VLAN to show up there and
work fine without any changes to that device. You will pay more for a
“smart switch” with 802.11q support, but you are going to save on the
number of switches, cabling, and ethernet cards. (e.g. in January 2013 I
paid $220 for a 24 port gigabit 802.11q switch.)
You will have to configure your switch. My NetGear switch is configured
through a web interface apparently writing by a maniacal sociopath, but
it can be made to do the job.
The Source Routing
We are going to need two auxiliary routing tables to hold rules for when
we know we have a U-verse address or a Charter address. These are going
to get names which means we add lines to /etc/iproute2/rt_tables,
(which is just a file mapping numbers to names)…
echo "200 att" >> /etc/iproute2/rt_tables
echo "201 charter" >> /etc/iproute2/rt_tables
When an interface comes up, we are going to add an ip routing rule to
force packets with a Charter source address to look in that charter
routing table and go out the right interface, likewise for AT&T… (Notice
the “throw” rules. Some people duplicate their main table here, but I’d
never keep that in sync, so I defer to the main table instead.)
# This is what makes source routing happen
ip rule add from 99.178.257.57/29 table att
# get a fresh start on the routing table
ip route flush table att
ip route add default via 99.178.257.62 dev eth2.4 table att
# get the RFC1812 private networks out, they don't want to go out this interface
# the "throw" will make them go back to your regular routing tables.
ip route add throw 10.0.0.0/8 table att
ip route add throw 172.16.0.0/12 table att
ip route add throw 192.168.0.0/16 table att
The SNAT For Our Private Addresses
Nothing new here, yet…
iptables -t nat -A POSTROUTING -o eth2.4 -s 172.16.0.0/12 -j SNAT --to-source 99.178.257.57
iptables -t nat -A POSTROUTING -o eth2.4 -s 192.168.0.0/16 -j SNAT --to-source 99.178.257.57
iptables -t nat -A POSTROUTING -o eth2.4 -s 10.0.0.0/8 -j SNAT --to-source 99.178.257.57
But wait! Now we have a problem. iptables connection tracking is going
to learn these SNAT rules, and for instance, if you have a ping running,
it will happily keep trying the dead interface after you take one down.
The fix I’m using is to clear the SNAT connection tracking information
with an interface goes up or down. I use this in my
/etc/network/interfaces stanzas (install
# We need to make NAT'd addresses choose a new path
# e.g. ICMP echo will be stuck on a dead interface if it was using this one
up conntrack -D --src-nat
down conntrack -D --src-nat
Choosing the Best Interface for Outgoing Traffic
You will want to use a metric on your default routes in order to choose
the best one. (Alternatively you can get into load balancing, but my
asymmetry is too high to care about that.)
I do this by not using the
gateway declaration in my
but just do a up command instead…
#gateway 99.178.257.62 --- but we want an explicit metric, so we do it this way
up ip route add default via 99.178.257.62 dev eth2.4 metric 1 || true
… that is my shunned AT&T connection. I use a metric of zero on the
Charter line so traffic prefers it, but will use AT&T if Charter goes
IPv6 gets the same treatment, except you don’t have to screw with SNAT
and conntrack, unless you really want to. Also, you will need some “-6”
keystrokes. It helps to remember that those routing tables for att and
charter are really four tables, two for IPv4 and two for IPv6.
I’ll just show you my Charter 6rd stanza, you can work it out from
iface charter6rd inet6 v4tunnel
# Force 6rd gateway to be on the Charter interface
pre-up ip route add 18.104.22.168 via 96.35.289.49 || true
# 2nd 32bits of this is my IPv4 address
up ip -6 rule add from 2602:100:6023:gd32::/59 table charter || true
down ip -6 rule del from 2602:100:6023:gd32::/59 table charter || true
up ip -6 route add default dev charter6rd table charter
post-down ip route del 22.214.171.124 via 96.35.289.49
up ip -6 route add 2000::/3 dev charter6rd metric 5
down ip -6 route flush dev charter6rd
What Is Wrong With This Strategy
When one of the ISPs is broken, I need to bring down their interface,
otherwise traffic will happily still try to use it. There may be
automated ways to do this, but I’m a simple barbarian and given the
rarity of the events, I just use a little cron job that if it can’t see
some portion of the internet out a particular interface, brings that
interface down for a little while. I suppose playing with the default
route metrics would be nicer, but like I said, simple barbarian. (I do
have a nagging suspicion that if I were smarter about the load balancing
it would “just work”. But I’m not.)