Hi,

This is a long message. It comes down to "is there a good way to make a box always use a specific default source address?"

The longer version is below to give you some more background information :-)

I am talking to a customer who has the following situation:

They have multiple dual stack networks. I'm only looking at the IPv6 part here. The servers (Linux boxes) all have multiple IPv6 addresses. Every box has its own address (iron address) and each service has a separate address (service address) so that it can be moved to a different box if necessary (unexpected high number of visitors etc). For some of the addresses load balancers are used to distribute the load.

All of the load balancing is done using Direct-Routing (one box responds to NDP for that address and then forwards the packet to the right back-end server by leaving the layer-3 addresses in place and using the layer-2 address of the chosen back-end server, the back-end server has the address on its loopback interface, accepts the packet and sends the reply directly to the original layer-3 source address) with VRRP between load balancers for redundancy.

Now the incoming requests are no problem at all. The problem is with the source address selection when the boxes initiate a connection. We want outgoing connections to always use the iron address as default source address. Using one of the service addresses is undesirable for several reasons:
- If the address is load balanced then initiating outbound connections from that address will break as return traffic might be load balanced to one of the other boxes
- If the application running service A creates an outbound connection using service B's address, then moving service B to another box will break application A's established connections
- Determining which machine sent which packet is harder
- Firewall policies are much harder to write if connections are initiated from an unpredictable source address

With IPv6 this is much harder to accomplish than I ever expected. Putting the addresses in a configuration file is not an option because the services can be migrated to another box (= another iron address).

When looking at the source address selection mechanism in RFC 6724 I see a few possibilities to adjust the algorithm:

Rule 1: Prefer same address
- N/A here: Localhost traffic isn't important here

Rule 2: Prefer appropriate scope
- All addresses involved have global scope

Rule 3: Avoid deprecated addresses
- Making all service addresses deprecated prevents them from being used as a source address by default, so this accomplishes what we want, but it feels like a hack...

Rule 4: Prefer home addresses.
- N/A here: Mobile IPv6 not in use

Rule 5: Prefer outgoing interface.
- Lots of addresses on the same interface (does work for addresses on the loopback interface)

Rule 5.5: Prefer addresses in a prefix advertised by the next-hop.
- N/A: Possible source addresses are in the same prefix

Rule 6: Prefer matching label.
- Giving all service addresses a different label from the iron addresses seemed a good choice. We considered renumbering all services to use addresses in a specific part of the /64 and then configure Linux to use a separate label for that prefix (i.e. `ip addrlabel add prefix 2001:db8:123:abc:1000::/68 label 1000`). This works for connections to the outside world, but if one service connects to another service then it still would choose a service address as source address. Giving every service address its own label would solve that, but with the potentially hundreds of addresses on each box (yes, this is a big server farm) this becomes unmanageable.

Rule 7: Prefer temporary addresses.
- N/A: temporary addresses are disabled on the boxes

Rule 8: Use longest matching prefix.
- When connecting to external services all addresses on the LAN have the same match-length (all shorter than a /64). When connecting to other services on the same LAN it becomes unpredictable again (depends on the other service addresses on the same box)


Based on this it seems that making all service addresses deprecated immediately and leaving the iron address as the only preferred address would be the only way to solve this. It does feel like a hack though. RFC 4862 says that "an address becomes "deprecated" in anticipation that its current interface binding will become invalid". These addresses are not anticipated to become invalid for a very long time (= indefinite). I can already imagine a kernel deciding to free up some resources by dropping addresses that are marked as deprecated for example.

So: does anybody have any advice on this? Is it safe/future-proof/etc to mark addresses as deprecated to avoid them being used as default source addresses? Do we need some other way to steer the default source address selection procedure? Or is the only 'official' way to give each and every /128 its own label (so that rule 6 becomes equal to rule 1 for service addresses)?

Cheers,
Sander