[nicolaskagami]

Context

In order to implement RFD 662 - Modular Router Configuration, the switch needs to support separate routing contexts. For context, RFD 404 - Boundary Tunnel Routing defines how we use tunneling to achieve external reachability. We're modularizing external reachability (for tunneled traffic), not implementing VRFs.

This PR focuses on separating these routing contexts while keeping strong backwards compatibility. We'll now support multiple TEP endpoints, each redirecting to a separate router configuration (endpoint selection is done in OPTE). A "default" routing context is kept for backwards compatibility, allowing legacy APIs to remain functional. Thus, it's still used for both underlay routing and the default router configuration (which we'll likely want to remove later).

Internal Changes

• Route lookup keys (route_index) are enriched with an exact match on routing_table_id, used to select the routing context (lpm -> (exact, lpm)).
• This routing_table_id lives entirely inside dendrite, associated with the router configuration uuid provided by the control plane.
• meta.routing_table_id (8 bits) is added to ingress metadata and defaults to 0.
• The claimv6 action now takes a routing_table_id and associates the registered address (TEP endpoint or other loopback) with it, enriching the packet metadata. Non-default tunnel endpoints use ids 1-255. Default endpoints and underlay use 0.
• Apparently adding 8 bits exact to the keys costs 1 entry in IPV4_LPM_SIZE. IPV6 seems to be unaffected.

API Changes

(v14, VERSION_MULTI_ROUTER)

Routers

Method	Path	Body / response
GET	/router	BTreeMap<Uuid, Ipv6Addr>
GET	/router/{router_id}	Ipv6Addr
PUT	/router/{router_id}	Ipv6Addr
DELETE	/router/{router_id}

Routes

Method	Path
GET	/router/{router_id}/route/{ip_family}
GET	/router/{router_id}/route/{ip_family}/{cidr}
POST	/router/{router_id}/route/{ip_family}
PUT	/router/{router_id}/route/{ip_family}
DELETE	/router/{router_id}/route/{ip_family}/{cidr}
DELETE	/router/{router_id}/route/{ip_family}/{cidr}/{port_id}/{link_id}/{tgt_ip}

Architectural decisions (in decreasing order of confidence)

• routing_table_id lives in dpd, not defined by the control plane (or derived from the uuid): This has many benefits, such as not introducing needless coupling, allowing switches to have different IDs and capacities in the long run and not limiting the total number of routing configurations to the minimum number any switch can support. For example, we could easily reassign the routing tables for different purposes in the future without breaking compatibility with the current API.
• Endpoints vs other loopbacks: It seems that we only use loopback addresses for endpoints at the moment, though the switch seems to support them. This PR is a first step in allowing us to separate our boundary services loopbacks from whatever else we might want to use loopbacks for. Endpoints must be unique to select the routing context.
• Default Router: This is the trick that allows backwards compatibility. The legacy API will implicitly use this routing context, both for the underlay routes as well as for the default router configuration. Every other router configuration goes elsewhere. We'll likely want to remove the default router in the future so we know underlay traffic routing is segregated. We can't do this now without re-routing every loopback address registration to it. /route/* returns both underlay and default router routes.
• Names: Naming things is really hard. I accept suggestions here.
  • The router_id name clashes with the Router ID concept. While it's identifying a (logical) router's context, it invites the wrong inference regarding its lifetime and purpose.
  • The default router could be called the system routing table or something else.
  • Depending on how we separate things, we could just talk about TunnelEndpoints instead of "routers".
• Locking and new structs: I put the new Routers inside RouteData so they share a lock because otherwise I'd have to hold an async lock through await points because, for some reason, RouteData is locked with an async lock. Loopbacks now also have a slight dependence on this data, which just means it needs to be locked in order. There are many improvements one can do to the overall structure depending on how we want to deal with endpoints x loopbacks. The current approach is trying to impose as few new architectural decisions as possible.

companion sidecar-lite branch: nsk/662-modular-router-config