net.add_addr_allfibs=1 behaviour deprecation

Dear FreeBSD users,

I would like to make net.add_addr_allfibs=0 as the default system behaviour and remove net.add_addr_allfibs.
To do so, I would like to collect use cases with net.add_addr_allfibs=1 and multiple fibs, to ensure they can still be supported after removal.

Background:

Multi-fib support was added in r178888 [1], 12 years ago. Addition of interface addresses to all fibs was a feature from day 1.
The `net.add_addr_allfibs` sysctl  was added in r180840 [2], 12 years ago.

Problem:
The goal of the fib support is to provide multiple independent routing tables, isolated from each other.
`net.add_addr_allfibs` default tries to shift gears in the opposite direction, unconditionally inserting all addresses to all of the fibs.

It complicates the logic, kernel code and makes control plane performance decrease with the number of fibs.
It make impossible to use the same prefixes in multiple fibs, which may be desired given shortage of IPv4 address space.

I do understand that there are some cases where such behaviour is desired.
For example, it can be used to achieve VRF route leaking or binding on address from different fibs.
I would like to collect such cases to consider supporting them in a different way.


The goal is to make net.add_addr_allfibs=0 default behaviour and remove net.add_addr_allfibs.
It will simplify kernel fib-related code and allow bringing more fib-related features. It will also improve fib scaling.


Timeline:
Aug 1: summarising feedback and the usecases, decision on proceeding further
Aug 20 (tentative):  patches for supported usecases
Sep 15 (tentative):  net.add_addr_allfibs removal.

[1]: [base Contents of /head/sys/net/route.c](https://svnweb.freebsd.org/base/head/sys/net/route.c?revision=178888&view=markup)
[2]: [base Diff of /head/sys/net/route.c](https://svnweb.freebsd.org/base/head/sys/net/route.c?r1=180839&r2=180840&)

/Alexander