Define per-service mixerclient runtime configuration

[ayj]

A single mixerclient may serve multiple destination services. Aspects
of runtime configuration must therefore be defined per-service,
e.g. Quota, APISpec. Express this per-service configuration as a map
of config elements indexed by the fully qualified destination service,
e.g. destination.service.

As an implementation note, the mixerclient could derive
destination.service using the host header. Alternatively, Pilot can
populate inbound route's opaque_config with the canonical destination
service name. The mixerclient can use the destination service
name as key into the runtime configuration to determine the
approptiate quota, apispec, etc. This has the nice property of
automatically handling all domain name variations, e.g. my-service,
my-service.my-namespace, my-service.my-namespace.svc.local.

[ZackButcher]

Don't renumber fields, add the already used field numbers to the reserved list inside the message. (Renumbering fields is a breaking change in protos.)

[qiwzhang]

I believe nobody is using this proto file so we can renumber fields any way we want.

[ayj]

+1 for raising this point during review. In this particular case, the configuration isn't used yet and is conceptually still in alpha. We definitely shouldn't make this kind of change once its used by proxy.

[ZackButcher]

Same, this is a breaking change, use 8 instead and add 5 to the reserved list in this message.

[qiwzhang]

What about single backend case? Does "Host" header always present even for single backend?

[ayj]

Yes, I'm assuming we can make that guarantee. We could add a default config as part of the map if necessary later.

[qiwzhang]

Maybe adding a default config outside of the map. Otherwise, we have to use empty string as key for default config in the map.

[ayj]

I would prefer to add that if/when its needed. Alternatively, we could use a special key that is not a valid service name, e.g. ":default".

[qiwzhang]

If we rely on "Host" header to find the config, I am afraid of many of our tests will fail. They are just using "curl" or go http client, without "Host" header in the request.
I think using default config is a "normal" case, using multiple configs is a special case. So I prefer that there are a default config and a map. If "Host" header presents, cheek the map, found, use that config, otherwise, use default config.

[ayj]

I added a default destination service name which can be used if client request doesn't specify a host. This allows the MixerControlConfig map value to be shared between the default and service specified case.

[qiwzhang]

If we are going to use Envoy xDS to distribute these config, LDS update is very expensive, RDS is fast. If any per route config change, you prefer to update RDS than LDS. But with this approach, you have to update LDS. @kyessenov

[rshriram]

But you replicate the same definition across 100s of routes. Also, fault filter is updated via LDS. So cost is same

[qiwzhang]

@rshriram I did not understand of your "replication" statement. could you clarify?
My statement is to compare with original plan of using per-route opaque data to send MixerControlConfig distributed by RDS and MixerFilterConfig send by LDS will not have this map. In this way, if a per-route config is changed, only RDS updates it. With a map in the MixerFilterConfig, the map need to be changed, and LDS needs to send the updated map.

[rshriram]

Right.. and my point was that there can be 100s of routes.. So, if we ship API definitions as opaque configs per route, then the amount of traffic for RDS will be pretty high and slow things down, as we have API definitions embedded in each route. While LDS may be slower, its fine for this use case, because API definitions do not change as frequently as routes.

[ayj]

Mixerclient config changes should be relatively rare in practice. Distribution through filter config (via LDS) shouldn't be an issue. Managing control config per filter rather than per-route also avoids unnecessary coupling of the config with Envoy configuration model. Pilot will likely always be responsible for bootstrapping mixerclient (via filter config) and providing some basic identity information (destination.uid, per-route destination.service). On the other hand, quota/apispec might eventually be managed by another component with better support for staged rollout.

[qiwzhang]

@rshriram I am more worry about tearing down/starting up listeners inside Envoy than amount of data in the wire. It is very expensive. You don't want to do that often.

[qiwzhang]

Another issue: per-route forwarding flag and per-route forwarding attributes are removed. Are you sure we are not using it? or we don't need to use it in the future? @kyessenov

[ayj]

We can add them later if they're needed. At present, forwarding attributes only includes per-proxy identity (source.ip and source.uid) and not per-service identity.