Unification of config API

[kyessenov]

Some decisions from a meeting:

1. We do not need transactional semantics for the config updates, a regular key value store is sufficient for our purpose in Mixer and Manager
2. We may need to explore PATCH semantics for updates to allow long-lasting rollouts (since parent config might be concurrently updated).
3. Need to unify the key naming for config artifacts, in particular separate the query for mixer artifacts from the id of the artifact.

@andraxylia @mandarjog @ayj

[ayj]

Open items that require further discussion (please add/remove as necessary):

1. Versioning
2. Validation
3. Consistency guarantees
4. Availability
5. Performance

[kyessenov]

Proposal to unify config documents and config keys

After discussion with @mandarjog @andraxylia @ayj , it seems we can start unification from the consistent conceptual model of the config, putting storage, perf, versioning, validation, and roll-outs aside. We have a common theme in Manager and Mixer that configuration consists of individual documents:

• route rules, destination policies, mesh config in Manager
• policy rules in Mixer, individual aspects in rules
• descriptors and adapter configuration in Mixer

The granularity of the document is dictated by two factors:

• External factor: the user should be able to deploy an individual configuration setting as a single operation. For example, for Mixer that means applying a blacklist with a selector is an operation and should not involve multiple steps such as listing all rules, appending an elements, and then submitting a new list.
• Internal factor: the boundary between selecting rules and selecting within rules should be conceptually simple. It seems that for Mixer rules that means that "selectors" belong to a configuration unit since it operates in the data plane; while "scope", "subject" and "ruleId" are used to find rules that need to be enforced, and are effectively selectors in the control plane. For Manager, "destination" field is the sole field for selecting rules and policies to apply.

Every configuration document should carry its metadata such as the "type" of the document, and a collection of "keys" that are used to select documents. All remaining fields compose the document content, as specified by the proto definitions: that means selector/aspects are mixer rule content, routes/matches are proxy rule content. The configuration key has to be unique globally, and that is the only guaranteed constraint that is easy to enforce with a key-value storage. We should utilize consistent fields such as "type" and "name" for uniformity.

Applying these ideas to current config protos, I suggest we structure them as follows:

1. Route rule:

# meta: note the lack of namespace
type: route-rule
destination: a.default.svc.cluster.local
name: favorite-rule
# content
precedence: 2
match: 
  httpHeaders:
    version:
      exact: v1
route:
  - tags:
      version: v1

Sample REST: get /config/route-rule/a.default.svc.cluster.local/favorite-rule, list /config/route-rule/a.default.svc.cluster.local

2. Destination policy:

# meta: note lack of namespace, name since only one policy per destination
type: destination-rule
destination: a.default.svc.cluster.local
# content
- tags:
     version: v1
  circuitBreaker:
     ...

Sample REST key: get /config/destination-rule/a.default.svc.cluster.local, list /config/destination-rule/

3. Mesh config:

# meta: single object per installation
type: mesh-config
# content
discoveryAddress: istio-manager:8080
authPolicy: MUTUAL_TLS

Sample REST key: get /config/mesh-config

4. Mixer rule:

# meta
type: mixer-rule
scope: global
subject: a.default.svc.cluster.local
name: clueless-pet-rule
# content
selector: source.labels["app"]=="reviews" && source.labels["version"] == "v3"
aspects:
  - kind: quota
    params:
      quota:
      - descriptorName: RequestCount
        maxAmount: 5
        expiration: 1s

Sample REST key: get /config/mixer-rule/global/a.default.svc.cluster.local/clueless-pet-rule

5. Descriptor (need help from mixer folks)

[kyessenov]

@geeknoid @mandarjog @ayj @andraxylia

[mandarjog]

(edited by @kyessenov to fit the story of rate limit)

Mixer has the following additional types.

5. Mixer adapter:

# meta: seems like unique per scope
type: mixer-adapter   
scope: global
name: shiny-quota-adaptor
# annotation: satisfies aspect quota
# aspects:
#  - quota
# content
plugin: memQuota # this can be a statically linked plug-in, a webhook, or an actual plugin
parameters:
  ...

REST key: get /config/mixer-adapter/global/shiny-quota-adaptor

edited by @mandarjog to align with api spec /scopes/{scope}/adapters/{impl_name}/{config_name}
@kyessenov: this misses my point of strict separation of content from name: memQuota does not belong to the name here; it's clearly an implementation detail. It's even called "impl".

I find it useful to think about this as a code statement that instantiates a plugin to satisfy an aspect:

global.shinyQuotaAdaptor := memQuota(parameters).(quota)

6. Mixer descriptor:

# meta: seems like unique per scope
type: mixer-descriptor
scope: global
name: quota
# content
- name: RequestCount
  rate_limit: true

REST key: GET /config/mixer-descriptor/global/quota/RequestCount
edited by @mandarjog to align with api spec /scopes/{scope}/descriptors/{descriptor_type}/{descriptor_name}

@ZackButcher @geeknoid @douglas-reid does this make sense?

[geeknoid]

​We're also going to need to define dictionaries within config...​

…

On Fri, May 5, 2017 at 2:53 PM, mandarjog ***@***.***> wrote: Mixer has the following additional types. type: mixer-adapters # note no subject scope: global name: rate-limit type: mixer-descriptors # note no subject scope: global name: rate-limit — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVucHR7XIHOKgvLjHU_IBehvw2FRa_c4ks5r25pTgaJpZM4NP88l> .

[mandarjog]

type: mixer-aspects-version-rule  
scope: global
subject: ns.svc.cluster.local
name: rate-limit
version-rule:
- version: v1
   weight: 50
- version: v2
   weight: 50

type: mixer-adapters-version-rule   # note no subject
scope: global
name: rate-limit
version-rule:
- version: v1
   weight: 100
- version: v2
   weight: 0

[kyessenov]

what is this last post about?

[mandarjog]

Adapters are grouped on the 'impl' key And descriptors have types. Descriptor types are already used as keys. https://github.com/istio/api/blob/master/mixer/v1/config_api_swagger.yaml Look at replaceAdapterConfig and replaceDescriptor Using "iPhone 9";

…

On May 5, 2017, at 3:45 PM, Kuat ***@***.***> wrote: These configs are enormous, let's think how to break apart adapters and descriptors. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[douglas-reid]

Re: naming

is the name for the descriptor:

• /config/mixer-descriptor/global/quota/RequestCount (or perhaps: config/mixer/descriptor/global/quota/RequestCount) or
• RequestCount or
• a completely different namespaced thing (like istio.io/api/request_count)
  ?

I feel like we want the resource names to be the first (and we should have that reflect in the documents themselves). At least, we should be consistent with: https://cloud.google.com/apis/design/resource_names

[kyessenov]

I'm not sure what the descriptor actually does. Is there significance to the name RequestCount, or it is just chosen for human-readability? The config key (the unique name) has to include the scope since Mixer's overriding semantics is encapsulated to the scope/subject hierarchy.

[kyessenov]

@mandarjog please stop editing my examples :) I thought quota as a name is important since it's just a symbolic name to connect an aspect to an implementation. Why did you rename it to quota-default?

[douglas-reid]

@kyessenov my naming thought applies to more than just descriptors.

in a bunch of these examples, we could leave out the lines for type, destination,scope, subject and just have one name that is the relative resource name.

So, name: /v1/config/mixer/rules/global/a.default.svc.cluster.local/clueless-pet-rule or name: /v1/config/destinations/a.default.svc.cluster.local.

[kyessenov]

True, but I think there is benefit to structured names. It's immediately clear what list operations do if you break the name into resource type, and primary key properties.

[mandarjog]

quota-default reflects the fact that there is a 1:n relationship between the adapter impl and the name of the configuration block. It is also *not* the name of the aspect. The impl is implicitly connected to the aspect because it registers itself as such. That name is important because the aspect can use it to explicitly associate with the adapter. Re: editing "your" example, looks like I wrote that comment originally. Edit it back if you need to, now that you know the reason. Using "iPhone 9";

…

On May 5, 2017, at 8:35 PM, Kuat ***@***.***> wrote: @mandarjog please stop editing my examples :) I thought quota as a name is important since it's just a symbolic name to connect an aspect to an implementation. Why did you rename it to quota-default? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kyessenov]

Ok, updating examples to reflect that there could be multiple adaptors for an aspect.

[kyessenov]

Regarding expicit/implicit declaration of the fact that memQuota satisfies quota aspect: I think either way is fine with statically-linked plugins. However, for a separate system than mixer, it would be nice to be able to query adaptors for an aspect without having to run mixer to get that information.

[ayj]

The association of rule (route/destination/mixer) to service (destination/scope/subject) is currently encoded in the rule. Does it make sense for this association to be distinct resource?

An association would associate a service (or set of services via labels) to a set of rules based on service name, scope, etc. Associations could map one or more service to one or more rules. This composability might be easier user to manage than, for example, creating N rules with identical content except for the subject.

This might also be useful for versioning and staged rollouts where rules could be considered immutable and only the association is updated (or new associations added), e.g. Association maps 5% of services to RulesV1 and 95% to RulesV2.

[kyessenov]

I think it's an interesting idea to think about an "association rule" (presumably not an individual binding of a rule to a service but some rule saying "match these ____ services to these ____ rules"). Mixer design uses config overrides that are scoped by authority. A cluster admin rule overrides a service producer rule, which is reflected by the fact that the higher authority "scope" rule is used for the same "subject". I'm not sure how to express that easily with more general associations.

[ayj]

Yes, service-to-rule binding could be N:N. Wouldn't authority belong with the binding with this model? It could either be an explicit field in the binding resource or perhaps the binding themselves are organized under hierarchical scope/authority.

[kyessenov]

Is the below example similar to what you have in mind:

# meta
type: mixer-rule
name: clueless-pet-rule
# content
selector: source.labels["app"]=="reviews" && source.labels["version"] == "v3"
aspects:
  - kind: quota
    params:
      quota:
      - descriptorName: RequestCount
        maxAmount: 5
        expiration: 1s

# meta
type: mixer-rule-binding
name: apply-to-default
# content
selector: service == "*.default.svc.cluster.local" && rule == "clueless-pet-rule" # select which service and rules to apply
authority: root # utmost authority, root decision precedes decisions by lower authority

The two selectors are different: the rule selector applies to data path (request attributes) while the binding selector applies to the control plane (hierarchy of services and authorities, collection of rules). I'd want to hear a viable example where data path dictates the control plane rule - I can only come up with the following: "if a request contains a magic string, apply the rule that increases the quota for the rate limit".

[rshriram]

While you guys discuss the high level constructs and ideas, how about doing the basic sensible cleanups that should have been there in the first place for istioctl ?
istio/old_pilot_repo#710

The command set is confusing to say the least. It blatantly and very obviously misses the simplest of usability aspects. Istioctl create and istioctl mixer rule create. I suggest we fix this first (hack or not) and then worry about all the internal formats.

[kyessenov]

Abandoning since beta-mixer is completely revamping its terminology.