Make lambdas able to change other trace attributes

[FrnchFrgg]

Some plotly trace attributes can be arrays to be per-point. Extracting those out of the object returned by the lambda could enable using different attributes (say, text or marker.size) for specific points (local extrema come to mind).

[FrnchFrgg]

What I envision: in https://github.com/dbuezas/lovelace-plotly-graph-card/blob/master/src/plotly-graph-card.ts#L585 check for an attributes entry, or settings or whatever you find most suitable, and merge it with the trace at https://github.com/dbuezas/lovelace-plotly-graph-card/blob/master/src/plotly-graph-card.ts#L603 (probably in prioritary position but not overriding x and y).

The user would thus specify the lambda

(xs, ys) => { return {
    x: [0, 1, 2],
    y: [3, 5, 1],
    attributes: { text: ["a", "b", "c"] }
}

It would not be very compatible with the solitary points patching because of the index shifting, but maybe in those cases it should be the users responsibility to cope (by ensuring there are no such points to patch maybe).

Another possible API would be to just accept whatever member of the returned object and merge that directly instead of a sub-member. In that case the lambda would be:

(xs, ys) => { return {
    x: [0, 1, 2],
    y: [3, 5, 1],
    text: ["a", "b", "c"]
}

[dbuezas]

Good idea! I'm also thinking about passing more stuff as parameter (like the visible range)
BTW, I have a branch that admits lambdas everywhere (not only inside entities), I think we should revive that.
I'd like to think this through a bit, maybe lambdas anywhere should be able to return any value or object in any part of the yaml, and that should be merged inside that very part of the yaml.

[FrnchFrgg]

I suppose that having lambdas everywhere is a good way to go too. I looked at your branch and it is clever, but a) you test twice for strings starting with lambda, and b) I think that if the window.eval throws you should return the string as-is.

[dbuezas]

I think that if the window.eval throws you should return the string as-is.

I've been moving to show the errors in the screen to improve the yaml editing experience.
I intend to use a proper json validator at some point.

I think in the experimental branch I used a prefix to indicate "this is a function", kind of like !lambda in esphome (except that that exact modifier requires a yaml plugin in HA, so it will have to be slightly different. Maybe $lambda or $fn or something like that.

[FrnchFrgg]

You used the prefix lambda but any prefix unless very obnoxious could very well be a legit parameter. For instance, lambda could be the name of a sensor, or some unit, or something like that.

[dbuezas]

One thing that I haven't quite figured out yet, are the parameters to pass to the lambdas.
Ideally, I'd like them to be uniform all across, so I was thinking about passing an object that contains absolutely everything and it is organized by key. If you are inside an entity, you get an extra one with the "current" trace.
Also, it may be interesting to pass the full hass object, and evaluate the full config on each plot. This way one could add the state of an entity to labels, etc.

[midiwidi]

It would be really great and add so much more flexibility if attributes/parameters could be specified dynamically bases on the trace data or other HA entity state/attributes. That would be very useful for x/y range limits, positions of annotations / shapes / images ... to name a view.
You said "lambdas everywhere" would be a way of doing it. With lambdas (if I understand this correctly) the data processing happens in the JavaScript domain. Would it also be possible to do that in the HA / Jinja2 domain using templates?

[dbuezas]

It would be really great and add so much more flexibility

I agree! I actually never moved forward for that because I assumed almost nobody would go this deep. I'm pleased to see how much I underestimated this card's users!

---

With lambdas (if I understand this correctly) the data processing happens in the JavaScript domain.

Correct

---

Would it also be possible to do that in the HA / Jinja2 domain using templates?

You can do that already with:

• Javascript templates: https://github.com/iantrich/config-template-card
• Jinja2 yaml templates: https://github.com/gadgetchnnel/lovelace-card-templater

[dbuezas]

Btw, I welcome yaml examples of how you envision the API.
I'm thinking of:

entities:
  - entity: sensor.my_other_sensor
     fn: ({x, y, attributes, ...a_lot_more, vars, ... }) => {
         vars.maxOther = Math.max(...y) // not returning anything has no effect
     }

  - entity: sensor.mysensor
     y: $fn ({x, y, attributes, ...a_lot_more, vars, ... }) =>
         parseFloat(y)*2 + vars.maxOther 
  - entity: sensor.mysensor_2
     fn: ({x, y, attributes, ...a_lot_more, vars, ... }) => ({
         y: parseFloat(y)*2 + vars.maxOther // returns an object, so this is merged with the entity
     })
title: $fn ({hass}) => hass.states["sensor.any_sensor"].state # that's the current state
layout:
  yaxis:
    range: $fn ({states}) => [Math.min(...states["sensor.mysensor"], 100]

[dbuezas]

The one thing I need to figure out is an elegant way to handle merging for "root lambdas", where the key is fn.
The attributes defined before the fn should be overwritten by the object keys returned by fn, but those defined after it should override the fn.
Not very important, but it would make it a tad more intuitive

entities:
  - entity: sensor.my_other_sensor
    name: name1  # lambda overrides yaml
    fn: |- 
      ({x, y, attributes, ...a_lot_more, vars, ... }) => ({
         name: name2, 
      })

entities:
  - entity: sensor.my_other_sensor
    fn: |-
      ({x, y, attributes, ...a_lot_more, vars, ... }) => ({
         name: name,
      })
    name: name2  # <-- yaml overrides lambda

Probably I just need to construct a new object, deep merging each key at time

[midiwidi]

It would be really great and add so much more flexibility

I agree! I actually never moved forward for that because I assumed almost nobody would go this deep. I'm pleased to see how much I underestimated this card's users!

With lambdas (if I understand this correctly) the data processing happens in the JavaScript domain.

Correct

Would it also be possible to do that in the HA / Jinja2 domain using templates?

You can do that already with:

* Javascript templates: https://github.com/iantrich/config-template-card

* Jinja2 yaml templates: https://github.com/gadgetchnnel/lovelace-card-templater

Thank you for the links. I wasn't aware of them. Especially the second one (lovelace-card-templater) sounds very useful for me.

I'm playing around trying to find a nice design to display weather forecast data. I put this together with static parameters

There are several parameters which need to change dynamically depending on the data and with the lovelace-card-templater I might be able to achieve that.

[dbuezas]

That looks really cool!

[midiwidi]

Regarding the lambdas:

Thinking more about my question if Jinja2 templates could be used, I realise that in most cases lambdas would be the best way to do it. This is because in most cases the data series contain historical data which is not that easy to access via Jinja2 templates. One would have to issue SQL queries to do that. So yes, I think lambdas are the way to go.

From your examples above, it looks like your plan is to provide 1 lambda function which would output all the different attributes. With this approach you have the problem to merge the attributes specified in the lambda with attributes specified in YAML.

entities:
  - entity: sensor.my_other_sensor
    fn: |-
      ({x, y, attributes, ...a_lot_more, vars, ... }) => ({
         x: ....
         y: y*2-5,
         name: name,
         other attribute: value
      })

What about leaving the attributes in YAML exclusively and use multiple lambdas to modify each?

entities:
  - entity: sensor.my_other_sensor
    x: $fn ({x}) => ...
    y: $fn ({y}) => ({y*2-5})
    name: $fn ({attributes}) => ({...})
    other attributes: $fn ({...})

Then you don't have to merge. You just take the static value or evaluate the lambda.

It gets a bit trickier in the other sections such as layout or shapes. In one of your examples from above you had

layout:
  yaxis:
    range: $fn ({states}) => [Math.min(...states["sensor.mysensor"], 100]

What is states? An array which contains the y data of all traces which are using the first y axis?
Could the data come from a different source not being plotted on that axis?
Maybe one could specify the whole path

layout:
  yaxis:
    range: $fn ({}) =>   
          [Math.min(sensor.not_plotted_01.y], 
           sensor.not_plotted_02.attribute01]

Maybe that could also be useful for plotting 1 sensor versus another one, e.g.

    . . .
    x: sensor.my_voltage_sensor.y
    y: sensor.my_current_sensor.y
    name: "V-I transfer function"
    . . .

[dbuezas]

it looks like your plan is to provide 1 lambda function which would output all the different attributes.

The plan is to do both:

• If one prefixes a value with $fn, then that's the value of the stated key (e.g: name: $fn (...)=> ...)
• If one uses it as the key, then it will be merged

[dbuezas]

Could the data come from a different source not being plotted on that axis?

That's the plan, one of the params will have all fetched data. There are a couple of issues, because multiple series can ve rxtracted from the entity name may (statistics, different periods, states and attributes)

[dbuezas]

Maybe that could also be useful for plotting 1 sensor versus another one

You nailed it. That's exactly why I originally started playing with the idea, driven by this request #21

[FrnchFrgg]

How do you plan to solve the chicken-and-egg problem, that is if some lambda function sets something which changes the visible range thus the fetched data and may thus change what the lambda would return...

Also, I think that having separate attributes being able to be computed with the $fn prefix, while maybe interesting, is much less important than making lambda merge with the trace config all attributes returned in the object. Most of the times, the same code will generate every interesting config at the same time, especially for configuration entries that are arrays to do something different to every point.

If the only mechanism we have is $fn prefix, then a loop finding "interesting points" would have to be copied and run for each of textposition, text, marker.size.

And the solution to say "just make the parent object a $fn" is very cumbersome too, because that means one will have to specify all configuration in the lambda instead of using standard yaml for constant config.

And it is easier to just merge the lambda result's attributes (if the result is not iterable, as is currently checked). This is a low-risk and small change.

[zanna-37]

The plan is to do both:

• If one prefixes a value with $fn, then that's the value of the stated key (e.g: name: $fn (...)=> ...)
• If one uses it as the key, then it will be merged

Isn't it better, easier, and cleaner to only stick with the first?

[dbuezas]

Yeah, you're probably right. One can still return an object in the parent element if that's needed

[FrnchFrgg]

With the recent filters work, I think that it would be better to merge into the trace a data.config field if present. This is a small scale and low-risk change, will enable advanced uses (e.g. define per-point radius from the data, or add text to only some points) and maybe some built-in filter can benefit from being able to change the entry configuration too.

A bit of a chicken-and-egg problem: if one returns config.unit_of_measurement in the filter should it take precedence over the trace configured one (which already takes precedence over what's stored in the entity) ? I would say yes but that may be debatable. And what if one specifies both config.unit_of_measurement and attributes.unit_of_measurement ? I'd say the latter takes precedence but that's starting to become complicated.

[dbuezas]

Ohhh, config is a very good name! Laugh about it, but I really struggled with naming that object! Particularly because it is just the attributes of the last state, so it is actually very related to state.attrubutes.
I think the precedence should be:

1. Trace.unit_of_measurement (manually set in yaml)
2. Filter data.config.unit_of_measurement
3. Last raw entity from HA.attributes.unit_of_measurement

This way, if an inbuilt filter changes it, the user can still override it easily.

But why do you think it should ve merged inside the trace instead of having it being an "ephemeral" value inside the data going in and out of filters?

[dbuezas]

Btw, i intend to make data.vars available to the future lambda style functions that will be available outside of entities (like in layout, etc). That way one will be able to do things like setting an axis range based on entity data, but without having to expose the raw fetched data or any other internal stuff. Wdyt?

Another thing I am not sure yet is what to do if one needs the history of one entity but doesn't want to plot it. One could hide it via plotlyjs parameters, but that would disable fetching and whatnot. WDYT about another trace option like fetch_but_dont_plot: true? Of maybe discard it if an inbuilt filter returns a special value instead of the output data?

[dbuezas]

Maybe meta instead of config. It is in the end data about the data

[dbuezas]

I'm starting to spend some thoughts on the implementation.
The approach I have in mind is to parse and evaluate the yaml on each plot update.

• Create an empty vars object
• Grab the defaults object and merge into each entity (and axis) without evaluating it.
• Traverse the rest of the yaml tree, depth first
  • If a node is an entitiy, evaluate the fetch modifyiers first (entity name, offset, fetch states only, attributes or statistics?, which period?, global minimum_response and significant_changes_only ).
    • Then fetch data
    • Evaluate the rest in depth first order, passing the data object with xs, ys, etc
  • Else, Each other fn encountered is evaluated passing vars and hass only

Finally the data, layout and config objects to pass to plotlyjs can be generated. Alternatively, these are generated while evaluating the tree.

Docs wise:

Functions evaluation order only matters for vars, and will will be done top to bottom, with two exceptions:

• If they affect what is to be fetched, they will be evaluated right before each entity ([list of keys here])
• Defaults will be evaluated right before the object they configure. If the object they configure is not explicitly present in the yaml, they will be evaluated in the position the defaults are found.

You are quite right that it is non trivial, but I think it will be a fun algorithmic challenge :)

I think the corner cases of function evaluation order that require explanations will be very seldom encountered by users.

@FrnchFrgg what do you think? Do you have a better idea? Any caveats or gotchas you can see?

[dbuezas]

Alternatively, throw an exception if the evaluation order would be different than the written order. This may be a better way to avoid confusing behavior

[dbuezas]

Nice, this is an absolute brainfuck, I like it a lot

[dbuezas]

First prototype shows some promise:
https://gist.github.com/dbuezas/41846c03030b7fb8fc1b5d9aa610911a

[dbuezas]

I think I'll have to rearchitect the fetch and plot functions, but they'll likely end up better than they were. What I have in mind is that the fetched and transformed data (i.e through filters), will end up inside the parsed_config as x and y. Then to plot it is just a matter of passing the whole thing to plotly as-is.

[dbuezas]

It should be general enough that one could generate the list of entities via code, and generate them with code inside too.
And execution strictly follows the yaml top to bottom (except for the defaults object ofc,).

There will be some complexity with the auto generation of axes (each different unit_of_measurement gets its own yaxis) but i think that will just be a second pass over the evaluated "yaml tree".

[dbuezas]

In the master branch the processing of the data, and merging different defaults in different places has become quite complex and the code hard to follow (very spaghetti like).
I need to unravel and simplify all of that and make data dependencies very explicit for this to work. Because of that I made myself a rough dependency graph that I'll post here for future reference.

src

[dbuezas]

The approach I'm following to avoid forward dependencies in function evaluation (*) is that no functions are allowed on parameters that appear later-than-needed in the yaml. An error will tell the user "move this up in the yaml". This should result in less user confusion as the execution order is always known (top to bottom), without getting in the way of plots that don't use functions to set some parameters.

@FrnchFrgg you were right this is quite a big undertaking, I hope I finish it within the next couple of weeks, before my schedule has to catch up with adult life again. Code wise I'll be exchanging unnecessary complexity (bad code) with useful complexity.

(*) e.g the offset to be defined before a filter is evaluated, because the offset affects what gets fetched

[dbuezas]

I'm getting closer to the first beta. In some sense it feels similar to implementing an interpreter. It's quite a rewrite, probably around 50% of all the code different, and kind of dense. Once it starts running again I'll make another pass to try to make it a bit more intuitive.
The hardest part was dealing with the interdependencies between defaults and different parts of the yaml. The yAxis titles for example are by default the unit_of_measurement of each trace. And each unit is by default grabbed from the attributes of the entity. So some defaults ended up being $fn themselves, which makes a mess if what they need is also an $fn, particularly because the order of evaluation of each $fn is enforced to be top to bottom (in case vars is used).
Another couple of late evenings and there should be some screenshots already.

[dbuezas]

One negative effect of my approach is that fetching needs to be done sequentially, in case somebody uses $fn to compute an offset, entity name, statistic or period. One could proceed with the next trace until a $fn or filter is found.

[dbuezas]

First time it runs: the title of the plot comes from the evaluated entity. 🎉

[dbuezas]

With arbitrary functions, one can use entity data to fill the other axes of plots. This means other more fancy stuff will be possible very soon!

Temperature vs Humidity vs Pressure

[dbuezas]

[dbuezas]

[dbuezas]

Indicators

[dbuezas]

[midiwidi]

The new possibilities are awesome. Thank you so much for refactoring the code to make this possible.

[dbuezas]

[dbuezas]

type: custom:plotly-graph-dev
entities:
  - entity: sensor.openweathermap_temperature
    period: day
    statistic: max
    x: $fn ({ ys,vars }) => ys
    type: histogram
    name: max
    opacity: 0.8
    marker:
      color: rgb(206,43,30)
  - entity: sensor.openweathermap_temperature
    period: day
    statistic: mean
    x: $fn ({ ys,vars }) => ys
    type: histogram
    name: mean
    opacity: 0.7
    marker:
      color: yellow
  - entity: sensor.openweathermap_temperature
    period: day
    statistic: min
    x: $fn ({ ys,vars }) => ys
    type: histogram
    name: min
    opacity: 0.6
    marker:
      color: blue
title: Temperature Histogram last 100 days
hours_to_show: 2400
raw_plotly_config: true
layout:
  barmode: overlay
  margin:
    t: 20
    l: 50
    b: 40
  height: 285
  xaxis:
    autorange: true

[dbuezas]

My free time ends next week, I hope I get some feedback during the weekend :)

https://github.com/dbuezas/lovelace-plotly-graph-card/releases/tag/v3.0.0-beta

@FrnchFrgg You were very right, this was a titanic effort, but I think it will be worth it.

[dbuezas]

Closed by the release of v3.0.0.

[dbuezas]

@FrnchFrgg, I haven't heard your feedback on the feature, what do you think?

[FrnchFrgg]

I have ported my previous setup to the new system and it works great. It feels really powerful.

At first I thought that the dependency order would cause problems: I need to set several settings in the entity from a single loop, but cannot use a $fn for the whole entity entry because it wouldn't have access to the xs and ys since these are only available after fetching thus after the entity id and time_offset and similar have been specified.

But the vars feature solves it because I can have one loop for the first setting, and the other just piggyback into its work by reusing the stashed away values.

Playing a bit more with it, I feel a lot of the card features could be in fact implemented as $fn now. It is that powerful. (BTW, my gut feeling is that extend_to_present should be a filter, but that is not possible since it would change the default behavior of non-statistics traces).

In particular, I always wanted a way to align the right of the graph to the last statistic entry instead of "now". I now can by using a $fn for the global time_offset (due to dependencies I cannot use the actual xs[xs.length-1] but I know the last known value is snapped at the previous full hour).

All in all it feels like a solid and powerful feature. I didn't have time to look at the implementation yet.

[dbuezas]

Thanks for sharing! The implementation is kind of dense, I'll welcome readability suggestions if you have them.

Regarding the vars, your approach sounds quite appropriate.

On aligning to last timestamp, that's a clever trick! Let me know if you reach its limits and need some inbuilt support.

And finally, about extend_to_present you are absolutely right, I hadn't thought of that and it would simplify the code and make it more intuitive. I may do that change in the future, but will first let people forget all the last breaking changes 😬

[sedado22]

Temperature vs Humidity vs Pressure

Hello, First of all, thanks a lot for your hard work.
Is it by any chance possible that you could share this card configuration ? it would be really helpful to me.
Thanks in advance!

[dbuezas]

3d scatter: #298