Thread-local append_keys for logger

[Dunedan]

I have some code running on AWS Lambda which uses threads and I want to use different values for additional keys with append_keys() per thread to be able to group logs generated by different threads. I haven't seen anything in the docs suggesting that this might be possible. Am I missing something or is that indeed not possible right now?

The feature I'm looking for would be something like structlog's Thread-Local Context.

One way I can imagine the interface for using that would be something like shown below, which would append/remove additional keys only for logs generated by the current thread:

 logger.append_keys_thread_local(order_id=order_id)
 logger.remove_keys_thread_local(["order_id"])

Is that something which would make sense for AWS Lambda Powertools?

[heitorlessa]

Hey Daniel! You’re correct, this isn’t possible today as append_keys persists the keys into the Logger. This means it’s not thread safe. Would you like to create a RFC and we can discuss potential implementations & trade-offs? I can see a few ways to solve this: unbounded logger with a copy of the current log_format so they don’t update bi-directionally like Child Logger, etc self.log_format is what makes it non-thread safe: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/aws_lambda_powertools/logging/formatter.py

…

On Fri, 28 Jan 2022 at 07:59, Daniel Roschka ***@***.***> wrote: I have some code running on AWS Lambda which uses threads and I want to use different values for additional keys with append_keys() per thread to be able to group logs generated by different threads. I haven't seen anything in the docs suggesting that this might be possible. Am I missing something or is that indeed not possible right now? The feature I'm looking for would be something like structlog's Thread-Local Context <https://www.structlog.org/en/stable/thread-local.html>. One way I can imagine the interface for using that would be something like shown below, which would append/remove additional keys only for logs generated by the current thread: logger.append_keys_thread_local(order_id=order_id) logger.remove_keys_thread_local(["order_id"]) Is that something which would make sense for AWS Lambda Powertools? — Reply to this email directly, view it on GitHub <#982>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBDB6SPIRKTGDVKDC3DUYI5D7ANCNFSM5M76XFCA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[heitorlessa]

And yes it would make sense in Powertools. I can totally see people using Futures to handle a batch of orders and wanting to temporarily persist "order_id" and "thread_id" instead of using extra={} for every log statement (thread-safe).

The interfaces you suggested could create a temporary copy of log_format, isolate them per thread_id, so you wouldn't need multiple unbounded loggers.

RFC would be great to flesh out these details and understand common use cases.

Thanks a lot for kicking this off :)

[Dunedan]

Here is the RFC: #991
I'm still not sure how the interface to use it would look best, but I guess we can figure that out when discussing the RFC.
Looking forward to comments there.

[Dunedan]

I just noticed that the same topic probably applies to metrics as well.