Skip to content

Aj/proactive init #343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 28, 2023
Merged

Aj/proactive init #343

merged 6 commits into from
Jun 28, 2023

Conversation

astuyve
Copy link
Contributor

@astuyve astuyve commented Jun 23, 2023

What does this PR do?

This adds support for tagging proactively initialized lambda functions.

Motivation

Testing Guidelines

I built a pre-release and tested it, verifying we're correctly tagging proactively initialized lambda functions
image

Additional Notes

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)

@astuyve astuyve requested a review from a team as a code owner June 23, 2023 17:39
@astuyve astuyve changed the title Aj/practive init Aj/proactive init Jun 26, 2023
Copy link
Contributor

@joeyzhao2018 joeyzhao2018 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@astuyve astuyve merged commit 9aef51d into main Jun 28, 2023
@astuyve astuyve deleted the aj/practive-init branch June 28, 2023 18:12
if not _lambda_container_initialized:
now = time.time_ns()
if (now - init_timestamp_ns) // 1_000_000_000 > 10:
_cold_start = False
Copy link

@pvicente pvicente Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @astuyve @joeyzhao2018 should it also be _cold_start = True? I'm getting use cold_start stats from apms and with this change I will need to take into account new proactive_initialization= true to measure it like before.

Based on AWS docs I'm not sure if in this situation cold_start should be true or false. Any thoughts?

Thanks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @pvicente - We're actively working with AWS on documentation for this and I can't say more right now, but you should open a support ticket with them and tell them AJ Stuyvenberg sent you ;)

A bit more about what I can say:
With this change you'll still get cold_start:true if the AWS Lambda function actually experienced a cold start, like so:
image

However if the function is proactively initialized, then it's not a cold start in the sense that there was no additional latency caused by sandbox initialization (as init occurred seconds, hours, or minutes before the first request was served by the sandbox).

We now tag those correctly, so when we use APM to select this python invocation:
image
We can see that initialization occurred 3m59s before the function was executed:
image

I hope I've helped!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation @astuyve 🥇

_cold_start = False
_proactive_initialization = True
else:
_cold_start = not _lambda_container_initialized
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@astuyve I think on this case _proactive_initialization = False see my case from APM stats:

Screenshot 2023-07-07 at 14 28 28

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We initialize _proactive_initialization to False on line 9 so we shouldn't need to redeclare that here.

In this case rolling up to top 10 is skewing the results:
image
I can see something similar in one of my orgs:
image
But when I remove the pie chart grouping, I see many cold_start: true spans:
image

However for clarity, line 27 should always be false as we're already inside of a conditional checking for _lambda_container_initialized == false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants