URLLib3 is pinned below version 2

[LewisCowlesMotive]

Expected Behavior

Stable dependencies that don't downgrade without explanation.

Actual Behavior

Urllib3 downgrades from 2.x after installing datadog-lambda

Steps to Reproduce the Problem

1. poetry init
2. poetry add boto3
3. poetry install
4. poetry lock
5. poetry show | grep urllib
6. poetry add datadog-lambda
7. poetry install
8. poetry lock
9. poetry show | grep urllib

Specifications

The below details are provided as asked for, but are irrelevant

• Datadog Lambda Layer version: N/A
• Python version: 3.11

Stacktrace

N/A; this bug is in the packaging 🙃

[astuyve]

Hi, thanks for reaching out!

Yes this is the result of #329 and caused by a dependency on botocore, which also is pinned below v2 of urllib3. This seems to cause other issues with amazon's internal infrastructure.

You can follow along this thread for more: boto/botocore#2926

[LewisCowlesMotive]

Their latest comment is confusing then boto/botocore#2926 (comment)
It mentions now being pinned below 2.1

[LewisCowlesMotive]

It seems upon further examination, your python range may be why you have a simpler limit
https://github.com/boto/botocore/blob/1.31.63/setup.py#L29
P.S. I love that I didn't have to go through so many files in this repo to get to a dependency. Botocore seems mildly cursed in that regard.

[astuyve]

Only for 3.10 and above.
This library has to support 3.7 and above, which is why we can't change the pin yet.

I'm more than happy to remove it when AWS can fully support all current Lambda runtimes.

Edit:
Just saw your new comments, yeah. Happy to fix this when they do 😃

[LewisCowlesMotive]

Do you think you'd accept a patch for using https://python-poetry.org/docs/dependency-specification/#multiple-constraints-dependencies so that 3.10+ can use <2.1?

I run 3.11; and it might be appreciated

[astuyve]

We would but that change is complex as our build process isn't presently aware of variations across multiple poetry.lock files, which may be tricky to add. Additionally it may complicate our coordination with dd-trace-py. But I'm happy to test out whatever you'd like to contribute.

[LewisCowlesMotive]

lets see if / where it blows up and revisit 😄 ; PR above #377

[LewisCowlesMotive]

@astuyve it looks like it passes tests; flakes on integrations are AWS permissions related, no idea how you'll overcome those, but they do not relate to the python code.

[astuyve]

Yeah, I have to run them myself. I'll let you know, thanks!

[astuyve]

Hi, sorry this change seems to have broken python 3.10 with the same error reported in #329, see the screenshot:

I'll investigate if we need to continue patching requests for hybrid tracing, and see if we can tighten this to python 3.11 only.

Thanks!

[LewisCowlesMotive]

So you're importing from an internal package and it breaks things... Can you describe what that import is used for?

[astuyve]

Hi, no sorry for the missing context - it's thrown by boto3 (same error we've been discussing) when it is imported from the runtime.
The patch looks good in 3.11 though.

[astuyve]

Hi, to update - I created a brand new serverless application using python3.10, boto3 (from the 3.10 runtime), and my own version of urllib3==2.0.7, and without datadog's instrumentation.

It crashes with the same error:

Python 3.11 works fine though.

Looks like it has a newer version of botocore/boto3:

link
Instead I've opened #380 which allows urllib3 2.x for python 3.11 only and marks it as optional.

[astuyve]

Additionally I've opened aws/aws-lambda-base-images#118 to track this fix in the Lambda runtime itself

[LewisCowlesMotive]

Can I ask when the new release will be cut? #380 looks like it solves it.

[astuyve]

Should go out today!