Merge branch 'develop' into time2

Author: ran-isenberg

CHANGELOG.md

@@ -4,6 +4,11 @@
 <a name="unreleased"></a>
 # Unreleased
 
+## Bug Fixes
+
+* **apigateway:** support nested router decorators ([#1709](https://github.com/awslabs/aws-lambda-powertools-python/issues/1709))
+* **parameters:** get_secret correctly return SecretBinary value ([#1717](https://github.com/awslabs/aws-lambda-powertools-python/issues/1717))
+
 ## Documentation
 
 * project name consistency
@@ -14,23 +19,39 @@
 ## Features
 
 * **apigateway:** multiple exceptions in exception_handler ([#1707](https://github.com/awslabs/aws-lambda-powertools-python/issues/1707))
+* **event_sources:** extract CloudWatch Logs in Kinesis streams ([#1710](https://github.com/awslabs/aws-lambda-powertools-python/issues/1710))
+* **logger:** log uncaught exceptions via system's exception hook ([#1727](https://github.com/awslabs/aws-lambda-powertools-python/issues/1727))
+* **parser:** extract CloudWatch Logs in Kinesis streams ([#1726](https://github.com/awslabs/aws-lambda-powertools-python/issues/1726))
 
 ## Maintenance
 
-* **ci:** revert custom hw for E2E due to lack of hw
-* **ci:** prevent dependabot updates to trigger E2E
-* **ci:** use new custom hw for E2E
+* apigw test event wrongly set with base64
 * **ci:** limit to src only to prevent dependabot failures
+* **ci:** use new custom hw for E2E
+* **ci:** prevent dependabot updates to trigger E2E
+* **ci:** revert custom hw for E2E due to lack of hw
 * **deps:** bump dependabot/fetch-metadata from 1.3.4 to 1.3.5 ([#1689](https://github.com/awslabs/aws-lambda-powertools-python/issues/1689))
+* **deps-dev:** bump flake8-builtins from 2.0.0 to 2.0.1 ([#1715](https://github.com/awslabs/aws-lambda-powertools-python/issues/1715))
+* **deps-dev:** bump mypy-boto3-appconfigdata from 1.25.0 to 1.26.0.post1 ([#1704](https://github.com/awslabs/aws-lambda-powertools-python/issues/1704))
+* **deps-dev:** bump mypy-boto3-xray from 1.25.0 to 1.26.0.post1 ([#1703](https://github.com/awslabs/aws-lambda-powertools-python/issues/1703))
+* **deps-dev:** bump mypy-boto3-cloudwatch from 1.25.0 to 1.26.0.post1 ([#1714](https://github.com/awslabs/aws-lambda-powertools-python/issues/1714))
+* **deps-dev:** bump flake8-bugbear from 22.10.25 to 22.10.27 ([#1665](https://github.com/awslabs/aws-lambda-powertools-python/issues/1665))
+* **deps-dev:** bump mypy-boto3-s3 from 1.25.0 to 1.26.0.post1 ([#1716](https://github.com/awslabs/aws-lambda-powertools-python/issues/1716))
+* **deps-dev:** bump types-requests from 2.28.11.3 to 2.28.11.4 ([#1701](https://github.com/awslabs/aws-lambda-powertools-python/issues/1701))
 * **deps-dev:** bump mypy-boto3-logs from 1.25.0 to 1.26.3 ([#1702](https://github.com/awslabs/aws-lambda-powertools-python/issues/1702))
+* **deps-dev:** bump mypy-boto3-lambda from 1.25.0 to 1.26.0.post1 ([#1705](https://github.com/awslabs/aws-lambda-powertools-python/issues/1705))
+* **deps-dev:** bump mypy-boto3-xray from 1.26.0.post1 to 1.26.9 ([#1720](https://github.com/awslabs/aws-lambda-powertools-python/issues/1720))
+* **deps-dev:** bump mypy-boto3-ssm from 1.26.0.post1 to 1.26.4 ([#1721](https://github.com/awslabs/aws-lambda-powertools-python/issues/1721))
+* **deps-dev:** bump mypy-boto3-appconfig from 1.25.0 to 1.26.0.post1 ([#1722](https://github.com/awslabs/aws-lambda-powertools-python/issues/1722))
+* **deps-dev:** bump pytest-xdist from 2.5.0 to 3.0.2 ([#1655](https://github.com/awslabs/aws-lambda-powertools-python/issues/1655))
 * **deps-dev:** bump mkdocs-material from 8.5.7 to 8.5.9 ([#1697](https://github.com/awslabs/aws-lambda-powertools-python/issues/1697))
 * **deps-dev:** bump flake8-comprehensions from 3.10.0 to 3.10.1 ([#1699](https://github.com/awslabs/aws-lambda-powertools-python/issues/1699))
 * **deps-dev:** bump types-requests from 2.28.11.2 to 2.28.11.3 ([#1698](https://github.com/awslabs/aws-lambda-powertools-python/issues/1698))
 * **deps-dev:** bump pytest-benchmark from 3.4.1 to 4.0.0 ([#1659](https://github.com/awslabs/aws-lambda-powertools-python/issues/1659))
 * **deps-dev:** bump mypy-boto3-secretsmanager from 1.25.0 to 1.26.0.post1 ([#1691](https://github.com/awslabs/aws-lambda-powertools-python/issues/1691))
-* **deps-dev:** bump pytest-xdist from 2.5.0 to 3.0.2 ([#1655](https://github.com/awslabs/aws-lambda-powertools-python/issues/1655))
-* **deps-dev:** bump flake8-bugbear from 22.10.25 to 22.10.27 ([#1665](https://github.com/awslabs/aws-lambda-powertools-python/issues/1665))
+* **deps-dev:** bump pytest-asyncio from 0.20.1 to 0.20.2 ([#1723](https://github.com/awslabs/aws-lambda-powertools-python/issues/1723))
 * **deps-dev:** bump mypy-boto3-ssm from 1.25.0 to 1.26.0.post1 ([#1690](https://github.com/awslabs/aws-lambda-powertools-python/issues/1690))
+* **logger:** overload inject_lambda_context with generics ([#1583](https://github.com/awslabs/aws-lambda-powertools-python/issues/1583))
 
 
 <a name="v2.2.0"></a>

aws_lambda_powertools/event_handler/api_gateway.py

@@ -798,6 +798,7 @@ def register_route(func: Callable):
             # Convert methods to tuple. It needs to be hashable as its part of the self._routes dict key
             methods = (method,) if isinstance(method, str) else tuple(method)
             self._routes[(rule, methods, cors, compress, cache_control)] = func
+            return func
 
         return register_route
 

aws_lambda_powertools/logging/logger.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import functools
 import inspect
 import io
@@ -96,6 +98,11 @@ class Logger(logging.Logger):  # lgtm [py/missing-call-to-init]
         custom logging formatter that implements PowertoolsFormatter
     logger_handler: logging.Handler, optional
         custom logging handler e.g. logging.FileHandler("file.log")
+    log_uncaught_exceptions: bool, by default False
+        logs uncaught exception using sys.excepthook
+
+        See: https://docs.python.org/3/library/sys.html#sys.excepthook
+
 
     Parameters propagated to LambdaPowertoolsFormatter
     --------------------------------------------------
@@ -203,6 +210,7 @@ def __init__(
         stream: Optional[IO[str]] = None,
         logger_formatter: Optional[PowertoolsFormatter] = None,
         logger_handler: Optional[logging.Handler] = None,
+        log_uncaught_exceptions: bool = False,
         json_serializer: Optional[Callable[[Dict], str]] = None,
         json_deserializer: Optional[Callable[[Union[Dict, str, bool, int, float]], str]] = None,
         json_default: Optional[Callable[[Any], Any]] = None,
@@ -222,6 +230,8 @@ def __init__(
         self.child = child
         self.logger_formatter = logger_formatter
         self.logger_handler = logger_handler or logging.StreamHandler(stream)
+        self.log_uncaught_exceptions = log_uncaught_exceptions
+
         self.log_level = self._get_log_level(level)
         self._is_deduplication_disabled = resolve_truthy_env_var_choice(
             env=os.getenv(constants.LOGGER_LOG_DEDUPLICATION_ENV, "false")
@@ -244,6 +254,10 @@ def __init__(
 
         self._init_logger(formatter_options=formatter_options, **kwargs)
 
+        if self.log_uncaught_exceptions:
+            logger.debug("Replacing exception hook")
+            sys.excepthook = functools.partial(log_uncaught_exception_hook, logger=self)
+
     # Prevent __getattr__ from shielding unknown attribute errors in type checkers
     # https://github.com/awslabs/aws-lambda-powertools-python/issues/1660
     if not TYPE_CHECKING:
@@ -735,3 +749,8 @@ def _is_internal_frame(frame):  # pragma: no cover
     """Signal whether the frame is a CPython or logging module internal."""
     filename = os.path.normcase(frame.f_code.co_filename)
     return filename == logging._srcfile or ("importlib" in filename and "_bootstrap" in filename)
+
+
+def log_uncaught_exception_hook(exc_type, exc_value, exc_traceback, logger: Logger):
+    """Callback function for sys.excepthook to use Logger to log uncaught exceptions"""
+    logger.exception("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))  # pragma: no cover

aws_lambda_powertools/utilities/data_classes/kinesis_stream_event.py

@@ -1,7 +1,11 @@
 import base64
 import json
-from typing import Iterator
+import zlib
+from typing import Iterator, List
 
+from aws_lambda_powertools.utilities.data_classes.cloud_watch_logs_event import (
+    CloudWatchLogsDecodedData,
+)
 from aws_lambda_powertools.utilities.data_classes.common import DictWrapper
 
 
@@ -43,6 +47,11 @@ def data_as_json(self) -> dict:
         """Decode binary encoded data as json"""
         return json.loads(self.data_as_text())
 
+    def data_zlib_compressed_as_json(self) -> dict:
+        """Decode binary encoded data as bytes"""
+        decompressed = zlib.decompress(self.data_as_bytes(), zlib.MAX_WBITS | 32)
+        return json.loads(decompressed)
+
 
 class KinesisStreamRecord(DictWrapper):
     @property
@@ -98,3 +107,11 @@ class KinesisStreamEvent(DictWrapper):
     def records(self) -> Iterator[KinesisStreamRecord]:
         for record in self["Records"]:
             yield KinesisStreamRecord(record)
+
+
+def extract_cloudwatch_logs_from_event(event: KinesisStreamEvent) -> List[CloudWatchLogsDecodedData]:
+    return [CloudWatchLogsDecodedData(record.kinesis.data_zlib_compressed_as_json()) for record in event.records]
+
+
+def extract_cloudwatch_logs_from_record(record: KinesisStreamRecord) -> CloudWatchLogsDecodedData:
+    return CloudWatchLogsDecodedData(data=record.kinesis.data_zlib_compressed_as_json())

aws_lambda_powertools/utilities/parameters/secrets.py

@@ -96,7 +96,12 @@ def _get(self, name: str, **sdk_options) -> str:
         # Explicit arguments will take precedence over keyword arguments
         sdk_options["SecretId"] = name
 
-        return self.client.get_secret_value(**sdk_options)["SecretString"]
+        secret_value = self.client.get_secret_value(**sdk_options)
+
+        if "SecretString" in secret_value:
+            return secret_value["SecretString"]
+
+        return secret_value["SecretBinary"]
 
     def _get_multiple(self, path: str, **sdk_options) -> Dict[str, str]:
         """

aws_lambda_powertools/utilities/parser/models/kinesis.py

@@ -1,8 +1,13 @@
-from typing import List, Type, Union
+import json
+import zlib
+from typing import Dict, List, Type, Union
 
 from pydantic import BaseModel, validator
 
 from aws_lambda_powertools.shared.functions import base64_decode
+from aws_lambda_powertools.utilities.parser.models.cloudwatch import (
+    CloudWatchLogsDecode,
+)
 from aws_lambda_powertools.utilities.parser.types import Literal
 
 
@@ -28,6 +33,21 @@ class KinesisDataStreamRecord(BaseModel):
     eventSourceARN: str
     kinesis: KinesisDataStreamRecordPayload
 
+    def decompress_zlib_record_data_as_json(self) -> Dict:
+        """Decompress Kinesis Record bytes data zlib compressed to JSON"""
+        if not isinstance(self.kinesis.data, bytes):
+            raise ValueError("We can only decompress bytes data, not custom models.")
+
+        return json.loads(zlib.decompress(self.kinesis.data, zlib.MAX_WBITS | 32))
+
 
 class KinesisDataStreamModel(BaseModel):
     Records: List[KinesisDataStreamRecord]
+
+
+def extract_cloudwatch_logs_from_event(event: KinesisDataStreamModel) -> List[CloudWatchLogsDecode]:
+    return [CloudWatchLogsDecode(**record.decompress_zlib_record_data_as_json()) for record in event.Records]
+
+
+def extract_cloudwatch_logs_from_record(record: KinesisDataStreamRecord) -> CloudWatchLogsDecode:
+    return CloudWatchLogsDecode(**record.decompress_zlib_record_data_as_json())

docs/core/logger.md

@@ -291,6 +291,30 @@ Use `logger.exception` method to log contextual information about exceptions. Lo
     --8<-- "examples/logger/src/logging_exceptions_output.json"
     ```
 
+#### Uncaught exceptions
+
+Logger can optionally log uncaught exceptions by setting `log_uncaught_exceptions=True` at initialization.
+
+!!! info "Logger will replace any exception hook previously registered via [sys.excepthook](https://docs.python.org/3/library/sys.html#sys.excepthook){target='_blank'}."
+
+??? question "What are uncaught exceptions?"
+
+    It's any raised exception that wasn't handled by the [`except` statement](https://docs.python.org/3.9/tutorial/errors.html#handling-exceptions){target="_blank"}, leading a Python program to a non-successful exit.
+
+    They are typically raised intentionally to signal a problem (`raise ValueError`), or a propagated exception from elsewhere in your code that you didn't handle it willingly or not (`KeyError`, `jsonDecoderError`, etc.).
+
+=== "logging_uncaught_exceptions.py"
+
+    ```python hl_lines="7"
+    --8<-- "examples/logger/src/logging_uncaught_exceptions.py"
+    ```
+
+=== "logging_uncaught_exceptions_output.json"
+
+    ```json hl_lines="7-8"
+    --8<-- "examples/logger/src/logging_uncaught_exceptions_output.json"
+    ```
+
 ### Date formatting
 
 Logger uses Python's standard logging date format with the addition of timezone: `2021-05-03 11:47:12,494+0200`.