From baff051b64afde951ada89140c96de80384fc0e1 Mon Sep 17 00:00:00 2001 From: Falcao Date: Fri, 27 Sep 2024 13:36:11 -0300 Subject: [PATCH 01/13] changed parser documentation --- docs/utilities/parser.md | 861 +++++++++++++++++++++------------------ 1 file changed, 473 insertions(+), 388 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 204295a6ece..aa5916d6198 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -2,168 +2,116 @@ title: Parser (Pydantic) description: Utility --- + -This utility provides data parsing and deep validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="_blank" rel="nofollow"}. +The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. ## Key features -* Defines data in pure Python classes, then parse, validate and extract only what you want -* Built-in envelopes to unwrap, extend, and validate popular event sources payloads -* Enforces type hints at runtime with user-friendly errors -* Support only Pydantic v2 +- Define data models using Python classes +- Parse and validate Lambda event payloads +- Built-in support for common AWS event sources +- Runtime type checking with user-friendly error messages +- Compatible with Pydantic v2 ## Getting started ### Install -!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" - -You need to bring Pydantic v2.0.3 or later as an external dependency. - -Add `aws-lambda-powertools[parser]` as a dependency in your preferred tool: _e.g._, _requirements.txt_, _pyproject.toml_. +Powertools for AWS Lambda (Python) supports Pydantic v2. Each Pydantic version requires different dependencies before you can use Parser. -### Defining models +!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="\_blank"}" -You can define models to parse incoming events by inheriting from `BaseModel`. +???+ warning + This will increase the compressed package size by >10MB due to the Pydantic dependency. -```python title="Defining an Order data model" -from aws_lambda_powertools.utilities.parser import BaseModel -from typing import List, Optional + To reduce the impact on the package size at the expense of 30%-50% of its performance[Pydantic can also be + installed without binary files](https://pydantic-docs.helpmanual.io/install/#performance-vs-package-size-trade-off){target="_blank" rel="nofollow"}: -class OrderItem(BaseModel): - id: int - quantity: int - description: str + Pip example:`SKIP_CYTHON=1 pip install --no-binary pydantic aws-lambda-powertools[parser]` -class Order(BaseModel): - id: int - description: str - items: List[OrderItem] # nesting models are supported - optional_field: Optional[str] = None # this field may or may not be available when parsing +```python +pip install aws-lambda-powertools ``` -These are simply Python classes that inherit from BaseModel. **Parser** enforces type hints declared in your model at runtime. +You can also add as a dependency in your preferred tool: e.g., requirements.txt, pyproject.toml. -### Parsing events +### Data Model with Parse -You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. Both are also able to parse either dictionary or JSON string as an input. +Define models by inheriting from `BaseModel` to parse incoming events. Pydantic then validates the data, ensuring all fields adhere to specified types and guaranteeing data integrity. -#### event_parser decorator +#### Event parser -Use the decorator for fail fast scenarios where you want your Lambda function to raise an exception in the event of a malformed payload. - -`event_parser` decorator will throw a `ValidationError` if your event cannot be parsed according to the model. - -???+ note - **This decorator will replace the `event` object with the parsed model if successful**. This means you might be careful when nesting other decorators that expect `event` to be a `dict`. - -```python hl_lines="19" title="Parsing and validating upon invocation with event_parser decorator" -from aws_lambda_powertools.utilities.parser import event_parser, BaseModel -from aws_lambda_powertools.utilities.typing import LambdaContext -from typing import List, Optional +The `event_parser` decorator automatically parses and validates the event. -import json - -class OrderItem(BaseModel): - id: int - quantity: int - description: str - -class Order(BaseModel): - id: int - description: str - items: List[OrderItem] # nesting models are supported - optional_field: Optional[str] = None # this field may or may not be available when parsing - - -@event_parser(model=Order) -def handler(event: Order, context: LambdaContext): - print(event.id) - print(event.description) - print(event.items) - - order_items = [item for item in event.items] - ... - -payload = { - "id": 10876546789, - "description": "My order", - "items": [ - { - "id": 1015938732, - "quantity": 1, - "description": "item xpto" - } - ] -} - -handler(event=payload, context=LambdaContext()) -handler(event=json.dumps(payload), context=LambdaContext()) # also works if event is a JSON string +```python +from aws_lambda_powertools.utilities.parser import BaseModel, event_parser, ValidationError + +class MyEvent(BaseModel): + id: int + name: str + +@event_parser(model=MyEvent) +def lambda_handler(event: MyEvent, context): + try: + return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} + except ValidationError as e: + return {"statusCode": 400, "body": f"Invalid input: {str(e)}"} ``` -Alternatively, you can automatically extract the model from the `event` without the need to include the model parameter in the `event_parser` function. - -```python hl_lines="23 24" - --8<-- "examples/parser/src/using_the_model_from_event.py" -``` +The `@event_parser(model=MyEvent)` automatically parses the event into the specified Pydantic model MyEvent. -#### parse function +The function catches **ValidationError**, returning a 400 status code with an error message if the input doesn't match the `MyEvent` model. It provides robust error handling for invalid inputs. -Use this standalone function when you want more control over the data validation process, for example returning a 400 error for malformed payloads. +#### Parse function -```python hl_lines="21 31" title="Using standalone parse function for more flexibility" -from aws_lambda_powertools.utilities.parser import parse, BaseModel, ValidationError -from typing import List, Optional +The `parse()` function allows you to manually control when and how an event is parsed into a Pydantic model. This can be useful in cases where you need flexibility, such as handling different event formats or adding custom logic before parsing. -class OrderItem(BaseModel): - id: int - quantity: int - description: str +```python +from aws_lambda_powertools.utilities.parser import BaseModel, parse, ValidationError + +# Define a Pydantic model for the expected structure of the input +class MyEvent(BaseModel): + id: int + name: str + +def lambda_handler(event: dict, context): + try: + # Manually parse the incoming event into MyEvent model + parsed_event: MyEvent = parse(model=MyEvent, event=event) + return { + "statusCode": 200, + "body": f"Hello {parsed_event.name}, your ID is {parsed_event.id}" + } + except ValidationError as e: + # Catch validation errors and return a 400 response + return { + "statusCode": 400, + "body": f"Validation error: {str(e)}" + } -class Order(BaseModel): - id: int - description: str - items: List[OrderItem] # nesting models are supported - optional_field: Optional[str] = None # this field may or may not be available when parsing +``` +--- -payload = { - "id": 10876546789, - "description": "My order", - "items": [ - { - # this will cause a validation error - "id": [1015938732], - "quantity": 1, - "description": "item xpto" - } - ] -} +**Should I use parse() or @event_parser? šŸ¤”** -def my_function(): - try: - parsed_payload: Order = parse(event=payload, model=Order) - # payload dict is now parsed into our model - return parsed_payload.items - except ValidationError: - return { - "status_code": 400, - "message": "Invalid order" - } -``` +TheĀ `parse()`Ā function offers more flexibility and control: -#### Primitive data model parsing +- It allows parsing different parts of an event using multiple models. +- You can conditionally handle events before parsing them. +- It's useful for integrating with complex workflows where a decorator might not be sufficient. +- It provides more control over the validation process. -The parser allows you parse events into primitive data types, such as `dict` or classes that don't inherit from `BaseModel`. The following example shows you how to parse a [`Union`](https://docs.pydantic.dev/latest/api/standard_library_types/#union): +TheĀ `@event_parser`Ā decorator is ideal for: -```python ---8<-- "examples/parser/src/multiple_model_parsing.py" -``` +- Fail-fast scenarios where you want to immediately stop execution if the event payload is invalid. +- Simplifying your code by automatically parsing and validating the event at the function entry point. ### Built-in models -Parser comes with the following built-in models: +Parser provides built-in models for parsing events from AWS services. You don't need to worry about creating these models yourself - we've already done that for you, making it easier to process AWS events in your Lambda functions. | Model name | Description | | ------------------------------------------- | ------------------------------------------------------------------------------------- | @@ -204,156 +152,139 @@ You can extend them to include your own models, and yet have all other known fie ???+ tip For Mypy users, we only allow type override for fields where payload is injected e.g. `detail`, `body`, etc. -```python hl_lines="16-17 28 41" title="Extending EventBridge model as an example" -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.utilities.parser.models import EventBridgeModel - -from typing import List, Optional +**Example: custom data model with Amazon EventBridge** +Use the model to validate and extract relevant information from the incoming event. This can be useful when you need to handle events with a specific structure or when you want to ensure that the event data conforms to certain rules. -class OrderItem(BaseModel): - id: int - quantity: int - description: str +```python +from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel -class Order(BaseModel): - id: int - description: str - items: List[OrderItem] +# Define a custom EventBridge model by extending the built-in EventBridgeModel +class MyCustomEventBridgeModel(EventBridgeModel): + detail_type: str = Field(alias="detail-type") + source: str + detail: dict + +def lambda_handler(event: dict, context): + try: + # Manually parse the incoming event into the custom model + parsed_event: MyCustomEventBridgeModel = parse(model=MyCustomEventBridgeModel, event=event) + + return { + "statusCode": 200, + "body": f"Event from {parsed_event.source}, type: {parsed_event.detail_type}" + } + except ValidationError as e: + return { + "statusCode": 400, + "body": f"Validation error: {str(e)}" + } -class OrderEventModel(EventBridgeModel): - detail: Order +``` -payload = { - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "OrderPurchased", - "source": "OrderService", - "account": "111122223333", - "time": "2020-10-22T18:43:48Z", - "region": "us-west-1", - "resources": ["some_additional"], - "detail": { - "id": 10876546789, - "description": "My order", - "items": [ - { - "id": 1015938732, - "quantity": 1, - "description": "item xpto" - } - ] - } +You can simulate an EventBridge event like the following to test the Lambda function: + +**Sample event:** + +```json +{ + "version": "0", + "id": "abcd-1234-efgh-5678", + "detail-type": "order.created", + "source": "my.order.service", + "account": "123456789012", + "time": "2023-09-10T12:00:00Z", + "region": "us-west-2", + "resources": [], + "detail": { + "orderId": "O-12345", + "amount": 100.0 + } } - -ret = parse(model=OrderEventModel, event=payload) - -assert ret.source == "OrderService" -assert ret.detail.description == "My order" -assert ret.detail_type == "OrderPurchased" # we rename it to snake_case since detail-type is an invalid name - -for order_item in ret.detail.items: - ... ``` -**What's going on here, you might ask**: - -1. We imported our built-in model `EventBridgeModel` from the parser utility -2. Defined how our `Order` should look like -3. Defined how part of our EventBridge event should look like by overriding `detail` key within our `OrderEventModel` -4. Parser parsed the original event against `OrderEventModel` - -???+ tip - When extending a `string` field containing JSON, you need to wrap the field - with [Pydantic's Json Type](https://pydantic-docs.helpmanual.io/usage/types/#json-type){target="_blank" rel="nofollow"}: - - ```python hl_lines="14 18-19" - --8<-- "examples/parser/src/extending_built_in_models_with_json_mypy.py" - ``` - - Alternatively, you could use a [Pydantic validator](https://pydantic-docs.helpmanual.io/usage/validators/){target="_blank" rel="nofollow"} to transform the JSON string into a dict before the mapping: - - ```python hl_lines="18-20 24-25" - --8<-- "examples/parser/src/extending_built_in_models_with_json_validator.py" - ``` +## Advanced ### Envelopes -When trying to parse your payloads wrapped in a known structure, you might encounter the following situations: - -* Your actual payload is wrapped around a known structure, for example Lambda Event Sources like EventBridge -* You're only interested in a portion of the payload, for example parsing the `detail` of custom events in EventBridge, or `body` of SQS records +Envelopes use JMESPath expressions to extract specific portions of complex, nested JSON structures. This feature simplifies processing events from various AWS services by allowing you to focus on core data without unnecessary metadata or wrapper information. -You can either solve these situations by creating a model of these known structures, parsing them, then extracting and parsing a key where your payload is. +**Purpose of the Envelope** -This can become difficult quite quickly. Parser makes this problem easier through a feature named `Envelope`. +- Data Extraction: The envelope helps extract the specific data we need from a larger, more complex event structure. +- Standardization: It allows us to handle different event sources in a consistent manner. +- Simplification: By using an envelope, we can focus on parsing only the relevant part of the event, ignoring the surrounding metadata. Envelopes can be used via `envelope` parameter available in both `parse` function and `event_parser` decorator. -Here's an example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key. +Here's an example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key, from the payload below: -```python hl_lines="18-22 25 31" title="Parsing payload in a given key only using envelope feature" +```json +payload = { + "version": "0", + "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", + "detail-type": "CustomerSignedUp", + "source": "CustomerService", + "account": "111122223333", + "time": "2020-10-22T18:43:48Z", + "region": "us-west-1", + "resources": ["some_additional_"], + "detail": { + "username": "universe", + "parentid_1": "12345", + "parentid_2": "6789" + } + } +``` + +An example using `@event_parser` decorator to automatically parse the EventBridge event and extract the UserModel data. The envelope, specifically `envelopes.EventBridgeEnvelope` in this case, is used to extract the relevant data from a complex event structure. It acts as a wrapper or container that holds additional metadata and the actual payload we're interested in. + +```python from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes from aws_lambda_powertools.utilities.typing import LambdaContext class UserModel(BaseModel): - username: str - password1: str - password2: str - -payload = { - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "CustomerSignedUp", - "source": "CustomerService", - "account": "111122223333", - "time": "2020-10-22T18:43:48Z", - "region": "us-west-1", - "resources": ["some_additional_"], - "detail": { - "username": "universe", - "password1": "myp@ssword", - "password2": "repeat password" - } -} - -ret = parse(model=UserModel, envelope=envelopes.EventBridgeEnvelope, event=payload) + username: str + parentid_1: str + parentid_2: str -# Parsed model only contains our actual model, not the entire EventBridge + Payload parsed -assert ret.password1 == ret.password2 - -# Same behaviour but using our decorator @event_parser(model=UserModel, envelope=envelopes.EventBridgeEnvelope) -def handler(event: UserModel, context: LambdaContext): - assert event.password1 == event.password2 +def lambda_handler(event: UserModel, context: LambdaContext): + if event.parentid_1!= event.parentid_2: + return { + "statusCode": 400, + "body": "Parent ids do not match" + } + + # If parentids match, proceed with user registration + # Add your user registration logic here + + return { + "statusCode": 200, + "body": f"User {event.username} registered successfully" + } ``` -**What's going on here, you might ask**: - -1. We imported built-in `envelopes` from the parser utility -2. Used `envelopes.EventBridgeEnvelope` as the envelope for our `UserModel` model -3. Parser parsed the original event against the EventBridge model -4. Parser then parsed the `detail` key using `UserModel` - #### Built-in envelopes -Parser comes with the following built-in envelopes, where `Model` in the return section is your given model. - -| Envelope name | Behaviour | Return | -| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | -| **DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`.
2. Parses records in `NewImage` and `OldImage` keys using your model.
3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` | -| **EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`.
2. Parses `detail` key using your model and returns it. | `Model` | -| **SqsEnvelope** | 1. Parses data using `SqsModel`.
2. Parses records in `body` key using your model and return them in a list. | `List[Model]` | -| **CloudWatchLogsEnvelope** | 1. Parses data using `CloudwatchLogsModel` which will base64 decode and decompress it.
2. Parses records in `message` key using your model and return them in a list. | `List[Model]` | -| **KinesisDataStreamEnvelope** | 1. Parses data using `KinesisDataStreamModel` which will base64 decode it.
2. Parses records in in `Records` key using your model and returns them in a list. | `List[Model]` | -| **KinesisFirehoseEnvelope** | 1. Parses data using `KinesisFirehoseModel` which will base64 decode it.
2. Parses records in in `Records` key using your model and returns them in a list. | `List[Model]` | -| **SnsEnvelope** | 1. Parses data using `SnsModel`.
2. Parses records in `body` key using your model and return them in a list. | `List[Model]` | -| **SnsSqsEnvelope** | 1. Parses data using `SqsModel`.
2. Parses SNS records in `body` key using `SnsNotificationModel`.
3. Parses data in `Message` key using your model and return them in a list. | `List[Model]` | -| **ApiGatewayEnvelope** | 1. Parses data using `APIGatewayProxyEventModel`.
2. Parses `body` key using your model and returns it. | `Model` | -| **ApiGatewayV2Envelope** | 1. Parses data using `APIGatewayProxyEventV2Model`.
2. Parses `body` key using your model and returns it. | `Model` | -| **LambdaFunctionUrlEnvelope** | 1. Parses data using `LambdaFunctionUrlModel`.
2. Parses `body` key using your model and returns it. | `Model` | -| **KafkaEnvelope** | 1. Parses data using `KafkaRecordModel`.
2. Parses `value` key using your model and returns it. | `Model` | -| **VpcLatticeEnvelope** | 1. Parses data using `VpcLatticeModel`.
2. Parses `value` key using your model and returns it. | `Model` | -| **BedrockAgentEnvelope** | 1. Parses data using `BedrockAgentEventModel`.
2. Parses `inputText` key using your model and returns it. | `Model` | +Parsers provides built-in envelopes to extract and parse specific parts of complex event structures. These envelopes simplify handling nested data in events from various AWS services, allowing you to focus on the relevant information for your Lambda function. + +| Envelope name | Behaviour | Return | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | +| **DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`. `` 2. Parses records in `NewImage` and `OldImage` keys using your model. `` 3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` | +| **EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`. ``2. Parses`detail` key using your model and returns it. | `Model` | +| **SqsEnvelope** | 1. Parses data using `SqsModel`. ``2. Parses records in`body` key using your model and return them in a list. | `List[Model]` | +| **CloudWatchLogsEnvelope** | 1. Parses data using `CloudwatchLogsModel` which will base64 decode and decompress it. ``2. Parses records in`message` key using your model and return them in a list. | `List[Model]` | +| **KinesisDataStreamEnvelope** | 1. Parses data using `KinesisDataStreamModel` which will base64 decode it. ``2. Parses records in in`Records` key using your model and returns them in a list. | `List[Model]` | +| **KinesisFirehoseEnvelope** | 1. Parses data using `KinesisFirehoseModel` which will base64 decode it. ``2. Parses records in in`Records` key using your model and returns them in a list. | `List[Model]` | +| **SnsEnvelope** | 1. Parses data using `SnsModel`. ``2. Parses records in`body` key using your model and return them in a list. | `List[Model]` | +| **SnsSqsEnvelope** | 1. Parses data using `SqsModel`. `` 2. Parses SNS records in `body` key using `SnsNotificationModel`. `` 3. Parses data in `Message` key using your model and return them in a list. | `List[Model]` | +| **ApiGatewayEnvelope** | 1. Parses data using `APIGatewayProxyEventModel`. ``2. Parses`body` key using your model and returns it. | `Model` | +| **ApiGatewayV2Envelope** | 1. Parses data using `APIGatewayProxyEventV2Model`. ``2. Parses`body` key using your model and returns it. | `Model` | +| **LambdaFunctionUrlEnvelope** | 1. Parses data using `LambdaFunctionUrlModel`. ``2. Parses`body` key using your model and returns it. | `Model` | +| **KafkaEnvelope** | 1. Parses data using `KafkaRecordModel`. ``2. Parses`value` key using your model and returns it. | `Model` | +| **VpcLatticeEnvelope** | 1. Parses data using `VpcLatticeModel`. ``2. Parses`value` key using your model and returns it. | `Model` | +| **BedrockAgentEnvelope** | 1. Parses data using `BedrockAgentEventModel`. ``2. Parses`inputText` key using your model and returns it. | `Model` | #### Bringing your own envelope @@ -361,205 +292,359 @@ You can create your own Envelope model and logic by inheriting from `BaseEnvelop Here's a snippet of how the EventBridge envelope we demonstrated previously is implemented. -=== "EventBridge Model" - - ```python - from datetime import datetime - from typing import Any, Dict, List - - from aws_lambda_powertools.utilities.parser import BaseModel, Field - - - class EventBridgeModel(BaseModel): - version: str - id: str # noqa: A003,VNE003 - source: str - account: str - time: datetime - region: str - resources: List[str] - detail_type: str = Field(None, alias="detail-type") - detail: Dict[str, Any] - ``` - -=== "EventBridge Envelope" - - ```python hl_lines="8 10 25 26" - from aws_lambda_powertools.utilities.parser import BaseEnvelope, models - from aws_lambda_powertools.utilities.parser.models import EventBridgeModel - - from typing import Any, Dict, Optional, TypeVar - - Model = TypeVar("Model", bound=BaseModel) - - class EventBridgeEnvelope(BaseEnvelope): - - def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> Optional[Model]: - """Parses data found with model provided +```python +import json +from typing import Any, Dict, Optional, TypeVar, Union +from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser, BaseModel +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel +from aws_lambda_powertools.utilities.typing import LambdaContext - Parameters - ---------- - data : Dict - Lambda event to be parsed - model : Model - Data model provided to parse after extracting data using envelope +Model = TypeVar("Model", bound=BaseModel) + +class EventBridgeEnvelope(BaseEnvelope): + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: type[Model]) -> Optional[Model]: + if data is None: + return None + + parsed_envelope = EventBridgeModel.parse_obj(data) + return self._parse(data=parsed_envelope.detail, model=model) + +class OrderDetail(BaseModel): + order_id: str + amount: float + customer_id: str + +@event_parser(model=OrderDetail, envelope=EventBridgeEnvelope) +def lambda_handler(event: OrderDetail, context: LambdaContext): + try: + # Process the order + print(f"Processing order {event.order_id} for customer {event.customer_id}") + print(f"Order amount: ${event.amount:.2f}") + + # Your business logic here + # For example, you might save the order to a database or trigger a payment process + + return { + "statusCode": 200, + "body": json.dumps({ + "message": f"Order {event.order_id} processed successfully", + "order_id": event.order_id, + "amount": event.amount, + "customer_id": event.customer_id + }) + } + except Exception as e: + print(f"Error processing order: {str(e)}") + return { + "statusCode": 500, + "body": json.dumps({"error": "Internal server error"}) + } +``` - Returns - ------- - Any - Parsed detail payload with model provided - """ - parsed_envelope = EventBridgeModel.model_validate(data) - return self._parse(data=parsed_envelope.detail, model=model) - ``` +You can use the following test event: + +```json +{ + "version": "0", + "id": "12345678-1234-1234-1234-123456789012", + "detail-type": "Order Placed", + "source": "com.mycompany.orders", + "account": "123456789012", + "time": "2023-05-03T12:00:00Z", + "region": "us-west-2", + "resources": [], + "detail": { + "order_id": "ORD-12345", + "amount": 99.99, + "customer_id": "CUST-6789" + } +} +``` **What's going on here, you might ask**: -1. We defined an envelope named `EventBridgeEnvelope` inheriting from `BaseEnvelope` -2. Implemented the `parse` abstract method taking `data` and `model` as parameters -3. Then, we parsed the incoming data with our envelope to confirm it matches EventBridge's structure defined in `EventBridgeModel` -4. Lastly, we call `_parse` from `BaseEnvelope` to parse the data in our envelope (.detail) using the customer model +- **EventBridgeEnvelope**: extracts the detail field from EventBridge events. +- **OrderDetail Model**: defines and validates the structure of order data. +- **@event_parser**: decorator automates parsing and validation of incoming events using the specified model and envelope. ### Data model validation ???+ warning - This is radically different from the **Validator utility** which validates events against JSON Schema. +This is radically different from the **Validator utility** which validates events against JSON Schema. -You can use parser's validator for deep inspection of object values and complex relationships. +You can use Pydantic's validator for deep inspection of object values and complex relationships. There are two types of class method decorators you can use: -* **`validator`** - Useful to quickly validate an individual field and its value -* **`root_validator`** - Useful to validate the entire model's data +- **`field_validator`** - Useful to quickly validate an individual field and its value +- **`model_validator`** - Useful to validate the entire model's data Keep the following in mind regardless of which decorator you end up using it: -* You must raise either `ValueError`, `TypeError`, or `AssertionError` when value is not compliant -* You must return the value(s) itself if compliant +- You must raise either `ValueError`, `TypeError`, or `AssertionError` when value is not compliant +- You must return the value(s) itself if compliant -#### validating fields +#### Field Validator -Quick validation to verify whether the field `message` has the value of `hello world`. +Quick validation using decorator `field_validator` to verify whether the field `message` has the value of `hello world`. -```python hl_lines="6" title="Data field validation with validator" -from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator +```python +from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.typing import LambdaContext class HelloWorldModel(BaseModel): - message: str + message: str + + @field_validator('message') + def is_hello_world(cls, v): + if v != "hello world": + raise ValueError("Message must be hello world!") + return v + +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=HelloWorldModel, event=event) + return { + "statusCode": 200, + "body": f"Received message: {parsed_event.message}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } - @validator('message') - def is_hello_world(cls, v): - if v != "hello world": - raise ValueError("Message must be hello world!") - return v - -parse(model=HelloWorldModel, event={"message": "hello universe"}) ``` -If you run as-is, you should expect the following error with the message we provided in our exception: +If you run using a test event `{"message": "hello universe"}` you should expect the following error with the message we provided in our exception: -```python title="Sample validation error message" +```python message Message must be hello world! (type=value_error) ``` Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. -```python hl_lines="7" title="Validating all data fields with custom logic" -from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator +```python +from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.typing import LambdaContext class HelloWorldModel(BaseModel): - message: str - sender: str + message: str + sender: str + + @field_validator('*') + def has_whitespace(cls, v): + if ' ' not in v: + raise ValueError("Must have whitespace...") + return v + +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=HelloWorldModel, event=event) + return { + "statusCode": 200, + "body": f"Received message: {parsed_event.message}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } - @validator('*') - def has_whitespace(cls, v): - if ' ' not in v: - raise ValueError("Must have whitespace...") +``` - return v +Try with `event={"message": "hello universe", "sender": "universe"}` to get a validation error, as "sender" does not contain white spaces. -parse(model=HelloWorldModel, event={"message": "hello universe", "sender": "universe"}) -``` +#### Model validator -#### validating entire model +`model_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted or comparing field values. -`root_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted, comparing field values, etc. +**Key points about model_validator:** -```python title="Comparing and validating multiple fields at once with root_validator" +- It runs after all other validators have been called. +- It receives all the values of the model as a dictionary. +- It can modify or validate multiple fields at once. +- It's useful for validations that depend on multiple fields. + +```python +from aws_lambda_powertools.utilities.typing import LambdaContext from aws_lambda_powertools.utilities.parser import parse, BaseModel, root_validator class UserModel(BaseModel): username: str - password1: str - password2: str - - @root_validator - def check_passwords_match(cls, values): - pw1, pw2 = values.get('password1'), values.get('password2') - if pw1 is not None and pw2 is not None and pw1 != pw2: - raise ValueError('passwords do not match') + parentid_1: str + parentid_2: str + + @model_validator(mode='after') + def check_parents_match(cls, values): + pi1, pi2 = values.get('parentid_1'), values.get('parentid_2') + if pi1 is not None and pi2 is not None and pi1 != pi2: + raise ValueError('Parent ids do not match') return values +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=UserModel, event=event) + return { + "statusCode": 200, + "body": f"Received parent id from: {parsed_event.username}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } -payload = { - "username": "universe", - "password1": "myp@ssword", - "password2": "repeat password" -} - -parse(model=UserModel, event=payload) ``` +- The keyword argument `mode='after'` will cause the validator to be called after all field-level validation and parsing has been completed. + ???+ info - You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in Pydantic's documentation. +You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in [Pydantic's documentation](`https://pydantic-docs.helpmanual.io/usage/validators/`). -### Advanced use cases +**String fields that contain JSON data** -???+ tip "Tip: Looking to auto-generate models from JSON, YAML, JSON Schemas, OpenApi, etc?" - Use Koudai Aono's [data model code generation tool for Pydantic](https://github.com/koxudaxi/datamodel-code-generator){target="_blank" rel="nofollow"} +Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanual.io/usage/types/#json-type){target="\_blank" rel="nofollow"}. This approach allows Pydantic to properly parse and validate the JSON content, ensuring type safety and data integrity. -There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="_blank" rel="nofollow"}. +```python +from typing import Any, Type +from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel +from aws_lambda_powertools.utilities.typing import LambdaContext +from aws_lambda_powertools.utilities.parser.types import Json -???+ tip "Pydantic helper functions" - Pydantic also offers [functions](https://pydantic-docs.helpmanual.io/usage/models/#helper-functions){target="_blank" rel="nofollow"} to parse models from files, dicts, string, etc. +class CancelOrder(BaseModel): + order_id: int + reason: str -Two possible unknown use cases are Models and exception' serialization. Models have methods to [export them](https://pydantic-docs.helpmanual.io/usage/exporting_models/){target="_blank" rel="nofollow"} as `dict`, `JSON`, `JSON Schema`, and Validation exceptions can be exported as JSON. +class CancelOrderModel(BaseModel): + body: Json[CancelOrder] -```python hl_lines="21 28-31" title="Converting data models in various formats" -from aws_lambda_powertools.utilities import Logger -from aws_lambda_powertools.utilities.parser import parse, BaseModel, ValidationError, validator +class CustomEnvelope(BaseEnvelope): + def parse(self, data: dict, model: Type[BaseModel]) -> Any: + return model.parse_obj({"body": data.get("body", {})}) -logger = Logger(service="user") +@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) +def lambda_handler(event: CancelOrderModel, context: LambdaContext): + cancel_order: CancelOrder = event.body -class UserModel(BaseModel): - username: str - password1: str - password2: str + assert cancel_order.order_id is not None -payload = { - "username": "universe", - "password1": "myp@ssword", - "password2": "repeat password" + # Process the cancel order request + print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") + + return { + "statusCode": 200, + "body": f"Order {cancel_order.order_id} cancelled successfully" + } +``` + +Alternatively, you could use a [Pydantic validator](https://pydantic-docs.helpmanual.io/usage/validators/){target="\_blank" rel="nofollow"} to transform the JSON string into a dict before the mapping. + +```python +import json +from typing import Any, Type +from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel, validator +from aws_lambda_powertools.utilities.typing import LambdaContext + +class CancelOrder(BaseModel): + order_id: int + reason: str + +class CancelOrderModel(BaseModel): + body: CancelOrder + + @validator("body", pre=True) + def transform_body_to_dict(cls, value): + if isinstance(value, str): + return json.loads(value) + return value + +class CustomEnvelope(BaseEnvelope): + def parse(self, data: dict, model: Type[BaseModel]) -> Any: + return model.parse_obj({"body": data.get("body", {})}) + +@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) +def lambda_handler(event: CancelOrderModel, context: LambdaContext): + cancel_order: CancelOrder = event.body + + assert cancel_order.order_id is not None + + # Process the cancel order request + print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") + + return { + "statusCode": 200, + "body": json.dumps({"message": f"Order {cancel_order.order_id} cancelled successfully"}) + } + +``` + +To test both examples above, you can use: + +```json +{ + "body": "{\"order_id\": 12345, \"reason\": \"Changed my mind\"}" } +``` + +### Serialization + +Models in Pydantic offer more than direct attribute access. They can be transformed, serialized, and exported in various formats. + +Pydantic's definition of _serialization_ is broader than usual. It includes converting structured objects to simpler Python types, not just data to strings or bytes. This reflects the close relationship between these processes in Pydantic. + +Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="\_blank" rel="nofollow"}. -def my_function(): - try: - return parse(model=UserModel, event=payload) - except ValidationError as e: - logger.exception(e.json()) - return { - "status_code": 400, - "message": "Invalid username" - } - -User: UserModel = my_function() -user_dict = User.dict() -user_json = User.json() -user_json_schema_as_dict = User.schema() -user_json_schema_as_json = User.schema_json(indent=2) +```python +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.logging import Logger +from aws_lambda_powertools.utilities.typing import LambdaContext + +logger = Logger() + +class UserModel(BaseModel): + username: str + parentid_1: str + parentid_2: str + +def validate_user(event): + try: + user = parse(model=UserModel, event=event) + return { + "statusCode": 200, + "body": user.model_dump_json() + } + except Exception as e: + logger.exception("Validation error") + return { + "statusCode": 400, + "body": str(e) + } + +@logger.inject_lambda_context +def lambda_handler(event: dict, context: LambdaContext) -> dict: + logger.info("Received event", extra={"event": event}) + + result = validate_user(event) + + if result["statusCode"] == 200: + user = UserModel.model_validate_json(result["body"]) + logger.info("User validated successfully", extra={"username": user.username}) + + # Example of serialization + user_dict = user.model_dump() + user_json = user.model_dump_json() + + logger.debug("User serializations", extra={ + "dict": user_dict, + "json": user_json + }) + + return result ``` -These can be quite useful when manipulating models that later need to be serialized as inputs for services like DynamoDB, EventBridge, etc. +???+ info +There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="\_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="\_blank" rel="nofollow"}. ## FAQ @@ -573,8 +658,8 @@ Parser is best suited for those looking for a trade-off between defining their m We export most common classes, exceptions, and utilities from Pydantic as part of parser e.g. `from aws_lambda_powertools.utilities.parser import BaseModel`. -If what you're trying to use isn't available as part of the high level import system, use the following escape hatch mechanism: +If what you're trying to use isn't available as part of the high level import system, use the following escape _most_ hatch mechanism: -```python title="Pydantic import escape hatch" +```python from aws_lambda_powertools.utilities.parser.pydantic import ``` From 9d5acc1c09a168b0eeab1851bb6a0ae4e6523460 Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Fri, 27 Sep 2024 13:53:28 -0300 Subject: [PATCH 02/13] changed parser documentation --- docs/utilities/parser.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index aa5916d6198..5a15d3c87fc 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -5,7 +5,7 @@ description: Utility -The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. +The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. Test It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. ## Key features From 7141b4049ab3fcba8121b53fdcc210c8c9673e6e Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Fri, 27 Sep 2024 14:50:29 -0300 Subject: [PATCH 03/13] changed parser documentation --- docs/utilities/parser.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 5a15d3c87fc..507b79e331c 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -21,6 +21,10 @@ The Parser utility in Powertools for AWS Lambda simplifies data parsing and vali Powertools for AWS Lambda (Python) supports Pydantic v2. Each Pydantic version requires different dependencies before you can use Parser. +```python +pip install aws-lambda-powertools +``` + !!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="\_blank"}" ???+ warning @@ -31,9 +35,7 @@ Powertools for AWS Lambda (Python) supports Pydantic v2. Each Pydantic version r Pip example:`SKIP_CYTHON=1 pip install --no-binary pydantic aws-lambda-powertools[parser]` -```python -pip install aws-lambda-powertools -``` + You can also add as a dependency in your preferred tool: e.g., requirements.txt, pyproject.toml. @@ -370,7 +372,7 @@ You can use the following test event: ### Data model validation ???+ warning -This is radically different from the **Validator utility** which validates events against JSON Schema. + This is radically different from the **Validator utility** which validates events against JSON Schema. You can use Pydantic's validator for deep inspection of object values and complex relationships. @@ -419,7 +421,6 @@ def lambda_handler(event: dict, context: LambdaContext): If you run using a test event `{"message": "hello universe"}` you should expect the following error with the message we provided in our exception: ```python -message Message must be hello world! (type=value_error) ``` @@ -500,7 +501,7 @@ def lambda_handler(event: dict, context: LambdaContext): - The keyword argument `mode='after'` will cause the validator to be called after all field-level validation and parsing has been completed. ???+ info -You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in [Pydantic's documentation](`https://pydantic-docs.helpmanual.io/usage/validators/`). + You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in [Pydantic's documentation](`https://pydantic-docs.helpmanual.io/usage/validators/`). **String fields that contain JSON data** @@ -644,7 +645,7 @@ def lambda_handler(event: dict, context: LambdaContext) -> dict: ``` ???+ info -There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="\_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="\_blank" rel="nofollow"}. + There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="\_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="\_blank" rel="nofollow"}. ## FAQ From d985d74064463d91d3462eb671b0004a33ab1439 Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Fri, 27 Sep 2024 14:52:15 -0300 Subject: [PATCH 04/13] changed parser documentation --- docs/utilities/parser.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 507b79e331c..399f2a02a70 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -5,7 +5,7 @@ description: Utility -The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. Test It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. +The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. ## Key features From 6514d48d871953f08fbf6e645680400b746fded7 Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Thu, 17 Oct 2024 11:20:23 -0300 Subject: [PATCH 05/13] change the code from the doc to examples folder --- docs/utilities/parser.md | 459 +++--------------- .../parser/src/bring_your_own_envelope.json | 15 + .../parser/src/bring_your_own_envelope.py | 46 ++ .../src/custom_data_model_with_eventbridge.py | 23 + .../parser/src/data_model_eventbridge.json | 14 + examples/parser/src/envelope_payload.json | 17 + .../parser/src/envelope_with_event_parser.py | 23 + ...xtending_built_in_models_with_json_mypy.py | 21 - ...ing_built_in_models_with_json_validator.py | 27 -- examples/parser/src/field_validator.py | 24 + .../parser/src/field_validator_all_values.py | 25 + .../parser/src/getting_started_with_parser.py | 12 + examples/parser/src/json_data_string.json | 3 + examples/parser/src/model_validator.py | 26 + examples/parser/src/multiple_model_parsing.py | 33 -- examples/parser/src/parser_function.py | 21 + examples/parser/src/serialization_parser.py | 45 ++ .../parser/src/string_fields_contain_json.py | 29 ++ ..._fields_contain_json_pydantic_validator.py | 35 ++ .../parser/src/using_the_model_from_event.py | 27 -- 20 files changed, 424 insertions(+), 501 deletions(-) create mode 100644 examples/parser/src/bring_your_own_envelope.json create mode 100644 examples/parser/src/bring_your_own_envelope.py create mode 100644 examples/parser/src/custom_data_model_with_eventbridge.py create mode 100644 examples/parser/src/data_model_eventbridge.json create mode 100644 examples/parser/src/envelope_payload.json create mode 100644 examples/parser/src/envelope_with_event_parser.py delete mode 100644 examples/parser/src/extending_built_in_models_with_json_mypy.py delete mode 100644 examples/parser/src/extending_built_in_models_with_json_validator.py create mode 100644 examples/parser/src/field_validator.py create mode 100644 examples/parser/src/field_validator_all_values.py create mode 100644 examples/parser/src/getting_started_with_parser.py create mode 100644 examples/parser/src/json_data_string.json create mode 100644 examples/parser/src/model_validator.py delete mode 100644 examples/parser/src/multiple_model_parsing.py create mode 100644 examples/parser/src/parser_function.py create mode 100644 examples/parser/src/serialization_parser.py create mode 100644 examples/parser/src/string_fields_contain_json.py create mode 100644 examples/parser/src/string_fields_contain_json_pydantic_validator.py delete mode 100644 examples/parser/src/using_the_model_from_event.py diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 399f2a02a70..afe5a291e1d 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -5,7 +5,7 @@ description: Utility -The Parser utility in Powertools for AWS Lambda simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. +The Parser utility simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. ## Key features @@ -19,7 +19,7 @@ The Parser utility in Powertools for AWS Lambda simplifies data parsing and vali ### Install -Powertools for AWS Lambda (Python) supports Pydantic v2. Each Pydantic version requires different dependencies before you can use Parser. +Parser supports Pydantic v2. Each Pydantic version requires different dependencies before you can use Parser. ```python pip install aws-lambda-powertools @@ -36,7 +36,6 @@ pip install aws-lambda-powertools Pip example:`SKIP_CYTHON=1 pip install --no-binary pydantic aws-lambda-powertools[parser]` - You can also add as a dependency in your preferred tool: e.g., requirements.txt, pyproject.toml. ### Data Model with Parse @@ -47,19 +46,9 @@ Define models by inheriting from `BaseModel` to parse incoming events. Pydantic The `event_parser` decorator automatically parses and validates the event. -```python -from aws_lambda_powertools.utilities.parser import BaseModel, event_parser, ValidationError - -class MyEvent(BaseModel): - id: int - name: str - -@event_parser(model=MyEvent) -def lambda_handler(event: MyEvent, context): - try: - return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} - except ValidationError as e: - return {"statusCode": 400, "body": f"Invalid input: {str(e)}"} + +```python title="getting_started_with_parser.py" +--8<-- "examples/parser/src/getting_started_with_parser.py" ``` The `@event_parser(model=MyEvent)` automatically parses the event into the specified Pydantic model MyEvent. @@ -70,31 +59,9 @@ The function catches **ValidationError**, returning a 400 status code with an er The `parse()` function allows you to manually control when and how an event is parsed into a Pydantic model. This can be useful in cases where you need flexibility, such as handling different event formats or adding custom logic before parsing. -```python -from aws_lambda_powertools.utilities.parser import BaseModel, parse, ValidationError - -# Define a Pydantic model for the expected structure of the input -class MyEvent(BaseModel): - id: int - name: str - -def lambda_handler(event: dict, context): - try: - # Manually parse the incoming event into MyEvent model - parsed_event: MyEvent = parse(model=MyEvent, event=event) - return { - "statusCode": 200, - "body": f"Hello {parsed_event.name}, your ID is {parsed_event.id}" - } - except ValidationError as e: - # Catch validation errors and return a 400 response - return { - "statusCode": 400, - "body": f"Validation error: {str(e)}" - } - +```python title="parser_function.py" +--8<-- "examples/parser/src/parser_function.py" ``` - --- **Should I use parse() or @event_parser? šŸ¤”** @@ -157,53 +124,18 @@ You can extend them to include your own models, and yet have all other known fie **Example: custom data model with Amazon EventBridge** Use the model to validate and extract relevant information from the incoming event. This can be useful when you need to handle events with a specific structure or when you want to ensure that the event data conforms to certain rules. -```python -from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field -from aws_lambda_powertools.utilities.parser.models import EventBridgeModel - -# Define a custom EventBridge model by extending the built-in EventBridgeModel -class MyCustomEventBridgeModel(EventBridgeModel): - detail_type: str = Field(alias="detail-type") - source: str - detail: dict - -def lambda_handler(event: dict, context): - try: - # Manually parse the incoming event into the custom model - parsed_event: MyCustomEventBridgeModel = parse(model=MyCustomEventBridgeModel, event=event) - - return { - "statusCode": 200, - "body": f"Event from {parsed_event.source}, type: {parsed_event.detail_type}" - } - except ValidationError as e: - return { - "statusCode": 400, - "body": f"Validation error: {str(e)}" - } +=== "Custom data model" -``` + ```python + --8<-- "examples/parser/src/custom_data_model_with_eventbridge.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/data_model_eventbridge.json" + ``` -You can simulate an EventBridge event like the following to test the Lambda function: - -**Sample event:** - -```json -{ - "version": "0", - "id": "abcd-1234-efgh-5678", - "detail-type": "order.created", - "source": "my.order.service", - "account": "123456789012", - "time": "2023-09-10T12:00:00Z", - "region": "us-west-2", - "resources": [], - "detail": { - "orderId": "O-12345", - "amount": 100.0 - } -} -``` ## Advanced @@ -219,53 +151,22 @@ Envelopes use JMESPath expressions to extract specific portions of complex, nest Envelopes can be used via `envelope` parameter available in both `parse` function and `event_parser` decorator. -Here's an example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key, from the payload below: - -```json -payload = { - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "CustomerSignedUp", - "source": "CustomerService", - "account": "111122223333", - "time": "2020-10-22T18:43:48Z", - "region": "us-west-1", - "resources": ["some_additional_"], - "detail": { - "username": "universe", - "parentid_1": "12345", - "parentid_2": "6789" - } - } -``` +All you want is what's inside the `detail` key, from the payload in the sample example. -An example using `@event_parser` decorator to automatically parse the EventBridge event and extract the UserModel data. The envelope, specifically `envelopes.EventBridgeEnvelope` in this case, is used to extract the relevant data from a complex event structure. It acts as a wrapper or container that holds additional metadata and the actual payload we're interested in. +Using `@event_parser` decorator to automatically parse the EventBridge event and extract the UserModel data. The envelope, specifically `envelopes.EventBridgeEnvelope` in this case, is used to extract the relevant data from a complex event structure. It acts as a wrapper or container that holds additional metadata and the actual payload we're interested in. + +=== "Envelopes using event parser decorator" + + ```python + --8<-- "examples/parser/src/envelope_with_event_parser.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/envelope_payload.json" + ``` -```python -from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes -from aws_lambda_powertools.utilities.typing import LambdaContext - -class UserModel(BaseModel): - username: str - parentid_1: str - parentid_2: str - -@event_parser(model=UserModel, envelope=envelopes.EventBridgeEnvelope) -def lambda_handler(event: UserModel, context: LambdaContext): - if event.parentid_1!= event.parentid_2: - return { - "statusCode": 400, - "body": "Parent ids do not match" - } - - # If parentids match, proceed with user registration - # Add your user registration logic here - - return { - "statusCode": 200, - "body": f"User {event.username} registered successfully" - } -``` #### Built-in envelopes @@ -294,74 +195,18 @@ You can create your own Envelope model and logic by inheriting from `BaseEnvelop Here's a snippet of how the EventBridge envelope we demonstrated previously is implemented. -```python -import json -from typing import Any, Dict, Optional, TypeVar, Union -from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser, BaseModel -from aws_lambda_powertools.utilities.parser.models import EventBridgeModel -from aws_lambda_powertools.utilities.typing import LambdaContext - -Model = TypeVar("Model", bound=BaseModel) - -class EventBridgeEnvelope(BaseEnvelope): - def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: type[Model]) -> Optional[Model]: - if data is None: - return None - - parsed_envelope = EventBridgeModel.parse_obj(data) - return self._parse(data=parsed_envelope.detail, model=model) - -class OrderDetail(BaseModel): - order_id: str - amount: float - customer_id: str - -@event_parser(model=OrderDetail, envelope=EventBridgeEnvelope) -def lambda_handler(event: OrderDetail, context: LambdaContext): - try: - # Process the order - print(f"Processing order {event.order_id} for customer {event.customer_id}") - print(f"Order amount: ${event.amount:.2f}") - - # Your business logic here - # For example, you might save the order to a database or trigger a payment process - - return { - "statusCode": 200, - "body": json.dumps({ - "message": f"Order {event.order_id} processed successfully", - "order_id": event.order_id, - "amount": event.amount, - "customer_id": event.customer_id - }) - } - except Exception as e: - print(f"Error processing order: {str(e)}") - return { - "statusCode": 500, - "body": json.dumps({"error": "Internal server error"}) - } -``` +=== "Bring your own envelope with Event Bridge" + + ```python + --8<-- "examples/parser/src/bring_your_own_envelope.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/bring_your_own_envelope.json" + ``` -You can use the following test event: - -```json -{ - "version": "0", - "id": "12345678-1234-1234-1234-123456789012", - "detail-type": "Order Placed", - "source": "com.mycompany.orders", - "account": "123456789012", - "time": "2023-05-03T12:00:00Z", - "region": "us-west-2", - "resources": [], - "detail": { - "order_id": "ORD-12345", - "amount": 99.99, - "customer_id": "CUST-6789" - } -} -``` **What's going on here, you might ask**: @@ -390,32 +235,8 @@ Keep the following in mind regardless of which decorator you end up using it: Quick validation using decorator `field_validator` to verify whether the field `message` has the value of `hello world`. -```python -from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator -from aws_lambda_powertools.utilities.typing import LambdaContext - -class HelloWorldModel(BaseModel): - message: str - - @field_validator('message') - def is_hello_world(cls, v): - if v != "hello world": - raise ValueError("Message must be hello world!") - return v - -def lambda_handler(event: dict, context: LambdaContext): - try: - parsed_event = parse(model=HelloWorldModel, event=event) - return { - "statusCode": 200, - "body": f"Received message: {parsed_event.message}" - } - except ValueError as e: - return { - "statusCode": 400, - "body": str(e) - } - +```python title="field_validator.py" +--8<-- "examples/parser/src/field_validator.py" ``` If you run using a test event `{"message": "hello universe"}` you should expect the following error with the message we provided in our exception: @@ -426,33 +247,8 @@ If you run using a test event `{"message": "hello universe"}` you should expect Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. -```python -from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator -from aws_lambda_powertools.utilities.typing import LambdaContext - -class HelloWorldModel(BaseModel): - message: str - sender: str - - @field_validator('*') - def has_whitespace(cls, v): - if ' ' not in v: - raise ValueError("Must have whitespace...") - return v - -def lambda_handler(event: dict, context: LambdaContext): - try: - parsed_event = parse(model=HelloWorldModel, event=event) - return { - "statusCode": 200, - "body": f"Received message: {parsed_event.message}" - } - except ValueError as e: - return { - "statusCode": 400, - "body": str(e) - } - +```python title="field_validator_all_values.py" +--8<-- "examples/parser/src/field_validator_all_values.py" ``` Try with `event={"message": "hello universe", "sender": "universe"}` to get a validation error, as "sender" does not contain white spaces. @@ -468,34 +264,8 @@ Try with `event={"message": "hello universe", "sender": "universe"}` to get a va - It can modify or validate multiple fields at once. - It's useful for validations that depend on multiple fields. -```python -from aws_lambda_powertools.utilities.typing import LambdaContext -from aws_lambda_powertools.utilities.parser import parse, BaseModel, root_validator - -class UserModel(BaseModel): - username: str - parentid_1: str - parentid_2: str - - @model_validator(mode='after') - def check_parents_match(cls, values): - pi1, pi2 = values.get('parentid_1'), values.get('parentid_2') - if pi1 is not None and pi2 is not None and pi1 != pi2: - raise ValueError('Parent ids do not match') - return values -def lambda_handler(event: dict, context: LambdaContext): - try: - parsed_event = parse(model=UserModel, event=event) - return { - "statusCode": 200, - "body": f"Received parent id from: {parsed_event.username}" - } - except ValueError as e: - return { - "statusCode": 400, - "body": str(e) - } - +```python title="model_validator.py" +--8<-- "examples/parser/src/model_validator.py" ``` - The keyword argument `mode='after'` will cause the validator to be called after all field-level validation and parsing has been completed. @@ -507,86 +277,33 @@ def lambda_handler(event: dict, context: LambdaContext): Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanual.io/usage/types/#json-type){target="\_blank" rel="nofollow"}. This approach allows Pydantic to properly parse and validate the JSON content, ensuring type safety and data integrity. -```python -from typing import Any, Type -from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel -from aws_lambda_powertools.utilities.typing import LambdaContext -from aws_lambda_powertools.utilities.parser.types import Json - -class CancelOrder(BaseModel): - order_id: int - reason: str -class CancelOrderModel(BaseModel): - body: Json[CancelOrder] +=== "Validate string fields containing JSON data" -class CustomEnvelope(BaseEnvelope): - def parse(self, data: dict, model: Type[BaseModel]) -> Any: - return model.parse_obj({"body": data.get("body", {})}) + ```python + --8<-- "examples/parser/src/string_fields_contain_json.py" + ``` -@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) -def lambda_handler(event: CancelOrderModel, context: LambdaContext): - cancel_order: CancelOrder = event.body +=== "Sample event" - assert cancel_order.order_id is not None - - # Process the cancel order request - print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") - - return { - "statusCode": 200, - "body": f"Order {cancel_order.order_id} cancelled successfully" - } -``` + ```json + --8<-- "examples/parser/src/json_data_string.json" + ``` Alternatively, you could use a [Pydantic validator](https://pydantic-docs.helpmanual.io/usage/validators/){target="\_blank" rel="nofollow"} to transform the JSON string into a dict before the mapping. -```python -import json -from typing import Any, Type -from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel, validator -from aws_lambda_powertools.utilities.typing import LambdaContext - -class CancelOrder(BaseModel): - order_id: int - reason: str - -class CancelOrderModel(BaseModel): - body: CancelOrder +=== "Validate string fields containing JSON data using Pydantic validator" - @validator("body", pre=True) - def transform_body_to_dict(cls, value): - if isinstance(value, str): - return json.loads(value) - return value + ```python + --8<-- "examples/parser/src/string_fields_contain_json_pydantic_validator.py" + ``` -class CustomEnvelope(BaseEnvelope): - def parse(self, data: dict, model: Type[BaseModel]) -> Any: - return model.parse_obj({"body": data.get("body", {})}) +=== "Sample event" -@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) -def lambda_handler(event: CancelOrderModel, context: LambdaContext): - cancel_order: CancelOrder = event.body + ```json + --8<-- "examples/parser/src/json_data_string.json" + ``` - assert cancel_order.order_id is not None - - # Process the cancel order request - print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") - - return { - "statusCode": 200, - "body": json.dumps({"message": f"Order {cancel_order.order_id} cancelled successfully"}) - } - -``` - -To test both examples above, you can use: - -```json -{ - "body": "{\"order_id\": 12345, \"reason\": \"Changed my mind\"}" -} -``` ### Serialization @@ -596,52 +313,8 @@ Pydantic's definition of _serialization_ is broader than usual. It includes conv Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="\_blank" rel="nofollow"}. -```python -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.logging import Logger -from aws_lambda_powertools.utilities.typing import LambdaContext - -logger = Logger() - -class UserModel(BaseModel): - username: str - parentid_1: str - parentid_2: str - -def validate_user(event): - try: - user = parse(model=UserModel, event=event) - return { - "statusCode": 200, - "body": user.model_dump_json() - } - except Exception as e: - logger.exception("Validation error") - return { - "statusCode": 400, - "body": str(e) - } - -@logger.inject_lambda_context -def lambda_handler(event: dict, context: LambdaContext) -> dict: - logger.info("Received event", extra={"event": event}) - - result = validate_user(event) - - if result["statusCode"] == 200: - user = UserModel.model_validate_json(result["body"]) - logger.info("User validated successfully", extra={"username": user.username}) - - # Example of serialization - user_dict = user.model_dump() - user_json = user.model_dump_json() - - logger.debug("User serializations", extra={ - "dict": user_dict, - "json": user_json - }) - - return result +```python title="serialization_parser.py" +--8<-- "examples/parser/src/serialization_parser.py" ``` ???+ info diff --git a/examples/parser/src/bring_your_own_envelope.json b/examples/parser/src/bring_your_own_envelope.json new file mode 100644 index 00000000000..f905c7b5b16 --- /dev/null +++ b/examples/parser/src/bring_your_own_envelope.json @@ -0,0 +1,15 @@ +{ + "version": "0", + "id": "12345678-1234-1234-1234-123456789012", + "detail-type": "Order Placed", + "source": "com.mycompany.orders", + "account": "123456789012", + "time": "2023-05-03T12:00:00Z", + "region": "us-west-2", + "resources": [], + "detail": { + "order_id": "ORD-12345", + "amount": 99.99, + "customer_id": "CUST-6789" + } +} \ No newline at end of file diff --git a/examples/parser/src/bring_your_own_envelope.py b/examples/parser/src/bring_your_own_envelope.py new file mode 100644 index 00000000000..548e1b096b8 --- /dev/null +++ b/examples/parser/src/bring_your_own_envelope.py @@ -0,0 +1,46 @@ +import json +from typing import Any, Dict, Optional, TypeVar, Union +from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser, BaseModel +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel +from aws_lambda_powertools.utilities.typing import LambdaContext + +Model = TypeVar("Model", bound=BaseModel) + +class EventBridgeEnvelope(BaseEnvelope): + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: type[Model]) -> Optional[Model]: + if data is None: + return None + + parsed_envelope = EventBridgeModel.parse_obj(data) + return self._parse(data=parsed_envelope.detail, model=model) + +class OrderDetail(BaseModel): + order_id: str + amount: float + customer_id: str + +@event_parser(model=OrderDetail, envelope=EventBridgeEnvelope) +def lambda_handler(event: OrderDetail, context: LambdaContext): + try: + # Process the order + print(f"Processing order {event.order_id} for customer {event.customer_id}") + print(f"Order amount: ${event.amount:.2f}") + + # Your business logic here + # For example, you might save the order to a database or trigger a payment process + + return { + "statusCode": 200, + "body": json.dumps({ + "message": f"Order {event.order_id} processed successfully", + "order_id": event.order_id, + "amount": event.amount, + "customer_id": event.customer_id + }) + } + except Exception as e: + print(f"Error processing order: {str(e)}") + return { + "statusCode": 500, + "body": json.dumps({"error": "Internal server error"}) + } \ No newline at end of file diff --git a/examples/parser/src/custom_data_model_with_eventbridge.py b/examples/parser/src/custom_data_model_with_eventbridge.py new file mode 100644 index 00000000000..d5dfc4e4085 --- /dev/null +++ b/examples/parser/src/custom_data_model_with_eventbridge.py @@ -0,0 +1,23 @@ +from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel + +# Define a custom EventBridge model by extending the built-in EventBridgeModel +class MyCustomEventBridgeModel(EventBridgeModel): + detail_type: str = Field(alias="detail-type") + source: str + detail: dict + +def lambda_handler(event: dict, context): + try: + # Manually parse the incoming event into the custom model + parsed_event: MyCustomEventBridgeModel = parse(model=MyCustomEventBridgeModel, event=event) + + return { + "statusCode": 200, + "body": f"Event from {parsed_event.source}, type: {parsed_event.detail_type}" + } + except ValidationError as e: + return { + "statusCode": 400, + "body": f"Validation error: {str(e)}" + } \ No newline at end of file diff --git a/examples/parser/src/data_model_eventbridge.json b/examples/parser/src/data_model_eventbridge.json new file mode 100644 index 00000000000..2e05f0f8fa7 --- /dev/null +++ b/examples/parser/src/data_model_eventbridge.json @@ -0,0 +1,14 @@ +{ + "version": "0", + "id": "abcd-1234-efgh-5678", + "detail-type": "order.created", + "source": "my.order.service", + "account": "123456789012", + "time": "2023-09-10T12:00:00Z", + "region": "us-west-2", + "resources": [], + "detail": { + "orderId": "O-12345", + "amount": 100.0 + } +} \ No newline at end of file diff --git a/examples/parser/src/envelope_payload.json b/examples/parser/src/envelope_payload.json new file mode 100644 index 00000000000..68e1a454868 --- /dev/null +++ b/examples/parser/src/envelope_payload.json @@ -0,0 +1,17 @@ +{ + "version": "0", + "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", + "detail-type": "CustomerSignedUp", + "source": "CustomerService", + "account": "111122223333", + "time": "2020-10-22T18:43:48Z", + "region": "us-west-1", + "resources": [ + "some_additional_" + ], + "detail": { + "username": "universe", + "parentid_1": "12345", + "parentid_2": "6789" + } +} \ No newline at end of file diff --git a/examples/parser/src/envelope_with_event_parser.py b/examples/parser/src/envelope_with_event_parser.py new file mode 100644 index 00000000000..ce3ae84be5c --- /dev/null +++ b/examples/parser/src/envelope_with_event_parser.py @@ -0,0 +1,23 @@ +from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes +from aws_lambda_powertools.utilities.typing import LambdaContext + +class UserModel(BaseModel): + username: str + parentid_1: str + parentid_2: str + +@event_parser(model=UserModel, envelope=envelopes.EventBridgeEnvelope) +def lambda_handler(event: UserModel, context: LambdaContext): + if event.parentid_1!= event.parentid_2: + return { + "statusCode": 400, + "body": "Parent ids do not match" + } + + # If parentids match, proceed with user registration + # Add your user registration logic here + + return { + "statusCode": 200, + "body": f"User {event.username} registered successfully" + } \ No newline at end of file diff --git a/examples/parser/src/extending_built_in_models_with_json_mypy.py b/examples/parser/src/extending_built_in_models_with_json_mypy.py deleted file mode 100644 index 813f757ad79..00000000000 --- a/examples/parser/src/extending_built_in_models_with_json_mypy.py +++ /dev/null @@ -1,21 +0,0 @@ -from pydantic import BaseModel, Json - -from aws_lambda_powertools.utilities.parser import event_parser -from aws_lambda_powertools.utilities.parser.models import APIGatewayProxyEventV2Model -from aws_lambda_powertools.utilities.typing import LambdaContext - - -class CancelOrder(BaseModel): - order_id: int - reason: str - - -class CancelOrderModel(APIGatewayProxyEventV2Model): - body: Json[CancelOrder] # type: ignore[assignment] - - -@event_parser(model=CancelOrderModel) -def handler(event: CancelOrderModel, context: LambdaContext): - cancel_order: CancelOrder = event.body - - assert cancel_order.order_id is not None diff --git a/examples/parser/src/extending_built_in_models_with_json_validator.py b/examples/parser/src/extending_built_in_models_with_json_validator.py deleted file mode 100644 index acd4f3fc825..00000000000 --- a/examples/parser/src/extending_built_in_models_with_json_validator.py +++ /dev/null @@ -1,27 +0,0 @@ -import json - -from pydantic import BaseModel, validator - -from aws_lambda_powertools.utilities.parser import event_parser -from aws_lambda_powertools.utilities.parser.models import APIGatewayProxyEventV2Model -from aws_lambda_powertools.utilities.typing import LambdaContext - - -class CancelOrder(BaseModel): - order_id: int - reason: str - - -class CancelOrderModel(APIGatewayProxyEventV2Model): - body: CancelOrder # type: ignore[assignment] - - @validator("body", pre=True) - def transform_body_to_dict(cls, value: str): - return json.loads(value) - - -@event_parser(model=CancelOrderModel) -def handler(event: CancelOrderModel, context: LambdaContext): - cancel_order: CancelOrder = event.body - - assert cancel_order.order_id is not None diff --git a/examples/parser/src/field_validator.py b/examples/parser/src/field_validator.py new file mode 100644 index 00000000000..13dde60653f --- /dev/null +++ b/examples/parser/src/field_validator.py @@ -0,0 +1,24 @@ +from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.typing import LambdaContext + +class HelloWorldModel(BaseModel): + message: str + + @field_validator('message') + def is_hello_world(cls, v): + if v != "hello world": + raise ValueError("Message must be hello world!") + return v + +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=HelloWorldModel, event=event) + return { + "statusCode": 200, + "body": f"Received message: {parsed_event.message}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } \ No newline at end of file diff --git a/examples/parser/src/field_validator_all_values.py b/examples/parser/src/field_validator_all_values.py new file mode 100644 index 00000000000..04ab0850244 --- /dev/null +++ b/examples/parser/src/field_validator_all_values.py @@ -0,0 +1,25 @@ +from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.typing import LambdaContext + +class HelloWorldModel(BaseModel): + message: str + sender: str + + @field_validator('*') + def has_whitespace(cls, v): + if ' ' not in v: + raise ValueError("Must have whitespace...") + return v + +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=HelloWorldModel, event=event) + return { + "statusCode": 200, + "body": f"Received message: {parsed_event.message}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } \ No newline at end of file diff --git a/examples/parser/src/getting_started_with_parser.py b/examples/parser/src/getting_started_with_parser.py new file mode 100644 index 00000000000..deae31fd106 --- /dev/null +++ b/examples/parser/src/getting_started_with_parser.py @@ -0,0 +1,12 @@ +from aws_lambda_powertools.utilities.parser import BaseModel, event_parser, ValidationError + +class MyEvent(BaseModel): + id: int + name: str + +@event_parser(model=MyEvent) +def lambda_handler(event: MyEvent, context): + try: + return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} + except ValidationError as e: + return {"statusCode": 400, "body": f"Invalid input: {str(e)}"} \ No newline at end of file diff --git a/examples/parser/src/json_data_string.json b/examples/parser/src/json_data_string.json new file mode 100644 index 00000000000..9cd4ba447be --- /dev/null +++ b/examples/parser/src/json_data_string.json @@ -0,0 +1,3 @@ +{ + "body": "{\"order_id\": 12345, \"reason\": \"Changed my mind\"}" +} \ No newline at end of file diff --git a/examples/parser/src/model_validator.py b/examples/parser/src/model_validator.py new file mode 100644 index 00000000000..a66abbd0429 --- /dev/null +++ b/examples/parser/src/model_validator.py @@ -0,0 +1,26 @@ +from aws_lambda_powertools.utilities.typing import LambdaContext +from aws_lambda_powertools.utilities.parser import parse, BaseModel, root_validator + +class UserModel(BaseModel): + username: str + parentid_1: str + parentid_2: str + + @model_validator(mode='after') + def check_parents_match(cls, values): + pi1, pi2 = values.get('parentid_1'), values.get('parentid_2') + if pi1 is not None and pi2 is not None and pi1 != pi2: + raise ValueError('Parent ids do not match') + return values +def lambda_handler(event: dict, context: LambdaContext): + try: + parsed_event = parse(model=UserModel, event=event) + return { + "statusCode": 200, + "body": f"Received parent id from: {parsed_event.username}" + } + except ValueError as e: + return { + "statusCode": 400, + "body": str(e) + } \ No newline at end of file diff --git a/examples/parser/src/multiple_model_parsing.py b/examples/parser/src/multiple_model_parsing.py deleted file mode 100644 index 556848bbff6..00000000000 --- a/examples/parser/src/multiple_model_parsing.py +++ /dev/null @@ -1,33 +0,0 @@ -from typing import Any, Literal, Union - -from pydantic import BaseModel, Field -from typing_extensions import Annotated - -from aws_lambda_powertools.utilities.parser import event_parser - - -class Cat(BaseModel): - animal: Literal["cat"] - name: str - meow: int - - -class Dog(BaseModel): - animal: Literal["dog"] - name: str - bark: int - - -Animal = Annotated[ - Union[Cat, Dog], - Field(discriminator="animal"), -] - - -@event_parser(model=Animal) -def lambda_handler(event: Animal, _: Any) -> str: - if isinstance(event, Cat): - # we have a cat! - return f"🐈: {event.name}" - - return f"🐶: {event.name}" diff --git a/examples/parser/src/parser_function.py b/examples/parser/src/parser_function.py new file mode 100644 index 00000000000..2e2eab4e5c7 --- /dev/null +++ b/examples/parser/src/parser_function.py @@ -0,0 +1,21 @@ +from aws_lambda_powertools.utilities.parser import BaseModel, parse, ValidationError + +# Define a Pydantic model for the expected structure of the input +class MyEvent(BaseModel): + id: int + name: str + +def lambda_handler(event: dict, context): + try: + # Manually parse the incoming event into MyEvent model + parsed_event: MyEvent = parse(model=MyEvent, event=event) + return { + "statusCode": 200, + "body": f"Hello {parsed_event.name}, your ID is {parsed_event.id}" + } + except ValidationError as e: + # Catch validation errors and return a 400 response + return { + "statusCode": 400, + "body": f"Validation error: {str(e)}" + } \ No newline at end of file diff --git a/examples/parser/src/serialization_parser.py b/examples/parser/src/serialization_parser.py new file mode 100644 index 00000000000..5f0dd86eade --- /dev/null +++ b/examples/parser/src/serialization_parser.py @@ -0,0 +1,45 @@ +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.logging import Logger +from aws_lambda_powertools.utilities.typing import LambdaContext + +logger = Logger() + +class UserModel(BaseModel): + username: str + parentid_1: str + parentid_2: str + +def validate_user(event): + try: + user = parse(model=UserModel, event=event) + return { + "statusCode": 200, + "body": user.model_dump_json() + } + except Exception as e: + logger.exception("Validation error") + return { + "statusCode": 400, + "body": str(e) + } + +@logger.inject_lambda_context +def lambda_handler(event: dict, context: LambdaContext) -> dict: + logger.info("Received event", extra={"event": event}) + + result = validate_user(event) + + if result["statusCode"] == 200: + user = UserModel.model_validate_json(result["body"]) + logger.info("User validated successfully", extra={"username": user.username}) + + # Example of serialization + user_dict = user.model_dump() + user_json = user.model_dump_json() + + logger.debug("User serializations", extra={ + "dict": user_dict, + "json": user_json + }) + + return result \ No newline at end of file diff --git a/examples/parser/src/string_fields_contain_json.py b/examples/parser/src/string_fields_contain_json.py new file mode 100644 index 00000000000..4ba73a4b248 --- /dev/null +++ b/examples/parser/src/string_fields_contain_json.py @@ -0,0 +1,29 @@ +from typing import Any, Type +from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel +from aws_lambda_powertools.utilities.typing import LambdaContext +from aws_lambda_powertools.utilities.parser.types import Json + +class CancelOrder(BaseModel): + order_id: int + reason: str + +class CancelOrderModel(BaseModel): + body: Json[CancelOrder] + +class CustomEnvelope(BaseEnvelope): + def parse(self, data: dict, model: Type[BaseModel]) -> Any: + return model.model_validate({"body": data.get("body", {})}) + +@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) +def lambda_handler(event: CancelOrderModel, context: LambdaContext): + cancel_order: CancelOrder = event.body + + assert cancel_order.order_id is not None + + # Process the cancel order request + print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") + + return { + "statusCode": 200, + "body": f"Order {cancel_order.order_id} cancelled successfully" + } \ No newline at end of file diff --git a/examples/parser/src/string_fields_contain_json_pydantic_validator.py b/examples/parser/src/string_fields_contain_json_pydantic_validator.py new file mode 100644 index 00000000000..d4e9e595490 --- /dev/null +++ b/examples/parser/src/string_fields_contain_json_pydantic_validator.py @@ -0,0 +1,35 @@ +import json +from typing import Any, Type +from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel, validator +from aws_lambda_powertools.utilities.typing import LambdaContext + +class CancelOrder(BaseModel): + order_id: int + reason: str + +class CancelOrderModel(BaseModel): + body: CancelOrder + + @validator("body", pre=True) + def transform_body_to_dict(cls, value): + if isinstance(value, str): + return json.loads(value) + return value + +class CustomEnvelope(BaseEnvelope): + def parse(self, data: dict, model: Type[BaseModel]) -> Any: + return model.parse_obj({"body": data.get("body", {})}) + +@event_parser(model=CancelOrderModel, envelope=CustomEnvelope) +def lambda_handler(event: CancelOrderModel, context: LambdaContext): + cancel_order: CancelOrder = event.body + + assert cancel_order.order_id is not None + + # Process the cancel order request + print(f"Cancelling order {cancel_order.order_id} for reason: {cancel_order.reason}") + + return { + "statusCode": 200, + "body": json.dumps({"message": f"Order {cancel_order.order_id} cancelled successfully"}) + } \ No newline at end of file diff --git a/examples/parser/src/using_the_model_from_event.py b/examples/parser/src/using_the_model_from_event.py deleted file mode 100644 index 41e3116c61a..00000000000 --- a/examples/parser/src/using_the_model_from_event.py +++ /dev/null @@ -1,27 +0,0 @@ -import json - -from pydantic import BaseModel, validator - -from aws_lambda_powertools.utilities.parser import event_parser -from aws_lambda_powertools.utilities.parser.models import APIGatewayProxyEventV2Model -from aws_lambda_powertools.utilities.typing import LambdaContext - - -class CancelOrder(BaseModel): - order_id: int - reason: str - - -class CancelOrderModel(APIGatewayProxyEventV2Model): - body: CancelOrder # type: ignore[assignment] - - @validator("body", pre=True) - def transform_body_to_dict(cls, value: str): - return json.loads(value) - - -@event_parser -def handler(event: CancelOrderModel, context: LambdaContext): - cancel_order: CancelOrder = event.body - - assert cancel_order.order_id is not None From d2288a455562ac775d5a2494ba4fce9588958a5b Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Tue, 22 Oct 2024 15:14:27 -0300 Subject: [PATCH 06/13] add line highlights and fix sub header typo --- docs/utilities/parser.md | 28 +++++++++---------- .../parser/src/bring_your_own_envelope.py | 3 +- .../parser/src/envelope_with_event_parser.py | 3 +- examples/parser/src/field_validator.py | 3 +- .../parser/src/field_validator_all_values.py | 3 +- .../parser/src/getting_started_with_parser.py | 3 +- examples/parser/src/model_validator.py | 3 +- examples/parser/src/parser_function.py | 3 +- ..._fields_contain_json_pydantic_validator.py | 5 ++-- 9 files changed, 31 insertions(+), 23 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index afe5a291e1d..b462c29bab8 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -47,7 +47,7 @@ Define models by inheriting from `BaseModel` to parse incoming events. Pydantic The `event_parser` decorator automatically parses and validates the event. -```python title="getting_started_with_parser.py" +```python title="getting_started_with_parser.py" hl_lines="2 8" --8<-- "examples/parser/src/getting_started_with_parser.py" ``` @@ -59,10 +59,10 @@ The function catches **ValidationError**, returning a 400 status code with an er The `parse()` function allows you to manually control when and how an event is parsed into a Pydantic model. This can be useful in cases where you need flexibility, such as handling different event formats or adding custom logic before parsing. -```python title="parser_function.py" +```python title="parser_function.py" hl_lines="2 12" --8<-- "examples/parser/src/parser_function.py" ``` ---- + **Should I use parse() or @event_parser? šŸ¤”** @@ -124,9 +124,9 @@ You can extend them to include your own models, and yet have all other known fie **Example: custom data model with Amazon EventBridge** Use the model to validate and extract relevant information from the incoming event. This can be useful when you need to handle events with a specific structure or when you want to ensure that the event data conforms to certain rules. -=== "Custom data model" +=== "Custom data model" - ```python + ```python hl_lines="2 5 13" --8<-- "examples/parser/src/custom_data_model_with_eventbridge.py" ``` @@ -157,7 +157,7 @@ Using `@event_parser` decorator to automatically parse the EventBridge event and === "Envelopes using event parser decorator" - ```python + ```python hl_lines="2 10" --8<-- "examples/parser/src/envelope_with_event_parser.py" ``` @@ -197,7 +197,7 @@ Here's a snippet of how the EventBridge envelope we demonstrated previously is i === "Bring your own envelope with Event Bridge" - ```python + ```python hl_lines="3 4 8-16" --8<-- "examples/parser/src/bring_your_own_envelope.py" ``` @@ -235,7 +235,7 @@ Keep the following in mind regardless of which decorator you end up using it: Quick validation using decorator `field_validator` to verify whether the field `message` has the value of `hello world`. -```python title="field_validator.py" +```python title="field_validator.py" hl_lines="2 8" --8<-- "examples/parser/src/field_validator.py" ``` @@ -247,7 +247,7 @@ If you run using a test event `{"message": "hello universe"}` you should expect Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. -```python title="field_validator_all_values.py" +```python title="field_validator_all_values.py" hl_lines="2 9" --8<-- "examples/parser/src/field_validator_all_values.py" ``` @@ -264,7 +264,7 @@ Try with `event={"message": "hello universe", "sender": "universe"}` to get a va - It can modify or validate multiple fields at once. - It's useful for validations that depend on multiple fields. -```python title="model_validator.py" +```python title="model_validator.py" hl_lines="3 10" --8<-- "examples/parser/src/model_validator.py" ``` @@ -280,7 +280,7 @@ Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanu === "Validate string fields containing JSON data" - ```python + ```python hl_lines="4 6-11" --8<-- "examples/parser/src/string_fields_contain_json.py" ``` @@ -294,7 +294,7 @@ Alternatively, you could use a [Pydantic validator](https://pydantic-docs.helpma === "Validate string fields containing JSON data using Pydantic validator" - ```python + ```python hl_lines="4 14" --8<-- "examples/parser/src/string_fields_contain_json_pydantic_validator.py" ``` @@ -313,7 +313,7 @@ Pydantic's definition of _serialization_ is broader than usual. It includes conv Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="\_blank" rel="nofollow"}. -```python title="serialization_parser.py" +```python title="serialization_parser.py" hl_lines="37-38" --8<-- "examples/parser/src/serialization_parser.py" ``` @@ -335,5 +335,5 @@ We export most common classes, exceptions, and utilities from Pydantic as part o If what you're trying to use isn't available as part of the high level import system, use the following escape _most_ hatch mechanism: ```python -from aws_lambda_powertools.utilities.parser.pydantic import +from aws_lambda_powertools.utilities.parser.pydantic import <"what you'd like to import"> ``` diff --git a/examples/parser/src/bring_your_own_envelope.py b/examples/parser/src/bring_your_own_envelope.py index 548e1b096b8..66e2af7e20d 100644 --- a/examples/parser/src/bring_your_own_envelope.py +++ b/examples/parser/src/bring_your_own_envelope.py @@ -1,7 +1,8 @@ import json from typing import Any, Dict, Optional, TypeVar, Union -from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser, BaseModel +from aws_lambda_powertools.utilities.parser import BaseEnvelope, BaseModel from aws_lambda_powertools.utilities.parser.models import EventBridgeModel +from aws_lambda_powertools.utilities.parser import event_parser from aws_lambda_powertools.utilities.typing import LambdaContext Model = TypeVar("Model", bound=BaseModel) diff --git a/examples/parser/src/envelope_with_event_parser.py b/examples/parser/src/envelope_with_event_parser.py index ce3ae84be5c..b1d1d6f1ccd 100644 --- a/examples/parser/src/envelope_with_event_parser.py +++ b/examples/parser/src/envelope_with_event_parser.py @@ -1,4 +1,5 @@ -from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes +from aws_lambda_powertools.utilities.parser import event_parser, BaseModel +from aws_lambda_powertools.utilities.parser import envelopes from aws_lambda_powertools.utilities.typing import LambdaContext class UserModel(BaseModel): diff --git a/examples/parser/src/field_validator.py b/examples/parser/src/field_validator.py index 13dde60653f..9b1682d31d4 100644 --- a/examples/parser/src/field_validator.py +++ b/examples/parser/src/field_validator.py @@ -1,4 +1,5 @@ -from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.utilities.parser import field_validator from aws_lambda_powertools.utilities.typing import LambdaContext class HelloWorldModel(BaseModel): diff --git a/examples/parser/src/field_validator_all_values.py b/examples/parser/src/field_validator_all_values.py index 04ab0850244..a7ca7cebce4 100644 --- a/examples/parser/src/field_validator_all_values.py +++ b/examples/parser/src/field_validator_all_values.py @@ -1,4 +1,5 @@ -from aws_lambda_powertools.utilities.parser import parse, BaseModel, field_validator +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.utilities.parser import field_validator from aws_lambda_powertools.utilities.typing import LambdaContext class HelloWorldModel(BaseModel): diff --git a/examples/parser/src/getting_started_with_parser.py b/examples/parser/src/getting_started_with_parser.py index deae31fd106..40b43b06e86 100644 --- a/examples/parser/src/getting_started_with_parser.py +++ b/examples/parser/src/getting_started_with_parser.py @@ -1,4 +1,5 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, event_parser, ValidationError +from aws_lambda_powertools.utilities.parser import BaseModel, ValidationError +from aws_lambda_powertools.utilities.parser import event_parser class MyEvent(BaseModel): id: int diff --git a/examples/parser/src/model_validator.py b/examples/parser/src/model_validator.py index a66abbd0429..05f87fb96f6 100644 --- a/examples/parser/src/model_validator.py +++ b/examples/parser/src/model_validator.py @@ -1,5 +1,6 @@ from aws_lambda_powertools.utilities.typing import LambdaContext -from aws_lambda_powertools.utilities.parser import parse, BaseModel, root_validator +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.utilities.parser import model_validator class UserModel(BaseModel): username: str diff --git a/examples/parser/src/parser_function.py b/examples/parser/src/parser_function.py index 2e2eab4e5c7..fbf3ba932b7 100644 --- a/examples/parser/src/parser_function.py +++ b/examples/parser/src/parser_function.py @@ -1,4 +1,5 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, parse, ValidationError +from aws_lambda_powertools.utilities.parser import BaseModel, ValidationError +from aws_lambda_powertools.utilities.parser import parse # Define a Pydantic model for the expected structure of the input class MyEvent(BaseModel): diff --git a/examples/parser/src/string_fields_contain_json_pydantic_validator.py b/examples/parser/src/string_fields_contain_json_pydantic_validator.py index d4e9e595490..4d06f871dcb 100644 --- a/examples/parser/src/string_fields_contain_json_pydantic_validator.py +++ b/examples/parser/src/string_fields_contain_json_pydantic_validator.py @@ -1,6 +1,7 @@ import json from typing import Any, Type -from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel, validator +from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel +from aws_lambda_powertools.utilities.validation import validator from aws_lambda_powertools.utilities.typing import LambdaContext class CancelOrder(BaseModel): @@ -18,7 +19,7 @@ def transform_body_to_dict(cls, value): class CustomEnvelope(BaseEnvelope): def parse(self, data: dict, model: Type[BaseModel]) -> Any: - return model.parse_obj({"body": data.get("body", {})}) + return model.model_validate({"body": data.get("body", {})}) @event_parser(model=CancelOrderModel, envelope=CustomEnvelope) def lambda_handler(event: CancelOrderModel, context: LambdaContext): From 19694b667bf6fe1a18bb71e444b668d07fdbb533 Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Tue, 29 Oct 2024 17:11:58 -0300 Subject: [PATCH 07/13] fix typos, pydantic install, and add built in envelope example --- docs/utilities/parser.md | 56 +++++++++++++------ .../parser/src/envelope_with_event_parser.py | 1 - examples/parser/src/example_event_parser.json | 4 ++ .../parser/src/getting_started_with_parser.py | 7 +-- examples/parser/src/sqs_model_event.json | 26 +++++++++ examples/parser/src/sqs_model_event.py | 15 +++++ 6 files changed, 87 insertions(+), 22 deletions(-) create mode 100644 examples/parser/src/example_event_parser.json create mode 100644 examples/parser/src/sqs_model_event.json create mode 100644 examples/parser/src/sqs_model_event.py diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index b462c29bab8..3b942ec2222 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -19,7 +19,7 @@ The Parser utility simplifies data parsing and validation using [Pydantic](https ### Install -Parser supports Pydantic v2. Each Pydantic version requires different dependencies before you can use Parser. +Parser supports Pydantic v2. To use Parser, you'll need to install the necessary dependencies for Pydantic v2 beforehand. ```python pip install aws-lambda-powertools @@ -27,15 +27,6 @@ pip install aws-lambda-powertools !!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="\_blank"}" -???+ warning - This will increase the compressed package size by >10MB due to the Pydantic dependency. - - To reduce the impact on the package size at the expense of 30%-50% of its performance[Pydantic can also be - installed without binary files](https://pydantic-docs.helpmanual.io/install/#performance-vs-package-size-trade-off){target="_blank" rel="nofollow"}: - - Pip example:`SKIP_CYTHON=1 pip install --no-binary pydantic aws-lambda-powertools[parser]` - - You can also add as a dependency in your preferred tool: e.g., requirements.txt, pyproject.toml. ### Data Model with Parse @@ -46,12 +37,20 @@ Define models by inheriting from `BaseModel` to parse incoming events. Pydantic The `event_parser` decorator automatically parses and validates the event. +=== "getting_started_with_parser.py" -```python title="getting_started_with_parser.py" hl_lines="2 8" ---8<-- "examples/parser/src/getting_started_with_parser.py" -``` + ```python hl_lines="2 8" + --8<-- "examples/parser/src/getting_started_with_parser.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/example_event_parser.json" + ``` -The `@event_parser(model=MyEvent)` automatically parses the event into the specified Pydantic model MyEvent. + +The `@event_parser(model=MyEvent)` automatically parses the event into the specified Pydantic model `MyEvent`. The function catches **ValidationError**, returning a 400 status code with an error message if the input doesn't match the `MyEvent` model. It provides robust error handling for invalid inputs. @@ -59,9 +58,17 @@ The function catches **ValidationError**, returning a 400 status code with an er The `parse()` function allows you to manually control when and how an event is parsed into a Pydantic model. This can be useful in cases where you need flexibility, such as handling different event formats or adding custom logic before parsing. -```python title="parser_function.py" hl_lines="2 12" ---8<-- "examples/parser/src/parser_function.py" -``` +=== "parser_function.py" + + ```python hl_lines="2 12" + --8<-- "examples/parser/src/parser_function.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/example_event_parser.json" + ``` **Should I use parse() or @event_parser? šŸ¤”** @@ -172,6 +179,21 @@ Using `@event_parser` decorator to automatically parse the EventBridge event and Parsers provides built-in envelopes to extract and parse specific parts of complex event structures. These envelopes simplify handling nested data in events from various AWS services, allowing you to focus on the relevant information for your Lambda function. + +=== "sqs_model_event.py" + + ```python hl_lines="2 7" + --8<-- "examples/parser/src/sqs_model_event.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/sqs_model_event.json" + ``` + +The example above uses `SqsEnvelope` by importing `SqsModel`. Other built-in envelopes can be found below. + | Envelope name | Behaviour | Return | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | | **DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`. `` 2. Parses records in `NewImage` and `OldImage` keys using your model. `` 3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` | diff --git a/examples/parser/src/envelope_with_event_parser.py b/examples/parser/src/envelope_with_event_parser.py index b1d1d6f1ccd..5de0ca8148f 100644 --- a/examples/parser/src/envelope_with_event_parser.py +++ b/examples/parser/src/envelope_with_event_parser.py @@ -16,7 +16,6 @@ def lambda_handler(event: UserModel, context: LambdaContext): } # If parentids match, proceed with user registration - # Add your user registration logic here return { "statusCode": 200, diff --git a/examples/parser/src/example_event_parser.json b/examples/parser/src/example_event_parser.json new file mode 100644 index 00000000000..1dcc13a2e3e --- /dev/null +++ b/examples/parser/src/example_event_parser.json @@ -0,0 +1,4 @@ +{ + "id": "12345", + "name": "Jane Doe" +} \ No newline at end of file diff --git a/examples/parser/src/getting_started_with_parser.py b/examples/parser/src/getting_started_with_parser.py index 40b43b06e86..660606e2303 100644 --- a/examples/parser/src/getting_started_with_parser.py +++ b/examples/parser/src/getting_started_with_parser.py @@ -7,7 +7,6 @@ class MyEvent(BaseModel): @event_parser(model=MyEvent) def lambda_handler(event: MyEvent, context): - try: - return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} - except ValidationError as e: - return {"statusCode": 400, "body": f"Invalid input: {str(e)}"} \ No newline at end of file + #if your model is valid, you can return + return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} + \ No newline at end of file diff --git a/examples/parser/src/sqs_model_event.json b/examples/parser/src/sqs_model_event.json new file mode 100644 index 00000000000..08d6e28e0ac --- /dev/null +++ b/examples/parser/src/sqs_model_event.json @@ -0,0 +1,26 @@ +{ + "Records": [ + { + "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d", + "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...", + "body": "Test message hello!", + "attributes": { + "ApproximateReceiveCount": "1", + "SentTimestamp": "1545082649183", + "SenderId": "AIDAIENQZJOLO23YVJ4VO", + "ApproximateFirstReceiveTimestamp": "1545082649185" + }, + "messageAttributes": { + "testAttr": { + "stringValue": "100", + "binaryValue": "base64Str", + "dataType": "Number" + } + }, + "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3", + "eventSource": "aws:sqs", + "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue", + "awsRegion": "us-east-2" + } + ] +} \ No newline at end of file diff --git a/examples/parser/src/sqs_model_event.py b/examples/parser/src/sqs_model_event.py new file mode 100644 index 00000000000..80976711946 --- /dev/null +++ b/examples/parser/src/sqs_model_event.py @@ -0,0 +1,15 @@ +from aws_lambda_powertools.utilities.parser import parse +from aws_lambda_powertools.utilities.parser.models import SqsModel +from aws_lambda_powertools.utilities.typing import LambdaContext + + +def lambda_handler(event: dict, context: LambdaContext) -> None: + parsed_event = parse(model=SqsModel, event=event) + + results = [] + for record in parsed_event.Records: + results.append({ + 'Message ID':record.messageId, + 'body':record.body + }) + return results \ No newline at end of file From 9f140925be54de900cf920b763ac0e72690de1cc Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Tue, 29 Oct 2024 17:48:21 -0300 Subject: [PATCH 08/13] changed the example from built in env to models --- docs/utilities/parser.md | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 3b942ec2222..8845517fcbb 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -89,6 +89,20 @@ TheĀ `@event_parser`Ā decorator is ideal for: Parser provides built-in models for parsing events from AWS services. You don't need to worry about creating these models yourself - we've already done that for you, making it easier to process AWS events in your Lambda functions. +=== "sqs_model_event.py" + + ```python hl_lines="2 7" + --8<-- "examples/parser/src/sqs_model_event.py" + ``` + +=== "Sample event" + + ```json + --8<-- "examples/parser/src/sqs_model_event.json" + ``` + +The example above uses `SqsModel`. Other built-in envelopes can be found below. + | Model name | Description | | ------------------------------------------- | ------------------------------------------------------------------------------------- | | **AlbModel** | Lambda Event Source payload for Amazon Application Load Balancer | @@ -179,21 +193,6 @@ Using `@event_parser` decorator to automatically parse the EventBridge event and Parsers provides built-in envelopes to extract and parse specific parts of complex event structures. These envelopes simplify handling nested data in events from various AWS services, allowing you to focus on the relevant information for your Lambda function. - -=== "sqs_model_event.py" - - ```python hl_lines="2 7" - --8<-- "examples/parser/src/sqs_model_event.py" - ``` - -=== "Sample event" - - ```json - --8<-- "examples/parser/src/sqs_model_event.json" - ``` - -The example above uses `SqsEnvelope` by importing `SqsModel`. Other built-in envelopes can be found below. - | Envelope name | Behaviour | Return | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | | **DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`. `` 2. Parses records in `NewImage` and `OldImage` keys using your model. `` 3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` | From 506205679eb96429d5a1b963c65178b6f67a8cd3 Mon Sep 17 00:00:00 2001 From: Ana Falcao Date: Wed, 30 Oct 2024 08:51:09 -0300 Subject: [PATCH 09/13] fix validationerror import --- examples/parser/src/custom_data_model_with_eventbridge.py | 2 +- examples/parser/src/getting_started_with_parser.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/parser/src/custom_data_model_with_eventbridge.py b/examples/parser/src/custom_data_model_with_eventbridge.py index d5dfc4e4085..6cacc2e12a0 100644 --- a/examples/parser/src/custom_data_model_with_eventbridge.py +++ b/examples/parser/src/custom_data_model_with_eventbridge.py @@ -1,4 +1,4 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field +from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field, ValidationError from aws_lambda_powertools.utilities.parser.models import EventBridgeModel # Define a custom EventBridge model by extending the built-in EventBridgeModel diff --git a/examples/parser/src/getting_started_with_parser.py b/examples/parser/src/getting_started_with_parser.py index 660606e2303..0349c5b26b3 100644 --- a/examples/parser/src/getting_started_with_parser.py +++ b/examples/parser/src/getting_started_with_parser.py @@ -1,4 +1,4 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, ValidationError +from aws_lambda_powertools.utilities.parser import BaseModel from aws_lambda_powertools.utilities.parser import event_parser class MyEvent(BaseModel): From dc5bcae7f2d0a48b586361dc065673d92819e3f2 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Thu, 7 Nov 2024 11:16:27 +0000 Subject: [PATCH 10/13] Changing highlights + import --- docs/utilities/parser.md | 131 ++++++------------ .../parser/src/bring_your_own_envelope.py | 34 +++-- .../src/custom_data_model_with_eventbridge.py | 16 +-- .../parser/src/envelope_with_event_parser.py | 25 ++-- examples/parser/src/field_validator.py | 19 ++- .../parser/src/getting_started_with_parser.py | 8 +- examples/parser/src/model_validator.py | 32 +++-- examples/parser/src/parser_function.py | 15 +- examples/parser/src/serialization_parser.py | 24 ++-- .../parser/src/string_fields_contain_json.py | 14 +- 10 files changed, 136 insertions(+), 182 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 8845517fcbb..096c39873b8 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -5,7 +5,7 @@ description: Utility -The Parser utility simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="\_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. +The Parser utility simplifies data parsing and validation using [Pydantic](https://pydantic-docs.helpmanual.io/){target="_blank" rel="nofollow"}. It allows you to define data models in pure Python classes, parse and validate incoming events, and extract only the data you need. ## Key features @@ -19,27 +19,30 @@ The Parser utility simplifies data parsing and validation using [Pydantic](https ### Install -Parser supports Pydantic v2. To use Parser, you'll need to install the necessary dependencies for Pydantic v2 beforehand. +Powertools only supports Pydantic v2, so make sure to install the required dependencies for Pydantic v2 before using the Parser. ```python -pip install aws-lambda-powertools +pip install aws-lambda-powertools[tracer] ``` -!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="\_blank"}" +!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" -You can also add as a dependency in your preferred tool: e.g., requirements.txt, pyproject.toml. +You can also add as a dependency in your preferred tool: `e.g., requirements.txt, pyproject.toml`. ### Data Model with Parse -Define models by inheriting from `BaseModel` to parse incoming events. Pydantic then validates the data, ensuring all fields adhere to specified types and guaranteeing data integrity. +You can define models by inheriting from `BaseModel` or any other supported type through `TypeAdapter` to parse incoming events. Pydantic then validates the data, ensuring that all fields conform to the specified types and maintaining data integrity. + +???+ info + The new TypeAdapter feature provide a flexible way to perform validation and serialization based on a Python type. Read more in the [Pydantic documentation](https://docs.pydantic.dev/latest/api/type_adapter/){target="_blank" rel="nofollow"}. #### Event parser -The `event_parser` decorator automatically parses and validates the event. +The `@event_parser` decorator automatically parses the incoming event into the specified Pydantic model `MyEvent`. If the input doesn't match the model's structure or type requirements, we raises a `ValidationError` directly from Pydantic. -=== "getting_started_with_parser.py" +=== "getting_started_with_parser.py" - ```python hl_lines="2 8" + ```python hl_lines="3 10" --8<-- "examples/parser/src/getting_started_with_parser.py" ``` @@ -49,18 +52,13 @@ The `event_parser` decorator automatically parses and validates the event. --8<-- "examples/parser/src/example_event_parser.json" ``` - -The `@event_parser(model=MyEvent)` automatically parses the event into the specified Pydantic model `MyEvent`. - -The function catches **ValidationError**, returning a 400 status code with an error message if the input doesn't match the `MyEvent` model. It provides robust error handling for invalid inputs. - #### Parse function -The `parse()` function allows you to manually control when and how an event is parsed into a Pydantic model. This can be useful in cases where you need flexibility, such as handling different event formats or adding custom logic before parsing. +You can use the `parse()` function when you need to have flexibility with different event formats, custom pre-parsing logic, and better exception handling. -=== "parser_function.py" +=== "parser_function.py" - ```python hl_lines="2 12" + ```python hl_lines="3 14" --8<-- "examples/parser/src/parser_function.py" ``` @@ -70,15 +68,14 @@ The `parse()` function allows you to manually control when and how an event is p --8<-- "examples/parser/src/example_event_parser.json" ``` - -**Should I use parse() or @event_parser? šŸ¤”** +#### Keys difference between parse and event_parser TheĀ `parse()`Ā function offers more flexibility and control: - It allows parsing different parts of an event using multiple models. - You can conditionally handle events before parsing them. - It's useful for integrating with complex workflows where a decorator might not be sufficient. -- It provides more control over the validation process. +- It provides more control over the validation process and handling exceptions. TheĀ `@event_parser`Ā decorator is ideal for: @@ -87,9 +84,9 @@ TheĀ `@event_parser`Ā decorator is ideal for: ### Built-in models -Parser provides built-in models for parsing events from AWS services. You don't need to worry about creating these models yourself - we've already done that for you, making it easier to process AWS events in your Lambda functions. +You can use pre-built models provided by the Parser for parsing events from AWS services, so you don’t need to create these models yourself, we’ve already done that for you. -=== "sqs_model_event.py" +=== "sqs_model_event.py" ```python hl_lines="2 7" --8<-- "examples/parser/src/sqs_model_event.py" @@ -101,7 +98,7 @@ Parser provides built-in models for parsing events from AWS services. You don't --8<-- "examples/parser/src/sqs_model_event.json" ``` -The example above uses `SqsModel`. Other built-in envelopes can be found below. +The example above uses `SqsModel`. Other built-in models can be found below. | Model name | Description | | ------------------------------------------- | ------------------------------------------------------------------------------------- | @@ -145,9 +142,9 @@ You can extend them to include your own models, and yet have all other known fie **Example: custom data model with Amazon EventBridge** Use the model to validate and extract relevant information from the incoming event. This can be useful when you need to handle events with a specific structure or when you want to ensure that the event data conforms to certain rules. -=== "Custom data model" +=== "Custom data model" - ```python hl_lines="2 5 13" + ```python hl_lines="4 8 16" --8<-- "examples/parser/src/custom_data_model_with_eventbridge.py" ``` @@ -157,41 +154,29 @@ Use the model to validate and extract relevant information from the incoming eve --8<-- "examples/parser/src/data_model_eventbridge.json" ``` - ## Advanced ### Envelopes -Envelopes use JMESPath expressions to extract specific portions of complex, nested JSON structures. This feature simplifies processing events from various AWS services by allowing you to focus on core data without unnecessary metadata or wrapper information. - -**Purpose of the Envelope** - -- Data Extraction: The envelope helps extract the specific data we need from a larger, more complex event structure. -- Standardization: It allows us to handle different event sources in a consistent manner. -- Simplification: By using an envelope, we can focus on parsing only the relevant part of the event, ignoring the surrounding metadata. +You can use **Envelopes**, which are **JMESPath expressions**, to extract specific portions of complex, nested JSON structures. This is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like **EventBridge**. Envelopes can be used via `envelope` parameter available in both `parse` function and `event_parser` decorator. -All you want is what's inside the `detail` key, from the payload in the sample example. - -Using `@event_parser` decorator to automatically parse the EventBridge event and extract the UserModel data. The envelope, specifically `envelopes.EventBridgeEnvelope` in this case, is used to extract the relevant data from a complex event structure. It acts as a wrapper or container that holds additional metadata and the actual payload we're interested in. - === "Envelopes using event parser decorator" - ```python hl_lines="2 10" + ```python hl_lines="3 6-10 12" --8<-- "examples/parser/src/envelope_with_event_parser.py" ``` === "Sample event" - ```json + ```json hl_lines="12-16" --8<-- "examples/parser/src/envelope_payload.json" ``` - #### Built-in envelopes -Parsers provides built-in envelopes to extract and parse specific parts of complex event structures. These envelopes simplify handling nested data in events from various AWS services, allowing you to focus on the relevant information for your Lambda function. +You can use pre-built envelopes provided by the Parser to extract and parse specific parts of complex event structures. | Envelope name | Behaviour | Return | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | @@ -212,13 +197,13 @@ Parsers provides built-in envelopes to extract and parse specific parts of compl #### Bringing your own envelope -You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method. +You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method or `@event_parser` decorator. Here's a snippet of how the EventBridge envelope we demonstrated previously is implemented. === "Bring your own envelope with Event Bridge" - ```python hl_lines="3 4 8-16" + ```python hl_lines="6 12-18" --8<-- "examples/parser/src/bring_your_own_envelope.py" ``` @@ -228,7 +213,6 @@ Here's a snippet of how the EventBridge envelope we demonstrated previously is i --8<-- "examples/parser/src/bring_your_own_envelope.json" ``` - **What's going on here, you might ask**: - **EventBridgeEnvelope**: extracts the detail field from EventBridge events. @@ -256,7 +240,7 @@ Keep the following in mind regardless of which decorator you end up using it: Quick validation using decorator `field_validator` to verify whether the field `message` has the value of `hello world`. -```python title="field_validator.py" hl_lines="2 8" +```python title="field_validator.py" hl_lines="1 10-14" --8<-- "examples/parser/src/field_validator.py" ``` @@ -266,42 +250,26 @@ If you run using a test event `{"message": "hello universe"}` you should expect Message must be hello world! (type=value_error) ``` -Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. - -```python title="field_validator_all_values.py" hl_lines="2 9" ---8<-- "examples/parser/src/field_validator_all_values.py" -``` - -Try with `event={"message": "hello universe", "sender": "universe"}` to get a validation error, as "sender" does not contain white spaces. - #### Model validator `model_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted or comparing field values. -**Key points about model_validator:** - -- It runs after all other validators have been called. -- It receives all the values of the model as a dictionary. -- It can modify or validate multiple fields at once. -- It's useful for validations that depend on multiple fields. - -```python title="model_validator.py" hl_lines="3 10" +```python title="model_validator.py" hl_lines="1 12-17" --8<-- "examples/parser/src/model_validator.py" ``` -- The keyword argument `mode='after'` will cause the validator to be called after all field-level validation and parsing has been completed. +1. The keyword argument `mode='after'` will cause the validator to be called after all field-level validation and parsing has been completed. ???+ info - You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in [Pydantic's documentation](`https://pydantic-docs.helpmanual.io/usage/validators/`). + You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in [Pydantic's documentation](`https://pydantic-docs.helpmanual.io/usage/validators/`){target="_blank" rel="nofollow"}. -**String fields that contain JSON data** - -Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanual.io/usage/types/#json-type){target="\_blank" rel="nofollow"}. This approach allows Pydantic to properly parse and validate the JSON content, ensuring type safety and data integrity. +#### String fields that contain JSON data +Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanual.io/usage/types/#json-type){target="_blank" rel="nofollow"}. This approach allows Pydantic to properly parse and validate the JSON content, ensuring type safety and data integrity. === "Validate string fields containing JSON data" - ```python hl_lines="4 6-11" + ```python hl_lines="3 14" --8<-- "examples/parser/src/string_fields_contain_json.py" ``` @@ -311,35 +279,20 @@ Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanu --8<-- "examples/parser/src/json_data_string.json" ``` -Alternatively, you could use a [Pydantic validator](https://pydantic-docs.helpmanual.io/usage/validators/){target="\_blank" rel="nofollow"} to transform the JSON string into a dict before the mapping. - -=== "Validate string fields containing JSON data using Pydantic validator" - - ```python hl_lines="4 14" - --8<-- "examples/parser/src/string_fields_contain_json_pydantic_validator.py" - ``` - -=== "Sample event" - - ```json - --8<-- "examples/parser/src/json_data_string.json" - ``` - - ### Serialization Models in Pydantic offer more than direct attribute access. They can be transformed, serialized, and exported in various formats. Pydantic's definition of _serialization_ is broader than usual. It includes converting structured objects to simpler Python types, not just data to strings or bytes. This reflects the close relationship between these processes in Pydantic. -Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="\_blank" rel="nofollow"}. +Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="_blank" rel="nofollow"}. -```python title="serialization_parser.py" hl_lines="37-38" +```python title="serialization_parser.py" hl_lines="39-40" --8<-- "examples/parser/src/serialization_parser.py" ``` ???+ info - There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="\_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="\_blank" rel="nofollow"}. + There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability){target="_blank" rel="nofollow"}, [declaring fields with dynamic values](https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value){target="_blank" rel="nofollow"}. ## FAQ @@ -351,10 +304,10 @@ Parser is best suited for those looking for a trade-off between defining their m **How do I import X from Pydantic?** -We export most common classes, exceptions, and utilities from Pydantic as part of parser e.g. `from aws_lambda_powertools.utilities.parser import BaseModel`. - -If what you're trying to use isn't available as part of the high level import system, use the following escape _most_ hatch mechanism: +We recommend importing directly from Pydantic to access all features and stay up-to-date with the latest Pydantic updates. For example: ```python -from aws_lambda_powertools.utilities.parser.pydantic import <"what you'd like to import"> +from pydantic import BaseModel, Field, ValidationError ``` + +While we export some common Pydantic classes and utilities through the parser for convenience (e.g., `from aws_lambda_powertools.utilities.parser import BaseModel`), importing directly from Pydantic ensures you have access to all features and the most recent updates. diff --git a/examples/parser/src/bring_your_own_envelope.py b/examples/parser/src/bring_your_own_envelope.py index 66e2af7e20d..1fb5dea0045 100644 --- a/examples/parser/src/bring_your_own_envelope.py +++ b/examples/parser/src/bring_your_own_envelope.py @@ -1,25 +1,30 @@ import json -from typing import Any, Dict, Optional, TypeVar, Union -from aws_lambda_powertools.utilities.parser import BaseEnvelope, BaseModel +from typing import Any, Dict, Optional, Type, TypeVar, Union + +from pydantic import BaseModel + +from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser from aws_lambda_powertools.utilities.parser.models import EventBridgeModel -from aws_lambda_powertools.utilities.parser import event_parser from aws_lambda_powertools.utilities.typing import LambdaContext Model = TypeVar("Model", bound=BaseModel) + class EventBridgeEnvelope(BaseEnvelope): - def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: type[Model]) -> Optional[Model]: + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Type[Model]) -> Optional[Model]: if data is None: return None - parsed_envelope = EventBridgeModel.parse_obj(data) + parsed_envelope = EventBridgeModel.model_validate(data) return self._parse(data=parsed_envelope.detail, model=model) + class OrderDetail(BaseModel): order_id: str amount: float customer_id: str + @event_parser(model=OrderDetail, envelope=EventBridgeEnvelope) def lambda_handler(event: OrderDetail, context: LambdaContext): try: @@ -32,16 +37,15 @@ def lambda_handler(event: OrderDetail, context: LambdaContext): return { "statusCode": 200, - "body": json.dumps({ - "message": f"Order {event.order_id} processed successfully", - "order_id": event.order_id, - "amount": event.amount, - "customer_id": event.customer_id - }) + "body": json.dumps( + { + "message": f"Order {event.order_id} processed successfully", + "order_id": event.order_id, + "amount": event.amount, + "customer_id": event.customer_id, + }, + ), } except Exception as e: print(f"Error processing order: {str(e)}") - return { - "statusCode": 500, - "body": json.dumps({"error": "Internal server error"}) - } \ No newline at end of file + return {"statusCode": 500, "body": json.dumps({"error": "Internal server error"})} diff --git a/examples/parser/src/custom_data_model_with_eventbridge.py b/examples/parser/src/custom_data_model_with_eventbridge.py index 6cacc2e12a0..b9d0c4593b0 100644 --- a/examples/parser/src/custom_data_model_with_eventbridge.py +++ b/examples/parser/src/custom_data_model_with_eventbridge.py @@ -1,23 +1,21 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, parse, Field, ValidationError +from pydantic import Field, ValidationError + +from aws_lambda_powertools.utilities.parser import parse from aws_lambda_powertools.utilities.parser.models import EventBridgeModel + # Define a custom EventBridge model by extending the built-in EventBridgeModel class MyCustomEventBridgeModel(EventBridgeModel): detail_type: str = Field(alias="detail-type") source: str detail: dict + def lambda_handler(event: dict, context): try: # Manually parse the incoming event into the custom model parsed_event: MyCustomEventBridgeModel = parse(model=MyCustomEventBridgeModel, event=event) - return { - "statusCode": 200, - "body": f"Event from {parsed_event.source}, type: {parsed_event.detail_type}" - } + return {"statusCode": 200, "body": f"Event from {parsed_event.source}, type: {parsed_event.detail_type}"} except ValidationError as e: - return { - "statusCode": 400, - "body": f"Validation error: {str(e)}" - } \ No newline at end of file + return {"statusCode": 400, "body": f"Validation error: {str(e)}"} diff --git a/examples/parser/src/envelope_with_event_parser.py b/examples/parser/src/envelope_with_event_parser.py index 5de0ca8148f..ba222ff1190 100644 --- a/examples/parser/src/envelope_with_event_parser.py +++ b/examples/parser/src/envelope_with_event_parser.py @@ -1,23 +1,20 @@ -from aws_lambda_powertools.utilities.parser import event_parser, BaseModel -from aws_lambda_powertools.utilities.parser import envelopes +from pydantic import BaseModel + +from aws_lambda_powertools.utilities.parser import envelopes, event_parser from aws_lambda_powertools.utilities.typing import LambdaContext + class UserModel(BaseModel): - username: str - parentid_1: str - parentid_2: str + username: str + parentid_1: str + parentid_2: str + @event_parser(model=UserModel, envelope=envelopes.EventBridgeEnvelope) def lambda_handler(event: UserModel, context: LambdaContext): - if event.parentid_1!= event.parentid_2: - return { - "statusCode": 400, - "body": "Parent ids do not match" - } + if event.parentid_1 != event.parentid_2: + return {"statusCode": 400, "body": "Parent ids do not match"} # If parentids match, proceed with user registration - return { - "statusCode": 200, - "body": f"User {event.username} registered successfully" - } \ No newline at end of file + return {"statusCode": 200, "body": f"User {event.username} registered successfully"} diff --git a/examples/parser/src/field_validator.py b/examples/parser/src/field_validator.py index 9b1682d31d4..5af46bb4f41 100644 --- a/examples/parser/src/field_validator.py +++ b/examples/parser/src/field_validator.py @@ -1,25 +1,22 @@ -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.utilities.parser import field_validator +from pydantic import BaseModel, field_validator + +from aws_lambda_powertools.utilities.parser import parse from aws_lambda_powertools.utilities.typing import LambdaContext + class HelloWorldModel(BaseModel): message: str - @field_validator('message') + @field_validator("message") def is_hello_world(cls, v): if v != "hello world": raise ValueError("Message must be hello world!") return v + def lambda_handler(event: dict, context: LambdaContext): try: parsed_event = parse(model=HelloWorldModel, event=event) - return { - "statusCode": 200, - "body": f"Received message: {parsed_event.message}" - } + return {"statusCode": 200, "body": f"Received message: {parsed_event.message}"} except ValueError as e: - return { - "statusCode": 400, - "body": str(e) - } \ No newline at end of file + return {"statusCode": 400, "body": str(e)} diff --git a/examples/parser/src/getting_started_with_parser.py b/examples/parser/src/getting_started_with_parser.py index 0349c5b26b3..64625f8c87a 100644 --- a/examples/parser/src/getting_started_with_parser.py +++ b/examples/parser/src/getting_started_with_parser.py @@ -1,12 +1,14 @@ -from aws_lambda_powertools.utilities.parser import BaseModel +from pydantic import BaseModel + from aws_lambda_powertools.utilities.parser import event_parser + class MyEvent(BaseModel): id: int name: str + @event_parser(model=MyEvent) def lambda_handler(event: MyEvent, context): - #if your model is valid, you can return + # if your model is valid, you can return return {"statusCode": 200, "body": f"Hello {event.name}, your ID is {event.id}"} - \ No newline at end of file diff --git a/examples/parser/src/model_validator.py b/examples/parser/src/model_validator.py index 05f87fb96f6..8f9bd2d2d77 100644 --- a/examples/parser/src/model_validator.py +++ b/examples/parser/src/model_validator.py @@ -1,27 +1,31 @@ +from pydantic import BaseModel, model_validator + +from aws_lambda_powertools.utilities.parser import parse from aws_lambda_powertools.utilities.typing import LambdaContext -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.utilities.parser import model_validator + class UserModel(BaseModel): - username: str - parentid_1: str - parentid_2: str + username: str + parentid_1: str + parentid_2: str + + @model_validator(mode="after") # (1)! + def check_parents_match(cls, values): + pi1, pi2 = values.get("parentid_1"), values.get("parentid_2") + if pi1 is not None and pi2 is not None and pi1 != pi2: + raise ValueError("Parent ids do not match") + return values + - @model_validator(mode='after') - def check_parents_match(cls, values): - pi1, pi2 = values.get('parentid_1'), values.get('parentid_2') - if pi1 is not None and pi2 is not None and pi1 != pi2: - raise ValueError('Parent ids do not match') - return values def lambda_handler(event: dict, context: LambdaContext): try: parsed_event = parse(model=UserModel, event=event) return { "statusCode": 200, - "body": f"Received parent id from: {parsed_event.username}" + "body": f"Received parent id from: {parsed_event.username}", } except ValueError as e: return { "statusCode": 400, - "body": str(e) - } \ No newline at end of file + "body": str(e), + } diff --git a/examples/parser/src/parser_function.py b/examples/parser/src/parser_function.py index fbf3ba932b7..713bc2f5045 100644 --- a/examples/parser/src/parser_function.py +++ b/examples/parser/src/parser_function.py @@ -1,22 +1,19 @@ -from aws_lambda_powertools.utilities.parser import BaseModel, ValidationError +from pydantic import BaseModel, ValidationError + from aws_lambda_powertools.utilities.parser import parse + # Define a Pydantic model for the expected structure of the input class MyEvent(BaseModel): id: int name: str + def lambda_handler(event: dict, context): try: # Manually parse the incoming event into MyEvent model parsed_event: MyEvent = parse(model=MyEvent, event=event) - return { - "statusCode": 200, - "body": f"Hello {parsed_event.name}, your ID is {parsed_event.id}" - } + return {"statusCode": 200, "body": f"Hello {parsed_event.name}, your ID is {parsed_event.id}"} except ValidationError as e: # Catch validation errors and return a 400 response - return { - "statusCode": 400, - "body": f"Validation error: {str(e)}" - } \ No newline at end of file + return {"statusCode": 400, "body": f"Validation error: {str(e)}"} diff --git a/examples/parser/src/serialization_parser.py b/examples/parser/src/serialization_parser.py index 5f0dd86eade..ed6b16ca304 100644 --- a/examples/parser/src/serialization_parser.py +++ b/examples/parser/src/serialization_parser.py @@ -1,27 +1,26 @@ -from aws_lambda_powertools.utilities.parser import parse, BaseModel +from pydantic import BaseModel + from aws_lambda_powertools.logging import Logger +from aws_lambda_powertools.utilities.parser import parse from aws_lambda_powertools.utilities.typing import LambdaContext logger = Logger() + class UserModel(BaseModel): username: str parentid_1: str parentid_2: str + def validate_user(event): try: user = parse(model=UserModel, event=event) - return { - "statusCode": 200, - "body": user.model_dump_json() - } + return {"statusCode": 200, "body": user.model_dump_json()} except Exception as e: logger.exception("Validation error") - return { - "statusCode": 400, - "body": str(e) - } + return {"statusCode": 400, "body": str(e)} + @logger.inject_lambda_context def lambda_handler(event: dict, context: LambdaContext) -> dict: @@ -37,9 +36,6 @@ def lambda_handler(event: dict, context: LambdaContext) -> dict: user_dict = user.model_dump() user_json = user.model_dump_json() - logger.debug("User serializations", extra={ - "dict": user_dict, - "json": user_json - }) + logger.debug("User serializations", extra={"dict": user_dict, "json": user_json}) - return result \ No newline at end of file + return result diff --git a/examples/parser/src/string_fields_contain_json.py b/examples/parser/src/string_fields_contain_json.py index 4ba73a4b248..12446c081f2 100644 --- a/examples/parser/src/string_fields_contain_json.py +++ b/examples/parser/src/string_fields_contain_json.py @@ -1,19 +1,25 @@ from typing import Any, Type -from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel + +from pydantic import BaseModel, Json + +from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser from aws_lambda_powertools.utilities.typing import LambdaContext -from aws_lambda_powertools.utilities.parser.types import Json + class CancelOrder(BaseModel): order_id: int reason: str + class CancelOrderModel(BaseModel): body: Json[CancelOrder] + class CustomEnvelope(BaseEnvelope): def parse(self, data: dict, model: Type[BaseModel]) -> Any: return model.model_validate({"body": data.get("body", {})}) + @event_parser(model=CancelOrderModel, envelope=CustomEnvelope) def lambda_handler(event: CancelOrderModel, context: LambdaContext): cancel_order: CancelOrder = event.body @@ -25,5 +31,5 @@ def lambda_handler(event: CancelOrderModel, context: LambdaContext): return { "statusCode": 200, - "body": f"Order {cancel_order.order_id} cancelled successfully" - } \ No newline at end of file + "body": f"Order {cancel_order.order_id} cancelled successfully", + } From ebf6eda59f416416b6bffddde678d554d3385ff2 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Thu, 7 Nov 2024 11:37:05 +0000 Subject: [PATCH 11/13] Changing highlights + import --- .github/workflows/quality_check.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/quality_check.yml b/.github/workflows/quality_check.yml index f6f50c25abc..22561070bfd 100644 --- a/.github/workflows/quality_check.yml +++ b/.github/workflows/quality_check.yml @@ -20,6 +20,7 @@ on: paths: - "aws_lambda_powertools/**" - "tests/**" + - "examples/**" - "pyproject.toml" - "poetry.lock" - "mypy.ini" @@ -30,6 +31,7 @@ on: paths: - "aws_lambda_powertools/**" - "tests/**" + - "examples/**" - "pyproject.toml" - "poetry.lock" - "mypy.ini" From 9d3c809104d83f08a69f0e76252fdffafe451bb0 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Thu, 7 Nov 2024 11:38:28 +0000 Subject: [PATCH 12/13] Changing highlights + import --- docs/utilities/parser.md | 14 ++++---- .../parser/src/field_validator_all_values.py | 15 +++++---- examples/parser/src/sqs_model_event.py | 14 ++++---- .../parser/src/string_fields_contain_json.py | 16 +++++++-- ..._fields_contain_json_pydantic_validator.py | 33 +++++++++++++------ 5 files changed, 59 insertions(+), 33 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 096c39873b8..87a21d1e512 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -42,7 +42,7 @@ The `@event_parser` decorator automatically parses the incoming event into the s === "getting_started_with_parser.py" - ```python hl_lines="3 10" + ```python hl_lines="3 11" --8<-- "examples/parser/src/getting_started_with_parser.py" ``` @@ -58,7 +58,7 @@ You can use the `parse()` function when you need to have flexibility with differ === "parser_function.py" - ```python hl_lines="3 14" + ```python hl_lines="3 15" --8<-- "examples/parser/src/parser_function.py" ``` @@ -144,7 +144,7 @@ Use the model to validate and extract relevant information from the incoming eve === "Custom data model" - ```python hl_lines="4 8 16" + ```python hl_lines="4 8 17" --8<-- "examples/parser/src/custom_data_model_with_eventbridge.py" ``` @@ -164,7 +164,7 @@ Envelopes can be used via `envelope` parameter available in both `parse` functio === "Envelopes using event parser decorator" - ```python hl_lines="3 6-10 12" + ```python hl_lines="3 7-10 13" --8<-- "examples/parser/src/envelope_with_event_parser.py" ``` @@ -203,7 +203,7 @@ Here's a snippet of how the EventBridge envelope we demonstrated previously is i === "Bring your own envelope with Event Bridge" - ```python hl_lines="6 12-18" + ```python hl_lines="6 13-19" --8<-- "examples/parser/src/bring_your_own_envelope.py" ``` @@ -269,7 +269,7 @@ Wrap these fields with [Pydantic's Json Type](https://pydantic-docs.helpmanu === "Validate string fields containing JSON data" - ```python hl_lines="3 14" + ```python hl_lines="5 24" --8<-- "examples/parser/src/string_fields_contain_json.py" ``` @@ -287,7 +287,7 @@ Pydantic's definition of _serialization_ is broader than usual. It includes conv Read more at [Serialization for Pydantic documentation](https://docs.pydantic.dev/latest/concepts/serialization/#model_copy){target="_blank" rel="nofollow"}. -```python title="serialization_parser.py" hl_lines="39-40" +```python title="serialization_parser.py" hl_lines="36-37" --8<-- "examples/parser/src/serialization_parser.py" ``` diff --git a/examples/parser/src/field_validator_all_values.py b/examples/parser/src/field_validator_all_values.py index a7ca7cebce4..9a89b5495c4 100644 --- a/examples/parser/src/field_validator_all_values.py +++ b/examples/parser/src/field_validator_all_values.py @@ -1,26 +1,27 @@ -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.utilities.parser import field_validator +from aws_lambda_powertools.utilities.parser import BaseModel, field_validator, parse from aws_lambda_powertools.utilities.typing import LambdaContext + class HelloWorldModel(BaseModel): message: str sender: str - @field_validator('*') + @field_validator("*") def has_whitespace(cls, v): - if ' ' not in v: + if " " not in v: raise ValueError("Must have whitespace...") return v + def lambda_handler(event: dict, context: LambdaContext): try: parsed_event = parse(model=HelloWorldModel, event=event) return { "statusCode": 200, - "body": f"Received message: {parsed_event.message}" + "body": f"Received message: {parsed_event.message}", } except ValueError as e: return { "statusCode": 400, - "body": str(e) - } \ No newline at end of file + "body": str(e), + } diff --git a/examples/parser/src/sqs_model_event.py b/examples/parser/src/sqs_model_event.py index 80976711946..8093a230df6 100644 --- a/examples/parser/src/sqs_model_event.py +++ b/examples/parser/src/sqs_model_event.py @@ -3,13 +3,15 @@ from aws_lambda_powertools.utilities.typing import LambdaContext -def lambda_handler(event: dict, context: LambdaContext) -> None: +def lambda_handler(event: dict, context: LambdaContext) -> list: parsed_event = parse(model=SqsModel, event=event) results = [] for record in parsed_event.Records: - results.append({ - 'Message ID':record.messageId, - 'body':record.body - }) - return results \ No newline at end of file + results.append( + { + "message_id": record.messageId, + "body": record.body, + }, + ) + return results diff --git a/examples/parser/src/string_fields_contain_json.py b/examples/parser/src/string_fields_contain_json.py index 12446c081f2..3055bed7e7d 100644 --- a/examples/parser/src/string_fields_contain_json.py +++ b/examples/parser/src/string_fields_contain_json.py @@ -1,10 +1,19 @@ -from typing import Any, Type +from __future__ import annotations + +from typing import TYPE_CHECKING, Any from pydantic import BaseModel, Json from aws_lambda_powertools.utilities.parser import BaseEnvelope, event_parser +from aws_lambda_powertools.utilities.parser.functions import ( + _parse_and_validate_event, + _retrieve_or_set_model_from_cache, +) from aws_lambda_powertools.utilities.typing import LambdaContext +if TYPE_CHECKING: + from aws_lambda_powertools.utilities.parser.types import T + class CancelOrder(BaseModel): order_id: int @@ -16,8 +25,9 @@ class CancelOrderModel(BaseModel): class CustomEnvelope(BaseEnvelope): - def parse(self, data: dict, model: Type[BaseModel]) -> Any: - return model.model_validate({"body": data.get("body", {})}) + def parse(self, data: dict[str, Any] | Any | None, model: type[T]): + adapter = _retrieve_or_set_model_from_cache(model=model) + return _parse_and_validate_event(data=data, adapter=adapter) @event_parser(model=CancelOrderModel, envelope=CustomEnvelope) diff --git a/examples/parser/src/string_fields_contain_json_pydantic_validator.py b/examples/parser/src/string_fields_contain_json_pydantic_validator.py index 4d06f871dcb..5c19606736d 100644 --- a/examples/parser/src/string_fields_contain_json_pydantic_validator.py +++ b/examples/parser/src/string_fields_contain_json_pydantic_validator.py @@ -1,25 +1,38 @@ +from __future__ import annotations + import json -from typing import Any, Type -from aws_lambda_powertools.utilities.parser import event_parser, BaseEnvelope, BaseModel -from aws_lambda_powertools.utilities.validation import validator +from typing import TYPE_CHECKING, Any + +from aws_lambda_powertools.utilities.parser import BaseEnvelope, BaseModel, event_parser +from aws_lambda_powertools.utilities.parser.functions import ( + _parse_and_validate_event, + _retrieve_or_set_model_from_cache, +) from aws_lambda_powertools.utilities.typing import LambdaContext +from aws_lambda_powertools.utilities.validation import validator + +if TYPE_CHECKING: + from aws_lambda_powertools.utilities.parser.types import T + class CancelOrder(BaseModel): order_id: int reason: str + class CancelOrderModel(BaseModel): body: CancelOrder @validator("body", pre=True) def transform_body_to_dict(cls, value): - if isinstance(value, str): - return json.loads(value) - return value + return json.loads(value) if isinstance(value, str) else value + class CustomEnvelope(BaseEnvelope): - def parse(self, data: dict, model: Type[BaseModel]) -> Any: - return model.model_validate({"body": data.get("body", {})}) + def parse(self, data: dict[str, Any] | Any | None, model: type[T]): + adapter = _retrieve_or_set_model_from_cache(model=model) + return _parse_and_validate_event(data=data, adapter=adapter) + @event_parser(model=CancelOrderModel, envelope=CustomEnvelope) def lambda_handler(event: CancelOrderModel, context: LambdaContext): @@ -32,5 +45,5 @@ def lambda_handler(event: CancelOrderModel, context: LambdaContext): return { "statusCode": 200, - "body": json.dumps({"message": f"Order {cancel_order.order_id} cancelled successfully"}) - } \ No newline at end of file + "body": json.dumps({"message": f"Order {cancel_order.order_id} cancelled successfully"}), + } From 4d10ff8acada4644d3bb0d249ba8d639b458c107 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Thu, 7 Nov 2024 17:01:53 +0000 Subject: [PATCH 13/13] Addressing Andrea's feedback --- docs/utilities/parser.md | 42 ++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 87a21d1e512..4c86c983d31 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -13,7 +13,7 @@ The Parser utility simplifies data parsing and validation using [Pydantic](https - Parse and validate Lambda event payloads - Built-in support for common AWS event sources - Runtime type checking with user-friendly error messages -- Compatible with Pydantic v2 +- Compatible with Pydantic v2.x ## Getting started @@ -22,14 +22,14 @@ The Parser utility simplifies data parsing and validation using [Pydantic](https Powertools only supports Pydantic v2, so make sure to install the required dependencies for Pydantic v2 before using the Parser. ```python -pip install aws-lambda-powertools[tracer] +pip install aws-lambda-powertools[parser] ``` !!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" -You can also add as a dependency in your preferred tool: `e.g., requirements.txt, pyproject.toml`. +You can also add as a dependency in your preferred tool: `e.g., requirements.txt, pyproject.toml`, etc. -### Data Model with Parse +### Data Model with Parser You can define models by inheriting from `BaseModel` or any other supported type through `TypeAdapter` to parse incoming events. Pydantic then validates the data, ensuring that all fields conform to the specified types and maintaining data integrity. @@ -38,7 +38,7 @@ You can define models by inheriting from `BaseModel` or any other supported type #### Event parser -The `@event_parser` decorator automatically parses the incoming event into the specified Pydantic model `MyEvent`. If the input doesn't match the model's structure or type requirements, we raises a `ValidationError` directly from Pydantic. +The `@event_parser` decorator automatically parses the incoming event into the specified Pydantic model `MyEvent`. If the input doesn't match the model's structure or type requirements, it raises a `ValidationError` directly from Pydantic. === "getting_started_with_parser.py" @@ -68,7 +68,7 @@ You can use the `parse()` function when you need to have flexibility with differ --8<-- "examples/parser/src/example_event_parser.json" ``` -#### Keys difference between parse and event_parser +#### Keys differences between parse and event_parser TheĀ `parse()`Ā function offers more flexibility and control: @@ -84,7 +84,7 @@ TheĀ `@event_parser`Ā decorator is ideal for: ### Built-in models -You can use pre-built models provided by the Parser for parsing events from AWS services, so you don’t need to create these models yourself, we’ve already done that for you. +You can use pre-built models to work events from AWS services, so you don’t need to create them yourself. We’ve already done that for you! === "sqs_model_event.py" @@ -158,7 +158,7 @@ Use the model to validate and extract relevant information from the incoming eve ### Envelopes -You can use **Envelopes**, which are **JMESPath expressions**, to extract specific portions of complex, nested JSON structures. This is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like **EventBridge**. +You can use **Envelopes** to extract specific portions of complex, nested JSON structures. This is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like **EventBridge**. Envelopes can be used via `envelope` parameter available in both `parse` function and `event_parser` decorator. @@ -181,23 +181,23 @@ You can use pre-built envelopes provided by the Parser to extract and parse spec | Envelope name | Behaviour | Return | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | | **DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`. `` 2. Parses records in `NewImage` and `OldImage` keys using your model. `` 3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` | -| **EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`. ``2. Parses`detail` key using your model and returns it. | `Model` | -| **SqsEnvelope** | 1. Parses data using `SqsModel`. ``2. Parses records in`body` key using your model and return them in a list. | `List[Model]` | -| **CloudWatchLogsEnvelope** | 1. Parses data using `CloudwatchLogsModel` which will base64 decode and decompress it. ``2. Parses records in`message` key using your model and return them in a list. | `List[Model]` | -| **KinesisDataStreamEnvelope** | 1. Parses data using `KinesisDataStreamModel` which will base64 decode it. ``2. Parses records in in`Records` key using your model and returns them in a list. | `List[Model]` | -| **KinesisFirehoseEnvelope** | 1. Parses data using `KinesisFirehoseModel` which will base64 decode it. ``2. Parses records in in`Records` key using your model and returns them in a list. | `List[Model]` | -| **SnsEnvelope** | 1. Parses data using `SnsModel`. ``2. Parses records in`body` key using your model and return them in a list. | `List[Model]` | +| **EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`. ``2. Parses `detail` key using your model`` and returns it. | `Model` | +| **SqsEnvelope** | 1. Parses data using `SqsModel`. ``2. Parses records in `body` key using your model`` and return them in a list. | `List[Model]` | +| **CloudWatchLogsEnvelope** | 1. Parses data using `CloudwatchLogsModel` which will base64 decode and decompress it. ``2. Parses records in `message` key using your model`` and return them in a list. | `List[Model]` | +| **KinesisDataStreamEnvelope** | 1. Parses data using `KinesisDataStreamModel` which will base64 decode it. ``2. Parses records in in `Records` key using your model`` and returns them in a list. | `List[Model]` | +| **KinesisFirehoseEnvelope** | 1. Parses data using `KinesisFirehoseModel` which will base64 decode it. ``2. Parses records in in` Records` key using your model`` and returns them in a list. | `List[Model]` | +| **SnsEnvelope** | 1. Parses data using `SnsModel`. ``2. Parses records in `body` key using your model`` and return them in a list. | `List[Model]` | | **SnsSqsEnvelope** | 1. Parses data using `SqsModel`. `` 2. Parses SNS records in `body` key using `SnsNotificationModel`. `` 3. Parses data in `Message` key using your model and return them in a list. | `List[Model]` | -| **ApiGatewayEnvelope** | 1. Parses data using `APIGatewayProxyEventModel`. ``2. Parses`body` key using your model and returns it. | `Model` | -| **ApiGatewayV2Envelope** | 1. Parses data using `APIGatewayProxyEventV2Model`. ``2. Parses`body` key using your model and returns it. | `Model` | -| **LambdaFunctionUrlEnvelope** | 1. Parses data using `LambdaFunctionUrlModel`. ``2. Parses`body` key using your model and returns it. | `Model` | -| **KafkaEnvelope** | 1. Parses data using `KafkaRecordModel`. ``2. Parses`value` key using your model and returns it. | `Model` | -| **VpcLatticeEnvelope** | 1. Parses data using `VpcLatticeModel`. ``2. Parses`value` key using your model and returns it. | `Model` | -| **BedrockAgentEnvelope** | 1. Parses data using `BedrockAgentEventModel`. ``2. Parses`inputText` key using your model and returns it. | `Model` | +| **ApiGatewayEnvelope** | 1. Parses data using `APIGatewayProxyEventModel`. ``2. Parses `body` key using your model`` and returns it. | `Model` | +| **ApiGatewayV2Envelope** | 1. Parses data using `APIGatewayProxyEventV2Model`. ``2. Parses `body` key using your model`` and returns it. | `Model` | +| **LambdaFunctionUrlEnvelope** | 1. Parses data using `LambdaFunctionUrlModel`. ``2. Parses `body` key using your model`` and returns it. | `Model` | +| **KafkaEnvelope** | 1. Parses data using `KafkaRecordModel`. ``2. Parses `value` key using your model`` and returns it. | `Model` | +| **VpcLatticeEnvelope** | 1. Parses data using `VpcLatticeModel`. ``2. Parses `value` key using your model`` and returns it. | `Model` | +| **BedrockAgentEnvelope** | 1. Parses data using `BedrockAgentEventModel`. ``2. Parses `inputText` key using your model`` and returns it. | `Model` | #### Bringing your own envelope -You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method or `@event_parser` decorator. +You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method. Here's a snippet of how the EventBridge envelope we demonstrated previously is implemented.