InsuranceLake Deployment Validation

1. If you have not previously set the default AWS region for your session, you must set it now.

   The default region for InsuranceLake is us-east-2.

    export AWS_DEFAULT_REGION=<replace with your region>
   

2. Transfer the sample claim data to the Collect bucket (Source system: SyntheticData, Table: ClaimData).

    export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    aws s3 cp resources/syntheticgeneral-claim-data.csv s3://dev-insurancelake-${AWS_ACCOUNT_ID}-${AWS_DEFAULT_REGION}-collect/SyntheticGeneralData/ClaimData/
   

3. The claim data workflow must complete loading of the data into the Cleanse bucket to successfully load the policy data in the following step.

4. Transfer the sample policy data to the Collect bucket (Source system: SyntheticData, Table: PolicyData).

    aws s3 cp resources/syntheticgeneral-policy-data.csv s3://dev-insurancelake-${AWS_ACCOUNT_ID}-${AWS_DEFAULT_REGION}-collect/SyntheticGeneralData/PolicyData/
   

5. Upon successful transfer of the file, an event notification from S3 will trigger the state-machine-trigger Lambda function.

6. The Lambda function will insert a record into the DynamoDB table {environment}-{resource_name_prefix}-etl-job-audit to track job start status.

7. The Lambda function will also trigger the Step Functions State Machine. The State Machine execution name will be <filename>-<YYYYMMDDHHMMSSxxxxxx> and have the required metadata as input parameters.

8. The State Machine will trigger the AWS Glue job for Collect to Cleanse data processing.

9. The Collect to Cleanse AWS Glue job will execute the transformation logic defined in configuration files.

10. The AWS Glue job will load the data into the Cleanse bucket using the provided metadata. The data will be stored in S3 as s3://{environment}-{resource_name_prefix}-{account}-{region}-cleanse/syntheticgeneraldata/claimdata/year=YYYY/month=MM/day=DD in Apache Parquet format.

11. The AWS Glue job will create or update the AWS Glue Catalog table using the table name passed as a parameter based on the folder name (PolicyData and ClaimData).

12. After the Collect to Cleanse AWS Glue job completes, the State Machine will trigger the Cleanse to Consume AWS Glue job.

13. The Cleanse to Consume AWS Glue job will execute the SQL logic defined in configuration files.

14. The Cleanse to Consume AWS Glue job will store the resulting data set in S3 as s3://{environment}-{resource_name_prefix}-{account}-{region}-consume/syntheticgeneraldata/claimdata/year=YYYY/month=MM/day=DD in Apache Parquet format.

15. The Cleanse to Consume AWS Glue job will create or update the AWS Glue Catalog table.

16. After successful completion of the Cleanse to Consume AWS Glue job, the State Machine will trigger the etl-job-auditor Lambda function to update the DynamoDB table {environment}-{resource_name_prefix}-etl-job-audit with the latest status.

17. An Amazon Simple Notification Service (Amazon SNS) notification will be sent to all subscribed users.

18. To validate the data load, use Athena and execute the following query:

     select * from syntheticgeneraldata_consume.policydata limit 100