InsuranceLake Quickstart with CI/CD Guide

If you’ve determined that InsuranceLake is a good starting point for your own serverless data lake and would like to rapidly iterate through development cycles with one or more teammates, we recommend deploying it with a CI/CD pipeline.

The steps in this section assume you want to develop and deploy locally. Follow the steps to create your CodePipeline stack and to use it to deploy the InsuranceLake resources.

If this is your first time using the application in this environment, follow the CDK Instructions steps to setup your local environment for AWS CDK development.
You will likely need to make configuration changes to the code. Ensure that you have forked the repository on Github or copied the source code to a separate repository.
Refer to Copy OpenSource Git Repositories for steps on copying the source code to a local repository.
Clone the repositories locally.
Modify the below commands to use the correct URL for your repositories. Subsequent steps will assume you are using the repository names aws-insurancelake-infrastructure and aws-insurancelake-etl.
```
 git clone https://path/to/aws-insurancelake-infrastructure
 git clone https://path/to/aws-insurancelake-etl
```
Use a terminal or command prompt and change the working directory to the location of the infrastruture code.
```
 cd aws-insurancelake-infrastructure
```

Open the lib/configuration.py file using vi or nano.

 nano +77 lib/configuration.py

 vi +77 lib/configuration.py

Review the local_mapping structure in the get_local_configuration() function and make changes where necessary.
- Specifically, the Regions and account IDs should make sense for your environments. These values in the repository (not locally) will be used by AWS CodePipeline and need to be maintained in the repository.
- The values for the Test and Production environments can be ommitted at this time, because we will only be deploying the Deployment and Development environments in the same account.
- You must explicitly specify the account and Region for each environment so that the infrastructure virtual private clouds (VPCs) get three Availability Zones. Review the reference documentation.
- Refer to Application Configuration from the Full Deployment Guide for more details on configuration parameters.
If you modified the configuration, copy configuration.py file to the ETL repository.
```
 cp lib/configuration.py ../aws-insurancelake-etl/lib/
```
Edit the configuration.py in the ETL repository and modify the repository configuration parameters to reference the ETL code repository.
We recommend that you keep the logical ID prefix and resource name prefix consistent between repositories.
Create a CodeConnections connection for your forked repository and configure the repository parameters by following the AWS CodePipeline and Git Integration guide.
Bootstrap CDK in your AWS account.
```
 cdk bootstrap
```
Deploy the Infrastructure CodePipeline stack in the development environment (one stack).
```
 cdk deploy Dev-InsuranceLakeInfrastructurePipeline
```
Review and accept IAM credential creation for the CodePipeline stack.
- Wait for deployment to finish (approximately 5 minutes).
Open CodePipeline in the AWS Console and select the dev-insurancelake-infrastructure-pipeline Pipeline.
- The first run of the pipeline starts automatically after the Pipeline stack is deployed.
Monitor the status of the pipeline until complete.
Switch the working directory to the location of the ETL code.
```
 cd ../aws-insurancelake-etl
```
Deploy the ETL CodePipeline stack in the development environment (one stack).
```
 cdk deploy Dev-InsuranceLakeEtlPipeline
```
Review and accept IAM credential creation for the CodePipeline stack.
- Wait for deployment to finish (approximately 5 minutes).
Open CodePipeline in the AWS Console and select the dev-insurancelake-etl-pipeline Pipeline.
- The first run of the pipeline starts automatically after the Pipeline stack is deployed.
Monitor the status of the pipeline until completed.

You can now make changes locally, commit them to the repository, and CodePipeline will automatically deploy them to your AWS account.

InsuranceLake Quickstart with CI/CD Guide

Further Reading