InsuranceLake Quickstart with CI/CD Guide

If you’ve determined that InsuranceLake is a good starting point for your own serverless data lake and would like to rapidly iterate through development cycles with one or more teams, we recommend deploying with a CI/CD pipeline. Follow the steps in this section to create your CodePipeline stack, and to use it to deploy the InsuranceLake resources:

1. If this is your first time using the application, follow the Python/CDK Basics steps.
2. Use a terminal or command prompt and change the working directory to the location of the infrastruture code.

    cd aws-insurancelake-infrastructure
   

3. Open the lib/configuration.py file using vi or nano.

    nano +81 lib/configuration.py
   

    vi +81 lib/configuration.py
   

4. Review the local_mapping structure in the get_local_configuration() function.
   • Specifically, the Regions and account IDs should make sense for your environments. These values in the repository (not locally) will be used by AWS CodeCommit and need to be maintained in the repository.
   • The values for the Test and Production environments can be ommitted at this time, because we will only be deploying the Deployment and Development environments.
   • You must explicitly specify the account and Region for each environment so that the infrastructure virtual private clouds (VPCs) get three Availability Zones (if the Region has them available). Review the reference documentation.
5. Deploy the CodeCommit repository stack.

    cdk deploy Deploy-InsuranceLakeInfrastructureMirrorRepository
   

   • While this stack is designed for a mirror repository, it can also be used as a main repository for your InsuranceLake code. You can follow links to help setup other repository types here:
     • GitHub
     • Bitbucket
     • GitLab
6. If you plan to use CodeCommit as the main repository, install the Git CodeCommit Helper:

    sudo pip install git-remote-codecommit
   

7. Initialize git, create a develop branch, perform initial commit, and push to remote.
   • We are using the develop branch because the Dev environment deployment is triggered by commits to the develop branch.
   • Edit the repository URL to correspond to your version control system if you are not using CodeCommit.

      git init
      git branch -M develop
      git add .
      git commit -m 'Initial commit'
      git remote add origin codecommit::us-east-2://aws-insurancelake-infrastructure
      git push --set-upstream origin develop
     

8. Deploy the Infrastructure CodePipeline stack in the development environment (one stack).

    cdk deploy Dev-InsuranceLakeInfrastructurePipeline
   

9. Review and accept IAM credential creation for the CodePipeline stack.
   • Wait for deployment to finish (approximately 5 minutes).
10. Open CodePipeline in the AWS Console and select the dev-insurancelake-infrastructure-pipeline Pipeline.
    • The first run of the pipeline starts automatically after the Pipeline stack is deployed.
11. Monitor the status of the pipeline until complete.
12. Switch the working directory to the location of the ETL code.

     cd ../aws-insurancelake-etl
    

13. In lib/configuration.py, review the local_mapping structure in the get_local_configuration() function, and ensure this matches the Infrastructure configuration, or differs if specifically needed.
14. Deploy the CodeCommit repository stack.

     cdk deploy Deploy-InsuranceLakeEtlMirrorRepository
    

15. Initialize git, create a develop branch, perform initial commit, and push to remote.
    • We are using the develop branch because the Dev environment deployment is triggered by commits to the develop branch.
    • Edit the repository URL to correspond to your version control system if you are not using CodeCommit.

       git init
       git branch -M develop
       git add .
       git commit -m 'Initial commit'
       git remote add origin codecommit::us-east-2://aws-insurancelake-etl
       git push --set-upstream origin develop
      

16. Deploy the ETL CodePipeline stack in the development environment (one stack).

     cdk deploy Dev-InsuranceLakeEtlPipeline
    

17. Review and accept IAM credential creation for the CodePipeline stack.
    • Wait for deployment to finish (approximately 5 minutes).
18. Open CodePipeline in the AWS Console and select the dev-insurancelake-etl-pipeline Pipeline.
    • The first run of the pipeline starts automatically after the Pipeline stack is deployed.
19. Monitor the status of the pipeline until completed.