Overview
This Guidance recommends using Index State Management from Amazon OpenSearch Service to optimize OpenSearch costs. Since there is no CDK support yet, the datastream/index/lifecycle policy were created through Open Search Dashboards. This guide walkthroughd the manual steps.
Pre-requisites:
VideoLogisticsOpensearchStack from VideoAnalyticsVideoLogisticsCDK has been deployed.
Use OpenSearch Dashboards for Operations
It is easier to run operations like index management, index query from dashboards. Steps to set up:
- Login to your AWS account
- Access Amazon Opensearch Service on the console and select on
valopensearchdomain - Temporarily enable Cognito auth to access OpenSearch dashboards by clicking Security Configuration → Edit → Enable Amazon Cognito Authentication
- Select the proper region, and create a new user pool, identity pool, and user in the user pool
- Add following statement in the access policy in Security Configuration:
{ "Effect": "Allow", "Principal": { "AWS": {IAM role name of Authentication for OpenSearch Dashboards/Kibana} }, "Action": "es:ESHttp", "Resource": [ "arn:aws:es:{region}:{accountId}:domain/valopensearchdomain", "arn:aws:es:{region}:{accountId}:domain/valopensearchdomain/*" ] } - Wait for the domain status to update
- Enter OpenSearch dashboard by clicking the url
- Click Interact with the OpenSearch API in OpenSearch dashboard
- After the operations, disable cognito authentication (or else you might get some security violations/warnings)
Create Index Templates
There are 3 data streams, each with an index template.
- Trajectory Template
PUT _index_template/trajectory-1.0-template { "index_patterns": "trajectory-1.0", "data_stream": { "timestamp_field": { "name": "timestamp" } }, "template": { "settings": { "index": { "max_inner_result_window": 1000 } }, "mappings": { "properties": { "timestamp": { "type": "date" }, "metadata": { "type": "object", "properties": { "deviceId": { "type": "keyword" }, "thumbnailS3Paths": { "type": "keyword" } } }, "modelOutput": { "type": "object", "properties": { "MetadataStream": { "type": "object", "properties": { "VideoAnalytics": { "type": "object", "properties": { "Frame": { "type": "object", "properties": { "Source": { "type": "keyword" }, "UtcTime": { "type": "date" }, "Transformation": { "type": "object", "properties": { "Translate": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } }, "Scale": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Object": { "type": "object", "properties": { "ObjectId": { "type": "long" }, "Appearance": { "type": "object", "properties": { "Shape": { "type": "object", "properties": { "BoundingBox": { "type": "object", "properties": { "left": { "type": "double" }, "right": { "type": "double" }, "top": { "type": "double" }, "bottom": { "type": "double" } } }, "CenterOfGravity": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Class": { "type": "object", "properties": { "Type": { "type": "object", "properties": { "Likelihood": { "type": "double" }, "txt": { "type": "keyword" } } } } } } }, "Extension": { "type": "object", "properties": { "timeText": { "type": "keyword" }, "TriggerDateTime": { "type": "keyword" }, "BestThumbnailCapDateTimeMs": { "type": "keyword" }, "BestThumbnailCapDateTime": { "type": "keyword" } } } } } } } } } } } } } } } } }Put some dummy data to trigger data stream creation (don’t need this if we already have camera setup to ingest inference into cloud. The data stream creation is required for PIT lambda to succeed).
POST _bulk {"create":{"_index":"trajectory-1.0"}} {"timestamp":1586672099,"metadata":{"deviceId":"TestDevice","thumbnailS3Paths":["testThumbnailPath"]},"modelOutput": {"MetadataStream": {"VideoAnalytics": {"Frame": {"Source": "VA_TRAJ_BASIC","UtcTime": "2023-10-18T21:55:13","Transformation": {"Translate": {"x": -1,"y": 1},"Scale": {"x": 1,"y": 1}},"Object": {"ObjectId": 6,"Appearance": {"Shape": {"BoundingBox": {"bottom": 0.2160579115152359,"left": 0.37358295917510986,"right": 0.39994168281555176,"top": 0.12755586206912994},"CenterOfGravity": {"x": 0.3867623209953308,"y": 0.17180688679218292}},"Class": {"Type": {"Likelihood": 0.8054792284965515,"txt": "Person"}}},"Extension": {"timeText": "2023-06-30T10:44:13+09:00","TriggerDateTime": "2023-06-30T10:44:13.564+0900","BestThumbnailCapDateTimeMs": "2023-06-30T10:44:13.564+0900","BestThumbnailCapDateTime": "2023-06-30T10:44:13.564+0900 2023-06-30T11:44:13.564+0900"}}}}}}} - Event Template
PUT _index_template/event-1.0-template { "index_patterns": "event-1.0", "data_stream": { "timestamp_field": { "name": "timestamp" } }, "template": { "settings": { "index": { "max_inner_result_window": 1000 } }, "mappings": { "properties": { "timestamp": { "type": "date" }, "metadata": { "type": "object", "properties": { "deviceId": { "type": "keyword" }, "thumbnailS3Paths": { "type": "keyword" } } }, "modelOutput": { "type": "object", "properties": { "MetadataStream": { "type": "object", "properties": { "VideoAnalytics": { "type": "object", "properties": { "Frame": { "type": "object", "properties": { "Source": { "type": "keyword" }, "UtcTime": { "type": "date" }, "Transformation": { "type": "object", "properties": { "Translate": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } }, "Scale": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Object": { "type": "nested", "properties": { "ObjectId": { "type": "long" }, "Appearance": { "type": "object", "properties": { "Shape": { "type": "object", "properties": { "BoundingBox": { "type": "object", "properties": { "left": { "type": "double" }, "right": { "type": "double" }, "top": { "type": "double" }, "bottom": { "type": "double" } } }, "CenterOfGravity": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Class": { "type": "object", "properties": { "Type": { "type": "object", "properties": { "Likelihood": { "type": "double" }, "txt": { "type": "keyword" } } } } } } }, "Extension": { "type": "object", "properties": { "RoiId": { "type": "long" }, "time": { "type": "long" }, "timeText": { "type": "keyword" }, "TriggerDateTime": { "type": "keyword" }, "Sensitivity": { "type": "double" }, "BestThumbnailCapDateTimes": { "type": "keyword" }, "direction": { "type": "byte" } } } } } } } } } } } } } } } } }Put some dummy data to trigger data stream creation (don’t need this if we already have camera setup to ingest inference into cloud. The data stream creation is required for PIT lambda to succeed).
POST _bulk {"create":{"_index":"event-1.0"}} {"timestamp":1586672099,"metadata":{"deviceId":"TestDevice","thumbnailS3Paths":["testThumbnailPath"]},"modelOutput": {"MetadataStream": {"VideoAnalytics": {"Frame": {"Source": "VA_HEAT_BI","UtcTime": "2023-10-18T21:55:13","Transformation": {"Translate": {"x": -1,"y": 1},"Scale": {"x": 1,"y": 1}},"Object": {"ObjectId": 6,"Appearance": {"Shape": {"BoundingBox": {"bottom": 0.2160579115152359,"left": 0.37358295917510986,"right": 0.39994168281555176,"top": 0.12755586206912994},"CenterOfGravity": {"x": 0.3867623209953308,"y": 0.17180688679218292}},"Class": {"Type": {"Likelihood": 0.8054792284965515,"txt": "Person"}}},"Extension": {"RoiId": 3,"time": 1688089453,"timeText": "2023-06-30T10:44:13+09:00","TriggerDateTime": "2023-06-30T10:44:13.564+0900","Sensitivity": 0.6231524348258972,"direction": 1}}}}}}} UpdateAttributes Template
PUT _index_template/updateattributes-1.0-template { "index_patterns": "updateattributes-1.0", "data_stream": { "timestamp_field": { "name": "timestamp" } }, "template": { "settings": { "index": { "max_inner_result_window": 1000 } }, "mappings": { "properties": { "timestamp": { "type": "date" }, "metadata": { "type": "object", "properties": { "deviceId": { "type": "keyword" }, "thumbnailS3Paths": { "type": "keyword" } } }, "modelOutput": { "type": "object", "properties": { "MetadataStream": { "type": "object", "properties": { "VideoAnalytics": { "type": "object", "properties": { "Frame": { "type": "object", "properties": { "Source": { "type": "keyword" }, "UtcTime": { "type": "date" }, "Transformation": { "type": "object", "properties": { "Translate": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } }, "Scale": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Object": { "type": "object", "properties": { "ObjectId": { "type": "long" }, "Appearance": { "type": "object", "properties": { "Shape": { "type": "object", "properties": { "BoundingBox": { "type": "object", "properties": { "left": { "type": "double" }, "right": { "type": "double" }, "top": { "type": "double" }, "bottom": { "type": "double" } } }, "CenterOfGravity": { "type": "object", "properties": { "x": { "type": "double" }, "y": { "type": "double" } } } } }, "Color": { "type": "object", "properties": { "Extension": { "type": "object", "properties": { "topColorStep1": { "type": "short" }, "topColorStep2": { "type": "short" }, "botColorStep1": { "type": "short" }, "botColorStep2": { "type": "short" }, "topColor1_H": { "type": "short" }, "topColor1_S": { "type": "short" }, "topColor1_V": { "type": "short" }, "topColor2_H": { "type": "short" }, "topColor2_S": { "type": "short" }, "topColor2_V": { "type": "short" }, "botColor1_H": { "type": "short" }, "botColor1_S": { "type": "short" }, "botColor1_V": { "type": "short" }, "botColor2_H": { "type": "short" }, "botColor2_S": { "type": "short" }, "botColor2_V": { "type": "short" } } } } }, "Class": { "type": "object", "properties": { "Type": { "type": "object", "properties": { "Likelihood": { "type": "double" }, "txt": { "type": "keyword" } } } } }, "Extension": { "type": "object", "properties": { "age": { "type": "short" }, "ageGroup": { "type": "short" }, "gender": { "type": "short" } } } } }, "Extension": { "type": "object", "properties": { "cls": { "type": "long" }, "EnterDt": { "type": "long" }, "EnterDtText": { "type": "keyword" }, "EnterDtTextMs": { "type": "keyword" }, "ExitDt": { "type": "long" }, "ExitDtText": { "type": "keyword" }, "ExitDtTextMs": { "type": "keyword" }, "BestThumbnailCapDateTime": { "type": "keyword" }, "BestThumbnailCapDateTimeMs": { "type": "keyword" }, "timeText": { "type": "keyword" }, "TriggerDateTime": { "type": "keyword" } } } } } } } } } } } } } } } } }Put some dummy data to trigger data stream creation (don’t need this if we already have camera setup to ingest inference into cloud. The data stream creation is required for PIT lambda to succeed).
POST _bulk {"create":{"_index":"updateattributes-1.0"}} {"timestamp":1586672099,"metadata":{"deviceId":"TestDevice","thumbnailS3Paths":["testThumbnailPath"]},"modelOutput":{"MetadataStream":{"VideoAnalytics":{"Frame":{"Object":{"Appearance":{"Class":{"Type":{"Likelihood":0.5679365992546082,"txt":"Person"}},"Color":{"Extension":{"botColor1_H":176,"botColor1_S":60,"botColor1_V":21,"botColor2_H":0,"botColor2_S":0,"botColor2_V":0,"botColorStep1":0,"botColorStep2":-1,"topColor1_H":7,"topColor1_S":98,"topColor1_V":98,"topColor2_H":0,"topColor2_S":0,"topColor2_V":0,"topColorStep1":10,"topColorStep2":-1}},"Shape":{"BoundingBox":{"bottom":0.9854872226715088,"left":0.4674275815486908,"right":0.87078857421875,"top":0.0164659284055233},"CenterOfGravity":{"x":0.6691080927848816,"y":0.5009765625}}},"Extension":{"BestThumbnailCapDateTime":"2023-10-11T07:22:46+09:00","BestThumbnailCapDateTimeMs":"2023-10-11T07:22:46.753+0900","EnterDt":1696976564,"EnterDtText":"2023-10-11T07:22:44+09:00","EnterDtTextMs":"2023-10-11T07:22:44.531+0900","ExitDt":1696976571,"ExitDtText":"2023-10-11T07:22:51+09:00","ExitDtTextMs":"2023-10-11T07:22:51.943+0900","TriggerDateTime":"2023-10-11T07:22:51.943+0900","cls":0,"timeText":"2023-10-11T07:22:51+09:00"},"ObjectId":10191},"Source":"VA_ATTR_BASIC","Transformation":{"Scale":{"x":1,"y":1},"Translate":{"x":-1,"y":1}},"UtcTime":"2023-10-10T22:22:52"}}}}}
Apply Index State Management
Create ISM Policy
- Rollover to a new index for write when index size reaches 100 GB (20 GB per shard * 5 shards)
- Move from Hot to Warm after 12 hours
- Delete inference from Warm when lifetime is over 30 days (the lifetime starts from the time inference ingested in the Video Analytics Guidance, not the time put into Warm)
PUT _plugins/_ism/policies/inference_retention_policy { "policy": { "description": "Inference index state management policy for retention", "default_state": "hot", "schema_version": 1, "states": [{ "name": "hot", "actions": [{ "rollover": { "min_size": "100gb" } }], "transitions": [{ "state_name": "warm", "conditions": { "min_index_age": "12h" } }] }, { "name": "warm", "actions": [{ "warm_migration": {}, "retry": { "count": 5, "delay": "30m", "backoff": "exponential" } }], "transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "30d" } }] }, { "name": "delete", "actions": [{ "delete": {} }] }], "ism_template": { "index_patterns": ["*ds-*"], "priority": 100 } } }
Attach ISM Policy
POST _plugins/_ism/add/*ds-*
{
"policy_id": "inference_retention_policy"
}
Set Lifecycle Rule for S3 Bucket
Once OpenSearch is configured to only retain data for 30 days, it is also advised to configure the S3 bucket containing AI events to also delete thumbnails after 30 days (instructions).