AWS TAMS Tools

AWS has released a number of tools for working in a TAMS environment to help customers start their journey of adoption. These are available as a separate repository on Github.

Figure 17: Overview of the different components in the AWS TAMS Tools repo

This repository includes the following capabilities:

• The TAMS Tools UI to enable users to view a TAMS store through a web browser rather than directly through the API. The Tools UI also integrates with other components within the Tools repository to provide additional capabilities, such as viewing content directly and monitoring the MediaLive and MediaConvert ingest processes.

• An HLS end point, which is a light-weight layer that converts the TAMS API into a basic HLS implementation. This allows content with the TAMS store to be played back through an HLS player.

• The ability to use an AWS Elemental MediaLive channel as an ingest tool into a TAMS store. While MediaLive does not natively support TAMS integration, this ingest harness uses the HLS output and then ingests this into a TAMS store to allow a simple way of testing live ingest into a store.

• A workflow to use AWS Elemental MediaConvert and process an existing file to create chunks suitable to loading into a TAMS store. This can be a useful tool to populate a store with test content.

Instructions for deploying the tools can be found in the TAMS Tools repository.

TAMS Tools UI

The TAMS Tools UI is provided to allow a simple way to view the TAMS API and carry out a few basic tasks such as creating tags or deleting content. It also provides interactions with some of the other tools, such as the MediaLive and MediaConvert ingest processes.

The UI is designed to be an engineering style tool and is appropriate for use in demonstrations and proof of concepts rather than for day-to-day operations. It is expected that in a typical deployment, there would be some form of MAM or PAM alongside the TAMS store. The user would search, view, and interact with content through the MAM or PAM UI rather than the TAMS Tools API.

The UI allows listing and searching at both the flow and source levels. It supports nesting the hierarchy of both flows and sources to simply the initial view. Any sources or flows are listed as children of the source or flow that collects them.

Figure 18: Screenshot of the TAMS Tools UI showing sources

Clicking through to the required source or flow will provide all the details that the API knows. From here, a user can click through the provided tabs for additional metadata or to view the linked sources and flows. Users can also add, remove, and edit tags from the UI.

Figure 19: Screenshot of the TAMS Tools UI showing details of a source

The View Diagram link provides a visual representation of all the TAMS sources and flows for a given piece of content. This can be useful for identifying issues with the data structures. Double clicking on an element will take the user directly to the screen for that element, which is a quick way of navigating between the different elements.

Figure 20: Screenshot of the TAMS Tools UI showing the hierarchy view

The View HLS link will open a player screen to allow the user to view the content in TAMS. This uses the HLS endpoint, which needs to be deployed to enable this option. Additionally, it will only work with content that is compatible with the HLS specifications and is subject to the limitations of the end point.

If the user clicks on the View HLS link from a source, this will open the multi-variant manifest and will provide each suitable flow as an HLS rendition. This will allow the user to view the combined audio and video elements as well as selecting which rendition to view.

Clicking the View HLS link from a flow will open the manifest for that flow directly. For mono-essence workflows, this means it will only contain either video or audio and not both.

Figure 21: Screenshot of the TAMS Tools UI showing the HLS player

The TAMS Tools UI provides a simple method of deleting content from the store. This can be done in bulk from the flows screen. The user has the option to delete the whole of the flow or to provide a timerange for the portion that they wish to delete.

Figure 22: Screenshot of the TAMS Tools UI showing the ability to delete flows

HLS Endpoint

TAMS works with small chunks of media, so as long as the choice of media format is compatible, it is relatively simple to convert from the TAMS API to HLS for streaming content to devices. The HLS endpoint in the tools repository is an example of how this can be implemented.

The HLS endpoint is built using API Gateway and Lambda. It reads the source, flow, and segments metadata from the TAMS API and then creates the relevant HLS manifests. The media remains in the original location in the TAMS-managed object storage with the pre-signed URLs supplied by the TAMS API passed through to the HLS manifest.

Figure 23: Architecture diagram of the HLS Endpoint

There are two endpoints on the HLS endpoint, both of which need to be provided with a suitable ID to generate the relevant manifest:

• /sources/{sourceId}/output.m3u8
  The sources endpoint will generate a multi-variant manifest file based on the available video and audio flows. This then allows the player or user to select which elements or TAMS flows they are interested in.
• /flows/{flowId}/output.m3u8
  If the flow specified is a “multi” flow, then this endpoint behaves in a similar manner to the Sources one described above. If a “mono” flow is specified, then the endpoint provides the list of segments that are available for the content. This is a direct conversion of the TAMS segments endpoint to an HLS manifest. If the flow is tagged as ingesting, then the HLS manifest will be left open, indicating to the player that is is live content and that it should be regularly refreshed for new segments.

The following flow level tags are supported by the HLS endpoint:

There are a few known limitations of the HLS endpoint which come from restrictions in the HLS specification rather than limitations in the TAMS API:

• Long running flows: The TAMS API can support almost unlimited duration flows. However, HLS does not support any form of pagination within the manifest. This means that at some point, the manifest will become too long and the end point will time out. This will work for typical VOD length assets (minutes to hours) and where a suitable segment length is used. For long running flows, either a TAMS native player is required or the tag hls_segments can be used to only request the most recent segments for the flow.

• Sub segment playback: While the TAMS API supports the playback of just part of a media segment, the HLS specification does not support this natively. To achieve sub segment playback, it would be necessary to render the required sub segments and register them as new segments into the store.

MediaLive Ingest

The AWS Elemental MediaLive ingest functionality in the TAMS Tools repository is designed to provide a simple way of getting a live stream into TAMS for the purposes of testing workflows. While MediaLive does not support TAMS natively, the Tools repo provides a surrounding set of tools to take an HLS output from MediaLive and ingest it into a TAMS store.

The process is split into three main stages:

• An AWS Step Functions workflow that sets up the TAMS sources and flows, configures Eventbridge rules for the correct bucket, and stores the configuration in Parameter Store, a capability of AWS Systems Manager, to be used by the ingest process.
• An ingest Lambda function that is triggered by an object arriving in an S3 bucket. This is then compared with the configuration and ingested into TAMS.
• A second Step Functions workflow to remove the remaining elements, including the EventBridge rules and the Parameter Store.

The repository does not set up the actual Elemental MediaLive channel. This needs to be done directly either through the AWS Console or through the API. The MediaLive channel should have one output group configured to generate an HLS output. This can contain multiple video renditions as well as separate audio channels.

The first major process is the Step Functions workflow that sets up the whole process. This needs to be provided the label for the new sources and flows to be created as well as the MediaLive channel details. It then reads the MediaLive configuration and, from this, creates the flows in the TAMS API using the technical characteristics from the configuration. Step Functions also writes a set of metadata in the Parameter Store, which provides the link between the S3 bucket location that MediaLive will write to and the appropriate TAMS flows. It also creates an EventBridge rule on the S3 bucket to forward new object events to the ingest Lambda.

Figure 24: Key stages in the MediaLive setup Step Functions workflow

The actual ingest process is triggered by the HLS segments being written out from the MediaLive system to the S3 bucket. The notification is sent to Eventbridge and triggers an ingest Lambda function. Lambda will read the configuration stored in Parameter Store, which will then enable it to parse the object name and location and match that to the TAMS flows. It then runs the usual TAMS ingest process to request a storage location, uploads the segment to the TAMS-managed S3 bucket, and registers it with the TAMS API.

Figure 25: Architecture of the ingest process

The process of starting and stopping the ingest process, as well as starting and stopping the MediaLive channel, can all be carried out within the TAMS Tools UI. When setting up the UI, it is necessary to provide the ARNs of the Step Functions workflows. From this, the UI will then enable the MediaLive screen.

Figure 26: TAMS Tools UI showing the ingest setup stage

For the MediaLive workflow to be successful, it is important to ensure the following:

• Each MediaLive channel must write the HLS out to a separate Amazon S3 folder.
• The Amazon S3 output locations for MediaLive must not include any underscore characters.
• The EventBridge rules for ingest must be in place to ensure notifications are received for all new segments.

If any of these are not in place, then the TAMS Tools UI will not allow an ingest process to be manually started.

Figure 27: TAMS Tools UI showing an ingest configuration error

MediaConvert Import

The TAMS Tools includes the ability to import existing files into TAMS using AWS Elemental MediaConvert. This tool is useful for quickly and easily populating a store with example content.

For existing flat files, it is possible to import these into TAMS either by importing as a single segment or by processing into small chunks and then uploading these. Alternatively, if there is a MAM already in place, then it may be easier to store the content in there alongside the references to the live content in TAMS. The decision on how to handle this will need to be made in context of the workflow and understanding the benefits and impacts of each approach.

The import process is based on a single Step Functions workflow, which is triggered by the originating content arriving in an S3 bucket watch folder. In the folder, there should be a corresponding mediaconvert.json file which provides the configuration for the MediaConvert transcode job. Multiple folders can be created with different job specifications if required.

Since MediaConvert allows many of the media parameter fields to be set to same as source (for example, frame rate), the first stage of the workflow is to run a MediaInfo process using Lambda on the source file. The workflow then reads the MediaConvert job definition file to work out what flows need to be created in TAMS. Where the MediaConvert job definition specifies using the same settings as the source, these are taken from the MediaInfo of the original file.

Once the flows have been created, the ingest process uses the same core ingest process as for the MediaLive. This includes setting up configuration in Parameter Store and creating appropriate EventBridge rules. The same ingest Lambda function is then called for each segment, and it is registered into TAMS.

Once the MediaConvert transcode job is complete, the Step Functions workflow cleans up the remaining elements, including removing the EventBridge rules, Parameter Store, and original source file.

Figure 28: Key stages in the MediaConvert Step Functions workflow

The TAMS Tools UI provides a basic set of monitoring of jobs running through the MediaConvert ingest process. This is activated by adding the appropriate Step Functions ARN into the environment configuration file. There is also a link from the UI to the MediaConvert job within the AWS Console.

Figure 29: TAMS Tools UI showing MediaConvert import jobs