The solution revolved around the common use of Amazon Simple Storage Service(Amazon S3) and adjustments to the pipeline that train on unstructured data (such as video, audio and free-form text). The AWS refined solution demonstrated a clear pattern for a significant reduction in complexity, cost and centrally managing the second step (when it comes to Data Scientist/Engineer inputs).

“When ML algorithms process unstructured data like images and video, it requires various normalisation tasks (such as grey-scaling and resizing). This step exists to accelerate model convergence, avoid overfitting and improve prediction accuracy. You often perform these preprocessing steps on instances that later run the AI training. That approach creates inefficiencies, because those resources typically have more expensive processors (for example, GPUs) than these tasks require. Instead, our solution externalizes those operations across economic, horizontally scalable Amazon S3 Object Lambda functions.

This design pattern has three critical benefits. To begin, it centralises the shared data transformation steps, such as image normalisation and removing ML pipeline code duplication. Secondly, S3 Object Lambda functions avoid data consistency issues in derived data through JIT conversions. Finally, the serverless infrastructure reduces operational overhead, increases access time and limits costs to the per-millisecond time when running your code.

An elegant solution exists in which you can centralise these data preprocessing and data conversion operations with S3 Object Lambda. S3 Object Lambda enables you to add code that modifies data from Amazon S3 before returning it to an application. The code runs within an AWS Lambda function, a serverless compute service. Lambda can instantly scale to tens of thousands of parallel runs while supporting dozens of programming languages and even custom containers.” – AWS Solution Architecture team

In the solution shown in the infrastructure diagram above, the S3 bucket contains the raw images to be processed. You’ll then need to create an S3 Access Point for the images. By building multiple levels of Machine learning models, you should create separate S3 Access Points for each model.

Alternatively, AWS Identity and Access Management (IAM) policies for access points support sharing reusable functions across ML pipelines. Then you attach a Lambda function that has your preprocessing business logic to the S3 Access Point. After you retrieve the data, you call the S3 Access Point to perform JIT data transformations. Finally, you update your ML model to use the new S3 Object Lambda Access Point to retrieve data from Amazon S3.

The original article then walks through the motions of creating the S3 Object Lambda access point as well as reviewing the typical cost savings analysis that take into account the standard model training inefficiencies. We recommend checking it out and getting access to the scrollable Python language (helping you create a Lambda function that performs the image resizing and conversion).