AWS AI/ML Model Customization
OfficialFreeStreamline AI model deployment and fine-tuning on SageMaker.
Free · Opens the source repo
What AWS AI/ML Model Customization does
The AWS AI/ML Model Customization skill provides a comprehensive framework for developers and data scientists looking to fine-tune and deploy AI models using Amazon SageMaker. This skill encompasses the entire lifecycle of model customization, from initial planning through to production deployment. It is particularly useful for those who need to select base models from SageMaker Hub, prepare datasets for training, and evaluate model performance before deployment.
Users can leverage this skill to navigate the complexities of fine-tuning techniques such as Supervised Fine-Tuning (SFT), Deep Prompt Optimization (DPO), Reinforcement Learning with Human Feedback (RLVR), and Reinforcement Learning from AI Feedback (RLAIF). The skill also includes tools for validating dataset quality and transforming datasets into the required formats for training. By utilizing the provided scripts and templates, users can efficiently manage their projects and streamline their workflows.
The skill is designed for those who are already familiar with Amazon SageMaker and are looking for a structured approach to model customization. It helps users set up necessary resources such as IAM roles and S3 buckets, and manage MLflow applications within the SageMaker environment. Additionally, it offers diagnostic tools to monitor and troubleshoot SageMaker endpoints, ensuring that users can maintain optimal performance and quickly address any issues that arise during deployment.
Overall, this skill is an essential resource for anyone involved in machine learning projects on AWS, providing the necessary tools and guidance to effectively customize and deploy AI models.
When to use it
Use this skill when you need to fine-tune models, prepare datasets, or deploy AI solutions on SageMaker.
When not to use it
This skill is not suitable for tasks related to Ground Truth labeling, Feature Store management, or general AWS infrastructure management.
What you can build with it
Fine-Tuning a Model
Use the skill to select a base model and apply fine-tuning techniques to tailor it for specific tasks.
Preparing Datasets
Transform and validate datasets to ensure they meet the requirements for training before initiating a fine-tuning job.
Deploying to SageMaker Endpoints
Deploy your trained models to SageMaker endpoints or Bedrock for real-time inference, with tools for monitoring and diagnostics.
How to install AWS AI/ML Model Customization
View source1. Install with the skills CLI
npx skills add aws/agent-toolkit-for-aws/aws-ai-ml --agent claude-code2. Or install it manually
Download the skill folder and drop it into ~/.claude/skills/ for all projects, or .claude/skills/ to scope it to one repo. Restart Claude Code so it picks up the new skill.
Anthropic's agentic coding CLI, and the reference implementation of Agent Skills. Drop a skill folder into ~/.claude/skills and Claude Code loads it automatically whenever a task matches the skill's description. Claude Code docs
Inside SKILL.md
Written by awsAWS AI/ML Model Customization
Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.
Routing
Match the user's intent to the appropriate reference folder and load only that content.
| User intent | Reference | When to use |
|---|---|---|
| Plan a model customization project, discover scope of work, resume or modify a plan | references/planning/ | User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference. |
| Define the business problem, success criteria, or use case spec | references/use-case-specification/ | User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines. |
| Select or change a base model | references/model-selection/ | User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation. |
| Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF) | references/finetuning-technique/ | User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first. |
| Validate dataset quality and format | references/dataset-evaluation/ | User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job. |
| Transform or convert a dataset between formats | references/dataset-transformation/ | User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code. |
| Generate fine-tuning code and start training | references/finetuning/ | User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers. |
| Evaluate or benchmark a trained model | references/model-evaluation/ | User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer. |
| Deploy a model to endpoint or Bedrock | references/model-deployment/ | User says "deploy my model", "create an endpoint", "make it available". Handles Nova vs OSS deployment pathways. |
| Set up IAM roles, S3 buckets, SDK configuration | references/sdk-getting-started/ | User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment. |
| Manage project directory and artifacts | references/directory-management/ | Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory. |
| Set up, update, or delete a SageMaker Managed MLflow app | references/manage-mlflow/ | User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected. |
| Diagnose a failing or unhealthy SageMaker endpoint | references/endpoint-diagnostics/ | User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments. |
Rules
- Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.
- Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.
Frequently asked questions about AWS AI/ML Model Customization
Similar skills
Spring Boot Testing
Master testing techniques for Spring Boot 4 applications.
GitHub Issues
Manage GitHub issues efficiently with MCP tools.
Geofeed Tuner
Optimize your IP geolocation feeds in CSV format.
Batch Files
Master Windows batch scripting for automation and task management.
Adobe Illustrator Scripting
Automate your Illustrator workflows with ExtendScript.
Plugin Structure
Create and organize Claude Code plugins effectively.