Vector Index Tuning
FreeOptimize vector index performance for better search.
Free · Opens the source repo
What Vector Index Tuning does
Vector Index Tuning provides a comprehensive guide for optimizing vector indexes to enhance performance in production environments. This skill is particularly useful for developers and data engineers who need to fine-tune their vector search infrastructure, especially when working with large datasets. By focusing on parameters such as HNSW (Hierarchical Navigable Small World) settings and quantization strategies, users can achieve significant improvements in search latency and memory usage.
The skill outlines best practices for selecting the appropriate index type based on data size, ranging from flat indexes for smaller datasets to more complex structures like IVF (Inverted File) with Product Quantization for larger datasets. It also details the critical HNSW parameters, including M, efConstruction, and efSearch, which directly affect the quality of search and index construction. Understanding these parameters allows users to balance recall and speed effectively, which is essential for applications requiring rapid response times.
In addition to tuning parameters, the skill emphasizes the importance of continuous monitoring and benchmarking with real queries to ensure that the index performs optimally as data evolves. The included templates and worked examples in the references/details.md file provide practical guidance for implementing these concepts, making it easier for users to apply the recommendations directly to their projects. This skill is designed for those who are looking to scale their vector search capabilities while maintaining high performance and efficiency.
Overall, Vector Index Tuning is an essential resource for anyone involved in managing vector databases or search systems, offering actionable insights that can lead to improved application performance and user satisfaction.
When to use it
Use this skill when you need to tune HNSW parameters, implement quantization strategies, or scale your vector search infrastructure effectively.
When not to use it
This skill may not be suitable for smaller datasets where simple indexing methods suffice, or if you're not dealing with vector search applications.
What you can build with it
Tuning a Large Vector Database
When managing a vector database with millions of entries, use this skill to optimize HNSW parameters for improved search performance.
Implementing Quantization Strategies
In scenarios where memory usage is critical, apply the quantization techniques outlined in this skill to reduce the storage requirements of your vector data.
Scaling Search Infrastructure
As your application grows to handle billions of vectors, leverage the guidance in this skill to select the appropriate index types and configurations.
How to install Vector Index Tuning
View source1. Install with the skills CLI
npx skills add wshobson/agents/vector-index-tuning --agent claude-code2. Or install it manually
Download the skill folder and drop it into ~/.claude/skills/ for all projects, or .claude/skills/ to scope it to one repo. Restart Claude Code so it picks up the new skill.
Anthropic's agentic coding CLI, and the reference implementation of Agent Skills. Drop a skill folder into ~/.claude/skills and Claude Code loads it automatically whenever a task matches the skill's description. Claude Code docs
Inside SKILL.md
Written by wshobsonVector Index Tuning
Guide to optimizing vector indexes for production performance.
When to Use This Skill
- Tuning HNSW parameters
- Implementing quantization
- Optimizing memory usage
- Reducing search latency
- Balancing recall vs speed
- Scaling to billions of vectors
Core Concepts
1. Index Type Selection
Data Size Recommended Index
────────────────────────────────────────
< 10K vectors → Flat (exact search)
10K - 1M → HNSW
1M - 100M → HNSW + Quantization
> 100M → IVF + PQ or DiskANN
2. HNSW Parameters
| Parameter | Default | Effect |
|---|---|---|
| M | 16 | Connections per node, ↑ = better recall, more memory |
| efConstruction | 100 | Build quality, ↑ = better index, slower build |
| efSearch | 50 | Search quality, ↑ = better recall, slower search |
3. Quantization Types
Full Precision (FP32): 4 bytes × dimensions
Half Precision (FP16): 2 bytes × dimensions
INT8 Scalar: 1 byte × dimensions
Product Quantization: ~32-64 bytes total
Binary: dimensions/8 bytes
Templates and detailed worked examples
Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.
Best Practices
Do's
- Benchmark with real queries - Synthetic may not represent production
- Monitor recall continuously - Can degrade with data drift
- Start with defaults - Tune only when needed
- Use quantization - Significant memory savings
- Consider tiered storage - Hot/cold data separation
Don'ts
- Don't over-optimize early - Profile first
- Don't ignore build time - Index updates have cost
- Don't forget reindexing - Plan for maintenance
- Don't skip warming - Cold indexes are slow
Frequently asked questions about Vector Index Tuning
Similar skills
ClickHouse Logs Queries
Efficiently manage Supabase logs with ClickHouse SQL.
EF Core D2 Database Diagram Generator
Visualize your EF Core models as D2 diagrams effortlessly.
Safe SQL Execution
Ensure secure SQL execution in Supabase applications.
Oracle to PostgreSQL Migration
Identify migration risks between Oracle and PostgreSQL.
SSMA Console
Streamline Oracle to SQL Server migrations with ease.
SQL Performance Optimization
Enhance SQL query efficiency across all databases.