Scalable Vector Databases for Enterprise-Scale AI Retrieval
DOI:
https://doi.org/10.15662/IJRAI.2024.0704001Keywords:
Vector Databases, Enterprise AI, Semantic Search, Retrieval-Augmented Generation (RAG), Scalability, Data Governance, Embedding Models, Approximate Nearest Neighbor Search (ANNS)Abstract
Scalable vector databases have emerged as a cornerstone for enterprise-scale AI retrieval, enabling efficient storage, management, and querying of high-dimensional vector embeddings derived from unstructured data. These databases facilitate semantic search, recommendation systems, and retrieval-augmented generation (RAG) by transforming complex data into numerical representations. This paper examines the evolution, architecture, and performance of scalable vector databases, highlighting their significance in enterprise AI applications. We explore the challenges associated with scalability, consistency, and data governance, and propose solutions to address these issues. Through comparative analysis of leading vector database systems, we provide insights into their capabilities and limitations. The findings underscore the critical role of scalable vector databases in unlocking the potential of AI-driven enterprise solutions.
References
1. Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3), 535-547. FAISS
2. Guo, Y., et al. (2020). Milvus: A cloud-native vector database. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. https://dl.acm.org/doi/10.1145/3318464.3389722
3. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824-836.
4. Kiela, D., et al. (2021). Supervised multimodal bitransformers for classifying images and text. Proceedings of the 37th International Conference on Machine Learning. Multimodal embeddings
5. Zhou, Y., et al. (2021). Neural index for billion-scale vector search. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. Neural Index
6. Charikar, M. (2002). Similarity estimation techniques from rounding algorithms. Proceedings of the 34th Annual ACM Symposium on Theory of Computing, 380-388. [Locality-Sensitive Hashing]
7. Chen, J., et al. (2018). Learning to hash for scalable vector search. ACM Computing Surveys, 50(3), 1-36.
8. Li, X., et al. (2020). Vector search engines: A tutorial and survey. ACM Computing Surveys. (Preprint)
9. Tschannen, M., Bachem, O., & Lucic, M. (2019). Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069.
10. Gionis, A., Indyk, P., & Motwani, R. (1999). Similarity search in high dimensions via hashing Proceedings of the 25th International Conference on Very Large Data Bases (VLDB), 518-529.
