Retrieval Augmented Generation (RAG)¶
This work is licensed under a Creative Commons Attribution 4.0 International License.
One of the more recent applications of GPTs is the use of vector databases with specific documentation text or imagery.
Retrieval Augmented Generation (RAG) can be used to search through large corpuses of text, or specific texts for data mining and research applications.
Real world examples of Agentic AI include self-driving vehicles, warehouse robots, and high frequency trading bots.
How it works¶
RAG databases allow you to host your own private data
Open Source projects like Weaviate or Pinecone allow you to self-host private data.
Embeddings¶
Text passages (or other data, like images) are transformed into numerical representations known as “embeddings.” These embeddings capture semantic meaning, so similar concepts end up being close together in vector space.
Understanding Embeddings
What are Embeddings? - Vicki Boykis - download PDF
Embeddings are a way to represent data (words, images, etc.) as numerical vectors in a multi-dimensional space. These vectors capture semantic relationships between data points, meaning similar items are located closer together in the embedding space.
Embedded space for geospatial applications:
Visualizing how embeddings can organize satellite imagery. Millions of points covering the state of Alabama move between their geographic position and their location in the embedding space. pic.twitter.com/Z6FtoMQ84B
— Caleb Kruse (@clkruse) May 15, 2023
Embedded space for natural language:
Why are Embeddings Important?
- Machine Learning: Embeddings are essential for training machine learning models, as they provide a way to represent complex data in a format that algorithms can understand.
- Semantic Search: Embeddings enable semantic search, where you can find information based on meaning rather than just keyword matching.
- Recommendation Systems: Embeddings help power recommendation systems by identifying items with similar characteristics.
- Data Visualization: Embeddings can be used to visualize relationships between data points in a lower-dimensional space.
Vector Database¶
A vector database stores embeddings efficiently, often using specialized data structures to handle large-scale, high-dimensional searches.
OpenAI Platform allows you to upload your own data to a vector store and file search in custom assistants