Semantic Search and RAG

Giulio Alessandrini, Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

This session will highlight new tools for creating and managing vector 
databases and semantic indices. We'll also discuss their integration with 
large language models to generate contextually relevant responses, showcasing 
improved data handling and search capabilities.

Outline

What is Semantic Search?

The Limitations of Traditional Search

Embeddings: The Core of Semantic Search

What is a Vector Database?

Work with Vector Databases

Semantic Search

Context Augmentation

What is Retrieval-Augmented Generation (RAG)?

What is Semantic Search?

Semantic search retrieves relevant results based on meaning and context, 
rather than simple keyword matching.

The Limitations of Traditional Search

Traditional search relies on exact word matching. Misses context, synonyms, 
and meaning.

searchQuery="writing code";
InputField[Dynamic[searchQuery],String,BaseStyle->{"ControlStyle"},ContinuousAction->True]

Traditional Search

Dynamic[Column[Pane[#,{500,30},Alignment->{Left,Center}]&/@If[StringQ[searchQuery]&&StringLength[searchQuery]>0,
MapThread[HighlightString,{documents, StringPosition[documents,StringSplit[searchQuery],IgnoreCase->True]}],
documents
],{Left,Center},BaseStyle->"ControlStyle"]
]

The Limitations of Traditional Search

Traditional search relies on exact word matching. Misses context, synonyms, 
and meaning.

searchQuery="writing code";
InputField[Dynamic[searchQuery],String,BaseStyle->{"ControlStyle"},ContinuousAction->True]

Traditional Search

documents={
  "Mount Everest is the highest mountain in the world.",
  "The Wolfram Language is a great programming language.",
  "The Eiffel Tower is located in Paris."
};
Dynamic[Column[Pane[#,{500,30},Alignment->{Left,Center}]&/@If[StringQ[searchQuery]&&StringLength[searchQuery]>0,
MapThread[HighlightString,{documents, StringPosition[documents,StringSplit[searchQuery],IgnoreCase->True]}],
documents
],{Left,Center},BaseStyle->"ControlStyle"]
]

Semantic Search

documents={
  "Mount Everest is the highest mountain in the world.",
  "The Wolfram Language is a great programming language.",
  "The Eiffel Tower is located in Paris."
};
index=CreateSemanticSearchIndex[documents];
Dynamic[If[StringQ[searchQuery]&&StringLength[searchQuery]>0,
Dataset@SemanticSearch[index,searchQuery,{"Item","Distance"}],
Transpose@Dataset[<|"Item"->documents,"Distance"->ConstantArray[Missing[],Length[documents]]|>]
]]

Embeddings: The Core of Semantic Search

Embeddings are numerical representations of words, sentences, or documents in 
high-dimensional space.

Grid[Thread[
{Join[documents,{"what's in the capital of France?","tall","write code"}],
Short[Normal[#],4]&/@embeddings}],Alignment->Left,Spacings->{1, 1},BaseStyle->{"ControlStyle",10}]

Similar concepts are mapped closer together.

ListPlot[DimensionReduce[embeddings,2]->Join[documents,{"what's in the capital of France?","tall","write code"}],Axes->False,BaseStyle->12,LabelingFunction->Callout,ImageSize->Large]

What is a Vector Database?

A specialized database for storing and retrieving high-dimensional vectors 
efficiently.

Traditional Database

Documents are stored in a tabular like structure together with relevant 
metadata.

ndocs=Length[documents];
Row[{Grid[Partition[documents,1],BaseStyle->"ControlStyle",Spacings->{1, 1}],Dataset[Transpose[Association["ID"->RandomInteger[10^6,ndocs],"documents"->Pane/@documents,"date"->RandomDate[ndocs],"author"->ResourceFunction["RandomPetName"][ndocs]],AllowedHeads->All]]},Style[" -> ",20,Gray]]

Search is performed matching the query literally against the index

Bag of tricks to improve results (e.g. tokenization, stop word removal, 
stemming & lemmatization, synonym matching, fuzzy matching, query expansion, 
spell correction)

Boolean operators are used to refine the query (e.g., "apple AND computer," 
"apple OR orange," "apple NOT fruit")

query1="what is the highest moutain in the world?";
query2="what is the highest mountain in the world?";
Row[{
Style[query1,"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Spell Correction","Label"]],
Style[query2,"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Tokenization","Label"]],
Style[TextWords[query2],"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Stopword Removal","Label"]],
Style[TextWords[DeleteStopwords@query2],"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Stemming and Lemmatization","Label"]],
Grid[Partition[{{"high","higher","highest"},"mountain","world"},1],BaseStyle->"ControlStyle",Spacings->{1, 1}],
Labeled[Style["->",20,Gray],Style["Query Expansion","Label"]],
Grid[Partition[{{"high","higher","highest"},{"mountain","peak"},{"world","Earth"}},1],BaseStyle->"ControlStyle",Spacings->{1, 1}]
},Spacer[5]]

What is a Vector Database?

A specialized database for storing and retrieving high-dimensional vectors 
efficiently.

Vector Database

Documents are stored as vectors in a high dimensional space and encoded in 
graph representation for efficient search

ndocs=Length[documents];
Row[{Grid[Partition[documents,1],BaseStyle->"ControlStyle",Spacings->{1, 1}],Grid[RandomReal[{-1,1},{Length[documents],1,5}],BaseStyle->"ControlStyle",Spacings->{1, 1}],
BlockRandom[RandomGraph[SpatialGraphDistribution[100,.2]],RandomSeeding->12]
},Style[" -> ",20,Gray]]

Search is performed by encoding the query to the same space

Matches are found via approximate nearest neighbours

query1="what is the highest moutain in the world?";
Row[{
Style[query1,"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Encoding","Label"]],
Style[RandomReal[{-1,1},5],"ControlStyle"],
Labeled[Style["->",20,Gray],Style["Vector Search","Label"]],
Style[BlockRandom[RandomGraph[SpatialGraphDistribution[100,.2]],RandomSeeding->12],"ControlStyle"]
},Spacer[5]]

Key Capabilities of Vector Databases

Efficient search in high-dimensional space

Scalable and optimized for large datasets

Support real time update with minimal performance cost

Work with Vector Databases

Create a new database

DeleteObject[VectorDatabaseObjects["myDB*"]]

db=CreateVectorDatabase["myDB"]

Add elements to the database

AddToVectorDatabase[db,{{1,2,3},{-1,0,4},{2,3,1}}]

Search the database

VectorDatabaseSearch[db,{1,1,1}]

VectorDatabaseSearch[db,{1,1,1},2]

Copy the database to a new location

db2=VectorDatabaseObject[db,GeneratedAssetLocation->File["mynewfile"]]

db["Location"]
db2["Location"]

Provide a custom feature extractor to handle non-vector elements:

imagedb=CreateVectorDatabase[{,,,,,},FeatureExtractor->"ImageFeatures"]

Search directly with images:

VectorDatabaseSearch[imagedb,]

Semantic Search

Automatically build vector databases from a list of documents

Create a new index:

alice=CreateSemanticSearchIndex@ResourceData["Alice in Wonderland"]

Search the index:

SemanticSearch[alice,"a sport game",MaxItems->3]//Map[Framed[Style[#,8]]&]

Add new documents to the index

UpdateSemanticSearchIndex[alice,WikipediaData["Alice In Wonderland"]]

Semantic Search

This is the code for the initial slide

searchQuery="writing code";
InputField[Dynamic[searchQuery],String,BaseStyle->{"ControlStyle"},ContinuousAction->True]

documents={
  "Mount Everest is the highest mountain in the world.",
  "The Wolfram Language is a great programming language.",
  "The Eiffel Tower is located in Paris."
};
index=CreateSemanticSearchIndex[documents];
Dynamic[If[StringQ[searchQuery]&&StringLength[searchQuery]>0,
Dataset@SemanticSearch[index,searchQuery,{"Item","Distance"}],
Transpose@Dataset[<|"Item"->documents,"Distance"->ConstantArray[Missing[],Length[documents]]|>]
]]

Context Augmentation

Augment the prompt with additional context or data that improves the quality 
of the response

Add contextual information to a prompt:

gen=LLMPromptGenerator[{"User: ",$CloudAccountName,"Date: ",Now,"Place:",$GeoLocationCity}&]

LLMSynthesize["Hi there! how can you help me?",LLMEvaluator-><|"Prompts"->{"Always greet the person you are talking to","add contextual information in a seamless way",gen}|>]

Run arbitrary code on the prompt:

lengthcheck=LLMPromptGenerator[StringRiffle@{"Check for the input to be longer then 100 characters and only 
answer in that case. The input length is ",ToString@StringLength[#]}&]

LLMSynthesize["The sky is blue",LLMEvaluator-><|"Prompts"->lengthcheck|>]

Define special syntaxes:

gen=LLMPromptGenerator["Talk like a "<>First[StringCases[#, StartOfString~~"@"~~role:(LetterCharacter..)~~WordBoundary:>role], "pirate"]&]

LLMSynthesize["@cowboy hi there! what ya doin?",LLMEvaluator-><|"Prompts"->gen|>]

What is Retrieval-Augmented Generation (RAG)?

Combines traditional information retrieval with language models to answer 
queries by generating context-aware responses using retrieved documents.

LLMPromptGenerator was built to synergize well with SemanticSearchIndex:

docsindex=ResourceData["Semantic Index of Wolfram Language Documentation"]

Combine the "CodeWriter" prompt with dynamic suggestions from the Wolfram 
Language documentation:

LLMSynthesize["Create a new document with a plot of the sin function",
 LLMEvaluator->
<|"Prompts"->{LLMPrompt["CodeWriter"],LLMPromptGenerator[docsindex]}
|>]

DocumentNotebook[{
  TextCell["Plot of the Sin Function", "Title"],
  ExpressionCell[Plot[Sin[x], {x, 0, 10}], "Output"]
}]

Resources

Guides

LLM-Related Functionality

Machine Learning

Repositories

Wolfram Prompt Repository

Wolfram LLM Tool Repository

Wolfram Data Repository

THANK YOU

INIT

DeleteObject@SemanticSearchIndices[]
DeleteObject@VectorDatabaseObjects[]

documents={
  "Mount Everest is the highest mountain in the world.",
  "The Wolfram Language is a great programming language.",
  "The Eiffel Tower is located in Paris."
};
index = CreateSemanticSearchIndex[documents];
(*embeddings = LLMServices`EmbedText[Join[documents, {"what's in the capital of France?", "tall", "write code"}], $LLMEvaluator]["Content"];*)
embeddings = BinaryDeserialize[ByteArray[]];