State of LLMFunctions

Timothée Verdier, Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Outline

Interacting with a large language model (LLM) in Wolfram Language

LLMSynthesize, LLMSynthesizeSubmit

LLMFunction

ChatObject, ChatEvaluate, ChatSubmit

Defining the working environment

LLMEvaluator, LLMConfiguration

Prompts

Tools

Connecting to LLMs

Wolfram LLM

Local LLM evaluation

Service Connections

Interacting with a large language model (LLM) in Wolfram Language

LLMSynthesize, LLMSynthesizeSubmit

LLMFunction

ChatObject, ChatEvaluate, ChatSubmit

LLMSynthesize, LLMSynthesizeSubmit

Large Language Models (LLMs) are trained to generate text conditioned by a context, or by a set of instructions: the prompt

LLMSynthesize["Give me the names of three famous physics Nobel prize"]

LLMSynthesize, LLMSynthesizeSubmit

LLMs are very good at figuring out natural language instructions.

LLMSynthesize["Complete the sentence: The last Nobel prize in physics is awarded to <<completion>>"]

An LLMs generated completion is a randomly drawn from the “distribution of possible completions”.

the more open the instruction, the more possible outcome

LLMSynthesize["Complete the sentence: The last Nobel prize in physics is awarded to <<completion>>"]

well-crafted prompts can be used to control the completion behavior:

Table[
LLMSynthesize[{
"Complete the sentence by just adding the missing names.","Do not repeat the sentence.",
"Answer to the best of our current knowledge without justyfying it.",
"Sentence:
The last Nobel prize in physics is awarded to",
"Completion:"
}]
,
5
]

LLMSynthesize, LLMSynthesizeSubmit

LLM Queries can be submitted asynchronously [14.2]

task = LLMSynthesizeSubmit["Complete the sentence: The last Nobel prize in physics is awarded to <<completion>>"]
2+2
task["TaskStatus"]

Results are accessible to HandlerFunctions

task = LLMSynthesizeSubmit["Complete the sentence: The last Nobel prize in physics is awarded to <<completion>>"
,
HandlerFunctions -> <|"CompletionGenerated" -> Function[result = #CompletionText]
|>
]
task["TaskStatus"]

result

LLMSynthesize, LLMSynthesizeSubmit

LLM Queries can be submitted asynchronously [14.2]

liveresult1={};
Dynamic[StringJoin[liveresult1]]

task=LLMSynthesizeSubmit["Complete the sentence: The last Nobel prize in physics is awarded to <<completion>>",
HandlerFunctions -> <|
"ContentChunkReceived" -> Function@AppendTo[liveresult1,#ContentChunk]
|>
]

LLMSynthesize, LLMSynthesizeSubmit

Run independent LLM queries in parallel [14.2]

resultList = Table[Missing[],5];
Dynamic[resultList]

tasks=Table[
With[{pos = iter},
LLMSynthesizeSubmit[{
"Complete the sentence by just adding the missing names.",
"Do not repeat the sentence.",
"Answer to the best of our current knowledge without justyfying it.",
"Sentence:
The last Nobel prize in physics is awarded to",
"Completion:"
},
HandlerFunctions -> <|
"CompletionGenerated" -> Function[resultList[[pos]] = #CompletionText]
|>
]]
,
{iter,1, 5}
]

LLMSynthesize, LLMSynthesizeSubmit

If the LLM supports it, from Version 14.1, image expressions can be used as prompts:

LLMSynthesize[]

Text and images can be mixed together an combined into a single prompt

LLMSynthesize[{"how many animals are in this image?",}]

LLMFunction

Use the prompt in LLMFunction to convert an immediate request into a reusable function:

fun=LLMFunction["Complete the sentence by just adding the missing names.
Do not repeat the sentence.
Use semicolumn as list delimiter.
Answer to the best of our current knowledge without justyfying it.

Sentence:
The last Nobel prize in physics is awarded to

Completion:"]

Apply the function to evaluate the call:

fun[]

LLMFunction

Function parameters: the default prompt interpretation is StringTemplate

fun2=LLMFunction["Complete the sentence by just adding the missing names.
Do not repeat the sentence.
Use semicolumn as list delimiter.
Answer to the best of our current knowledge without justifying it.

Sentence:
The last Nobel prize in `` is awarded to

Completion:"]

fun2["peace"]

LLMFunction

On the fly interpretation of the returned results

fun3=LLMFunction["Complete the sentence by just adding the missing names.
Do not repeat the sentence.
Use semicolumn as list delimiter.
Answer to the best of our current knowledge without justifying it.

Sentence:
The last Nobel prize in `` is awarded to

Completion:",
DelimitedSequence["Person"]
]

fun3["biology"]

LLMFunction

LLM model permitting, LLMFunction can take images passed as  TemplateObject:

imageCaption=LLMFunction[TemplateObject[{
"Generate a list of object present in the following image:",
TemplateSlot[1],
"The output must contain only a list of object separated by newlines without any punctuation"
}],
DelimitedSequence["String"]
]

imageCaption[]

ChatEvaluate, ChatSubmit

Initiate a new chat conversation with a prompt of your choice for the LLM

chat=ChatObject["Talk like a member of the Nobel prize comitee."]

Now both text and images can be used in the conversation [14.1]

chat=ChatEvaluate[chat,{"what is this picture?",(Entity["domestic cat", Entity["TaxonomicSpecies","FelisCatus::ddvt3"], "Entity[\"TaxonomicSpecies\", \"FelisCatus::ddvt3\"]", "species specification"])[(EntityProperty["image", EntityProperty["TaxonomicSpecies","Image"], "EntityProperty[\"TaxonomicSpecies\", \"Image\"]"])]}]

Continue the conversation

chat=ChatEvaluate[chat,{"Would you support this candidate? Or rather this second one? Which category of the prize would you see fit?",}]

ChatEvaluate, ChatSubmit

Chat evaluations can be submitted asynchronously [14.2]

liveresult=.
Dynamic[liveresult["MessageGrid"]]

tasks=ChatSubmit[
ChatObject["Talk like a member of the Noble price comitee."],
"The Nobel Prize in Physics 2024 was awarded to John J. Hopfield and Geoffrey E. Hinton, please write the short corresponding anouncement summary",
HandlerFunctions -> Function[liveresult=#ChatObject]
]

tasks["TaskStatus"]

Define the Working Environment

LLMEvaluator

Prompts

Tools

Retrieval-Augmented Generation, Vector Databases, Semantic Search

LLMEvaluator, LLMConfiguration

LLM settings can be passed as an LLMConfiguration object:

config = LLMConfiguration[<|
"Model"-> <|"Service"-> "Anthropic", "Name"-> "claude-3-5-sonnet-20240620"|>,
"Temperature"->.7,
"MaxTokens"->32
|>]

The interaction with LLMs can be controlled via the LLMEvaluator option:

LLMSynthesize["A new Nobel price category has been eventually added for Mathematics", LLMEvaluator->LLMConfiguration[]]

Settings and parameters are contained in an LLMConfiguration object. A configuration can be set as a global default in $LLMEvaluator.

$LLMEvaluator

Prompts

One very important part of the configuration is the “Prompt” setting (If a service supports it, they will be assigned the System role or a similar one.)

This sets major instructions for the LLMs besides the input.

Use the "ELI5" prompt to steer the LLM toward simple answers:

LLMSynthesize["In short, What can I do today?"]

Use the "ELI5" prompt to steer the LLM toward a simple kid tailored answers:

LLMSynthesize["What can I do today?",LLMEvaluator->"Prompts"->LLMPrompt["ELI5"]]

Multiple prompts can be combined:

LLMSynthesize["What can I do today?", LLMEvaluator->"Prompts"->{"Answer all the questions in Spanish",LLMPrompt["ELI5"]}]

Prompts

It’s not always possible to know beforehand what the best settings for a conversation are. LLMPromptGenerator can be used to define prompts on the fly by applying a function on the LLM input.

Prompts can be enriched with contextual information:

gen=LLMPromptGenerator[{
"User: ",$CloudAccountName,
"Date: ",Now,
"Place:",$GeoLocationCity
}&]

ChatEvaluate[
ChatObject[LLMEvaluator-><|"Prompts"->{"Always greet the person you are talking to by their name",gen}|>],
"What can I do today?"
]

But above all, LLMPromptGenerator was designed with Retrieval-Augmented Generation (RAG) in mind - to see it shine, go to the dedicated talk:

Semantic Search and RAG (Friday, 9:00 AM - 4:30 AM CT)

Tools

Calling function (aka tools), LLM can query extra pieces of (controlled) information it needs

user=LLMPromptGenerator[{
"User: ",$CloudAccountName,
"Date: ",Now,
"Place:",$GeoLocationCity}&
]
thermometer=LLMTool[
{"thermometer","gives the day temperature"},
{"date"->("DateTime"|"Date"),"where"->"Location"},Mean@WeatherForecastData[#where, "Temperature", DateObject[#date, "Day"]]&]
raingauge = LLMTool[
{"raingauge","gives the daily forcasted precipitation"},{"date"->("DateTime"|"Date"),"where"->"Location"}, Mean@WeatherForecastData[#where, "TotalPrecipitation", DateObject[#date, "Day"]]&
]

liveresult3=.
Dynamic[liveresult3["MessageGrid"]]

liveresult3["Usage"]

ChatSubmit[
ChatObject[LLMEvaluator-><|
"Prompts"->{"Always greet the person you are talking to",user}, "Tools"-> {thermometer,raingauge},
"ToolMethod"-> "Service"
|>],
"What can I do today?",
HandlerFunctions-> Function[liveresult3 = #ChatObject]
]

Model permitting, LLMConfiguration can target the API Tool calling mechanisms specifying 
“ToolMethod” -> “Service”

Retrieval-Augmented Generation (RAG), Vector Databases, Semantic Search...

Semantic Search and RAG

Giulio Alessandrini

Friday from 9:00 AM - to 9:30 AM CT

Connecting to LLMs

Wolfram LLM Kit

Service Connections

Local LLMs

Wolfram LLM Kit

Enable all the LLM functionality in the language without any external account.

Service Connections

Underlying service connect framework - direct API calls

ServiceExecute["OpenAI",
"Chat",
{"Messages"->{<|"Role"->"User","Content"->"Hello there!"|>}}
]

New and updated LLM services [14.1]

• AlephAlpha
• Anthropic (updated)
• GoogleGemini (replaces Google PaLM)
• MistralAI
• Cohere
• DeepSeek
• TogetherAI
• Groq
• OpenAI (updated)

Local LLMs [14.2]

LLMSynthesize[
"make a short poem about sharks",
LLMEvaluator->LLMConfiguration[
<|"Model"-><|
"Service"->"LocalEvaluator",
"Name"->"/Users/timotheev/Downloads/tinyllama-2-1b-miniguanaco.Q3_K_M.gguf",
"MessageTemplate"->StringTemplate@"### Human: {`Content`}\n### Assistant:"
|>,
"MaxTokens"->50,
"Temperature"->0.5
|>]
]

State of LLMFunctions

Thank you!