Advanced Automated Assessment
for Math & More

Christophe Pakleza

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

The Wolfram Language is a powerful tool for comparing symbolic expressions, especially mathematical expressions. In this talk, I will show how to generate quiz questions with automated assessments. These assessments will include built-in advanced comparisons for common educational math problems as well as user-customized comparison methods.  Also see how to deploy quiz questions to the cloud for integration with other systems

1. SETTING AND RUNNING 
Assessments for Math

2. DEPLOYMENT

Question Framework quick recap

Tools provided :

User interface

“QuestionNotebook authoring”: to write quizzes (with no coding)

QuestionGenerator: to generate random quizzes based on a template
QuestionSelector: to select quizzes from a pool of questions

QuestionInterface: to customize input for QuestionObject
QuestionObject: central interface to:
			- submit answers
			- retrieve the assessment result

Assessment

AssessmentFunction: perform and returns the assessments
AssessmentResultObject: contains the assessment result and details

AssessmentFunction

Documentation update in 14.1 web docs »

AssessmentFunction[...]

Assessment settings

AssessmentFunction[
			{
			key1->"Score1", 
			key2->"Score2", ...}             (* answer key *)
			,
			<| 
				       "ComparisonMethod"          ->       (*built in method*)
				      "specific method options" ->        (*built in option*)
			(*or*)      "Comparator"                           ->        (*user custom comparison method*)
			(*or*)      "Selector"                                   ->        (*user custom selection method*)
                                                                        "ListAssessment"                    ->        (*enable multiple part answers*)
				      "other settings"                      ->        {(*...*)}
			|>
		            
	                                 , Tolerance                                                         -> 100(*numeric value*)
	                                 , DistanceFunction                                         -> func(*distance metric function*)
		     ]

where “ListAssessment” enables different multi-part answer modes:
- “SeparatelyScoreElements” : elementwise assessment
- “AllElementsOrdered” or “AllElementsOrderless” : taking into account the order of the answers

Built-in comparison “math” methods

Used for Free-Form assessments

Simple Types

“Number”

“Vector”

Complex Expressions

“ArithmeticResult”

“PolynomialResult”

“CalculusResult”

“AlgebraicValue”

⚠️ Fixed-Choice assessment

(No specific comparison)

QuestionObject[
	QuestionInterface["ChooseMultiple",HoldForm[Sqrt[4]]]
	,AssessmentFunction[{-2 -> True, 0-> False, 2-> True, 1-> False}
						,<|"ListAssessment"->"SeparatelyScoreElements"|>]]

Other available Fixed Choice interfaces

QuestionInterface[...]

Free From assessment

QuestionObject[
	QuestionInterface["MultipleShortAnswers",HoldForm[Sqrt[4]]]
	,AssessmentFunction[{-2->True,2->True,0->False,1->False}
						,<|"ListAssessment"->"SeparatelyScoreElements"|>,Tolerance->0.01]]

“Number”

to compare numbers!

comparison is based on a distance metric (default is absolute difference) and a Tolerance

when multiple correct answers are possible, chooses the closest

Multipart answer with all elements required

QuestionObject[
	Column[{StringForm["Find all the roots of the polynomial p(x) = ``"
						,pol=Expand[(x-1/3)(x-2)(x-11/2)]]
		   ,OpenerView[{Style["Hint:",Blue,10],},True]}]
	,AssessmentFunction[{{1/3, 2, 11/2}}
						,<|"ComparisonMethod"->"Number","ListAssessment"->"AllElementsOrderless"|>
						,Tolerance->0.05]]

“Number”

code example 1: multipart answer

roots=AssessmentFunction[{{1,2,11/2}},<|"ListAssessment"->"AllElementsOrderless"|>,Tolerance->0.01]

roots[{2,1}]

roots[{2,1,5.49}]

Tolerance is a most important setting, by default it is 0.

AssessmentFunction[{{1,2,11/2}},<|"ComparisonMethod"->"Number","ListAssessment"->"AllElementsOrderless"|>][{2,1,5.49}]

“Number”

code example 2: multiple possible answers

pi=AssessmentFunction[{ 3->1 ,3.1->2,3.2->0.5,Pi->5},Tolerance->0.05]

result=pi[3.14]

result["Score"]

“Vector”

to compare vectors!

comparison is based on a distance metric (default is euclidean distance) and a Tolerance.

when multiple answers are possible, choose the closest

Single answer. Custom distance metric

QuestionObject[
	Column[{StringForm["Give any vector orthogonal to ``",MatrixForm@{0,0,1}]
		   ,OpenerView[{Style["Hint:",Blue,10],},False]}]
	,AssessmentFunction[{{0,0,1}},"Vector", DistanceFunction->Dot,Tolerance->0.01]]

Built-in comparison “math” methods

Used for Free-Form assessments

Simple Types

"Number"

“Vector”

Complex Expressions

“ArithmeticResult”

“PolynomialResult”

“CalculusResult”

“AlgebraicValue”

-> Within each method:

the evaluation of the expression is more or less restrained

custom comparison is performed to determine equivalence

“ArithmeticResult”

best suited for arithmetic exercises

no evaluation

allows any ordering of the terms

ignores “unnecessary” parenthesis (-> no impact on evaluation)

Single free form answer

QuestionObject[
StringForm["How much is ``? \n``",HoldForm[🍏+3*🍊-2🍒+🍏-🍊+🍒-🍊],],
AssessmentFunction[2 🍏+🍊-🍒,"ArithmeticResult"]]

“ArithmeticResult”

code example 1: Preventing the answer from evaluating. Allowing reordering of the terms

AssessmentFunction[3+2*a,"ArithmeticResult"][1+a+a+2]

In AssessmentFunction Hold is only preventing specific transformations from being performed, depending on the comparison method

AssessmentFunction[3+2 a,"ArithmeticResult"][Hold[1+a+a+2]]

AssessmentFunction[3+2a,"ArithmeticResult"][Hold[3+2a]]

AssessmentFunction[3+2 a,"ArithmeticResult"][Hold[2a+3]]

AssessmentFunction[3+2 a,"ArithmeticResult"][Hold[2a+3+1-1]]

For comparison:  the “HeldExpression” method

AssessmentFunction[Hold[3+2a],"HeldExpression"][Hold[2a+3]]

“ArithmeticResult”

code example 2: automatically simplifies parenthesis (when there is no impact on evaluation)

AssessmentFunction[Hold[(3+4)*x],"ArithmeticResult"][Hold[3+4*x]]

AssessmentFunction[Hold[3*4*x],"ArithmeticResult"][Hold[(3*4)*x]]

AssessmentFunction[Hold[1+2+3*(4*x)],"ArithmeticResult"][Hold[1+(2+3*(4*x))]]

AssessmentFunction[Hold[1+2+3*4*x],"ArithmeticResult"][Hold[1+(2+3*(4*x))]]

AssessmentFunction[Hold[1+2+(3*(4*x))],"ArithmeticResult"][Hold[1+(2+3*(4*x))]]

AssessmentFunction[1+Cos[x+2*Pi],"ArithmeticResult"][Cos[(Pi*2)+x]+1]

“ArithmeticResult”

code example 3: not suited when too many multiple correct answers !

"Task: Factor the polynomial"

-12+10 x-2 (x^2)//Factor

AssessmentFunction[-12+10 x-2 (x^2)//Factor,"ArithmeticResult"][-2 (-3+x) (-2+x)]

AssessmentFunction[-12+10 x-2 (x^2)//Factor,"ArithmeticResult"][2(x-3)(2-x)]

AssessmentFunction[-12+10 x-2 (x^2)//Factor,"ArithmeticResult"][(2x-6)(2-x)]

Adding a new answer in the answer key:

result=AssessmentFunction[
		{-12+10 x-2 (x^2)//Factor,
		2(x-3)(2-x),
		(2x-6)(2-x)
		(* ... infinite correct answers*)
		}
		,"ArithmeticResult"][2(x-3)(2-x)]

“PolynomialResult”

best suited for polynomial exercises

blocks evaluation

allows any ordering of the terms

allow equivalent distribution of coefficients

supports pattern matching

Free form answer

QuestionObject[
StringForm["Factor the polynomial `1`",Expand[poly=(1-x)(2-x)(3-x)]],
AssessmentFunction[poly,"PolynomialResult"]
]

“PolynomialResult”

code example 1: holding evaluation but allowing equivalent forms

factorisation=AssessmentFunction[(1+x)(2-x)(3-x),"PolynomialResult"]

Factor[6+x-4 (x^2)+(x^3)]

factorisation[Factor[6+x-4 (x^2)+(x^3)]]

factorisation[Hold[Factor[6+x-4 (x^2)+(x^3)]]]

factorisation[Hold[(3-x)(1+x)(1+1-x)]]

“PolynomialResult” considers equivalent  expressions which differs by different distributions of the signs or coefficients

-(-3+x)(1+x)(2-x)

factorisation[Hold[-(-3+x)(1+x)(2-x)]]

factorisation[Hold[-2(-3+x)(1+x)(1-x/2)]]

factorisation[Hold[-6(-1+x/3)(1+x)(1-x/2)]]

factorisation[Hold[-6(-1+x/3)(1+1+x)(1-x/2)]]

“PolynomialResult”

code example 2: Support for “Blank” pattern matching

MatchQ[1+x+a,a+1+_]

“PolynomialResult” support “blank” pattern matching: _

partialcredit=AssessmentFunction[{(1+x)(2-x)(3-x) -> 1,
								  (1+x)(_-x)(3-x) -> 1/2}
								,"PolynomialResult"]

result=partialcredit[(1+x)(2-x)(3-x)]

result["Score"]

result2=partialcredit[(1+x)(-2-x)(3-x)]

result2["Score"]

partialcredit[Hold[(1+x)(1+1-x)(3-x)]]["Score"]

Equivalent distribution of the coefficient for “blank” pattern matching is currently supported only for integers and symbols

partialcredit[3(1+x)(-2-x)(1-x/3)]

partialcredit[-3(1+x)(whatever+x)(1-x/3)]

“PolynomialResult”

Coming soon: new method options to enable sign and/or all coefficients equivalence

AssessmentFunction[(1+x)(2-x)(3-x),<|"ComparisonMethod"->"PolynomialResult", "AllowEquivalentDistribution"->Sign|>][Hold[(1+x)(-2+x)(-3+x)]]

AssessmentFunction[(1+x)(2-x)(3-x),<|"ComparisonMethod"->"PolynomialResult", "AllowEquivalentDistribution"->Sign|>][Hold[2(1+x)(-1+x/2)(-3+x)]]

AssessmentFunction[(1+x)(2-x)(3-x),<|"ComparisonMethod"->"PolynomialResult", "AllowEquivalentDistribution"->None|>][(1+x)(-2+x)(-3+x)]

“CalculusResult”

best suited for exercises about calculus

only blocks calculus operations (derivation, integration)

allows equivalence for expressions with different integration constants names.

allows broad math expressions equivalence

Free form input

qg=QuestionGenerator[{"trig":>RandomChoice[{Sin,Cos}],"a":>RandomInteger[{1,4}]},
Module[{f, der, int},
f=#trig[x]^#a;
der=D[f,x];
int=Integrate[f,x];
plot=Plot[Evaluate[{Legended[f,"f"[x]==f],Legended[der,D["f"[x],x]],Legended[int,Integrate["f"[x],x]]}],];
QuestionObject[QuestionInterface["TextCompletion",<|"Prompt"->plot,"Template"->{"What is f'[x]==",TemplateSlot[1],"and ∫f[x]𝕕x == ",TemplateSlot[2]},"Interpreter"->"HeldMathExpression"(*,"FieldSize"->{15,1}*)|>],AssessmentFunction[{{der,int}},<|"ComparisonMethod"->"CalculusResult","ListAssessment"->"AllElementsOrdered"|>]]]&];qg[]

“CalculusResult”

code example 1: math expression equivalence

answerkey = Integrate[Sin[x]^2,x]

AssessmentFunction[answerkey,"CalculusResult"][Hold[Integrate[Sin[x]^2,x]]]

AssessmentFunction[answerkey,"CalculusResult"][Hold[(x/2)-(1/4) Sin[2 x]]]

AssessmentFunction[answerkey,"CalculusResult"][Hold[(2x-Sin[2 x])/4]]

AssessmentFunction[answerkey,"CalculusResult"][Hold[(2x-2Sin[x]Cos[x])/4]]

2Sin[x]Cos[x]==Sin[2*x]//Simplify

“CalculusResult”

code example 2: automatic equivalence for integration constants names

Optional Constant term

integral=Integrate[x^2,x,GeneratedParameters->C]

AssessmentFunction[integral,"CalculusResult"][x^3/3]

AssessmentFunction[integral,"CalculusResult"][x^3/3+a]

AssessmentFunction[integral,"CalculusResult"][x^3/3+whateversymbol]

Multitple Constants

solution=DSolveValue[y''''[x]==y[x],y[x],x]

diffeqtest=AssessmentFunction[solution,"CalculusResult"]

diffeqtest[ Hold[DSolveValue[y''''[x]==y[x],y[x],x]]]

diffeqtest[ (E^x)a+ (E^(-x))c+b Cos[x]+d Sin[x]]

diffeqtest[ (E^x)a+ (E^(-x))a+b Cos[x]+d Sin[x]]

diffeqtest[ (E^x)a+ (E^(-x))x+b Cos[x]+d Sin[x]]

diffeqtest[ (E^x)a+ (E^(-x))c+b (Cos[x]+d Sin[x]/b)]

diffeqtest[ (E^x)a+ (E^(-x))c+b (Cos[x]+2d Sin[x/2]Cos[x/2]/b)]

“AlgebraicValue”

best suited for exercises about solving equations

only blocks algebra solving functions

allows setting names to solutions

allows broad math expressions equivalence

Free form answer

A={{u,2*u},{u,u^2}}; b={26,19};
QuestionObject[
StringForm["Give the solution `1` and `2` to the linear system of equations:\n\t\t `3`",Style[x,Bold,Italic,Blue],Style[y,Bold,Italic,Blue],MatrixForm/@(system=Equal[A.{x,y} ,b])],
AssessmentFunction[Solve[system,{x,y}],<|"ComparisonMethod"->"AlgebraicValue", "OptionalResultName"->True, "ListAssessment"->"AllElementsOrdered"|>]
]

>>>Cheat code

A={{u,2*u},{u,u^2}};

RowReduce@A

b={26,19};

MatrixForm[RowReduce@Join[A,Transpose[{b}],2]]

A.{x,y}==b

Solve[A.{x,y}==b,{x,y}]

Inverse[A].b

Inverse[A].b//Simplify

Inverse[A].b//FullSimplify

“AlgebraicValue”

Code example: named solutions

Named solutions only works using “OptionalResultName” + “AllElementsOrdered”

Generate the answer key and the assessment function

solution=Solve[{x,y}∈InfiniteLine[{{0,0},{2,1}}]&&{x,y}∈Circle[],{x,y}]

lineCircleIntersection=AssessmentFunction[solution,<|"ComparisonMethod"->"AlgebraicValue",
"OptionalResultName"->True,"ListAssessment"->"AllElementsOrdered"|>];

Assessments

lineCircleIntersection[{-(2/(Sqrt[5])),-(1/(Sqrt[5]))}]

lineCircleIntersection[{-(1/(Sqrt[5])),-(2/(Sqrt[5]))}]

lineCircleIntersection[{x->-(2/(Sqrt[5])),y==-(1/(Sqrt[5]))}]

lineCircleIntersection[{Hold[x=-((2(Sqrt[5]))/5)],y==-(1/(Sqrt[5]))}]

Custom comparison

AssessmentFunction[
			{
			key1->"Score1", 
			key2->"Score2", ...}             (* answer key *)
			,
			<| 
				      "ComparisonMethod"        ->         (*built in method*)
				      "specific method options" ->        (*built in option*)
			(*or*)      "Comparator"                          ->        (*user custom comparison method*)
			(*or*)      "Selector"                                   ->        (*user custom selection method*)
                                                                        "ListAssessment"                   ->        (*enable multiple part answers*)
				      "other settings"                      ->        {(*...*)}
			|>
		            
	                                 , Tolerance                                                         -> 100(*numeric value*)
	                                 , DistanceFunction                                         -> func(*distance metric function*)
		     ]

-> Implementing a custom method is necessary as soon as too many correct answers are possible and when no built in method meets the needs.

Custom comparison

Code Example: Quizz-> Give any orthonormal vector to {0,1,0}

1/ Using “Vector” method

orthonormalvector=AssessmentFunction[{{0,1,0}},"Vector", DistanceFunction->(#1.#2 +Abs[(Norm[#1]-1)]&)]

orthonormalvector[{1.01,0,0}]

orthonormalvector[{1.0,0,0}]

2/ Using a custom “Comparator” method

orthonormalvectorComparator=AssessmentFunction[{{0,1,0}},<|"Comparator"->(And[#1.#2==0, Norm[#1]==1]&)|>]

orthonormalvectorComparator[{0,0,1.1}]

orthonormalvectorComparator[{0,0,1.}]

orthonormalvectorComparatorWithTolerance=AssessmentFunction[{{0,1,0}},<|"Comparator"->(And[Abs[#1.#2]<0.01, Abs[Norm[#1]-1]<0.001]&)|>]

orthonormalvectorComparatorWithTolerance[{1.001,0,0}]

mycomparator[givenanswer_, answerkey_]:=And[Abs[givenanswer.answerkey]<0.01, Abs[Norm[givenanswer]-1]<0.001]

AssessmentFunction[{{0,1,0}},<|"Comparator"->mycomparator|>][{0,0,1.001}]

3/ Using pattern matching -> custom scores

unitvector[vec_]:=Abs[Norm[vec]-1]<0.001
orthogonal[refvec_][ vec_]:=And[Abs[refvec.vec]<0.01]

and[tests__][vec_]:=Apply[And,Through[{tests}[vec]]]

customscore=AssessmentFunction[{
					_?(and[unitvector,orthogonal[{1,0,0}]])->2,
					_?(orthogonal[{1,0,0}])->1,
					_?unitvector->1/2
					}]

customscore[{0,0,1}]["Score"]

customscore[{0,0,2}]["Score"]

customscore[{1/Sqrt[2],1/Sqrt[2],0}]["Score"]

1. SETTING AND RUNNING 
Assessments for Math

2. DEPLOY ASSESSMENTS

CLOUD Deployement

1/ Running cloud deployed quizzes in the notebook environment

AssessmentFunction

define the assessment

myelementwiseassessment=AssessmentFunction[(x-1)(x-2),<|"ComparisonMethod"->"PolynomialResult","ListAssessment"->"SeparatelyScoreElements"|>]

check the assessment (run locally)

myelementwiseassessment[{(x-1)(x-2),(x-1)(x-2),(-x+1)(-x+2)}][All]

deploy the assessment to the cloud

mycloudAF=ResourceFunction["QuestionDeploy"][myelementwiseassessment]

run the deployed assessment inside the notebook

submittedAnswer={(x-1)(x-2),(x-1)(x-2),(-x+1)(-x+2)};

mycloudAF[submittedAnswer]

%[All]

QuestionObject

define the QuestionObject

myquiz=QuestionObject[QuestionInterface["MultipleShortAnswers",<|"Prompt"->StringForm["Give equivalent factorized forms of the polynomial: `1`",Expand[(x-1)(x-2)]],"Interpreter"->"HeldMathExpression"|>],myelementwiseassessment]

deploy the question object to the cloud

mydeployedquiz=ResourceFunction["QuestionDeploy"][myquiz];

mydeployedquiz["QuestionObject"]//InputForm

run the deployed question object inside the notebook

mydeployedquiz["QuestionObject"]

CLOUD Deployement

2/ HTTPRequest and JSON response

Using the previously defined assessment:

myelementwiseassessment

Deploying to the cloud and specify an Interpreter

mycloudAFwithInterpreter=ResourceFunction["QuestionDeploy"][myelementwiseassessment,"AnswerInterpretation"->Interpreter[DelimitedSequence["MathExpression",","]]]

Making the HTTPRequest and reading the response

baseurl=mycloudAFwithInterpreter[[1,1]]

myresponse=URLRead[
HTTPRequest[CloudObject@URLBuild[{baseurl,"assess"},{"ExpressionFormat"->"TeXFragment","ResponseFormat"->"JSON"}],
<|Method->"POST","Body"->"(x-1)(x-2),(x-1)(x-2),-2(x-1)(1-x/2)","ContentType"->"text/plain"|>
]
]

Import[myresponse,"String"]

Import[myresponse,"RawJSON"]

Question Framework Applications

CBM: Computer Based Math Educational curriculum and software

WELE / Wolfram EdTech: full education solutions

Wolfram Tutor: an upcoming AI-based tutoring platform

Other WTC 2024 Talks

Creating Quizzes With QuestionObject (->QuestionFramework)

Wolfram Consulting AI Math Assistant and Enabling Components

JerryBot: A Proactive AI Tutor for Algebra One

Converting OpenStax Textbooks for Use by LLMs, with an Example Application to Chemistry Education