Estimation Methods – catsim.estimation

Estimators are the objects responsible for estimating of examinees ability values, given a dichotomous (binary) response vector and an array of the items answered by the examinee. In the domain of IRT, there are two main types of ways of estimating \(\hat{\theta}\): and these are the Bayesian methods and maximum-likelihood ones.

Maximum-likelihood methods choose the \(\hat{\theta}\) value that maximizes the likelihood (see catsim.irt.log_likelihood()) of an examinee having a certain response vector, given the corresponding item parameters.

Bayesian methods used a priori information (usually assuming ability and parameter distributions) to make new estimations. The knowledge of new estimations is then used to make new assumptions about the parameter distributions, refining future estimations.

All implemented classes in this module inherit from a base abstract class Estimator. Simulator allows that a custom estimator be used during the simulation, as long as it also inherits from Estimator.

catsim implements a few types of maximum-likelihood estimators.

class catsim.estimation.NumericalSearchEstimator(precision: int = 6, dodd: bool = True, verbose: bool = False, method='bounded')

Bases: Estimator

Estimator that implements multiple search algorithms in unimodal functions to find the maximum of the log-likelihood function. There are implementations of ternary search, dichotomous search, Fibonacci search and golden-section search, according to [Veliz20]. Also check [Brent02]. It is also possible to use the methods from scipy.optimize.minimize_scalar().

Parameters:

• precision (int, optional) – number of decimal points of precision, defaults to 6

• dodd (bool, optional) – whether to employ Dodd’s estimation heuristic [Dod90] when the response vector only has one kind of response (all correct or all incorrect, see catsim.cat.dodd()), defaults to True

• verbose (bool, optional) – verbosity level of the maximization method

• method (str, optional) – the search method to employ, one of ‘ternary’, ‘dichotomous’, ‘fibonacci’, ‘golden’, ‘brent’, ‘bounded’ and ‘golden2’, defaults to bounded

property dodd: bool

Whether Dodd’s estimation heuristic [Dod90] will be used by estimator in case the response vector is composed solely of right or wrong answers.

Returns:

boolean value indicating if Dodd’s method will be used or not.

See:

catsim.cat.dodd()

estimate(index: int | None = None, items: ndarray | None = None, administered_items: List[int] | None = None, response_vector: List[bool] | None = None, est_theta: float | None = None, **kwargs) → float

Returns the theta value that maximizes the log-likelihood function, given the current state of the

test for the given examinee.

Parameters:

• index – index of the current examinee in the simulator

• items – a matrix containing item parameters in the format that catsim understands (see: catsim.cat.generate_item_bank())

• administered_items – a list containing the indexes of items that were already administered

• response_vector – a boolean list containing the examinee’s answers to the administered items

• est_theta – a float containing the current estimated ability

Returns:

the current \(\hat\theta\)

property method: str

Get the estimator search method selected during instantiation.

Returns:

search method

Comparison between estimators

The chart below displays the execution times of the same simulation (100 examinees, an item bank of 300 questions and 20 items per test) using different \(\hat{\theta}\) estimation methods.

(Source code, png, hires.png, pdf)

The charts below show the \(\hat{\theta}\) found by the different estimation methods, given three dichotomous response vectors with different numbers of correct answers. All estimators reach comparable results, maximizing the log-likelihood function . The main difference among them is in the number of calls to catsim.irt.log_likelihood(). Methods that require less calls to maximize the function are usually more time efficient.

(Source code)

Fig. 1 (png, hires.png, pdf)

Fig. 2 (png, hires.png, pdf)

Fig. 3 (png, hires.png, pdf)

Fig. 4 (png, hires.png, pdf)

Fig. 5 (png, hires.png, pdf)

Fig. 6 (png, hires.png, pdf)

Fig. 7 (png, hires.png, pdf)