Computerized Adaptive Testing Simulation – `catsim.simulation`

Module containing functions relevant to the process of simulating the application of adaptive tests.

Most of this module is based on the work of [Bar10].

class catsim.simulation.Estimator(verbose: bool = False)[source]

Base class for ability estimators.

Estimators are responsible for computing ability estimates based on examinees’ responses to administered items.

Parameters:

verbosebool, optional: Whether to be verbose during execution. Default is False.

property avg_evaluations: float

Get the average number of function evaluations for all tests the estimator has been used.

Returns:

float: Average number of function evaluations per test.

property calls: int

Get how many times the estimator has been called to maximize/minimize the log-likelihood function.

Returns:

int: Number of times the estimator has been called to maximize/minimize the log-likelihood function.

Compute the theta value that maximizes the log-likelihood function for the given examinee.

When this method is used inside a simulator, its arguments are automatically filled. Outside of a simulation, the user can also specify the arguments to use the Estimator as a standalone object.

Parameters:

indexint or None, optional: Index of the current examinee in the simulator. Default is None.
item_bankItemBank or None, optional: An ItemBank containing item parameters. Default is None.
administered_itemslist[int] or None, optional: A list containing the indexes of items that were already administered. Default is None.
response_vectorlist[bool] or None, optional: A boolean list containing the examinee’s answers to the administered items. Default is None.
est_thetafloat or None, optional: A float containing the current estimated ability. Default is None.

Returns:

float: The current estimated ability \(\hat\theta\).

property evaluations: int

Get the total number of times the estimator has evaluated the log-likelihood function during its existence.

Returns:

int: Number of function evaluations.

class catsim.simulation.FiniteSelector(test_size: int)[source]

Base class representing a CAT item selector for fixed-length tests.

Parameters:

test_sizeint: Number of items to be administered in the test.

property overlap_rate: float | None

Get the overlap rate of the test, if it is of finite length.

Returns:

float or None: Overlap rate of the test, or None if not yet computed.

property test_size: int

Get the number of items to be administered in the test.

Returns:

int: Number of items to be administered in the test.

class catsim.simulation.Initializer[source]

Base class for CAT initializers.

Initializers are responsible for selecting examinees’ initial ability estimates before any items are administered.

abstractmethod initialize(**kwargs: Any) → float[source]

Select an examinee’s initial \(\theta\) value.

Parameters:

**kwargsdict: Arguments used by the Initializer implementation.

Returns:

float: Examinee’s initial \(\theta\) value.

class catsim.simulation.Selector[source]

Base class representing a CAT item selector.

Selectors are responsible for choosing which item to administer next to an examinee based on their current estimated ability and test progress.

Return the index of the next item to be administered.

Parameters:

indexint or None, optional: The index of the current examinee in the simulator. Default is None.
item_bankItemBank or None, optional: An ItemBank containing item parameters. Default is None.
administered_itemslist[int] or None, optional: A list containing the indexes of items that were already administered. Default is None.
est_thetafloat or None, optional: A float containing the current estimated ability. Default is None.
**kwargsdict: Additional keyword arguments.

Returns:

int or None: Index of the next item to be applied, or None if there are no more items to be presented.

class catsim.simulation.Simulable[source]

Base class representing one of the Simulator components that will receive a reference back to it.

This class provides the infrastructure for components (Initializers, Selectors, Estimators, and Stoppers) to access the Simulator instance they belong to.

Notes

This class inherits from ABC to indicate it’s meant to be subclassed, though it doesn’t define abstract methods itself. Concrete abstract methods are defined in its subclasses (Initializer, Selector, Estimator, Stopper).

preprocess() → None[source]

Override this method to perform any initialization the Simulable might need for the simulation.

preprocess is called after a value is set for the simulator property. If a new value is attributed to simulator, this method is called again, guaranteeing that internal properties of the Simulable are re-initialized as necessary.

Notes

The default implementation does nothing. Subclasses should override this method if they need to perform setup operations that require access to the simulator.

property simulator: Simulator

Set or get the simulator object.

Returns:

Simulator: The Simulator instance tied to this Simulable.

Raises:

TypeError: If the simulator is not of type catsim.simulation.Simulator or is None.

Class representing the CAT simulator.

The Simulator gathers several objects that describe the full simulation process (initializer, selector, estimator, stopper) and simulates one or more computerized adaptive tests.

Parameters:

item_bankItemBank or numpy.ndarray: An ItemBank object containing item parameters. If a numpy.ndarray is provided, it will be automatically converted to an ItemBank.
examineesint or npt.ArrayLike: Either an integer with the number of examinees (whose real \(\theta\) values will be sampled from a normal distribution), or an array-like (list, tuple, or numpy array) containing the examinees’ true \(\theta\) values (float type).
initializerInitializer or None, optional: Initializer to use during the simulation. Default is None.
selectorSelector or None, optional: Selector to use during the simulation. Default is None.
estimatorEstimator or None, optional: Estimator to use during the simulation. Default is None.
stopperStopper or None, optional: Stopper to use during the simulation. Default is None.
seedint, optional: Seed used by the numpy random number generator during the simulation procedure. Default is 0.

property administered_items: list[list[int]]: A list of lists with the indexes of items administered to each examinee during the simulation.

property bias: float

Get the bias between the estimated and true abilities.

This property is only available after simulate() has been successfully called. For more information on estimation bias, see catsim.cat.bias()

property duration: float: Duration of the simulation, in seconds.

property estimations: list[list[float]]: A list of lists with all estimated \(\hat\theta\) values for all examinees during each step of the test.

property estimator: Estimator | None

Get the estimator used during the simulation.

Returns:

Estimator or None: The estimator used during the simulation.

property examinees: ndarray[tuple[Any, ...], dtype[floating[Any]]]: :py:type:numpy.ndarray containing examinees true ability values (\(\theta\)).

property initializer: Initializer | None

Get the initializer used during the simulation.

Returns:

Initializer or None: The initializer used during the simulation.

property item_bank: ItemBank

ItemBank used by the simulator.

Returns:

ItemBank: The ItemBank containing all item parameters and exposure rates.

property items: ndarray[tuple[Any, ...], dtype[floating[Any]]]

Item matrix used by the simulator (for backward compatibility).

Returns:

npt.NDArray[numpy.floating[Any]]: The underlying item parameter matrix from the ItemBank.

property latest_estimations: list[float]: Final estimated \(\hat\theta\) values for all examinees.

property mse: float

Get the mean-squared error between the estimated and true abilities.

This property is only available after simulate() has been successfully called. For more information on the mean-squared error of estimation, see catsim.cat.mse().

property overlap_rate: float: Overlap rate of the test, if it is of finite length.

property response_vectors: list[list[bool]]: List of boolean lists containing the examinees answers to all items.

property rmse: float

Get the root mean-squared error between the estimated and true abilities.

This property is only available after simulate() has been successfully called. For more information on the root mean-squared error of estimation, see catsim.cat.rmse().

property rng: Generator: Get the random number generator used by the simulator.

property selector: Selector | None

Get the selector used during the simulation.

Returns:

Selector or None: The selector used during the simulation.

simulate(initializer: Initializer | None = None, selector: Selector | None = None, estimator: Estimator | None = None, stopper: Stopper | None = None, verbose: bool = False) → None[source]

Simulate a computerized adaptive testing application to one or more examinees.

The simulation process iterates through each examinee, initializing their ability, selecting items, recording responses, estimating abilities, and stopping when the criterion is met.

Parameters:

initializerInitializer or None, optional: An initializer that selects examinees’ initial \(\theta_0\). Default is None.
selectorSelector or None, optional: A selector that selects new items to be presented to examinees. Default is None.
estimatorEstimator or None, optional: An estimator that reestimates examinees abilities after each item is applied. Default is None.
stopperStopper or None, optional: An object with a stopping criterion for the test. Default is None.
verbosebool, optional: Whether to periodically print a message regarding the progress of the simulation. Good for longer simulations. Default is False.

Raises:

ValueError: If any of initializer, selector, estimator, or stopper is None (either passed or from constructor).

Examples

>>> from catsim.initialization import RandomInitializer
>>> from catsim.selection import MaxInfoSelector
>>> from catsim.estimation import NumericalSearchEstimator
>>> from catsim.stopping import MaxItemStopper
>>> from catsim.simulation import Simulator
>>> from catsim.item_bank import ItemBank
>>> initializer = RandomInitializer()
>>> selector = MaxInfoSelector()
>>> estimator = NumericalSearchEstimator()
>>> stopper = MaxItemStopper(20)
>>> Simulator(ItemBank.generate_item_bank(100), 10).simulate(initializer, selector, estimator, stopper)

property stopper: Stopper | None

Get the stopper used during the simulation.

Returns:

Stopper or None: The stopper used during the simulation.

class catsim.simulation.Stopper[source]

Base class for CAT stopping criteria.

Stoppers determine when a test should end based on specific criteria such as test length, measurement precision, or other conditions.

abstractmethod stop(index: int | None = None, **kwargs: Any) → bool[source]

Check whether the test reached its stopping criterion for the given user.

Parameters:

indexint or None, optional

The index of the current examinee in the simulator. When used within a simulation, this parameter is provided automatically. When used standalone, other parameters may be provided via kwargs. Default is None.

**kwargsdict

Additional keyword arguments that specific Stopper implementations may require. Common arguments include:

administered_items: Item parameters or indices that were administered
theta: Current ability estimate
item_bank: ItemBank for accessing item parameters

Returns:

bool: True if the test met its stopping criterion, False otherwise.

Computerized Adaptive Testing Simulation – catsim.simulation

Computerized Adaptive Testing Simulation – `catsim.simulation`