Item Response Theory Functions – catsim.irt

Item Response Theory is a group of statistical tools created with the purpose of measuring latent traits, like personality traits or knowledge in specific areas. IRT allows for the creation of a measuring instrument (the test), under which every examinee may be positioned and compared with others. The masuring instrument is created by items, which can be most commonly represented as tasks in a test.

On the mathematical domain, IRT allows us to calculate the probability with which an examinee might answer an item correctly. For that both items and examinees are modelled using numerical parameters. This allows for the estimation and analysis of examinees proficiencies on the same scale as items difficulties. It also allows for the quality of items to be judged by the values of their parameters.

This module containing functions pertaining to the Item Response Theory logistic models.

catsim.irt.icc(theta: float, a: float, b: float, c: float = 0, d: float = 1) → float[source]

Item Response Theory four-parameter logistic function [Ayala2009], [Magis13]:

\[P(X_i = 1| \theta) = c_i + \frac{d_i-c_i}{1+ e^{a_i(\theta-b_i)}}\]
Return type:

float

Parameters:
  • theta (float) – the individual’s proficiency value. This parameter value has no boundary, but if a distribution of the form \(N(0, 1)\) was used to estimate the parameters, then \(-4 \leq \theta \leq 4\).
  • a (float) – the discrimination parameter of the item, usually a positive value in which \(0.8 \leq a \leq 2.5\).
  • b (float) – the item difficulty parameter. This parameter value has no boundaries, but it is necessary that it be in the same value space as theta (usually \(-4 \leq b \leq 4\)).
  • c (float) – the item pseudo-guessing parameter. Being a probability, \(0\leq c \leq 1\), but items considered good usually have \(c \leq 0.2\).
  • d (float) – the item upper asymptote. Being a probability, \(0\leq d \leq 1\), but items considered good usually have \(d \approx 1\).
catsim.irt.icc_hpc(theta: float, items: numpy.ndarray) → numpy.ndarray[source]

Implementation of icc() using numpy and numexpr in which the characteristic function for all items in a numpy.ndarray are calculated at once.

Return type:

ndarray

Parameters:
  • theta (float) – the individual’s proficiency value.
  • items (ndarray) – array containing the four item parameters.
Returns:

an array of all item characteristic functions, given the current theta

catsim.irt.inf(theta: float, a: float, b: float, c: float = 0, d: float = 1) → float[source]

Calculates the information value of an item using the Item Response Theory four-parameter logistic model function [Ayala2009], [Magis13]:

\[I_i(\theta) = \frac{a^2[(P(\theta)-c)]^2[d - P(\theta)]^2}{(d-c)^2[1-P(\theta)]P(\theta)}\]
Return type:

float

Parameters:
  • theta (float) – the individual’s proficiency value. This parameter value has no boundary, but if a distribution of the form \(N(0, 1)\) was used to estimate the parameters, then \(-4 \leq \theta \leq 4\).
  • a (float) – the discrimination parameter of the item, usually a positive value in which \(0.8 \leq a \leq 2.5\).
  • b (float) – the item difficulty parameter. This parameter value has no boundary, but if a distribution of the form \(N(0, 1)\) was used to estimate the parameters, then \(-4 \leq b \leq 4\).
  • c (float) – the item pseudo-guessing parameter. Being a probability, \(0\leq c \leq 1\), but items considered good usually have \(c \leq 0.2\).
  • d (float) – the item upper asymptote. Being a probability, \(0\leq d \leq 1\), but items considered good usually have \(d \approx 1\).
Returns:

the information value of the item at the designated theta point.

catsim.irt.inf_hpc(theta: float, items: numpy.ndarray)[source]

Implementation of inf() using numpy and numexpr in which the information value for all items in a numpy.ndarray are calculated at once.

Parameters:
  • theta (float) – the individual’s proficiency value.
  • items (ndarray) – array containing the four item parameters.
Returns:

an array of all item information values, given the current theta

catsim.irt.log_likelihood(est_theta: float, response_vector: list, administered_items: numpy.ndarray) → float[source]

Calculates the log-likelihood of an estimated proficiency, given a response vector and the parameters of the answered items [Ayala2009].

The likelihood function of a given \(\theta\) value given the answers to \(I\) items is given by:

\[L(X_{Ij} | \theta_j, a_I, b_I, c_I, d_I) = \prod_{i=1} ^ I P_{ij}(\theta)^{X_{ij}} Q_{ij}(\theta)^{1-X_{ij}}\]

For mathematical reasons, finding the maximum of \(L(X_{Ij}\) includes using the product rule of derivations. Since \(L(X_{Ij}\) has \(j\) parts, it can be quite complicated to do so. Also, for computational reasons, the product of probabilities can quickly tend to 0, so it is common to use the log-likelihood in maximization/minimization problems, transforming the product of probabilities in a sum of probabilities:

\[\log L(X_{Ij} | \theta_j, a_I, b_I, c_I, d_I) = \sum_{i=1} ^ I \left\lbrace x_{ij} \log P_{ij}(\theta)+ (1 - x_{ij}) \log Q_{ij}(\theta) \right\rbrace\]
Return type:

float

Parameters:
  • est_theta (float) – estimated proficiency value.
  • response_vector (list) – a Boolean list containing the response vector.
  • administered_items (ndarray) – a numpy array containing the parameters of the answered items.
Returns:

log-likelihood of a given proficiency value, given the responses to the administered items.

catsim.irt.max_info(a: float = 1, b: float = 0, c: float = 0, d: float = 1) → float[source]

Returns the \(\theta\) value to which the item with the given parameters gives maximum information. For the 1-parameter and 2-parameter logistic models, this \(\theta\) corresponds to where \(b = 0.5\). In the 3-parameter and 4-parameter logistic models, however, this value is given by ([Magis13])

\[argmax_{\theta}I(\theta) = b + \frac{1}{a} log \left(\frac{x^* - c}{d - x^*}\right)\]

where

\[x^* = 2 \sqrt{\frac{-u}{3}} cos\left\{\frac{1}{3}acos\left(-\frac{v}{2}\sqrt{\frac{27}{-u^3}}\right)+\frac{4 \pi}{3}\right\} + 0.5\]
\[u = -\frac{3}{4} + \frac{c + d - 2cd}{2}\]
\[v = -\frac{c + d - 1}{4}\]

A few results can be seen in the plots below:

(Source code)

_images/irt-1_00.png

Fig. 3 (png, hires.png, pdf)

_images/irt-1_01.png

Fig. 4 (png, hires.png, pdf)

Return type:

float

Parameters:
  • a (float) – item discrimination parameter
  • b (float) – item difficulty parameter
  • c (float) – item pseudo-guessing parameter
  • d (float) – item upper asymptote
catsim.irt.max_info_hpc(items: numpy.ndarray)[source]

Implementation of max_info() using numpy and numexpr in which the theta value that maximizes the information function for all items is returned in a numpy.ndarray.

Parameters:items (ndarray) – array containing the four item parameters.
Returns:an array of all theta values that maximize the information function of each item.
catsim.irt.negative_log_likelihood(est_theta: float, *args) → float[source]

Function used by scipy.optimize optimization functions that tend to minimize values, instead of maximizing them. Calculates the negative log-likelihood of a proficiency value, given a response vector and the parameters of the administered items. The value of negative_log_likelihood() is simply the value of \(-\) log_likelihood() or, mathematically:

\[- \log L(X_{Ij} | \theta_j, a_I, b_I, c_I, d_I)\]
Return type:float
Parameters:est_theta (float) – estimated proficiency value

args:

Parameters:
  • list (response_vector) – a Boolean list containing the response vector
  • numpy.ndarray (administered_items) – a numpy array containing the parameters of the answered items
Returns:

negative log-likelihood of a given proficiency value, given the responses to the administered items

catsim.irt.normalize_item_bank(items: numpy.ndarray) → numpy.ndarray[source]

Normalize an item matrix so that it conforms to the standard used by catsim. The item matrix must have dimension nx3, in which column 1 represents item discrimination, column 2 represents item difficulty, column 3 represents the pseudo-guessing parameter and column 4 represents the item upper asymptote.

If the matrix has one column, it is assumed to be the difficulty column and the other two columns are added such that items simulate the 1-parameter logistic model.

If the matrix has two columns, they are assumed to be the discrimination and difficulty columns, respectively. The pseudo-guessing column is added such that items simulate the 2-parameter logistic model.

If the matrix has three columns, they are assumed to be the discrimination, difficulty and pseudo-guessing columns, respectively. The upper asymptote column is added such that items simulate the 3-parameter logistic model.

Return type:ndarray
Parameters:items (ndarray) – the item matrix.
Returns:an nx4 item matrix conforming to the 4 parameter logistic model.
catsim.irt.reliability(theta: float, items: numpy.ndarray)[source]

Computes test reliability [Thissen00], given by:

\[Rel = 1 - \frac{1}{I(\theta)}\]

Test reliability is a measure of internal consistency for the test, similar to Cronbach’s \(\alpha\) in Classical Test Theory. Its value is always lower than 1, with values close to 1 indicating good reliability. If \(I(\theta) < 1\), \(Rel < 0\) and in these cases it does not make sense, but usually the application of additional items solves this problem.

Parameters:
  • theta (float) – a proficiency value.
  • items (ndarray) – a matrix containing item parameters.
Returns:

the test reliability at theta for a test represented by items.

catsim.irt.see(theta: float, items: numpy.ndarray) → float[source]

Computes the standard error of estimation (\(SEE\)) of a test at a specific \(\theta\) value [Ayala2009]:

\[SEE = \sqrt{\frac{1}{I(\theta)}}\]

where \(I(\theta)\) is the test information (see test_info()).

Return type:

float

Parameters:
  • theta (float) – a proficiency value.
  • items (ndarray) – a matrix containing item parameters.
Returns:

the standard error of estimation at theta for a test represented by items.

catsim.irt.test_info(theta: float, items: numpy.ndarray) → float[source]

Computes the test information of a test at a specific \(\theta\) value [Ayala2009]:

\[I(\theta) = \sum_{j \in J} I_j(\theta)\]

where \(J\) is the set of items in the test and \(I_j(\theta)\) is the item information of \(j\) at aspecific \(\theta\) value.

Return type:

float

Parameters:
  • theta (float) – a proficiency value.
  • items (ndarray) – a matrix containing item parameters.
Returns:

the test information at theta for a test represented by items.

catsim.irt.validate_item_bank(items: numpy.ndarray, raise_err: bool = False)[source]

Validates the shape and parameters in the item matrix so that it conforms to the standard used by catsim. The item matrix must have dimension nx4, in which column 1 represents item discrimination, column 2 represents item difficulty, column 3 represents the pseudo-guessing parameter and column 4 represents the item upper asymptote.

The item matrix must have at least one line, exactly four columns and \(\forall i \in I , a_i > 0 \wedge 0 < c_i < 1 \wedge 0 < d_i < 1\)

Parameters:
  • items (ndarray) – the item matrix.
  • raise_err (bool) – whether to raise an error in case the validation fails or just print the error message to standard output.
catsim.irt.var(theta: float, items: numpy.ndarray) → float[source]

Computes the variance (\(Var\)) of the proficiency estimate of a test at a specific \(\theta\) value [Ayala2009]:

\[Var = \frac{1}{I(\theta)}\]

where \(I(\theta)\) is the test information (see test_info()).

Return type:

float

Parameters:
  • theta (float) – a proficiency value.
  • items (ndarray) – a matrix containing item parameters.
Returns:

the variance of proficiency estimation at theta for a test represented by items.