diff --git a/learn-stan/case-studies.qmd b/learn-stan/case-studies.qmd index e105c0c4..05e7ba30 100644 --- a/learn-stan/case-studies.qmd +++ b/learn-stan/case-studies.qmd @@ -9,6 +9,41 @@ aliases: The case studies on this page are intended to reflect best practices in Bayesian methodology and Stan programming. We aim to keep them current with the latest version of the Stan language, but there may be times when case studies need updating to reflect the latest Stan features and syntax. +## Volume 12 (2026) + +### Bayesian Probabilistic Knowledge Structures + +Knowledge structure theory (KST) seeks to provide procedures for +the effective diagnosis of a student's knowledge state in a certain domain +(such as algebra, geometry, physics, or statistics; Doignon & Falmagne, 1999; +Heller & Stefanutti, 2024). Here, we introduce Bayesian versions of two +prevalent probabilistic KST models: the basic local independence model and the +simple learning model. We demonstrate how these models can be coded in Stan +and interfaced with R. The Bayesian framework can then be leveraged to assess +model adequacy via posterior predictive checks and for model comparison via +cross-validation. These procedures are illustrated with simulated responses +and a real-data example. + +View HTML +*(link opens in new tab)* + +Authors +: Florian Wickelmaier, Julian Mollenhauer, and Alice Jenisch + +Keywords +: psychometrics, knowledge structures, knowledge spaces, BLIM, SLM + +Source Repository +: [fwickel/BayesianPKS](https://codeberg.org/fwickel/BayesianPKS){target="_blank"} +(Codeberg) + +Dependencies +: `rstan`, `pks`, `loo` + +License +: BSD (3 clause), CC-BY + + ## Volume 11 (2025) ### The Sum-to-Zero Constraint in Stan diff --git a/learn-stan/case-studies/probabilistic_knowledge_structures.html b/learn-stan/case-studies/probabilistic_knowledge_structures.html new file mode 100644 index 00000000..06c17d37 --- /dev/null +++ b/learn-stan/case-studies/probabilistic_knowledge_structures.html @@ -0,0 +1,5390 @@ + + + + + + + + + + + + + +Bayesian Probabilistic Knowledge Structures + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+
+

Bayesian Probabilistic Knowledge Structures1

+
+ + + +
+ +
+
Authors
+
+

Florian Wickelmaier

+

Julian Mollenhauer

+

Alice Jenisch

+
+
+ +
+
Published
+
+

October 20, 2026

+
+
+ + +
+ +
+
+
Abstract
+ Knowledge structure theory (KST) seeks to provide procedures for the effective diagnosis of a student’s knowledge state in a certain domain (such as algebra, geometry, physics, or statistics; Doignon & Falmagne, 1999; Heller & Stefanutti, 2024). Here, we introduce Bayesian versions of two prevalent probabilistic KST models: the basic local independence model and the simple learning model. We demonstrate how these models can be coded in Stan and interfaced with R. The Bayesian framework can then be leveraged to assess model adequacy via posterior predictive checks and for model comparison via cross-validation. These procedures are illustrated with simulated responses and a real-data example. +
+
+ + +
+ + +
+
+R setup +
rerun_dgp <- FALSE
+refit_model <- FALSE
+
+library(pks)
+library(rstan)
+options(mc.cores = 4)
+set.seed(1617)
+
+## For plotting with Rgraphviz
+myattrs <- list(
+  graph = list(rankdir = "BT", bgcolor = "white"),
+   edge = list(arrowsize = NULL),
+   node = list(fillcolor = "lightblue", fontsize = 10,
+               shape = "rectangle", fixedsize = FALSE)
+)
+
+
+ +
+

Probabilistic models in knowledge structure theory

+ +

Knowledge structure theory (KST) seeks to provide procedures for the effective diagnosis of a student’s knowledge state in a certain domain, such as algebra, geometry, physics, or statistics (Doignon and Falmagne 1999; Falmagne et al. 1990; Heller and Stefanutti 2024).

+

KST offers an alternative approach to the representation of knowledge that is distinct from classical test theory (Lord and Novick 1968) and from item response theory (IRT, Fischer and Molenaar 1995; Rasch 1960): While the latter theories seek to establish a continuous numerical representation of the ability of a student (the true score or the latent trait), KST provides a discrete non-numerical characterization of the strengths and weaknesses of a student within a set-theoretic framework. Probabilistic KST models are latent class models and related to cognitive diagnostic assessment models (CDMs, Junker and Sijtsma 2001; de la Torre 2009), which were originally devised to identify the qualitative attributes necessary for a student to solve certain problems. The connections between KST, IRT, and CDMs have recently been explicated (Heller et al. 2015, 2016; Noventa et al. 2024, 2025, 2026).

+

The basic notions in KST are

+
    +
  • the knowledge domain: a finite set \(Q\) of (dichotomous) problems (also called items);

  • +
  • the knowledge state: the subset \(K \subseteq Q\) of problems in the domain \(Q\) a student masters;

  • +
  • the knowledge structure: a collection \(\mathcal{K}\) of knowledge states that contains at least the empty set \(\emptyset\) and the set \(Q\).

  • +
+

Only a subset of all possible response patterns \(\mathcal{R}\) are also knowledge states in \(\mathcal{K}\).

+
+

Example: High school geometry

+

Lakshminarayan (1995) presented 959 undergraduate students with the five problems in high school geometry shown in Figure 1.

+
+
+
+
+
+
+
+
+ +
+
+(a) In the figure above, what is the measure of angle p? Give your answer in degrees. +
+
+
+
+
+
+
+
+ +
+
+(b) In the figure above, what is the measure of angle y in degrees? +
+
+
+
+
+
+
+
+
+
+ +
+
+(c) In the figure above, line L is parallel to line M. Angle x is 55 degrees. What is the measure of angle y in degrees? +
+
+
+
+
+
+
+
+ +
+
+(d) In the triangle shown above, side AB has the length of 3 inches, and side AC has the length of 5 inches. Angle ABC is 90 degrees. What is the area of the triangle? +
+
+
+
+
+
+
+
+
+
+ +
+
+(e) In the above quadrilateral, side AB \(=\) 1 inch. The angles are as marked in the figure. The angles marked X are all equal to each other. What is the perimeter of the figure ABCD? +
+
+
+
+
+
+
+Figure 1: Five high school geometry problems. +
+
+
+

With the knowledge domain \(Q = \{a,b,c,d,e\}\) consisting of five items, there are \(2^{|Q|} = 32\) possible response patterns ranging from no problem solved (\(00000\)) to all problems solved (\(11111\)). Not all of them are plausible knowledge states because solving a more difficult problem may require that a simpler problem is solved first. Doignon and Falmagne (1999) introduced a knowledge structure that is used here for illustration; it consists of \(|\mathcal{K}| = +9\) knowledge states (Figure 2).

+
+
data(DoignonFalmagne7, package = "pks")  # provides knowledge structure
+K <- DoignonFalmagne7$K
+rownames(K) <- as.pattern(K, useNames = TRUE)
+rownames(K)[rownames(K) %in% c("{}", "abcde")] <- c("\u2205", "Q")
+plot(relations::as.relation(is.subset(K)), attrs = myattrs, main = "")
+
+
+
+
+ +
+
+Figure 2: Graph of a knowledge structure with nine knowledge states. A label such as \(ab\) is an abbreviation for the set \(\{a, b\}\). An ascending path between two states represents set inclusion. +
+
+
+
+
+

A knowledge structure like the one displayed in Figure 2 may be obtained in different ways: informally, by postulating plausible implications among problems; empirically, by analyzing all pairwise solution patterns and establishing a binary precedence relation between problems (Van Leeuwe 1974; Schrepp 1999); by systematically querying experts (like teachers) whether failing certain problems would entail – based on logical or curricular dependencies – failing other problems (Falmagne et al. 1990; Kambouri et al. 1994).

+
+
+

Deterministic and probabilistic theory

+

The knowledge structure in Figure 2 is special in two ways: First, it is closed under union and thus called a knowledge space. Second, it is well-graded: it can be decomposed into learning paths along which a student can traverse – by learning how to solve one problem at a time – from a state of ignorance \(\emptyset\) to full mastery \(Q\). Consequently, \(\mathcal{K}\) is downgradable: For any nonempty state \(K\) there is some problem \(q\) such that \(K \smallsetminus \{q\} \in \mathcal{K}\). If learning proceeds by mastering one additional problem in each step, the outer fringe collects the problems a student in state \(K\) is ready to learn next; it is defined as \[ + K^\mathcal{O} = \{q \notin K \mid K \cup \{q\} \in \mathcal{K}\}. +\] Figure 3 shows one possible learning path through \(\mathcal{K}\) and illustrates the outer fringe of the knowledge state \(\{a, b\}\): \(c\) and \(d\) are the problems a student in this state is ready to learn.

+
+Code +
rel <- relations::as.relation(is.subset(K))
+mycol <- setNames(rep("white", 9),
+                  unlist(relations::relation_domain(rel)[[1]]))
+mycol["\u2205"] <-
+mycol["b"] <-
+mycol["ab"] <-
+mycol["abc"] <-
+mycol["abcd"] <-
+mycol["Q"] <- "lightblue"
+plot(rel,
+     nodeAttrs = list(fillcolor = mycol),
+     attrs = myattrs, main = "")
+mycol <- setNames(rep("white", 9),
+                  unlist(relations::relation_domain(rel)[[1]]))
+mycol["ab"] <- "lightblue"
+mycol["abd"] <-
+mycol["abc"] <- "lightgreen"
+plot(rel,
+     nodeAttrs = list(fillcolor = mycol),
+     attrs = myattrs, main = "")
+
+
+
+
+
+
+
+
+
+ +
+
+(a) Learning path \(b \to a \to c \to d \to e\) +
+
+
+
+
+
+
+
+ +
+
+(b) Outer fringe \(\{a, b\}^\mathcal{O} = \{c, d\}\) +
+
+
+
+
+
+
+Figure 3: Panel a: Knowledge states marked in blue are along a learning path from \(\emptyset\) to \(Q.\) Panel b: Knowledge states marked in green are the ones relevant for obtaining the outer fringe of \(\{a, b\}\). +
+
+
+

In deterministic KST, there is no explicit formulation of response errors. For example, the pattern \(10010\) does not coincide with any state in \(\mathcal{K}\). The pattern could arise if a student in state \(\{a, b, d\}\) makes a mistake at \(b\) or if a student in state \(\{a\}\) guesses the solution to problem \(d\). In order to formulate a probabilistic model, the implicit distinction between knowledge state and response pattern has to be made explicit (Falmagne and Doignon 1988a, 1988b). A knowledge state \(K\) is a latent class, and the response pattern \(R\) is a manifest indicator of the knowledge state. As a consequence, this gives rise to two types of response errors,

+
    +
  • a careless error, which occurs if the response is incorrect although the problem is contained in \(K\);

  • +
  • a lucky guess, which occurs if the response is correct although the problem is not contained in \(K\).

  • +
+

The latent knowledge states themselves are considered to be random variables with a probability distribution. Formally, a probabilistic knowledge structure consists of

+
    +
  • a knowledge structure \(\mathcal{K}\) on a knowledge domain \(Q\) of problems;

  • +
  • the conditional probabilities \(P(R \mid K)\) to observe response pattern \(R\) given state \(K\);

  • +
  • the marginal distribution \(P(K)\) on the knowledge states \(K \in +\mathcal{K}\).

  • +
+

The probability of the response pattern \(R \in \mathcal{R} = 2^Q\) then is predicted by \[ + P(R) = \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K), +\] and the response frequencies \(y = (N_R)_{R \in \mathcal{R}}\) follow a multinomial likelihood: \[ + p(y \mid \theta) = \prod_{R \in \mathcal{R}} P(R)^{N_R}. +\]

+

Two prevalent models that make specific assumptions about the conditional and marginal probabilities are the basic local independence model and the simple learning model.

+
+

The basic local independence model (BLIM)

+

For the conditional distribution \(P(R \mid K)\), the BLIM assumes that given the knowledge state \(K\) of a student, responses are stochastically independent over problems (local independence). This implies that the response to each problem \(q\) only depends on the probabilities of a careless error \(\beta_q\) and a lucky guess \(\eta_q\). Accordingly, the probability of the response pattern \(R\) given the knowledge state \(K\) becomes \[ + P(R \mid K) = + \prod_{q \in K \setminus R} \beta_q + \cdot \prod_{q \in K \cap R} (1 - \beta_q) + \cdot \prod_{q \in R \setminus K} \eta_q + \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q). +\] For example, the probability that a student in state \(\{a,b\}\) solves the problems \(b\) and \(d\) is \[ + P(01010 \mid \{a,b\}) = + \beta_a \cdot (1 - \beta_b) \cdot (1 - \eta_c) \cdot \eta_d \cdot + (1 - \eta_e). +\] For the marginal distribution \(P(K)\), the BLIM does not imply any restrictions; for each knowledge state \(K\) it introduces a parameter \(\pi_K\) with \(\pi_K \in (0, 1)\) and \(\sum_K \pi_K = 1\). Thus, in total it has \(2 +\cdot |Q| + |\mathcal{K}| - 1\) free parameters.

+
+
+

The simple learning model (SLM)

+

Just like the BLIM, the SLM assumes local independence for the conditional distribution of responses given a knowledge state \(P(R \mid K)\). On top of that, the SLM establishes restrictions on the marginal distribution of knowledge states \(P(K)\). It requires \(\mathcal{K}\) to be a well-graded knowledge space, that is a knowledge space that is downgradable. For each problem, it introduces a solvability parameter \(g_q\), and the probability of a knowledge state \(K\) becomes \[ + P(K) = \prod_{q \in K} g_q \! + \prod_{q \in K^\mathcal{O}} \! (1 - g_q). +\] This is the probability of mastering all problems in \(K\) while not mastering any problems accessible from \(K\). Accessible here means: problems that are in the outer fringe \(K^\mathcal{O}\) of \(K\) as defined above. For example, with the well-graded knowledge space in Figure 3, the probability of being in state \(\{a,b\}\) is \[ + P(\{a,b\}) = g_a \cdot g_b \cdot (1 - g_c) \cdot (1 - g_d). +\] In total, the SLM has \(3 \cdot |Q|\) free parameters.

+ +
+
+
+
+

Fitting models to artificial responses

+ +
+

The BLIM as data-generating model

+
+

Simulating responses

+

Using the simulate function from the pks package, we generate response patterns for 2000 students using the knowledge structure in Figure 2 and the BLIM as data generating process (dgp). We assume a fixed distribution over the knowledge states and set all careless error and guessing probabilities to 0.1.

+
+
dgp <- function(K) {
+  nitem  <- ncol(K)
+  m0 <- list(
+       P.K = setNames(c(0.15, 0.1, 0.1, 0.08, 0.1, 0.12, 0.1, 0.1, 0.15),
+                      as.pattern(K)),
+      beta = rep(0.1, nitem),
+       eta = rep(0.1, nitem),
+         K = K,
+    ntotal = 2000
+  )
+  class(m0) <- "blim"
+  y <- simulate(m0, method = "multinomial", zeropad = TRUE)
+  attr(y, "dgp") <- m0
+  y
+}
+
+
+
N.R <- dgp(K)
+dput(N.R, file = "data/data_dgp_blim.R")
+
+
+
N.R <- dget("data/data_dgp_blim.R")
+datlist <- list(
+       R = as.binmat(N.R),                   # distinct response patterns
+       K = K,                                # knowledge states
+      Ko = getKFringe(K),                    # outer fringes
+       y = c(N.R),                           # by-pattern multinomial count
+    ycat = match(rep(names(N.R), N.R), names(N.R)), # by-subj response category
+  nstate = nrow(K),
+   nitem = ncol(K),
+       N = sum(N.R)
+)
+datlist$npattern <- nrow(datlist$R)
+datlist$y |> print()
+
+
00000 10000 01000 11000 00100 10100 01100 11100 00010 10010 01010 11010 00110 
+  207   141   152   173    23    36    37   179    27    33    33   160     8 
+10110 01110 11110 00001 10001 01001 11001 00101 10101 01101 11101 00011 10011 
+   22    17   166    18    25    14    33     3    17    19   145     5    10 
+01011 11011 00111 10111 01111 11111 
+    8    47     0    15    24   203 
+
+
datlist$ycat |> table()
+
+

+  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19  20 
+207 141 152 173  23  36  37 179  27  33  33 160   8  22  17 166  18  25  14  33 
+ 21  22  23  24  25  26  27  28  30  31  32 
+  3  17  19 145   5  10   8  47  15  24 203 
+
+
+

The list of data that is passed on to Stan includes two versions of the generated responses: y is a named vector that contains the multinomial counts \(N_R\), some may be zero, but its length is always npattern, which is the number of all possible distinct response patterns; ycat is a vector that contains the response category of each student as a position index in y, it must be a number between 1 and npattern. Whenever some response patterns were not observed, the response frequency vector y should be completed with zeros, as is achieved by the zeropad = TRUE setting above. In addition, we assume that every student provides a response to every problem as the models below do not account for missingness of single item responses (see de Chiusole et al. 2015).

+
+
+

Fitting the BLIM

+

In Stan, the BLIM can be written as follows:

+
+
functions {
+  matrix getPRK(vector beta, vector eta, matrix K_t, matrix K1m_t,
+                matrix R) {  // P(R|K)
+    return exp(
+        (1 - R) * diag_pre_multiply(log(beta), K_t)
+      + R * diag_pre_multiply(log1m(beta), K_t)
+      + R * diag_pre_multiply(log(eta), K1m_t)
+      + (1 - R) * diag_pre_multiply(log1m(eta), K1m_t)
+    );
+  }
+}
+data {
+  int<lower=1> N;                               // number of students
+  int<lower=1> nstate;                          // number of knowledge states
+  int<lower=1> nitem;                           // number of problems
+  int<lower=1> npattern;                        // number of distinct patterns
+  array[npattern] int<lower=0> y;               // response frequencies
+  array[N] int<lower=1, upper=npattern> ycat;   // response categories
+  matrix<lower=0, upper=1>[nstate, nitem] K;    // knowledge structure matrix
+  matrix<lower=0, upper=1>[npattern, nitem] R;  // response pattern matrix
+}
+transformed data {
+  matrix[nitem, nstate] K_t = K';               // cache transpose
+  matrix[nitem, nstate] K1m_t = (1 - K)';
+}
+parameters {
+  vector<lower=0, upper=1>[nitem] beta;         // error probabilities
+  vector<lower=0, upper=1 - beta>[nitem] eta;   // guess probabilities
+  simplex[nstate] pi;                           // state probabilities
+}
+model {
+  // Priors
+  beta ~ beta(2, 9);
+  eta ~ beta(2, 9);
+  // Likelihood
+  y ~ multinomial(getPRK(beta, eta, K_t, K1m_t, R) * pi);
+}
+generated quantities {
+  vector[npattern] PR = getPRK(beta, eta, K_t, K1m_t, R) * pi;  // P(R)
+  array[npattern] int yrep = multinomial_rng(PR, N);
+  vector[N] log_lik;
+  for (i in 1:N) {
+    log_lik[i] = categorical_lpmf(ycat[i] | PR);
+  }
+}
+
+

The conditional distribution \(P(R \mid K)\) is implemented as a separate getPRK function, which is called in the model block to build the multinomial likelihood. The weakly informative priors for the \(\beta_q\) and \(\eta_q\) parameters are beta distributions assuming low error rates. In addition, the error rates are restricted to satisfy \(\beta_q + \eta_q < 1\). This ensures that a correct response is more likely if the problem is mastered than if it is not mastered, and it prevents the case \(\beta_q = \eta_q = 0.5\) for all \(q +\in Q\), which would render the model nonidentifiable (Heller 2017). The prior for the \(\pi_K\) parameters is a diffuse Dirichlet distribution, as both high and low probabilities for the knowledge states are plausible a priori. In Stan, a \(\mathrm{dirichlet}(1, \dots, 1)\) distribution corresponds to a constant added to the log-posterior, so it can be omitted. If a more informative prior is desired, it can be added to the model block as pi ~ dirichlet(...); with a suitable row vector of constants.

+

In summary, the Bayesian BLIM can be written as: \[\begin{align*} + \beta_q &\sim \mathrm{beta}(2, 9), \quad \beta_q + \eta_q < 1\\ + \eta_q &\sim \mathrm{beta}(2, 9)\\ + \pi_K &\sim \mathrm{dirichlet}(\mathbf{\alpha}), + \quad \mathbf{\alpha} = (1, \dots, 1)\\ + y &\sim \mathrm{multinomial}(P(R)), + \quad y = (N_R)_{R \in \mathcal{R}}, + \quad \sum_{R \in \mathcal{R}} N_R = N\\ + P(R) &= \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K)\\ + P(R \mid K) &= + \prod_{q \in K \setminus R} \beta_q + \cdot \prod_{q \in K \cap R} (1 - \beta_q) + \cdot \prod_{q \in R \setminus K} \eta_q + \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q)\\ + P(K) &= \pi_K, \quad \pi_K \in (0, 1), + \quad \sum_{K \in \mathcal{K}} \pi_K = 1, +\end{align*}\] where \(R\) denotes the response pattern, \(N_R\) its frequency, and \(K\) a knowledge state.

+

Stan fits the model to the generated responses and draws posterior samples for its parameters:

+
+
## Code takes time to run; instead use saved output
+blim1 <- stan("stan/blim.stan", data = datlist, seed = 1634)
+saveRDS(blim1, "saved_fit/blim1.rds")
+
+
+
blim1 <- readRDS(file = "saved_fit/blim1.rds")
+print(blim1, pars = c("beta", "eta", "pi"), probs = c(.025, .975))
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+        mean se_mean   sd 2.5% 97.5% n_eff Rhat
+beta[1] 0.10       0 0.01 0.08  0.12  3961    1
+beta[2] 0.09       0 0.01 0.07  0.12  4203    1
+beta[3] 0.14       0 0.03 0.08  0.19  2737    1
+beta[4] 0.20       0 0.10 0.03  0.38  1314    1
+beta[5] 0.20       0 0.10 0.03  0.40  1415    1
+eta[1]  0.19       0 0.10 0.03  0.38  1667    1
+eta[2]  0.19       0 0.10 0.03  0.39  1848    1
+eta[3]  0.11       0 0.02 0.08  0.15  3641    1
+eta[4]  0.13       0 0.02 0.09  0.16  5122    1
+eta[5]  0.10       0 0.01 0.07  0.13  3737    1
+pi[1]   0.20       0 0.04 0.14  0.30  1388    1
+pi[2]   0.09       0 0.03 0.02  0.16  1757    1
+pi[3]   0.09       0 0.03 0.01  0.15  1762    1
+pi[4]   0.05       0 0.03 0.00  0.10  1360    1
+pi[5]   0.09       0 0.03 0.04  0.14  1479    1
+pi[6]   0.12       0 0.02 0.08  0.17  1726    1
+pi[7]   0.08       0 0.04 0.01  0.15  1450    1
+pi[8]   0.08       0 0.04 0.01  0.15  1462    1
+pi[9]   0.20       0 0.05 0.13  0.31  1250    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 17:54:04 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

The convergence diagnostics and effective sample sizes do not indicate problems for any of the parameters. Figure 4 shows the posterior distribution of the BLIM parameters and their true values.

+
+
+Code +
posteriorBlimPlot <- function(samples, npar = 5 + 5 + 9,
+                              N.R = NULL, dgp = FALSE) {
+  with(samples, {
+    boxplot(cbind(beta, eta, pi)[, npar:1], horizontal = TRUE,
+            border = "gray", axes = FALSE, xlab = "Parameter value")
+    rev(colMeans(cbind(beta, eta, pi))) |>
+      points(y = 1:npar, col = "darkblue")
+  })
+  points(rev(coef(attr(N.R, "dgp"))), 1:npar, pch = 4, col = "darkblue")
+  axis(1)
+  axis(2, at = 1:npar, rev(c(
+    expression(beta[a]),
+    expression(beta[b]),
+    expression(beta[c]),
+    expression(beta[d]),
+    expression(beta[e]),
+    expression(eta[a]),
+    expression(eta[b]),
+    expression(eta[c]),
+    expression(eta[d]),
+    expression(eta[e]),
+    expression(pi[symbol("\306")]),
+    expression(pi[a]),
+    expression(pi[b]),
+    expression(pi[ab]),
+    expression(pi[abc]),
+    expression(pi[abd]),
+    expression(pi[abcd]),
+    expression(pi[abce]),
+    expression(pi[Q])
+  )), las = 1)
+  legend("topright", if(dgp) "BLIM (dgp)" else "BLIM", bty = "n")
+  box()
+}
+par(mai = c(.6, .5, .02, .07),
+    mgp = c(2, .7, 0))
+posteriorBlimPlot(extract(blim1), N.R = N.R, dgp = TRUE)
+legend("bottomright", inset = c(-.14, -.02),
+       legend = c("posterior mean", "true"), pch = c(1, 4),
+       col = "darkblue", bty = "n", ncol = 2, xpd = NA)
+
+
+
+
+
+ +
+
+Figure 4: Box plot of the posterior distribution of BLIM parameters, their posterior mean, and their true value. +
+
+
+
+
+

Regarding parameter recovery, we obtain mixed results: On one hand, for about half of the parameters the 50% regions of the box plots overlap with the true values, which indicates successful recovery. On the other hand, systematic deviations from the true values suggest that there are certain tradeoffs between parameters due to nonidentifiability. As a consequence of the knowledge structure being a well-graded knowledge space, some parameters of the BLIM are not identifiable (Heller 2017; Stefanutti et al. 2012).

+
+
c(
+         space = pks:::is.knowledgespace(K),
+  downgradable = is.downgradable(K),
+       forward = names(which(is.forward.graded(K))),
+      backward = names(which(is.backward.graded(K)))
+)
+
+
       space downgradable     forward1     forward2    backward1    backward2 
+      "TRUE"       "TRUE"          "a"          "b"          "d"          "e" 
+
+
+

The knowledge structure (Figure 2) is forward graded in problems \(a\) and \(b\) (they can be added to any state and still yield a state) and backward graded in \(d\) and \(e\) (they can be removed from any state and still yield a state). Thus, the error and guess probabilities of these problems trade off with some of the state probabilities. The BLIM’s partial nonidentifiability, however, does not compromise its predictive performance as shown below (Figure 5).

+
+
+

Fitting the SLM

+

In Stan, the SLM can be written like this:

+
+
functions {
+  matrix getPRK(vector beta, vector eta, matrix K_t, matrix K1m_t,
+                matrix R) {
+    return exp(
+        (1 - R) * diag_pre_multiply(log(beta), K_t)
+      + R * diag_pre_multiply(log1m(beta), K_t)
+      + R * diag_pre_multiply(log(eta), K1m_t)
+      + (1 - R) * diag_pre_multiply(log1m(eta), K1m_t)
+    );
+  }
+  vector getPK(row_vector g, matrix K, matrix Ko, int nstate) {  // P(K)
+    vector[nstate] log_pi;
+    for (i in 1:nstate) {
+      log_pi[i] = sum(K[i, ] .* log(g)) + sum(Ko[i, ] .* log1m(g));
+    }
+    return exp(log_pi);
+  }
+}
+data {
+  int<lower=1> N;
+  int<lower=1> nstate;
+  int<lower=1> nitem;
+  int<lower=1> npattern;
+  array[npattern] int<lower=0> y;
+  array[N] int<lower=1, upper=npattern> ycat;
+  matrix<lower=0, upper=1>[nstate, nitem] K;
+  matrix<lower=0, upper=1>[nstate, nitem] Ko;  // outer fringes
+  matrix<lower=0, upper=1>[npattern, nitem] R;
+}
+transformed data {
+  matrix[nitem, nstate] K_t = K';
+  matrix[nitem, nstate] K1m_t = (1 - K)';
+}
+parameters {
+  vector<lower=0, upper=1>[nitem] beta;
+  vector<lower=0, upper=1 - beta>[nitem] eta;
+  row_vector<lower=0, upper=1>[nitem] g;
+}
+model {
+  beta ~ beta(2, 9);
+  eta ~ beta(2, 9);
+  g ~ beta(5, 1);
+  y ~ multinomial(getPRK(beta, eta, K_t, K1m_t, R) * getPK(g, K, Ko, nstate));
+}
+generated quantities {
+  vector[npattern] PR = getPRK(beta, eta, K_t, K1m_t, R)
+                      * getPK(g, K, Ko, nstate);
+  array[npattern] int yrep = multinomial_rng(PR, N);
+  vector[N] log_lik;
+  for (i in 1:N) {
+    log_lik[i] = categorical_lpmf(ycat[i] | PR);
+  }
+}
+
+

For the marginal state distribution \(P(K)\), the code includes the getPK function that is called in the model block. The solvability parameters \(g_q\) have a beta prior. Its parameters should be chosen to produce an a priori plausible marginal state distribution. The model-implied marginal distribution evaluated at the prior mean can be checked via:

+
+
getSlmPK(g = rep(5/6, 5), K = K, Ko = getKFringe(K)) |> print(digits = 2)
+
+
    ∅     a     b    ab   abc   abd  abcd  abce     Q 
+0.028 0.139 0.139 0.019 0.016 0.096 0.080 0.080 0.402 
+
+
+

In summary, the Bayesian SLM can be written as: \[\begin{align*} + \beta_q &\sim \mathrm{beta}(2, 9), \quad \beta_q + \eta_q < 1\\ + \eta_q &\sim \mathrm{beta}(2, 9)\\ + g_q &\sim \mathrm{beta}(5, 1)\\ + y &\sim \mathrm{multinomial}(P(R)), + \quad y = (N_R)_{R \in \mathcal{R}}, + \quad \sum_{R \in \mathcal{R}} N_R = N\\ + P(R) &= \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K)\\ + P(R \mid K) &= + \prod_{q \in K \setminus R} \beta_q + \cdot \prod_{q \in K \cap R} (1 - \beta_q) + \cdot \prod_{q \in R \setminus K} \eta_q + \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q)\\ + P(K) &= \prod_{q \in K} g_q \! \prod_{q \in K^\mathcal{O}} \! (1 - g_q), +\end{align*}\] where \(R\) denotes the response pattern, \(N_R\) its frequency, \(K\) a knowledge state, and \(K^\mathcal{O}\) its outer fringe.

+

Fitting in Stan yields:

+
+
+Code +
slm1 <- readRDS(file = "saved_fit/slm1.rds")
+print(slm1, pars = c("beta", "eta", "g"), probs = c(.025, .975))
+
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+        mean se_mean   sd 2.5% 97.5% n_eff Rhat
+beta[1] 0.13       0 0.01 0.10  0.16  4023    1
+beta[2] 0.12       0 0.02 0.09  0.15  4022    1
+beta[3] 0.14       0 0.03 0.09  0.19  3870    1
+beta[4] 0.26       0 0.11 0.05  0.44  2051    1
+beta[5] 0.27       0 0.12 0.05  0.46  2494    1
+eta[1]  0.11       0 0.05 0.02  0.22  3195    1
+eta[2]  0.11       0 0.05 0.02  0.23  2933    1
+eta[3]  0.12       0 0.02 0.08  0.15  3889    1
+eta[4]  0.12       0 0.02 0.09  0.16  4421    1
+eta[5]  0.10       0 0.01 0.07  0.12  4002    1
+g[1]    0.78       0 0.02 0.73  0.82  3274    1
+g[2]    0.77       0 0.02 0.72  0.81  3281    1
+g[3]    0.73       0 0.03 0.66  0.80  4011    1
+g[4]    0.72       0 0.13 0.51  0.98  2064    1
+g[5]    0.71       0 0.13 0.50  0.97  2225    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 17:55:47 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

Also for the SLM, the convergence diagnostics and effective sample sizes do not indicate problems for any of the parameters.

+
+
+

Fitting the saturated multinomial model (MNM)

+

The MNM serves as a reference for the two substantive models. It can be written like this:

+
+
data {
+  int<lower=1> N;
+  int<lower=1> npattern;
+  array[npattern] int<lower=0> y;
+  array[N] int<lower=1, upper=npattern> ycat;
+}
+parameters {
+  simplex[npattern] theta;
+}
+model {
+  y ~ multinomial(theta);
+}
+generated quantities {
+  array[npattern] int yrep = multinomial_rng(theta, N);
+  vector[N] log_lik;
+  for (i in 1:N) {
+    log_lik[i] = categorical_lpmf(ycat[i] | theta);
+  }
+}
+
+

The MNM has one parameter for each distinct response pattern, so in total \(2^{|Q|} - 1\) free parameters. We obtain the following abbreviated posterior summary:

+
+
+Code +
mnm1 <- readRDS(file = "saved_fit/mnm1.rds")
+print(mnm1, pars = c("theta[1]", "theta[2]", "theta[32]"),
+      probs = c(.025, .975))
+
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+          mean se_mean   sd 2.5% 97.5% n_eff Rhat
+theta[1]  0.10       0 0.01 0.09  0.12  7228    1
+theta[2]  0.07       0 0.01 0.06  0.08  5320    1
+theta[32] 0.10       0 0.01 0.09  0.11  6056    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 17:56:57 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

Again convergence diagnostics and effective sample sizes look good.

+
+
+

Posterior predictive checks

+

Figure 5 shows the BLIM-generated and predicted frequencies for each distinct response pattern. The BLIM achieves a close fit between observations and predictions, not much worse than the MNM. The SLM’s predictions are further off; this is especially visible for patterns with higher frequency.

+
+
+Code +
postPredPlot <- function(blim, slm, mnm, N.R, dgp = 1) {
+  par(mfrow = c(1, 3), omi = c(.4, .25, 0, .03), mai = c(.1, .1, .3, .02),
+      mgp = c(2, .7, 0))
+  blim$yrep |>
+    boxplot(border = "gray", horizontal = TRUE, axes = FALSE,
+            main = if(dgp == 1) "BLIM (dgp)" else "BLIM")
+  points(N.R, 1:32, pch = 4, col = "darkblue")
+  axis(1)
+  axis(2, 1:32, names(N.R), las = 1, tick = FALSE, line = -1,
+       outer = TRUE, cex.axis = 0.82)
+  box()
+  #
+  slm$yrep |>
+    boxplot(border = "gray", horizontal = TRUE, axes = FALSE,
+            main = if(dgp == 2) "SLM (dgp)" else "SLM")
+  points(N.R, 1:32, pch = 4, col = "darkblue")
+  axis(1)
+  axis(2, 1:32, labels = FALSE)
+  box()
+  #
+  mnm$yrep |>
+    boxplot(border = "gray", horizontal = TRUE, axes = FALSE, main = "MNM")
+  points(N.R, 1:32, pch = 4, col = "darkblue")
+  axis(1)
+  axis(2, 1:32, labels = FALSE)
+  box()
+  mtext(paste0("Frequency (N = ", sum(N.R), ")"),
+        side = 1, line = 1.5, outer = TRUE, cex = .8)
+}
+postPredPlot(extract(blim1), extract(slm1), extract(mnm1), N.R)
+
+
+
+
+
+ +
+
+Figure 5: Box plot of the posterior predictive distribution of the response pattern frequencies for all three models. Data (marked by crosses) were generated by the BLIM. +
+
+
+
+
+
+
+

LOO-CV model comparison

+

When each model computes the observation-wise log-likelihood in the generated quantities block, approximate leave-one-out cross-validation (LOO-CV, Vehtari et al. 2017) is available to compare the models with respect to their ability to predict out-of-sample observations. This ability is quantified by the model’s expected log predictive density (ELPD), which is defined as \[ + \int \log(p(\tilde{y} | y)) p(\tilde{y}) d\tilde{y}, +\] where \(p(\tilde{y} | y)\) is the posterior predictive distribution. The true data distribution \(p(\tilde{y})\) is unknown, so the LOO estimate of ELPD is \[ + \frac{1}{m} \sum_{i=1}^m \log p(y_i | y_{-i}), +\] where \(m\) is the number of observations and \(y_{-i}\) are the data without the \(i\)th data point. Actual LOO-CV would require fitting the model \(m\) times, the loo package avoids this by approximating the ELPD estimate via importance sampling. For the three models, we obtain the following summary:

+
+
+Code +
l1 <- loo(blim1)
+l2 <- loo(slm1)
+l3 <- loo(mnm1)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+  _[1:8] |> print(simplify = FALSE)
+
+
+
 model elpd_diff se_diff p_worse diag_diff diag_elpd elpd_loo se_elpd_loo
+  BLIM       0.0     0.0      NA                      -5853.5        38.6
+   MNM      -7.6     3.7    0.98                      -5861.2        38.4
+   SLM     -82.0    13.0    1.00                      -5935.5        36.6
+
+
+

The elpd_loo and se_elpd_loo columns show the mean ELPD values and their standard error. The model with the largest ELPD is on top. The elpd_diff and se_diff columns show the mean ELPD difference values and their standard error relative to the top model. The standard error of the differences is smaller than the sum of the individual standard errors because the ELPD values are correlated. The p_worse column contains the probability for each model of having lower ELPD than the top model based on a normal approximation. The diag_diff and diag_elpd columns may contain diagnostic information about the ELPD difference being small or about the approximation to LOO-CV being non-trustworthy. More explanation of the output can be found in the documentation via ?loo::loo_compare.

+

For none of the three models are there any diagnostic flags, so these columns are empty here. The model most successful at predicting new response frequencies is the BLIM, followed by the MNM. The SLM is the least successful of the three. Remember that all three models were estimated based on the BLIM-generated data.

+
+
+
+

The SLM as data-generating model

+
+

Simulating responses

+

We generate a new set of response frequencies from a rudimentary SLM. The solvability parameters \(g_q\) are all set to 0.8; this gives rise to a distribution over the knowledge states that is different to the one above, where the BLIM was the data generator. Otherwise, the settings are the same as above.

+
+
dgp <- function(K) {
+  nitem  <- ncol(K)
+  m0 <- list(
+      beta = rep(0.1, nitem),
+       eta = rep(0.1, nitem),
+         g = rep(0.8, nitem),
+         K = K,
+    ntotal = 2000
+  )
+  m0$P.K <- getSlmPK(g = m0$g, K = K, Ko = getKFringe(K))
+  class(m0) <- c("slm", "blim")
+  y <- simulate(m0, method = "multinomial", zeropad = TRUE)
+  attr(y, "dgp") <- m0
+  y
+}
+
+

We obtain the following response pattern frequencies:

+
+
+Code +
N.R <- dget("data/data_dgp_slm.R")
+datlist$ycat <- match(rep(names(N.R), N.R), names(N.R))
+datlist$N <- sum(N.R)
+datlist$y <- c(N.R) |> print()
+
+
+
00000 10000 01000 11000 00100 10100 01100 11100 00010 10010 01010 11010 00110 
+   83   207   190    86     8    27    22    69    17    40    45   133     5 
+10110 01110 11110 00001 10001 01001 11001 00101 10101 01101 11101 00011 10011 
+   19    17   178    11    16    24    39     5    20    14   149     0     5 
+01011 11011 00111 10111 01111 11111 
+   12    46     4    33    54   422 
+
+
+
+
+

SLM parameter recovery

+
+
+Code +
slm2 <- readRDS(file = "saved_fit/slm2.rds")
+print(slm2, pars = c("beta", "eta", "g"), probs = c(.025, .975))
+
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+        mean se_mean   sd 2.5% 97.5% n_eff Rhat
+beta[1] 0.10       0 0.01 0.08  0.13  4092    1
+beta[2] 0.08       0 0.01 0.07  0.10  4038    1
+beta[3] 0.10       0 0.01 0.07  0.13  3751    1
+beta[4] 0.17       0 0.07 0.04  0.28  1845    1
+beta[5] 0.17       0 0.07 0.04  0.29  2301    1
+eta[1]  0.11       0 0.05 0.02  0.22  1865    1
+eta[2]  0.11       0 0.05 0.02  0.21  1839    1
+eta[3]  0.09       0 0.01 0.06  0.12  3810    1
+eta[4]  0.14       0 0.02 0.10  0.17  4131    1
+eta[5]  0.10       0 0.01 0.07  0.12  3846    1
+g[1]    0.81       0 0.02 0.77  0.84  1956    1
+g[2]    0.79       0 0.02 0.76  0.82  2007    1
+g[3]    0.83       0 0.02 0.80  0.87  3271    1
+g[4]    0.86       0 0.08 0.71  0.99  1983    1
+g[5]    0.86       0 0.08 0.71  0.99  2365    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 17:58:13 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

Figure 6 shows the posterior distribution of SLM parameters and their true values. Again, the results are mixed. About half of the parameters are covered by their 50% posterior intervals as expected. But there are also systematic deviations from the true values visible as a tradeoff between the solvability and careless-error parameters for problems \(d\) and \(e\): Higher solvability goes along with more careless errors and vice versa.

+
+
+Code +
posteriorSlmPlot <- function(samples, npar = 5 + 5 + 5,
+                             N.R = NULL, dgp = FALSE) {
+  with(samples, {
+    boxplot(cbind(beta, eta, g)[, npar:1], horizontal = TRUE,
+            border = "gray", axes = FALSE, xlab = "Parameter value")
+    rev(colMeans(cbind(beta, eta, g))) |>
+      points(y = 1:npar, col = "darkblue")
+  })
+  points(rev(coef(attr(N.R, "dgp"))), 1:npar, pch = 4, col = "darkblue")
+  axis(1)
+  axis(2, at = 1:npar, rev(c(
+    expression(beta[a]),
+    expression(beta[b]),
+    expression(beta[c]),
+    expression(beta[d]),
+    expression(beta[e]),
+    expression(eta[a]),
+    expression(eta[b]),
+    expression(eta[c]),
+    expression(eta[d]),
+    expression(eta[e]),
+    expression(g[a]),
+    expression(g[b]),
+    expression(g[c]),
+    expression(g[d]),
+    expression(g[e])
+  )), las = 1)
+  legend("topright", if(dgp) "SLM (dgp)" else "SLM", bty = "n")
+  box()
+}
+par(mai = c(.6, .5, .02, .03),
+    mgp = c(2, .7, 0))
+posteriorSlmPlot(extract(slm2), N.R = N.R, dgp = TRUE)
+legend("bottomright", inset = c(-.14, -.02),
+       legend = c("posterior mean", "true"), pch = c(1, 4),
+       col = "darkblue", bty = "n", ncol = 2, xpd = NA)
+
+
+
+
+
+ +
+
+Figure 6: Box plot of the posterior distribution of SLM parameters, their posterior mean, and their true value. +
+
+
+
+
+
+
+

Posterior predictive checks

+

In addition to the SLM, the BLIM and the MNM were fitted to the new data. Figure 7 shows the SLM-generated and predicted frequencies for each distinct response pattern. All three models achieve a close fit between observations and predictions.

+
+
+Code +
postPredPlot(extract(blim2), extract(slm2), extract(mnm2), N.R, dgp = 2)
+
+
+
+
+
+ +
+
+Figure 7: Box plot of the posterior predictive distribution of the response pattern frequencies for all three models. Data (marked by crosses) were generated by the SLM. +
+
+
+
+
+
+
+

LOO-CV model comparison

+
+
+Code +
l1 <- loo(blim2)
+l2 <- loo(slm2)
+l3 <- loo(mnm2)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+  _[1:8] |> print(simplify = FALSE)
+
+
+
 model elpd_diff se_diff p_worse       diag_diff diag_elpd elpd_loo se_elpd_loo
+   SLM       0.0     0.0      NA                            -5617.6        45.8
+  BLIM      -1.9     0.5    1.00 |elpd_diff| < 4            -5619.4        45.7
+   MNM      -5.0     5.0    0.84                            -5622.6        45.6
+
+
+

Now that the SLM has generated the responses that were used for parameter estimation, it also turns out to be the model with the highest predictive accuracy, closely followed by the BLIM. The MNM is worst of the three. Considering the uncertainty as indicated by the se_diff and p_worse values, the differences are not large.

+ +
+
+
+
+

An empirical application

+ +
+

High school geometry problems: Responses and knowledge structure

+

Lakshminarayan (1995, 1996) presented the high school geometry problems shown in Figure 1 to 959 undergraduate students before and after a lesson. The data were split into a training and a held-out set; only the 800 responses in the training data set are reported. In this application, the posttest responses (Figure 8) will be analyzed. They are referred to by Doignon and Falmagne (1999, chap. 8). The response frequencies are:

+
+
+Code +
data(hsgeometry, package = "pks")
+dotchart(angles$N.R[, "posttest"], pch = 4,
+         main = "High school geometry problems",
+         xlab = "Response frequency (pre- [o] and posttest [x])")
+points(angles$N.R[, "pretest"], 1:32)
+mtext("Lakshminarayan (1995)", side = 3, line = 0.5)
+N.R <- angles$N.R[, "posttest"] |>
+  print()
+
+
+
00000 00001 00010 00011 00100 00101 00110 00111 01000 01001 01010 01011 01100 
+   12     0     1     0     3     0     4     0    14     1     9     0    24 
+01101 01110 01111 10000 10001 10010 10011 10100 10101 10110 10111 11000 11001 
+    2    15     2     5     0     2     0     7     0     7     3    23     5 
+11010 11011 11100 11101 11110 11111 
+   31    10   160    22   288   150 
+
+
+
+
+
+ +
+
+Figure 8: Response frequencies before and after a lesson on geometry. +
+
+
+
+
+

Figure 9 displays the knowledge structure proposed by Lakshminarayan (1995) based on the content of the problems.

+
+
+Code +
K <- angles$K
+rownames(K) <- as.pattern(K, useNames = TRUE)
+rownames(K)[rownames(K) %in% c("{}", "abcde")] <- c("\u2205", "Q")
+plot(relations::as.relation(is.subset(K)), attrs = myattrs, main = "")
+
+
+
+
+
+ +
+
+Figure 9: Graph of a knowledge structure with eleven knowledge states. +
+
+
+
+
+
+
+

Fitting no-guessing models in Stan

+

With an open-answer format like in this problem set, it is plausible to assume that guessing the correct response is impossible or very unlikely. There are two options to implement this in Stan: Option 1 is to keep the guessing parameters in the model together with strong priors that favor no guessing. Option 2 is to set all guessing parameters \(\eta_q\) to zero (as in Doignon and Falmagne 1999, chap. 8). For Option 2, the getPRK function has to be adapted, so a no-guessing BLIM can be written in Stan like this:

+
+
functions {
+  matrix getPRK(vector beta, matrix eta_filter, matrix K_t, matrix R) {
+    return exp(
+        (1 - R) * diag_pre_multiply(log(beta), K_t)
+      + R * diag_pre_multiply(log1m(beta), K_t)
+    ) .* eta_filter;
+  }
+}
+data {
+  int<lower=1> N;
+  int<lower=1> nstate;
+  int<lower=1> nitem;
+  int<lower=1> npattern;
+  array[npattern] int<lower=0> y;
+  array[N] int<lower=1, upper=npattern> ycat;
+  matrix<lower=0, upper=1>[nstate, nitem] K;
+  matrix<lower=0, upper=1>[npattern, nitem] R;
+}
+transformed data {  // or do it in R and pass as data: R %*% t(1 - K) == 0
+  matrix<lower=0>[npattern, nstate] eta_filter = R * (1 - K)';
+  for (i in 1:npattern) {                  // if item in R but not in K:
+    for (j in 1:nstate) {                  //   set P(R|K) to zero
+      eta_filter[i, j] = eta_filter[i, j] == 0;
+    }
+  }
+  matrix[nitem, nstate] K_t = K';
+}
+parameters {
+  vector<lower=0, upper=1>[nitem] beta;
+  simplex[nstate] pi;
+}
+model {
+  beta ~ beta(2, 9);
+  y ~ multinomial(getPRK(beta, eta_filter, K_t, R) * pi);
+}
+generated quantities {
+  vector[npattern] PR = getPRK(beta, eta_filter, K_t, R) * pi;
+  array[npattern] int yrep = multinomial_rng(PR, N);
+  vector[N] log_lik;
+  for (i in 1:N) {
+    log_lik[i] = categorical_lpmf(ycat[i] | PR);
+  }
+}
+
+

The above code assumes that when response patterns are missing, their frequencies are set to zero such that the frequency vector is always complete. An alternative strategy is to leave out missing response patterns: R only contains the observed patterns and y their frequencies; npattern \(< +2^{|Q|}\) is their number. Then, the likelihood in the model block should avoid the multinomial distribution statement (which requires a simplex as an argument). Instead, a custom likelihood can be coded like this:

+
+
transformed parameters {
+  vector[npattern] logPR = log(getPRK(beta, eta_filter, K_t, R) * pi);
+}
+model {
+  beta ~ beta(2, 9);
+  target += sum(y .* logPR);  // custom likelihood
+}
+
+
+

Posterior distributions

+

We pass the data to Stan and obtain posterior samples for the no-guessing version of the BLIM.

+
+
+Code +
blim3 <- readRDS(file = "saved_fit/blim3.rds")
+print(blim3, pars = c("beta", "pi"), probs = c(.025, .975))
+
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+        mean se_mean   sd 2.5% 97.5% n_eff Rhat
+beta[1] 0.04       0 0.01 0.02  0.07  2603    1
+beta[2] 0.04       0 0.01 0.02  0.05  3538    1
+beta[3] 0.09       0 0.02 0.06  0.12  2422    1
+beta[4] 0.16       0 0.03 0.11  0.21  3620    1
+beta[5] 0.20       0 0.12 0.03  0.47  1751    1
+pi[1]   0.02       0 0.00 0.01  0.02  6199    1
+pi[2]   0.01       0 0.00 0.00  0.01  3013    1
+pi[3]   0.02       0 0.00 0.01  0.03  4714    1
+pi[4]   0.01       0 0.01 0.00  0.02  2702    1
+pi[5]   0.00       0 0.00 0.00  0.01  3418    1
+pi[6]   0.02       0 0.01 0.01  0.04  4328    1
+pi[7]   0.16       0 0.02 0.11  0.20  3941    1
+pi[8]   0.01       0 0.01 0.00  0.04  2491    1
+pi[9]   0.02       0 0.01 0.00  0.04  2783    1
+pi[10]  0.42       0 0.06 0.26  0.51  1438    1
+pi[11]  0.31       0 0.06 0.24  0.46  1402    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 18:17:19 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

We also estimate the no-guessing version of the SLM.

+
+
+Code +
slm3 <- readRDS(file = "saved_fit/slm3.rds")
+print(slm3, pars = c("beta", "g"), probs = c(.025, .975))
+
+
+
Inference for Stan model: anon_model.
+4 chains, each with iter=2000; warmup=1000; thin=1; 
+post-warmup draws per chain=1000, total post-warmup draws=4000.
+
+        mean se_mean   sd 2.5% 97.5% n_eff Rhat
+beta[1] 0.03       0 0.01 0.01  0.06  2769    1
+beta[2] 0.04       0 0.01 0.02  0.05  3663    1
+beta[3] 0.09       0 0.02 0.06  0.12  2410    1
+beta[4] 0.17       0 0.03 0.12  0.23  2261    1
+beta[5] 0.31       0 0.16 0.05  0.62  1669    1
+g[1]    0.92       0 0.01 0.89  0.94  2728    1
+g[2]    0.98       0 0.01 0.97  0.99  4641    1
+g[3]    0.96       0 0.02 0.93  0.99  1999    1
+g[4]    0.80       0 0.03 0.75  0.86  2259    1
+g[5]    0.53       0 0.15 0.35  0.92  1302    1
+
+Samples were drawn using NUTS(diag_e) at Thu Oct  8 18:19:20 2026.
+For each parameter, n_eff is a crude measure of effective sample size,
+and Rhat is the potential scale reduction factor on split chains (at 
+convergence, Rhat=1).
+
+
+

Figure 10 displays the posterior distribution of the BLIM and SLM parameters. The pattern for the careless-error parameters \(\beta_q\) is similar for both models. In order to compare the results with maximum likelihood estimation (MLE), the models were fitted using the pks package. When MLE is repeated with different starting values, the estimates show high variability for those parameters with the highest posterior uncertainty, indicating tradeoffs due to nonidentifiability.

+
+
+Code +
posteriorBlimPlot <- function(samples, mlfit, npar = 5 + 11,
+                              N.R = NULL, dgp = FALSE) {
+  with(samples, {
+    boxplot(cbind(beta, pi)[, npar:1], horizontal = TRUE,
+            xlim = c(1, 16), ylim = 0:1, border = "gray", axes = FALSE)
+    for(m in mlfit)
+      rev(coef(m)[!grepl("^eta\\.", names(coef(m)))]) |>
+        points(y = jitter(1:npar), pch = 16, cex = 0.6, col = "tan2")
+    rev(colMeans(cbind(beta, pi))) |>
+      points(y = 1:npar, col = "darkblue")
+  })
+  axis(1)
+  axis(2, at = 1:npar, rev(c(
+    expression(beta[a]),
+    expression(beta[b]),
+    expression(beta[c]),
+    expression(beta[d]),
+    expression(beta[e]),
+    expression(pi[symbol("\306")]),
+    expression(pi[a]),
+    expression(pi[b]),
+    expression(pi[ab]),
+    expression(pi[ad]),
+    expression(pi[bc]),
+    expression(pi[abc]),
+    expression(pi[abd]),
+    expression(pi[bcd]),
+    expression(pi[abcd]),
+    expression(pi[Q])
+  )), las = 1)
+  legend("topright", if(dgp) "BLIM (dgp)" else "BLIM", bty = "n")
+  box()
+}
+posteriorSlmPlot <- function(samples, mlfit, npar = 5 + 5,
+                             N.R = NULL, dgp = FALSE) {
+  with(samples, {
+    boxplot(cbind(beta, g)[, npar:1], at = 1:npar + 6, horizontal = TRUE,
+            xlim = c(1, 16), ylim = 0:1, border = "gray", axes = FALSE)
+    for(m in mlfit)
+      rev(coef(m)[!grepl("^eta\\.", names(coef(m)))]) |>
+        points(y = jitter(1:npar) + 6, pch = 16, cex = 0.6, col = "tan2")
+    rev(colMeans(cbind(beta, g))) |>
+      points(y = 1:npar + 6, col = "darkblue")
+  })
+  axis(1)
+  axis(2, at = 1:npar + 6, rev(c(
+    expression(beta[a]),
+    expression(beta[b]),
+    expression(beta[c]),
+    expression(beta[d]),
+    expression(beta[e]),
+    expression(g[a]),
+    expression(g[b]),
+    expression(g[c]),
+    expression(g[d]),
+    expression(g[e])
+  )), las = 1)
+  legend("topright", if(dgp) "SLM (dgp)" else "SLM", bty = "n")
+  box()
+}
+blim3_ml <- readRDS(file = "saved_fit/blim3_ml.rds")
+slm3_ml <- readRDS(file = "saved_fit/slm3_ml.rds")
+par(mfrow = 1:2, mai = c(.4, .5, .02, .03), omi = c(.3, 0, 0, 0),
+    mgp = c(2, .7, 0))
+posteriorBlimPlot(extract(blim3), blim3_ml)
+posteriorSlmPlot(extract(slm3), slm3_ml)
+mtext("Parameter value", side = 1, outer = TRUE)
+legend("bottomright", inset = c(-.28, -.02), pch = c(1, 16),
+       legend = c("posterior mean", "MLE"), pt.cex = c(1, 0.6),
+       col = c("darkblue", "tan2"), bty = "n", ncol = 2, xpd = NA)
+
+
+
+
+
+ +
+
+Figure 10: Box plot of the posterior distribution of BLIM and SLM parameters, their posterior mean, and maximum likelihood estimates (ten replications with different starting values). +
+
+
+
+
+
+
+

Posterior predictive checks

+

Figure 11 shows the observed response frequencies and their predictions from the BLIM, SLM, and MNM. The BLIM predictions are almost as close to the observations as the MNM predictions. The SLM is further off, especially for response patterns with high frequency.

+
+
+Code +
postPredPlot(extract(blim3), extract(slm3), extract(mnm3), N.R, dgp = 0)
+
+
+
+
+
+ +
+
+Figure 11: Box plot of the posterior predictive distribution of the response pattern frequencies for all three models. Crosses denote the observed response frequencies. +
+
+
+
+
+
+
+

LOO-CV model comparison

+
+
+Code +
l1 <- loo(blim3)
+l2 <- loo(slm3)
+l3 <- loo(mnm3)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+  _[1:8] |> print(simplify = FALSE)
+
+
+
 model elpd_diff se_diff p_worse       diag_diff diag_elpd elpd_loo se_elpd_loo
+   MNM       0.0     0.0      NA                            -1651.1        35.8
+  BLIM      -3.3     7.0    0.68 |elpd_diff| < 4            -1654.4        37.1
+   SLM     -32.7    10.8    1.00                            -1683.7        38.1
+
+
+

According to LOO-CV, the BLIM is not much worse than the saturated multinomial model at predicting replication data. The uncertainty reflected by the standard error of this difference is high. The SLM is noticeably worse than the other two models.

+
+
+
+
+

Summary

+

In this case study, we introduced Bayesian versions of two prevalent models in knowledge structure theory, the basic local independence model and the simple learning model. We showed how to code these models in Stan, obtain posterior estimates, run diagnostic checks, and compare them using cross validation. For the two models illustrated here, there are classical methods available for estimating their parameters and evaluating their fit. We see the greatest benefit from adopting a Bayesian framework when we want to extend these models: (1) Bayesian methods allow us to implement models for which classical methods do not exist and would be difficult to obtain. (2) Because the prior distribution serves as a regularization mechanism, models with many parameters (as are common in KST) become feasible to estimate.

+
+
+

Computational details

+

The pks package version 0.8-0 (Heller and Wickelmaier 2013) was used to generate artificial responses, for data handling, for providing the high school geometry data, and for plotting knowledge structures leveraging facilities from relations (Meyer and Hornik 2026) and Rgraphviz (Hansen et al. 2025). Stan was interfaced from R using RStan version 2.32.7 (Stan Development Team 2025). The loo package version 2.10.1 (Vehtari et al. 2026) was used for running LOO-CV model comparison.

+
+
+ + + + +
+ + +

References

+
+de Chiusole, D., L. Stefanutti, P. Anselmi, and E. Robusto. 2015. “Modeling Missing Data in Knowledge Space Theory.” Psychological Methods 20 (4): 506–22. https://doi.org/10.1037/met0000050. +
+
+de la Torre, J. 2009. “DINA Model and Parameter Estimation: A Didactic.” Journal of Educational and Behavioral Statistics 34 (1): 115–30. https://doi.org/10.3102/1076998607309474. +
+
+Doignon, J.-P., and J.-C. Falmagne. 1999. Knowledge Spaces. Springer. +
+
+Falmagne, J.-C., and J.-P. Doignon. 1988a. “A Class of Stochastic Procedures for the Assessment of Knowledge.” British Journal of Mathematical and Statistical Psychology 41 (1): 1–23. https://doi.org/10.1111/j.2044-8317.1988.tb00884.x. +
+
+Falmagne, J.-C., and J.-P. Doignon. 1988b. “A Markovian Procedure for Assessing the State of a System.” Journal of Mathematical Psychology 32 (3): 232–58. https://doi.org/10.1016/0022-2496(88)90011-9. +
+
+Falmagne, J.-C., M. Koppen, M. Villano, J.-P. Doignon, and L. Johannesen. 1990. “Introduction to Knowledge Spaces: How to Build, Test, and Search Them.” Psychological Review 97 (2): 201–24. https://doi.org/10.1037/0033-295X.97.2.201. +
+
+Fischer, G. H., and I. W. Molenaar. 1995. Rasch Models: Foundations, Recent Developments, and Applications. Springer. +
+
+Hansen, K. D., J. Gentry, L. Long, et al. 2025. Rgraphviz: Provides Plotting Capabilities for R Graph Objects. https://doi.org/10.18129/B9.bioc.Rgraphviz. +
+
+Heller, J. 2017. “Identifiability in Probabilistic Knowledge Structures.” Journal of Mathematical Psychology 77: 46–57. https://doi.org/10.1016/j.jmp.2016.07.008. +
+
+Heller, J., and L. Stefanutti. 2024. Knowledge Structures: Recent Developments in Theory and Application. World Scientific. https://doi.org/10.1142/13519. +
+
+Heller, J., L. Stefanutti, P. Anselmi, and E. Robusto. 2015. “On the Link Between Cognitive Diagnostic Models and Knowledge Space Theory.” Psychometrika 80 (4): 995–1019. https://doi.org/https://doi.org/10.1007/s11336-015-9457-x. +
+
+Heller, J., L. Stefanutti, P. Anselmi, and E. Robusto. 2016. “Erratum to: On the Link Between Cognitive Diagnostic Models and Knowledge Space Theory.” Psychometrika 81 (1): 250–51. https://doi.org/10.1007/s11336-015-9494-5. +
+
+Heller, J., and F. Wickelmaier. 2013. “Minimum Discrepancy Estimation in Probabilistic Knowledge Structures.” Electronic Notes in Discrete Mathematics 42: 49–56. https://doi.org/10.1016/j.endm.2013.05.145. +
+
+Junker, B. W., and K. Sijtsma. 2001. “Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory.” Applied Psychological Measurement 25 (3): 258–72. https://doi.org/10.1177/01466210122032064. +
+
+Kambouri, M., M. Koppen, M. Villano, and J.-C. Falmagne. 1994. “Knowledge Assessment: Tapping Human Expertise by the QUERY Routine.” International Journal of Human-Computer Studies 40 (1): 119–51. https://doi.org/10.1006/ijhc.1994.1006. +
+
+Lakshminarayan, K. 1995. “Theoretical and Empirical Aspects of Some Stochastic Learning Models.” PhD thesis, University of California, Irvine. +
+
+Lakshminarayan, K. 1996. A Hybrid Latent Trait and Latent Class Model of Learning - Theoretical Details and Empirical Application. Technical report No. MBS 96-07. University of Califomia, Irvine. +
+
+Lord, F. M., and M. R. Novick. 1968. Statistical Theories of Mental Test Scores. Addison-Wesley. +
+
+Meyer, D., and K. Hornik. 2026. relations: Data Structures and Algorithms for Relations. https://doi.org/10.32614/CRAN.package.relations. +
+
+Noventa, S., J. Heller, and A. Kelava. 2024. “Toward a Unified Perspective on Assessment Models, Part I: Foundations of a Framework.” Journal of Mathematical Psychology 122 (102872): 1–20. https://doi.org/10.1016/j.jmp.2024.102872. +
+
+Noventa, S., J. Heller, T. F. Schaffland, and A. Kelava. 2026. “Toward a Unified Perspective on Assessment Models, Part III: Polytomous and Continuous Latent Variables.” Journal of Mathematical Psychology 129 (102993): 1–24. https://doi.org/10.1016/j.jmp.2026.102993. +
+
+Noventa, S., J. Heller, S. Ye, and A. Kelava. 2025. “Toward a Unified Perspective on Assessment Models, Part II: Dichotomous Latent Variables.” Journal of Mathematical Psychology 125 (102926): 1–25. https://doi.org/10.1016/j.jmp.2025.102926. +
+
+Rasch, G. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. Danish Institute for Educational Research. +
+
+Schrepp, M. 1999. “On the Empirical Construction of Implications Between Bi-Valued Test Items.” Mathematical Social Sciences 38 (3): 361–75. https://doi.org/10.1016/S0165-4896(99)00025-6. +
+
+Stan Development Team. 2025. RStan: The R Interface to Stan. https://mc-stan.org/. +
+
+Stefanutti, L., J. Heller, P. Anselmi, and E. Robusto. 2012. “Assessing the Local Identifiability of Probabilistic Knowledge Structures.” Behavior Research Methods 44 (4): 1197–211. https://doi.org/10.3758/s13428-012-0187-z. +
+
+Van Leeuwe, J. F. 1974. “Item Tree Analysis.” Nederlands Tijdschrift Voor de Psychologie En Haar Grensgebieden 29 (6): 475–83. +
+
+Vehtari, A., J. Gabry, M. Magnusson, et al. 2026. loo: Efficient Leave-One-Out Cross-Validation and WAIC for Bayesian Models. https://mc-stan.org/loo/. +
+
+Vehtari, A., A. Gelman, and J. Gabry. 2017. “Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC.” Statistics and Computing 27 (5): 1413–32. https://doi.org/10.1007/s11222-016-9696-4. +
+

Footnotes

+ +
    +
  1. Correspondence to: Florian Wickelmaier, Department of Psychology, University of Tübingen, Schleichstr. 4, 72076 Tübingen, Germany, wickelmaier@web.de. Portions of this material were presented at the Psychoco International Workshop on Psychometric Computing, February 5-6, 2026, Padua, Italy. We thank Bob Carpenter for his comments on an earlier version of this manuscript.↩︎

  2. +
+
+ + +
+ + + + + \ No newline at end of file