When Knowledge-Tracing Rankings Depend on Available History
Earlier ASSISTments 2009 benchmark overview; not the matched-history experimentEarlier benchmark calculation pipeline
This 2026 methodological study examines whether knowledge-tracing rankings depend on dataset provenance and the amount and representation of learners' previous responses. In an ASSISTments 2009 source audit, 817 of 4,148 shared learners had different normalized ordered skill-response sequences between a corrected official release and a pinned processed mirror.
In a separate ASSISTments 2015 experiment, a nonoverlapping cohort of 4,000 learners provided 20,592 common evaluation responses from 582 evaluable test learners. With 20 prior events, Brier scores were 0.176674 for count-based logistic regression, 0.173885 for order-aware logistic regression and 0.175140 for a GRU. The GRU minus order-aware difference was +0.001255 (95% learner-cluster bootstrap interval [0.000075, 0.002426]). Lower Brier is better. The relative ranking depends on the comparator, not evidence of a causal effect of model architecture.
The earlier 2009-only benchmark found a GRU advantage over its PFA comparator. That remains a separate historical experiment using different model information. The current paper does not claim generalization to modern classrooms, other institutions or teaching effectiveness. Preprint research is not peer review.
817 / 4,1482009 sequences differ
20,592common 2015 targets
0.173885order-aware Brier
0.175140GRU Brier
Evidence: Official ASSISTments source provenance and frozen computational protocols are documented in the repository. No student-level raw records are hosted here. The arXiv identifier will be linked once verified.
Early Assessment Submission and Later Outcomes in OULAD
Full study story: question, data, design, result and interpretation limitData processing and analysis workflow
This study examines whether submitting at least one qualifying non-banked assessment by day 30 is associated with a more favorable final outcome among learners who are still eligible at that point. Using the OULAD CCC 2014J presentation, the day-30 landmark is fixed before exposure and outcome are defined. The analysis starts with 1,983 eligible learners and retains 1,820 complete supported cases: 1,624 had submitted by the landmark and 196 had not. Pre-exposure covariates are then used to construct inverse-probability weights for the comparison.
The normalized Hájek contrast is 0.3861, with a full-refit bootstrap interval of approximately 0.3242 to 0.4447; the overlap-weighted sensitivity estimate is 0.3261. Those numbers are reported together with the less comfortable diagnostics: extreme weights, an effective sample size of about 509.5 and residual imbalance remain. The study therefore supports an adjusted observational association, not the claim that making students submit earlier would cause better outcomes.
1,820analysis cases
1,624exposed
196control
0.3861Hájek contrast
Data and provenance: Open University Learning Analytics Dataset, OULAD / UCI 349, using the frozen CCC 2014J day 30 landmark design. The recorded analysis files preserve the cohort definition, weighting diagnostics, uncertainty procedure and source fingerprint.
Full study story: question, data, design, result and interpretation limitData processing and analysis workflow
This study tests one specific idea behind a misconception-aware tutor: does the wrong answer itself contain useful information for retrieving the evidence that should support a response? From the SciQ test split, 884 supported questions provide 884 support passages and 2,652 question–wrong-answer proxy cases. A fixed BM25 setup compares three queries for the same problem: the question alone, the question plus its observed wrong answer, and the question plus a shuffled wrong answer. The shuffled condition acts as a negative control for the simple effect of adding more lexical material.
Question-only retrieval achieves MRR 0.9472, compared with 0.9433 for observed wrong-answer expansion and 0.9447 for the shuffled control. The observed wrong-answer condition is 0.0039 below the baseline, and its difference from the shuffled control is small with a 95% interval spanning zero. In this benchmark, the wrong answer does not provide a distinct retrieval advantage. That conclusion is deliberately narrow: SciQ distractors are answer options, not validated learner misconceptions, and the experiment evaluates retrieval rather than learning gains from a tutor.
884eligible questions
2,652wrong-answer cases
0.9472baseline MRR
−0.0039wrong-answer Δ MRR
Data and provenance: SciQ test split from Allen Institute for AI, pinned to revision 2c94ad3… with byte-level SHA-256 verification. The empirical corpus uses 884 supported questions, and the shuffled condition is a deterministic lexical-expansion control.
Enrollment-Time Dropout Risk Model Audit on UCI 697
Full study story: question, data, design, result and interpretation limitData processing and analysis workflow
This study audits an enrollment-time dropout-risk model rather than presenting it as a deployment-ready predictor. UCI dataset 697 contains 4,424 historical student records. Variables that describe first- or second-semester performance are removed so the prediction boundary stays at enrollment, while protected attributes are kept out of the model and used only for auditing. The resulting model uses 20 predictors and is examined for discrimination, calibration, subgroup error patterns, threshold-dependent allocation and sensitivity to alternative data splits.
On the primary 885-record holdout, ROC-AUC is 0.8316 and Brier score is 0.1447. The gender audit has enough support under the study's stated rules, while international status, intersectional groups and special-needs comparisons do not; those cases are marked not evaluable rather than being interpreted as zero disparity. Repeated splits and bootstrap analyses add uncertainty information, but the evidence still belongs to one historical Portuguese higher-education dataset. The result is an audit record, not a fairness certificate or permission to deploy the model.
4,424records
20retained predictors
0.8316ROC-AUC
0.1447Brier score
Data and provenance: UCI Machine Learning Repository dataset 697, Predict Students' Dropout and Academic Success, DOI 10.24432/C5MC89. Each empirical run records a SHA-256 fingerprint of the normalized source table and target.
Of the 26 projects, most are synthetic or rule-based prototypes rather than validated educational interventions. Classroom Discourse Intelligence includes an empirical benchmark on the TalkMoves corpus.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
GenAI Learning Observatory asks a practical instrumentation question: when learners use generative AI, can an analysis layer distinguish verification, revision and reflection from simple answer adoption? The current repository uses synthetic prompt-event records grouped by learner and session. Transparent lexical rules tag interaction intent, and those events are converted into session-level verification, revision, reflection, adoption and intent-diversity measures.
The resulting agency-oriented index is deliberately presented as a design heuristic, not a psychological scale. Its weights are visible, every component can be traced back to the underlying synthetic events, and no real learner claim is made. The project is therefore most useful as a reproducible test bed for deciding what should be logged and validated before collecting consented learner–AI interaction data.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Teacher–AI Assessment Studio explores how an AI score can remain subordinate to auditable human review. Synthetic paired human and AI ordinal scores are compared using exact agreement, agreement within one point and quadratic kappa. A separate routing rule sends cases to review when disagreement is large or the simulated AI uncertainty crosses a declared threshold, so the reason for every handoff remains visible.
The prototype evaluates routing behavior rather than proving that either scorer is correct. Agreement is not accuracy against a gold standard, the uncertainty values are synthetic, and the review threshold has not been validated for real student work. Its value is the decision architecture: two judgments stay separate, uncertainty is explicit, and ambiguous cases do not disappear behind a single automated score.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Privacy-Preserving Learning Analytics demonstrates how model training can be organized while synthetic learner records remain in local client shards. Each client computes a local gradient, the update can be norm-clipped and perturbed with Gaussian noise, and the server aggregates the resulting updates with explicit client weights. A separate synthetic test set shows how those engineering choices affect predictive utility.
The repository demonstrates data-local computation, not a formal privacy guarantee. It does not yet implement privacy accounting, ε/δ reporting, secure aggregation or adversarial threat-model evaluation. That distinction is important: the project is a transparent federated-learning scaffold for studying utility and information flow before stronger privacy claims are attempted.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Multimodal Self-Regulation Lab tests a leakage-safe way to combine synthetic interaction, attention-proxy and self-report signals. All modalities share one held-out partition, and imputation and scaling are fitted only on the training rows before they are applied to the held-out data. The main classifier pipeline is compared with a reliability-weighted rank-fusion alternative so the effect of different fusion strategies can be inspected.
The project is about analysis design, not validated measurement of self-regulation or attention. Both predictors and outcomes are simulated, and proxy signals should not be interpreted as psychological constructs. Its strongest contribution is methodological discipline: the train–test boundary is explicit, preprocessing cannot borrow information from the full sample, and competing fusion approaches are evaluated on the same held-out cases.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Learning Design Process Mining turns synthetic instructional-design event logs into ordered traces that can be compared across projects. The pipeline reconstructs process variants, counts stage-to-stage transitions, estimates transition entropy and calculates a clearly defined repetition rate. These outputs make it possible to see where design workflows converge, branch or revisit earlier stages.
The analysis intentionally avoids equating repetition with inefficiency. Returning to an earlier stage may represent productive iteration rather than rework, and the bundled logs are synthetic. Authentic design-process data and defensible stage definitions would be needed before drawing conclusions about efficiency, expertise or organizational performance.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Hybrid Intelligence Lab asks when human and AI decisions might complement one another rather than simply compete. Synthetic cases contain a human decision, an AI decision and confidence values, allowing the system to measure overlapping versus complementary errors. A transparent routing policy keeps agreements, uses confidence gaps on disagreements and can default to the human when the gap is small.
The simulation is intentionally revealing about its own assumptions: confidence is generated partly from correctness, so strong hybrid performance is easier to obtain than it might be in practice. Threshold sweeps therefore show how the policy behaves under the simulator, not which policy a classroom should adopt. The repository is best read as an experimental harness for studying routing logic, deferral and error complementarity.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Explanation Faithfulness for AIED separates a convincing explanation from one that actually tracks a model decision. Synthetic cases provide an original prediction probability, probabilities after selected features are removed or retained, and feature sets from perturbed cases. From these inputs the repository calculates comprehensiveness, sufficiency gap and feature-set stability, while keeping the measures distinct before any combined summary is formed.
The current software accepts supplied perturbation outputs rather than intervening on a production model, and its combined faithfulness index is a hand-weighted heuristic. A stronger study would connect the harness to a real model, perform actual feature interventions and compare against appropriate random-feature controls. The repository therefore demonstrates an auditable evaluation framework, not a certification that an explanation is trustworthy.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Collaborative Reasoning Analytics examines observable structure in synthetic dialogue without pretending that surface patterns equal deep reasoning. Explicit rules label selected reasoning moves, adjacent turns are compared for lexical uptake, and speaker participation is summarized to show whether one participant dominates the conversation. Each quantity can be traced back to the turns that produced it.
These signals are descriptive proxies. Shared vocabulary does not prove conceptual uptake, balanced participation does not guarantee equitable reasoning, and rule-based move labels require human validation. The project is useful as a transparent coding and aggregation baseline that can later be compared with expert annotations on authentic collaborative-learning conversations.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Cognitive Offloading Analytics explores how several observable AI-use behaviors can be combined into an explicit offloading proxy. Copying similarity, revision, verification, delayed recall and confidence change are transformed in declared directions, combined with visible weights and normalized within the synthetic cohort. The pipeline then assigns heuristic bands and tests how much the result changes under alternative weighting scenarios.
The score is not a diagnosis of dependence on AI or a direct measure of cognition. Its weights, normalization and bands are design choices, and the synthetic cohort defines the scale. The important feature is sensitivity: the repository shows whether a conclusion remains stable when reasonable assumptions change instead of presenting one arbitrary weighting as ground truth.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Classroom Discourse Intelligence is the empirical benchmark in this section. It uses the TalkMoves corpus to classify teacher discourse moves, after removing duplicate transcript copies and grouping matching transcript text before the split. The final benchmark contains 565 transcript groups and 175,129 labeled teacher utterances; a TF–IDF logistic-regression model is evaluated against a majority-class baseline on 113 held-out groups.
On the held-out groups, logistic regression reaches macro-F1 0.5198 compared with 0.1152 for the majority baseline, while overall accuracy remains similar at about 0.678. That contrast matters because macro-F1 is more sensitive to minority move classes. The result is evidence about coded teacher-talk classification under this corpus and split; it is not a measure of teaching quality, pedagogical effectiveness or student learning.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Adaptive Socratic Tutor implements a transparent scaffolding policy around a supplied mastery estimate and attempt count. Instead of jumping directly to an answer, the policy moves from a focused question toward progressively more explicit hints and worked substeps. Each decision records which rule fired, making the boundary between light support and stronger intervention inspectable.
The repository does not estimate mastery itself and does not show that its hint sequence improves learning. The bundled cases are synthetic tests of policy behavior and edge conditions. Its purpose is to make a tutoring strategy explicit enough to critique and validate before connecting it to a real learner model or experimental study.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Instructor Insight Engine converts synthetic activity events into compact learner and class summaries for human review. Correct and attempted responses are aggregated by learner, accuracy is calculated from observed attempts, and a low-accuracy flag is allowed only after a minimum number of attempts. The same records are summarized at class level so the instructor can see both individual evidence and broader context.
The thresholds are deliberately simple and unvalidated. Time spent is not treated as learning, sparse activity is not converted into a confident label, and a flag is only a prompt for review. The repository is therefore a dashboard-logic prototype that emphasizes evidentiary sufficiency and interpretability rather than automated diagnosis.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Learner State Sequence Model provides a descriptive baseline for already-labeled learner-state sequences. It counts adjacent transitions, normalizes outgoing transition probabilities, calculates entropy for states with observed outgoing transitions and measures how often consecutive observations remain in the same state. When the sequence cannot support a quantity, the implementation leaves it undefined instead of inventing a value.
The model summarizes supplied labels; it does not infer a learner's hidden mental state. Any interpretation therefore depends on how those state labels were produced and validated. This makes the repository useful as a transparent first-order sequence analysis that can serve as a reference point before more complex hidden-state or neural models are introduced.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Multimodal Learning Analytics focuses on the data-engineering problem that comes before multimodal prediction. Supplied speech, gaze and click observations are kept within learner and session boundaries, assigned to fixed time windows and summarized with modality-appropriate rules: observed continuous features are averaged, clicks are counted, and each modality keeps an explicit availability flag.
The output is a clean, auditable window-level table rather than an inference about attention, emotion or learning. Missing modalities remain different from observed zeros, and sessions cannot be accidentally mixed. The repository is therefore a preprocessing baseline for later multimodal studies, where construct validity and predictive value would need to be established separately.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Constructive Alignment Auditor helps reviewers inspect whether learning outcomes, activities and assessments point in compatible directions. It uses an explicit Bloom-verb lexicon to identify supported cognitive demand and compares content terms with a transparent overlap measure. The evidence behind each inferred level is retained, while uncertain, missing or mismatched cases are surfaced for review.
The lexical rules are intentionally modest. A verb can be ambiguous, content overlap can miss semantic equivalence, and an automated mismatch does not prove poor instructional design. The tool is therefore best understood as a curriculum-review assistant that makes potential alignment problems easier to find while leaving the pedagogical judgment with an expert.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Assessment Design Lab turns a synthetic response matrix into familiar item and test diagnostics. For each item it reports the proportion correct, constructs respondent rest scores that exclude the focal item, forms upper and lower groups while preserving boundary ties, and calculates an upper-minus-lower discrimination signal when the data support it. Internal consistency is summarized separately with Cronbach's alpha.
None of these statistics is treated as proof of assessment validity. Alpha can be high for a weak instrument, item difficulty is sample dependent, and undefined quantities remain explicit when groups cannot be formed responsibly. The repository is designed to support item review by showing how each statistic was produced and where expert content judgment is still required.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Learning Experiment Platform keeps the key pieces of a small learning experiment connected from assignment through analysis. Participant IDs receive a seeded treatment or control assignment, baseline and follow-up outcomes remain attached to that assignment, and missing follow-up values are preserved as attrition. The analysis reports group summaries, raw mean differences, pooled-SD Cohen's d and bootstrap intervals for the raw difference.
The current handling of missing outcomes is a complete-case analysis within assigned groups, so the documentation does not present it as a full intention-to-treat analysis. Baseline imbalance, attrition and sparse groups are surfaced as review flags rather than hidden. The repository is a reproducible experimental scaffold, not a substitute for a defensible protocol, missing-data strategy or causal analysis plan.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Lesson Design Agent converts a supplied objective and teaching constraints into a structured lesson-plan scaffold using inspectable rules. It validates the design brief, identifies explicit objective verbs, selects a phase sequence that matches the supported cognitive demand and allocates the exact requested duration. Assessment-evidence prompts, accessibility considerations and missing-context flags remain attached to the generated plan.
The generated activities are proposals, not expert-approved instruction. Objective verbs can be ambiguous, contextual information may be incomplete and local feasibility cannot be inferred from text alone. The value of the repository is that its planning logic is visible and editable, making it easier for an educator to challenge the assumptions rather than accept an opaque generated lesson.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Learner Agency Simulator compares adaptive-support policies while giving synthetic learners explicit choices. A trajectory contains the current learner state, a learner action such as requesting or declining help, and a system action such as offering support or waiting. Across seeded runs the simulator records state occupancy, intervention burden, support acceptance, recovery and time to first mastery.
The results describe the consequences of the transition rules built into the simulator. They do not establish that a policy improves agency or learning in real learners, and conclusions can shift when assumptions about persistence, help-seeking or support acceptance are changed. Sensitivity scenarios are therefore part of the core analysis rather than an optional afterthought.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Competency Gap Intelligence connects role requirements with dated competency evidence while keeping uncertainty visible. Multiple evidence records can be aggregated with confidence weights, missing and stale evidence remain distinct from low proficiency, and confirmed positive gaps are prioritized using their declared importance. Prerequisite relationships then shape the order in which development needs should be addressed.
The resulting priorities are decision-support outputs, not validated measurements of human capability. Evidence levels, confidence values and importance weights all depend on organizational judgment. The repository's strength is auditability: a reviewer can trace a recommendation back to the evidence, see prerequisite blockers and test whether the ranking survives alternative importance assumptions.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Workplace Learning Recommender starts only after a development need has been confirmed. It combines the learner's current and target level with workplace constraints and structured resource metadata, filters out resources that are not feasible, and then scores the remaining options on declared dimensions such as gap coverage, task fit, quality and effort. Every score component remains visible.
The ranking estimates fit to the supplied metadata, not learning impact. Weak metadata, arbitrary weights or an incomplete catalog can change the recommendation, so the repository includes coverage and sensitivity checks rather than presenting the first ranking as definitive. Its purpose is transparent resource selection, not automated proof that a course will improve performance.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Training Transfer Analytics follows workplace application over time without collapsing it into business outcomes or contextual conditions. Synthetic self-report, manager and behavioral evidence can be combined with declared weights at baseline and follow-up points, while transfer conditions and optional performance outcomes remain separate. The analysis tracks change, persistence and disagreement among evidence sources.
The resulting trajectory does not show that training caused the observed change. The data are synthetic, the evidence weights are design choices and workplace conditions may explain persistence or decline. The repository demonstrates how longitudinal transfer evidence can be organized so that application, context and business outcomes remain analytically distinct.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Engagement Early-Warning System is built around a strict temporal rule: every predictor must be available by the prediction cutoff, and the outcome window begins afterward. Synthetic activity, task and assessment histories are converted into time-safe features, which feed a transparent logistic risk equation. The resulting probabilities can be placed into a capacity-limited review queue and examined with calibration and subgroup diagnostics.
The repository demonstrates leakage control and support-oriented reporting, not validated prediction of real learners. Its coefficients are synthetic and a high risk score is not a diagnosis or justification for punitive action. The central research contribution is architectural: the evidence boundary, prediction date and future outcome window are explicit enough to audit.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Curriculum Knowledge Graph represents courses, concepts, outcomes, assessments and resources as typed entities connected by provenance-labeled relationships. The implementation checks whether relation types are semantically allowed, traverses prerequisite paths, detects cycles and disconnected entities, and measures whether eligible outcomes or concepts have required relationships such as assessment coverage.
These are structural diagnostics, not a judgment that every relationship is pedagogically correct. A graph can be internally consistent and still encode a questionable prerequisite or weak assessment link. The repository therefore supports expert curriculum review by making dependencies and gaps queryable while preserving the source of each relationship.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Self-Regulated Learning Copilot models support as a decision about when to offer help, delay it or leave the learner in control. Its state includes supplied learning evidence, planning fields, progress, preferences, explicit help requests or refusals and prompt history. Transparent rules use that information to limit prompt burden, respect learner control and connect reflection to later strategy choices.
The bundled examples are synthetic, and operational labels such as plan completeness or struggle are not validated measures of self-regulation. The system is not intended to diagnose a learner or override an explicit refusal. Its value is to make autonomy-preserving support logic concrete enough to inspect before any real learner study.
Full project story: purpose, evidence, logic, outputs and interpretation limitData processing and decision workflow
Feedback Quality Evaluator breaks feedback review into separate dimensions instead of hiding everything inside one overall score. Supplied task goals, learner work, evidence and feedback text are examined for properties such as alignment, specificity, actionability and grounding, while missing evidence remains explicitly unevaluable. Optional external judgments can be compared with the rule-based output using exact agreement and Cohen's kappa where the statistic is defined.
The lexical rules are a baseline for validation and error analysis, not an authority on disciplinary accuracy or feedback quality. Agreement between two raters or systems is also not the same as correctness. The implementation makes those limits visible by reporting degenerate cases, such as constant labels where exact agreement can be perfect while kappa is undefined.