Decoding the Robotic Mind: Using Machine Learning to Assess Problem-Solving in the Classroom

TL;DR

Educational Robotics (ER) offers a rich, open-ended learning environment, but assessing it objectively is a major challenge for educators. This paper introduces an automated system that logs students' programming behaviors with Lego Mindstorms and uses a Mixed Machine Learning approach (K-Means + SVM) to identify three distinct problem-solving styles. The system not only predicts success with high accuracy but also provides a "virtual sensor" for teachers to detect which students are struggling in real-time.

The Assessment Gap: Beyond Subjective Observation

In a typical robotics classroom, students learn through "bricolage"—a process of trial, error, and physical feedback. While this is pedagogically powerful, it is a nightmare to grade. Teachers often rely on subjective observations which are prone to bias and impossible to scale. Prior works have attempted to track student code, but they often ignore the physical interaction with the robot. The authors of this study argue that to truly understand learning, we must capture the process, not just the final product.

Methodology: The "Mixed" Intelligence Approach

The core innovation lies in the data pipeline. Instead of just looking at the final code, the researchers modified the Lego Mindstorms EV3 software to log every single change a student made during two tasks:

1. Exercise A: Moving a motor a precise distance (1m).
2. Exercise B: Using an ultrasonic sensor to stop before an obstacle.

1. Feature Engineering

The researchers defined 12 specific indicators, such as the number of motors changed, blocks deleted, and "Delta" parameters (how much a value was tweaked from one test to the next).

2. The Mixed Pipeline

• Unsupervised Stage: K-Means clustering was applied to the programming sequences to find natural groupings of behavior.
• Supervised Stage: The output of these clusters was fed into four classifiers: Logistic Regression (LR), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Random Forest (RF).

Figure 1: The proposed system loop: from classroom activity to data logging, ML processing, and finally teacher feedback.

Three Styles of Problem Solving

Through the data mining process, three "personalities" of learners emerged:

• The Mathematical Planner: These students perform fewer trials and make almost no changes to their parameters. They calculate the result first. High success rate, but they may struggle if their initial formula is wrong.
• The Tinkerer with Refining: These students use a heuristic approach. They test the robot, see it’s a bit off, and make small, logical tweaks. This "steady incremental" style was highly successful.
• The Tinkerer with High Changes: These students are "lost in the woods." They change parameters drastically between tests (e.g., jumping from power level 10 to 100). This behavior showed a strong negative correlation with performance, indicating a failure to understand the robot’s feedback.

Figure 2: Example of visual Lego code being transformed into a structured log for analysis.

Results: SVM Takes the Lead

The experiment involved 197 Italian students across 56 groups. The results were clear: The SVM (Support Vector Machine) using the Mixed Approach outperformed all other combinations.

• Key Finding: The number of trials had a negative correlation (PCC = -0.48) with success in simple tasks, suggesting that "blind" tinkering is less effective than targeted refining.
• Accuracy: By analyzing the percentage of sequences belonging to specific clusters, the ML model could accurately predict whether a team would fail or succeed before the educator even made a final assessment.

Figure 3: Comparative performance of different ML models across the two exercises.

Deep Insight: The Future of the "Virtual Sensor"

This paper marks a shift from "Assessment of Learning" (grading) to "Assessment for Learning" (guiding). By identifying the "Tinkerer with High Changes" early, a teacher can intervene before the student becomes frustrated.

Limitations & Future Work: While powerful, the "Mixed Approach" currently lacks stability—adding new data might shift the K-Means clusters, requiring the model to be retrained. Future iterations will likely look into more stable, cross-domain features that can work across different robotics platforms (Lego, Arduino, etc.).

Final Takeaway: In the future, the classroom will be equipped with digital "dashboards" that don't just show if a student is right or wrong, but how they are thinking. This study provides the mathematical foundation for that reality.