Automated Descriptive Grading: Beyond Keyword Matching in Technical Education

TL;DR

Evaluating comprehensive, descriptive answers—those ranging from 500 to 1000 words—remains a bottleneck in online education. This paper proposes a score recommendation system that moves beyond simple string matching by ensemble-learning five distinct NLP similarity metrics. Tested on over 35,000 engineering exam records, the system provides a robust framework for automating assessment in "professional" courses where language is direct and objective.

Context & Positioning

In the landscape of Automated Short Answer Scoring (ASAS), most solutions are optimized for one-sentence responses. This work positions itself as a bridge between simple ASAS and complex essay grading. It focuses on Professional Courses (Applied Sciences/Engineering), capitalizing on the lack of metaphorical or subjective language to apply rigorous linguistic similarity measures.

The Core Motivation: Why Existing Metrics Fail Alone

The authors argue that no single metric can capture the multifaceted nature of a student's answer:

• Lexical Metrics (LCS) catch verbatim copying but miss synonyms.
• Knowledge-Based Metrics (Wu & Palmer) understand word relationships but ignore sentence structure.
• Statistical Metrics (LSA/Cosine) capture the "vibe" but might miss specific technical keywords.

Their insight was to treat these metrics not as competitors, but as features in a machine learning classifier.

Methodology: The Ensemble Architecture

The system follows a pipeline: Pre-processing (Lemmatization, Stop-word removal) → Feature Extraction (5 Baselines) → Classification (Logistic Regression).

The Five Pillars of Similarity:

1. Wu and Palmer: Uses WordNet-style hierarchies to score noun similarity.
2. LCS (Longest Common Substring): Measures consecutive word overlaps.
3. LSA (Latent Semantic Analysis): Uses SVD to find hidden thematic structures.
4. Cosine Similarity: Measures the angle between one-hot encoded vectors.
5. PMI-IR: Uses statistical co-occurrence to determine word relatedness.

Fig 1: The high-level workflow from student input to the final recommended score.

Experimental Insights

The research utilized a massive dataset of 35,240 records from engineering courses.

Key Performance Data:

Fig 2: Comparison of different machine learning heads. While K-Star performed slightly better in accuracy, Logistic Regression remained a more interpretable choice for score recommendation.

One critical finding was the 1-Marker Accuracy (66.44%). The system is significantly more effective at grading short, factual points than complex 3-mark descriptive syntheses, where academic nuance is higher.

Critical Analysis & Future Directions

While the system is a strong step toward automation, the authors identify a "Semantic Trap": the Negation Problem. A student writing "The system is stable" and "The system is not stable" would receive a high similarity score because the word overlap is nearly identical.

Limitations:

• Dataset Bias: The system was trained on a dataset where teachers primarily gave full marks or zeros, leading to a "U-shaped" bias in recommendations.
• Subjectivity: The "ground truth" (human graders) is inherently subjective, injecting noise into the training labels.

Future Outlook: The inclusion of Sentiment Polarity to detect "not" or "never" and the use of subject-specific corpora (Inverse Document Frequency) could significantly sharpen the system's precision.

Conclusion

This work demonstrates that for professional, objective-led courses, we don't need "Magic AI" to start automating grading. Regulated, multi-factor NLP similarity combined with standard regressors provides a transparent and "good enough" recommendation engine to assist human educators.