Evaluating LLMs Through the Lens of Core Competency: A New Standard for AGI

TL;DR

The explosion of Large Language Models (LLMs) has outpaced our ability to measure them. This survey from the Harbin Institute of Technology argues that we need to stop treating tasks as isolated metrics and instead evaluate LLMs through four Core Competencies: Knowledge, Reasoning, Reliability, and Safety. By shifting the focus to these internal "cognitive" pillars, we can better understand whether a model is truly intelligent or just a sophisticated pattern matcher.

The "Saturation" Crisis in NLP

For years, benchmarks like GLUE and SuperGLUE were the gold standard. However, the sheer scale of LLMs has led to a phenomenon called Emergence, where models suddenly gain abilities (like coding or complex reasoning) that smaller models lack. Traditional tasks are now too easy, and the industry is plagued by "training set leakage," where models memorize test answers. The authors suggest that evaluation should mirror corporate recruitment: testing the underlying competency rather than just the specific output.

Methodology: The Four Pillars of Intelligence

The survey restructures the chaotic landscape of 540+ tasks into a logical hierarchy.

1. Knowledge (Linguistic & World)

Knowledge is the foundation. The authors split this into:

• Linguistic Competency: Grammar, semantics, and pragmatics (tested by BLiMP or MetaphorBoolean).
• World Knowledge: General facts and specialized domain knowledge (tested by LexGLUE for legal or WikiFact for general facts).

2. Reasoning: The Engine of AGI

Reasoning is where the "intelligence" of an LLM is truly tested. The survey breaks this down into six sophisticated sub-parts:

• Causal Reasoning: Understanding cause-and-effect (COPA).
• Inductive Reasoning: Deriving general rules from specific examples.
• Analogical Reasoning: The logic behind In-Context Learning (SAT Analogies).

Table 1: Examples of Knowledge Competency benchmarks categorized by linguistic and world knowledge.

3. Reliability: Can We Trust the Machine?

Reliability is arguably the biggest bottleneck for commercial deployment. The survey focuses on:

• Hallucinations: Distinguishing between intrinsic (ignoring context) and extrinsic (making up facts about the world).
• Calibration: Does the model "know what it doesn't know"? A reliable model should verbally express uncertainty when it is likely to be wrong.

4. Safety: Ethical Guardrails

Safety evaluation is subjective and evolving. It covers:

• Harmfulness: Detecting violence, crime, and unauthorized expertise.
• Social Bias: Measuring if the model treats demographic groups differently (CrowS-Pairs).

Table 2: Key Reasoning datasets mapped against specific reasoning competencies.

Experiments and Results: The Current SOTA

The survey identifies that while models like GPT-4 show "sparks of AGI," they are still brittle.

• Knowledge: LLMs have largely "solved" linguistic knowledge, but world knowledge requires constant updates to avoid staleness.
• Reasoning: Chain-of-Thought (CoT) has significantly boosted deductive performance, but inductive and multi-hop reasoning remain challenging.
• Reliability vs. Instruction Fine-Tuning: Interestingly, the paper notes that Reinforcement Learning from Human Feedback (RLHF) can actually worsen a model's calibration, making it more confident but less honest.

Critical Insight & Future Outlook

The most profound takeaway is that LLM evaluation is moving toward process-based assessment. We aren't just looking at the final answer anymore; we are looking at the Chain-of-Thought, the uncertainty probability, and the ethical alignment.

Future Directions to Watch:

1. Planning: Moving from "thinking" to "acting" in environments.
2. Coding: Not just writing snippets, but software engineering and self-evolution.
3. Sentiment and Theory of Mind: Can models truly understand the emotional subtext of human communication?

Conclusion

This survey serves as a vital compass for researchers lost in the sea of LLM benchmarks. By focusing on Core Competencies, we can move toward building models that are not just capable, but also reliable and safe enough to integrate into the fabric of society.

Limitations

As the authors admit, safety and ethics are highly culture-dependent. What is "safe" in one society may be "censorship" in another. The competency framework must remain flexible enough to adapt to these shifting human values.