Decoding the Machine’s Mind: A Systematic Study of LLM-Generated Competency Questions

TL;DR

In the world of Knowledge Engineering, Competency Questions (CQs) are the blueprints of an ontology. This paper introduces CompCQ, a rigorous framework to audit how LLMs "think" when generating these blueprints. By testing 5 models—ranging from the 1-trillion parameter KimiK2 to GPT-4 and Llama 3—the researchers found that while closed models are clearer, open models trade readability for complexity. Crucially, they prove that you shouldn't rely on just one model if you want a complete set of requirements.

Background Positioning

Traditionally, ontology engineers sat with domain experts for weeks to draft CQs. Generative AI has turned this bottleneck into a "push-button" task. However, is a Llama-generated question as good as a GPT-generated one? This paper moves beyond simple "feasibility" (can it do it?) to "characterization" (how does it do it differently across domains?).

Problem & Motivation: The Black Box of Automation

The transition from manual pattern-based generation to LLM prompt-engineering has been swift but unguided. We don't know the Inductive Bias of these models in the context of Knowledge Engineering. Is a model making the ontology too complex? Is it missing key requirements? Prior work relied on subjective human evaluation, which doesn't scale. The authors noticed that "as the domain complexity increases, models behave wildly differently," necessitating a quantitative yardstick like CompCQ.

Methodology: The CompCQ Framework

The researchers didn't just look at word counts. They decomposed CQs into three specific dimensions of complexity:

1. Requirement Complexity (c1): Counts the "ontological primitives" (classes, properties, filters) implied by the question.
2. Linguistic Complexity (c2): Analyzes POS tagging and noun-chunk density.
3. Syntactic Complexity (c3): Measures the depth of the dependency tree.

Model Architecture and Strategy

The study utilized a zero-shot prompting strategy across two requirement formats: User Stories (narrative-driven) and Use Cases (structured flow).


Experiments & Results: The "Model-Domain Interplay"

The results reveal a fascinating split in the LLM landscape:

• The Concise Generalists: Gemini 2.5 Pro and GPT-4 produce the most readable and concise CQs. They stick closely to the text but might miss the "edges" of the domain.
• The Verbose Specialists: KimiK2 and Llama models often generate CQs with high syntactic complexity—sometimes reaching a US grade level of 18 (Post-grad) for a single question.
• The Divergence Gap: In broad domains like Cultural Heritage, different models generated almost 0% semantic overlap. This means GPT and Kimi effectively "saw" different parts of the same problem.

Table 2 above highlights how domain complexity (like PDTO) drives up syntactic and linguistic scores across all models.

Key Insight: Convergence vs. Novelty

In narrow, technical tasks (e.g., finding a national park based on weather), models converge on the same CQs. But in creative domains, they diverge. The study's Shannon Entropy analysis shows that GPT's outputs are often tightly clustered, while Gemini and Kimi show higher internal diversity but are less consistent.

Critical Analysis & Conclusion

Takeaway

The core contribution of this work is the debunking of the "one model fits all" myth in Knowledge Engineering. To build a robust ontology, engineers should use an ensemble of LLMs:

1. Use Gemini/GPT to establish a readable "Basics" set.
2. Use KimiK2 to explore complex, diverse "Edge Cases."
3. Apply Human-in-the-loop to prune and unify.

Limitations & Future Work

While CompCQ provides a massive leap in quantitative auditing, it still relies on other LLMs (like Gemini) for some feature extractions (e.g., relevance). Future research needs to investigate how these "expert-proxy" LLMs might introduce their own biases into the evaluation framework. Additionally, exploring how these CQs translate directly into SPARQL queries or OWL axioms is the logical next step.

Final Thought: LLMs are no longer just tools; they are "stakeholders" with specific linguistic personalities that we must learn to manage.