Question:

Based on these annotations. Please analyse the accuracy of each and every LLM per task type in a separate table. Please also produce a table summarising the performance of all LLMs per task type and per Example. Please note that "Judgment" and "Judgement" should be considered the same in counting the number of correct and incorrect tasks. If there is any discrepancy among the number or type of tasks across models, please report them. Please note that all models should be tested on the same (number of) tasks.

Answer:

I’ll treat “Judgment” and “Judgement” as the same label, inspect every annotation file in the zip, and check whether each model has the same task coverage before calculating accuracy.

I found 129 annotated task judgments across the 27 DOCX files. Coverage is balanced by model: each model has 43 judged tasks, with the same task counts per example.

Thought for 2m 19s

Done. I treated “Judgment” and “Judgement” as the same field, and counted any “Partially Incorrect” / “Incorrect?” as Incorrect.

You can download the full workbook here:

llm_annotation_accuracy_analysis.xlsx

SummaryCoverage notesGPT-5.4-Thinking DeepGemini 3 - ThinkingQwen 3.6 Plus - ThinkingRaw extracted judgments

Coverage check

No discrepancy was found in the number of tasks across models after normalization.

I normalized these task-type variants for counting: “Identifying event order”, “Identifying event orders”, and “Event ordering” were counted as Identifying event orders. “Question type” was treated as equivalent to “Task type”.

GPT-5.4-Thinking Deep — accuracy by task type

Gemini 3 - Thinking — accuracy by task type

Qwen 3.6 Plus - Thinking — accuracy by task type

Summary performance by task type

Summary performance by example

Key takeaways

Gemini has the best overall accuracy at 55.8%, followed by Qwen at 48.8%, then GPT-5.4 at 46.5%.

Gemini is strongest on event identification/order and composition, but weakest on LTS and abstraction. Qwen is best on abstraction among the three, though still low at 30.0%. GPT-5.4 is comparatively strongest on traces and identifying events, but weak on abstraction and composition.

Question:

Please extract these into a Word and a LaTex file.

Answer:

Summary of Annotation Tables

This document summarises the annotation-derived performance tables. The analysis treats “Judgment” and “Judgement” as the same field and counts partially incorrect judgements as incorrect. Task-type variants were normalised, including variants of event ordering.

Coverage and Normalisation

Accuracy by Model and Task Type

GPT-5.4-Thinking Deep

Gemini 3 - Thinking

Qwen 3.6 Plus - Thinking

Cross-Model Summary by Task Type

Cross-Model Summary by Example

Task Count Coverage Checks

Coverage by task type

Coverage by example

Interpretive Summary

• Gemini 3 - Thinking had the highest overall accuracy at 55.8%, followed by Qwen 3.6 Plus - Thinking at 48.8% and GPT-5.4-Thinking Deep at 46.5%.

• All models performed well on basic MSC semantic tasks, especially identifying events and event orders.

• Abstraction was the most difficult task category across all models, with only 20.0% aggregate accuracy.

• LTS reasoning was also a recurring weakness, with 25.0% aggregate accuracy.

• Composition was model-dependent: Gemini performed strongly at 71.4%, while GPT-5.4 and Qwen both scored 35.7%.