✒

LLM-as-Annotator

LREC 2026 · ACL Anthologypaper: Vidal-Gorène et al. 2026, LoResLM →
·
📍 · sentences, tokens · ⚠ disagreements ICL pool · v · ex ·
·

🧠
Mixture-of-Experts is ON. models will run in parallel — answers are voted per token, and contested tokens (⚠) are highlighted for your review.
💡
Single-model mode. Add a 2nd model in to compare per token (Mixture-of-Experts).

Task & label schema

edit fields…

JSON of the internal schema (fields with name/type/values/nullable/aggregator). Click outside to apply.

Models

🧠 OpenRouter supports MoE — you can pick several models. Single-model only. For Mixture-of-Experts (parallel models + vote), use OpenRouter.

API key

🔒 Tab-scoped storage. Keys live only in this browser tab's sessionStorage (one slot per provider). Sent as X-API-Key + X-LLM-Provider headers — never stored on the server.

Get a key → openrouter.ai/keys console.mistral.ai/api-keys platform.openai.com/api-keys ilaas.fr/services-inference

Load your own corpus

Press Enter, , or ; to add. You can also paste PER, LOC, ORG, WORK and press Enter to add them all at once.

Need lemmatization, morphology, or several tag columns? Use one of the POS / Lemma presets above, or open 📋 Task after loading for finer control.

ICL pool · v

Corrected sentences live here and are re-injected as few-shot examples on the next annotation run. The pool filters by (language, schema_hash) so cross-task contamination cannot happen.

📥 Import a PIE TSV (text_form/form/lemma/pos)

Sentences separated by blank lines. Header row (text_form form lemma pos) is optional and may be repeated before every sentence — duplicates are skipped, so the app's own TSV exports round-trip without manual cleanup. Requires an active POS+lemma schema.

💡 The companion paper reports that 5–50 tokens per example is generally enough — beyond that, returns diminish and the prompt grows costly.

# lang source preview
empty — correct a sentence and click 📥 to ICL.
⬇ Download .jsonl

Export corpus

All sentences in the corpus, with their current (possibly corrected) annotations.

⬇TSV round-trip with PIE baseline ⬇JSON schema-conformant ⬇CoNLL-U UD-standard ⬇JSONL fine-tune format

Apply to tokens

⚙ Advanced settings

ℹ About this app

LLM-as-annotator

This workbench treats a frontier LLM as an annotator. You pick a task (POS, lemma, NER, sentence classification, …), hand the model a few tokens or sentences, and it returns a structured JSON annotation that you can review, correct, and export.

In-context learning helps the prompt

The few-shot examples you validate are kept in a session-scoped ICL pool (filtered by language + schema). They are re-injected at the top of every subsequent prompt — the model imitates your conventions instead of having to be re-trained. You can pre-seed the pool by importing a PIE TSV in the ICL modal.

Data bootstrapping — any language

The loop is designed for low-resource scenarios. Even with zero training data, a pre-trained LLM can produce a usable first pass on Ancient Greek, Old Armenian, Old Georgian, Syriac, modern languages, or anything else. You correct the few contested tokens (highlighted by MoE voting), feed corrections back into the pool, and the next batch comes out cleaner.

When you have enough data, switch back to a classical model

Once you've collected a few hundred to a few thousand gold sentences this way, training a lightweight task-specific model (CRF, BiLSTM, fine-tuned BERT) is usually cheaper, faster, fully reproducible, and offline-friendly compared to running an LLM in production. The exports here (TSV / CoNLL-U / JSONL) are ready for that next step.

Companion paper: Vidal-Gorène et al., LoResLM 2026 — see the 📚 Cite pill.

💰 Session cost

Cost in USD is only reported by OpenRouter (the response includes the actual price billed to your key). Other providers report token counts only — cost stays at $0.00 here even if your account is billed.

model calls prompt tok completion tok cost
No calls yet — annotate a sentence to populate this log.

Help

Keyboard shortcuts

j / k focus next / previous token
e / ↵ edit focused token
1 – 9 (in editor) assign the i-th visible tag
x toggle selection of focused token
r re-annotate the focused sentence
↵ save token edit and advance to next disagreement
Esc close popup

Workflow

  1. Pick a sandbox example (sidebar) or paste your text.
  2. Choose a task preset (top bar → 📋 Task).
  3. Set OpenRouter API key (top bar → 🔑) and one or more models.
  4. Click ▶ Annotate all. Disagreement tokens turn amber.
  5. Click a token to fix it. Click 📥 to ICL to feed corrections back.
  6. Export TSV / JSON / CoNLL-U / JSONL when done.