Exam.Ninja · Editorial policy
Methodology and editorial policy
How the questions on Exam.Ninja are specified, drafted, verified, validated, reviewed and corrected, where AI fits into that, and who is responsible for them. It applies to every question bank on the site, including DipIMC.Ninja and MSRA.Ninja.
Who is responsible for the questions
Each question bank has a named editor. The editor decides what is published, answers for every question in the bank, and has the final say on corrections.
- DipIMC.Ninja (the RCSEd Diploma in Immediate Medical Care): Leigh Rogers, consultant paramedic and lecturer. About Leigh.
- MSRA.Ninja (the Multi-Specialty Recruitment Assessment): Dr Rory Irwin.
AI-assisted, editor-led: how the pipeline works
Exam.Ninja uses AI inside a structured, multi-stage authoring and verification pipeline. It is not a chatbot writing questions on request. Generation is constrained by the exam's published blueprint and a formal authoring specification, grounded in named primary sources, and passed through independent verification, deterministic validation and human editorial oversight before a question reaches a candidate. Each stage has a different job, and a question that fails any of them does not move on.
- Blueprint-led specification. Work starts from coverage, not from a prompt. The bank is measured against every unit of the exam's published curriculum and domain structure, and new items are specified where coverage is thin: the unit, the domain, the difficulty band and the decision the question must test.
- Source-grounded drafting. Items are drafted with AI assistance against the current UK source hierarchy for that topic: national guidelines and consensus statements first, then legislation, then standard texts at edition, chapter and page. The draft must carry its citations, with an anchoring passage wherever the source allows, so every claim in the explanation is tied to a source a candidate can open.
- Independent verification. A separate verification pass re-opens every cited source, confirms the page numbers, sections and links, checks that the keyed answer follows from the cited text, and looks for a second arguable answer. Where a newer guideline and an older text disagree, the item is aligned to current guidance and the explanation says so.
- Deterministic validation. Structural rules that do not depend on any model are then enforced in the database: option counts for the exam, a keyed answer inside them, a domain belonging to that exam, references in a standard form, and for a station a complete decision tree in which every path ends and every option is scored and explained.
- Editorial accountability. The bank's named editor sets the specification, owns the source hierarchy and answers for every published item. Publication is a recorded, versioned event, never an automatic side effect of generation.
- Human-in-the-loop revision. After publication, candidates' flags and question reviewers' audits feed a revision loop. Reviewers can work through an item with an AI assistant that sees the item, the flags and its audit history, but never who raised them. The assistant can only propose a revision; the proposal is checked against the same validation rules and the item's current version, and a named human reviewer decides whether it is applied. Every applied change increments the version number and is written to an audit log.
Restraint under uncertainty. The pipeline is designed to withhold rather than overstate. An item whose sources conflict, whose answer is arguable, or which fails verification is rewritten or held back, not published with a caveat. A published item that is later shown to be wrong is corrected or withdrawn, and the change is logged.
Human judgement stays in charge. AI accelerates drafting, verification and review; it does not decide what is true. The editor and reviewers do, with the primary sources open. This page describes the methodology rather than the underlying models or infrastructure.
How a question is written
- To the real exam's format. Questions follow the format the exam itself uses: for the DipIMC, single best answer questions with five options and timed, branching OSPE-style stations of 8 and 16 minutes; for the MSRA, clinical problem solving questions, extended matching questions and professional dilemmas (ranking and choose three).
- To the published syllabus. Every question is tagged to one unit of the exam's published curriculum or specification (for the DipIMC, phase 1 of the UK Pre-hospital Emergency Medicine curriculum, which the exam is blueprinted against) and to one domain, so coverage can be checked and gaps filled.
- One best answer, explained. Each question has a single keyed answer and an explanation of why it is the best answer. Distractors are meant to be plausible, not silly.
- Current UK practice. Questions are written to current UK guidance, because that is what the exams test.
- Our own questions. We do not publish questions from any exam paper. Where a question is adapted from openly licensed material, a credit line says so under the explanation.
How answers are sourced
Every explanation ends with "Where this comes from": the sources the question was written from, precise enough to check.
- Guidelines and consensus statements (for example Resuscitation Council UK, NICE, JRCALC and the Faculty of Pre-Hospital Care), named with the section and linked where they are published online, opening at the right page or section where we can.
- Textbooks, named with edition, chapter and page.
- Legislation, named with the Act and section and linked to legislation.gov.uk.
- Papers, where a guideline rests on one.
Where an older textbook and a newer guideline disagree, the explanation says so and follows the current guideline. Ten Second Triage replacing the triage sieve in 2023 is one example.
Checks before a question is published
Each question is checked independently before it is published: the check re-opens every cited source, confirms page numbers and links, and looks for a second arguable answer. Before a question is published it is checked against a set of automated rules: the right number of options for the exam, a keyed answer that is one of them, a domain that belongs to that exam, and references recorded in a standard form. A station must also be a complete path: every decision leads to another step or to the end, and every option carries feedback and a score. Anything the rules report is fixed before publication.
Review after publication
- Anyone can flag a question. Every question in the app has a "Flag this" control for an error, an outdated answer, a typo, or a question that is too easy or too hard. Flags go to the bank's editor and reviewers.
- Question reviewers audit the bank. Named reviewers read questions against their sources and record an outcome for each at its current version: passed, returned for rewriting, or revised.
- Every change is versioned. A correction increases the question's version number, which is shown beneath it in the app, and is written to a log of who changed what and when.
Corrections and withdrawals
A question found to be wrong is corrected or withdrawn, not quietly left in place. A withdrawn question stops being served but is kept on record, so past attempts still make sense. If you think something is wrong, flag it in the app or contact us.
Keeping up to date
Guidelines change and so do exam arrangements. Pages about an exam's dates, fees and format say when they were last checked against the organising body's own pages, and each bank shows when its content was last updated. Where the organising body's current rules differ from anything here, the organising body is right.
Independence
- Exam.Ninja is independent. It is not issued, accredited or endorsed by the Royal College of Surgeons of Edinburgh, its Faculty of Pre-Hospital Care, NHS England or any other body that sets or runs an exam covered here. Exam names are used only to describe what each bank helps you revise for, and no College marks are used.
- There is no advertising, no sponsorship and no tracking. DipIMC.Ninja is free. Some banks are a subscription; paying for one never changes what a question says.
Methodology FAQ
Does Exam.Ninja use AI to write questions?
Yes, as one stage of a larger pipeline. Items are drafted with AI assistance to a blueprint-led specification and against named primary sources, then independently verified against those sources, validated by deterministic rules and held to account by a named human editor.
Can AI publish or change a question on its own?
No. Publication is a recorded editorial step. After publication, an AI assistant can only propose a revision; the proposal must pass the same validation rules against the item's current version, and a named human reviewer decides whether it is applied. Every change is versioned and logged.
What happens when sources conflict?
Current UK national guidance takes precedence over older texts, and the explanation says where they differ. An item whose answer remains arguable, or whose sources cannot be reconciled, is rewritten or held back rather than published with a caveat.
Does this page name model vendors or infrastructure?
No. It describes the methodology and the standards that shape each question, not the underlying models or infrastructure.
Not clinical guidance
Exam.Ninja is for exam revision. In clinical practice, follow your own organisation's guidelines.