CELPIP scoring accuracy: practicecelpip.ca vs ChatGPT

ChatGPT does well on weak answers and badly on strong ones. On speaking answers CELPIP scored 9 to 12, ChatGPT landed within one level 14% of the time. Its average grade for them was 7.3, where CELPIP's average was 10.5. Our grader averaged 9.4 and landed within one level 74% of the time.

On answers CELPIP scored 0 to 5, ChatGPT was slightly ahead, 99% against our 93%. Writing follows the same pattern with a smaller gap: at levels 9 to 12, 54% for ChatGPT and 90% for us. Our previous grader, which ran until October 13, 2026, scored 59% of speaking answers within one level, so the changes on this page took speaking from 59% to 81%.

On price, yes. ChatGPT Plus costs US$20 a month, about CA$28, and ChatGPT Pro costs US$100 or US$200, about CA$140 or CA$280. Premium on practicecelpip.ca is CA$49.99 a month, or CA$14.99 a week.

What the money buys is different. On the official answers on this page, our grader landed within one level of CELPIP's score on 81% of speaking answers and 94% of writing answers, and gave the same level on all 3 runs for 69% of speaking answers. ChatGPT managed 58%, 77% and 45%.

We tested ChatGPT the way students use it, in an ordinary chat, so we can't say which plan's model answered. Paying for Plus or Pro buys more messages and newer models. It doesn't add a grader tested against official CELPIP answers, a CELPIP question bank or the exam timer.

We tested 141 language models, most of them on a handful of answers first. The 24 that held up for speaking and 13 for writing graded the whole test set, and each dot below is one of them. On speaking they ranged from 59% to 83% within one level on the same prompt.

We chose on the average distance from CELPIP's score over three runs, so the model we use is not the top dot on every measure. We keep its name to ourselves.

Speaking is graded from a transcript, so the transcript has to keep what the candidate said. We ran 7 speech-to-text models over 97 official recordings and compared each with CELPIP's own transcript. The transcriber we used until October got 13.4% of words wrong. The one we use now gets 6.7%.

The old transcriber also tidied speech. CELPIP's transcripts hold 176 repeated words, like "the the". Our current transcriber keeps 173 of them, and the old one kept 36. It also dropped false starts and fixed small grammar slips, so the grader was reading cleaner English than the candidate spoke.

We switched transcribers on October 13, 2026. On the same answers our grader's exact matches went from 39% to 45%, and its average grade moved from 0.20 of a level above CELPIP's to 0.16 below.

Fine-tuning takes an existing AI model and trains it further on worked examples until it does one job. Here the job was to read an answer and return a CELPIP level. We expected it to win. General models have never been taught the CELPIP scale and drift toward its middle, and a model trained on graded answers learns the scale from the answers themselves.

We trained a small open model on 143 official CELPIP answers, each with the level qualified CELPIP raters gave it. To keep the test fair we trained 3 copies, each on two thirds of the answers, and graded every answer with the copy that had never seen it.

It reached 62% within one level on speaking and 80% on writing, against 81% and 94% for our grader. The gap was widest on strong speakers: on answers CELPIP scored 9 to 12 it landed within one level 52% of the time, against 74% for our grader.

The limit was data. Those 143 answers are spread over 94 combinations of task and level, and 46 of those combinations had a single example. With so little to learn from, the model settled on a few safe grades: 93% of its 429 grades were a 5, an 8 or an 11. General models arrive already knowing English and how language is assessed, and a small model trained on this little data had to learn the whole scale from it.