UCAT Scoring System

[ucatboy]

This is how FunTimes replied to the question if they felt they got all correct for the 900s
> Post-UCAT Discussion 2019

"Yes for QR and AR. My calculations in QR all gave me answers which were an answer option."

The most astounding score for me isn't the AR and QR personally but the verbal reasoning instead - a 890 is equal to 39/40 which is insane for one of the most time pressured and difficult sections which prep companies don't seem to be able to fully replicate...I could only dream of achieving 750 in the final never find 890 haha.

 

[CatzDreaming]

Do we know definitively if percentile will be calculated with, or without SJT?

Most likely without SJT
I believe they'll include percentiles for SJT separately

 

[KM7]

I understand the UCAT creates 3 'versions' of the test and that each section's score is 300-900 based on a raw mark.

I have heard speculation that some questions contribute more to the scaled score than others. Can someone clarify this?

EX: 2 people both do Test A and both get 30/44 (disregard trial Qs), but they got different questions correct and incorrect. Do they get the same scaled score /900 or does it depend on the difficulty of questions they answered correctly?

Thanks in advance...

 

[TMG]

I understand the UCAT creates 3 'versions' of the test and that each section's score is 300-900 based on a raw mark.

I have heard speculation that some questions contribute more to the scaled score than others. Can someone clarify this?

EX: 2 people both do Test A and both get 30/44 (disregard trial Qs), but they got different questions correct and incorrect. Do they get the same scaled score /900 or does it depend on the difficulty of questions they answered correctly?

Thanks in advance...

Every question on the VR, QR and AR is wroth exactly the same amount of marks. The only section where the number of marks you get for each question is different is in DM, where drag and drop questions are 2 marks instead of 1.
Confirmed by the following table:

UCAT Scoring | UCAT ANZ Consortium

www.ucat.edu.au

In your example, both people who got 30/44 will get the same scaled score out of 900 and will not depend on 'difficulty' of their correct answers. The 3 different tests for the UCAT are designed to be of equal difficulty they really are as seen by the normal distribution for each type of test. Even if one test was of higher difficulty than another test, they will not account for this as their intention is to make them of equal difficulty.

Hope this helps.

 

[A1]

EX: 2 people both do Test A and both get 30/44 (disregard trial Qs), but they got different questions correct and incorrect. Do they get the same scaled score /900 or does it depend on the difficulty of questions they answered correctly?

We are certain it is the latter. It can be proved mathematically as follows: Take QR with 36 questions minus say 6 unmarked trial Qs for a round 30 marked Qs. If of equal marks they'd be 20 marks each and the QR marks must go 300, 320, 340... in steps of 20. But we have seen QR marks like 650, 660, 670, 680 in steps of 10.

(Similar argument for 4 unmarked Qs : average 600/32= 18.75 per Q, due to rounding QR marks would go *mostly* in steps of 20).

 

[ucatboy]

Every question on the VR, QR and AR is wroth exactly the same amount of marks. The only section where the number of marks you get for each question is different is in DM, where drag and drop questions are 2 marks instead of 1.
Confirmed by the following table:

UCAT Scoring | UCAT ANZ Consortium

www.ucat.edu.au

In your example, both people who got 30/44 will get the same scaled score out of 900 and will not depend on 'difficulty' of their correct answers. The 3 different tests for the UCAT are designed to be of equal difficulty they really are as seen by the normal distribution for each type of test. Even if one test was of higher difficulty than another test, they will not account for this as their intention is to make them of equal difficulty.

Hope this helps.

I wouldn't be so quick to jump conclusions - notice the word "conversion" from "raw marks" to "scale scores". Scaled, in my experience (and probably yours too - does VCE have "scaled scores"?) very rarely means a directly proportional/equal translation of marks.

The UCAT Consortium has, to my knowledge, NEVER confirmed that this is the case - in fact, they very rarely put "raw marks" and "scale scores" in the same sentence, which suggests that they have some algorithm/method of calculating test-takers' scaled scores.

Refer to the following document: https://www.ucat.ac.uk/media/1306/ukcat-2018-technical-report-summary.pdf

I would like to draw your attention to the following two tables:





By your logic, since every mark is equal, and the total number of marks up for grabs in VR is 600 (900 − 300), then each of the 40 questions would be 600 / 40 = 15 marks each. By your description, the mean of 22.14/40 for VR Form 1 would be equal to ((22.14 ÷ 40) × 600) + 300 = 632.1. But refer to the second table, and the mean is instead 570.26. Therefore each question cannot be worth an equal number of marks and there is something else at play here.

Furthermore, the mean raw mark for test-takers of VR 1 is 22.14, less than the mean raw mark for test-takers of VR 3 (22.18), yet the average scale score of VR1 is ever-so-slightly higher than VR 3 (570.26 versus 569.92).

I hope this helped KM7! The main takeaway here is that no, questions are not worth the same number of marks, but that should be a secondary concern as you should try to attempt every single question on the exam regardless of how many marks it's worth.

 

[Some guy]

I wouldn't be so quick to jump conclusions - notice the word "conversion" from "raw marks" to "scale scores". Scaled, in my experience (and probably yours too - does VCE have "scaled scores"?) very rarely means a diirectly proportional/equal translation of marks.

The UCAT Consortium has, to my knowledge, NEVER confirmed that this is the case - in fact, they very rarely put "raw marks" and "scale scores" in the same sentence, which suggests that they have some algorithm/method of calculating test-takers' scaled scores.

Refer to the following document: https://www.ucat.ac.uk/media/1306/ukcat-2018-technical-report-summary.pdf

I would like to draw your attention to the following two tables:

View attachment 3383

View attachment 3381

By your logic, since every mark is equal, and the total number of marks up for grabs in VR is 600 (900-300), then each of the 40 questions would be 600 / 40 = 15 marks each. By your description, the mean of 22.14/40 for VR Form 1 would be equal to ((22.14 ÷ 40) × 600) + 300 = 632.1. But refer to the second table, and the mean is instead 570.26. Therefore each question cannot be worth an equal number of marks and there is something else at play here.

Furthermore, the mean raw mark for test-takers of VR 1 is 22.14, less than the mean raw mark for test-takers of VR3 (22.18), yet the average scale score of VR1 is higher (albeit marginally) than VR 3 (570.26 versus 569.92).

I hope this helped KM7! The main takeaway here is that no, questions are not worth the same number of marks, but that should be a secondary concern as you should try to attempt every single question on the exam regardless of how many marks it's worth.

In passing the above would also make a fine QR question.

 

[ucatboy]

I would also like to add that the average test-taker score being 570 instead of the theoretical 630 makes intuitive sense and provides further support for the hypothesis of harder questions = more marks.

If the average person is barely able to get 50% of the questions in the subtest right, it's pretty unlikely they'll only be attempting the hardest questions right? Chances are they're doing the easiest questions in the subtest and leaving it at that. Hence, the 21-22 questions they got right out of 40 are more likely to be of the easier or lower-valued variety, which fits in neatly with the lower scale score average of ~570 (less than 50%).

Expanding upon this, it shows that there are pretty significant differences in the value of some questions compared to others. Let's, for the sake of argument, assume that the 22 questions gotten right on average by test-takers are the same and of the easiest variety, while the remaining 18 are equally harder.

The first 22 questions are on average, only worth 270 ÷ 22 = 12.27 marks. In contrast, the last 18 are worth 330 ÷ 18 = 18.33 marks, a startling 50% increase. But this is the average over 22 and 18 questions respectively - I doubt the 1st and 22nd question are both 12.27 marks. On the contrary, it could be something like 10 marks for the easiest question in the subtest and 14 marks for the 22nd easiest question for the subtest. Apply the same logic to the hardest 18 questions, and the hardest question in the subtest could easily exceed 20 marks in value. Of course, this is all conjecture and pure speculation, but it should hopefully reassure everyone that the extra time they spend on a particularly difficult question will be rewarded accordingly rather than wasted.

If you guys can make any sort of sense of my incoherent rambling with numbers and figures, you deserve a 900 in QR tbh

 

[KM7]

Ok - thanks for the input everyone. If this is so, would the process/strategy of prioritising the easier questions first not be as ideal as it seems on the surface (i.e. if the harder ones you *may* not get to are potentially worth up to double towards your scaled score)?

 

[Fili]

Imho, it's probably still best to do the easier questions first rather than last. At the moment, all this "harder questions are worth more" is just hypothetical and it doesn't mean that you'll also get them right if you focus om them.

In the time span it takes you to do 1 hard question, you could have done 2 to 3 easier questions which you're more likely to get right since they're easier. It's not worth it to risk going for a hard question, getting it wrong and not having any points under your belt at all. Whereas if you go for an easier question first, you're more likely to have points under your belt + more time to go for the harder ones.

Personally, I also like to go for the easier questions first since I'm less likely to stress early on in the section. Imagine the stress of not being confident with your answers and not having answered that many at all since you went for all the hard ones, realizing all the easy questions you couldn't do due to lack of time.

 

[KM7]

Imho, it's probably still best to do the easier questions first rather than last. At the moment, all this "harder questions are worth more" is just hypothetical and it doesn't mean that you'll also get them right if you focus om them.

In the time span it takes you to do 1 hard question, you could have done 2 to 3 easier questions which you're more likely to get right since they're easier. It's not worth it to risk going for a hard question, getting it wrong and not having any points under your belt at all. Whereas if you go for an easier question first, you're more likely to have points under your belt + more time to go for the harder ones.

Personally, I also like to go for the easier questions first since I'm less likely to stress early on in the section. Imagine the stress of not being confident with your answers and not having answered that many at all since you went for all the hard ones, realizing all the easy questions you couldn't do due to lack of time.

Lots of good reasons there, thanks!

 

[DrDrLMG!]

Having not done the UCAT, I'm not sure on this, but is it obvious which questions are the 'hard' ones and which are the 'easy' ones? There was variable question worth seen in UMAT, too, but it was impossible to know which were more likely to have a higher weighting (for want of a better word), and, in fact, it seemed likely that the weighting wasn't even finalised until the exam had been sat and everyone had answered all the questions.

I guess they must be pre-determined in UCAT, given you get your scores immediately. But my questions is: how likely is it that you could pick a 'hard' from an 'easy' question (given what one person finds easy others might find hard) in the UCAT?

(PS. This whole conversation gives me JOY and is one of the (largely pointless but very fascinating!) things I used to love talking about with regard to UMAT *heart eyes*)

 

[KM7]

Having not done the UCAT, I'm not sure on this, but is it obvious which questions are the 'hard' ones and which are the 'easy' ones? There was variable question worth seen in UMAT, too, but it was impossible to know which were more likely to have a higher weighting (for want of a better word), and, in fact, it seemed likely that the weighting wasn't even finalised until the exam had been sat and everyone had answered all the questions.

I guess they must be pre-determined in UCAT, given you get your scores immediately. But my questions is: how likely is it that you could pick a 'hard' from an 'easy' question (given what one person finds easy others might find hard) in the UCAT?

(PS. This whole conversation gives me JOY and is one of the (largely pointless but very fascinating!) things I used to love talking about with regard to UMAT *heart eyes*)

I see your point in that difficulty varies amongst all the test takers. Another big thing is that with much stricter time limits in the UCAT, it makes it even more difficult to allow yourself time to see whether a question is going to be 'hard' or not. In VR and DM, usually the size/complexity of the stimulus can indicate the difficulty of the question, but in my preparation so far, I have prioritised completing as many questions as I can in as little time (therefore attempting the 'easier' ones first). In AR, if a pattern doesn't click within 30sec I tend to move on. So, in short, with most questions, you can distinguish harder questions from easier ones.

Although, when I see a hard question unit, for example on MedEntry with regard to QR, it is almost entirely based on the size of the stimulus, except if some of the questions are able to be completed in ~20sec (e.g. % of A compared to B). I find myself skipping 3-4 units of 4 Q's in QR and averaging about 25/36 (~97%ile on that platform). I seem to always leave these units until last and normally miss out 5-6 Q's that are probably more difficult than others - but this triaging is what helps me get a better raw score, so I think it's best not to let this discussion change my strategy. This does confuse me a lot, considering on Medify, my QR average is 895/900...

I'm hoping in the real exam, I will have time to attempt all Q's, especially in QR, because it is a strong point for me.

 

[ucatboy]

Having not done the UCAT, I'm not sure on this, but is it obvious which questions are the 'hard' ones and which are the 'easy' ones? There was variable question worth seen in UMAT, too, but it was impossible to know which were more likely to have a higher weighting (for want of a better word), and, in fact, it seemed likely that the weighting wasn't even finalised until the exam had been sat and everyone had answered all the questions.

I guess they must be pre-determined in UCAT, given you get your scores immediately. But my questions is: how likely is it that you could pick a 'hard' from an 'easy' question (given what one person finds easy others might find hard) in the UCAT?

(PS. This whole conversation gives me JOY and is one of the (largely pointless but very fascinating!) things I used to love talking about with regard to UMAT *heart eyes*)

If I recall correctly, the reason why scaled scores can be given to you on the spot (rather than waiting until everyone's finished, like the UMAT) is that all the UCAT questions aren't new but have been given before in one form or another (think of UKCATs from the past 10+ years). Pearson already has pre-existing statistics on all of the questions so they can allocate a scaled score to each one beforehand.

It also explains every year, the average score on the three test sets in circulation are basically the same (differ by 1 mark at the most), ensuring fairness. Every year, Pearson has searched through its existing question database and compiled 3 sets of questions which they know have the same total difficulty (i.e. ask the question database to spit out 40 random VR questions that total 600 scale score), rather than writing 3 new test sets, which could be catastrophic if one ends up being way harder than the other.

I see your point in that difficulty varies amongst all the test takers. Another big thing is that with much stricter time limits in the UCAT, it makes it even more difficult to allow yourself time to see whether a question is going to be 'hard' or not. In VR and DM, usually the size/complexity of the stimulus can indicate the difficulty of the question, but in my preparation so far, I have prioritised completing as many questions as I can in as little time (therefore attempting the 'easier' ones first). In AR, if a pattern doesn't click within 30sec I tend to move on. So, in short, with most questions, you can distinguish harder questions from easier ones.

Although, when I see a hard question unit, for example on MedEntry with regard to QR, it is almost entirely based on the size of the stimulus, except if some of the questions are able to be completed in ~20sec (e.g. % of A compared to B). I find myself skipping 3-4 units of 4 Q's in QR and averaging about 25/36 (~97%ile on that platform). I seem to always leave these units until last and normally miss out 5-6 Q's that are probably more difficult than others - but this triaging is what helps me get a better raw score, so I think it's best not to let this discussion change my strategy. This does confuse me a lot, considering on Medify, my QR average is 895/900...

I'm hoping in the real exam, I will have time to attempt all Q's, especially in QR, because it is a strong point for me.

You're on track to get that 900 in QR I had a similar story to you, averaging 891 on Medify QR, and I found the questions in the real thing to be considerably easier than those during practice, finishing with time to spare. Don't change a thing about your strategy and keep it up!

 

[Jangko]

Do the scoring of UCAT sections vary? Like say you get 25/44 for VR would you either get 630 or 650 depending on the cohorts result? Or would you get 650 straight up?

 

[ucatboy]

Do the scoring of UCAT sections vary? Like say you get 25/44 for VR would you either get 630 or 650 depending on the cohorts result? Or would you get 650 straight up?

Probably anything between 600 and 700 depending on which 25 questions you got right i.e. were they particularly difficult ones, easy ones etc.