Metacognition: judging what you know¶

Level: 101 · for anyone who has handed in an exam sure it went well and got it back covered in red

One line: Of Flavell's four parts of metacognition, the last one, judging accurately how much you have learned, is a claim that can be tested: say how sure you are before you answer, and compare. A learner who does this sees their overconfidence in numbers, and the score that measures it rewards nothing but the truth.

The idea¶

Metacognition is John Flavell's word (1976) for thinking about your own thinking. Saundra McGuire's Teach Yourself How to Learn (Figure 3.2) lists four aspects of his definition, the ability to:

think about your own thinking;
be consciously aware of yourself as a problem solver;
monitor, plan, and control your mental processing;
accurately judge your level of learning.

The first three are habits: asking "what exactly don't I get?", "which strategies do I have for this?", "is this working?". McGuire's version of the first is the question to ask while studying: "Am I understanding this material, or just memorizing it?"

The fourth is different, because it is a claim, and a claim can be checked. "I know this" is a prediction about what will happen when someone asks. This lesson is about checking that prediction, because it is the one that decides when you stop studying.

Why the fourth ability fails¶

McGuire describes a scene most students know. You hand in an exam "beaming with pride, knowing that you've done a great job", and it comes back with a C, a D or an F. The obvious conclusion is "I'm not smart enough" or "I need to work harder". The accurate one is narrower: your judgement of your own learning was wrong, and it was wrong in a predictable direction.

The usual cause is rereading. Reading a page for the second time is easy, and the ease feels like knowing. But recognising something when you see it and producing it when you do not are different skills, and an exam asks for the second. In Roediger and Karpicke's 2006 experiment, students who reread a passage four times predicted they would remember more of it a week later than students who read it once and then tested themselves three times. A week later they remembered less.

The cure is not to try harder to feel the right amount of sure. It is to stop relying on the feeling: answer before you look, and keep score.

Keeping score¶

Before each answer, write down a confidence: 50% (a guess between two options) up to 100% (certain). Afterwards mark it right or wrong. Two numbers then say how well you judge yourself:

The gap: mean confidence minus the share you got right. Positive means overconfident.
The calibration table: of the answers you called 90%, what share was right? A calibrated learner's 90% answers are right about 9 times in 10.

One more number combines them. The Brier score is the mean of (confidence − outcome)², with outcome 1 for right and 0 for wrong. Zero is perfect; saying 50% every time scores 0.25; saying 100% and being wrong costs a full 1.

Why this particular formula? Because it cannot be gamed. If you really get a kind of question right with probability p and you report q, the expected score is p(1 − q)² + (1 − p)q². Its derivative in q is 2(q − p), which is zero only at q = p. The score is lowest when you report exactly what you believe. Overclaiming does not pay, and neither does false modesty. A scoring rule with this property is called proper, and it is what turns "accurately judge your level of learning" from a matter of opinion into a measurement.

What the program prints¶

The program scores one learner on twenty questions from this library, twice: after rereading the notes, and after a week of self-testing with flashcards. The data are invented, and the page says so. The pattern is the one the research reports. To score yourself, replace the list at the top of the program with your own (confidence, right) pairs.

Verified output of metacognition.py — regenerated by tools/run_examples.py, never hand-typed.

1. FLAVELL'S FOUR ABILITIES, AND THE ONE A PROGRAM CAN CHECK
   1. think about your own thinking                      -> ask: what exactly don't I get?
   2. be aware of yourself as a problem solver           -> ask: which strategies do I have for this?
   3. monitor, plan and control your mental processing   -> ask: is this working, and what now?
   4. accurately judge your level of learning            -> measure it: confidence against results
   The first three are habits. The fourth is a claim, so it can be tested:
   say how sure you are before you answer, then compare.

2. ONE LEARNER, TWENTY QUESTIONS, TWO WAYS OF STUDYING (invented data)
   after rereading the notes twice
     mean confidence  91.5%   share right  55.0%   gap  36.5%
          said | answers | right | share right
          100% |       7 |     4 |       57.1%  ###########
           90% |       9 |     5 |       55.6%  ###########
           80% |       4 |     2 |       50.0%  ##########
     Brier score 0.383   (0 is perfect; saying 50% every time scores 0.250)

   after a week of self-testing with flashcards
     mean confidence  80.0%   share right  75.0%   gap   5.0%
          said | answers | right | share right
          100% |       5 |     5 |      100.0%  ####################
           90% |       5 |     4 |       80.0%  ################
           80% |       2 |     2 |      100.0%  ####################
           70% |       3 |     2 |       66.7%  #############
           60% |       3 |     1 |       33.3%  #######
           50% |       2 |     1 |       50.0%  ##########
     Brier score 0.149   (0 is perfect; saying 50% every time scores 0.250)

   After rereading, the learner says 90-100% and is right about half the time:
   the page felt familiar, and familiar felt like known. After self-testing,
   the shares right climb with the confidence, and the gap all but closes.
   The score improved on both counts: more answers right, and better judged.

3. WHY HONEST CONFIDENCE IS THE BEST STRATEGY
   Suppose you will in fact get a kind of question right 70.0% of the time.
   The expected Brier score of reporting q is  p(1 - q)^2 + (1 - p)q^2:
     report  50%   expected score 0.250
     report  60%   expected score 0.220
     report  70%   expected score 0.210  <- smallest
     report  80%   expected score 0.220
     report  90%   expected score 0.250
     report 100%   expected score 0.300
   Its derivative, -2p(1 - q) + 2(1 - p)q = 2(q - p), is zero only at q = p.
   Neither bravado nor modesty pays: the score is lowest for the truth. A rule
   with that property is called proper, and it is what makes the fourth
   ability measurable rather than a matter of opinion.

4. WHAT MISJUDGING COSTS: WHEN DOES A LEARNER STOP STUDYING?
   A learner stops a topic once they feel 90% sure. Their real chance of
   answering right when they feel that sure is the 90% row above:
     after rereading      feels >= 90% sure on 16, right on  9:  56.2%
     after self-testing   feels >= 90% sure on 10, right on  9:  90.0%
   The rereader walks into the exam sure of sixteen answers and gets nine.
   That is the returned paper covered in red ink that McGuire describes, and
   only a test taken before the exam could have warned of it.

Section 2 is the red-ink exam in numbers. After rereading, the learner is 91.5% sure on average and right 55% of the time, and the calibration table is flat: their 100% answers are no more reliable than their 80% ones. The confidence carries no information. After self-testing, the share right rises with the confidence, and the gap shrinks from 36.5 points to 5.

Section 4 is where it costs. A learner who stops studying a topic once they feel 90% sure stops, after rereading, on topics they would get right barely half the time.

Your study habits, scored¶

McGuire also gives students an inventory: eleven statements about how they study, to be marked true or false, and a scale that turns the number of trues into a predicted grade (9 or more an A, 6–8 a B, 4–5 a C, 2–3 a D, fewer an F). The scale is her rule of thumb, not a validated test. What makes it worth running is what it leaves out: every statement is something you do, such as previewing before class, reviewing the same day, doing homework without looking at the worked examples, or reworking every missed question before the next class. None asks how clever you are. In her words, "you can change your predicted grade at any point by changing your behavior such that more of the statements are true."

The second program is a tool: run it with your own eleven answers, such as python3 study_inventory.py TFTTFFTTFTT. Without arguments it prints the scale and an invented student before and after, and marks each false answer that one of this chapter's lessons explains.

Verified output of study_inventory.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE SCALE
   11 true  ->  A
   10 true  ->  A
    9 true  ->  A
    8 true  ->  B
    7 true  ->  B
    6 true  ->  B
    5 true  ->  C
    4 true  ->  C
    3 true  ->  D
    2 true  ->  D
    1 true  ->  F
    0 true  ->  F

2. AN EXAMPLE STUDENT, BEFORE AND AFTER (invented answers)
   Before: crams, reworking the homework and making mnemonics the night before.
    1. F  I always preview the material before I go to class.   <- study cycle: preview
    2. F  I go over my lecture notes as soon as possible after lecture.   <- study cycle: review
    3. F  I do homework without example problems or copying from notes or book.   <- metacognition: answer without looking
    4. F  I regularly go to office hours or tutoring about the homework.
    5. T  I rework all of the homework problems before the test or quiz.
    6. F  I study for this class at least five days per week, outside class.   <- spaced retrieval: spread out
    7. T  I make mnemonics to help me remember facts and equations.
    8. F  I make diagrams or mental pictures of the concepts from class.
    9. F  I am in a study group where we do homework and quiz ourselves.
   10. F  I rework every missed quiz and test item before the next class.   <- metacognition: analyse the gap
   11. F  I can still do well even if I have done poorly so far.   <- attribute results to behavior
   2 true: predicted grade D

   After: the study cycle and self-testing (the arrows above).
    1. T  I always preview the material before I go to class.
    2. T  I go over my lecture notes as soon as possible after lecture.
    3. T  I do homework without example problems or copying from notes or book.
    4. F  I regularly go to office hours or tutoring about the homework.
    5. T  I rework all of the homework problems before the test or quiz.
    6. T  I study for this class at least five days per week, outside class.
    7. T  I make mnemonics to help me remember facts and equations.
    8. F  I make diagrams or mental pictures of the concepts from class.
    9. F  I am in a study group where we do homework and quiz ourselves.
   10. T  I rework every missed quiz and test item before the next class.
   11. T  I can still do well even if I have done poorly so far.
   8 true: predicted grade B

   Same student, 6 behaviors changed, D -> B.
   Nothing on the list asks how clever you are.

Questions¶

1. Which of Flavell's four abilities can you measure, and how?

Answer

The fourth, judging your level of learning. Write down a confidence before each answer, mark the answers, and compare confidence with results: the gap, the calibration table, or the Brier score.

2. A student says 90% on ten questions and gets six right. Overconfident, underconfident, or calibrated? By how much?

Answer

Overconfident by 30 points: 90% claimed, 60% right.

3. What is the Brier score of answering "100% sure" and being wrong? Of "50%" and being wrong?

Answer

(1 − 0)² = 1, the worst possible. (0.5 − 0)² = 0.25.

4. You believe you get a type of question right 70% of the time. Why not report 100%, to look confident?

Answer

Because it raises your expected Brier score from 0.21 to 0.30. The expected score p(1 − q)² + (1 − p)q² is smallest at q = p, so the honest report is the best one.

5. Rereading a chapter makes it feel familiar. Why is that feeling a poor guide?

Answer

Familiarity is recognition: the material feels known when it is in front of you. An exam asks for recall, producing the material when it is not in front of you. Only answering without looking tests recall.

6. What is the question McGuire suggests asking while studying?

Answer

"Am I understanding this material, or just memorizing it?"

7. Why does McGuire's inventory predict a grade from behaviours instead of from ability?

Answer

Because behaviours can be changed, and the prediction changes with them: each statement made true moves the count up. It is strategy 18 of her list, attribute results to actions, not ability, turned into a checklist.

8. A student's 100% answers are right 55% of the time and their 60% answers are right 55% of the time. What does that say about their confidence?

Answer

It says nothing about their answers. A useful confidence separates the answers that will be right from those that will not, and this one does not.

How to practise¶

Before every answer, write a number. On homework, flashcards and the questions on these pages: 50, 60, 70, 80, 90 or 100.
Keep the tally for a week, then fill in a calibration table like the program's.
Stop studying a topic on a test result, not on a feeling. "I got 9 of 10 without looking" is a reason to move on. "It looks familiar" is not.
When a grade surprises you, look at the gap before the grade. The grade says how much you knew; the gap says how well you knew it, and you can fix that.

The same idea in Zakrajsek¶

Todd Zakrajsek's The New Science of Learning (third edition, 2022), chapter 5, splits metacognitive regulation into three skills (after Tanner, 2012): planning before you study (what the exam expects, where, how long, alone or with others), monitoring while you study (every few pages, look away and explain the main idea to yourself, more often when the material is hard), and evaluating afterwards (was the time well spent; what do I understand least?). For the last he gives the muddiest point (Angelo and Cross, 1993): end each session by writing down what you understand least, and start the next one there.

His student Jules had dropped statistics three times, each time after failing the first exam, and called themselves "bad at math". Two thoughts were banned for the fourth attempt, "I can't do math" and "statistics is stupid". "All of math" became one concept at a time, the median first, and "I can't" became "I can't yet". Jules also kept a log of two predictions per exam: the grade they expected after reading the first question, and the grade they expected after answering the last. As the skill grows, the two estimates should come closer together. That log is a calibration table with one row per exam, the thing this page's program builds. Jules passed with a C, stayed in the psychology programme, and at graduation said the metacognitive skills were what carried them through. Study Tip 5.2: when you evaluate, note what you are doing well too.

Flashcards¶

The page as a deck of Anki cards: metacognition.txt. Import with File → Import. Tags: definition, calibration, scoring, trap.

Po polsku, w skrócie¶

Metakognicja to według Johna Flavella „myślenie o własnym myśleniu". Saundra McGuire wymienia cztery jej elementy: myśleć o swoim myśleniu, być świadomym siebie jako kogoś, kto rozwiązuje problemy, kontrolować i planować własną pracę umysłową oraz trafnie oceniać, ile się naprawdę nauczyło.

Pierwsze trzy to nawyki. Czwarty to twierdzenie, a twierdzenie można sprawdzić: zanim odpowiesz na pytanie, zapisz, na ile procent jesteś pewien, a potem porównaj z wynikiem. Kto przeczytał notatki dwa razy, zwykle jest pewny na 90% i ma rację w połowie przypadków, bo znajome wydaje się znane. Rozpoznać coś, gdy się to widzi, a przypomnieć sobie bez podglądania, to dwie różne umiejętności, a egzamin sprawdza tę drugą.

Miarą jest wynik Briera: średnia z (pewność − wynik)². Ma on ważną cechę: najmniejszą wartość oczekiwaną daje podanie dokładnie tej pewności, którą się naprawdę ma. Przechwalanie się nie opłaca, fałszywa skromność też nie. Wniosek praktyczny: kończ naukę tematu po teście, a nie po uczuciu, że „już umiem". Todd Zakrajsek dodaje trzy umiejętności: planowanie przed nauką, sprawdzanie siebie w trakcie i ocenę po, na przykład zapisanie na koniec, czego rozumiemy najmniej. Jego student Jules prowadził dziennik: przewidywana ocena przed egzaminem i po nim. To ta sama tabela kalibracji, którą buduje program.

Metacognition: judging what you know¶

The idea¶

Why the fourth ability fails¶

Keeping score¶

What the program prints¶

Your study habits, scored¶

Questions¶

How to practise¶

The same idea in Zakrajsek¶

Flashcards¶

Po polsku, w skrócie¶

See also¶