Elon Musk: Test an AI Answer Against a Known Result
How can I check an AI answer when its confident wording gives me no reason to trust it?
Choose, practice, reflectA practical learning exercise
The Lesson
An assistant gives you a neat table and a confident total. The explanation sounds sensible. You still need to know whether it counted the right items, used the supplied values and admitted when a value was missing.
In his 2023 conversation with Lex Fridman, Elon Musk discusses the problem of confidently wrong AI answers and the need to test conclusions against reality. In the transcript’s sections around 35:59 and 40:01, he frames reliability as an aspiration that must contend with errors. These are his views in that interview, not a measurement of the accuracy of today’s Grok.
A practical place to begin is a question whose answer you can establish independently. Make a small control case: a task with explicit inputs and a result you have already calculated or observed. Keep your expected result outside the assistant’s conversation until you have examined its answer.
For example, a fictional supply list has six blue tokens, four green tokens and three blue tokens. You want the blue-token total. You can check it yourself: six plus three is nine. An answer of thirteen might be mathematically tidy but ignore the color condition. A correct result with no explanation can also hide a misunderstanding that appears in the next case.
Check the number, the interpretation and the treatment of missing information separately. Then change one detail. Replace three with five, remove a count or change the requested color. The new case helps reveal whether the assistant followed the rule or merely produced one lucky answer. A handful of checks is a learning exercise, not proof of general reliability.
Retrieval is a separate capability. Current Grok Web Search documentation describes searching and reading web pages. That can provide material to examine, but access to sources does not establish a correct interpretation of your task. For the control below, all the necessary evidence is in the supplied fictional list; no browsing is needed.
Reflection
- Which part of an answer do you currently trust because it sounds confident?
- What small result can you check without asking another AI to approve it?
- Which instruction would change the answer if it were ignored?
- When should missing information lead to a question instead of an estimate?
- What would this small test leave unproven about your real task?
Practice
Original SelfGrowthVideos exercise: make a known-answer control. This is our exercise, not Musk’s benchmark or an endorsed evaluation method. Use paper or an assistant you already have. No paid account is required.
- Create the input. Write three fictional entries: blue tokens, 6; green tokens, 4; blue tokens, 3. The task is to total only blue tokens and show which entries were counted.
- Keep an answer key. Record 9 and the two blue entries in your own notes. Do not include the answer key in the task you give the assistant. Ask it to use only the supplied list and flag missing counts.
- Check the first response. Does the total match 9? Are the included entries blue? Did the explanation invent another entry or a rule? Mark each check separately; confident wording earns no extra credit.
- Change one value. Make the final blue count 5. Your independently checked total is now 11. Repeat the task with the revised list and save both responses. Confirm that the assistant used the fresh input.
- Remove one value. Replace that final count with “unknown.” The exact blue total is now unavailable. A useful response can report the known subtotal of 6 while asking for the missing count. An exact total supplied without an explicit, authorized assumption fails this check.
- Record the limits. Note the date, the tool or model if shown, the inputs, the outputs and your corrections. Keep the original answer even if a later retry succeeds. These cases do not measure performance on larger lists, images, unfamiliar subjects or consequential decisions.
Review
Return to the task you hoped to delegate. Can you construct a similarly small case with an independent answer? If the assistant failed, investigate the input, instruction and result before widening its role. If it passed, keep the control for future changes and continue checking real outputs. A passing control earns a narrow piece of evidence, not automatic trust.