Back around February 25th of this year, the California Academy of Sciences released the results of a poll conducted for them in December 2008 by Harris Interactive. The results were not good:
A new national survey commissioned by the California Academy of Sciences and conducted by Harris Interactive® reveals that the U.S. public is unable to pass even a basic scientific literacy test.
The survey revealed the distressing news that Americans couldn’t properly identify the length of time for the earth to go around the sun, estimate the amount of fresh water on the surface of the earth, or decide whether to go along with the idea that we once walked with dinosaurs. The Cal Academy had designed and administered a “basic scientific literacy test” and the country that developed a pill to produce erections on demand had failed it.
The reaction from the science blogosphere: a sad acknowledgement that while the merit of such a survey was debatable, its conclusions were unsurprising and probably correct.
Sheril Kirshenbaum at The Intersection wrote:
But, wait a second… Before rushing to attack the American education system, first consider: What do such quizzes actually reveal? Is it fair to use the results as evidence ofscientific illiteracy? Furthermore, what does that term really mean?
Over at NeuroLogica Blog, Steven Novella questioned these surveys as a measure of science literacy:
Also, a metaquestion raised by this survey is how good these surveys are themselves. What do they really tell us? If anything they overestimate scientific literacy because they are multiple choice – so some of the percentage correct were due to lucky guessing. Also, they don’t really test scientific understanding, just random bits of scientific trivia. There probably is some correlation between knowing basic science facts and being interested in and understanding science, but these are not the same thing.
Adult science literacy is the name of the game here at Science4Grownups, but it’s a little beyond the scope of this blog to solve that problem. What I could do is find out how hard the test was, and whether I could pass it. Going to the Cal Academy website reveals a little quiz in a corner of the home page entitled “Test Your Science.” Answering 4 questions will get you the results of how respondents answered those same questions on the survey.
But wait. 4 questions? Really? The California Academy of Sciences has determined the U.S. is scientifically illiterate based upon 4 questions? That can’t be right.
My curiosity got the better of me, so I went ahead and called up Steven Ng at California Academy of Sciences, and after exchanging several emails over the course of a week, I was able to get the most of the poll questions, some aggregate results, and Harris Interactive’s methodology.
The questions are included below, and the results will be included in an upcoming post. Demographic questions are not included, nor is the script questioners are instructed to read. I also don’t know for certain whether this is the order in which the questions were asked.
My thoughts:
Out of 32 questions, 6 or 18% are factual: Q10-Q15; the rest are opinion questions. To me, this hardly qualifies it a test of anything, much less scientific literacy. While Q10-Q12 could be consider questions of “basic” science literacy (I think Q11 about the dinosaurs is a bit a stretch), I’m not sure how knowing what percentage of the Earth’s water is fresh constitutes anything other than geographical trivia.
Q2 appears to be a relatively reasonable question about interest in various scientific issues. However, it seems odd they asked one general question about “science discoveries,” one about genetic discoveries (which are a form of science discovery last I checked) and then 4 questions about environment-related issues. Is the idea to identify what environmental issues interest people most, or just to confirm that people have some interest in various environmental issues?
For some reason Q3 doesn’t then ask about knowledgeability for the exact same areas as in Q2.
Q2-Q5 and Q16 are all questions begging for response bias. Of course, puppies are a good thing, or in these cases how important interest and knowledge of science is, right?
In Q6, whether or not you have any inkling of what scientists are doing, you’re quite likely to agree with this statement, there wouldn’t be anything wrong with that. Scientists today work in a myriad of areas of study, each requiring specialized terminology, techniques, and expertise. Again, this seems to be a question subject to response bias. That they didn’t do a better job of neutralizing the question was bad enough (eg, “I understand a lot, some, not much, or nothing at all about what scientists are doing today”), but there is no followup question asking whether the respondent is comfortable or uncomfortable with the implications of the answer given.
Quite frankly, Q9 begins with a scientifically illiterate statement. It is indeed the scientific consensus is that many species go extinct every year, but that’s normal – species go extinct all the time, otherwise we probably wouldn’t be here.
By the time I got to the end of it, the survey really began to grate on me. Essentially, it asks a lot of relatively vague or open-to-interpretation questions mostly about the environment (which you can find out more about by visiting the California Academy of Sciences, open M-S from…), but when challenged about the lack of clarity, the out seems to be: “well, you’d understand what we mean if you were properly scientifically literate.” I would argue that scientific inquiry is about making sure you carefully ask questions in the right way and this respect, the design of this survey fails.
Next up, we look at the results. Thanks to Steven Ng of the California Academy of Sciences for helping me pull together the questions and the results.
Q1. How connected do you feel to the natural world – very connected, somewhat connected, not very connected, or not at all connected?
2. How interested are you in … – very interested, moderately interested, or not at all interested?
Q2a. New scientific discoveries
Q2b. Environmental issues
Q2c. Discoveries in genetics
Q2d. The discovery of new species
Q2e. The extinction of species
Q2f. The future of energy resources (such as oil and natural gas)
3. How knowledgeable do you feel about … – very knowledgeable, moderately knowledgeable, or not at all knowledgeable?
Q3a. New scientific discoveries
Q3b. Environmental issues
Q3c. What you can do personally to protect the natural world
4. Would knowing more about science and the way it works help you to …? (Yes, No, Not Sure)
Q4a. Make more informed voting decisions
Q4b. Make more informed choices as a consumer
Q4c. Make more informed choices for you children
Q4d. Make more informed choices for your own health and physical well being
Q4e. Satisfy your own curiosity
Q5. Who do you trust most as a source of reliable information about the environment and the natural world?
Answer volunteered by respondent.
Q6. In general, do you agree or disagree with the following statement – I understand less and less of what scientists are doing today? (Agree, Disagree, Not Sure)
Q7. How much stress do you believe there is on the natural world today – a lot of stress, some stress, a little stress, or no stress at all?
Q8. Do you think the actions of humans are a major factor, a moderate factor, a minor factor, or no factor in impacting the natural world?
Q9. There is a consensus among scientists that many species are going extinct each year. How much of a negative impact do you think the current extinction of species has on humans – very negative, somewhat negative, slightly negative, or not at all negative?
Q10. How long does it take for the Earth to go around the Sun – one day, one week, one month, or one year?
Q11. Is the following statement true or false? The earliest humans lived at the same time as dinosaurs.
Q12. What percentage of the Earth’s surface is covered by water?
1-9%
10-19%
20-29%
30-39%
40-49%
50-59%
60-69%
70-79%
80-89%
90-99%
100%
Not sure
Q13. What percentage of the Earth’s water is fresh water?
0%
1-9%
10-19%
20-29%
30-39%
40-49%
50-59%
60-69%
70-79%
80-89%
90-99%
100%
Not sure
Q14. Do you think that evolution is currently occurring? (Yes, No, Not Sure)
Q15. Do you think humans are influencing the evolution of other species? (Yes, No, Not Sure)
16. In your opinion, how important is science education to … – absolutely essential, very important, somewhat important, or not at all important?
Q16a. The U.S. Economy
Q16b. The U.S. healthcare system
Q16c. The U.S. global reputation
17. How knowledgeable would you say you are about each of the following scientific concepts … – very knowledgeable, moderately knowledgeable, or not at all knowledgeable?
Q17a. Sustainability
Q17b. Biodiversity
Q17c. Evolution
{ 1 comment… read it below or add one }
Cool, thanks for doing some original reporting on this subject. We tend to take these poll results as gospel, when it’s really not clear at all what methodology the pollsters used to come up with their results. Very enlightening.
You must log in to post a comment.
{ 2 trackbacks }