California Academy of Sciences Scientific Literacy Poll – Part 1

by Eric on April 10, 2009

Back around February 25th of this year, the California Academy of Sciences released the results of a poll conducted for them in December 2008 by Harris Interactive. The results were not good:

A new national survey commissioned by the California Academy of Sciences and conducted by Harris Interactive® reveals that the U.S. public is unable to pass even a basic scientific literacy test.

poll-graphic

The survey revealed the distressing news that Americans couldn’t properly identify the length of time for the earth to go around the sun, estimate the amount of fresh water on the surface of the earth, or decide whether to go along with the idea that we once walked with dinosaurs. The Cal Academy had designed and administered a “basic scientific literacy test” and the country that developed a pill to produce erections on demand had failed it.

 

The reaction from the science blogosphere: a sad acknowledgement that while the merit of such a survey was debatable, its conclusions were unsurprising and probably correct.

Sheril Kirshenbaum at The Intersection wrote:

But, wait a second… Before rushing to attack the American education system, first consider: What do such quizzes actually reveal? Is it fair to use the results as evidence ofscientific illiteracy? Furthermore, what does that term really mean?

Over at NeuroLogica Blog, Steven Novella questioned these surveys as a measure of science literacy:

Also, a metaquestion raised by this survey is how good these surveys are themselves. What do they really tell us? If anything they overestimate scientific literacy because they are multiple choice – so some of the percentage correct were due to lucky guessing. Also, they don’t really test scientific understanding, just random bits of scientific trivia. There probably is some correlation between knowing basic science facts and being interested in and understanding science, but these are not the same thing.

Adult science literacy is the name of the game here at Science4Grownups, but it’s a little beyond the scope of this blog to solve that problem. What I could do is find out how hard the test was, and whether I could pass it. Going to the Cal Academy website reveals a little quiz in a corner of the home page entitled “Test Your Science.” Answering 4 questions will get you the results of how respondents answered those same questions on the survey.

But wait. 4 questions? Really? The California Academy of Sciences has determined the U.S. is scientifically illiterate based upon 4 questions? That can’t be right.

My curiosity got the better of me, so I went ahead and called up Steven Ng at California Academy of Sciences, and after exchanging several emails over the course of a week, I was able to get the most of the poll questions, some aggregate results, and Harris Interactive’s methodology.

The questions are included below, and the results will be included in an upcoming post. Demographic questions are not included, nor is the script questioners are instructed to read. I also don’t know for certain whether this is the order in which the questions were asked.

My thoughts:

Out of 32 questions, 6 or 18% are factual: Q10-Q15; the rest are opinion questions. To me, this hardly qualifies it a test of anything, much less scientific literacy. While Q10-Q12 could be consider questions of “basic” science literacy (I think Q11 about the dinosaurs is a bit a stretch), I’m not sure how knowing what percentage of the Earth’s water is fresh constitutes anything other than geographical trivia.

Q2 appears to be a relatively reasonable question about interest in various scientific issues. However, it seems odd they asked one general question about “science discoveries,” one about genetic discoveries (which are a form of science discovery last I checked) and then 4 questions about environment-related issues. Is the idea to identify what environmental issues interest people most, or just to confirm that people have some interest in various environmental issues?

For some reason Q3 doesn’t then ask about knowledgeability for the exact same areas as in Q2.

Q2-Q5 and Q16 are all questions begging for response bias. Of course, puppies are a good thing, or in these cases how important interest and knowledge of science is, right?

In Q6, whether or not you have any inkling of what scientists are doing, you’re quite likely to agree with this statement, there wouldn’t be anything wrong with that. Scientists today work in a myriad of areas of study, each requiring specialized terminology, techniques, and expertise. Again, this seems to be a question subject to response bias. That they didn’t do a better job of neutralizing the question was bad enough (eg, “I understand a lot, some, not much, or nothing at all about what scientists are doing today”), but there is no followup question asking whether the respondent is comfortable or uncomfortable with the implications of the answer given.

Quite frankly, Q9 begins with a scientifically illiterate statement. It is indeed the scientific consensus is that many species go extinct every year, but that’s normal – species go extinct all the time, otherwise we probably wouldn’t be here.

By the time I got to the end of it, the survey really began to grate on me. Essentially, it asks a lot of relatively vague or open-to-interpretation questions mostly about the environment (which you can find out more about by visiting the California Academy of Sciences, open M-S from…), but when challenged about the lack of clarity, the out seems to be: “well, you’d understand what we mean if you were properly scientifically literate.” I would argue that scientific inquiry is about making sure you carefully ask questions in the right way and this respect, the design of this survey fails.

Next up, we look at the results. Thanks to Steven Ng of the California Academy of Sciences for helping me pull together the questions and the results.

Q1. How connected do you feel to the natural world – very connected, somewhat connected, not very connected, or not at all connected?

2. How interested are you in … – very interested, moderately interested, or not at all interested?

Q2a. New scientific discoveries

Q2b. Environmental issues

Q2c. Discoveries in genetics

Q2d. The discovery of new species

Q2e. The extinction of species

Q2f. The future of energy resources (such as oil and natural gas)

3. How knowledgeable do you feel about … – very knowledgeable, moderately knowledgeable, or not at all knowledgeable?

Q3a. New scientific discoveries

Q3b. Environmental issues

Q3c. What you can do personally to protect the natural world

4. Would knowing more about science and the way it works help you to …? (Yes, No, Not Sure)

Q4a. Make more informed voting decisions

Q4b. Make more informed choices as a consumer

Q4c. Make more informed choices for you children

Q4d. Make more informed choices for your own health and physical well being

Q4e. Satisfy your own curiosity

Q5. Who do you trust most as a source of reliable information about the environment and the natural world?

Answer volunteered by respondent.

Q6. In general, do you agree or disagree with the following statement – I understand less and less of what scientists are doing today? (Agree, Disagree, Not Sure)

Q7. How much stress do you believe there is on the natural world today - a lot of stress, some stress, a little stress, or no stress at all?

Q8. Do you think the actions of humans are a major factor, a moderate factor, a minor factor, or no factor in impacting the natural world?

Q9. There is a consensus among scientists that many species are going extinct each year. How much of a negative impact do you think the current extinction of species has on humansvery negative, somewhat negative, slightly negative, or not at all negative?

Q10. How long does it take for the Earth to go around the Sun - one day, one week, one month, or one year?

Q11. Is the following statement true or false? The earliest humans lived at the same time as dinosaurs.

Q12. What percentage of the Earth’s surface is covered by water?

1-9%
10-19%
20-29%
30-39%
40-49%
50-59%
60-69%
70-79%
80-89%
90-99%
100%
Not sure

Q13. What percentage of the Earth’s water is fresh water?

0%
1-9%
10-19%
20-29%
30-39%
40-49%
50-59%
60-69%
70-79%
80-89%
90-99%
100%
Not sure

Q14. Do you think that evolution is currently occurring? (Yes, No, Not Sure)

Q15. Do you think humans are influencing the evolution of other species? (Yes, No, Not Sure)

16. In your opinion, how important is science education to … - absolutely essential, very important, somewhat important, or not at all important?

Q16a. The U.S. Economy

Q16b. The U.S. healthcare system

Q16c. The U.S. global reputation

17. How knowledgeable would you say you are about each of the following scientific concepts … - very knowledgeable, moderately knowledgeable, or not at all knowledgeable?

Q17a. Sustainability

Q17b. Biodiversity

Q17c. Evolution


Share Me:
  • Google Bookmarks
  • Digg
  • del.icio.us
  • StumbleUpon
  • Sphinn
  • Technorati
  • Reddit
  • Facebook
  • LinkedIn
  • TwitThis
  • Mixx

{ 1 comment… read it below or add one }

susansls April 10, 2009 at 8:55 pm

Cool, thanks for doing some original reporting on this subject. We tend to take these poll results as gospel, when it’s really not clear at all what methodology the pollsters used to come up with their results. Very enlightening.

{ 2 trackbacks }

Previous post:

Next post: