I often hear rumors of a perceived dichotomy between the depth of understanding and quantity of fixed data. Comments like “The number of participants involved in a study seems to indicate validity” and “this attitude might be detrimental to the initiative to improve the classroom in the minds of a single teacher”, abound.
I believe that these assumptions might be widespread. The ideal is that, as Silverman says in Interpreting Qualitative Data(2006) , “qualitative research should go for “Authenticity’ rather than sample size”. But, later he bemoaned that, “…in qualitative research, observation is not generally seen as a very important method of data collection. This is because it is difficult to conduct observational studies in large samples.” This leaves one with the impression that there is a premium in the presentation of lots of data or in conducting large group studies.
Jakob Nielsen, Ph.D., a user advocate and principal of the Nielsen Norman Group writes about “numbers fetishism” in his biweekly newsletter and admonishes against it. He advises, like David Silverman, that despite the ethos of statistics, quantity of numbers or even the use of quantitative research methods, does not automatically make research more valid. Mr. Nielson points out that if one were to study the 2004 U.S. presidential election in Florida to determine why George Bush won, working in a strictly quantitative method (with 100 persons sampled) they would not have had statistically significant evidence to indicate that some persons voted for Pat Buchanan mistakenly on the butterfly ballot (less than 1 % of the electorate did that). However, as the author states “A qualitative study, on the other hand, would likely have revealed some voters saying something like, "Okay, I want to vote for Gore, so I'm punching the second hole ... oh, wait, it looks like Buchanan's arrow points to that hole. I have to go down one for Gore's hole."
The whole idea of the size of statistical groups and relevancy reminds me of a book I read four years ago called ‘The Wisdom of Crowds’ by James Surowiecki (2004) [See audio excerpts 1,2,3,4]. Mr. Surowiecki presents research that size does matter in that large groups of people are collectively more intelligent and savvy than experts are individually. He says that there are three kinds of inquiries where this phenomenon is apparent – cognitive, cooperative and those instances requiring coordination between people. The foundation of his thesis is based on an observation by Francis Galton in 1906 in which 787 local farmers and townspersons at the annual West of England Fat Stock and Poultry Exhibition bet on the ‘slaughtered and dressed’ weight of an ox. The mean wager of the crowd was 1197 pounds. The actual dressed weight was 1198 pounds. In another instance, in May of 1968 the USS Scorpion was lost at sea in a possible area of 20 miles in diameter by thousands of feet deep. Sontag and Drew explain in their book “Blind Man’s Bluff” (2000) that against popular convention, John Craven, the naval officer in charge of locating the wreckage did not hire an expert in submarines or oceanography, but rather, he created several different scenarios of what might have happened to the ship, probable submarine speed and angle of descent. He then surveyed a large group individually as to their estimate of the plausibility of each scenario and then used Bayes’s theorem (which allows for new information and its effect on pre-existing expectations) to calculate the results. Although no one person in the group had selected the discovery point as their likely spot, the calculation of Craven’s group was only 220 yards from where submarine was recovered. Therefore, I hope Surowiecki’s conclusion of ‘individual power’ heartens the singular classroom teacher !
Hopefully, this assumption of the inherent validity of a large study population or voluminous statistics can be debunked and take with it the ethos of the intellectualist (so evidenced in the way my parent's eyes glaze over when I say the word “pedagogical”). In its place, I hope, will come an affinity for the appropriate design of the methodology to the research.
I believe that these assumptions might be widespread. The ideal is that, as Silverman says in Interpreting Qualitative Data(2006) , “qualitative research should go for “Authenticity’ rather than sample size”. But, later he bemoaned that, “…in qualitative research, observation is not generally seen as a very important method of data collection. This is because it is difficult to conduct observational studies in large samples.” This leaves one with the impression that there is a premium in the presentation of lots of data or in conducting large group studies.
Jakob Nielsen, Ph.D., a user advocate and principal of the Nielsen Norman Group writes about “numbers fetishism” in his biweekly newsletter and admonishes against it. He advises, like David Silverman, that despite the ethos of statistics, quantity of numbers or even the use of quantitative research methods, does not automatically make research more valid. Mr. Nielson points out that if one were to study the 2004 U.S. presidential election in Florida to determine why George Bush won, working in a strictly quantitative method (with 100 persons sampled) they would not have had statistically significant evidence to indicate that some persons voted for Pat Buchanan mistakenly on the butterfly ballot (less than 1 % of the electorate did that). However, as the author states “A qualitative study, on the other hand, would likely have revealed some voters saying something like, "Okay, I want to vote for Gore, so I'm punching the second hole ... oh, wait, it looks like Buchanan's arrow points to that hole. I have to go down one for Gore's hole."
The whole idea of the size of statistical groups and relevancy reminds me of a book I read four years ago called ‘The Wisdom of Crowds’ by James Surowiecki (2004) [See audio excerpts 1,2,3,4]. Mr. Surowiecki presents research that size does matter in that large groups of people are collectively more intelligent and savvy than experts are individually. He says that there are three kinds of inquiries where this phenomenon is apparent – cognitive, cooperative and those instances requiring coordination between people. The foundation of his thesis is based on an observation by Francis Galton in 1906 in which 787 local farmers and townspersons at the annual West of England Fat Stock and Poultry Exhibition bet on the ‘slaughtered and dressed’ weight of an ox. The mean wager of the crowd was 1197 pounds. The actual dressed weight was 1198 pounds. In another instance, in May of 1968 the USS Scorpion was lost at sea in a possible area of 20 miles in diameter by thousands of feet deep. Sontag and Drew explain in their book “Blind Man’s Bluff” (2000) that against popular convention, John Craven, the naval officer in charge of locating the wreckage did not hire an expert in submarines or oceanography, but rather, he created several different scenarios of what might have happened to the ship, probable submarine speed and angle of descent. He then surveyed a large group individually as to their estimate of the plausibility of each scenario and then used Bayes’s theorem (which allows for new information and its effect on pre-existing expectations) to calculate the results. Although no one person in the group had selected the discovery point as their likely spot, the calculation of Craven’s group was only 220 yards from where submarine was recovered. Therefore, I hope Surowiecki’s conclusion of ‘individual power’ heartens the singular classroom teacher !
Hopefully, this assumption of the inherent validity of a large study population or voluminous statistics can be debunked and take with it the ethos of the intellectualist (so evidenced in the way my parent's eyes glaze over when I say the word “pedagogical”). In its place, I hope, will come an affinity for the appropriate design of the methodology to the research.
No comments:
Post a Comment