Interesting study. I think you could, if you had an interest, to expand the study to see if different users (based on a variable, such as gender, subject matter knowledge, demographics, etc.) used one cognitive processing method more than others.
Hi, thank you for your question! Participants were 48% female and 52% male. Also 60% belonged to a STEM field and 40% were in the Arts. 40% of the students received their high school education outside of British Columbia – predominantly from the US, China and India, but also Korea, Singapore, and England amongst other countries. Also 88% of participants had training or attended a class in digital literacy education. In addition, when participants were asked to rank their confidence in their search skills (i.e., using a Likert scale from very low, low, moderate, high, to very high), 92% ranked it as high or very high and 8% ranked their skills as moderate. No participants lacked confidence in their search skill.
Prior scholars looking at cognitive biases on Google search pages in regards to political issues did not find statistically significant differences amongst gender or culture (See: Epstein 2017, “Suppressing the Search Engine Manipulation Effect (SEME)”). The findings were also replicated by an independent study looking at science issues (See: Huang 2020, “Exploring Students’ Search Behavior and the Effect of Epistemological Beliefs on Contradictory Issues”).
The study presented here also found the four cognitive biases had similar affects on genders, country-of-origin, and discipline. Both novices and experts (i.e., participants with higher-domain knowledge and intellectual-investment on a topic) will also exhibit similar signs of cognitive bias on general topics. However, there is one situation where the difference between experts and novices becomes statistically significant: While both group exhibit similar levels of cognitive biases on general topics, the novices exhibit higher levels of cognitive bias as the topic becomes more focused. In other words, an expert’s search strategy is less biased than novices on focused topics, but not on general topics. The difference in effect being a matter of degrees of bias, not kind of bias.
wourmajj
1 year ago
Could you discuss a little about the participants, how they were selected and do you think any of their demographic background (age, gender, etc) had a role in the results?
Hi, thank you for your question! 47% of participants preferred pronouns were she/her and identified as female, 49% of participants preferred pronouns were he/him and identified as male, and 4% were non-binary. Also, 60% were studying within a STEM field and 40% were in the Arts. 40% of the students received their high school education outside of British Columbia – predominantly from the US and China, but also from England, India, Korea, and Singapore, amongst other countries. Also, 90% of participants had educational training in digital literacy. In addition, when participants were asked to rank their confidence in their search skills (i.e., using a Likert scale from very low, low, moderate, high, to very high), 92% ranked it as high or very high and 8% ranked their skills as moderate. No participants ranked their search skills as below moderate.
The framing effects and four cognitive biases had similar affects on genders, country-of-origin, and discipline. Both novices and experts (i.e., participants with higher-domain knowledge and intellectual-investment on a topic) will also exhibit similar signs of cognitive bias on general topics. However, there is one situation where the difference between experts and novices becomes statistically significant: While both groups exhibit similar levels of cognitive biases on general topics, the novices exhibit higher levels of cognitive bias as the topic becomes more focused. In other words, an expert’s search strategy is less biased than novices on focused topics, but not on general topics. The difference is a matter of degrees of bias, not kind of bias.
Very interesting, thank you for this study. excellent work!
Brandon Strubberg
1 year ago
Hello, Alamir. Thank you for sharing your research about how users experience information via frames. Our students need to be adept at both deciphering how information generates meaning and designing and managing digital information. Based on your research, what takeaways would you most want a student to understand about frames and networks of information as they enter the professional sphere?
This is a great question! As for the takeaways I’d like students to understand about framing, it would relate to my recommendations in Table 4 of the poster. The participants in the study had a good understanding on how sources can have biases, which I credit to educators. However, educators should also focus on both how a search algorithm and a search strategy interact together to create biases. For the search algorithm, this means providing more sophisticated forms of search literacy that explain how the document-order and document-genre on a search page can affect their understanding of a topic [Figure 2].
For the search strategy, students should also consider how their prior-knowledge can be used to either broaden or narrow frames. To demonstrate this difference with specific examples from the study: Students often used their prior positive experience with cellphones to frame the topic of “cellphones in a classroom” as a generally positive experience. When they queried the topic, they found a search page with a list of benefits and used them in a write-up. They also knew cellphones could be distractions to students, but suggested ways that the problem could be fixed (e.g., by putting cellphones on silent-mode). Conversely, another student, Participant51, went through the same process as the other students but used her prior knowledge to expand her framing. Participant51 had the same positive experience and used the same search strategy to find positive results on the search page. However, she then considered a prior memory when a cellphone did not just distract the class, but her teacher’s lecture structure. Participant51 used this experience to query for more perspectives from professors on cellphones. She then expanded the framing of her write-up from the positive student-perspective to include a list of perspectives from professors who saw cellphones as a distraction to their lecture – regardless of whether phones are put on silent. Thus, Participant51’s framing of the issue expanded from being a mostly positive one for students to one that also included some of the negative impacts it can have on a professor’s lecture. In summary, students who use their prior experiences to frame an answer only exhibited more biased answers than those who use their prior experiences to broaden the framing of their query.
Thank you for doing this study, Alamir. Capturing participants’ cognitive interactions and multiple dimensions of frames was very interesting. Could you add a bit more about the implications of this study for educators or other fields? Thank you!
This is a great question! This research examined digital literacy amongst a specific demographic of high-achieving students that not only received educational training in digitally literacy, but ranked their confidence in their search skills as high. However, the paper found that the current depth of digital literacy of students is inadequate for responding to the four cognitive biases encountered during a search. However, not all students were influenced to the same degree. The students who were less influenced by biases exhibited behavior that led to four recommendations on how to mitigate the four cognitive biases. In brief, students can 1) first understand that some document-genres are pre-determined by search algorithms. Countering priming requires setting a prior base for valued genres (e.g., if you need to see the results of a poll from an academic-genre, do not settle for another genre, such as a blog, that can frame the poll with a bias). 2) To counter the anchoring bias that emerges from the search algorithm’s order, students can scan the range of potential range of answers – since those answers could have potentially appeared at the top depending on the algorithm’s personalizations (or a query’s wording). 3) While concluding a search, students can counter framing-effects by considering overlooked genres that broaden the frame of debate (e.g., a topic framed by a blog may be framed differently by the news, Wikipedia, or an academic journal). 4) To counter the availability bias that occurs from a search page’s document-order students can search for counter-points to their original query (e.g., querying for “benefits of clean coal” should be countered with querying for “problems with clean coal”). These are four brief ones, but I’m happy to expand on these four recommendations.
In general, educators should focus on both how a search algorithm and a search strategy interact together to create biases. For the search algorithm, this means providing more sophisticated forms of search literacy that explain how the document-order and document-genre affect a search. For the search strategy, students should consider how their prior-knowledge can be used: Students who use their prior-knowledge as a guide for broadening their search strategy for information on a topic exhibit fewer biases. However, if their prior-knowledge is used to frame an answer (instead of their search strategy itself) they exhibit more biased answers.
Dan Richards
Admin
1 year ago
Insightful work, Alamir! And quite important. I kept thinking what role writing/TC/UX courses have in teaching how to re-frame. How do we reach meta-cognitive awareness to know what our frames are? Metaphor theory?
Hi Dan, thank you for your questions. Even though the two questions are related, I will answer both in separate paragraphs for clarity:
“How do we reach meta-cognitive awareness to know what our frames are?”
Insight on answering this question can come from the theory of Search-As-Learning (SAL), which conceptualizes ‘search’ as a learning experience. SAL theorizes that people do not only search for information, but metacognate as they complete a search task. My research hypothesized that just as cognitive tasks require framing, metacognition requires a metaframe (i.e., where people can metacognate about frames themselves and how they interact) and I will provide an example. Using a think-aloud protocol, several utterances that can be classified as metacognition were observed, such as when students would not only search a topic but then comment on how they think they ought to search for a topic. Sometimes these utterances referred to not only the information retrieved from a search, but other people’s biases or how information is framed by other factors. For example, students recognized that Google was framing their results about an issue. This metacognitive line of thinking about other sources of framing is where I think students can begin to consider the metaframe. The next step towards recognizing the metaframe requires being aware of how these frames interact with our own personal frames to create feedback loops. The most common occurrence of this line of thinking is when people recognize confirmation bias: In this case, the student realizes that not only is their query framed by their opinion, but so is the search page itself framed to provide search results that are framed – leading to a feedback loop. This is just one of the interactions that occur in the metaframe but there are others. However, it is one of the more recognizable biases in the cognitive literature so I think it is a good place to start teaching this concept. Fortunately, most students recognized biases in one of the three frames I mention in my introduction. Unfortunately, only one of the sixty students recognized that the biases in these frames can interact.
As for Conceptual Metaphor theory I actually have quite a bit to add about that line of theory but didn’t have room on my poster for it. To keep it brief: I would relateLakoff and Johnson (1980)’s claim on CMT to how people use a metaphor of a search process to guide their actual experience of searching. For example, when some participants thought about searching for a topic they didn’t always just have a question but also an idea of what type of results might show up (e.g., Wikipedia, a Google Answer, News, etc..) and even some fragments of the type of answer. This metaphor is always kept simple and roughly ordered in the way presented in Table 1 of my poster. As expected, the actual experience of their search process would always be situated in a way that the process does not align perfectly with it. For example, people need to re-query, repeat steps, skip steps, and some of these steps would be conducted simultaneously. That is why I captioned in the table that the frames of thought “can be used in parallel” during the information search experience. In other words, a rough/vague metaphor of an ordered search process is kept in a person’s mind to guide the actual situatedness (and messiness) of the experience itself. I have this graphed out in more detailed diagrams for my dissertation, but that is the general relationship.
Interesting study. I think you could, if you had an interest, to expand the study to see if different users (based on a variable, such as gender, subject matter knowledge, demographics, etc.) used one cognitive processing method more than others.
Hi, thank you for your question! Participants were 48% female and 52% male. Also 60% belonged to a STEM field and 40% were in the Arts. 40% of the students received their high school education outside of British Columbia – predominantly from the US, China and India, but also Korea, Singapore, and England amongst other countries. Also 88% of participants had training or attended a class in digital literacy education. In addition, when participants were asked to rank their confidence in their search skills (i.e., using a Likert scale from very low, low, moderate, high, to very high), 92% ranked it as high or very high and 8% ranked their skills as moderate. No participants lacked confidence in their search skill.
Prior scholars looking at cognitive biases on Google search pages in regards to political issues did not find statistically significant differences amongst gender or culture (See: Epstein 2017, “Suppressing the Search Engine Manipulation Effect (SEME)”). The findings were also replicated by an independent study looking at science issues (See: Huang 2020, “Exploring Students’ Search Behavior and the Effect of Epistemological Beliefs on Contradictory Issues”).
The study presented here also found the four cognitive biases had similar affects on genders, country-of-origin, and discipline. Both novices and experts (i.e., participants with higher-domain knowledge and intellectual-investment on a topic) will also exhibit similar signs of cognitive bias on general topics. However, there is one situation where the difference between experts and novices becomes statistically significant: While both group exhibit similar levels of cognitive biases on general topics, the novices exhibit higher levels of cognitive bias as the topic becomes more focused. In other words, an expert’s search strategy is less biased than novices on focused topics, but not on general topics. The difference in effect being a matter of degrees of bias, not kind of bias.
Could you discuss a little about the participants, how they were selected and do you think any of their demographic background (age, gender, etc) had a role in the results?
Hi, thank you for your question! 47% of participants preferred pronouns were she/her and identified as female, 49% of participants preferred pronouns were he/him and identified as male, and 4% were non-binary. Also, 60% were studying within a STEM field and 40% were in the Arts. 40% of the students received their high school education outside of British Columbia – predominantly from the US and China, but also from England, India, Korea, and Singapore, amongst other countries. Also, 90% of participants had educational training in digital literacy. In addition, when participants were asked to rank their confidence in their search skills (i.e., using a Likert scale from very low, low, moderate, high, to very high), 92% ranked it as high or very high and 8% ranked their skills as moderate. No participants ranked their search skills as below moderate.
The framing effects and four cognitive biases had similar affects on genders, country-of-origin, and discipline. Both novices and experts (i.e., participants with higher-domain knowledge and intellectual-investment on a topic) will also exhibit similar signs of cognitive bias on general topics. However, there is one situation where the difference between experts and novices becomes statistically significant: While both groups exhibit similar levels of cognitive biases on general topics, the novices exhibit higher levels of cognitive bias as the topic becomes more focused. In other words, an expert’s search strategy is less biased than novices on focused topics, but not on general topics. The difference is a matter of degrees of bias, not kind of bias.
Very interesting, thank you for this study. excellent work!
Hello, Alamir. Thank you for sharing your research about how users experience information via frames. Our students need to be adept at both deciphering how information generates meaning and designing and managing digital information. Based on your research, what takeaways would you most want a student to understand about frames and networks of information as they enter the professional sphere?
This is a great question! As for the takeaways I’d like students to understand about framing, it would relate to my recommendations in Table 4 of the poster. The participants in the study had a good understanding on how sources can have biases, which I credit to educators. However, educators should also focus on both how a search algorithm and a search strategy interact together to create biases. For the search algorithm, this means providing more sophisticated forms of search literacy that explain how the document-order and document-genre on a search page can affect their understanding of a topic [Figure 2].
For the search strategy, students should also consider how their prior-knowledge can be used to either broaden or narrow frames. To demonstrate this difference with specific examples from the study: Students often used their prior positive experience with cellphones to frame the topic of “cellphones in a classroom” as a generally positive experience. When they queried the topic, they found a search page with a list of benefits and used them in a write-up. They also knew cellphones could be distractions to students, but suggested ways that the problem could be fixed (e.g., by putting cellphones on silent-mode). Conversely, another student, Participant51, went through the same process as the other students but used her prior knowledge to expand her framing. Participant51 had the same positive experience and used the same search strategy to find positive results on the search page. However, she then considered a prior memory when a cellphone did not just distract the class, but her teacher’s lecture structure. Participant51 used this experience to query for more perspectives from professors on cellphones. She then expanded the framing of her write-up from the positive student-perspective to include a list of perspectives from professors who saw cellphones as a distraction to their lecture – regardless of whether phones are put on silent. Thus, Participant51’s framing of the issue expanded from being a mostly positive one for students to one that also included some of the negative impacts it can have on a professor’s lecture. In summary, students who use their prior experiences to frame an answer only exhibited more biased answers than those who use their prior experiences to broaden the framing of their query.
Great example. Thank you!
Thank you for doing this study, Alamir. Capturing participants’ cognitive interactions and multiple dimensions of frames was very interesting. Could you add a bit more about the implications of this study for educators or other fields? Thank you!
This is a great question! This research examined digital literacy amongst a specific demographic of high-achieving students that not only received educational training in digitally literacy, but ranked their confidence in their search skills as high. However, the paper found that the current depth of digital literacy of students is inadequate for responding to the four cognitive biases encountered during a search. However, not all students were influenced to the same degree. The students who were less influenced by biases exhibited behavior that led to four recommendations on how to mitigate the four cognitive biases. In brief, students can 1) first understand that some document-genres are pre-determined by search algorithms. Countering priming requires setting a prior base for valued genres (e.g., if you need to see the results of a poll from an academic-genre, do not settle for another genre, such as a blog, that can frame the poll with a bias). 2) To counter the anchoring bias that emerges from the search algorithm’s order, students can scan the range of potential range of answers – since those answers could have potentially appeared at the top depending on the algorithm’s personalizations (or a query’s wording). 3) While concluding a search, students can counter framing-effects by considering overlooked genres that broaden the frame of debate (e.g., a topic framed by a blog may be framed differently by the news, Wikipedia, or an academic journal). 4) To counter the availability bias that occurs from a search page’s document-order students can search for counter-points to their original query (e.g., querying for “benefits of clean coal” should be countered with querying for “problems with clean coal”). These are four brief ones, but I’m happy to expand on these four recommendations.
In general, educators should focus on both how a search algorithm and a search strategy interact together to create biases. For the search algorithm, this means providing more sophisticated forms of search literacy that explain how the document-order and document-genre affect a search. For the search strategy, students should consider how their prior-knowledge can be used: Students who use their prior-knowledge as a guide for broadening their search strategy for information on a topic exhibit fewer biases. However, if their prior-knowledge is used to frame an answer (instead of their search strategy itself) they exhibit more biased answers.
Insightful work, Alamir! And quite important. I kept thinking what role writing/TC/UX courses have in teaching how to re-frame. How do we reach meta-cognitive awareness to know what our frames are? Metaphor theory?
Hi Dan, thank you for your questions. Even though the two questions are related, I will answer both in separate paragraphs for clarity:
“How do we reach meta-cognitive awareness to know what our frames are?”
Insight on answering this question can come from the theory of Search-As-Learning (SAL), which conceptualizes ‘search’ as a learning experience. SAL theorizes that people do not only search for information, but metacognate as they complete a search task. My research hypothesized that just as cognitive tasks require framing, metacognition requires a metaframe (i.e., where people can metacognate about frames themselves and how they interact) and I will provide an example. Using a think-aloud protocol, several utterances that can be classified as metacognition were observed, such as when students would not only search a topic but then comment on how they think they ought to search for a topic. Sometimes these utterances referred to not only the information retrieved from a search, but other people’s biases or how information is framed by other factors. For example, students recognized that Google was framing their results about an issue. This metacognitive line of thinking about other sources of framing is where I think students can begin to consider the metaframe. The next step towards recognizing the metaframe requires being aware of how these frames interact with our own personal frames to create feedback loops. The most common occurrence of this line of thinking is when people recognize confirmation bias: In this case, the student realizes that not only is their query framed by their opinion, but so is the search page itself framed to provide search results that are framed – leading to a feedback loop. This is just one of the interactions that occur in the metaframe but there are others. However, it is one of the more recognizable biases in the cognitive literature so I think it is a good place to start teaching this concept. Fortunately, most students recognized biases in one of the three frames I mention in my introduction. Unfortunately, only one of the sixty students recognized that the biases in these frames can interact.
As for Conceptual Metaphor theory I actually have quite a bit to add about that line of theory but didn’t have room on my poster for it. To keep it brief: I would relate Lakoff and Johnson (1980)’s claim on CMT to how people use a metaphor of a search process to guide their actual experience of searching. For example, when some participants thought about searching for a topic they didn’t always just have a question but also an idea of what type of results might show up (e.g., Wikipedia, a Google Answer, News, etc..) and even some fragments of the type of answer. This metaphor is always kept simple and roughly ordered in the way presented in Table 1 of my poster. As expected, the actual experience of their search process would always be situated in a way that the process does not align perfectly with it. For example, people need to re-query, repeat steps, skip steps, and some of these steps would be conducted simultaneously. That is why I captioned in the table that the frames of thought “can be used in parallel” during the information search experience. In other words, a rough/vague metaphor of an ordered search process is kept in a person’s mind to guide the actual situatedness (and messiness) of the experience itself. I have this graphed out in more detailed diagrams for my dissertation, but that is the general relationship.