Accuracy and biases of output

Man, student and laptop hologram of internet, information technology and software development research on campus. African person on computer, data integration and plexus network

Summary

Though many of us may treat and view generative AI as an “expert” due to the datasets on which it has been trained, it is not: The internet, which often serves as the source of much if not all of a tool’s LLM, is not intrinsically an expert on any topic, as we all know. Generative AI can only generate content based on patterns it has detected and cannot appraise or determine the accuracy or social impact of what it creates. Only humans are capable of fully assessing the quality of material and of empathizing with others. Only we can understand. 

Therefore, in integrating and using generative AI as part of students’ learning and workflow, you should take the time to encourage, scaffold for, and discuss with students the critical engagement and use of these tools. There are two fundamental and necessary components of such critical engagement and use:

  1. AI literacy skills, which includes prompt engineering
  2. Social awareness and engagement (to identify biases and prejudices toward various groups)

The latter concern does not receive much attention in discussions about teaching students to use generative AI. However, developing this awareness will enable students to not only more effectively and comprehensively assess and judge the output of generative AI but will also foster and invite the development of empathy, something currently in decline (American Psychological Association, 2019) but increasingly needed in our complex and dynamic society.

Accuracy of output

Generative AI can create text based on what it has learned through its large language model. However, as the case study above demonstrates, it cannot assess the accuracy, validity, and truthfulness of the content it produces. As ChatGPT itself observed in discussing what generative AI cannot do, it cannot:

  • Guarantee complete accuracy and reliability in generated content.
  • Distinguish between real and generated content in all cases.

Generative AI merely uses its LLM to produce text relevant to the prompt from users. Often that output uses correct and true information borrowed from the web. Sometimes, though, that output contains “hallucinations.” The Kennesaw State University School of Data Science and Analytics (2023) states, “For the most part, when people talk about an AI hallucination, they mean that a generative AI process has responded to their prompt with what appears to be real, valid content, but which is not.”  Hiller (2023) found that in response to a prompt about teaching with technology in a science class, ChatGPT produced six citations, five of which were fake. This pattern of numerous false citations likely persists in the work generative AI may produce for students.

Though it is possible that generative AI will improve and gain effectiveness against producing hallucinations, for the time being faculty should teach students AI literacy skills. Key among these is the ability to analyze and critique the output from generative AI for quality, accuracy, and correctness. If students are working on a research project, such as the legal brief assignment described in the scenario below, it is important to ask and remind students to corroborate and check any citations or studies presented by generative AI.

Some argue that because of the possibility of hallucinations, students should not exclusively rely on generative AI for sources and citations for an assignment (Wellborn, 2023). Instead, students should review and check any citations or summaries created by generative AI for accuracy. Though generative AI may provide a basis for the discussion at hand, students should corroborate and expand on it. The technology is rapidly progressing in this area (Tay, 2023).

Biases of output

Generative AI simply analyzes a prompt and then “parrots” an answer back from its LLM (large language model). “The model (for ChatGPT) was trained using text databases from the internet. This included a whopping 570GB of data obtained from books, webtexts, Wikipedia, articles and other pieces of writing on the internet. To be even more exact, 300 billion words were fed into the system” (Hughes, 2023). If some biased or prejudiced sources are present among the training data and contribute to the construction of the LLM, then those biases or prejudices may appear once again in the output from generative AI.

It is also important to note that not including substantial content from or with certain perspectives or demographic groups in the training data may also produce biased content. Such content “can manifest in a myriad of ways, ranging from gender bias, racial and ethnic bias, socioeconomic bias, cultural bias, content bias, and ideological bias in terms of political, philosophical, and religious perspectives” (Trivedi, 2023, p. 23). Ultimately, generative AI may reinforce and perpetuate social marginalization. The research so far has established two relevant and urgent concerns:

  • Marginalization of conservative perspectives in the viewpoints generated by LLMs: “Models trained on the internet alone tend to be biased toward less educated, lower income, or conservative points of view. Newer models, on the other hand, further refined through curated human feedback tend to be biased toward more liberal, higher educated, and higher income audiences” (Myers, 2023). For example, the latest GPT model (GPT-4) relies on and uses RLHF (Reinforcement Learning from Human Feedback) (Malhotra, 2023). Because of this, conservatives have voiced and raised concerns about the potential bias displayed in the output of generative AI. ChatGPT has been found to display a “pro-environmental, left-libertarian ideology” (Hartmann et al., 2023). By asking questions related to the political compass test, Rutinowski et al. (2023) reached a similar conclusion: “ChatGPT seems to hold a bias towards progressive views” (p. 1). A study by Santurkar et al. (2023) also determined that the perspectives expressed by LLMs sharply diverged from those held by various demographic groups in the United States.
  • Dated and dangerous stereotypes based on gender and race, especially in images generated by AI: These concerns about bias also extend to artificially generated images. A Bloomberg analysis (Nicoletti & Bass, 2023) found that Stable Diffusion, an image generator driven by artificial intelligence, placed men with lighter skin tones in higher-paying jobs and women and individuals with darker skin tones in lower-paying or domestic jobs. Social stratification in these images was found to be higher than exists in the data from the Bureau of Labor Statistics. In addition, “more than 80% of the images generated for the keyword ‘inmate’ were of people with darker skin, even though people of color make up less than half of the US prison population, according to the Federal Bureau of Prisons” (Nicoletti & Bass, 2023).

The research has established that generative AI may create content with bias across multiple modalities. Therefore, faculty should draw students’ attention to these issues and invite students to appraise and review generative AI’s output for biased viewpoints or inaccurate and harmful stereotypes. Students should approach the content they receive from generative AI with a critical and socially engaged eye. With conservatives often expressing concerns about their views being marginalized on college campuses and with students becoming more diverse and non-traditional as society changes, it is important for us to treat them all with respect and to acknowledge when generative AI may fail them.

Scenario

You teach an online course on constitutional law for the University of Missouri School of Law. You have asked students to select a clause within a constitutional amendment of their choice, generate a hypothetical legal case, and then compose a corresponding legal brief describing the relevant precedents associated with the clause and/or amendment and established by the United States Supreme Court. This legal brief must cite specific decisions, discuss how the Supreme Court’s understanding of the clause and the amendment may have evolved over time, and develop an argument about how this understanding applies to and informs the case at hand.

You receive a legal brief from a student about the Equal Protection Clause of the Fourteenth Amendment, discussing how its role in society has expanded from its passage to the present day. However, as you review the brief, you notice something: In addition to including cases like Brown v. Board of Education, Obergefell v. Hodges, and others, the legal brief contains citations for cases you do not recognize at all. You begin doing research and realize these cases do not exist. They never happened. Concerned, you confront the student. Once you do so, the student admits that she relied on ChatGPT to write a good portion of the legal brief and did not bother to check if all the included cases were actually real.

You decide that while the student will lose points on the “relevant case law” section of the rubric, this could serve as a learning experience for her and others. What would you like to tell her and her fellow students?

Note: Actual events inspired this scenario.