ChatGPT may be learning personality from language alone

An Israeli study found that ChatGPT can predict how groups of people will respond to personality-test questions before they take the tests.This could help psychologists develop personality assessments more quickly and suggests that artificial intelligence (AI) models trained on human language can capture meaningful patterns in human behaviour.

The study did not show that ChatGPT understands individuals or can diagnose mental illness. Instead, it showed that an LLM could predict population-level patterns before collecting the human data needed to test those predictions.

“Given that personality traits are reflected in language, LLMs may have learned the structure of human personality as a natural byproduct of their training,” said lead author Dr. Rotem Monsa of the Hebrew University of Jerusalem. “So, while they were not taught specifically psychology or personality theories, these are already embedded in the language that LLMs learn from,” he said.

Testing ChatGPT’s Predictions

To test the idea, the researchers used GPT-4 to generate two personality questionnaires from very different sources. One was based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), a widely used reference for diagnosing mental disorders. The other was based on an astrology textbook, which the researchers selected because its descriptions of personality are not scientifically validated.

Both texts were converted into statements that participants rated on a scale from strongly disagree to strongly agree.

“We wanted to choose texts that describe human personality in very rich detail but also sit on opposite ends of a spectrum in terms of scientific grounding,” Monsa explained. “The DSM-5 was refined through decades of clinical research and is the standard diagnostic manual in clinical psychiatry, known worldwide. The astrology text is very culturally based but not scientifically validated.”

Before administering the questionnaires, the researchers asked GPT-4 to predict how people would respond to them. They then gave the questionnaires to 600 participants, alongside the Big Five Inventory (BFI), a widely validated assessment of five broad personality traits that served as a benchmark.

The participants’ responses were broadly consistent with the model’s predictions.

The DSM-5-based questionnaire showed strong internal consistency, meaning its questions tended to measure related aspects of personality. Its results also resembled those of the BFI, supporting its ability to capture meaningful patterns in personality.

The astrology-based questionnaire, by contrast, showed weak consistency among its purported personality traits.

“Our data suggest that the astrological elements don’t reflect coherent psychological dimensions,” Monsa said. “Personality traits that were together in, for example, the fire elements don’t actually go together in the population.”

Despite that, both questionnaires were able to predict outcomes including depression, anxiety and well-being at levels comparable to the BFI. The finding does not validate astrology. Rather, it suggests that ChatGPT was able to extract personality-related information from the language used to describe people, even when the underlying source was not scientifically grounded.

“The fact that LLMs can predict human response patterns before seeing any human data suggests that these models have observed something generally meaningful about human psychology,” Monsa said.

The researchers cautioned that the findings may not transfer directly to other languages and cultures because LLMs have been trained predominantly on English-language material from Western societies.

“We would assume that in other languages and cultures, the results will be not as strong as we saw here,” Monsa said.

The study was published in the peer-reviewed journal iScience.