AI involved in crafting questions for California Bar exam

    0
    0

    The California State Bar has revealed that some multiple-choice questions in a troubled bar exam were crafted with the help of artificial intelligence. In a news announcement on Monday, the legal licensing authority stated its intention to request that the California Supreme Court make adjustments to the exam scores for individuals who participated in the February bar exam.

    “The mishap surrounding the February 2025 bar exam is worse than anticipated,” expressed Mary Basick, assistant dean of academic skills at the University of California, Irvine, Law School. “It’s almost unbelievable to think questions were created by non-experts using artificial intelligence.”

    In February, the new format of the exam led to numerous complaints from test-takers. Many participants faced challenges in completing the bar exam due to online platform crashes, which occurred even before some had begun the test. Others encountered difficulties in finishing and saving their essays, experienced screen lags and error messages, and were unable to copy and paste text.

    According to recent information from the State Bar, out of 171 scored multiple-choice questions, 100 were provided by Kaplan, and 48 were sourced from a first-year law students exam. A smaller segment of 23 questions were formulated by ACS Ventures, the State Bar’s psychometrician, with the assistance of artificial intelligence.

    Leah Wilson, executive director of the State Bar, expressed confidence in the effectiveness of the multiple-choice questions to reliably and fairly evaluate the legal proficiency of the candidates.

    Katie Moran, an associate professor at the University of San Francisco School of Law known for her expertise in bar exam preparation, referred to the disclosure as “a staggering admission.” “The State Bar has acknowledged they engaged a company for a non-expert to use AI in drafting the questions that appeared on the bar exam,” she said. “Moreover, they paid the same company to review and eventually approve the questions, including those created by the company.”

    Andrew Perlman, dean of Suffolk University Law School and a member of the advisory council for the American Bar Association Task Force on the Law and Artificial Intelligence, mentioned that he too had not come across the usage of AI for crafting bar exam questions nor any set rules regulating such uses.

    Nonetheless, he wasn’t surprised considering the rapid evolution of AI technology. Perlman noted that AI could be beneficial in formulating questions for assessments but emphasized the importance of rigorous expert review of AI-generated content. He anticipates the use of AI in such applications to expand.

    While there may currently be some skepticism regarding the integration of AI technology within the legal field, Perlman predicted that in the future, there would be concerns over the capabilities of lawyers who choose not to utilize these technological tools.