Proceedings of the National Academy of Sciences (PNAS)<\/em>, could be useful in both informing AI literacy training, and help developers improve LLMs for a wider range of uses.<\/p>\nWhile LLMs are becoming increasingly powerful, they can also make mistakes and provide incorrect information, which could have negative consequences as these systems become more integrated into our everyday lives.<\/p>\n
\u201cLLMs have become wildly popular, and evaluating their performance in a quantitative way is important, but we also need to evaluate how well these systems work with and can support people,\u201d said co-first author Albert Jiang, from Cambridge\u2019s Department of Computer Science and Technology. \u201cWe don\u2019t yet have comprehensive ways of evaluating an LLM\u2019s performance when interacting with humans.\u201d<\/p>\n
The standard way to evaluate LLMs relies on static pairs of inputs and outputs, which disregards the interactive nature of chatbots, and how that changes their usefulness in different scenarios. The researchers developed CheckMate to help answer these questions, designed for but not limited to applications in mathematics.<\/p>\n
\u201cWhen talking to mathematicians about LLMs, many of them fall into one of two main camps: either they think that LLMs can produce complex mathematical proofs on their own, or that LLMs are incapable of simple arithmetic,\u201d said co-first author Katie Collins from the Department of Engineering. \u201cOf course, the truth is probably somewhere in between, but we wanted to find a way of evaluating which tasks LLMs are suitable for and which they aren\u2019t.\u201d<\/p>\n
The researchers recruited 25 mathematicians, from undergraduate students to senior professors, to interact with three different LLMs (InstructGPT, ChatGPT, and GPT-4) and evaluate their performance using CheckMate. Participants worked through undergraduate-level mathematical theorems with the assistance of an LLM and were asked to rate each individual LLM response for correctness and helpfulness. Participants did not know which LLM they were interacting with.<\/p>\n
The researchers recorded the sorts of questions asked by participants, how participants reacted when they were presented with a fully or partially incorrect answer, whether and how they attempted to correct the LLM, or if they asked for clarification. Participants had varying levels of experience with writing effective prompts for LLMs, and this often affected the quality of responses that the LLMs provided.<\/p>\n
An example of an effective prompt is \u201cwhat is the definition of X\u201d (X being a concept in the problem) as chatbots can be very good at retrieving concepts they know of and explaining it to the user.<\/p>\n
\u201cOne of the things we found is the surprising fallibility of these models,\u201d said Collins. \u201cSometimes, these LLMs will be really good at higher-level mathematics, and then they\u2019ll fail at something far simpler. It shows that it\u2019s vital to think carefully about how to use LLMs effectively and appropriately.\u201d<\/p>\n
However, like the LLMs, the human participants also made mistakes. The researchers asked participants to rate how confident they were in their own ability to solve the problem they were using the LLM for. In cases where the participant was less confident in their own abilities, they were more likely to rate incorrect generations by LLM as correct.<\/p>\n
\u201cThis kind of gets to a big challenge of evaluating LLMs, because they\u2019re getting so good at generating nice, seemingly correct natural language, that it\u2019s easy to be fooled by their responses,\u201d said Jiang. \u201cIt also shows that while human evaluation is useful and important, it\u2019s nuanced, and sometimes it\u2019s wrong. Anyone using an LLM, for any application, should always pay attention to the output and verify it themselves.\u201d<\/p>\n
Based on the results from CheckMate, the researchers say that newer generations of LLMs are increasingly able to collaborate helpfully and correctly with human users on undergraduate-level maths problems, as long as the user can assess the correctness of LLM-generated responses. Even if the answers may be memorised and can be found somewhere on the internet, LLMs have the advantage of being flexible in their inputs and outputs over traditional search engines (though should not replace search engines in their current form).<\/p>\n
While CheckMate was tested on mathematical problems, the researchers say their platform could be adapted to a wide range of fields. In the future, this type of feedback could be incorporated into the LLMs themselves, although none of the CheckMate feedback from the current study has been fed back into the models.<\/p>\n
\u201cThese kinds of tools can help the research community to have a better understanding of the strengths and weaknesses of these models,\u201d said Collins. \u201cWe wouldn\u2019t use them as tools to solve complex mathematical problems on their own, but they can be useful assistants if the users know how to take advantage of them.\u201d<\/p>\n
The research was supported in part by the Marshall Commission, the Cambridge Trust, Peterhouse, Cambridge, The Alan Turing Institute, the European Research Council, and the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).<\/p>\n
Reference:
<\/em>Katherine M. Collins, Albert Q. Jiang, et al. \u2018<\/em>Evaluating Language Models for Mathematics through Interactions<\/em><\/a>.\u2019 <\/em>Proceedings of the National Academy of Sciences (2024). DOI: 10.1073\/pnas.2318124121<\/em><\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":"
Researchers have developed a platform for the interactive evaluation of AI-powered chatbots such as ChatGPT A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The\u2026<\/p>\n","protected":false},"author":2,"featured_media":18301,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_scj_primary_category_id":55,"_scj_featured":false,"_scj_featured_from":"","_scj_featured_until":"","_scj_featured_order":0,"_scj_visibility_class":"current","_scj_layout_family":"standard","_scj_article_style":"","_scj_display_overrides":[],"_scj_intro_image_id":18301,"_scj_intro_alt":"New open-source platform allows users to evaluate performance of AI-powered chatbots","_scj_intro_caption":"","_scj_intro_class":"","_scj_intro_float":"","_scj_full_image_id":18301,"_scj_full_alt":"New open-source platform allows users to evaluate performance of AI-powered chatbots","_scj_full_caption":"","_scj_full_class":"","_scj_full_float":"","_scj_media_type":"","_scj_media_provider":"","_scj_media_external_id":"","_scj_media_url":"","_scj_media_poster_id":0,"_scj_media_width":0,"_scj_media_height":0,"_scj_media_aspect_ratio":"","_scj_media_description":"","_scj_gallery_items":[],"_scj_related_post_ids":[],"_scj_additional_authors":[],"scj_layout_family":"standard","scj_media_provider":"","scj_media_url":"","scj_full_caption":"","footnotes":""},"categories":[55],"tags":[],"class_list":["post-18302","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence"],"yoast_head":"\n
New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com<\/title>\n<meta name=\"description\" content=\"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com\" \/>\n<meta property=\"og:description\" content=\"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\" \/>\n<meta property=\"og:site_name\" content=\"thesmartcityjournal.com\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/thesmartcityjournal\" \/>\n<meta property=\"article:published_time\" content=\"2024-06-06T04:36:27+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-02-20T11:16:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"800\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Redacci\u00f3n de The Smart City Journal\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SmartCityJour_\" \/>\n<meta name=\"twitter:site\" content=\"@SmartCityJour_\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Redacci\u00f3n de The Smart City Journal\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\"},\"author\":{\"@type\":\"Organization\",\"name\":\"Redacci\u00f3n de The Smart City Journal\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/author\\\/smart-city\"},\"headline\":\"New open-source platform allows users to evaluate performance of AI-powered chatbots\",\"datePublished\":\"2024-06-06T04:36:27+00:00\",\"dateModified\":\"2025-02-20T11:16:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\"},\"wordCount\":1037,\"publisher\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/CheckMate.jpg\",\"articleSection\":[\"Artificial Intelligence\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\",\"name\":\"New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/CheckMate.jpg\",\"datePublished\":\"2024-06-06T04:36:27+00:00\",\"dateModified\":\"2025-02-20T11:16:10+00:00\",\"description\":\"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/CheckMate.jpg\",\"contentUrl\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/CheckMate.jpg\",\"width\":1280,\"height\":800,\"caption\":\"New open-source platform allows users to evaluate performance of AI-powered chatbots\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/artificial-intelligence\\\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"New open-source platform allows users to evaluate performance of AI-powered chatbots\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#website\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\",\"name\":\"thesmartcityjournal.com\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Organization\",\"NewsMediaOrganization\"],\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#organization\",\"name\":\"thesmartcityjournal.com\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/logo-smart-city-journal-1.png\",\"contentUrl\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/logo-smart-city-journal-1.png\",\"width\":346,\"height\":91,\"caption\":\"thesmartcityjournal.com\"},\"image\":{\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/thesmartcityjournal\",\"https:\\\/\\\/x.com\\\/SmartCityJour_\",\"https:\\\/\\\/www.instagram.com\\\/thesmartcityjournal_\\\/\",\"https:\\\/\\\/www.tiktok.com\\\/@thesmartcityjournal\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/the-smart-city-journal\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCIPTMzK206pVFxMNjO7SSxw\"]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#\\\/schema\\\/person\\\/39c9da1aeac1b5c2d179ef9e4e293cf4\",\"name\":\"Redacci\u00f3n de The Smart City Journal\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/?s=96&d=mm&r=g\",\"caption\":\"Redacci\u00f3n de The Smart City Journal\"},\"description\":\"Equipo de redacci\u00f3n de The Smart City Journal.\",\"url\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en\\\/author\\\/smart-city\"},{\"@type\":\"SiteNavigationElement\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#site-navigation\",\"cssSelector\":[\".cib-main-menu\"]},{\"@type\":\"WPHeader\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#header\",\"cssSelector\":[\".cib-header\"]},{\"@type\":\"WPFooter\",\"@id\":\"https:\\\/\\\/www.thesmartcityjournal.com\\\/en#footer\",\"cssSelector\":[\".cib-footer\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com","description":"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots","og_locale":"en_US","og_type":"article","og_title":"New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com","og_description":"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well","og_url":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots","og_site_name":"thesmartcityjournal.com","article_publisher":"https:\/\/www.facebook.com\/thesmartcityjournal","article_published_time":"2024-06-06T04:36:27+00:00","article_modified_time":"2025-02-20T11:16:10+00:00","og_image":[{"width":1280,"height":800,"url":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg","type":"image\/jpeg"}],"author":"Redacci\u00f3n de The Smart City Journal","twitter_card":"summary_large_image","twitter_creator":"@SmartCityJour_","twitter_site":"@SmartCityJour_","twitter_misc":{"Written by":"Redacci\u00f3n de The Smart City Journal","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#article","isPartOf":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots"},"author":{"@type":"Organization","name":"Redacci\u00f3n de The Smart City Journal","url":"https:\/\/www.thesmartcityjournal.com\/en\/author\/smart-city"},"headline":"New open-source platform allows users to evaluate performance of AI-powered chatbots","datePublished":"2024-06-06T04:36:27+00:00","dateModified":"2025-02-20T11:16:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots"},"wordCount":1037,"publisher":{"@id":"https:\/\/www.thesmartcityjournal.com\/en#organization"},"image":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage"},"thumbnailUrl":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg","articleSection":["Artificial Intelligence"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots","url":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots","name":"New open-source platform allows users to evaluate performance of AI-powered chatbots - thesmartcityjournal.com","isPartOf":{"@id":"https:\/\/www.thesmartcityjournal.com\/en#website"},"primaryImageOfPage":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage"},"image":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage"},"thumbnailUrl":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg","datePublished":"2024-06-06T04:36:27+00:00","dateModified":"2025-02-20T11:16:10+00:00","description":"A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs). The researchers tested CheckMate in an experiment where human participants used three LLMs \u2013 InstructGPT, ChatGPT and GPT-4 \u2013 as assistants for solving undergraduate-level mathematics problems. The team studied how well","breadcrumb":{"@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#primaryimage","url":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg","contentUrl":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2024\/06\/CheckMate.jpg","width":1280,"height":800,"caption":"New open-source platform allows users to evaluate performance of AI-powered chatbots"},{"@type":"BreadcrumbList","@id":"https:\/\/www.thesmartcityjournal.com\/en\/artificial-intelligence\/new-open-source-platform-allows-users-to-evaluate-performance-of-ai-powered-chatbots#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.thesmartcityjournal.com\/en"},{"@type":"ListItem","position":2,"name":"New open-source platform allows users to evaluate performance of AI-powered chatbots"}]},{"@type":"WebSite","@id":"https:\/\/www.thesmartcityjournal.com\/en#website","url":"https:\/\/www.thesmartcityjournal.com\/en","name":"thesmartcityjournal.com","description":"","publisher":{"@id":"https:\/\/www.thesmartcityjournal.com\/en#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.thesmartcityjournal.com\/en?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Organization","NewsMediaOrganization"],"@id":"https:\/\/www.thesmartcityjournal.com\/en#organization","name":"thesmartcityjournal.com","url":"https:\/\/www.thesmartcityjournal.com\/en","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.thesmartcityjournal.com\/en#\/schema\/logo\/image\/","url":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2026\/09\/logo-smart-city-journal-1.png","contentUrl":"https:\/\/www.thesmartcityjournal.com\/wp-content\/uploads\/2026\/09\/logo-smart-city-journal-1.png","width":346,"height":91,"caption":"thesmartcityjournal.com"},"image":{"@id":"https:\/\/www.thesmartcityjournal.com\/en#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/thesmartcityjournal","https:\/\/x.com\/SmartCityJour_","https:\/\/www.instagram.com\/thesmartcityjournal_\/","https:\/\/www.tiktok.com\/@thesmartcityjournal","https:\/\/www.linkedin.com\/company\/the-smart-city-journal\/","https:\/\/www.youtube.com\/channel\/UCIPTMzK206pVFxMNjO7SSxw"]},{"@type":"Organization","@id":"https:\/\/www.thesmartcityjournal.com\/en#\/schema\/person\/39c9da1aeac1b5c2d179ef9e4e293cf4","name":"Redacci\u00f3n de The Smart City Journal","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","caption":"Redacci\u00f3n de The Smart City Journal"},"description":"Equipo de redacci\u00f3n de The Smart City Journal.","url":"https:\/\/www.thesmartcityjournal.com\/en\/author\/smart-city"},{"@type":"SiteNavigationElement","@id":"https:\/\/www.thesmartcityjournal.com\/en#site-navigation","cssSelector":[".cib-main-menu"]},{"@type":"WPHeader","@id":"https:\/\/www.thesmartcityjournal.com\/en#header","cssSelector":[".cib-header"]},{"@type":"WPFooter","@id":"https:\/\/www.thesmartcityjournal.com\/en#footer","cssSelector":[".cib-footer"]}]}},"_links":{"self":[{"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/posts\/18302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/comments?post=18302"}],"version-history":[{"count":0,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/posts\/18302\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/media\/18301"}],"wp:attachment":[{"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/media?parent=18302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/categories?post=18302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thesmartcityjournal.com\/en\/wp-json\/wp\/v2\/tags?post=18302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}