TÜV NORD GROUP makes LLMs/GPTs safer with new test method

TÜVIT, a TÜV NORD GROUP company, develops test methods for Large Language Models (LLMs), which are used in common generative pretrained transformers (GPTs). Since the introduction of ChatGPT, generative artificial intelligence (AI) applications have been spreading rapidly. According to a Forsa survey commissioned by the TÜV Association, a good one in three people in Germany have already used ChatGPT. With the development of test methods, TÜVIT is making an important contribution to implementing the upcoming regulations from the European AI Act for developers and users.

“Potential risks of large language models are uncertain reliability, the poor quality of outputs and misleading and incorrect information from such AI systems. This creates the possibility of false information being disseminated,” explains Dirk Kretzschmar, Managing Director of TÜVIT. There is also a risk of bias in corresponding models if the training data is not checked appropriately. This mainly concerns racist or sexist output from AI systems. “Such risks could lead to social consequences as soon as such unchecked models are used by a global population,” continues Kretzschmar. Large language models are also exposed to attacks. These include privacy attacks, evasion attacks and poisoning attacks. These attacks are aimed at extracting personal data, misdirecting the model or deliberately introducing false data into the model.

To counteract these challenges, TÜVIT is developing suitable methods based on the latest research to test corresponding applications before they are used. “We don't test what such systems can do particularly well, but where their weak points lie,” says Kretzschmar. A large language model always has a specific application for the customer. That is why the experts at TÜVIT define the criteria to be tested according to customer requirements and the necessary regulatory requirements (EU AI Act). They then determine the relevant areas and potential attacks. Based on this, the auditors create data sets and methods and look at the quality of the answers and possible biases. To do this, they carry out a comprehensive data analysis that includes public and customer-specific data sets as well as specially generated attacks. This is similar to traditional pen testing (penetration testing), in which the TÜVIT experts continuously try to penetrate the system and identify vulnerabilities by carrying out various attacks. “We strive to minimize the risks of large language models through comprehensive testing. This is an important building block for the implementation of future regulations and also strengthens confidence in this new technology,” explains Kretzschmar.

Founded over 150 years ago, we stand for security and trust worldwide. As a knowledge company, we have our sights firmly set on the digital future. Whether engineers, IT security experts or specialists for the mobility of the future: in more than 100 countries, we ensure that our customers become even more successful in the networked world.