Cegeka boosts Terumo Europe’s regulatory efficiency with conversational AI

Terumo Europe, a leading healthcare company, understands the paramount importance of quality in medical equipment and services. As it wanted to improve efficiency of its regulatory affairs department, the company turned to Cegeka to implement an AI-based solution. Employees can now interrogate the Regulatory Intelligent Assistant about regulatory related documents. 

Terumo is a global healthcare company specializing in medical equipment and services. The company manufactures a wide range of products including catheters, filters, insulin delivery systems, needles, syringes, stents, and surgical products. With “Contributing to Society through Healthcare” as its corporate mission, Terumo is deeply committed to improving the health and well-being of patients through its products and services. 

The healthcare company’s European branch, headquartered in Leuven, has been a long-standing client of Cegeka, relying on them for its infrastructure needs as well as Microsoft analytics development and support. “It was a logical step for us to extend our collaboration with Cegeka and investigate possible gains from AI,” explains Kristof Geukens, ICT Applications Manager Analytics & AI at Terumo Europe.

Interrogating regulatory documents 

Inspired by one of Cegeka’s webinars about AI, Kristof Geukens started thinking about a potential business case. “We came up with a business case for the regulatory affairs department, aiming to automate repeatable complex tasks to facilitate our employees’ day-to-day operations,” he elaborates. “Together with our regulatory affairs intelligence specialist Darya Sayevich, we decided to concentrate on knowledge discovery and proposed three specific use cases with measurable objectives that we could evaluate.”

Within the regulatory affairs department, employees need to explore many different data sources and compare and combine documents about medical device regulation. This is complicated further by continuous evolution of these regulations. Therefore, Terumo Europe came up with three use cases to automate using a conversational interface. “In a first use case, we sought to enable our employees to inquire about regulations in different countries,” says Kristof Geukens. “Another use case concentrated on identifying changes between current and prior versions of regulations. Additionally, we wanted to be able to ask questions about regulations available only in foreign languages.”

Cegeka developed a conversational interface for the department’s documents, allowing personnel to ‘chat’ with this knowledge repository. The project was called Regulatory Intelligent Assistant (RIA). Terumo Europe has full control over which documents are incorporated into RIA, emphasizes Kristof Geukens: “We are always able to add new documents without depending on Cegeka.” Another important feature is that the assistant’s answers are annotated with links to the relevant sections in the source documents. “This cultivates trust in the assistant, because our employees can always verify its answers.”

Large language models commonly suffer from a flaw known as ‘hallucinating’, where they generate factually incorrect or nonsensical answers. However, RIA doesn’t have this tendency to hallucinate, because it’s trained solely on regulatory affairs documents and has a well defined regulatory context. As a result, the assistant doesn’t respond to inquiries beyond its domain.

“The annotations with links to the source documents aim to cultivate trust in the assistant, because our employees can always verify its answers.”

KRISTOF GEUKENS, ICT APPLICATIONS MANAGER ANALYTICS & AI AT TERUMO EUROPE

Learning on the go 

The initial phase was deployed to establish a proof of value. “The goal of this phase was to decide on the required technologies for implementing the three use cases,” explains Kristof Geukens. Cegeka developed each use case incrementally, each time gleaning insights from the previous use case. “Our business people were closely involved from the start, and as a result Cegeka was able to implement the project quickly.” 

While testing the proof of value implementations, it became clear that the assistant had some limitations. “RIA worked quite well for simple questions, but the answers on more complex questions weren’t accurate or complete enough,” says Kristof Geukens. After investigating this and discussing the results with Cegeka, it turned out that the issue was rooted in the limited context of the large language model used, GPT-3.5. Consequently, the model was replaced with GPT-4 Turbo for the production version. “I’m glad that Cegeka was always transparent about the models and technologies they were using, which made it easier for us to understand the limitations we experienced and how to overcome them.”

Feeding the data into the model posed another challenge, explains Kristof Geukens. “We initially attempted to incorporate some data from a huge spreadsheet into the model. This seemed logical at that time, but it turned out to require a lot of manual work to get the data in the right format. The lesson we learned there was to prefer PDF documents with natural language over tables in Excel files.” 

A digital assistant for all 

Upon completion of user acceptance testing, Kristof Geukens doesn’t anticipate any big problems in change management: “Initial feedback is positive. RIA is integrated in the user’s workflow as a chat box on a web page. Employees can easily ask questions and get answers with links to the relevant documents. They can also rate responses with a thumbs up or down, which we can analyze later for future improvements. Of course, the more specific your question, the more specific the answer you’ll get. Just like we all had to learn to use search engines like Google more than twenty years ago, employees will now have to learn to use the digital assistant efficiently.” 

Kristof Geukens is already considering the technology’s potential benefits in other departments. “The sales and marketing team has expressed interest in a digital assistant, and I also received questions from the legal department. Of course, each use case has its specific requirements. So when RIA is finished, we’ll first try to find a use case that ties in with what we have now to scale it up without too many changes. However, it’s my firm belief that digital assistants are the future. A time will come when everyone at Terumo will be used to working with a digital assistant.”

“A time will come when everyone at Terumo will be used to working with a digital assistant.”

KRISTOF GEUKENS, ICT APPLICATIONS MANAGER ANALYTICS & AI AT TERUMO EUROPE