Process Modeling with Large Language Models: Towards a Fine-Grained Evaluation Framework

Process Modeling with Large Language Models: Towards a Fine-Grained Evaluation Framework

Speaker: Alexis Brissard

Topics: Process Modeling with LLMs & ICL, Entity Disambiguation

Abstract

Process Modeling (PMo) consists in constructing process models from process descriptions in natural language. Two main approaches have been developed to automate this complex and time-intensive task leveraging Large Language Models (LLMs): Process Information Extraction (PIE) and Process Model Generation (PMG). However, they both have limitations and the field lacks a standardized and comprehensive evaluation framework to assess the real capabilities and compare these methods. This report presents 2 main contributions: a detailed comparison of Process Model Representations (PMRs) for PMo and a new method integrating PIE and PMG to provide a fine-grained evaluation of the generated process models.