MOTIV

Project Description

Im vom Bayerischen Forschungsinstitut für Digitale Transformation (bidt) geförderten Projekt „MOTIV - Digital Interaction Literacy: Monitor, Training, and Visibility” (Projektwebsite) analysiert das interdisziplinäre Team die Interaktion mit sprachbasierten interaktiven Technologien und KI-Anwendungen und erfasst Nutzungsgewohnheiten sowie vorherrschende (Fehl-)Vorstellungen über die Systeme. Dabei wird die „digital interaction literacy“ als eine digitale Schlüsselkompetenz für einen selbstbestimmten, kompetenten Umgang mit digitalen Technologien erkannt. Mit Blick auf heterogene Gesellschaftsgruppen entwickelt MOTIV adaptive Trainingsmodule, um das Bewusstsein der unterschiedlichen Akteur*innen zu schärfen sowie Entwicklungspotentiale zu identifizieren und auszubauen.

MOTIV ist ein Gemeinschaftsprojekt zwischen Astrid Carolus (Medienpsychologie), Carolin Wienrich (Psychologie Intelligenter Interaktiver Systeme) und Andreas Hotho (Data Science) und hat eine Laufzeit von drei Jahren und wird mit ca. 760.000 Euro gefördert.

Zeitraum:	2020 - 2023
Fördergeber:	Bayerischen Forschungsinstitut für Digitale Transformation (bidt)

Publications

2024[ to top ]

OtterlyObsessedWithSemantics at SemEval-2024 Task 4: Developing a Hierarchical Multi-Label Classification Head for Large Language Models. Wunderle, Julia; Schubert, Julian; Cacciatore, Antonella; Zehe, Albin; Pfister, Jan; Hotho, Andreas. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), A. K. Ojha, A. S. Dougruöz, H. Tayyar Madabushi, G. Da San Martino, S. Rosenthal, A. Rosá (eds.), pp. 602–612. Association for Computational Linguistics, Mexico City, Mexico, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
For our submission for Subtask 1, we developed a custom classification head that is designed to be applied atop of a Large Language Model. We reconstructed the hierarchy across multiple fully connected layers, allowing us to incorporate previous foundational decisions in subsequent, more fine-grained layers. To find the best hyperparameters, we conducted a grid-search and to compete in the multilingual setting, we translated all documents to English.

@inproceedings{wunderle-etal-2024-otterlyobsessedwithsemantics, abstract = {For our submission for Subtask 1, we developed a custom classification head that is designed to be applied atop of a Large Language Model. We reconstructed the hierarchy across multiple fully connected layers, allowing us to incorporate previous foundational decisions in subsequent, more fine-grained layers. To find the best hyperparameters, we conducted a grid-search and to compete in the multilingual setting, we translated all documents to English.}, address = {Mexico City, Mexico}, author = {Wunderle, Julia and Schubert, Julian and Cacciatore, Antonella and Zehe, Albin and Pfister, Jan and Hotho, Andreas}, booktitle = {Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)}, editor = {Ojha, Atul Kr. and Do{\u{g}}ru{\"o}z, A. Seza and Tayyar Madabushi, Harish and Da San Martino, Giovanni and Rosenthal, Sara and Ros{\'a}, Aiala}, keywords = {author:pfister}, month = {06}, pages = {602–612}, publisher = {Association for Computational Linguistics}, title = {{O}tterly{O}bsessed{W}ith{S}emantics at {S}em{E}val-2024 Task 4: Developing a Hierarchical Multi-Label Classification Head for Large Language Models}, year = 2024 }
Pollice Verso at SemEval-2024 Task 6: The Roman Empire Strikes Back. Kobs, Konstantin; Pfister, Jan; Hotho, Andreas. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), A. K. Ojha, A. S. Dougruöz, H. Tayyar Madabushi, G. Da San Martino, S. Rosenthal, A. Rosá (eds.), pp. 1529–1536. Association for Computational Linguistics, Mexico City, Mexico, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
We present an intuitive approach for hallucination detection in LLM outputs that is modeled after how humans would go about this task. We engage several LLM ``experts'' to independently assess whether a response is hallucinated. For this we select recent and popular LLMs smaller than 7B parameters. By analyzing the log probabilities for tokens that signal a positive or negative judgment, we can determine the likelihood of hallucination. Additionally, we enhance the performance of our ``experts'' by automatically refining their prompts using the recently introduced OPRO framework. Furthermore, we ensemble the replies of the different experts in a uniform or weighted manner, which builds a quorum from the expert replies. Overall this leads to accuracy improvements of up to 10.6 p.p. compared to the challenge baseline. We show that a Zephyr 3B model is well suited for the task. Our approach can be applied in the model-agnostic and model-aware subtasks without modification and is flexible and easily extendable to related tasks.

@inproceedings{kobs-etal-2024-pollice, abstract = {We present an intuitive approach for hallucination detection in LLM outputs that is modeled after how humans would go about this task. We engage several LLM {``}experts{''} to independently assess whether a response is hallucinated. For this we select recent and popular LLMs smaller than 7B parameters. By analyzing the log probabilities for tokens that signal a positive or negative judgment, we can determine the likelihood of hallucination. Additionally, we enhance the performance of our {``}experts{''} by automatically refining their prompts using the recently introduced OPRO framework. Furthermore, we ensemble the replies of the different experts in a uniform or weighted manner, which builds a quorum from the expert replies. Overall this leads to accuracy improvements of up to 10.6 p.p. compared to the challenge baseline. We show that a Zephyr 3B model is well suited for the task. Our approach can be applied in the model-agnostic and model-aware subtasks without modification and is flexible and easily extendable to related tasks.}, address = {Mexico City, Mexico}, author = {Kobs, Konstantin and Pfister, Jan and Hotho, Andreas}, booktitle = {Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)}, editor = {Ojha, Atul Kr. and Do{\u{g}}ru\"oz, A. Seza and Tayyar Madabushi, Harish and Da San Martino, Giovanni and Rosenthal, Sara and Ros\'a, Aiala}, keywords = {author:pfister}, month = {06}, pages = {1529–1536}, publisher = {Association for Computational Linguistics}, title = {Pollice Verso at SemEval-2024 Task 6: The Roman Empire Strikes Back}, year = 2024 }

2023[ to top ]

Pointer Networks: A Unified Approach to Extracting German Opinions. Wunderle, Julia; Pfister, Jan; Hotho, Andreas. In Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023), M. Georges, A. Herygers, A. Friedrich, B. Roth (eds.), pp. 127–138. Association for Computational Lingustics, Ingolstadt, Germany, 2023.
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
@inproceedings{wunderle-etal-2023-pointer, address = {Ingolstadt, Germany}, author = {Wunderle, Julia and Pfister, Jan and Hotho, Andreas}, booktitle = {Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023)}, editor = {Georges, Munir and Herygers, Aaricia and Friedrich, Annemarie and Roth, Benjamin}, keywords = {from:janpf}, month = {09}, pages = {127–138}, publisher = {Association for Computational Lingustics}, title = {Pointer Networks: A Unified Approach to Extracting {G}erman Opinions}, year = 2023 }
Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions. Wangsadirdja, Dirk; Pfister, Jan; Kobs, Konstantin; Hotho, Andreas. In Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pp. 1090–1095. Association for Computational Linguistics, Toronto, Canada, 2023.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023. The core idea behind our system is to leverage pre-trained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any task-specific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QA-capable models. Also, to fit as much relevant context into the model's limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text.Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.

@inproceedings{wangsadirdja-etal-2023-jack, abstract = {In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023. The core idea behind our system is to leverage pre-trained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any task-specific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QA-capable models. Also, to fit as much relevant context into the model's limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text.Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.}, address = {Toronto, Canada}, author = {Wangsadirdja, Dirk and Pfister, Jan and Kobs, Konstantin and Hotho, Andreas}, booktitle = {Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023)}, keywords = {from:janpf}, month = {07}, pages = {1090–1095}, publisher = {Association for Computational Linguistics}, title = {Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions}, year = 2023 }
Long-Term Effects of Perceived Friendship with Intelligent Voice Assistants on Usage Behavior, User Experience, and Social Perceptions. Wienrich, Carolin; Carolus, Astrid; Markus, André; Augustin, Yannik; Pfister, Jan; Hotho, Andreas. In Computers, 12(4). 2023.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
Social patterns and roles can develop when users talk to intelligent voice assistants (IVAs) daily. The current study investigates whether users assign different roles to devices and how this affects their usage behavior, user experience, and social perceptions. Since social roles take time to establish, we equipped 106 participants with Alexa or Google assistants and some smart home devices and observed their interactions for nine months. We analyzed diverse subjective (questionnaire) and objective data (interaction data). By combining social science and data science analyses, we identified two distinct clusters—users who assigned a friendship role to IVAs over time and users who did not. Interestingly, these clusters exhibited significant differences in their usage behavior, user experience, and social perceptions of the devices. For example, participants who assigned a role to IVAs attributed more friendship to them used them more frequently, reported more enjoyment during interactions, and perceived more empathy for IVAs. In addition, these users had distinct personal requirements, for example, they reported more loneliness. This study provides valuable insights into the role-specific effects and consequences of voice assistants. Recent developments in conversational language models such as ChatGPT suggest that the findings of this study could make an important contribution to the design of dialogic human–AI interactions.

@article{wienrich2023longterm, abstract = {Social patterns and roles can develop when users talk to intelligent voice assistants (IVAs) daily. The current study investigates whether users assign different roles to devices and how this affects their usage behavior, user experience, and social perceptions. Since social roles take time to establish, we equipped 106 participants with Alexa or Google assistants and some smart home devices and observed their interactions for nine months. We analyzed diverse subjective (questionnaire) and objective data (interaction data). By combining social science and data science analyses, we identified two distinct clusters—users who assigned a friendship role to IVAs over time and users who did not. Interestingly, these clusters exhibited significant differences in their usage behavior, user experience, and social perceptions of the devices. For example, participants who assigned a role to IVAs attributed more friendship to them used them more frequently, reported more enjoyment during interactions, and perceived more empathy for IVAs. In addition, these users had distinct personal requirements, for example, they reported more loneliness. This study provides valuable insights into the role-specific effects and consequences of voice assistants. Recent developments in conversational language models such as ChatGPT suggest that the findings of this study could make an important contribution to the design of dialogic human–AI interactions.}, author = {Wienrich, Carolin and Carolus, Astrid and Markus, André and Augustin, Yannik and Pfister, Jan and Hotho, Andreas}, journal = {Computers}, keywords = {from:janpf}, number = 4, title = {Long-Term Effects of Perceived Friendship with Intelligent Voice Assistants on Usage Behavior, User Experience, and Social Perceptions}, volume = 12, year = 2023 }
Higher-Order DeepTrails: Unified Approach to *Trails. Koopmann, Tobias; Pfister, Jan; Markus, André; Carolus, Astrid; Wienrich, Carolin; Hotho, Andreas. In Lernen, Wissen, Daten, Analysen (LWDA) Conference Proceedings, Marburg, Germany, October 9-11, 2023, Vol. 3630 of CEUR Workshop Proceedings, M. Leyer, J. Wichmann (eds.), pp. 372–386. CEUR-WS.org, 2023.
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
@inproceedings{DBLP:conf/lwa/Koopmann23, author = {Koopmann, Tobias and Pfister, Jan and Markus, André and Carolus, Astrid and Wienrich, Carolin and Hotho, Andreas}, booktitle = {Lernen, Wissen, Daten, Analysen {(LWDA)} Conference Proceedings, Marburg, Germany, October 9-11, 2023}, editor = {Leyer, Michael and Wichmann, Johannes}, keywords = {author:pfister}, pages = {372–386}, publisher = {CEUR-WS.org}, series = {{CEUR} Workshop Proceedings}, title = {Higher-Order DeepTrails: Unified Approach to *Trails}, volume = 3630, year = 2023 }

2022[ to top ]

Point me to your Opinion, SenPoi. Pfister, Jan; Wankerl, Sebastian; Hotho, Andreas. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 1313–1323. Association for Computational Linguistics, Seattle, United States, 2022.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
- [ BibSonomy-Post ]
Structured Sentiment Analysis is the task of extracting sentiment tuples in a graph structure commonly from review texts. We adapt the Aspect-Based Sentiment Analysis pointer network BARTABSA to model this tuple extraction as a sequence prediction task and extend their output grammar to account for the increased complexity of Structured Sentiment Analysis. To predict structured sentiment tuples in languages other than English we swap BART for a multilingual mT5 and introduce a novel Output Length Regularization to mitigate overfitting to common target sequence lengths, thereby improving the performance of the model by up to 70%. We evaluate our approach on seven datasets in five languages including a zero shot crosslingual setting.

@inproceedings{pfister-etal-2022-senpoi, abstract = {Structured Sentiment Analysis is the task of extracting sentiment tuples in a graph structure commonly from review texts. We adapt the Aspect-Based Sentiment Analysis pointer network BARTABSA to model this tuple extraction as a sequence prediction task and extend their output grammar to account for the increased complexity of Structured Sentiment Analysis. To predict structured sentiment tuples in languages other than English we swap BART for a multilingual mT5 and introduce a novel Output Length Regularization to mitigate overfitting to common target sequence lengths, thereby improving the performance of the model by up to 70{\%}. We evaluate our approach on seven datasets in five languages including a zero shot crosslingual setting.}, address = {Seattle, United States}, author = {Pfister, Jan and Wankerl, Sebastian and Hotho, Andreas}, booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)}, keywords = {from:janpf}, month = {07}, pages = {1313–1323}, publisher = {Association for Computational Linguistics}, title = {Point me to your Opinion, {S}en{P}oi}, year = 2022 }

Staff

The following persons are involved in this project:

MOTIV

Project Description

Publications

Staff

Hinweis zum Datenschutz

Hinweis zum Datenschutz

Bildnachweise