Large Language Models

• 
  PreAdapter: Pre-training Language Models on Knowledge Graphs. Omeliyanenko, Janna; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2024, to appear. 2024.
  • [ BibTeX ]
  • [ Download ]
  @article{omeliyanenko2024preadapter, author = {Omeliyanenko, Janna and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2024, to appear}, keywords = {selected}, title = {PreAdapter: Pre-training Language Models on Knowledge Graphs}, year = 2024 }

• 
  Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions. Wangsadirdja, Dirk; Pfister, Jan; Kobs, Konstantin; Hotho, Andreas. In Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pp. 1090–1095. Association for Computational Linguistics, Toronto, Canada, 2023.
  • [ Abstract ]
  • [ BibTeX ]
  • [ URL ]
  • [ Download ]
  In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023. The core idea behind our system is to leverage pre-trained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any task-specific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QA-capable models. Also, to fit as much relevant context into the model's limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text.Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.

  @inproceedings{wangsadirdja-etal-2023-jack, abstract = {In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023. The core idea behind our system is to leverage pre-trained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any task-specific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QA-capable models. Also, to fit as much relevant context into the model's limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text.Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.}, address = {Toronto, Canada}, author = {Wangsadirdja, Dirk and Pfister, Jan and Kobs, Konstantin and Hotho, Andreas}, booktitle = {Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023)}, keywords = {from:janpf}, month = {07}, pages = {1090–1095}, publisher = {Association for Computational Linguistics}, title = {Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions}, year = 2023 }
• 
  Large Language Models and Knowledge Graphs: Opportunities and Challenges. Pan, Jeff Z.; Razniewski, Simon; Kalo, Jan-Christoph; Singhania, Sneha; Chen, Jiaoyan; Dietze, Stefan; Jabeen, Hajira; Omeliyanenko, Janna; Zhang, Wen; Lissandrini, Matteo; Biswas, Russa; de Melo, Gerard; Bonifati, Angela; Vakaj, Edlira; Dragoni, Mauro; Graux, Damien. In Transactions on Graph Data and Knowledge, 1(1), pp. 2:1–2:38. Schloss Dagstuhl -- Leibniz-Zentrum f"u}r Informatik, Dagstuhl, Germany, 2023.
  • [ BibTeX ]
  • [ URL ]
  • [ Download ]
  @article{pan_et_al:TGDK.1.1.2, address = {Dagstuhl, Germany}, annote = {Keywords: Large Language Models, Pre-trained Language Models, Knowledge Graphs, Ontology, Retrieval Augmented Language Models}, author = {Pan, Jeff Z. and Razniewski, Simon and Kalo, Jan-Christoph and Singhania, Sneha and Chen, Jiaoyan and Dietze, Stefan and Jabeen, Hajira and Omeliyanenko, Janna and Zhang, Wen and Lissandrini, Matteo and Biswas, Russa and de Melo, Gerard and Bonifati, Angela and Vakaj, Edlira and Dragoni, Mauro and Graux, Damien}, journal = {Transactions on Graph Data and Knowledge}, keywords = {selected}, number = 1, pages = {2:1–2:38}, publisher = {Schloss Dagstuhl – Leibniz-Zentrum f{\"u}r Informatik}, title = {Large Language Models and Knowledge Graphs: Opportunities and Challenges}, volume = 1, year = 2023 }
• 
  CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion. Omeliyanenko, Janna; Zehe, Albin; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2023. 2023.
  • [ BibTeX ]
  • [ Download ]
  @article{noauthororeditor, author = {Omeliyanenko, Janna and Zehe, Albin and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2023}, keywords = {selected}, title = {CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion}, year = 2023 }
• 
  Optimizing Medical Service Request Processes through Language Modeling and Semantic Search. Schlör, Daniel; Pfister, Jan; Hotho, Andreas. In 2023 the 7th International Conference on Medical and Health Informatics (ICMHI), of ICMHI 2023, pp. 136–141. Association for Computing Machinery, Kyoto, Japan, 2023.
  • [ Abstract ]
  • [ BibTeX ]
  • [ URL ]
  • [ Download ]
  Medical service requests are a crucial part of the workflow in hospitals and healthcare organizations. However, the process of requesting medical services can be time consuming and can require physicians and medical personnel to navigate complex interfaces and enter detailed information about the requested service. In this paper, we propose a system that uses machine learning techniques such as large language models and semantic search to optimize the process of requesting medical services. Our approach enables physicians to request medical services using natural language rather than navigating complex interfaces, allowing for more efficient and flexible interactions with hospital information systems. We evaluate our approach on real-world data and discuss the implications of our work for the future of digital health care. Our results suggest that our approach has the potential to streamline the process of requesting medical services and reduce the time and manual effort required in the daily hospital routine.

  @inproceedings{10.1145/3608298.3608324, abstract = {Medical service requests are a crucial part of the workflow in hospitals and healthcare organizations. However, the process of requesting medical services can be time consuming and can require physicians and medical personnel to navigate complex interfaces and enter detailed information about the requested service. In this paper, we propose a system that uses machine learning techniques such as large language models and semantic search to optimize the process of requesting medical services. Our approach enables physicians to request medical services using natural language rather than navigating complex interfaces, allowing for more efficient and flexible interactions with hospital information systems. We evaluate our approach on real-world data and discuss the implications of our work for the future of digital health care. Our results suggest that our approach has the potential to streamline the process of requesting medical services and reduce the time and manual effort required in the daily hospital routine.}, address = {New York, NY, USA}, author = {Schlör, Daniel and Pfister, Jan and Hotho, Andreas}, booktitle = {2023 the 7th International Conference on Medical and Health Informatics (ICMHI)}, keywords = {app_medical}, pages = {136–141}, publisher = {Association for Computing Machinery}, series = {ICMHI 2023}, title = {Optimizing Medical Service Request Processes through Language Modeling and Semantic Search}, year = 2023 }
• 
  CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion. Omeliyanenko, Janna; Zehe, Albin; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2023, to appear. 2023.
  • [ Abstract ]
  • [ BibTeX ]
  • [ Download ]
  Automated completion of knowledge graphs is a popular topic in the Semantic Web community that aims to automatically and continuously integrate new appearing knowledge into knowledge graphs using artificial intelligence. Recently, approaches that leverage implicit knowledge from language models for this task have shown promising re- sults. However, by fine-tuning language models directly to the domain of knowledge graphs, models forget their original language representation and associated knowledge. An existing solution to address this issue is a trainable adapter, which is integrated into a frozen language model to extract the relevant knowledge without altering the model itself. How- ever, this constrains the generalizability to the specific extraction task and by design requires new and independent adapters to be trained for new knowledge extraction tasks. This effectively prevents the model from benefiting from existing knowledge incorporated in previously trained adapters. In this paper, we propose to combine the benefits of adapters for knowl- edge graph completion with the idea of integrating capsules, introduced in the field of continual learning. This allows the continuous integra- tion of knowledge into a joint model by sharing and reusing previously trained capsules. We find that our approach outperforms solutions using traditional adapters, while requiring notably fewer parameters for con- tinuous knowledge integration. Moreover, we show that this architecture benefits significantly from knowledge sharing in low-resource situations, outperforming adapter-based models on the task of link prediction.

  @article{noauthororeditor, abstract = {Automated completion of knowledge graphs is a popular topic in the Semantic Web community that aims to automatically and continuously integrate new appearing knowledge into knowledge graphs using artificial intelligence. Recently, approaches that leverage implicit knowledge from language models for this task have shown promising re- sults. However, by fine-tuning language models directly to the domain of knowledge graphs, models forget their original language representation and associated knowledge. An existing solution to address this issue is a trainable adapter, which is integrated into a frozen language model to extract the relevant knowledge without altering the model itself. How- ever, this constrains the generalizability to the specific extraction task and by design requires new and independent adapters to be trained for new knowledge extraction tasks. This effectively prevents the model from benefiting from existing knowledge incorporated in previously trained adapters. In this paper, we propose to combine the benefits of adapters for knowl- edge graph completion with the idea of integrating capsules, introduced in the field of continual learning. This allows the continuous integra- tion of knowledge into a joint model by sharing and reusing previously trained capsules. We find that our approach outperforms solutions using traditional adapters, while requiring notably fewer parameters for con- tinuous knowledge integration. Moreover, we show that this architecture benefits significantly from knowledge sharing in low-resource situations, outperforming adapter-based models on the task of link prediction.}, author = {Omeliyanenko, Janna and Zehe, Albin and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2023, to appear}, keywords = {graph}, title = {CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion}, year = 2023 }

• 
  Point me to your Opinion, SenPoi. Pfister, Jan; Wankerl, Sebastian; Hotho, Andreas. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 1313–1323. Association for Computational Linguistics, Seattle, United States, 2022.
  • [ Abstract ]
  • [ BibTeX ]
  • [ URL ]
  • [ Download ]
  Structured Sentiment Analysis is the task of extracting sentiment tuples in a graph structure commonly from review texts. We adapt the Aspect-Based Sentiment Analysis pointer network BARTABSA to model this tuple extraction as a sequence prediction task and extend their output grammar to account for the increased complexity of Structured Sentiment Analysis. To predict structured sentiment tuples in languages other than English we swap BART for a multilingual mT5 and introduce a novel Output Length Regularization to mitigate overfitting to common target sequence lengths, thereby improving the performance of the model by up to 70%. We evaluate our approach on seven datasets in five languages including a zero shot crosslingual setting.

  @inproceedings{pfister-etal-2022-senpoi, abstract = {Structured Sentiment Analysis is the task of extracting sentiment tuples in a graph structure commonly from review texts. We adapt the Aspect-Based Sentiment Analysis pointer network BARTABSA to model this tuple extraction as a sequence prediction task and extend their output grammar to account for the increased complexity of Structured Sentiment Analysis. To predict structured sentiment tuples in languages other than English we swap BART for a multilingual mT5 and introduce a novel Output Length Regularization to mitigate overfitting to common target sequence lengths, thereby improving the performance of the model by up to 70{\%}. We evaluate our approach on seven datasets in five languages including a zero shot crosslingual setting.}, address = {Seattle, United States}, author = {Pfister, Jan and Wankerl, Sebastian and Hotho, Andreas}, booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)}, keywords = {from:janpf}, month = {07}, pages = {1313–1323}, publisher = {Association for Computational Linguistics}, title = {Point me to your Opinion, {S}en{P}oi}, year = 2022 }

• 
  LM4KG: Improving Common Sense Knowledge Graphs with Language Models. Omeliyanenko, Janna; Zehe, Albin; Hettinger, Lena; Hotho, Andreas. In International Semantic Web Conference. Springer, 2020.
  • [ BibTeX ]
  • [ Download ]
  @inproceedings{omeliyanenko2020reweight, author = {Omeliyanenko, Janna and Zehe, Albin and Hettinger, Lena and Hotho, Andreas}, booktitle = {International Semantic Web Conference}, keywords = {selected}, organization = {Springer}, title = {LM4KG: Improving Common Sense Knowledge Graphs with Language Models}, year = 2020 }
• 
  Emote-Controlled: Obtaining Implicit Viewer Feedback through Emote based Sentiment Analysis on Comments of Popular Twitch.tv Channels. Kobs, Konstantin; Zehe, Albin; Bernstetter, Armin; Chibane, Julian; Pfister, Jan; Tritscher, Julian; Hotho, Andreas. In ACM Transactions on Social Computing. 2020.
  • [ BibTeX ]
  • [ Download ]
  @article{kobs2020emotecontrolled, author = {Kobs, Konstantin and Zehe, Albin and Bernstetter, Armin and Chibane, Julian and Pfister, Jan and Tritscher, Julian and Hotho, Andreas}, journal = {ACM Transactions on Social Computing}, keywords = {sentiment-analysis}, title = {Emote-Controlled: Obtaining Implicit Viewer Feedback through Emote based Sentiment Analysis on Comments of Popular Twitch.tv Channels}, year = 2020 }

• 
  Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels. Schöch, Christof; Schlör, Daniel; Popp, Stefanie; Brunner, Annelen; Henny, Ulrike; Tello, Jos’e} Calvo. In DH, pp. 346–353. 2016.
  • [ BibTeX ]
  • [ Download ]
  @inproceedings{schoch2016straight, author = {Schöch, Christof and Schlör, Daniel and Popp, Stefanie and Brunner, Annelen and Henny, Ulrike and Tello, Jos{\'e} Calvo}, booktitle = {DH}, keywords = {directspeech}, pages = {346–353}, title = {Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels.}, year = 2016 }