Knowledge Enriched Machine Learning

Leveraging additional domain knowledge for deep learning models, for example knowledge graphs for NLP, additional item attributes in recommendation or knowledge about physical characteristics in Neural PDEs.

Publications

2024[ to top ]

PreAdapter: Pre-training Language Models on Knowledge Graphs. Omeliyanenko, Janna; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2024, to appear. 2024.
- [ BibTeX ]
- [ Download ]
@article{omeliyanenko2024preadapter, author = {Omeliyanenko, Janna and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2024, to appear}, keywords = {selected}, title = {PreAdapter: Pre-training Language Models on Knowledge Graphs}, year = 2024 }

2023[ to top ]

ConvMOS: climate model output statistics with deep learning. Steininger, Michael; Abel, Daniel; Ziegler, Katrin; Krause, Anna; Paeth, Heiko; Hotho, Andreas. In Data Mining and Knowledge Discovery, 37(1), pp. 136–166. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Climate models are the tool of choice for scientists researching climate change. Like all models they suffer from errors, particularly systematic and location-specific representation errors. One way to reduce these errors is model output statistics (MOS) where the model output is fitted to observational data with machine learning. In this work, we assess the use of convolutional Deep Learning climate MOS approaches and present the ConvMOS architecture which is specifically designed based on the observation that there are systematic and location-specific errors in the precipitation estimates of climate models. We apply ConvMOS models to the simulated precipitation of the regional climate model REMO, showing that a combination of per-location model parameters for reducing location-specific errors and global model parameters for reducing systematic errors is indeed beneficial for MOS performance. We find that ConvMOS models can reduce errors considerably and perform significantly better than three commonly used MOS approaches and plain ResNet and U-Net models in most cases. Our results show that non-linear MOS models underestimate the number of extreme precipitation events, which we alleviate by training models specialized towards extreme precipitation events with the imbalanced regression method DenseLoss. While we consider climate MOS, we argue that aspects of ConvMOS may also be beneficial in other domains with geospatial data, such as air pollution modeling or weather forecasts.

@article{steininger2023convmos, abstract = {Climate models are the tool of choice for scientists researching climate change. Like all models they suffer from errors, particularly systematic and location-specific representation errors. One way to reduce these errors is model output statistics (MOS) where the model output is fitted to observational data with machine learning. In this work, we assess the use of convolutional Deep Learning climate MOS approaches and present the ConvMOS architecture which is specifically designed based on the observation that there are systematic and location-specific errors in the precipitation estimates of climate models. We apply ConvMOS models to the simulated precipitation of the regional climate model REMO, showing that a combination of per-location model parameters for reducing location-specific errors and global model parameters for reducing systematic errors is indeed beneficial for MOS performance. We find that ConvMOS models can reduce errors considerably and perform significantly better than three commonly used MOS approaches and plain ResNet and U-Net models in most cases. Our results show that non-linear MOS models underestimate the number of extreme precipitation events, which we alleviate by training models specialized towards extreme precipitation events with the imbalanced regression method DenseLoss. While we consider climate MOS, we argue that aspects of ConvMOS may also be beneficial in other domains with geospatial data, such as air pollution modeling or weather forecasts.}, author = {Steininger, Michael and Abel, Daniel and Ziegler, Katrin and Krause, Anna and Paeth, Heiko and Hotho, Andreas}, journal = {Data Mining and Knowledge Discovery}, keywords = {author:KRAUSE}, number = 1, pages = {136–166}, title = {ConvMOS: climate model output statistics with deep learning}, volume = 37, year = 2023 }
Large Language Models and Knowledge Graphs: Opportunities and Challenges. Pan, Jeff Z.; Razniewski, Simon; Kalo, Jan-Christoph; Singhania, Sneha; Chen, Jiaoyan; Dietze, Stefan; Jabeen, Hajira; Omeliyanenko, Janna; Zhang, Wen; Lissandrini, Matteo; Biswas, Russa; de Melo, Gerard; Bonifati, Angela; Vakaj, Edlira; Dragoni, Mauro; Graux, Damien. In Transactions on Graph Data and Knowledge, 1(1), pp. 2:1–2:38. Schloss Dagstuhl -- Leibniz-Zentrum f"u}r Informatik, Dagstuhl, Germany, 2023.
- [ BibTeX ]
- [ URL ]
- [ Download ]
@article{pan_et_al:TGDK.1.1.2, address = {Dagstuhl, Germany}, annote = {Keywords: Large Language Models, Pre-trained Language Models, Knowledge Graphs, Ontology, Retrieval Augmented Language Models}, author = {Pan, Jeff Z. and Razniewski, Simon and Kalo, Jan-Christoph and Singhania, Sneha and Chen, Jiaoyan and Dietze, Stefan and Jabeen, Hajira and Omeliyanenko, Janna and Zhang, Wen and Lissandrini, Matteo and Biswas, Russa and de Melo, Gerard and Bonifati, Angela and Vakaj, Edlira and Dragoni, Mauro and Graux, Damien}, journal = {Transactions on Graph Data and Knowledge}, keywords = {selected}, number = 1, pages = {2:1–2:38}, publisher = {Schloss Dagstuhl – Leibniz-Zentrum f{\"u}r Informatik}, title = {Large Language Models and Knowledge Graphs: Opportunities and Challenges}, volume = 1, year = 2023 }
DynaBench: A Benchmark Dataset for Learning Dynamical Systems from Low-Resolution Data. Dulny, Andrzej; Hotho, Andreas; Krause, Anna. In Machine Learning and Knowledge Discovery in Databases: Research Track, D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, F. Bonchi (eds.), pp. 438–455. Springer Nature Switzerland, Cham, 2023.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Previous work on learning physical systems from data has focused on high-resolution grid-structured measurements. However, real-world knowledge of such systems (e.g. weather data) relies on sparsely scattered measuring stations. In this paper, we introduce a novel simulated benchmark dataset, DynaBench, for learning dynamical systems directly from sparsely scattered data without prior knowledge of the equations. The dataset focuses on predicting the evolution of a dynamical system from low-resolution, unstructured measurements. We simulate six different partial differential equations covering a variety of physical systems commonly used in the literature and evaluate several machine learning models, including traditional graph neural networks and point cloud processing models, with the task of predicting the evolution of the system. The proposed benchmark dataset is expected to advance the state of art as an out-of-the-box easy-to-use tool for evaluating models in a setting where only unstructured low-resolution observations are available. The benchmark is available at https://professor-x.de/dynabench.

@inproceedings{10.1007/978-3-031-43412-9_26, abstract = {Previous work on learning physical systems from data has focused on high-resolution grid-structured measurements. However, real-world knowledge of such systems (e.g. weather data) relies on sparsely scattered measuring stations. In this paper, we introduce a novel simulated benchmark dataset, DynaBench, for learning dynamical systems directly from sparsely scattered data without prior knowledge of the equations. The dataset focuses on predicting the evolution of a dynamical system from low-resolution, unstructured measurements. We simulate six different partial differential equations covering a variety of physical systems commonly used in the literature and evaluate several machine learning models, including traditional graph neural networks and point cloud processing models, with the task of predicting the evolution of the system. The proposed benchmark dataset is expected to advance the state of art as an out-of-the-box easy-to-use tool for evaluating models in a setting where only unstructured low-resolution observations are available. The benchmark is available at https://professor-x.de/dynabench.}, address = {Cham}, author = {Dulny, Andrzej and Hotho, Andreas and Krause, Anna}, booktitle = {Machine Learning and Knowledge Discovery in Databases: Research Track}, editor = {Koutra, Danai and Plant, Claudia and Gomez Rodriguez, Manuel and Baralis, Elena and Bonchi, Francesco}, keywords = {selected}, pages = {438–455}, publisher = {Springer Nature Switzerland}, title = {DynaBench: A Benchmark Dataset for Learning Dynamical Systems from Low-Resolution Data}, year = 2023 }
CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion. Omeliyanenko, Janna; Zehe, Albin; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2023. 2023.
- [ BibTeX ]
- [ Download ]
@article{noauthororeditor, author = {Omeliyanenko, Janna and Zehe, Albin and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2023}, keywords = {selected}, title = {CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion}, year = 2023 }
TaylorPDENet: Learning PDEs from non-grid Data. Heinisch, Paul; Dulny, Andrzej; Krause, Anna; Hotho, Andreas. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Modeling data obtained from dynamical systems has gained attention in recent years as a challenging task for machine learning models. Previous approaches assume the measurements to be distributed on a grid. However, for real-world applications like weather prediction, the observations are taken from arbitrary locations within the spatial domain. In this paper, we propose TaylorPDENet - a novel machine learning method that is designed to overcome this challenge. Our algorithm uses the multidimensional Taylor expansion of a dynamical system at each observation point to estimate the spatial derivatives to perform predictions. TaylorPDENet is able to accomplish two objectives simultaneously: accurately forecast the evolution of a complex dynamical system and explicitly reconstruct the underlying differential equation describing the system. We evaluate our model on a variety of advection-diffusion equations with different parameters and show that it performs similarly to equivalent approaches on grid-structured data while being able to process unstructured data as well.

@misc{heinisch2023taylorpdenet, abstract = {Modeling data obtained from dynamical systems has gained attention in recent years as a challenging task for machine learning models. Previous approaches assume the measurements to be distributed on a grid. However, for real-world applications like weather prediction, the observations are taken from arbitrary locations within the spatial domain. In this paper, we propose TaylorPDENet - a novel machine learning method that is designed to overcome this challenge. Our algorithm uses the multidimensional Taylor expansion of a dynamical system at each observation point to estimate the spatial derivatives to perform predictions. TaylorPDENet is able to accomplish two objectives simultaneously: accurately forecast the evolution of a complex dynamical system and explicitly reconstruct the underlying differential equation describing the system. We evaluate our model on a variety of advection-diffusion equations with different parameters and show that it performs similarly to equivalent approaches on grid-structured data while being able to process unstructured data as well.}, author = {Heinisch, Paul and Dulny, Andrzej and Krause, Anna and Hotho, Andreas}, keywords = {from:martinr}, note = {cite arxiv:2306.14511}, title = {TaylorPDENet: Learning PDEs from non-grid Data}, year = 2023 }
CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion. Omeliyanenko, Janna; Zehe, Albin; Hotho, Andreas; Schlör, Daniel. In International Semantic Web Conference ISWC 2023, to appear. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ Download ]
Automated completion of knowledge graphs is a popular topic in the Semantic Web community that aims to automatically and continuously integrate new appearing knowledge into knowledge graphs using artificial intelligence. Recently, approaches that leverage implicit knowledge from language models for this task have shown promising re- sults. However, by fine-tuning language models directly to the domain of knowledge graphs, models forget their original language representation and associated knowledge. An existing solution to address this issue is a trainable adapter, which is integrated into a frozen language model to extract the relevant knowledge without altering the model itself. How- ever, this constrains the generalizability to the specific extraction task and by design requires new and independent adapters to be trained for new knowledge extraction tasks. This effectively prevents the model from benefiting from existing knowledge incorporated in previously trained adapters. In this paper, we propose to combine the benefits of adapters for knowl- edge graph completion with the idea of integrating capsules, introduced in the field of continual learning. This allows the continuous integra- tion of knowledge into a joint model by sharing and reusing previously trained capsules. We find that our approach outperforms solutions using traditional adapters, while requiring notably fewer parameters for con- tinuous knowledge integration. Moreover, we show that this architecture benefits significantly from knowledge sharing in low-resource situations, outperforming adapter-based models on the task of link prediction.

@article{noauthororeditor, abstract = {Automated completion of knowledge graphs is a popular topic in the Semantic Web community that aims to automatically and continuously integrate new appearing knowledge into knowledge graphs using artificial intelligence. Recently, approaches that leverage implicit knowledge from language models for this task have shown promising re- sults. However, by fine-tuning language models directly to the domain of knowledge graphs, models forget their original language representation and associated knowledge. An existing solution to address this issue is a trainable adapter, which is integrated into a frozen language model to extract the relevant knowledge without altering the model itself. How- ever, this constrains the generalizability to the specific extraction task and by design requires new and independent adapters to be trained for new knowledge extraction tasks. This effectively prevents the model from benefiting from existing knowledge incorporated in previously trained adapters. In this paper, we propose to combine the benefits of adapters for knowl- edge graph completion with the idea of integrating capsules, introduced in the field of continual learning. This allows the continuous integra- tion of knowledge into a joint model by sharing and reusing previously trained capsules. We find that our approach outperforms solutions using traditional adapters, while requiring notably fewer parameters for con- tinuous knowledge integration. Moreover, we show that this architecture benefits significantly from knowledge sharing in low-resource situations, outperforming adapter-based models on the task of link prediction.}, author = {Omeliyanenko, Janna and Zehe, Albin and Hotho, Andreas and Schlör, Daniel}, journal = {International Semantic Web Conference ISWC 2023, to appear}, keywords = {graph}, title = {CapsKG: Enabling Continual Knowledge Integration in Language Models for Automatic Knowledge Graph Completion}, year = 2023 }

2020[ to top ]

LM4KG: Improving Common Sense Knowledge Graphs with Language Models. Omeliyanenko, Janna; Zehe, Albin; Hettinger, Lena; Hotho, Andreas. In International Semantic Web Conference. Springer, 2020.
- [ BibTeX ]
- [ Download ]
@inproceedings{omeliyanenko2020reweight, author = {Omeliyanenko, Janna and Zehe, Albin and Hettinger, Lena and Hotho, Andreas}, booktitle = {International Semantic Web Conference}, keywords = {selected}, organization = {Springer}, title = {LM4KG: Improving Common Sense Knowledge Graphs with Language Models}, year = 2020 }
Emote-Controlled: Obtaining Implicit Viewer Feedback through Emote based Sentiment Analysis on Comments of Popular Twitch.tv Channels. Kobs, Konstantin; Zehe, Albin; Bernstetter, Armin; Chibane, Julian; Pfister, Jan; Tritscher, Julian; Hotho, Andreas. In ACM Transactions on Social Computing. 2020.
- [ BibTeX ]
- [ Download ]
@article{kobs2020emotecontrolled, author = {Kobs, Konstantin and Zehe, Albin and Bernstetter, Armin and Chibane, Julian and Pfister, Jan and Tritscher, Julian and Hotho, Andreas}, journal = {ACM Transactions on Social Computing}, keywords = {sentiment-analysis}, title = {Emote-Controlled: Obtaining Implicit Viewer Feedback through Emote based Sentiment Analysis on Comments of Popular Twitch.tv Channels}, year = 2020 }
iNALU: Improved Neural Arithmetic Logic Unit. Schlör, Daniel; Ring, Markus; Hotho, Andreas. In Frontiers in Artificial Intelligence, 3, p. 71. 2020.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Neural networks have to capture mathematical relationships in order to learn various tasks. They approximate these relations implicitly and therefore often do not generalize well. The recently proposed Neural Arithmetic Logic Unit (NALU) is a novel neural architecture which is able to explicitly represent the mathematical relationships by the units of the network to learn operations such as summation, subtraction or multiplication. Although NALUs have been shown to perform well on various downstream tasks, an in-depth analysis reveals practical shortcomings by design, such as the inability to multiply or divide negative input values or training stability issues for deeper networks. We address these issues and propose an improved model architecture. We evaluate our model empirically in various settings from learning basic arithmetic operations to more complex functions. Our experiments indicate that our model solves stability issues and outperforms the original NALU model in means of arithmetic precision and convergence.

@article{10.3389/frai.2020.00071, abstract = {Neural networks have to capture mathematical relationships in order to learn various tasks. They approximate these relations implicitly and therefore often do not generalize well. The recently proposed Neural Arithmetic Logic Unit (NALU) is a novel neural architecture which is able to explicitly represent the mathematical relationships by the units of the network to learn operations such as summation, subtraction or multiplication. Although NALUs have been shown to perform well on various downstream tasks, an in-depth analysis reveals practical shortcomings by design, such as the inability to multiply or divide negative input values or training stability issues for deeper networks. We address these issues and propose an improved model architecture. We evaluate our model empirically in various settings from learning basic arithmetic operations to more complex functions. Our experiments indicate that our model solves stability issues and outperforms the original NALU model in means of arithmetic precision and convergence.}, author = {Schlör, Daniel and Ring, Markus and Hotho, Andreas}, journal = {Frontiers in Artificial Intelligence}, keywords = {inalu}, pages = 71, title = {iNALU: Improved Neural Arithmetic Logic Unit}, volume = 3, year = 2020 }

Knowledge Enriched Machine Learning

Publications

Data privacy protection

Data privacy protection

Picture credits