mPlane - Building an Intelligent Measurement Plane for the Internet

Head

Phuoc Tran-Gia

Florian Wamser

Matthias Hirth

Researchers

Valentin Burger

Christian Schwartz

Summary

University of Würzburg is involved in the following mPlane work packages.

WP2 - Programmable Probes: Crowdsourcing based network measurements of selected cloud applications.
WP3 - Large-scale data analysis: Analysis of measurements traces from cloud applications on network layer and application layer.

The joint collaboration successfully led to the following joint publications with other mPlane project partners.

Crowdsourcing based network measurements [1]:

Several approaches exist for network measurements ranging from analysing live traffic traces from campus or Internet Service Providers (ISPs) networks, to performing active measurements on distributed testbeds like PlanetLab, or involving volunteers. Each method falls short in offering only a partial vision. For instance, the scope of traffic traces is limited to the ISP's network and customers' habits, while active measurements might be biased by the population or node location involved in the experiment. To complement these techniques we propose to use (commercial) crowdsourcing platforms for network measurements. They permit a controllable, diverse and realistic view on the Internet and provide better control than measurements with voluntary participants. In [1], we compare crowdsourcing with traditional measurement techniques, describe possible pitfalls and limitations and present best practices to overcome these issues. The major contribution of this paper comprises a guideline for researchers when and how to exploit crowdsourcing for network measurements.

On QoE for File Storage Services [2]:

Cloud computing is receiving growing attention by researchers from a variety of disciplines. However, so far only a few studies exist that investigate th eQoE of cloud services, including the category of personal file storage services like Dropbox, from an end user perspective. In [2], a methodology and the results of four different user studies towards a situational QoE model for file storage services are provided. In order to obtain insights in existing usage practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services in general and one specifically targeted at users of one of the major popular cloud storage services, namely Dropbox. A third study on mobile Dropbox further investigated QoE and specific use cases in a mobile context, via smartphones and tablets. Based on these results, typical usage situations as well as short-term and long-term QoE influence factors were derived which included user profile, context and situation, as well as system level influences. In a fourth study the impact of waiting times as short-term QoE influence on system level during the regular usage of Dropbox is investigated. As an outcome of our studies, we formulate research questions driving the agenda towards measuring and modeling QoE for cloud-based file storage services.

QoE Model for YouTube and Different Transport Protocols [3]:

Video streaming currently dominates global Internet traffic and will be of even increasing importance in the future. In [3], we assess the impact of the underlying transport protocol on the user perceived quality for video streaming using YouTube as example. In particular, we investigate whether UDP or TCP fits better for Video-on-Demand delivery from the end user's perspective, when the video is transmitted over a bottleneck link. For UDP based streaming, the bottleneck link results in spatial and temporal video artifacts, decreasing the video quality. In contrast, in the case of TCP based streaming, the displayed content itself is not disturbed but playback suffers from stalling due to re-buffering. The results of subjective user studies for both scenarios are analyzed in order to assess the transport protocol influences on Quality of Experience of YouTube. To this end, application-level measurements are conducted for YouTube streaming over a network bottleneck in order to develop models for realistic stalling patterns. Furthermore, mapping functions are derived that accurately describe the relationship between network-level impairments and QoE for both protocols.

Homepage: http://www.ict-mplane.eu/

Article[ to top ]

YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks. Casas, Pedro; Seufert, Michael; Schatz, Raimund. In ACM SIGMETRICS Performance Evaluation Review, 41, pp. 44–46. 2013.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
YouTube is changing the way operators manage network performance monitoring. In this paper we introduce YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G customers watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.

@article{info3-article-2013-114, abstract = {YouTube is changing the way operators manage network performance monitoring. In this paper we introduce YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G customers watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.}, author = {Casas, Pedro and Seufert, Michael and Schatz, Raimund}, journal = {ACM SIGMETRICS Performance Evaluation Review}, keywords = {sys:hidden:article}, month = {09}, pages = {44-46}, title = {YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks}, volume = 41, year = 2013 }

%0 Journal Article %1 info3-article-2013-114 %A Casas, Pedro %A Seufert, Michael %A Schatz, Raimund %D 2013 %J ACM SIGMETRICS Performance Evaluation Review %P 44-46 %T YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks %V 41 %X YouTube is changing the way operators manage network performance monitoring. In this paper we introduce YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G customers watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.

Inproceedings[ to top ]

Application-Layer Monitoring of QoE Parameters for Mobile YouTube Video Streaming in the Field. Seufert, Michael; Casas, Pedro; Wamser, Florian; Wehner, Nikolas; Schatz, Raimund; Tran-Gia, Phuoc. In IEEE 6th International Conference on Communications and Electronics (ICCE). Ha Long, Vietnam, 2016.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
YouTube video streaming is one of the most popular and most demanding services in cellular networks. Thus, operators are concerned about the quality of the streaming delivered by their networks and would like to monitor the Quality of Experience (QoE) of the end users. In this work, we conduct a field study of mobile YouTube video streaming, in which both network flow parameters and application-layer streaming parameters were monitored, and present the characteristics of current mobile YouTube streaming. The impact of both approaches is investigated showing that monitoring network parameters is not sufficient to directly infer the resulting QoE. In contrast, the streaming parameters, which can be obtained from application-layer monitoring, show high correlations to the subjectively experienced quality, and thus, are better suited for QoE monitoring.

@inproceedings{info3-inproceedings-2016-9, abstract = {YouTube video streaming is one of the most popular and most demanding services in cellular networks. Thus, operators are concerned about the quality of the streaming delivered by their networks and would like to monitor the Quality of Experience (QoE) of the end users. In this work, we conduct a field study of mobile YouTube video streaming, in which both network flow parameters and application-layer streaming parameters were monitored, and present the characteristics of current mobile YouTube streaming. The impact of both approaches is investigated showing that monitoring network parameters is not sufficient to directly infer the resulting QoE. In contrast, the streaming parameters, which can be obtained from application-layer monitoring, show high correlations to the subjectively experienced quality, and thus, are better suited for QoE monitoring.}, address = {Ha Long, Vietnam}, author = {Seufert, Michael and Casas, Pedro and Wamser, Florian and Wehner, Nikolas and Schatz, Raimund and Tran-Gia, Phuoc}, booktitle = {IEEE 6th International Conference on Communications and Electronics (ICCE)}, keywords = {can}, month = {07}, title = {Application-Layer Monitoring of QoE Parameters for Mobile YouTube Video Streaming in the Field}, year = 2016 }

%0 Conference Paper %1 info3-inproceedings-2016-9 %A Seufert, Michael %A Casas, Pedro %A Wamser, Florian %A Wehner, Nikolas %A Schatz, Raimund %A Tran-Gia, Phuoc %B IEEE 6th International Conference on Communications and Electronics (ICCE) %C Ha Long, Vietnam %D 2016 %T Application-Layer Monitoring of QoE Parameters for Mobile YouTube Video Streaming in the Field %X YouTube video streaming is one of the most popular and most demanding services in cellular networks. Thus, operators are concerned about the quality of the streaming delivered by their networks and would like to monitor the Quality of Experience (QoE) of the end users. In this work, we conduct a field study of mobile YouTube video streaming, in which both network flow parameters and application-layer streaming parameters were monitored, and present the characteristics of current mobile YouTube streaming. The impact of both approaches is investigated showing that monitoring network parameters is not sufficient to directly infer the resulting QoE. In contrast, the streaming parameters, which can be obtained from application-layer monitoring, show high correlations to the subjectively experienced quality, and thus, are better suited for QoE monitoring.
The Impact of Adaptation Strategies on Perceived Quality of HTTP Adaptive Streaming. Egger, Sebastian; Gardlo, Bruno; Seufert, Michael; Schatz, Raimund. In 1st Workshop on Design, Quality and Deployment of Adaptive Video Streaming (VideoNext). Sydney, Australia, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Changing network conditions like bandwidth fluctuations and resulting bad user experience issues (e.g. video freezes) pose severe challenges to Internet video streaming. To address this problem, an increasing number of video services utilizes HTTP adaptive streaming (HAS). HAS enables service providers to improve Quality of Experience (QoE) and resource utilization by incorporating information from different layers. However, these adaptation possibilities of HAS also introduce new perceivable impairments such as the fluctuation of audiovisual quality levels over time, which in turn lead to novel QoE-related research questions.The main contribution of this paper is the formulation of open research questions as well as a thorough systematic user-centric analysis of different quality adaptation dimensions and strategies. The underlying data has been acquired through two crowdsourcing and one lab study. The results provide guidance w.r.t. which encoding dimensions are combined best for the creation of the adaptation set and what type of adaptation strategy should be used. Furthermore it provides insights on the impact of adaptation frequency and the true QoE gain of adaptation over stallings.

@inproceedings{info3-inproceedings-2014-507, abstract = {Changing network conditions like bandwidth fluctuations and resulting bad user experience issues (e.g. video freezes) pose severe challenges to Internet video streaming. To address this problem, an increasing number of video services utilizes HTTP adaptive streaming (HAS). HAS enables service providers to improve Quality of Experience (QoE) and resource utilization by incorporating information from different layers. However, these adaptation possibilities of HAS also introduce new perceivable impairments such as the fluctuation of audiovisual quality levels over time, which in turn lead to novel QoE-related research questions.The main contribution of this paper is the formulation of open research questions as well as a thorough systematic user-centric analysis of different quality adaptation dimensions and strategies. The underlying data has been acquired through two crowdsourcing and one lab study. The results provide guidance w.r.t. which encoding dimensions are combined best for the creation of the adaptation set and what type of adaptation strategy should be used. Furthermore it provides insights on the impact of adaptation frequency and the true QoE gain of adaptation over stallings.}, address = {Sydney, Australia}, author = {Egger, Sebastian and Gardlo, Bruno and Seufert, Michael and Schatz, Raimund}, booktitle = {1st Workshop on Design, Quality and Deployment of Adaptive Video Streaming (VideoNext)}, keywords = {sys:hidden:inproceedings}, month = 12, title = {The Impact of Adaptation Strategies on Perceived Quality of HTTP Adaptive Streaming}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-507 %A Egger, Sebastian %A Gardlo, Bruno %A Seufert, Michael %A Schatz, Raimund %B 1st Workshop on Design, Quality and Deployment of Adaptive Video Streaming (VideoNext) %C Sydney, Australia %D 2014 %T The Impact of Adaptation Strategies on Perceived Quality of HTTP Adaptive Streaming %X Changing network conditions like bandwidth fluctuations and resulting bad user experience issues (e.g. video freezes) pose severe challenges to Internet video streaming. To address this problem, an increasing number of video services utilizes HTTP adaptive streaming (HAS). HAS enables service providers to improve Quality of Experience (QoE) and resource utilization by incorporating information from different layers. However, these adaptation possibilities of HAS also introduce new perceivable impairments such as the fluctuation of audiovisual quality levels over time, which in turn lead to novel QoE-related research questions.The main contribution of this paper is the formulation of open research questions as well as a thorough systematic user-centric analysis of different quality adaptation dimensions and strategies. The underlying data has been acquired through two crowdsourcing and one lab study. The results provide guidance w.r.t. which encoding dimensions are combined best for the creation of the adaptation set and what type of adaptation strategy should be used. Furthermore it provides insights on the impact of adaptation frequency and the true QoE gain of adaptation over stallings.
Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment. Hoßfeld, Tobias; Hirth, Matthias; Korshunov, Pavel; Hanhart, Philippe; Gardlo, Bruno; Keimel, Christian; Timmerer, Christian. In 16th International Workshop on Multimedia Signal Processing. Jakarta, Indonesia, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.

@inproceedings{info3-inproceedings-2014-498, abstract = {The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.}, address = {Jakarta, Indonesia}, author = {Hoßfeld, Tobias and Hirth, Matthias and Korshunov, Pavel and Hanhart, Philippe and Gardlo, Bruno and Keimel, Christian and Timmerer, Christian}, booktitle = {16th International Workshop on Multimedia Signal Processing}, keywords = {crowdsourcing-design}, month = {09}, title = {Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-498 %A Hoßfeld, Tobias %A Hirth, Matthias %A Korshunov, Pavel %A Hanhart, Philippe %A Gardlo, Bruno %A Keimel, Christian %A Timmerer, Christian %B 16th International Workshop on Multimedia Signal Processing %C Jakarta, Indonesia %D 2014 %T Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment %X The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.
Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. Hoßfeld, Tobias; Seufert, Michael; Sieber, Christian; Zinner, Thomas. In 6th International Workshop on Quality of Multimedia Experience (QoMEX). Singapore, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
HTTP Adaptive Streaming (HAS) is employed by more and more video streaming services in the Internet. It allows to adapt the downloaded video quality to the current network conditions, and thus, avoids stalling (i.e., playback interruptions) to the greatest possible extend. The adaptation of video streams is done by switching between different quality representation levels, which influences the user perceived quality of the video stream. In this work, the influence of several adaptation parameters, namely, switch amplitude (i.e., quality level difference), switching frequency, and recency effects, on Quality of Experience (QoE) is investigated. Therefore, crowdsourcing experiments were conducted in order to collect subjective ratings for different adaptation-related test conditions. The results of these subjective studies indicate the influence of the adaptation parameters, and based on these findings a simplified QoE model for HAS is presented, which only relies on the switch amplitude and the playback time of each layer.

@inproceedings{info3-inproceedings-2014-503, abstract = {HTTP Adaptive Streaming (HAS) is employed by more and more video streaming services in the Internet. It allows to adapt the downloaded video quality to the current network conditions, and thus, avoids stalling (i.e., playback interruptions) to the greatest possible extend. The adaptation of video streams is done by switching between different quality representation levels, which influences the user perceived quality of the video stream. In this work, the influence of several adaptation parameters, namely, switch amplitude (i.e., quality level difference), switching frequency, and recency effects, on Quality of Experience (QoE) is investigated. Therefore, crowdsourcing experiments were conducted in order to collect subjective ratings for different adaptation-related test conditions. The results of these subjective studies indicate the influence of the adaptation parameters, and based on these findings a simplified QoE model for HAS is presented, which only relies on the switch amplitude and the playback time of each layer.}, address = {Singapore}, author = {Hoßfeld, Tobias and Seufert, Michael and Sieber, Christian and Zinner, Thomas}, booktitle = {6th International Workshop on Quality of Multimedia Experience (QoMEX)}, keywords = {smartenit}, month = {09}, title = {Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-503 %A Hoßfeld, Tobias %A Seufert, Michael %A Sieber, Christian %A Zinner, Thomas %B 6th International Workshop on Quality of Multimedia Experience (QoMEX) %C Singapore %D 2014 %T Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming %X HTTP Adaptive Streaming (HAS) is employed by more and more video streaming services in the Internet. It allows to adapt the downloaded video quality to the current network conditions, and thus, avoids stalling (i.e., playback interruptions) to the greatest possible extend. The adaptation of video streams is done by switching between different quality representation levels, which influences the user perceived quality of the video stream. In this work, the influence of several adaptation parameters, namely, switch amplitude (i.e., quality level difference), switching frequency, and recency effects, on Quality of Experience (QoE) is investigated. Therefore, crowdsourcing experiments were conducted in order to collect subjective ratings for different adaptation-related test conditions. The results of these subjective studies indicate the influence of the adaptation parameters, and based on these findings a simplified QoE model for HAS is presented, which only relies on the switch amplitude and the playback time of each layer.
Performance Model for Waiting Times in Cloud File Synchronization Services. Schwartz, Christian; Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc. In 26th International Teletraffic Congress (ITC). Karlskrona, Sweden, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.

@inproceedings{info3-inproceedings-2014-495, abstract = {Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.}, address = {Karlskrona, Sweden}, author = {Schwartz, Christian and Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {26th International Teletraffic Congress (ITC)}, keywords = {smartenit}, month = {09}, title = {Performance Model for Waiting Times in Cloud File Synchronization Services}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-495 %A Schwartz, Christian %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B 26th International Teletraffic Congress (ITC) %C Karlskrona, Sweden %D 2014 %T Performance Model for Waiting Times in Cloud File Synchronization Services %X Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.
Dynamic Bandwidth Allocation for Multiple Network Connections: Improving User QoE and Network Usage of YouTube in Mobile Broadband. Wamser, Florian; Zinner, Thomas; Z Zhu, Jing; Tran-Gia, Phuoc. In ACM SIGCOMM Capacity Sharing Workshop (CSWS 2014). Chicago, IL, USA, 2014.
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
@inproceedings{info3-inproceedings-2014-500, address = {Chicago, IL, USA}, author = {Wamser, Florian and Zinner, Thomas and Z Zhu, Jing and Tran-Gia, Phuoc}, booktitle = {ACM SIGCOMM Capacity Sharing Workshop (CSWS 2014)}, keywords = {demobenefit}, month = {08}, title = {Dynamic Bandwidth Allocation for Multiple Network Connections: Improving User QoE and Network Usage of YouTube in Mobile Broadband}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-500 %A Wamser, Florian %A Zinner, Thomas %A Z Zhu, Jing %A Tran-Gia, Phuoc %B ACM SIGCOMM Capacity Sharing Workshop (CSWS 2014) %C Chicago, IL, USA %D 2014 %T Dynamic Bandwidth Allocation for Multiple Network Connections: Improving User QoE and Network Usage of YouTube in Mobile Broadband
Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring. Hirth, Matthias; Scheuring, Sven; Hoßfeld, Tobias; Schwartz, Christian; Tran-Gia, Phuoc. In 5th International Conference on Communications and Electronics (ICCE 2014). Da Nang, Vietnam, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.

@inproceedings{info3-inproceedings-2014-494, abstract = {Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.}, address = {Da Nang, Vietnam}, author = {Hirth, Matthias and Scheuring, Sven and Hoßfeld, Tobias and Schwartz, Christian and Tran-Gia, Phuoc}, booktitle = {5th International Conference on Communications and Electronics (ICCE 2014)}, keywords = {crowdsourcing-design}, month = {07}, title = {Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-494 %A Hirth, Matthias %A Scheuring, Sven %A Hoßfeld, Tobias %A Schwartz, Christian %A Tran-Gia, Phuoc %B 5th International Conference on Communications and Electronics (ICCE 2014) %C Da Nang, Vietnam %D 2014 %T Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring %X Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.
Crowdsourcing 2.0: Enhancing Execution Speed and Reliability of Web-based QoE Testing. Gardlo, Bruno; Egger, Sebastian; Seufert, Michael; Schatz, Raimund. In IEEE International Conference on Communications (ICC). Sydney, Australia, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Since its introduction a few years ago, the concept of 'Crowdsourcing' has been heralded as highly attractive alternative approach towards evaluating the Quality of Experience (QoE) of networked multimedia services. The main reason is that, in comparison to traditional laboratory-based subjective quality testing, crowd-based QoE assessment over the Internet promises to be not only much more cost-effective (no lab facilities required, less cost per subject) but also much faster in terms of shorter campaign setup and turnaround times. However, the reliability of remote test subjects and consequently, the trustworthiness of study results is still an issue that prevents the widespread adoption of crowd-based QoE testing. Various ideas for improving user rating reliability and test efficiency have been proposed, with the majority of them relying on a posteriori analysis of results. However, such methods introduce a major lag that significantly affects efficiency of campaign execution. In this paper we address these shortcomings by introducing in momento methods for crowdsourced video QoE assessment, which yield improvements of results reliability by the factor two and campaign execution efficiency by the factor ten. The proposed in momento methods are applicable to existing crowd-based QoE testing approaches and suitable for a variety of service scenarios.

@inproceedings{info3-inproceedings-2014-488, abstract = {Since its introduction a few years ago, the concept of 'Crowdsourcing' has been heralded as highly attractive alternative approach towards evaluating the Quality of Experience (QoE) of networked multimedia services. The main reason is that, in comparison to traditional laboratory-based subjective quality testing, crowd-based QoE assessment over the Internet promises to be not only much more cost-effective (no lab facilities required, less cost per subject) but also much faster in terms of shorter campaign setup and turnaround times. However, the reliability of remote test subjects and consequently, the trustworthiness of study results is still an issue that prevents the widespread adoption of crowd-based QoE testing. Various ideas for improving user rating reliability and test efficiency have been proposed, with the majority of them relying on a posteriori analysis of results. However, such methods introduce a major lag that significantly affects efficiency of campaign execution. In this paper we address these shortcomings by introducing in momento methods for crowdsourced video QoE assessment, which yield improvements of results reliability by the factor two and campaign execution efficiency by the factor ten. The proposed in momento methods are applicable to existing crowd-based QoE testing approaches and suitable for a variety of service scenarios.}, address = {Sydney, Australia}, author = {Gardlo, Bruno and Egger, Sebastian and Seufert, Michael and Schatz, Raimund}, booktitle = {IEEE International Conference on Communications (ICC)}, keywords = {sys:hidden:inproceedings}, month = {06}, title = {Crowdsourcing 2.0: Enhancing Execution Speed and Reliability of Web-based QoE Testing}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-488 %A Gardlo, Bruno %A Egger, Sebastian %A Seufert, Michael %A Schatz, Raimund %B IEEE International Conference on Communications (ICC) %C Sydney, Australia %D 2014 %T Crowdsourcing 2.0: Enhancing Execution Speed and Reliability of Web-based QoE Testing %X Since its introduction a few years ago, the concept of 'Crowdsourcing' has been heralded as highly attractive alternative approach towards evaluating the Quality of Experience (QoE) of networked multimedia services. The main reason is that, in comparison to traditional laboratory-based subjective quality testing, crowd-based QoE assessment over the Internet promises to be not only much more cost-effective (no lab facilities required, less cost per subject) but also much faster in terms of shorter campaign setup and turnaround times. However, the reliability of remote test subjects and consequently, the trustworthiness of study results is still an issue that prevents the widespread adoption of crowd-based QoE testing. Various ideas for improving user rating reliability and test efficiency have been proposed, with the majority of them relying on a posteriori analysis of results. However, such methods introduce a major lag that significantly affects efficiency of campaign execution. In this paper we address these shortcomings by introducing in momento methods for crowdsourced video QoE assessment, which yield improvements of results reliability by the factor two and campaign execution efficiency by the factor ten. The proposed in momento methods are applicable to existing crowd-based QoE testing approaches and suitable for a variety of service scenarios.
Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing. Burger, Valentin; Hirth, Matthias; Schwartz, Christian; Hoßfeld, Tobias. In Measurement, Modelling and Evaluation of Computing Systems (MMB 2014). Bamberg, Germany, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.

@inproceedings{info3-inproceedings-2014-486, abstract = {Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.}, address = {Bamberg, Germany}, author = {Burger, Valentin and Hirth, Matthias and Schwartz, Christian and Hoßfeld, Tobias}, booktitle = {Measurement, Modelling and Evaluation of Computing Systems (MMB 2014)}, keywords = {smartenit}, month = {03}, title = {Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-486 %A Burger, Valentin %A Hirth, Matthias %A Schwartz, Christian %A Hoßfeld, Tobias %B Measurement, Modelling and Evaluation of Computing Systems (MMB 2014) %C Bamberg, Germany %D 2014 %T Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing %X Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.
QoE of YouTube Video Streaming for Current Internet Transport Protocols. Hoßfeld, Tobias; Schatz, Raimund; Krieger, Udo. In Measurement, Modelling and Evaluation of Computing Systems (MMB 2014). Bamberg, Germany, 2014.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Video streaming currently dominates global Internet traffic and will be of even increasing importance in the future. In this paper we assess the impact of the underlying transport protocol on the user perceived quality for video streaming using YouTube as example. In particular, we investigate whether UDP or TCP fits better for Video-on-Demand delivery from the end user's perspective, when the video is transmitted over a bottleneck link. For UDP based streaming, the bottleneck link results in spatial and temporal video artifacts, decreasing the video quality. In contrast, in the case of TCP based streaming, the displayed content itself is not disturbed but playback suffers from stalling due to rebufferung. The results of subjective user studies for both scenarios are analyzed in order to assess the transport protocol influences on Quality of Experience of YouTube. To this end, application-level measurements are conducted for YouTube streaming over a network bottleneck in order to develop models for realistic stalling patterns. Furthermore, mapping functions are derived that accurately describe the relationship between network-level impairments and QoE for both protocols.

@inproceedings{info3-inproceedings-2014-485, abstract = {Video streaming currently dominates global Internet traffic and will be of even increasing importance in the future. In this paper we assess the impact of the underlying transport protocol on the user perceived quality for video streaming using YouTube as example. In particular, we investigate whether UDP or TCP fits better for Video-on-Demand delivery from the end user's perspective, when the video is transmitted over a bottleneck link. For UDP based streaming, the bottleneck link results in spatial and temporal video artifacts, decreasing the video quality. In contrast, in the case of TCP based streaming, the displayed content itself is not disturbed but playback suffers from stalling due to rebufferung. The results of subjective user studies for both scenarios are analyzed in order to assess the transport protocol influences on Quality of Experience of YouTube. To this end, application-level measurements are conducted for YouTube streaming over a network bottleneck in order to develop models for realistic stalling patterns. Furthermore, mapping functions are derived that accurately describe the relationship between network-level impairments and QoE for both protocols.}, address = {Bamberg, Germany}, author = {Hoßfeld, Tobias and Schatz, Raimund and Krieger, Udo}, booktitle = {Measurement, Modelling and Evaluation of Computing Systems (MMB 2014)}, keywords = {demobenefit}, month = {03}, title = {QoE of YouTube Video Streaming for Current Internet Transport Protocols}, year = 2014 }

%0 Conference Paper %1 info3-inproceedings-2014-485 %A Hoßfeld, Tobias %A Schatz, Raimund %A Krieger, Udo %B Measurement, Modelling and Evaluation of Computing Systems (MMB 2014) %C Bamberg, Germany %D 2014 %T QoE of YouTube Video Streaming for Current Internet Transport Protocols %X Video streaming currently dominates global Internet traffic and will be of even increasing importance in the future. In this paper we assess the impact of the underlying transport protocol on the user perceived quality for video streaming using YouTube as example. In particular, we investigate whether UDP or TCP fits better for Video-on-Demand delivery from the end user's perspective, when the video is transmitted over a bottleneck link. For UDP based streaming, the bottleneck link results in spatial and temporal video artifacts, decreasing the video quality. In contrast, in the case of TCP based streaming, the displayed content itself is not disturbed but playback suffers from stalling due to rebufferung. The results of subjective user studies for both scenarios are analyzed in order to assess the transport protocol influences on Quality of Experience of YouTube. To this end, application-level measurements are conducted for YouTube streaming over a network bottleneck in order to develop models for realistic stalling patterns. Furthermore, mapping functions are derived that accurately describe the relationship between network-level impairments and QoE for both protocols.
YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks. Casas, Pedro; Seufert, Michael; Schatz, Raimund. In 31st International Symposium on Computer Performance, Modeling, Measurements and Evaluation (IFIP Performance). Vienna, Austria, 2013.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
YouTube is changing the way operators manage network performance monitoring. In this paper we present YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G users watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.

@inproceedings{info3-inproceedings-2013-472, abstract = {YouTube is changing the way operators manage network performance monitoring. In this paper we present YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G users watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.}, address = {Vienna, Austria}, author = {Casas, Pedro and Seufert, Michael and Schatz, Raimund}, booktitle = {31st International Symposium on Computer Performance, Modeling, Measurements and Evaluation (IFIP Performance)}, keywords = {demobenefit}, month = {09}, title = {YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks}, year = 2013 }

%0 Conference Paper %1 info3-inproceedings-2013-472 %A Casas, Pedro %A Seufert, Michael %A Schatz, Raimund %B 31st International Symposium on Computer Performance, Modeling, Measurements and Evaluation (IFIP Performance) %C Vienna, Austria %D 2013 %T YOUQMON: A System for On-line Monitoring of YouTube QoE in Operational 3G Networks %X YouTube is changing the way operators manage network performance monitoring. In this paper we present YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G users watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.
Need for Speed? On Quality of Experience for File Storage Services. Amrehn, Philipp; Vandenbroucke, Karel; Hoßfeld, Tobias; de Moor, Katrien; Hirth, Matthias; Schatz, Raimund; Casas, Pedro. In 4th International Workshop on Perceptual Quality of Systems (PQS 2013). Vienna, Austria, 2013.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ Download ]
Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.

@inproceedings{info3-inproceedings-2013-469, abstract = {Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.}, address = {Vienna, Austria}, author = {Amrehn, Philipp and Vandenbroucke, Karel and Hoßfeld, Tobias and de Moor, Katrien and Hirth, Matthias and Schatz, Raimund and Casas, Pedro}, booktitle = {4th International Workshop on Perceptual Quality of Systems (PQS 2013)}, keywords = {smartenit}, month = {09}, title = {Need for Speed? On Quality of Experience for File Storage Services}, year = 2013 }

%0 Conference Paper %1 info3-inproceedings-2013-469 %A Amrehn, Philipp %A Vandenbroucke, Karel %A Hoßfeld, Tobias %A de Moor, Katrien %A Hirth, Matthias %A Schatz, Raimund %A Casas, Pedro %B 4th International Workshop on Perceptual Quality of Systems (PQS 2013) %C Vienna, Austria %D 2013 %T Need for Speed? On Quality of Experience for File Storage Services %X Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.

Presentation[ to top ]

Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. Seufert, Michael. 6th International Workshop on Quality of Multimedia Experience (QoMEX), Singapore, 2014, September 19.
- [ BibTeX ]
- [ EndNote ]
@presentation{info3-talk-2014-488, address = {6th International Workshop on Quality of Multimedia Experience (QoMEX), Singapore}, author = {Seufert, Michael}, keywords = {smartenit}, month = {09}, title = {Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming}, year = 2014 }

%0 Generic %1 info3-talk-2014-488 %A Seufert, Michael %C 6th International Workshop on Quality of Multimedia Experience (QoMEX), Singapore %D 2014 %T Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming
Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring. Hirth, Matthias. Qualinet General Meeting, Delft, Netherlands, 2014, October 8.
- [ BibTeX ]
- [ EndNote ]
@presentation{info3-talk-2014-483, address = {Qualinet General Meeting, Delft, Netherlands}, author = {Hirth, Matthias}, keywords = {crowdsourcing-design}, month = 10, title = {Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring}, year = 2014 }

%0 Generic %1 info3-talk-2014-483 %A Hirth, Matthias %C Qualinet General Meeting, Delft, Netherlands %D 2014 %T Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring