To arrive at an effective data monitoring model and process, a number of experiment ‘archetypes’, based on EXPERIMEDIA’s driving experiments, have been considered.
EXPERIMEDIA test-beds potentially offer a wide variety of sources of data for the generation QoS, QoE and QoC metrics. In the following sub-sections, a range of data sources are identified for each of the primary metric dimensions.
The set of potential data sources for QoS metrics that could be used for an EXPERIMEDIA test-bed is likely to be the largest of the three primary metric dimensions. This set has been divided into the following sub-sets:
Examples from each of these sub-categories are provided below, see appendix A for the QoS metrics proposed for the EXPERIMEDIA driving experiments.
Data from the real world environment will be captured by physical sensors connected to the EXPERIMEDIA data monitoring system. This connectivity is unlikely to be direct since it is expected that most QoS based data will be delivered to a monitor server via a digital network and so will initially pass through a third party computing device or ‘data handler’. A few examples of physical environment data sources are provided in the table below:
| Measure | Source | Data handler |
| Temperature | Embedded digital thermometer | Wirelessly connected mobile phone/tablet |
| Light array | Fixed video camera | Video stream proxy with network access |
| Light array | Embedded video camera | Wirelessly connected mobile phone/tablet |
| User count | Physical gate system | Physically connected PC with network access |
In the context of an EXPERIMEDIA test-bed, the scope of the term ‘physical infrastructure’ refers to the hardware directly engaged with the creation, storage, access and delivery of digital content associated with an experiment. The physical hardware falling within this scope may also require a third party data handler (this may be just software but it could also include additional hardware) to provide instrumentation. Examples of such data sources are provided below:
| Measure | Source | Data handler |
| Battery life | Mobile device | None – connected to data monitor server |
| Packet loss | Network router | Network monitor proxy |
| CPU idle time | Mobile device/server | None – connected to data monitor server |
| Disk reads | Server | None – connected to data monitor server |
Metrics in this category are focussed on the physical operation of a piece of infrastructure hardware without reference to a specific software or service – these metrics are described below.
Software and services that form a part of the test-bed infrastructure are identified as logical infrastructure data sources. Measurements taken from these sources are described as logical since they describe the performance of the infrastructure that is determined by the behaviour of the software or service that is run on the supporting physical infrastructure.
| Measure | Source | Data handler |
| Number of active VMs | Cloud service | None – connected to data monitor server |
| Connected WiFi users | Network service | Network monitor proxy |
| Bandwidth throttle time | Network service | Network monitor proxy |
The table above outlines a few examples; once again some intermediate data handlers may be required where the software or service itself may not be modified to interface with EXPERIMEDIA monitoring systems.
The final sub-category of QoS data sources focusses on the content or application domain that is particular to the experiment and venue itself. In this case, there will be a wide range of possible measurements (depending on the context of the experiment), but fewer direct and indirect sources from which the data can be gathered. Based on EXPERIMEDIA’s driving experiments, an illustration of the potential measurements from a variety of sources is presented in the table below.
| Measure | Source | Driving experiment |
| POI content accesses/user | Schladming POI database | Schladming |
| Rendered frames/second | Augmented reality mobile application | Schladming |
| Video transcoding target bitrate | AVC transcoding component | CAR |
| Video frame random access speed | AVC media distribution component | CAR |
| Expert SN posts/programme | SCC SN analytics component | FHW |
| Virtual location visits/programme | Tholos content repository | FHW |
In these cases, it is assumed that all monitoring of measures will be conducted by the content service or application and delivered to the data monitoring server.
EXPERIMEDIA frames quality of experience as a synthesis of objective and subjective measurements which, combined, provide the experimenter with a rich picture of a user’s experience within the context of their interaction with FMI technologies. Data sources acting as a basis for QoE metrics are sub-divided as follows:
As with some QoS monitoring scenarios, QoE metrics may be generated from direct and indirect sources. Indicative examples of each of the five QoE sub-categories are provided in the sections below.
Some venues may provide the opportunity to directly instrument users during an experiment – EXPERIMEDIA’s CAR venue is a good example. Physical data capture may vary in its methods including physiological sensors and remote sensing (using physical modelling and analytics).
| Measure | Source | Data handler |
| Heart-rate | Physically attached sensor & wireless transmitter | Networked monitor proxy (local PC) |
| Gesture | Human motion tracker | Networked monitor proxy (local PC) |
| Proximity | Embedded GPS receiver | Wirelessly connected mobile phone/tablet |
The table above presents a number of potential quantitative measures that could be used to augment a quality of experience data set. As with some QoS instrumentation techniques, some of the physical data sources will require a dedicated data handler.
An additional set of objective measures that can potentially provide insight into the quality of experience is that associated with logging human-computer interactions. Included in this set of measures are simple forms of device operation (physical button pushes) to more complex interactions, such as the completion of an information processing task via a human-machine dialogue or the movement and actions of an avatar in a virtual space.
| Measure | Source | Data handler |
| Data input error rate | Venue provider’s web application | Web application server |
| Task completion | Pervasive game engine | Game engine server |
| AR target selection | AR client viewer | Wirelessly connected mobile phone/tablet |
The table above provides some examples of different kinds of human-computer interaction logging in various contexts. Measures such as these would necessarily be gathered by instrumenting the software associated with the delivery of content within the experiment. This quantitative information provides an important contextualising dimension to the subjective data types discussed in the proceeding sections.
Many of the experimental scenarios envisaged within EXPERIMEDIA include tight coupling of on-line, user-generated content that includes textual, audio, pictorial and video streams. All of these mediums have the potential to offer self-reporting data relating to personal or group activities and thus can act as a source of quality of experience data; examples are provided in the table below.
| Measure | Source | Data handler/processor |
| User clustering/grouping | On-line posting/photo tagging | SCC analytics component |
| Activity/event frequency | On-line posting/photo tagging | SCC analytics component |
| Activity/event frequency | On-line video posting | AVC analytics component |
Due to the nature of the medium (informal and irregular human communication) from which activity data could be extracted, many of the activity measures will require an intermediary that is capable of formally classifying the subjective reporting of human activities for the purpose of providing data for later analysis.
Currently, it is impossible to directly measure a ‘human experience’ – evidence of this phenomenon can only be gathered by using self-reporting techniques that generate data which can be analysed within a theoretical framework. Unlike some of the other measures described above, many of the sampling techniques used in capturing QoE measures gather samples (typically along an ordinal scale) which are aggregated to derive a final metric.
| Measure | Source | Data handler |
| Perceived ease of use (PEU) | On-line questionnaire | Web application server |
| Positive/negative affect valences | QoE sampler | Wirelessly connected mobile phone/tablet |
Two examples of components used in QoE evaluation methods are presented in the table above. Perceived ease of use is one component that is derived from a sample of scaled questionnaire responses used in Davis’ Technology Acceptance model. Positive and negative affect valences are dimensions that encapsulate self-report samples of experience including bi-polar descriptors such as irritable and relaxed , attentive and distracted .
Projected EXPERIMEDIA experimental contexts imagine that users will engage with a range of online content from a variety of social media providers such as Facebook , Twitter or YouTube . During an experiment it will be important to recognise that differing online social content providers will be used, however from an experimental data monitoring point of view, the sources of such data will be abstracted by the SCC component. The focus for QoC data sources is therefore sub-divided into analytical dimensions: