(en)The present specification describes methods and systems for modifying a media, such as Virtual Reality, Augmented Reality, or Mixed Reality (VR/AR/MxR) media based on a vision profile and a target application. In embodiments of the specification, a Sensory Data Exchange (SDE) is created that enables identification of various vision profiles for users and user groups. The SDE may be utilized to modify one or more media in accordance with each type of user and/or user group.

1.ApplicationNumber: US-201715482544-A
1.PublishNumber: US-2017293356-A1
2.Date Publish: 20171012
3.Inventor: KHADERI SYED KHIZER RAHIM
REDDY MOHAN KOMALLA
McDermott Kyle Christopher

4.Inventor Harmonized: KHADERI SYED KHIZER RAHIM(US)
REDDY MOHAN KOMALLA(US)
McDermott Kyle Christopher(US)

5.Country: US
6.Claims:
(en)The present specification describes methods and systems for modifying a media, such as Virtual Reality, Augmented Reality, or Mixed Reality (VR/AR/MxR) media based on a vision profile and a target application. In embodiments of the specification, a Sensory Data Exchange (SDE) is created that enables identification of various vision profiles for users and user groups. The SDE may be utilized to modify one or more media in accordance with each type of user and/or user group.

7.Description:
(en)CROSS-REFERENCE
The present application relies on, for priority, the following United States Provisional Patent Applications, which are also herein incorporated by reference in their entirety:
U.S. Provisional Patent Application No. 62/425,736, entitled “Methods and Systems for Gathering Visual Performance Data and Modifying Media Based on the Visual Performance Data” and filed on Nov. 23, 2016;
U.S. Provisional Patent Application No. 62/381,784, of the same title and filed on Aug. 31, 2016;
U.S. Provisional Patent Application No. 62/363,074, entitled “Systems and Methods for Creating Virtual Content Representations Via A Sensory Data Exchange Platform” and filed on Jul. 15, 2016;
U.S. Provisional Patent Application No. 62/359,796, entitled “Virtual Content Representations” and filed on Jul. 8, 2016;
U.S. Provisional Patent Application No. 62/322,741, of the same title and filed on Apr. 14, 2016; and U.S. Provisional Patent Application No. 62/319,825, of the same title and filed on Apr. 8, 2016.


FIELD
The present specification relates generally to virtual environments, and more specifically to methods and systems for modifying media, such as virtual reality-based, augmented reality-based, or mixed reality-based (VR/AR/MxR) media, based on an individual's vision profile and/or a target application.
BACKGROUND
In recent years, Virtual Reality (VR) environments, Augmented Reality (AR), and Mixed Reality (MxR) applications have become more common. While VR is a non-invasive simulation technology that provides an immersive, realistic, three-dimensional (3D) computer-simulated environment in which people perform tasks and experience activities as if they were in the real world; AR depicts a real world environment that is augmented or supplemented by computer generated media. The most direct experience of VR/AR/MxR is provided by fully immersive VR/AR/MxR systems, and the most widely adopted VR/AR/MxR systems display their simulated environment through special wearable Head-Mounted visual Displays (HMDs). HMDs typically consist of screens and lenses fitted into glasses, helmets or goggles, with a display that may be monocular (display seen by one eye only), binocular (both eyes view a single screen), or dichoptic (each eye views a different screen or image that can be stereoscopic, which gives additional depth cues).
Although HMDs have recently been introduced to the general public, they are not a new phenomenon. As early as the 1960s, computer graphics pioneer Ivan Sutherland developed the first HMD, which made it possible to overlay virtual images on the real world. HMD technology gradually evolved through the 1970s with use across military, industry, scientific research and entertainment domains. The early commercially available HMDs, such as the Virtual Research Flight Helmet™, Virtual I/O I-Glasses™, and Dynovisor™, had limited applications due to their narrow field-of-view (FOV) and inherent cumbersomeness in weight, physical restrictions, and system parameters. Recent advancements have been directed toward making HMDs more comfortable for longer duration of use. Recent HMD products including Google Glass™, Epson Moverio™, Vuzix Wrap™, and Oculus Rift™ have become commercially available and increasingly commonplace as a result of technical advancements. For example, one version of the Oculus Rift™, the Development Kit 2 (DK2), has a high resolution, high refresh rate (i.e., the frequency with which a display's image is updated), low persistence (which aids in removing motion blur), and advanced positional tracking for lower latency and precise movement, when compared to its predecessors. HMD technology advancement and cost reduction has increased its potential for widespread use.
Unfortunately, a number of vision-related conditions are associated with the use of such technology. For example, visually induced motion sickness (VIMS) or simulation sickness, which is related to visual-vestibular mismatch, has been attributed to significant systemic and perceptual problems inherently associated with the use of HMDs and remains an obstacle to the widespread adoption and commercial development of technologies associated with VR/AR/MxR-based HMDs. The systemic and perceptual problems with HMDs, not typically associated with traditional displays, include nausea, stomach discomfort, disorientation, postural instability and visual discomfort.
It is commonly accepted that the symptoms of nausea and instability result from various sensory input conflicts, including conflicting position and movement cues, leading to a disharmonious effect on the visual and vestibular systems. In addition, specific types of HMDs and also other apparatuses that provide virtual environments, may have mismatch problems with the user's visual system due to improper optical design, resulting in convergence-accommodation conflict and visual discomfort or fatigue. Other studies have reported high incidence of visual discomfort including eyestrain, dry eye, tearing, foreign body sensation, feeling of pressure in the eyes, aching around the eyes, headache, blurred vision, and difficulty in focusing. Other visual problems such as myopia, heterophoria, fixation disparity, vergence-accommodation disorders, and abnormal tear break-up time (TBUT) also have been reported. Using HMDs may cause accommodative spasm that in turn may lead to a transient myopia. Continued conflict between convergence-accommodation, the user's inter-pupillary distance (IPD), and/or the systems' inter-optical distance (IOD) may lead to heterophoria and fixation disparity changes. Moreover, visual symptoms are not necessarily limited to the time of actual virtual environment (VE) immersion; rather, visual changes including visual fatigue, reduced visual acuity and heterophoria may continue after terminating exposure to HMD-based VE. Users are often required to avoid driving or operating heavy machinery after exposure to VR/AR/MxR until VIMS and postural instability resolve. Complex visual tasks and reading during and after exposure to VR/AR/MxR may increase severity of VIMS.
Advances in HMD technology have provided the potential for its widespread use in VR/AR/MxR. However, VIMS still remains an obstacle to public adoption and commercial development of this technology. Visual discomfort induced by VR/AR/MxR in VE may be reduced by optimizing quality and design of VR/AR/MxR apparatuses such as HMDs. However, there is still a need for methods and systems that can resolve visual-vestibular mismatch and adapt VR/AR/MxR to the visual capacity of a user and/or a group of users in order to minimize and/or eliminate VIMS. Current visual measures and rating systems for VR/AR/MxR are qualitative in nature. There is also a need to establish quantitative measures to improve the quality of the user experience in VR/AR/MxR environments.
What is also needed is a system that is capable of grouping individuals based on demographic or other common factors to identify an acceptable modifications of visual media, thus reducing the levels of discomfort. A system is also needed that may adapt to an identified user or group in order to modify and present VR/AR/MxR media that reduces discomfort. A system is also needed that may identify delay and other forms of data including biometric data, and their patterns, to recommend and/or automate or dynamically change a VR/AR/MxR environment based on the data and the patterns.
SUMMARY
In some embodiments, the present specification is directed toward a method of improving or treating a condition experienced by a user, while said user is experiencing media using a computing device with a display, such as, but not limited to, a conventional laptop, mobile phone, tablet computer, desktop top computer, gaming system, virtual reality, augmented reality, and mixed reality view device. The method comprises acquiring a first value for at least one of the plurality of data using said computing device; acquiring a second value for the at least one of the plurality of data using said computing device; using said first value and second value, determining a change in at least one of the plurality of data over time; based upon said change in the at least one of the plurality of data over time, determining a degree of said condition; and based upon determining a degree of said condition, modifying said media.
Optionally, the computing device is a virtual reality, augmented reality, or mixed reality view device.
Optionally, the virtual reality, augmented reality, or mixed reality view device comprises at least one of a camera configured to acquire eye movement data, a sensor configured to detect a rate and/or direction of head movement, a sensor configured to detect a heart rate, and an EEG sensor to detect brain waves.
Optionally, the eye movement data comprises rapid scanning, saccadic movement, blink rate data, fixation data, pupillary diameter, and palpebral fissure distance.
Optionally, the condition is at least one of comprehension, fatigue, engagement, performance, symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch, symptoms associated with post-traumatic stress disorder, double vision related to accommodative dysfunction, vection due to unintended peripheral field stimulation, vergence-accommodation disorders, fixation disparity, blurred vision and myopia, headaches, difficulties in focusing, disorientation, postural instability, visual discomfort, eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, aching around the eyes, nausea, stomach discomfort, potential phototoxicity from overexposure to screen displays, hormonal dysregulation arising from excessive blue light exposure, heterophoria, decrease in positive emotions, and increase in negative emotions.
Optionally, the plurality of data comprises at least one of rapid scanning, saccadic movement, fixation, blink rate, pupillary diameter, speed of head movement, direction of head movement, heart rate, motor reaction time, smooth pursuit, palpebral fissure distance, degree and rate of brain wave activity, degree of convergence, and degree of convergence.
Optionally, the modifying of media comprises at least one of increasing a contrast of the media, decreasing a contrast of the media, making an object of interest that is displayed in the media larger in size, making an object of interest that is displayed in the media smaller in size, increasing a brightness of the media, decreasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, decreasing an amount of an object of interest displayed in the media shown in a central field of view and increasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media changing RGB values of said media based upon external data, demographic or trending data.
Optionally, the condition is comprehension.
Optionally, the change is at least one of increased rapid scanning, increased saccadic movement, decreased fixation, increased blink rate, increased pupillary diameter, increased head movement, increased heart rate, decreased reaction time, decreased separation of the eyelids, changes in brain wave activity, and increased smooth pursuit.
Optionally, the degree of the condition is a decreased comprehension of the user.
Optionally, based on said decreased comprehension of the user, said media is modified by at least one of increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data, demographic or trending data.
Optionally, the condition is fatigue.
Optionally, the change is at least one of decreased fixation, increased blink rate, and changes in convergence and divergence.
Optionally, the degree of the condition is an increased fatigue of the user.
Optionally, based on said increased fatigue of the user, said media is modified by at least one of increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, and increasing or introduction more motion.
In some embodiments, the present specification is directed toward a method of improving comprehension experienced by a user, while the user is experiencing media through a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of reduced comprehension of the user; and modifying media based upon determining a degree of reduced comprehension.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined increase in comprehension.
In some embodiments, the present specification is directed toward a method of decreasing fatigue experienced by a user, while the user is experiencing media through a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of increased fatigue of the user; and modifying media based upon determining a degree of increased fatigue.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring using at least one or more of: using one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality data of comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, the modifying the media comprises modifying to provide a predefined decrease in fatigue.
In some embodiments, the present specification is directed toward a method of increasing engagement of a user, while the user is experiencing media through a computing device with a display, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decreased engagement of the user; and modifying media based upon determining a degree of decreased engagement.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, the modifying the media comprises modifying to provide a predefined increase in engagement.
In some embodiments, the present specification is directed toward a method of improving performance of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of improvement in performance of the user; and modifying media based upon determining a degree of improved performance.
Optionally, the acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: sensor configured to detect basal body temperature, heart rate, body movement/rotation/direction/velocity/amplitude; sensor configured to measure limb movement/rotation/direction/velocity/amplitude; sensor configured to measure pulse rate, and other parameters similar to a pulse oximeter; sensor configured to measure auditory processing; sensor configured to measure gustatory and olfactory processing; sensor to measure pressure; input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; electroencephalography; electrocardiography; electromyography; electrooculography; electroretinography; and sensor configured to measure Galvanic Skin Response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, the modifying the media comprises modifying to provide a predefined increase in performance.
In some embodiments, the present specification is directed toward a method of decreasing symptoms associated with Visually-Induced Motion Sickness (VIMS) secondary to visual-vestibular mismatch, of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in VIMS symptoms of the user; and modifying media based upon determining a degree of decrease in VIMS symptoms.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, the modifying the media comprises modifying to provide a predefined decrease in VIMS symptoms.
In some embodiments, the present specification is directed toward a method of decreasing symptoms associated with Post-Traumatic Stress Disorder (PTSD), of a user, while the user is experiencing media through a computing device with a display including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in PTSD symptoms of the user; and modifying media based upon determining a degree of decrease in PTSD symptoms.
Optionally, the method further includes combining at least one of image processing methods, machine learning methods, electronic stimulation, and chemical stimulation, with the change in the plurality of data over time, wherein the combining is used for purposes of neuro-programming.
Optionally, the method further comprises combining at least one of image processing methods, machine learning methods, electronic stimulation, and chemical stimulation, with the change in the plurality of data over time, wherein the combining is used to modify light stimuli while the user is experiencing media through the virtual reality, augmented reality, or mixed reality view device.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in PTSD symptoms.
In some embodiments, the present specification is directed toward a method of decreasing double vision related to accommodative dysfunction of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in double vision of the user; and modifying media based upon determining a degree of decrease in double vision.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in double vision.
In some embodiments, the present specification is directed toward a method of decreasing vection due to unintended peripheral field stimulation of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in vection of the user; and modifying media based upon determining a degree of decrease in vection.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in vection.
In some embodiments, the present specification is directed toward a method of decreasing hormonal dysregulation arising from excessive blue light exposure of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in hormonal dysregulation; and modifying media based upon determining a degree of decrease in hormonal dysregulation.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in hormonal dysregulation.
In some embodiments, the present specification is directed toward a method of decreasing phototoxicity from overexposure to screen displays of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in phototoxicity; and modifying media based upon determining a degree of decrease in phototoxicity.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in phototoxicity.
In some embodiments, the present specification is directed toward a method of decreasing nausea and stomach discomfort of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in nausea and stomach discomfort; and modifying media based upon determining a degree of decrease in nausea and stomach discomfort.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in nausea and stomach discomfort.
In some embodiments, the present specification is directed toward a method of decreasing visual discomfort of a user, including at least one of eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, or aching around the eyes, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in visual discomfort; and modifying media based upon determining a degree of decrease in visual discomfort.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in visual discomfort.
In some embodiments, the present specification is directed toward a method of decreasing disorientation and postural instability of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in disorientation and postural instability; and modifying media based upon determining a degree of decrease in disorientation and postural instability.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: using one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in disorientation and postural instability.
In some embodiments, the present specification is directed toward a method of decreasing headaches and difficulties in focusing of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in headaches and difficulties in focusing; and modifying media based upon determining a degree of decrease in headaches and difficulties in focusing.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in headaches and difficulties in focusing.
Optionally, the present specification is directed toward a method of decreasing blurred vision and myopia of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in blurred vision and myopia; and modifying media based upon determining a degree of decrease in blurred vision and myopia.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in blurred vision and myopia.
In some embodiments, the present specification is directed toward a method of decreasing heterophoria of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in heterophoria; and modifying media based upon determining a degree of decrease in heterophoria.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in heterophoria.
In some embodiments, the present specification is directed toward a method of decreasing fixation disparity of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in fixation disparity; and modifying media based upon determining a degree of decrease in fixation disparity.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in fixation disparity.
In some embodiments, the present specification is directed toward a method of decreasing vergence-accommodation disorders of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in vergence-accommodation disorders; and modifying media based upon determining a degree of decrease in vergence-accommodation disorders.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, changing RGB values of said media based upon external data (demographic or trending data), increasing use of longer viewing distances when possible, matching simulated distance with focal distance more closely, moving objects in and out of depth at a slower pace, and making existing object conflicts less salient.
Optionally, the modifying the media comprises modifying to provide a predefined decrease in vergence-accommodation disorders.
In some embodiments, the present specification is directed toward a method of increasing positive emotion of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of increase in positive emotion; and modifying media based upon determining a degree of increase in positive emotion.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined increase in positive emotion.
In some embodiments, the present specification is directed toward a method of decreasing negative emotion of a user, while the user is experiencing media through a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device, the method comprising: acquiring a first value for a plurality of data; acquiring a second value for the plurality of data; using the first value and the second value to determine a change in the plurality of data over time; based upon the change in the plurality of data over time, determining a degree of decrease in negative emotion; and modifying media based upon determining a degree of decrease in negative emotion.
Optionally, acquiring the first value and the second value of the plurality of data comprises acquiring at least one or more of: a sensor configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, body velocity, or body amplitude; a sensor configured to measure limb movement, limb rotation, limb direction, limb velocity, or limb amplitude; a pulse oximeter; a sensor configured to measure auditory processing; a sensor configured to measure gustatory and olfactory processing; a sensor to measure pressure; an input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; an electroencephalograph; an electrocardiograph; an electromyograph; an electrooculograph; an electroretinography; and a sensor configured to measure galvanic skin response.
Optionally, the plurality of data comprises at least one or more of: palpebral fissure, blink rate, pupil size, pupil position, gaze direction, gaze position, vergence, fixation position, fixation duration, fixation rate; fixation count; saccade position, saccade angle, saccade magnitude, pro-saccade, anti-saccade, inhibition of return, saccade velocity, saccade rate, screen distance, head direction, head fixation, limb tracking, weight distribution, frequency domain (Fourier) analysis, electroencephalography output, frequency bands, electrocardiography output, electromyography output, electrooculography output, electroretinography output, galvanic skin response, body temperature, respiration rate, oxygen saturation, heart rate, blood pressure, vocalizations, inferred efferent responses, respiration, facial expression, olfactory processing, gustatory processing, and auditory processing.
Optionally, modifying the media comprises modifying by at least one of: increasing a contrast of the media, making an object of interest that is displayed in the media larger in size, increasing a brightness of the media, increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view, changing a focal point of content displayed in the media to a more central location, removing objects from a field of view and measuring if a user recognizes said removal, increasing an amount of color in said media, increasing a degree of shade in objects shown in said media, and changing RGB values of said media based upon external data (demographic or trending data).
Optionally, modifying the media comprises modifying to provide a predefined decrease in negative emotion.
In some embodiments, the present specification is directed toward a method of performing a transaction with a user, while said user is experiencing media using a computing device with a display, including a virtual reality, augmented reality, or mixed reality view device comprising obtaining at least one of psychometric, sensory, and biometric information from the user, the at least one of psychometric, sensory, and biometric information comprising one or more values for at least one of the plurality of data using said virtual reality, augmented reality, or mixed reality view device; rewarding the user for the obtained at least one of psychometric, sensory, and biometric information; using said one or more values to determine a change in at least one of the plurality of data over time; based upon said change in the at least one of the plurality of data over time, modifying said media.
The aforementioned and other embodiments of the present shall be described in greater depth in the drawings and detailed description provided below.



BRIEF DESCRIPTION OF THE DRAWINGS
These and other features and advantages of the present specification will be appreciated, as they become better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
FIG. 1A shows a block diagram illustrating user interaction with an exemplary Sensory Data Exchange Platform (SDEP), in accordance with an embodiment of the present specification;
FIG. 1B illustrates an exemplary breakdown of functions performed by a data ingestion system and a data processing system;
FIG. 1C illustrates an exemplary machine learning system, in accordance with an embodiment of the present specification;
FIG. 2 is a block diagram illustrating processing of a sensor data stream before it reaches a query processor, in accordance with an embodiment of the present specification;
FIG. 3 illustrates an overview of sources of digital data, in accordance with an embodiment of the present specification;
FIG. 4A illustrates characteristic metrics for visual data, in accordance with an embodiment of the present specification;
FIG. 4B provides a graphical presentation of color pair confusion components, in accordance with an embodiment of the present specification;
FIG. 4C shows a graph illustrating how luminance may be found for a given chromaticity that falls on the top surface of the display gamut projected into 3D chromoluminance space;
FIG. 5 illustrates characteristic metrics for auditory information, in accordance with an embodiment of the present specification;
FIG. 6 illustrates characteristic metrics for eye tracking, in accordance with an exemplary embodiment of the present specification;
FIG. 7 illustrates characteristic metrics for manual input, in accordance with an embodiment of the present specification;
FIG. 8 illustrates characteristic metrics for head tracking, in accordance with an embodiment of the present specification;
FIG. 9 illustrates characteristic metrics for electrophysiological and autonomic monitoring data, in accordance with an embodiment of the present specification;
FIG. 10A illustrates an exemplary process of image analysis of building curated data, in accordance with an embodiment of the present specification;
FIG. 10B illustrates an exemplary process of image analysis of building curated data, in accordance with an embodiment of the present specification;
FIG. 10C illustrates an exemplary process of image analysis of building curated data, in accordance with an embodiment of the present specification;
FIG. 10D illustrates an exemplary process of image analysis of building curated data, in accordance with an embodiment of the present specification;
FIG. 11A illustrates pupil position and size and gaze position over time;
FIG. 11B illustrates pupil position and size and gaze position over time;
FIG. 12 is an exemplary outline of a data analysis chain;
FIG. 13 provides a table containing a list of exemplary metrics for afferent and efferent sources, in accordance with some embodiments of the present specification;
FIG. 14 is an exemplary flow chart illustrating an overview of the flow of data from a software application to the SDEP;
FIG. 15 is an exemplary outline of a pre-processing portion of a process flow, in accordance with an embodiment of the present specification;
FIG. 16 is an exemplary outline of a python scripting portion of the analysis chain;
FIG. 17 is a flow chart illustrating a method of modifying media, in accordance with an embodiment of the present specification;
FIG. 18 is a flow chart illustrating a method of modifying media, in accordance with another embodiment of the present specification;
FIG. 19 illustrates a flow chart describing an exemplary process for improving comprehension, in accordance with some embodiments of the present specification;
FIG. 20 illustrates a flow chart describing an exemplary process for decreasing fatigue, in accordance with some embodiments of the present specification;
FIG. 21 illustrates a flow chart describing an exemplary process for increasing engagement, in accordance with some embodiments of the present specification;
FIG. 22 illustrates a flow chart describing an exemplary process for improving performance, in accordance with some embodiments of the present specification;
FIG. 23 illustrates a flow chart describing an exemplary process for decreasing symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch, in accordance with some embodiments of the present specification;
FIG. 24 illustrates a flow chart describing an exemplary process for decreasing symptoms associated with post-traumatic stress disorder (PTSD), in accordance with some embodiments of the present specification;
FIG. 25 illustrates a flow chart describing an exemplary process for decreasing double-vision related to accommodative dysfunction, in accordance with some embodiments of the present specification;
FIG. 26 illustrates a flow chart describing an exemplary process for decreasing vection due to unintended peripheral field stimulation, in accordance with some embodiments of the present specification;
FIG. 27 illustrates a flow chart describing an exemplary process for decreasing hormonal dysregulation arising from excessive blue light exposure, in accordance with some embodiments of the present specification;
FIG. 28 illustrates a flow chart describing an exemplary process for decreasing potential phototoxicity from overexposure to screen displays, in accordance with some embodiments of the present specification;
FIG. 29 illustrates a flow chart describing an exemplary process for decreasing nausea and stomach discomfort, in accordance with some embodiments of the present specification;
FIG. 30 illustrates a flow chart describing an exemplary process for decreasing visual discomfort, including at least one of eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, or aching around the eyes, in accordance with some embodiments of the present specification;
FIG. 31 illustrates a flow chart describing an exemplary process for decreasing disorientation and postural instability, in accordance with some embodiments of the present specification;
FIG. 32 illustrates a flow chart describing an exemplary process for decreasing headaches and difficulties in focusing, in accordance with some embodiments of the present specification;
FIG. 33 illustrates a flow chart describing an exemplary process for decreasing blurred vision and myopia, in accordance with some embodiments of the present specification;
FIG. 34 illustrates a flow chart describing an exemplary process for decreasing heterophoria, in accordance with some embodiments of the present specification;
FIG. 35 illustrates a flow chart describing an exemplary process for decreasing fixation disparity, in accordance with some embodiments of the present specification;
FIG. 36 illustrates a flow chart describing an exemplary process for decreasing vergence-accommodation disorders, in accordance with some embodiments of the present specification;
FIG. 37 illustrates a flow chart describing an exemplary process for increasing positive emotion, in accordance with some embodiments of the present specification;
FIG. 38 illustrates a flow chart describing an exemplary process for decreasing negative emotion, in accordance with some embodiments of the present specification; and
FIG. 39 is a flow chart describing an exemplary process for modifying media while enabling a micro-transaction, in accordance with some embodiments of the present specification.



DETAILED DESCRIPTION
In various embodiments, the present specification provides methods and systems for enabling modification of media in accordance with a visual profile of a user and/or group of users.
In another embodiment, the present specification describes methods, systems and software that is provided to third party developers of media (advertising and entertainment) who then use the software and data to optimize the presentation of that media for a user's specific vision characteristics.
In yet another embodiment, the present specification describes methods, systems, and software for directly providing media (advertising and entertainment) that already incorporates software and data that, when experienced by a user, can be optimized for that user's specific vision characteristics in real-time.
In one embodiment, a Sensory Data Exchange Platform (SDEP) is provided, wherein the SDEP may enable developers of media for Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MxR) systems and/or software to optimize the media for a user and/or a group of users. In embodiments, users may include programmers and developers of mobile platforms and web sites. In embodiments, the VR, AR, and/or MxR media is presented to an end-user through one or more electronic media devices including computers, portable computing devices, mobile devices, or any other device that is capable of presenting VR, AR, and/or MxR media.
In an embodiment, a user interacts with a software program embodying at least a portion of the SDEP in a manner that enables the software to collect user data and provided it to the SDEP. In an embodiment, the user may interact directly or indirectly with a SDEP to facilitate data collection. In an embodiment, the SDEP is a dynamic, two-way data exchange platform with a plurality of sensory and biometric data inputs, a plurality of programmatic instructions for analyzing the sensory and biometric data, and a plurality of outputs for the delivery of an integrated visual assessment.
In some embodiments, the SDEP outputs as a general collective output a “visual data profile” or a “vision performance index” (VPI). The visual data profile or vision performance index may be used to optimize media presentations of advertising, gaming, or content in a VR/AR/MxR system or a conventional laptop, mobile phone, desktop or tablet computing environment. In embodiments, the platform of the present specification is capable of taking in a number of other data sets that may enhance the understanding of a person's lifestyle and habits. In addition, machine learning, computer vision, and deep learning techniques are employed to help monitor and predict health outcomes through the analysis of an individual's data.
In an embodiment, the SDEP is used via an operating system executed on hardware (such as mobile, computer or Head Mounted Display (HMD)). In another embodiment, the SDEP is used by one or more content developers. In one embodiment, both hardware and content developers use the SDEP. The SDEP may enable collection of data related to how the user is interfacing with the content presented, what aspects of the content they are most engaged with and how engaged they are. Data collected through the SDEP may be processed to create a profile for the user and or groups of users with similar demographics. The content may be represented, for a particular profile, in a way that conforms to the hardware capabilities of the VR/AR/MxR system in a manner to optimize experience of that user and other users with a similar profile.
For example, the experience may be optimized by representing the media in a manner that may decrease phoria movement—specifically, long periods of convergence with simultaneous head movement to minimize visual vestibular mismatch; blending optical zones/focal zones of objects in the VE to minimize accommodative decoupling/dysfunction; disabling large peripheral stimuli during central stimuli engagement, to decrease the experience of vection; among other methods that enable an enhanced VR/AR/MxR experience.
The present specification is directed towards multiple embodiments. The following disclosure is provided in order to enable a person having ordinary skill in the art to practice the invention. Language used in this specification should not be interpreted as a general disavowal of any one specific embodiment or used to limit the claims beyond the meaning of the terms used therein. The general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Also, the terminology and phraseology used is for the purpose of describing exemplary embodiments and should not be considered limiting. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications and equivalents consistent with the principles and features disclosed. For purpose of clarity, details relating to technical material that is known in the technical fields related to the invention have not been described in detail so as not to unnecessarily obscure the present invention.
The term “and/or” means one or all of the listed elements or a combination of any two or more of the listed elements.
The terms “comprises” and variations thereof do not have a limiting meaning where these terms appear in the description and claims.
Unless otherwise specified, “a,” “an,” “the,” “one or more,” and “at least one” are used interchangeably and mean one or more than one.
For any method disclosed herein that includes discrete steps, the steps may be conducted in any feasible order. And, as appropriate, any combination of two or more steps may be conducted simultaneously.
Also herein, the recitations of numerical ranges by endpoints include all whole or fractional numbers subsumed within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, 5, etc.). Unless otherwise indicated, all numbers expressing quantities of components, molecular weights, and so forth used in the specification and claims are to be understood as being modified in all instances by the term “about.” Accordingly, unless otherwise indicated to the contrary, the numerical parameters set forth in the specification and claims are approximations that may vary depending upon the desired properties sought to be obtained by the present invention. At the very least, and not as an attempt to limit the doctrine of equivalents to the scope of the claims, each numerical parameter should at least be construed in light of the number of reported significant digits and by applying ordinary rounding techniques.
Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. All numerical values, however, inherently contain a range necessarily resulting from the standard deviation found in their respective testing measurements.
It should be noted herein that any feature or component described in association with a specific embodiment may be used and implemented with any other embodiment unless clearly indicated otherwise.
It should be further appreciated that all the afferent data presented herein and efferent data collected are performed using a hardware device, such as a mobile phone, laptop, tablet computer, or specialty hardware device, executing a plurality of programmatic instructions expressly designed to present, track, and monitor afferent data and to monitor, measure, and track efferent data, as further discussed below.
General Problem
Potential inciting factors of VIMS may be broadly categorized into the following factor areas: Hardware, User, Task, and Service. Methods and systems are needed that may optimize one or a combination of these factors to ease development and adoption of VR/AR/MxR. These factors are briefly discussed here.
Hardware Factors
Hardware device system variables impacting VIMS include viewing mode (e.g. monocular, binocular or dichoptic), headset design (e.g. fit, weight), optics (e.g. misalignment in the optics; contrast, luminance), field of view (FOV), and time lag (i.e. transport delay). HMD weight has been associated with the experience of visual discomfort and injury. Thus, hardware factors include field of view, resolution and/or frame rate, and time lag or latency, among other variables.
For example, field of view (FOV), may be implicated in producing visual discomfort symptoms. The FOV studies show that narrow FOV (<50 degrees) reduces the perception of self-motion and wide FOV (>100 degrees) may increase the presence and level of simulator sickness. For a full immersion experience, a FOV of at least 60° is recommended. The sense of immersion can be provided by parsing horizontal and vertical FOVs, which allows for flexibility in the content presentation. In flight simulation applications, for example, segmenting object presentation within a horizontal FOV of 40° by a vertical FOV of 30° improves the ergonomics and improves pilot performance.
Aside from influencing overall image quality, resolution may also affect a user's experience of VIMS. It is often uncomfortable to view low-quality images that are noisy or blurry. The visual resolution in humans is 1 minute of arc and is a technological limitation to many HMD systems. Depending on perceived distance in the VR environment, increased resolution mitigates “pixel perception” as objects gets closer. Having said that, it is important to provide the highest possible resolution in the design process in order to better accomplish immersive tasks in virtual reality (and AR and MxR) applications. Refresh or frame rate is another factor affecting visual comfort.
Most VR/AR and mixed environment systems have similar problems related to motion sickness and visual discomfort. One of the sources of these problems is latency or delay. Delay refers to the time it takes from when a user triggers an action to when the results of the user-triggered action are visible through the VR/AR/MxR media. Delay may also be attributed to the screens that are used to display VR/AR/MxR media. Specifically, features of a screen such as screen refresh and response time may attribute to a measure of delay. Measures of an acceptable level of delay, or delay that may not result in motion sickness or other forms of discomfort, may vary for different individuals.
Time lag between the individual and the system's action and reaction potentially could influence a user's experience of VIMS symptoms, as it affects human perception of visual and vestibular cues. Therefore, reducing the sensor error of HMD systems may minimize the VIMS experience. HMD optical characteristics, such as eye relief (a fixed distance from the eyepiece lens to its exit pupil), convergence demand, horizontal disparity, vertical misalignment of displays, inter-ocular rotation difference, vertical-horizontal magnification differences, luminance, focus differences, temporal asynchrony, focal distance, field curvature difference and inter-pupillary distance (IPD), are all potential factors that can induce visual discomfort and headache when they are misaligned or not optimally calibrated.
User Factors
Another factor related to the impact of VIMS is user characteristics because it is known that individuals differ in their susceptibility to VIMS. User characteristics may include, among others, age and gender; visual deficits; plasticity; and posture.
Age has been shown to have a significant relationship with HMD-related eyestrain symptoms. Children 2-12 years of age have immature visual systems and binocular function that is worse than that of adults; this makes children more susceptible to both visual discomfort caused by HMDs and oculomotor side effects including reduced visual acuity, amblyopia, or strabismus. Adults with limited fusional ranges experienced more visual discomfort, specifically with convergent eye movement in response to stimuli in VEs. Therefore, age effect on HMDs needs to be further studied and incorporated into the design of future HMDs. In regards to gender, females reported more simulator sickness and more often withdrew from HMD-based VEs when compared to male participants. This difference may be due to under-reporting from self-reports by males (so-called “macho effect”) or hormonal effects. Another possibility is the gender difference in FOV, with female having a wider FOV, increasing the risk for flicker perception, vection and motion sickness susceptibility.
People with visual deficits may have an increased susceptibility to oculomotor side effects compared to those without such deficits, though more studies are needed in this area. A past history of motion sickness or conditions that preclude these symptoms (migraines), are also notable for predicting susceptibility to motion sickness in HMD-based VEs. Individuals may habituate or adapt to HMD-based VEs (i.e. plasticity), with improvement in symptoms after repeated exposure to virtual environments. However, this habituation is variable among groups, with certain individuals adapting much more readily than others following repeated exposures to stimuli.
Individuals with more plasticity may be less likely to experience VIMS, though there may be variability in the amount of time needed to adapt to the VE. Greater plasticity does not translate to reduction or lack of initial symptoms, but rather the ability to improve susceptibility to symptoms quicker, typically following repeated exposures to VEs.
Based on the postural instability theory, an individual's posture may also contribute to VIMS. Postural instability has a unique role in VIMS, as it can be a cause for and a result of VIMS. Studies have noted individuals with stable posture are less susceptible to VIMS, suggestive of an inverse relationship between postural stability and VIMS. Postural stability is a confluence of sensory inputs that are visual, vestibular and somatosensory in nature. Two neural reflexes involved in postural stability include the vestibular-ocular reflex (stabilizes objects on the retina) and the vestibular-spinal reflex (stabilizes posture while body in motion). When visual and vestibular inputs are not in synchrony, the result is postural instability, VIMS, or both. Postural instability may last for several hours after exposure. Special considerations for HMD user safety, as related to the risk of postural instability, must be kept in mind. A recommendation that HMD users allow for a reorientation/recovery time prior to engaging in potentially dangerous activities such as driving or sports may be in order.
Task Factors
Task characteristics have been also identified as potentially affecting VIMS. Among these tasks, duration of time in Virtual Environments (VE) is most notable. Longer exposure to VE increases the incidence of VIMS. These symptoms may persist up to 60 minutes after exposure. Another important factor shown to influence VIMS is vection (i.e. an illusion of self-motion), with faster vection resulting in greater sickness symptoms. Viewing HMD-based VR in a sitting position may reduce symptoms, as sitting reduces the demands on postural control. More complicated tasks, such as reading, may induce total symptom severity scores and oculomotor-related symptom scores that are significantly higher than those observed with movies or games. These findings imply that more demanding tasks probably will create some degree of eyestrain. Increased reading sensitivity, when compared to watching a movie or playing a game, might be due to activation of different areas of the brain, which may make reading more complex than other tasks. Alternatively, reading can affect attention and blink rate, which may also contribute to an increase in VIMS. Moreover, inappropriate vertical gaze angle may cause increased oculomotor changes and visual discomfort.
Service Factors
Technical advances in hardware components will reduce physiologic ergonomic issues including HMD weight, system time delay, and luminance. However, given the multifactorial nature of VIMS, the conflict between visual and vestibular input remains a significant problem. Further reduction in VIMS needs to take into consideration how content is created and how it influences service factors. Service factors, in turn, need to take into consideration the VIMS effect on the viewer.
Services can be created intended for a wide audience shared experience (e.g. Broadcast) or for narrow niche audience experience (e.g. longtail content in video on demand systems). For large scale audiences, further VIMS reduction will be highly dependent on content creation, where FOV creates an immersive environment where watching a movie or playing a game would be more preferred rather than reading. User factors could be mitigated by reducing visual cues that create sensory input conflicts. This could be done by making the viewer more of a detached fixed observer (i.e. reduced vection) and making the content more of a scenery type which has already been made popular in viewing UHD/HDR material.
Content may be transmitted via linear broadcast, on-demand, or downloaded among other avenues. Service factors for VR need to be cognizant of the bits delivered to HMDs. This stream of bits constrains resolutions and frame rates while also affecting time-lag or latency where frame rate is a gating factor to latency. Ways to create a service around this dimension are to permit progressive download, on-demand streaming, or real-time linear streaming deliveries. This progression tracks like the evolution of Video streaming where progressive download was initially done until codec compression efficiencies and bandwidth expansion advanced enough to allows for online streaming and ultimately real-time encoding/streaming to occur.
VR environments hinge on some level of interactivity between the viewer and the observed environment. The accepted amount of interactivity can be designed into the service and should consider a reduction of VIMS effects.
Optimized service factors can reduce VIMS effects by creating and making aware to users and the content community a set of guidelines of optimized visual ergonomics for HMD use.
Next generation computing will be dominated by immersive platforms that represent a new level of synergy between users, content, and technology. VR/AR/MxR is the fourth major platform shift after PC, web, and email. VR/AR/MxR is also a part of technology commonly known as Brain-Machine (Computer) Interface (BMI/BCI). BMI/BCI has clinical applications, such as and not limited to EEG, MRI, fMRI, and Ultrasound. Most of these clinical interfaces are modular in nature. These can be invasive, or non-invasive; portable, or non-portable. Non-clinical applications may include gaming, military, and others. Among these different potential interfaces, the division in clinical and non-clinical context, is in part limited to the portability of the interfaces, with non-clinical being traditionally more portable. It is expected that an area of most intensive future development and investment is likely to be portable non-invasive systems such as gaming, especially incorporating non-invasive BMI with AR, VR, and MxR. There is a need developing to standardize BMI for clinical and non-clinical applications.
Overall BMI standardization requires standardizing the interoperability, connectivity, and modularity of multiple sensory interfaces with the brain, with many being closed-looped. There is thus a need for methods and systems that, given the current limitations of closed-loop systems, can support a standardization of these requirements.
Current health risks of BMI include visually-induced motion sickness secondary to visual-vestibular mismatch, double vision related to accommodative dysfunction, vection to unintended peripheral field stimulation, hormonal dysregulation (circadian rhythm) from blue light exposure, and potential phototoxicity from overexposure to screen displays. HMDs are also influenced by user factors including gender, inter-pupillary distance variances, accommodative amplitude (age-dependent), postural stability and the type of software content being displayed—as more visually-task oriented content tend to be more disruptive.
Definitions
The term “Virtual Reality” or “VR” is used throughout this specification, and, in embodiments, refers to immersive computer-simulated reality, or the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside and/or gloves fitted with sensors.
In embodiments, Augmented Reality (AR), also used along with VR throughout this specification, is a technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view. In embodiments, a common helmet-like device is the HMD, which is a display device, worn on the head or as part of the helmet, that has a small display optic in front of one (monocular HMD) or each eye (binocular HMD). In embodiments, the SDEP is a cloud-based service that any party can access in order to improve or otherwise modify a visually presented product or service.
Further, in embodiments, Mixed Reality (MxR), is also used with VR and AR throughout this specification. MxR, also referred to as hybrid reality, is the merging of VR and/or AR environments with the real environment to produce new levels of visual-experiences where physical and digital objects co-exist and interact in real time.
In embodiments, VR, AR, and MxR devices could include one or more of electronic media devices, computing devices, portable computing devices including mobile phones, laptops, personal digital assistants (PDAs), or any other electronic device that can support VR, AR, or MxR media. It should be noted herein that while the present specification is disclosed in the context of Virtual Reality, any and all of the systems and methods described below may also be employed in an Augmented Reality environment as well as Mixed Reality environments. So, where a Virtual Reality (VR) system is described, it should be understood by those of ordinary skill in the art that the same concepts may apply to an Augmented Reality (AR) and a Mixed Reality (MxR) system.
Eye-Tracking Definitions
In terms of performance, several eye tracking measures could be put into the context of Vision Performance Index (VPI) components, which are defined and described in detail in subsequent section of the specification. Blink rate and vergence measures can feed into measures of Fatigue and Recovery. Gaze and, more specifically, fixation positions can be used to estimate Reaction and Targeting measures. Continuous error rates during pursuit eye movements can also become targeting measures.
Various examples of physical measures for eye tracking may be available with desired standard units, expected ranges for measured values and/or, where applicable, thresholds for various states or categories based on those measures. Some references are provided through sections that discuss various components and subcomponents of eye tracking.
The following terms are associated with eye-tracking measures as made from a combination of video recording and image processing techniques; expert human scoring; and/or from Electrooculography (EOG) recording. Video eye tracking (VET) techniques may use explicit algorithmic analysis and/or machine learning to estimate proportional eyelid opening/closure, pupil size, pupil position (relative to the face) and gaze direction independently for each eye. EOG recording may be used to estimate eyelid and eye motion and, with limited precision, eye gaze direction. Both recording modalities may sample at rates of tens to thousands of times per second and allow for analysis of position, velocity, direction, and acceleration for the various measures. Comparison between the two eyes allows for measures of vergence which in turn allows for a three-dimensional (3D) gaze direction to be estimated.
Palpebral Fissure refers to the opening of the eyelids. While typically about 30 millimeters (mm) wide by 10 mm tall, most measurements can be relative to baseline distances measured on video. Of particular interest is the height (interpalpebral fissure height) as it relates to the following terms:
Percent Open (p eye open ) refers to how open the left (p left eye open ), right (p right eye open ), or both (p both eyes open ) eyes are, relative to the maximum open distance and typically measured over a predefined period of time.
Proportion Open (P eyes open ) refers to the proportion of time the eyes are open over a span of time (for example, during a session (P_(eyes open|session))). The threshold for ‘open’ may be variable (for example, P eyes open (where p both eyes open ≧25%)).
Blink can be defined as a complete closure of both eyes (p both eye open =0%) for between roughly 10 to 400 milliseconds (ms), with a specific measured blink closure time being based on differences among users and the eye tracking method.
Blink Rate (Frequency) (f blink ) refers to the average number of blinks per second (s −1 or Hz) measured for all blinks and/or blinks over a period of time (e.g. f_(blink|target present)). The blink rate may be referred to as a rate of change of the blink rate or a ratio of partial blinks to full blinks.
Blink Count Number (N_blink) refers to the number of blinks measured for all blinks and/or blinks over a period of time (e.g. N_(blink|target present)).
Pupil Size (S_pupil) refers to the size of the pupil, typically the diameter in millimeters (mm).
Pupil Position ( [x, y] _pupil) refers to the position of the left ( [x, y] _(left pupil)) or right ( [x, y] _(right pupil)) pupil within the fixed reference frame of the face, typically as a function of time. The pupil position definition includes, and is dependent upon, an initial pupil position and a final pupil position.
Gaze Direction ( [θ,φ] _gaze) refers to the direction in 3D polar coordinates of left ( [θ,φ] _(left gaze)) or right ( [θ,φ] _(right gaze)) eye gaze relative to the face, typically as a function of time. This is a measure of where the eyes are facing without regard to what the eyes see. It may be further classified as relevant or irrelevant depending on a task or a target.
Gaze Position ( [x, y, z] _gaze or [r, θ, φ] _gaze) refers to the position (or destination) of gaze in the environment in Cartesian or spherical 3D coordinates, typically as a function of time. The reference frame may be with respect to the user, device or some other point in space, but most commonly the origin of a coordinate space will be the user's eyes (one or the other or a point halfway between). The gaze position definition includes, and is dependent upon, an initial gaze position and a final gaze position.
Vergence is derived from estimated gaze direction and may be quantified as the difference in angle of the two eyes (positive differences being divergence and negative being convergence). When derived from gaze position, vergence contributes to and may be quantified as the distance of the gaze position from the eyes/face. Convergence and divergence may each be defined by their duration and rate of change.
Fixation Position ([x, y, z] fixation or [r, θ, φ] fixation ) is the position of a fixation in Cartesian or spherical 3D space measured as the estimated position of the user's gaze at a point in time. The fixation position definition includes, and is dependent upon, an initial fixation position and a final fixation position.
Fixation Duration (D fixation ) is the duration of a fixation (i.e. the time span between when the gaze of the eye arrives at a fixed position and when it leaves), typically measured in milliseconds or seconds (s). The average duration is denoted with a bar D fixation and may represent all fixations, fixations over a period of time (e.g. _D_(fixation|target present)) and/or fixations within a particular region (e.g. _D_(fixation|display center)). The fixation duration definition includes, and is dependent upon, a rate of change in fixations.
Fixation Rate (Frequency) (f_fixation) refers to the average number of fixations per second (ŝ(−1) or Hz) measured for all fixations, fixations over a period of time (e.g. f_(fixation|target present)) and/or fixations within a particular region (e.g. f_(fixation|display center)).
Fixations Count (Number) (N fixation ) refers to the number of fixations measured for all fixations, fixations over a period of time (e.g. N_(fixation|target present)) and/or fixations within a particular region (e.g. N_(fixation|display center)).
Saccade Position ([x 1 , y 1 , z 1 |x 2 , y 2 , z 2 ] saccade or [r 1 , θ 1 , φ 1 |r 2 , θ 2 , θ 2 , φ 2 ] saccade ) refers to the starting (1) and ending (2) positions of a saccadic eye movement in Cartesian or spherical 3D space. The reference frame will generally be the same, within a given scenario, as that used for gaze position. The saccade position definition includes, and is dependent upon, a rate of change, an initial saccade position, and a final saccade position.
Saccade Angle (Θ accade ) refers to an angle describing the 2-dimensional (ignoring depth) direction of a saccade with respect to some reference in degrees (°) or radians (rad). Unless otherwise specified the reference is vertically up and the angle increases clockwise. The reference may be specified (e.g. (Θ accade target ) to denote the deviation of the saccade direction from some desired direction (i.e. towards a target). The average saccade direction is denoted with a bar Θ saccade and may represent all or a subset of saccades (e.g. _ Θ _(saccade|target present)); because the direction is angular (i.e. circular) the average direction may be random unless a relevant reference is specified (e.g. _Θ_(saccade-target|target present)). The saccade angle may be used to determine how relevant a target is to a user, also referred to as a context of relevancy towards a target.
Saccade Magnitude (M saccade ) refers to the magnitude of a saccade relating to the distance traveled; this may be given as a visual angle in degrees (°) or radians (rad), a physical distance with regard to the estimated gaze position (e.g. in centimeters (cm) or inches (in)) or a distance in display space with regard to the estimated gaze position on a display (e.g. in pixels (px)). In reference to a particular point (P) in space, the component of the saccade magnitude parallel to a direct line to that point may be given as:
M saccade-P =M saccade ·cos(Θ saccade-P )

where M saccade is the magnitude of the saccade and Θ accade-P is the angle between the saccade direction and a vector towards point P. The average saccade magnitude is denoted with a bar M saccade , and this notation may be applied to all saccades and/or a subset in time or space and with regard to saccade magnitudes or the components of saccade magnitude relative to a designated point.
Pro-Saccade refers to movement towards some point in space, often a target, area of interest or some attention-capturing event. By the above terminology a pro-saccade would have a relatively small saccadic angle and positive magnitude component relative to a designated position.
Anti-Saccade refers to movement away from some point in space, often due to aversion or based on a task (instruction to look away). By the above terminology an anti-saccade would have a relatively large saccadic angle (around ±180° or ±πrad) and a negative magnitude component relative to a designated position.
Inhibition of Return (IOR) is related to anti-saccades and describes a tendency during search or free viewing to avoid recently fixated regions which are less informative. IOR reflects a general strategy for efficient sampling of a scene. It may be furthered defined by, or a function of, anti-saccades.
Saccade Velocity (v saccade ) or the velocity of a saccade is taken as the change in magnitude over time (and not generally from magnitude components towards a reference point). Based on the degree of magnitude and direction of the saccade velocity, it may be indicative of a degree of relevancy of the target to the user. The average saccade velocity is denoted with a bar v saccade and may be applied to all saccades or a subset in time and/or space.
Saccade Rate (Frequency) (f saccade ) denotes the average number of saccades per second (s −1 or Hz) measured for all saccades, saccades over a period of time (e.g. f_(saccade|target present)), saccades within a particular region (e.g. f_(saccade|display center)) and/or saccades defined by their direction (e.g. f_(saccade|towards target)).
Saccade Count (Number) (N saccade ) is the number of saccades measured for all saccades, saccades over a period of time (e.g. N_(saccade|target present)), saccades within a particular region (e.g. N_(saccade|display center)) and/or saccades defined by their direction (e.g. N_(saccade|towards target)).
Pursuit Eye Movements (PEM) is used to refer to both smooth pursuit eye movements where gaze tracks a moving object through space and vestibulo-ocular movements that compensate for head or body movement. It may be further defined by data indicative of an initiation, a duration, and/or a direction of smooth PEM. Also included are compensatory tracking of stationary objects from a moving frame of reference. PEM generally do not consist of fixations and saccades but rather continuous, relatively slow motion interrupted by occasional error-correcting saccades. The smooth and saccadic portions of a PEM trace may be subtracted and analyzed separately.
Body Tracking Definitions
Body tracking entails measuring and estimating the position of the body and limbs as a function of time and/or discrete events in time associated with a class of movement (e.g. a nod of the head). Information sources include video tracking with and without worn markers to aid in image processing and analysis, position trackers, accelerometers and various hand-held or worn devices, platforms, chairs, or beds.
Screen Distance (d screen ) refers to the distance between the user's eyes (face) and a given display device. As a static quantity, it is important for determining the direction towards various elements on the screen (visual angle), but as a variable with time, screen distance can measure user movements towards and away from the screen. Screen distance is dependent upon a rate of change, an initial position, and a final position between the user's eyes (face) and a given display device. Combined with face detection algorithms, this measure may be made from device cameras and separate cameras with known position relative to displays.
Head Direction (Facing) ([θ, φ] facing ) refers to the direction in 3D polar coordinates of head facing direction relative to either the body or to a display or other object in the environment. Tracked over time this can be used to derive events like nodding (both with engagement and fatigue), shaking, bobbing, or any other form of orientation. Head direction is dependent upon a rate of change, an initial position, and a final position of head facing direction relative to either the body or to a display or other object in the environment.
Head Fixation, while similar to fixations and the various measures associated with eye movements, may be measured and behavior-inferred. Generally head fixations will be much longer than eye fixations. Head movements do not necessarily indicate a change in eye gaze direction when combined with vestibulo-ocular compensation. Head fixation is dependent upon a rate of change, an initial position, and a final position of head fixations.
Head Saccade, while similar to saccades and their various measures associated with eye movements, may be measured as rapid, discrete head movements. These will likely accompany saccadic eye movements when shifting gaze across large visual angles. Orienting head saccades may also be part of auditory processing and occur in response to novel or unexpected sounds in the environment.
Head Pursuit, while similar to pursuit eye movements, tend to be slower and sustained motion often in tracking a moving object and/or compensating for a moving frame of reference.
Limb Tracking refers to the various measures that may be made of limb position over time using video with image processing or worn/held devices that are themselves tracked by video, accelerometers or triangulation. This includes pointing devices like a computer mouse and hand-held motion controllers. Relative limb position may be used to derive secondary measures like pointing direction. Limb tracking is dependent upon a rate of change, an initial position, and a final position of the limbs.
Weight Distribution refers to the distribution of weight over a spatial arrangement of sensors while users stand, sit or lie down can be used to measure body movement, position and posture. Weight distribution is dependent upon a rate of change, an initial position, and a final position of weight.
Facial expressions including micro-expressions, positions of eyebrows, the edges, corners, and boundaries of a person's mouth, and the positions of a user's cheekbones, may also be recorded.
Electrophysiological and Autonomic Definitions
Electrophysiological measures are based on recording of electric potentials (voltage) or electric potential differences typically by conductive electrodes placed on the skin. Depending on the part of the body where electrodes are placed various physiological and/or behavioral measures may be made based on a set of metrics and analyses. Typically voltages (very small—microvolts μV) are recorded as a function of time with a sample rate in the thousands of times per second (kHz). While electrophysiological recording can measure autonomic function, other methods can also be used involving various sensors. Pressure transducers, optical sensors (e.g. pulse oxygenation), accelerometers, etc. can provide continuous or event-related data.
Frequency Domain (Fourier) Analysis allows for the conversion of voltage potentials as a function of time (time domain) into waveform energy as a function of frequency. This can be done over a moving window of time to create a spectrogram. The total energy of a particular frequency or range of frequencies as a function of time can be used to measure responses and changes in states.
Electroencephalography (EEG) refers to electrophysiological recording of brain function. Time averaged and frequency domain analyses (detailed below) provide measures of states. Combined with precise timing information about stimuli, event-related potentials (EEG-ERP) can be analyzed as waveforms characteristic of a particular aspect of information processing.
Frequency Bands are typically associated with brain activity (EEG) and in the context of frequency domain analysis different ranges of frequencies are commonly used to look for activity characteristic of specific neural processes or common states. Frequency ranges are specified in cycles per second (s −1 or Hz):
Delta—Frequencies less than 4 Hz. Typically associated with slow-wave sleep. Theta—Frequencies between 4 and 7 Hz. Typically associated with drowsiness. Alpha—Frequencies between 8 and 15 Hz. Beta—Frequencies between 16 and 31 Hz. Gamma—Frequencies greater than 32 Hz.

Electrocardiography (ECG) refers to electrophysiological recording of heart function. The primary measure of interest in this context is heart rate.
Electromyography (EMG) refers to electrophysiological recording of muscle tension and movement. Measures of subtle muscle activation, not necessarily leading to overt motion, may be made. Electrodes on the face can be used to detect facial expressions and reactions.
Electrooculography (EOG) refers to electrophysiological recording across the eye. This can provide sensitive measures of eye and eyelid movement, however with limited use in deriving pupil position and gaze direction.
Electroretinography (ERG) refers to electrophysiological recording of retinal activity.
Galvanic Skin Response (GSR) (Electrodermal response) is a measure of skin conductivity. This is an indirect measure of the sympathetic nervous system as it relates to the release of sweat.
Body Temperature measures may be taken in a discrete or continuous manner. Relatively rapid shifts in body temperature may be measures of response to stimuli. Shifts may be measured by tracking a rate of change of temperature, an initial temperature, and a final temperature.
Respiration Rate refers to the rate of breathing and may be measured from a number of sources including optical/video, pneumography and auditory and will typically be measured in breaths per minute (min −1 Brief pauses in respiration (i.e. held breath) may be measured in terms of time of onset and duration.
Oxygen Saturation (S O 2 ) is a measure of blood oxygenation and may be used as an indication of autonomic function and physiological state.
Heart Rate is measured in beats per minute (min −1 nd may be measured from a number of sources and used as an indication of autonomic function and physiological state.
Blood Pressure is typically measured with two values: the maximum (systolic) and minimum (diastolic) pressure in millimeters of mercury (mm Hg). Blood pressure may be used as an indication of autonomic function and physiological state.
Efferent Audio Recording Definitions
Audio recording from nearby microphones can measure behavioral and even autonomic responses from users. Vocal responses can provide measures of response time, response meaning or content (i.e. what was said) as well as duration of response (e.g. “yeah” vs. “yeeeeeeeaaaah”). Other utterances like yawns, grunts or snoring might be measured. Other audible behaviors like tapping, rocking, scratching or generally fidgety behavior may be measured. In certain contexts, autonomic behaviors like respiration may be recorded.
Vocalizations, such as spoken words, phrases and longer constructions may be recorded and converted to text strings algorithmically to derive specific responses. Time of onset and duration of each component (response, word, syllable) may be measured. Other non-lingual responses (yelling, grunting, humming, etc.) may also be characterized. Vocalizations may reflect a range of vocal parameters including pitch, loudness, and semantics.
Inferred Efferent Responses refer to certain efferent responses of interest that may be recorded by audio and indicate either discrete responses to stimuli or signal general states or moods. Behaviors of interest include tapping, scratching, repeated mechanical interaction (e.g. pen clicking) bouncing or shaking of limbs, rocking and other repetitive or otherwise notable behaviors.
Respiration, such as measures of respiration rate, intensity (volume) and potentially modality (mouth vs. nose) may also be made.
Afferent Classification/Definitions
The states discussed below are generally measured in the context of or response to various stimuli and combinations of stimuli and environmental states. A stimulus can be defined by the afferent input modality (visual, auditory, haptic, etc.) and described by its features. Features may be set by applications (e.g. setting the position, size, transparency of a sprite displayed on the screen) or inferred by image/audio processing analysis (e.g. Fourier transforms, saliency mapping, object classification, etc.).
Regions of interest as discussed below may be known ahead of time and set by an application, may be defined by the position and extent of various visual stimuli and/or may be later derived after data collection by image processing analysis identifying contiguous, relevant and/or salient areas. In addition to stimulus features, efferent measures may be used to identify regions of interest (e.g. an area where a user tends to fixate is defined by gaze position data). Likewise both afferent and efferent measures may be used to segment time into periods for summary analysis (e.g. total number of fixations while breath is held).
Sensory Data Exchange Platform Overview
Reference is made to FIG. 1A , which shows a block diagram 100 illustrating user interaction with an exemplary SDEP, in accordance with an embodiment of the present specification. In an embodiment, a user 102 interfaces with a VR/AR/MxR system 104 . VR/AR/MxR system 104 may include devices such as HMDs, sensors, and/or any other forms of hardware elements 106 that present VR/AR/MxR media to the user in the form of a stimulus, and enables collection of user response data during user interaction with the presented media. The media may be communicated by a server, through a network, or any other type of content platform that is capable of providing content to HMDs. Sensors may be physiological sensors, biometric sensors, or other basic and advanced sensors to monitor user 102 . Additionally, sensors may include environmental sensors that record audio, visual, haptic, or any other types of environmental conditions that may directly or indirectly impact the vision performance of user 102 . VR/AR/MxR system 104 may also include software elements 108 that may be executed in association with hardware elements 106 . Exemplary software elements 108 include gaming programs, software applications (apps), or any other types of software elements that may contribute to presentation of a VR/AR/MxR media to user 102 . Software elements 108 may also enable the system to collect user response data. Collected data may be tagged with information about the user, the software application, the game (if any), the media presented to the user, the session during which the user interacted with the system, or any other data. A combination of hardware elements 106 and software elements 108 may be used to present VR/AR/MxR media to user 102 .
In an embodiment, stimulus and response data collected from user's 102 interaction with VR/AR/MxR system 104 may constitute data sources 110 . Data sources 110 may be created within an SDEP 118 based on an interaction between software elements 108 and SDEP 118 . Software elements 108 may also interact with SDEP 118 through proprietary function calls included in a Software Development Kit (SDK) for developers (i.e. the developers may send/receive data to/from SDEP 118 using predefined functions). SDEP 118 may include storage and processing components and could be a computing system. The functionality of SDEP 118 may largely reside on one or more servers and the data stored and retrieved from cloud services. Sources of data may be in the form of visual data, audio data, data collected by sensors deployed with VR/AR/MxR system 104 , user profile data, or any other data that may be related to user 102 . Visual data may largely include stimulus data and may be sourced from cameras (such as cell phone cameras or other vision equipment/devices), or from other indirect sources such as games and applications (apps). Sensors may provide spatial and time series data. User data may pertain to login information, or other user-specific information derived from their profiles, from social media apps, or other personalized sources. In embodiments, data sources are broadly classified as afferent data sources and efferent data sources, which are described in more detail in subsequent sections of the specification. In an embodiment, user profile data may be collected from another database, or may be provided through a different source. In an exemplary embodiment user profile data may be provided by service providers including one or more vision care insurance provider. In other embodiments, the user profile data may be collected from other sources including user's device, opt-in options in apps/games, or any other source.
Data sources 110 may be provided to a data ingestion system 112 . Data ingestion system 112 may extract and/or transform data in preparation to process it further in a data processing system 114 . Data adapters, which are a set of objects used to communicate between a data source and a dataset, may constitute data ingestion system 112 . For example, an image data adapter module may extract metadata from images, and may also process image data. In another example, a video data adapter module may also extract metadata from video data sources, and may also include a video transcoder to store large volumes of video into distributed file system. In another example, a time series data adapter module parses sensor data to time series. In another embodiment, a spatial data adapter module may utilize data from relatively small areas such as skin, and spatially transform the data for area measurements. In another example, a user profile data adapter module may sort general user data, such as through a login, a social media connect API, unique identifiers on phone, and the like.
SDEP 118 may further comprise a data processing system 114 that receives conditioned data from data ingestion system 112 . A machine learning module 152 within data processing system 114 may communicate with a storage and a real time queue to output data to a data serving system 116 , which may include an Application Program Interface (API). In embodiments, the machine learning system may implement one or more known and custom models to process data output from data ingestion system 112 .
FIG. 1B illustrates an exemplary process of breakdown of functions performed by data ingestion system 112 and data processing system 114 . In embodiments, at 172 , an application residing at system 104 collects stimulus and response data. The stimulus and response data is forwarded in the form of information related to display, color, light, image, position, time, user, session, and other data related to the user interaction. Data may be represented in the application in the raw form used for presenting images on a display, playing sounds through speakers and taking in user input information relevant to the running of the application. Additional telemetry information and video and sound recording from more advanced systems (i.e. VR/AR/MxR) may also be included.
At 174 , a software toolkit may take in the raw programmatic information from the application and apply various conversions to represent data in a more physically and/or physiologically relevant form. Images and video, combined with information about the display hardware, may be converted from red, green and blue (RGB) values into CIE 1931 chromoluminance values (and/or some other physiologically relevant chromoluminance space). Spatial display information (horizontal and vertical pixel coordinates), combined with estimates of physical display size and user viewing distance, may be converted into head-centric visual angle and distance. The data may be combined further with estimated gaze direction from eye tracking and this may be further converted into retinal coordinates. Likewise user interface markers (e.g. mouse cursor) may have their position converted. In embodiments, some other relevant data may pass through without conversion. In some applications, information about the current and previous interactions may be utilized by the toolkit to provide the application with suggestions for efficient sampling towards estimating psychometric parameters (shown as Bracketing information).
At 176 , image processing and analysis, relying on machine learning or deep learning applications, may break down image or audio information into relevant features (for example, edges, contours, textures, and others) and objects for which parameters like identity, spatial location and extent, motion, and the like, may be estimated.
At 178 , raw, physical parameters of stimuli and responses may be combined and analyzed into psychometric estimates of detection, discrimination, reaction, accuracy, memory, and other derivative measures. In embodiments derivative measure may include measures for trend analysis.
User and session data may be forwarded throughout the process illustrated in FIG. 1B to tag stimulus, response, and analysis data, in order to provide context for later presentation and/or analysis.
In embodiments, data output from the analysis at 174 may include graphs for targeting, reaction, detection, discrimination, and other parameters that are useful to process and present vision data.
FIG. 1C illustrates an exemplary machine learning system 152 , in accordance with an embodiment of the present specification. As described above, input data in the forms of visual data, audio data, sensor data, and user data, interfaces with SDEP 118 and is pre-processed through data integration system 112 . Processed/transformed data is provided to machine learning system 152 . In embodiments, machine learning (ML) system processes transformed data using one or more known and customized data models, such as but not limited to naïve Bayes, decision trees, and others. In embodiments, ML system 152 creates a data pipeline based on software framework such as Keystone ML and Velox. Modelled data may be stored in a database 154 . In an embodiment, a combination of NoSQL (Accumulo/HBase), SQL (MySQL), and object storage (for raw image and video data) is used. In embodiments, cell-level security is provided to storage 154 in compliance with HIPAA.
In an embodiment, a real time queue 156 communicates with ML system 152 to stream processing pipelines. In an embodiment, real time queue 156 functions using a distributed, publish-subscribe messaging system such as Kafka. In an embodiment, a Kafka agent collects the images, videos, and time series data, from sources at a desired frequency and these are then processed using various OpenCV and custom image processing libraries at runtime.
SDEP 118 may be used via a hardware operating system of a user device (for example, HMD), and/or by content developers. In one example, both hardware and content developers may use the SDEP. In this example, data may be collected about how the user is interfacing with the content presented, what aspects of the content they are most engaged with and how engaged they are. Furthermore, engagement may be increased based on what is known of that user and/or similar users within the same demographic. The content may present in a way to conform to the hardware capabilities in a manner to optimize the experience from an ergonomic standpoint.
In embodiments, SDEP 118 may further include a module 120 for backend analytics that feeds another API 122 . API 122 may, in turn, interface with user 102 , providing modified media to user 102 .
FIG. 2 is a block diagram illustrating processing of a sensor data stream before it reaches a query processor, in accordance with an embodiment of the present specification. In an embodiment, FIG. 2 illustrates a lambda architecture 200 for a sensor data stream received by a SDEP. Data processing architecture 200 may be designed to handle large quantities of data by parallel processing of data stream and batch. In an embodiment, a sensor data stream 202 comprising sensor data collected from users in real time is provided to a real time layer 204 . Real time layer 204 may receive and process online data through a real time processor 214 . Data collected in batches may be provided to a batch layer 206 . Batch layer 206 comprises a master data set 222 to receive and utilize for processing time stamped events that are appended to existing events. Batch layer 206 may precompute results using a distributed processing system involving a batch processor 216 that can handle very large quantities of data. Batch layer 206 may be aimed at providing accurate data by being able to process all available sensor data, to generate batch views 218 . A bulk uploader 220 may upload output to be stored in a database 210 , with updates completely replacing existing precomputed batch views. Processed data from both layers may be uploaded to respective databases 208 and 210 for real time serving and batch serving. Data from databases 208 and 210 may subsequently be accessed through a query processor 212 , which may be a part of a serving layer. Query processor 212 may respond to ad-hoc queries by returning precomputed views or building views from the processed data. In embodiments, real-time layer 204 , batch layer 206 , and serving layer may be utilized independently.
Data Acquisition
Events may be coded within the stream of data, coming potentially from the app, the user and environmental sensors, and may bear timestamps indicating when things happen. Anything with an unambiguous time of occurrence may qualify as an “event”. Most events of interest may be discrete in time, with time stamps indicating either the start or the end of some state. As an exception, electrophysiological data may be recorded continuously and generally analyzed by averaging segments of data synchronized in time with other events or by some other analysis.
In an embodiment, data collected from interactions with user 102 is broadly classified as afferent data and efferent data, corresponding to afferent events and efferent events. In the peripheral nervous system, an afferent nerve fiber is the nerve fiber (axon) of an afferent neuron (sensory neuron). It is a long process (projection) extending far from the nerve cell body that carries nerve impulses from sensory receptors or sense organs toward the central nervous system. The opposite direction of neural activity is efferent conduction. Conversely, an efferent nerve fiber is the nerve fiber (axon) of an efferent neuron (motor neuron). It is a long process (projection) extending far from the nerve cell body that carries nerve impulses away from the central nervous system toward the peripheral effector organs (mainly muscles and glands).
A “stimulus” may be classified as one or more events, typically afferent, forming a discrete occurrence in the physical world to which a user may respond. A stimulus event may or may not elicit a response from the user and in fact may not even be consciously perceived or sensed at all; thus, if an event occurred, it is made available for analysis. Stimulus event classes may include “Application Specific Events” and “General and/or Derived Stimulus Events”.
Application Specific Events may include the many stimulus event classes that may be specific to the sights, sounds, and other sensory effects of a particular application. All of the art assets are potential visual stimuli, and all of the sound assets are potential auditory stimuli. There may be other forms of input including, but not limited to gustatory, olfactory, tactile, along with physiologic inputs—heart rate, pulse ox, basal body temperature, along with positional data—accelerometer, visual-motor—limb movement, gyroscope—head movements/body movement—direction, force, and timing. The sudden or gradual appearance or disappearance, motion onset or offset, playing or pausing or other change in state of these elements will determine their specific timestamp. Defining these stimulus event classes may require an app developer to collaborate with the SDE, and may include specific development of image/audio processing and analysis code.
General and/or Derived Stimulus Events are those stimulus events that may be generic across all applications. These may include those afferent events derived from video (e.g. head mounted camera) or audio data recorded of the scene and not coming directly from the app (which itself will provide a more accurate record of those events). Device specific, but not app specific, events may also be classified. Likewise calibration and other activities performed for all apps may be considered general (though perhaps still able to be categorized by the app about to be used).
Some stimulus events may not be apparent until after a large volume of data is collected and analyzed. Trends may be detected and investigated where new stimulus event classes are created to explain patterns of responding among users. Additionally, descriptive and predictive analysis may be performed in order to facilitate real-time exchange of stimuli/content depending on the trends/patterns so as to personalize user-experience.
A “response” may be classified as one or more events, typically efferent, forming a discrete action or pattern of actions by the user, potentially in response to a perceived stimulus (real or imagined). Responses may further include any changes in physiological state as measured by electrophysiological and/or autonomic monitoring sensors. Responses may not necessarily be conscious or voluntary, though they will be identified as conscious/unconscious and voluntary/involuntary whenever possible. Response events classes may include discrete responses, time-locked mean responses, time derivative responses, and/or derived response events.
“Discrete Responses” represent the most common response events associated with volitional user behavior and are discrete in time with a clear beginning and end (usually lasting on the order of seconds or milliseconds). These include, among others, mouse or touch screen inputs, vocalizations, saccadic and pursuit eye movements, eye blinks (voluntary or not), head or other body part movement and electrophysiologically detected muscle movements.
Due to the noisy nature of some data recording, notably electrophysiological recording, it is difficult to examine responses to individual stimulus events. A Time-Locked Mean Response refers to the pattern of responding to a particular stimulus event, which may be extracted from numerous stimulus response events by averaging. Data for a length of time (usually on the order of seconds) immediately following each presentation of a particular stimulus is put aside and then averaged over many “trials” so that the noise in the data (presumably random in nature) cancels itself out leaving a mean response whose characteristics may be measured.
Time Derivative Responses reflect that some responses, particularly autonomic responses, change slowly over time; Sometimes too slowly to associate with discrete stimulus events. However the average value, velocity of change or acceleration of velocity (and other derived measures) within certain periods of time may be correlated with other measured states (afferent or efferent).
As with stimulus events, some response events may not be apparent before data collection but instead reveal themselves over time. Whether through human or machine guided analysis, some characteristic responses may emerge in the data, hence may be termed Inferred Response Events.
Whenever possible, responses will be paired with the stimuli which (may have) elicited them. Some applications may make explicit in the data stream how stimuli and responses are paired (as would be the case in psychophysical experimentation). For the general case, stimulus event classes will be given a set period of time, immediately following presentation, during which a response is reasonably likely to be made. Any responses that occur in this time frame may be paired with the stimulus. If no responses occur then it will be assumed the user did not respond to that stimulus event. Likewise response events will be given a set period of time, immediately preceding the action, during which a stimulus is likely to have caused it. Windows of time both after stimuli and before responses may be examined in order to aid in the discovery of new stimulus and response event classes not previously envisioned.
Stimulus and Response Event Classes may be defined and differentiated by their features (parameters, values, categories, etc.). Some features of an event class may be used to establish groups or categories within the data. Some features may (also) be used to calculate various metrics. Features may be numeric in nature, holding a specific value unique to the event class or the individual instance of an event. Features may be categorical, holding a named identity either for grouping or potentially being converted later into a numerical representation, depending on the analysis.
The features of stimulus events may primarily constitute a physical description of the stimulus. Some of these features may define the event class of the stimulus, and others may describe a specific occurrence of a stimulus (e.g. the timestamp). The named identity of a stimulus (e.g. sprite file name) and state information (e.g. orientation or pose) are stimulus features. The pixel composition of an image or waveform of a sound can be used to generate myriad different descriptive features of a stimulus. Some stimulus features may require discovery through data analysis, just as some stimulus event classes themselves may emerge from analysis.
Response features may generally include the type or category of response made, positional information (e.g. where the mouse click occurred or where a saccade originated/landed, a touch, a gaze, a fixation, turn of head, turn of body, direction and velocity of head, or body/limb movement) and timing information. Some derived features may come from examining the stimulus to which a response is made; for example: whether the response was “correct” or “incorrect”.
FIG. 3 illustrates an overview 300 of sources of digital data. In embodiments, afferent data 304 may be collected from sources that provide visual information 307 , auditory information 308 , spatial information 310 , or other environmentally measured states including and not limited to temperature, pressure, and humidity. Sources of afferent data 304 may include events that are meant to be perceived by a user 302 . User 302 may be a user interfacing with a VR/AR/MxR system in accordance with various embodiments of the present specification.
Afferent and efferent data may be collected for a plurality of people and related to demographic data that correspond to the profiles for each of the plurality of people, wherein the demographic data includes at least the sex and the age of each of the plurality of people. Once such a database is created, visual content, electronic advertisements, and other personalized services can be created that are targeted to a group of people having at least one particular demographic attribute by causing the media content of that service to have a greater impact on the retino-geniculo-cortical pathway of the targeted group.
Afferent Data
Afferent (stimulus) events may be anything happening on a display provided to user 302 in the VE, events coming from speakers or head/earphones, or haptic inputs generated by an app. Data may also be collected by environment sensors including and not limited to head-mounted cameras and microphones, intended to keep a record of things that may have been seen, heard, or felt by user 302 but not generated by the app itself. Afferent data 304 may be a form of stimulus, which may be broken down into raw components (features or feature sets) that are used to build analytic metrics.
In embodiments, an afferent (stimulus) event is paired with an efferent (response) event. In the pairing, each of the component stimulus features may be paired with each of the component response features for analysis. In some cases pairs of stimulus features or pairs of response features may also be examined for correlations or dependencies. Stimulus/response feature pairs are at the root of most of the conceivable metrics to be generated. All analyses may be broken down by these feature pairs before being grouped and filtered according to various other of the event features available. In embodiments, for all data sources including afferent 304 and efferent 306 data sources, timing information is required to correlate inputs to, and outputs from, user's 302 sensory system. The correlations may be utilized to identify characteristic metrics or psychophysical metrics for the user. For example, if VR/AR/MxR system 104 records that an object was drawn on a screen at time tS (stimulus), and also that a user pressed a particular key at a time tR (response), the time it took the user to respond to the stimulus may be derived by subtracting tR-tS. In alternate embodiments, the user may press a key, or make a gesture, or interact with the AR/VR/MxR environment through a touch or a gesture. This example correlates afferent data 304 and efferent data 306 .
An example that correlates two types of afferent data 304 may be if a gaze tracker indicates that the gaze position of a user changed smoothly over a given period of time indicating that the user was tracking a moving object. However, if a head tracker also indicates smooth motion in the opposite direction, at the same time, it might also indicate that the user was tracking a stationary object while moving their head.
Another example that correlates two types of afferent data 304 may be if visual object appears at time t 1 , and a sound file is played at time t 2 . If the difference between t 1 and t 2 is small (or none), they may be perceived as coming from the same source. If the difference is large, they may be attributed to different sources.
The data taken from accumulated response events may be used to describe patterns of behavior. Patterns of responding, independent of what stimuli may have elicited them, can be used to categorize various behavioral or physiological states of the user. Grouping responses by the stimuli that elicited them can provide measures of perceptual function. In some cases analyses of stimulus events may provide useful information about the apps themselves, or in what experiences users choose to engage. The analysis may include following parameters: unique events, descriptive statistics, and/or psychometric functions.
Unique Events represent instances where raw data may be of interest. Some uncommon stimulus or response events may not provide opportunities for averaging, but instead are of interest because of their rarity. Some events may trigger the end of a session or time period of interest (e.g. the user fails a task and must start over) or signal the beginning of some phase of interaction.
Descriptive Statistics provide summarized metrics. Thus, if multiple occurrences of an event or stimulus/response event or feature pairing may be grouped by some commonality, measures of central tendency (e.g. mean) and variability (e.g. standard deviation) may be estimated. These summarized metrics may enable a more nuanced and succinct description of behavior over raw data. Some minimal level of data accumulation may be required to be reasonably accurate.
Psychometric Functions may form the basis of measures of perceptual sensitivity and ability. Whenever a particular class of stimulus event is shown repeatedly with at least one feature varying among presentations there is an opportunity to map users' pattern of responses against that stimulus feature (assuming responding varies as well). For example, if the size (stimulus feature) of a particular object in a game varies, and sometimes the user finds it and sometimes they don't (response feature), then the probability of the user finding that object may be plotted as a function of its size. This may be done for multiple stimulus/response feature pairs for a single stimulus/response event pairing or for many different stimulus/response event pairs that happen to have the same feature pairing (e.g. size/detection). When a response feature (detection, discrimination, preference, etc.) plotted against a stimulus feature (size, contrast, duration, velocity, etc.) is available with mean responses for multiple stimulus levels, a function to that data (e.g. detection vs. size) may be fitted. The variables that describe that function can themselves be descriptive of behavior. Thresholds may be defined where on one side is failure and the other side success, or on one side choice A and the other side choice B, among others.
Visual Data
Referring back to FIG. 3 , in an embodiment, for an application, visual information data 307 from physical display(s) and the visual environment is in the form of still image files and/or video files captured by one or more cameras. In an embodiment, data is in the form of instructions for drawing a particular stimulus or scene (far less data volume required, some additional time in rendering required).
FIG. 4A is a block diagram 400 illustrating characteristic metrics for visual data, in accordance with an embodiment of the present specification. Characteristic metrics may characterize a user session and may be time-averaged. Referring to FIG. 4A , scope 402 may refer to whether the visual data is for an entire scene (the whole visual display or the whole image from a user-head-mounted camera). Physical attributes 404 may refer to objective measures of the scene or objects within it. They may include location relative to the retina, head and body, an orthogonal 3-D chromoluminance; and contrast vs. spatial frequency vs. orientation. Categorical attributes 406 may be named properties of the image, which may include named identity of an object, and/or the group identity.
Visual stimuli may generally be taken in as digital, true color images (24-bit) either generated by an application (image data provided by app directly) or taken from recorded video (e.g. from a head mounted camera). Images and video may be compressed in a lossy fashion; where weighted averaging of data may account for lossy compression, but otherwise image processing would proceed the same regardless. A developer may choose to provide information about the presentation of a stimulus which may allow for the skipping of some image processing steps and/or allow for post hoc rendering of scenes for analysis. Visual stimuli may include, but are not limited to the following components: objects, size, chromatic distance, luminance contrast, chromatic contrast, spatial feature extraction, saliency maps and/or temporal dynamics.
Objects (stimuli) may be identified in an image (or video frame) either by information from the application itself or found via machine learning (Haar-like features classification cascade, or similar). Once identified, the pixels belonging to the object itself (or within a bounding area corresponding to a known size centered on the object) will be tagged as the “object”. The pixels in an annulus around the object (necessarily within the boundaries of the image/scene itself) with the same width/height of the object (i.e. an area 3× the object width and 3× the object height, excluding the central area containing the object) will be tagged as the “surround”. If another image exists of the same exact area of the surround, but without the object present (thus showing what is “behind” the object), that entire area without the object may be tagged as the “background”. Metrics may be calculated relative to the surround and also relative to the background when possible. Object segments or parts may be used to break objects down into other objects and may also be used for identity or category variables. Objects need not correspond to physical objects and may include regions or boundaries within a scene or comprise a single image feature (e.g. an edge).
Object size is an important feature for determining acuity, or from known acuity predicting whether a user will detect or correctly identify an object. The object size may be defined as a width and height, either based on the longest horizontal and vertical distance between pixel locations in the object or as the width and height of a rectangular bounding box defining the object's location. Smaller features that may be necessary to successfully detect or discriminate the object from others may be located within the object. It may be assumed that the smallest feature in an object is 10% of the smaller of its two dimensions (width and height). It may also be assumed the smallest feature size is proportional to the size of a pixel on the display for a given viewing distance. The smallest feature size may be more explicitly found either by analysis of a Fourier transform of the image or examining key features from a Harr-like feature classification cascade (or similar machine learning based object detection) trained on the object.
The first of two breakdowns by color, chromatic distance is a measure of the color difference between the object and its surround/background, independent of any luminance differences. Red, green and blue values may be independently averaged across all pixels of the object and all pixels of the surround/background. These mean RGB values will be converted into CIE Tristimulus values (X, Y and Z) and then into CIE chromaticity (x and y) using either standard conversion constants or constants specific to the display used (when available). In an embodiment, conversion constants for conversion from RGB to XYZ, taken from Open CV function ‘cvtColor’ based on standard primary chromaticities, a white point at D65, and a maximum, white luminance of 1, is:





[



X




Y




Z



]

←


[



0.412453


0.357580


0.180423




0.212671


0.715160


0.072169




0.019334


0.119193


0.950227



]

·

[



R




G




B



]






In this embodiment, RGB is converted to xy using the following:




x
=

X

X
+
Y
+
Z








y
=

Y

X
+
Y
+
Z






The absolute distance between the chromaticity of the object and that of the surround/background will be logged as the chromatic distance. Next, a line will be drawn from the midpoint between the two chromaticities and each of the three copunctal points for L, M and S cones. These lines are confusion lines for L, M and S cone deficiencies, along which someone missing one of those cone types would be unable to discriminate chromaticity. The component of the line between object and surround/background chromaticity parallel to each of these three confusion lines will be logged as the L, M and S specific chromatic distances.
FIG. 4B provides a graphical presentation of color pair confusion components, in accordance with an embodiment of the present specification. Referring to the figure, a line 1308 is drawn between the two chromaticities given. As seen in the figure, three large dots—red 410 , green 412 , and blue 414 are copunctal points for L, M and S cones, respectively. From each dot extends a similarly color-coded, dashed line. Bold line 416 has a mid-point where the three, dashed lines intersect. Based on the angle between line 416 and the lines drawn from the midpoint to each of the copunctal points, the parallel component of that line for each of the three resulting confusion lines is determined. In embodiments, the closer to the parallel line between the colors is to a particular confusion line, the more difficult it will be for someone with a deficiency of the corresponding cone to discriminate. The component length divided by the total length (the quotient will be in the interval [ 0 , 1 ]) would be roughly the probability of the colors being confused.
FIG. 4C shows a graph illustrating how luminance may be found for a given chromaticity that falls on the top surface of the display gamut projected into 3D chromoluminance space. The graph shows a projection of a full display gamut for a computer screen into CIE 1931 chromoluminance space. While the RGB space used to define the color of pixels on a display can be represented by a perfect cube, the actual physical property of luminance is somewhat complexly derived from those values, represented by the shape seen in FIG. 6 . Luminance contrast may be defined in three ways. Generally the context of an analysis will suggest which one of the three to use, but all three may be computed for any object and its surround/background. For instances where a small object is present on a large, uniform background (e.g. for text stimuli), Weber contrast may be computed using the CIE Tristimulus values Y (corresponding to luminance) calculated from the mean RGB of the object and of the surround/background. Here it is assumed that the average luminance is roughly equal to the surround luminance. Weber contrast can be positive or negative and is theoretically unbounded. For object/surrounds that are periodic in nature, and especially with gradients (e.g. a sine wave grating), Michelson contrast may be computed from the minimum and maximum luminance values in the stimulus. Michelson contrast will always be a value between 0 and 1. For most cases it will be necessary to compute contrast from all of the pixel values, instead of from a mean or from the minimum and maximum. The RMS contrast (root mean square, or standard deviation) can be found by taking the standard deviation of the CIE Tristimulus value Y for all pixels. The RMS contrast of the object is one measure. The RMS contrast of the object relative to the RMS contrast of the surround/background is another. Finally, the RMS contrast of the object and surround together is yet a third measure of RMS contrast that can be used.
Chromatic contrast may be calculated on any pair of chromaticity values, independently, in all of the ways described above for luminance contrast. The most useful of these will either be the a* and b* components of CIELAB color space, or the L vs. M and S vs. LM components of cone-opponent color space. For any pair of dimensions, the Weber, Michelson and/or RMS contrast may be calculated, depending on the type of stimulus being analyzed. In addition, RMS contrast will be calculated for L, M and S cone deficiencies. CIE chromaticity values for all pixels will be converted into three sets of polar coordinates centered on the L, M and S copunctal points. In an embodiment, the following equation is used to convert Cartesian coordinates to polar coordinates, with an option to provide center points other than [0,0]:




θ
=


tan

-
1




(


y
-

y
c



x
-

x
c



)








Radius
=




(

y
-

y
c


)

2

+


(

x
-

x
c


)

2







RMS contrast may be calculated based on the radius coordinates for each conversion.
In addition to finding objects, algorithms may also identify prominent features present in a scene, or within objects, that may capture attention, be useful for a task the user is performing or otherwise be of interest as independent variables to correlate with behavior. Edges, those inside identified objects and otherwise, may be targets for fixations or other responses and their positions may be responsible for observed positional errors in responding and be worth correlating with correct and incorrect responses. Regions, contours, surfaces, reflections, shadows and many other features may be extracted from this data.
Saliency Maps refer to data that are collected from user interactions to inform models of saliency for future analysis of stimulus scenes. Edges, contours and other image features may be used to measure saliency and predict where user responses, including eye gaze fixations, may fall. Multiple algorithms may be applied to highlight different types of features in a scene.
Temporal Dynamics are also important because features of a visual display or environment, and any objects and object features thereof, may change over time. It will be important to log the time of any change, notably: appearance/disappearance or change in brightness/contrast of objects or features, motion start/stop or abrupt position change (in x, y, z planes), velocity change (or acceleration or any higher order time derivative of position) and any and all changes in state or identity of objects or features. Changes in chromaticity or luminance of objects or features should also be logged. Secondary changes in appearance resulting from changes in orientation or pose of an object or the object's position relative to the surround/background may also be logged.
Auditory Data
Referring back to FIG. 3 , auditory information 308 may be received from audio output such as speakers, and the environment by using microphones. In an embodiment auditory information 308 may be available in raw, waveform files or in more descriptive terms (e.g. this audio file played at this time).
FIG. 5 illustrates characteristic metrics 500 for auditory information 308 (shown in FIG. 3 ), in accordance with an embodiment of the present specification. Referring to FIG. 5 , a positional reference 502 may be noted to identify the location of sounds. The position, relative to a user's head, of an object or speaker in the environment will vary as they move their head. The position of a virtual source perceived through headphones may not change as the user turns their head (unless head tracking and sound processing work together to mimic those changes).
The physical attributes 504 of sound may include their location (derived from intensity, timing and frequency differences between the ears), frequency composition (derived from the waveform), and the composition of different sources. Categorical attributes 506 may be named properties of the image, which may include named identity of an object, and/or the group identity and may follow a similar description as for visual stimuli.
Auditory (Sound) stimuli may generally be taken in as digital waveforms (with varying spatial and temporal resolution or bitrate and possible compression) either generated by an application or taken from recorded audio (e.g. head mounted microphones, preferably binaural). Compression parameters, if any, may be recorded. Developers may choose to provide information about the presentation of a stimulus which may allow for the skipping of some processing. Visual information may be used to model the audio environment so that sound reflections or obscurations can be taken into account. Audio stimuli may be broken down to include the following parameters: Fourier Decomposition, Head-Centric Position, Sound Environment, and/or Objects.
Fourier Decomposition may be performed to break sound waves into components based on sound objects. Time-domain waveform data may be transformed into the frequency domain such that the amplitude and phase of different audio frequencies over time may be analyzed. This will allow the utilization of sound parameters (e.g. frequency, amplitude, wavelength, shape and envelope, timbre, phase, etc.) as independent variables.
Head-Centric Position or head tracking data may be necessary for environmental sounds. The position of sound sources relative to a user's ears may be derived, and whenever possible the sound waveforms as they exist at the user's ears may be recorded (ideally from binaural, head-mounted microphones). Binaural headset sound sources (e.g. headphones/earphones) may obviate the necessity for this.
Similarly, tracking data for body and/or limbs may be necessary for environmental sounds. The position of sound sources relative to a user's body and limbs may be derived. This data may be related to head tracking data identified for environmental sounds. The data may enable understanding of how body and limbs react with the movement of head.
Sound Environment is not critical in most common use cases (e.g. sound is coming from headset or from directly in front of the user), but will be important for considering environmental sounds to which users are anticipated to respond. Objects in the environment that reflect and/or block sound (commonly frequency specific) may change the apparent source location and other frequency dependent features of a sound. It may be useful to roughly characterize the physical environment as it affects the propagation of sound from its sources to the user.
Audio objects may be detected and segmented out using the same type of machine learning algorithms (Haar-like feature classification cascades or similar) that are used for detecting and segmenting out visual objects. This should be used whenever possible to obtain accurate audio event details and may also be useful for extracting audio parameters used by the auditory system for localization.
Most analysis may revolve around visual and (to a lesser extent) auditory stimuli occurring discretely in time. Other stimuli may include those sensed in other modalities (e.g. touch, taste, smell, etc.) or general environmental state variables that define the context of user interactions with applications (e.g. ambient lighting and background audio).
Examples of other stimuli may include the following:
Haptic Stimuli, where developers may choose to use haptic feedback mechanisms and, if they so choose, provide details about the nature and timing of those events. Haptic stimulation may also be derived via direct recording (unlikely) or derived from other sources (e.g. hearing the buzz of a physical vibration via microphone).
Other Modality Stimuli, where developers may be able to initiate smell, taste, temperature, pressure, pain or other sensation at discrete times creating stimulus events not already discussed. As with haptic stimuli, any record of such stimulation would best come directly from the application itself via function calls.
Environmental Stimuli, or stimuli that do not occur discretely in time and are either of constant state or steadily repeating, may provide important context for the discrete stimuli and responses that occur in a session. Ambient light levels may affect contrast sensitivity, baseline pupil size, circadian patterns and other physiological states of the user. Ambient sounds may affect auditory sensitivity, may mask certain auditory stimuli and also affect physiological and other states of the user. The time of day may also be an important variable for categorization and correlation. Though perhaps not readily recorded by an application, user input could provide information about sleep patterns, diet and other physiologically relevant state variables as well as categorical descriptions of the space including temperature, pressure, humidity (which may also be derived from location and other services).
Spatial Information
Referring back to FIG. 3 , in an embodiment, spatial information 310 may consist of descriptions of the setting around user 302 . This may include spatial orientation of user 302 and physical space around user 302 .
In an embodiment, setting is an environment in which interactions between user 302 and the app take place. Setting data may refer to things that are mostly static during a session including the physical setting, ambient light levels, room temperature, and other types of setting information. In embodiments, spatial information 310 is a part of the setting data. Setting data may generally be constant throughout a session with user 302 and therefore may not be broken down into “events” as described earlier. Setting data may pertain to a physical setting or may relate to personal details of user 302 .
Physical setting data may correspond to any description of the physical space, such as and not limited to a room or an outdoor setting, and may be useful to categorize or filter data. In an exemplary embodiment, physical setting data such as the ambient lighting present, may directly affect measures of pupil size, contrast sensitivity and others. Lighting may affect quality of video eye tracking, as well as any afferent events derived from video recording of a scene. Similarly, environmental sounds may affect users' sensitivity as well as the ability to characterize afferent events derived from audio recording.
Personal details of a user may pertain to any personal, largely demographic, data about the user or information about their present physiological or perceptual state (those that will remain largely unchanged throughout the session). This data may also be useful for categorization and filtering. Personal details may include any information regarding optics of the user's eyes (for example, those derived from knowledge of the user's eyeglass or contact prescription). Personal details may also include diet related information, such as recent meal history. Further, time, duration, and quality of most recent sleep period, any psychoactive substances recently taken in (e.g. caffeine) and recent exercise or other physical activity may all impact overall data.
Efferent Data
Eye Tracking
Video eye tracking and electrooculography provide information about eye movements, gaze direction, blinking and pupil size. Derived from these are measures of vergence, fatigue, arousal, aversion and information about visual search behavior. Information pertaining to eye movements include initiation, duration, and types of pro-saccadic movements (toward targets), anti-saccadic movements (toward un-intended target), the amount of anti-saccadic error (time and direction from intended to unintended target), smooth pursuit, gaze with fixation duration, pupil changes during movement and during fixation, frequency and velocity of blink rate, as well as frequency and velocity of eye movements. Information pertaining to vergence may include both convergence and divergence—in terms of initiation and duration. Combined with information about the visual scene, measures of accuracy, search time and efficiency (e.g. minimizing number of saccades in search) can be made.
Autonomic measures derived from video eye tracking data may be used to guide stimulus selection towards those that increase or decrease arousal and/or aversion. Summary information about gaze position may indicate interest or engagement and likewise be used to guide stimulus selection.
Referring to FIG. 3 , efferent data sources 306 may include video eye tracking data 312 . Data 312 may measure gaze direction, pupil size, blinks, and any other data pertaining to user's 302 eyes that may be measured using a Video Eye Tracker (VET) or an electrooculogram. This is also illustrated in FIG. 6 , which shows characteristic metrics 600 for eye tracking, in accordance with an exemplary embodiment of the present specification. Video eye tracking 602 generally involves recording images of a user's eye(s) and using image processing to identify the pupil and specific reflections of known light sources (typically infrared) from which may be derived measures of pupil size and gaze direction. The angular resolution (of eye gaze direction) and temporal resolution (frames per second) may limit the availability of some measures. Some measures may be recorded as discrete events, and others recorded over time for analysis of trends and statistics over epochs of time.
Gaze Direction
Software, typically provided with the eye tracking hardware, may provide calibrated estimates of gaze direction in coordinates tied to the display used for calibration. It may be possible/necessary to perform some of this conversion separately. For head mounted units with external view cameras the gaze position may be in head centric coordinates or in coordinates relative to specific objects (perhaps provided reference objects) in the environment. It is assumed that gaze direction will be provided at some rate in samples per second. Most of the following metrics will be derived from this stream of gaze direction data: saccade, pursuit, vergence, patterns, and/or microsaccades.
Saccade: Prolonged periods of relatively fixed gaze direction separated by rapid changes in gaze (over a matter of milliseconds) may be logged as “fixations” and the jumps in between as “saccades”. Fixations will be noted for position, start and end time and duration. In some cases they may also be rated for stability (variability of gaze direction during fixation). Saccades will be noted for their direction (angle), speed and distance. It is worth noting, and it will generally be assumed, that there is a period of cortical suppression during saccades when visual information is not (fully) processed. This saccadic suppression may be exploited by developers to alter displays without creating a percept of motion, appearance or disappearance among display elements.
Pursuit: Pursuit eye movements may be characterized by smooth changes in gaze direction, slower than typical saccades (and without cortical suppression of visual processing). These smooth eye movements generally occur when the eyes are pursuing/tracking an object moving relative to head facing direction, a stationary object while the head moves or moving objects while the head also moves. Body or reference frame motion can also generate pursuit eye movements to track objects. Pursuit can occur in the absence of a visual stimulus based on the anticipated position of an invisible or obscured target.
Vergence: This measure may require relatively fine resolution gaze direction data for both eyes simultaneously so that the difference in gaze direction between eyes can be used to determine a depth coordinate for gaze. Vergence is in relation to the distance of the object in terms of the user to measure objects between the near point of convergence and towards infinity in the distance—all of which may be modelled based off the measurements of vergence between convergence and divergence.
Patterns: Repeated patterns of eye movements, which may be derived from machine learning analysis of eye gaze direction data, may be used to characterize response events, states of user interaction or to measure effects of adaptation, training or learning. Notable are patterns during visual search for targets or free viewing of scenes towards the completion of a task (e.g. learning of scene details for later recognition in a memory task). Eye movement patterns may also be used to generate models for creating saliency maps of scenes, guiding image processing.
Microsaccades: With relatively sensitive direction and time resolution it may be possible to measure and characterize microsaccadic activity. Microsaccades are generally present during fixation, and are of particular interest during rigid or prolonged fixation. Feedback into a display system may allow for creating images that remain static on the retina resulting in Troxler fading. Microsaccades are not subject to conscious control or awareness.
Sample questions concerning eye tracking metrics that may be answered over a period of time may include: where are users looking the most (potentially in response to repeating events), how fast and accurate are saccadic eye movements, how rapidly are users finding targets, are users correctly identifying targets, how accurate is pursuit/tracking, are there preferences for certain areas/stimuli.
During free viewing or search, fixations (relatively stable eye gaze direction) between saccades typically last on the order of 200-300 milliseconds. Saccades have a rapidly accelerating velocity, up to as high as 500 degrees per second, ending with a rapid deceleration. Pursuit eye movements occur in order to steadily fixate on a moving object, either from object motion or head motion relative to the object or both. Vergence eye movements are used to bring the eyes together to focus on near objects. Vestibular eye movements are compensatory eye movements derived from head and/or body movement.
Reference is made to WO2015003097A1 entitled “A Non-Invasive Method for Assessing and Monitoring Brain”, which has at least partial common inventorship with the present specification. In an example, a pro-saccade eye tracking test is performed. The pro-saccade test measures the amount of time required for an individual to shift his or her gaze from a stationary object towards a flashed target. The pro-saccade eye tracking test may be conducted as described in The Antisaccade: A Review of Basic Research and Clinical Studies, by S. Everling and B. Fischer, Neuropsychologia Volume 36, Issue 9, 1 Sep. 1998, pages 885-899 (“Everling”), for example.
The pro-saccade test may be performed while presenting the individual with a standardized set of visual stimuli. In some embodiments, the pro-saccade test may be conducted multiple times with the same or different stimuli to obtain an average result. The results of the pro-saccade test may comprise, for example, the pro-saccade reaction time. The pro-saccade reaction time is the latency of initiation of a voluntary saccade, with normal values falling between roughly 200-250 ms. Pro-saccade reaction times may be further sub-grouped into: Express Pro-Saccades: 80-134 ms; Fast regular: 135-175 ms; Slow regular: 180-399 ms; and Late: (400-699 ms).
Similarly, an anti-saccade eye tracking test may be performed. The anti-saccade test measures the amount of time required for an individual to shift his or her gaze from a stationary object away from a flashed target, towards a desired focus point. The anti-saccade eye tracking test can be conducted as described in Everling, for example. In some examples, the anti-saccade test may also measure an error time and/or error distance; that is, the amount of time or distance in which the eye moves in the wrong direction (towards the flashed target). The anti-saccade test may be performed using the standardized set of visual stimuli. The results of the anti-saccade test may comprise, for example, mean reaction times as described above for the pro-saccade test, with typical mean reaction times falling into the range of roughly 190 to 270 ms. Other results may include initial direction of eye motion, final eye resting position, time to final resting position, initial fovea distance (i.e., how far the fovea moves in the direction of the flashed target), final fovea resting position, and final fovea distance (i.e., how far the fovea moves in the direction of the desired focus point).
Also, a smooth pursuit test may be performed. The smooth pursuit test evaluates an individual's ability to smoothly track moving visual stimuli. The smooth pursuit test can be conducted by asking the individual to visually follow a target as it moves across the screen. The smooth pursuit test may be performed using the standardized set of visual stimuli, and may be conducted multiple times with the same or different stimuli to obtain an average result. In some embodiments, the smooth pursuit test may include tests based on the use of fade-in, fade-out visual stimuli, in which the target fades in and fades out as the individual is tracking the target. Data gathered during the smooth pursuit test may comprise, for example, an initial response latency and a number of samples that capture the fovea position along the direction of motion during target tracking. Each sampled fovea position may be compared to the position of the center of the target at the same time to generate an error value for each sample.
For more sensitive tracking hardware, it may also be possible to measure nystagmus (constant tremor of the eyes), drifts (due to imperfect control) and microsaccades (corrections for drift). These will also contribute noise to gross measurements of gaze position; as a result fixations are often characterized by the mean position over a span of relatively stable gaze position measures. Alternatively, a threshold of gaze velocity (degrees/second) can be set, below which any small movements are considered to be within a fixation.
Saccades require time to plan and execute, and a delay, or latency, of at least 150 ms is typical after, for example, the onset of a visual stimulus eliciting the saccade. Much can be said about the latency before a saccade and various contexts that may lengthen or shorten them. The more accurate information we have regarding the relative timing of eye movements and events occurring in the visual scene, the more we can say about the effect of stimulus parameters on saccades.
Although usually correlated, shifts in attention and eye gaze do not necessarily have to happen together. In some contexts it may be efficient for the user to direct attention to a point in their visual periphery, for example to monitor one location while observing another. These scenarios may be useful for generating measures related to Field of View and Multi-Tracking.
It is possible to use image processing techniques to highlight regions within a scene of greater saliency based on models of the visual system. For example areas of greater high-spatial-frequency contrast (i.e. edges and lines) tend to capture attention and fixations. It is possible within a specific context to use eye gaze direction to develop custom saliency maps based on the information available in the visual scene combined with whatever tasks in which an observer may be engaged. This tool can be used to highlight areas of interest or greater engagement.
Pupil Size
Pupil size may be measured as part of the image processing necessary to derive gaze direction. Pupil size may generally change in response to light levels and also in response to certain stimulus events via autonomic process. Pupil responses are not subject to conscious control or awareness (except secondarily in the case of extreme illumination changes). Sample questions concerning eye tracking metrics that may be answered over a period of time may include: how are the pupils responding to different stimuli, how are the pupils behaving over time.
Pupil diameter generally falls between 2 and 8 mm at the extremes in light and dark, respectively. The pupil dilates and constricts in response to various internal and external stimuli. Due to differences in baseline pupil diameter, both among observers and due to ambient lighting and physiological state, pupil responses may generally be measured as proportions of change from baseline. For example, the baseline pupil diameter might be the diameter at the moment of an external stimulus event (image appears), and the response is measured by the extent to which the pupil dilates or constricts during the 1 second after the stimulus event. Eye color may affect the extent of constriction, and age may also be a factor.
In addition to responding to light, accommodation for distance and other spatial and motion cues, pupil diameter will often be modulated by cognitive load, certain imagery and reading. Pupil diameter may be modulated during or at the termination visual search. Proportional changes can range from a few to tens of percentage points.
Thresholds for determining computationally if a response has been made will vary depending on the context and on the sensitivity of the hardware used. Variations in ambient lighting and/or the mean luminance of displays will also have a large influence on pupil diameter and proportional changes, so thresholds will need to be adaptable and likely determined by the data itself (e.g. threshold for dilation event itself being a percentage of the range of pupil diameter values recorded within a session for one user).
Reference is again made to WO2015003097A1 titled “A Non-Invasive Method for Assessing and Monitoring Brain”, which has at least partial common inventorship with the present specification. In an example, pupillary response is assessed. Pupillary response is often assessed by shining a bright light into the individual's eye and assessing the response. In field settings, where lighting is difficult to control, pupillary response may be assessed using a standardized set of photographs, such as the International Affective Picture System (TAPS) standards. These photographs have been determined to elicit predictable arousal patterns, including pupil dilation. The pupillary response test may be performed using a variety of stimuli, such as changes to lighting conditions (including shining a light in the individual's eyes), or presentation of photographs, videos, or other types of visual data. In some embodiments, the pupillary test may be conducted multiple times with the same or different stimuli to obtain an average result. The pupillary response test may be conducted by taking an initial reading of the individual's pupil diameter, pupil height, and/or pupil width, then presenting the individual with visual stimuli to elicit a pupillary response. The change in pupil dilation (e.g., the change in diameter, height, width, and/or an area calculated based on some or all of these measurements) and the time required to dilate are measured. The results of the pupillary response test may include, for example, a set of dilation (mydriasis) results and a set of contraction (miosis) results, where each set may include amplitude, velocity (speed of dilation/constriction), pupil diameter, pupil height, pupil width, and delay to onset of response.
Blinks
Video eye trackers, as well as less specialized video imaging of a user's face/eye region, may detect rapid or prolonged periods of eye closure. Precautions may be taken as loss of acquisition may also be a cause for periods of data loss. Blink events, conscious or reflexive, and blink rates over time related to measures of fatigue or irritation may be recorded. Sample questions concerning eye tracking metrics are mentioned in FIG. 6 . In embodiments, these are questions that may be answered over a period of time and may include: are the users blinking in response to the onset of stimuli, is the blink rate changing in response to the stimuli, is the blink rate changing overall, does the blink rate suggest fatigue.
Normal blinking rates among adults are around 10 blinks per minute at rest, and generally decreases to around 3 blinks per minute during focused attention (e.g. reading). Other properties of blinks, for example distance/speed of eyelid movement and durations of various stages within a blink, have been correlated with error rates in non-visual tasks (for example, using auditory stimulus discrimination) and other measures; whenever possible it may be advantageous to use video recordings to analyze eyelid position in detail (i.e. automated eyelid tracking). Blink durations longer than 150 ms may be considered long-duration blinks.
As with most measures, proportional changes from baseline may be more valuable than absolute measures of blink frequency or average duration. Generally, significance can be assigned based on statistical measures, meaning any deviation is significant if it is larger than the general variability of the measure (for example as estimated using a t-test).
Manual Inputs
Referring back to FIG. 3 , another efferent data source 306 may be manual input 314 . Which have been a traditional tool of computer interaction and may be available in many forms. Exemplary manual inputs 314 of interest include input identity (key pressed), any other gesture, position coordinates (x, y, z) on a touch screen or by a mouse, and/or (video) tracking of hand or other limb. FIG. 7 illustrates characteristic metrics 700 for manual inputs 702 , in accordance with an embodiment of the present specification.
Sample questions concerning manual input metrics that may be answered over a period of time may include: where are the users clicking/touching the most (potentially in response to repeating events), how fast and accurate are the clicks/touches, how rapidly are users finding targets, are users correctly identifying targets, how accurate is tracking, are there preferences for certain areas/stimuli, what kind of grasping/touching motions are the users making, how is the hand/eye coordination, are there reflexive actions to virtual stimuli.
Responses made with the fingers, hands and/or arms, legs, or any other part of the body of users may generally yield timing, position, trajectory, pressure and categorical data. These responses may be discrete in time, however some sustained or state variable may be drawn from manual data as well. Following analytic response metrics may be derived from manual responses: category, identity, timing, position, and/or trajectory.
Category: In addition to categories like click, touch, drag, swipe and scroll there may be sub categories like double click, tap or push, multi-finger input, etc. Any variable that differentiates one action from another by category that is detectable by an application may be important for differentiating responses (and will likely be used for that purpose by developers).
Identity: Whenever multiple input modalities exist for the same type of response event, most notably the keys on a computer keyboard, or any other gesture that may be possible in a VR/AR/MxR environment, the identity of the input may be recorded. This also includes directions indicated on a direction pad, mouse buttons clicked and, when possible, the area of a touchpad touched (independent of cursor position), or any other gesture.
Timing: The initiation and ending time of all responses may be recorded (e.g. a button press will log both the button-down event and the button-up event), and from that response durations can be derived. This timing information will be key to connecting responses to the stimuli that elicited them and correlating events in time.
Position: For visual interfaces, the position may be in display coordinates. Positions may be singular for discrete events like clicks or continuously recorded at some reasonable rate for tracing, dragging, etc. When possible these may also be converted to retinal coordinates (with the combination of eye gaze tracking). By understanding position, a topography of the retina may be done, and areas of the retina may be mapped in relationship to their specific functions further in relationship to the brain, body, endocrine, and autonomic systems. For gestures recorded by video/motion capture the body-centric position will be recorded along with the location of any cursor or other object being controlled by the user.
Trajectory: For swipe, scroll and other dynamic gestures it may be possible to record the trajectory of the response (i.e. the direction and speed as a vector) in addition to any explicit position changes that occur. This will, in fact, likely be derived from an analysis of rapid changes in position data, unless the device also provides event types for these actions.
Head Tracking
Head tracking measures are largely associated with virtual, augmented, and mixed reality displays. They can provide measures of synchrony with displayed visual environments, but also of users' reactions to those environments. Orienting towards or away from stimuli, compensatory movements in line or not in line with the displayed visual environments and other motion behavior can be used to derive similar, though less precise, measures similar to those from eye tracking. Those derived measures associated with arousal, fatigue and engagement can be modified as previously stated.
If head movements, particularly saccadic head movements, prove to be a source of mismatch and discomfort for users it may be desirable to modify displays to reduce the number of such head movements. Keeping display elements within a region near head-center and/or encouraging slower changes in head-facing may reduce large head movements. With regards to individual differences: some users will move their heads more than others for the same scenario. It may be possible to train head movers to reduce their movements.
Referring back to FIG. 3 , head tracking data 316 may be another form of efferent data 306 source. Head tracking data 316 may track user's 302 head orientation and physical position from either video tracking (VET or otherwise) or position sensors located on HMDs, headsets, or other worn devices. In addition to tracking user's 302 head, their body may be tracked. The position of users' 302 bodies and parts thereof may be recorded, likely from video based motion capture or accelerometers in wearable devices. This position data would commonly be used to encode manual response data (coming from finger, hand or arm tracking) and/or head orientation relative to the environment to aid in eye gaze measurements and updating of the user's visual environment. Head position data may also be used to model the effect of head shadow on sounds coming from the environment. FIG. 8 illustrates characteristic metrics 800 for head tracking, which may include head orientation 802 and/or physical position 804 , in accordance with an embodiment of the present specification.
Sample questions concerning head tracking metrics that may be answered over a period of time may include: where are the users looking most (potentially in response to repeating events), how fast and accurate are head movements, how accurate is pursuit/tracking, is there preference for certain areas/stimuli, are users accurately coordinating head and eye movements to direct gaze and/or track objects, are head movements reduced due to the hardware, are users making many adjustments to the hardware, are users measurably fatigued by the hardware.
Head movements may be specifically important in the realms of virtual, augmented, and mixed reality, and may generally be correlated with eye movements, depending upon the task. There is large individual variability in propensity for head movements accompanying eye movements. During tasks like reading, head movement can account for 5% to 40% of shifting gaze (combined with eye movements). The degree to which a user normally moves their head may prove a key indicator of susceptibility to sickness from mismatch of visual and vestibular sensation.
It is likely that saccadic and pursuit head movements may be qualitatively different in those two modalities. For example, a mismatch may be less jarring if users follow an object from body front, 90 degrees to the right, to body side using a pursuit movement as opposed to freely directing gaze from forward to the right. If the velocity of a pursuit object is relatively steady then the mismatch would be imperceptible through most of the motion.
Referring back to FIG. 3 , a user's 302 vocal responses may also be tracked via microphone. Speech recognition algorithms would extract semantic meaning from recorded sound and mark the time of responses (potentially of individual words or syllables). In less sophisticated scenarios the intensity of vocal responses may be sufficient to mark the time of response. In embodiments, voice and speech data is correlated with several other forms of data such as and not limited to head tracking, eye-tracking, manual inputs, in order to determine levels of perception.
Electrophysiology/Autonomous Recording
Electrophysiological and autonomic measures fall largely outside the realm of conscious influence and, therefore, performance. These measures pertain largely to states of arousal and may therefore be used to guide stimulus selection. Recounted for convenience here, the measures of interest would come from electroencephalography (EEG—specifically the activity of various frequency bands associated with arousal states), galvanic skin response (GSR—also associated with arousal and reaction to emotional stimuli), heart rate, respiratory rate, blood oxygenation, and potentially measures of skeletal muscle responses.
Reference is again made to WO2015003097A1 titled “A Non-Invasive Method for Assessing and Monitoring Brain”, which has at least partial common inventorship with the present specification. In an example, brain wave activity is assessed by performing an active brain wave test. The active brain wave test may be conducted using EEG (electroencephalography) equipment and following methods known in the art. The active brain wave test may be performed while the individual is presented with a variety of visual stimuli. In some embodiments, the active brain wave test is conducted while presenting a standardized set of visual stimuli that is appropriate for assessing active brain wave activity. In some embodiments, the active brain wave test may be conducted multiple times, using the same or different visual stimuli, to obtain an average result. The results of the active brain wave test may comprise, for example, temporal and spatial measurements of alpha waves, beta waves, delta waves, and theta waves. In some embodiments, the results of the active brain wave test may comprise a ratio of two types of brain waves; for example, the results may include a ratio of alpha/theta waves.
Similarly, a passive brain wave test may be performed. The passive brain wave test may be conducted using EEG (electroencephalography) equipment to record brain wave data while the individual has closed eyes; i.e., in the absence of visual stimuli. The results of the passive wave brain wave test may comprise, for example, temporal and spatial measurements of alpha waves, beta waves, delta waves, and theta waves, for example. In some embodiments, the results of the passive brain wave test may comprise a ratio of two types of brain waves; for example, the results may include a ratio of alpha/theta waves. In some embodiments, the passive brain wave test may be conducted multiple times to obtain an average result.
When possible, and reliant upon precise timing information for both electric potentials and stimulus displays/speakers, time-averaged responses can be generated from repeated trials. Characteristic waveforms associated with visual or auditory processing (Event Related Potentials, ERP) can be measured and manipulated in various ways. As these do not require volitional behavior from users they represent a lower-level, arguably more pure measure of perception.
Referring back to FIG. 3 , electrophysiological data 318 may be yet another efferent data source 306 , which may generally be available in the form of voltage potentials recorded at a rate on the order of kHz. This may include any and all measurements of voltage potentials among electrodes placed on the skin or other exposed tissue (notably the cornea of the eye). Most use cases would presumably involve noninvasive recording, however opportunities may arise to analyze data from implanted electrodes placed for other medically valid purposes. Data may generally be collected at rates in the hundreds or thousands of samples per second. Analyses may focus on either time-locked averages of responses to stimulus events to generate waveforms or on various filtered representations of the data over time from which various states of activity may be inferred. For example, Electroencephalogram (EEG) may be used to gather electrode recording from the scalp/head, to reveal electrical activity of the brain and other neural activity. Recording may focus on areas of primary sensory processing, secondary and later sensory processing, cognitive processing or response generation (motor processing, language processing). An Electrooculogram (EOG) may be utilized to gather electrode recording from near the eye to measure changes in field potential due to relative eye position (gaze direction) and can also measure properties of retinal function and muscle activity. EOG may provide a low spatial resolution substitute for video eye tracking. An Electroretinogram (ERG) may be used to gather electrode recording from the cornea (minimally invasive) to capture neural activity from the retina. Correlation with chromatic and spatial properties of stimuli may allow for the characterization of responses from different cone types and locations on the retina (this is also the case with visual evoked potentials recorded via EEG). An Electrocardiogram (ECG) may be used to gather neuromuscular activity corresponding to cardiac function and provide measures of autonomic states, potentially in response to stimuli. Measurement of neuromuscular potentials may involve electrodes placed anywhere to record neuromuscular activity from skeletal muscle flex and/or movement of body and limb (including electromyogram, or EMG). Measurement of Galvanic Skin Response (GSR) may involve electrodes that can measure potential differences across the skin which are subject to conductance variations due to sweat and other state changes of the skin. These changes are involuntary and may reveal autonomic responses to stimuli or scenarios.
Another source of efferent data 306 may be autonomic monitoring data 320 , including information about heart rate, respiratory rate, blood oxygenation, skin conductance, and other autonomic (unconscious) response data from user 302 in forms similar to those for electrophysiological data 318 . Pressure transducers or other sensors may relay data about respiration rate. Pulse oximetry can measure blood oxygenation. Pressure transducers or other sensors can also measure blood pressure. Any and all unconscious, autonomic measures may reveal responses to stimuli or general states for categorization of other data. FIG. 9 illustrates characteristic metrics 900 for electrophysiological monitoring data 902 and autonomic monitoring data 904 , in accordance with an embodiment of the present specification.
Sample questions concerning electrophysiological metrics 902 and autonomic metrics 904 that may be answered over a period of time may include: what are the characteristics of time-averaged responses to events, how do various frequency bands or other derived states change over time or in response to stimuli.
Sensors for collecting data may be a part of hardware 106 , described above in context of FIG. 1A . Some sensors can be integrated into an HMD (for example, sensors for electroencephalography, electrooculography, electroretinography, cardiovascular monitoring, galvanic skin response, and others). Referring back to FIG. 3 , some data may require sensors elsewhere on the body of user 302 . Non-contact sensors (even video) may be able to monitor some electrophysiological data 318 and autonomic monitoring data 320 . In embodiments, these sensors could be smart clothing and other apparel. It may be possible to use imaging data for users, to categorize users or their present state. Functional imaging may also provide data relating to unconscious responses to stimuli. Imaging modalities include X-Ray/Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ophthalmic Imaging, Ultrasound, and Magnetoencephalography (MEG). Structural data derived from imaging may be used to localize sources of electrophysiological data (e.g. combining one or more of structural, MRI EEG, and MEG data).
Metrics may be broken into direct measures that can be inferred from these stimulus/response feature pairs, and indirect measures that can be inferred from the direct measures. It should be understood that in most cases individual occurrences of stimulus/response feature pairings may be combined statistically to estimate central tendency and variability. There is potential value in data from a single trial, from descriptive statistics derived from multiple repeated trials of a particular description and from exploring stimulus and/or response features as continuous variables for modelling and prediction.
Facial Pattern Recognition Machine Learning
The SDEP may utilize its models and predictive components in combination with a product to enable development of a customized predictive component for the product. The SDEP predictive components may be built through a collection process by which a large dataset of vision data from naturalistic or unconstrained settings from both primary and secondary sources may be curated and labeled. The dataset may include photographs, YouTube videos, Twitch, Instagram, and facial datasets that are available through secondary research, such as through the Internet. The curated and labeled data may be utilized for further engagement, and to build a custom-platform for the product.
FIGS. 10A to 10D illustrate an exemplary process of image analysis of building curated data. The illustrations describe an exemplary mobile-based version of the model. In other embodiments, the model may be executed on the cloud. FIG. 10A illustrates an exemplary image of a subject for whom a customized predictive component may be developed. FIG. 10B illustrates an image of the subject where the SDEP identifies the eyes for eye tracking, blink detection, gaze direction, and other parameters and/or facial attributes. In embodiments, the eyes are continually identified for tracking purposes through a series of images or through a video of the subject.
FIG. 10C illustrates a dataset 1002 of vision data from naturalistic or unconstrained settings, which may be used for extracting face attributes in the context of eye tracking, blink, and gaze direction. In embodiments, the SDEP system is trained with a large data set 1002 under different conditions where the frames are extracted from videos. Different conditions may include among other, complex face variations, lighting conditions, occlusions, and general hardware used. In embodiments, various computer vision techniques and Deep Learning are used to train the system. Referring to FIGS. 10C and 10D , image 1004 is selected to extract face attributes for analyzing emotions of the subject. In embodiments, images from the dataset, including image 1004 , are curated and labelled.
The following steps outline an exemplary data curation and labelling process:
1. Identify desirable data sources 2. Concurrently, develop a pipeline to perform facial key point detection from video and still images. This may be achieved by leveraging facial key point localization to segment and select the ocular region from faces. Further key point features may be used to determine rotation, pitch, and lighting of images, as possible dimensions to marginalize over in downstream analysis. Facial expressions may be identified to analyze emotions. Blinks, eye movements, and microsaccades may also be identified as part of the key point detection system. 3. Scrapes of data sources may be identified and fed through the SDEP to obtain a normalized set of ocular region images. Final images may be segmented/cropped to include only the ocular region, such that information on pitch, rotation, and lighting is available upon return. 4. Output from the above processing may be combined with a product to label blink, coloration, strabismus, and other metrics of interest to the product.

The above-mentioned collected and labelled data may be leveraged to develop custom predictive models of the ocular region. Customized machine learning algorithms may be created to predict key parameters ranging from blink rate, fatigue, emotions, gaze direction, attention, phorias, convergence, divergence, fixation, gaze direction, pupil size, and others. In addition, multimodal approaches may leverage the SDEP in order to benefit from pixel level information in digital stimuli and jointly learn relationships with ocular response. The pixel level information may be broken down to RGB, luminance to fuse the same with existing visual modeling algorithms.
In embodiments, eye tracking parameters are extracted from eye tracking algorithms. In an embodiment, pupil position, relative to the face, provides one measure from which to classify eye movements as fixations, pursuits and saccades. In an embodiment, pupil size is also measured, independently for both eyes. In an embodiment, gaze direction is estimated from relative pupil position. Gaze position may be measured in 3D space using data from both eyes and other measures (i.e. relative position of the face and screen), including estimates of vergence. Gaze Position provides another measure from which to classify eye movements.
FIGS. 11A and 11B illustrate pupil position and size and gaze position over time. While FIG. 11A illustrates pupil position and size and gaze position in 3D 1104 A and 2D 1110 A, at a first time; FIG. 11B illustrates pupil position and size and gaze position in 3D 1104 B and 2D 1110 B, at a second time. In an embodiment the second time is later than the first time. At any given point in the image there is (up to) 1 second of data being shown, with older data shown in a different color, such as blue. The light blue square represents the display at which the observer was looking. Physical dimensions are not to scale (e.g. the viewing distance was greater than it appears to be in the left panel). The left panel 1104 A and 1104 B shows a 3D isometric view of space with user's eyes 1106 to the left and the display 1108 to the right.
On the left side, gaze position is shown in 3D 1104 A and 1104 B. A line is drawn from the surface of the observer's display 1108 to the gaze position; red indicates gaze position behind the display 1108 and green indicates gaze position in front of the display 1108 . Three circles convey information about the eyes 1106 :
1. The largest, dark grey outline circle represents the average position of the eyes and face, relatively fixed in space. 2. The light grey outline within represents the average pupil size and pupil position relative to the face (moves but doesn't change size). 3. The black filled circle shows relative pupil size as well as pupil position relative to the face (moves and changes size).

When the pupil information is missing it may be assumed that the eyes are closed (or otherwise obscured).
Gaze position in 3D 1104 A and 1104 B is shown by a black dot (connected by black lines), with gaze direction emanating from both eyes. Depth of gaze from the display is further indicated by a green (front) or red (behind) line from the display to the current gaze position.
On the right side, gaze position 1110 A and 1110 B is shown in 2D. Here information about the pupils is absent. Also, information classifying eye movements is added:
1. Black indicates fixation during which a grey outline grows indicating relative duration of the fixation. 2. Blue indicates pursuit. 3. Green (with connecting lines) indicates saccades with lines connecting points during the saccade.

Brain-Machine Interfaces
Overall Brain-Machine (Computer) Interface (BMI/BCI) standardization requires standardizing the interoperability, connectivity, and modularity of multiple sensory interfaces with the brain, with many being closed looped. Embodiments of the present specification provide an exchange platform, given the current limitations of closed-loop symptoms, in supporting a standardization of these requirements.
Additionally, current measures and rating systems for VR/AR/MxR are qualitative in nature. Embodiments of the present specification aid in establishing quantitative measures to improve the quality of the user experience in VR/AR/MxR environments. This standardization is necessary, as these HMDs are becoming more pervasive in the non-clinical settings.
Current BMI/BCI interfaces, including but not limited to EEG, MRI, EOG, MEG, fMRI, ultrasound, and microwaves, are modular in nature. Among these different potential interfaces, the division in clinical and non-clinical context, is in part limited to the portability of the interfaces, with non-clinical being traditionally more portable. Vision data may be learned and utilized within the SDEP for different means of connectivity and interoperability, that will translate to the larger equipment involved in BMI/BCI interfaces, including but not limited to MRI, MEG, and others.
Embodiments of the present specification describe exemplary use cases that may be utilized for standardization of both the hardware components that make up HMDs and the software requirements for apps used in AR/VR environments.
Referring to the hardware components, features of HMDs may be key for standardization. For example, HMD devices have built in cameras very well suited to capture vision related data and extract various parameters to glean information in ways which was not possible before. This, combined with contextual information and data from other allied sensors may provide a unique opportunity to study the data and put perceptual computing into BMI/BCI systems. Defining of minimal specifications of cameras may be required to achieve this type of data capture for perceptual computing.
Referring to software components, displaying stimuli in VR/AR/MxR environments the embodiments of present specification provide systems and methods for being cognizant of focal point position, crowding, vection, accommodative mismatch, prolonged convergence and divergence, chroma-luminance, frame rate, sequencing, and other factors. Ignoring them may lead to multiple points for visually-induced motion sickness, headaches and/or computer vision syndrome. Quantification methods to establish data collection and pattern recognition for best in practice developer design methods of software is needed and are part of embodiments of the present specification.
HMDs do provide conflict between visual and vestibular stimuli because of the variations in foveal and peripheral vision. Also HMD image does not move with the head motion of the wearer. Embodiments of the present specification may be able to measure torsional eye movements, vergence eye movements, gaze detection and pupillary response from the data captured by native eye-tracking components in a non-invasive way. Data capturing for data like the minimal frame rate and pupil capture, may thus be standardized for the SDEP, in accordance to the various embodiments.
The SDEP built as part of above embodiments may comprise data ingestion modules from different sources such as HMD, mobile, various eye trackers, image and video sources, traditional imaging systems such as fMRI, X-ray, and the like, and data from EEG, EOG, and others.
The machine learning modules may process data in batch and real-time mode and expose the same as API so that it can be integrated and interfaced with multiple applications. The machine learning system may use Deep Convolutional neural networks to detect pupillary metrics, blink detection and gaze accurately from any image or video source. The other machine learning components may then correlate this data with sensory data inputs such as EEG, EOG, EMG, head movement data, haptic data and build comprehensive perceptual models of human vision.
Vision Performance Index
An important class of metrics may be those relating to performance. The performance of a user may be determined in the form of Vision Performance Index (VPI), which is described in detail subsequently in embodiments of the present specification.
Referring back to FIG. 1A , in an embodiment, data collected from user 102 , such as by media system 104 , may be processed to identify a Vision Performance Index (VPI) for user 102 (also referring to 1210 of FIG. 12 ). The VPI may indicate a level of vision performance of user 102 assessed during user's 102 interaction with VR/AR/MxR system 104 . The VPI may be used to identify a group of users for user 102 that have a similar VPI. The VPI may be further utilized to modify VR/AR/MxR media for user 102 in order to minimize Visually Induced Motion Sickness (VIMS), or any other discomfort arising from the virtual experience. In an embodiment, media is modified in real time for user 102 . In another embodiment, VPI is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar VPI, or subsequently to user 102 .
VPI may be measured and manipulated in various ways. In general, the goal may be to improve user's vision performance, however manipulations may also be aimed at increasing challenge (e.g. for the sake of engagement) which may, at least temporarily, decrease performance. In alternate embodiments, performance indices other than or in addition to that related to vision may be measured and manipulated. For example, other areas such as design, engagement, and the like, may be measured and manipulated through performance indices.
Referring again to FIG. 12 , an exemplary outline of a data analysis chain is illustrated. The data analysis begins at the lowest level at 1202 where data level may not be simplified further. At 1202 , parameters of a single stimulus can be used for multiple measures based on different independent variables, which correspond to direct features of a stimulus. Parameters of a single response can be used for multiple measures based on different dependent variables. At 1204 independent and dependent variables may be paired to extract a measure of a user's vision performance, or combined with others and fit to a model to generate measures of the user's vision performance. In embodiments, pairing may involve combining a response event to one or more stimulus events through correlation or other statistical/non-statistical methods. Individual pairs may be filtered to arrive at 1206 , where, for a given type of interaction, many pairs of independent and dependent variables can be used to either estimate the parameters of a model distribution or estimate descriptive statistics. In embodiments, a model distribution is an expectation of how often a measure will be a specific value. In some instances a normal distribution, which has the classic shape of a ‘Bell curve’, may be used. Once the process of descriptive statistics or model fitting is completed, at 1208 , an individual estimate of a physical measure of a property of user's vision may be generated. The individual user estimate may be based on a single interaction or a summary measure from multiple interactions. The measures of at least one physical property may be normalized to contribute to sub-components of VPI, at 1210 . At 1212 , multiple VPI sub-components scores may be combined (for example, averaged) to generate component scores. In embodiments, component scores may be further combined to generate overall VPI. VPI, its subcomponents, and components are discussed in greater detail in subsequent sections of the present specification.
In embodiments, measures of vision performance may be presented as a normalized “score” with relative, but not absolute, meaning, to the users. This is also illustrated at 1210 and 1212 in context of FIG. 12 . Users may be able to gauge their level of performance against the general population, or specific subsets thereof. Due to the presumed high degree of measurement noise associated with data recording from non-specialized hardware (i.e. mobile devices used outside of a controlled experimental setting), precise measures of efferent phenomena (e.g. pupil size, gaze direction, blink detection) and afferent parameters (e.g. display chromoluminance, viewing distance, audio intensity) are unavailable. It may therefore be required to rely on estimates of central tendency (i.e. mean) and variability (i.e. standard deviation) from the accumulated data of all users to define “typical” ranges for each measure and to set reasonable goals for increasing or decreasing those measures.
Scores may be normalized independently for each type of measure, for each of a variety of types of tasks and generally for each unique scenario or context. This may enable easy comparison and averaging across measures taken in different units, to different stimuli, and from different kinds of user responses. Additionally, for any and all scores, performance may be categorized as being marginally or significantly above or below average. Set descriptive criteria may be decided based on percentiles (assuming a given measure will be distributed normally among the general population). The examples in the following sections use 10% and 90%, however the percentiles may be arbitrarily chosen and can be modified for specific contexts. It may be assumed that 10% of users' scores will fall in the bottom or top 10% of scores, and therefore be ‘abnormally’ low or high, respectively.
In an embodiment, VPI may be a combination of one or more of the following parameters and sub-parameters, which may be both afferent and efferent in nature. Direct measures generally relate a single response feature to a single stimulus feature. Whenever possible a psychometric function may be built up from the pattern of responses (average response, probability of response or proportion of a category of responses) as the stimulus feature value changes. Direct measure may include the following: detection, discrimination, response time, and/or error.
Indirect measures may be the higher level interpretations of the direct measures and/or combinations of direct measures. These may also generally include descriptions of direct measures within or across specific contexts and the interactions among variables. Indirect measures may include the following: multi-tracking, fatigue/endurance, adaptation/learning, preference, memory, and/or states.
In embodiments, other vision-related parameters may be used to calculate the VPI, and may include, but are not limited to field of view (F), accuracy (A), multi-tracking (M), endurance (E), and/or detection/discrimination (D), together abbreviated as FAMED, all described in greater detail below.
Field of View (F)
Referring back to FIG. 1A , the Field of View (F) may be described as the extent of visual world seen by user 102 at any given moment. Central vision represents a central part of the field of view of user 102 , where user 102 has the greatest acuity which is important for things like reading. Peripheral Vision is the external part of the field of view of user 102 , which is important for guiding future behavior and catching important events outside of user's 102 focus.
Field of View measures the relative performance of users when interacting with stimuli that are in their Central or Peripheral fields of view based on measures of Accuracy and Detection. It is assumed that performance should generally be worse in the periphery due to decreased sensitivity to most stimulus features as visual eccentricity increases. The ratio of performance with Central and Peripheral stimuli will have some mean and standard deviation among the general population; as with other measures, the normalized scores will be used to determine if users have abnormally low or high Field of View ability.
If a user's Field of View score is abnormally low it may be improved by increasing the Accuracy and Detection scores for stimuli presented in the periphery. This generally would entail increasing consistency of timing and position, increasing chromaticity and luminance differences (between and within objects), increasing the size of objects and slowing any moving targets when presented in the periphery.
Accuracy (A)
Referring back to FIG. 1A , accuracy (A) may be a combination of making the right choices and being precise in actions performed by user 102 . Measures of accuracy may be divided into two subcomponents: Reaction and Targeting. Reaction relates to the time it takes to process and act upon incoming information. Reaction may refer to ability of the user 102 to make speedy responses during the VR/AR/MxR experience. Reaction may be measured as the span of time between the point when enough information is available in the stimulus to make a decision (i.e. the appearance of a stimulus) and the time when the user's response is recorded. For a speeded response this will usually be less than one second.
If a user's Reaction is abnormally slow (abnormally low score) it may be that the task is too difficult and requires modification of stimulus parameters discussed later in the context of Targeting and Detection. In an embodiment, a model distribution for any given measure (for example, a log-normal distribution for reaction times) is estimated. A cut-off may be determined from the estimate, above which 5% (or any other percentage) slowest time spans are found. Any incoming measure of reaction time that is equal or greater to the cut-off is considered ‘slow’ (or ‘significantly slow’). However, if reaction alone is abnormally low, when other scores are normal, it may be a sign of poor engagement with the task or a distraction. It may be helpful to reduce the number of items presented simultaneously or add additional, congruent cues to hold attention (e.g. add a sound to accompany the appearance of visual stimuli). If the user is required to respond to the location of a moving object, it may be that they require longer to estimate trajectories and plan an intercepting response; slowing of the target may improve reaction.
Response Time may be important for detection related measures, but is relevant to any response to a stimulus. Response time is generally the time span between a stimulus event and the response to that event. Response time may be used to measure the time necessary for the brain to process information. As an example, the appearance of a pattern on a display may lead to a certain pattern of responding from the retina measurable by ERG. At some point after the stimulus processing is evident from an averaged ERG waveform, the processing of that same stimulus will become evident in an average visual evoked potential (VEP) waveform recorded from the back of the head. At some point after that the average time to a button press response from the user indicates that the stimulus was fully processed to the point of generating a motor response. Though multiple timestamps may be generated by stimulus and response events, the response time should generally be taken as the time between the earliest detectable change in the stimulus necessary to choose the appropriate response to the earliest indication that a response has been chosen. For example, if an object begins moving in a straight line towards some key point on the display, that initial bit of motion in a particular direction may be enough for the user to know where the object will end up. They need not wait for it to get there. Likewise the initiation of moving of the mouse cursor (or any other gesture acceptable in a VR/AR/MxR environment) towards a target to be clicked may indicate that a response has been chosen, well before the click event actually occurs.
In embodiments, other changes in patterns of responding, including improvements, decrements and general shifts, may occur as the result of perceptual adaptation, perceptual learning and training (higher order learning). Considering adaptation and learning by the user may account for any variability in responses that can be explained, and thereby reduce measures of statistical noise and improve inferential power.
Patterns in responding, and changes thereof, may also be related to high order processes within the system. Users have an occasional tendency to change their minds about how they perform a task while they're doing it. Therefore, in embodiments, every choice made by users is analyzed for preferences, regardless of whether it informs models of visual processing.
In embodiments, responses are used by the system to measure recall or recognition by a user. Recall is the accurate generation of information previously recorded. Recognition is the correct differentiation between information previously recorded and new information.
Derived from measures over time and in specific contexts, measures of memory recall and recognition and memory capacity can be made. These may generally fall under the performance category and users may improve memory performance with targeted practice. Recall and recognition are often improved by semantic similarity among stimuli. Memory span may, likewise, be improved by learning to associate items with one another. The span of time over which items must be remembered may also be manipulated to alter performance on memory tasks. Distracting tasks, or lack thereof, during the retention span may also heavily influence performance.
For long term memory there may be exercises to enhance storage and retrieval, both of specific items and more generally. It may also be possible to derive measures associated with muscle memory within the context of certain physical interactions. Perceptual adaptation and perceptual learning are also candidates for measurement and manipulation.
Targeting relates to measures of temporal and positional precision in the user's actions. Referring back to FIG. 1A , targeting may relate to the precision of the responses of user 102 relative to the position of objects in the VE. Targeting is measured as the error between the user's responses and an optimal value, in relation to stimuli. The response could be a click, touch, gesture, eye movement, pupil response, blink, head movement, body/limb movement, or any other. If the user is expected to respond precisely in time with some event (as opposed to acting in response to that event, leading to a Reaction measure), they may respond too early or too late. The variability in the precision of their response yields a Targeting time error measure (usually on the order of one second or less). Additionally the position of the user's responses may have either a consistent bias (mean error) and/or level of variability (standard deviation of error) measured in pixels on the screen or some other physical unit of distance.
In embodiments, the system analyzes data related to user errors, including incorrect choices and deviations made by the user from the ideal or an optimum response. Most commonly these may be misidentification of stimuli, responding at inappropriate times (false positive responses), failing to respond at appropriate times (false negatives) and inaccuracy of timing or position of responses. Variability in responses or measures of response features may also be indications of error or general inaccuracy or inconsistency.
If a user's targeting score is abnormally low it may be that targets are too small or variability of location is too great. For timing of responses, more consistent timing of events makes synchronizing responses easier. This may be in the form of a recurring rhythm or a cue that occurs at some fixed time before the target event. For position, errors can be reduced by restricting the possible locations of targets or, in the case of moving targets, using slower speeds. Particularly for touch interfaces or other contexts where responses may themselves obscure the target (i.e. finger covering the display), making the target larger may improve targeting scores.
Multi-Tracking (M)
Multi-tracking (M) may generally refer to instances in which users are making multiple, simultaneous responses and/or are responding to multiple, simultaneous stimuli. They also include cases where users are performing more than one concurrent task, and responses to stimulus events that occur in the periphery (presumably while attention is focused elsewhere). Combination measures of peripheral detection (detection as a function of eccentricity) and other performance measures in the context of divided attention may be included.
Multi-tracking (M) may represent the ability of the user to sense multiple objects at the same time. Divided attention tasks may require user to act upon multiple things happening at once. Multi-Tracking measures the relative performance of users when interacting with stimuli that are presented in the context of Focused or Divided Attention. With focused attention, users generally need to pay attention to one part of a scene or a limited number of objects or features. In situations requiring divided attention, users must monitor multiple areas and run the risk of missing important events despite vigilance. As with Field of View, measures of Accuracy and Detection are used to determine a user's performance in the different Multi-Tracking contexts.
If a user's Multi-Tracking score is abnormally low it may indicate that they are performing poorly with tasks requiring Divided Attention, or exceptionally well with tasks requiring Focused Attention. Therefore, making Divided Attention tasks easier or Focused Attention tasks more difficult may improve the Multi-Tracking score. In the context of Divided Attention, reducing the perceptual load by decreasing the number of objects or areas the user needs to monitor may help. Increasing durations (object persistence) and slowing speeds in Divided Attention may also improve scores.
Fatigue/Endurance (E)
Performance measures may become worse over time due to fatigue. This may become evident in reductions in sensitivity (detection), correct discrimination, increase in response time and worsening rates or magnitudes of error. The rate of fatigue (change over time) and magnitude of fatigue (maximum reduction in performance measures) may be tracked for any and all measures. The delay before fatigue onset, as well as rates of recovery with rest or change in activity, may characterize endurance.
Endurance (E) may be related to the ability of user to maintain a high level of performance over time. Endurance measures relate to trends of Accuracy and Detection scores over time. Two measures for Endurance are Fatigue and Recovery.
Fatigue is a measure of how much performance decreases within a span of time. Fatigue is the point at which the performance of user may begin to decline, with measures of a rate of decline and how poor the performance gets. The basic measure of fatigue may be based on the ratio of scores in the latter half of a span of time compared to the earlier half. We assume that, given a long enough span of time, scores will decrease over time as users become fatigued and therefore the ratio will be less than 1. A ratio of 1 may indicate no fatigue, and a ratio greater than 1 may suggest learning or training effects are improving performance along with a lack of fatigue. If a user's Fatigue score is abnormally low then they may want to decrease the length of uninterrupted time in which they engage with the task. Taking longer and/or more frequent breaks may improve Fatigue scores. Generally decreasing the difficulty of tasks should help as well.
Recovery is a measure of performance returning to baseline levels between spans of time, with an assumed period of rest in the intervening interval. Recovery may relate to using breaks provided to user effectively to return to optimum performance. The basic measure of recovery currently implemented is to compare the ratio of scores in the latter half of the first of two spans of time to the scores in the earlier half of the second span of time. The spans of time may be chosen with the intention of the user having had a bit of rest between them. We assume that, given long enough spans of time to ensure some fatigue is occurring, scores will be lower before a break compared to after and therefore the ratio will be less than 1. A ratio of 1 indicates no effect of taking a break, and a ratio greater than 1 may indicate a decrease in engagement after the break or the presence of fatigue across, and despite, the break.
If a user's Recovery score is abnormally low, they may want to take longer breaks. It's possible they are not experiencing sufficient fatigue in order for there to be measurable recovery. Challenging the user to engage for longer, uninterrupted spans of time may improve recovery scores. Likewise an increase in task difficulty may result in greater fatigue and more room for recovery.
Detection/Discrimination (D)
Detection/Discrimination (D) may refer to the ability of the user to detect the presence of an object, or to differentiate among multiple objects. This parameter may depend on the sensitivity of user to various attributes of the object. Whenever a response event signals awareness of a stimulus event it may be determined that a user detected that stimulus. Unconscious processing, perhaps not quite to the level of awareness, may also be revealed from electrophysiological or other responses. Detection can be revealed by responding to the location of the stimulus or by a category of response that is congruent with the presence of that stimulus (e.g. correctly identifying some physical aspect of the stimulus). The magnitude of a stimulus feature parameter/value necessary for detection may define the user's detection threshold. Any feature of a stimulus may be presumed to be used for detection, however it will only be possible to exclusively attribute detection to a feature if that feature was the only substantial defining characteristic of the stimulus or if that stimulus feature appears in a great variety of stimuli to which users have made responses.
Whenever users correctly identify a stimulus feature parameter/value or make some choice among multiple alternatives based on one or more stimulus features that interaction may contribute towards a measure of discrimination. In many cases the measure of interest may be how different two things need to be before a user can tell they are different (discrimination threshold). Discrimination measures may indicate a threshold for sensitivity to certain features, but they may also be used to identify category boundaries (e.g. the border between two named colors). Unlike detection measures, discrimination measures need not necessarily depend upon responses being correct/incorrect. Discrimination measures may indicate subjective experience instead of ability.
Measures of Detection/Discrimination may be divided into three subcomponents: measures related to detecting and/or discriminating Color (chromoluminance), Contrast (chromoluminant contrast), and Acuity measures based on the smallest features of a stimulus. These afferent properties, in combination with efferent measures from manual or vocal responses, eye tracking measures (initiation of pro-saccade, decrease in anti-saccade, sustained fixation and decreased blink response), gaze direction, pupil size, blinks, head tracking measures, electrophysiological and/or autonomously recorded measures, measures from facial pattern recognition and machine learning, and others are used to determine sensitivity. All measures may be based on a user's ability to detect faintly visible stimuli or discriminate nearly identical stimuli. These measures are tied to the different subcomponents based on differences (between detected objects and their surroundings or between discriminated objects) in their features. Stimulus objects can differ in more than one feature and therefore contribute to measures of more than one subcomponent at a time.
Color differences may refer specifically to differences in chromaticity and/or luminance. If a user's Color score is abnormally low, tasks can be made easier by increasing differences in color. Specific color deficiencies may lead to poor color scores for specific directions of color differences. Using a greater variety of hues will generally allow specific deficiencies to have a smaller impact and stabilize scores.
Contrast differs from Color in that contrast refers to the variability of chromaticity and/or luminance within some visually defined area, whereas measures relating to Color in this context refer to the mean chromaticity and luminance. If a user's Contrast score is abnormally low it may be improved by increasing the range of contrast that is shown. Contrast sensitivity varies with spatial frequency, and so increasing or decreasing spatial frequency (making patterns more fine or coarse, respectively) may also help. Manipulations that improve Color scores will also generally improve Contrast scores.
Acuity measures derive from the smallest features users can use to detect and discriminate stimuli. It is related to contrast in that spatial frequency is also a relevant physical feature for measures of acuity. If a user's Acuity score is abnormally low it may be that objects are generally too small and should be enlarged overall. It may also help to increase differences in size, increase contrast and decrease spatial frequency. More so with Acuity than Color or Contrast, the speed of moving stimuli can be a factor and slowing moving targets may help improve Acuity scores.
The above parameters are all based on measuring features. In embodiments, their patterns may be noted over time. Trends and patterns may enable predictive analytics and also help personalize the user experience based on detection capabilities and other VPI/FAMED capabilities of the end user.
A great many general states of being may be inferred from the direct measures discussed. States may be estimated once per session, for certain segments of time or on a continuous basis, and in response to stimulus events. These may commonly relate to rates of responding or changes in behavior. FIG. 13 provides a table containing a list of exemplary metrics for afferent and efferent sources, in accordance with some embodiments of the present specification. The table illustrates that an afferent source may result in a stimulus event and feature. The combination of afferent source, stimulus events and feature, when combined further with a response (efferent source), may indicate a response event and feature. These combinations may hint at a psychometric measure. In the last column, the table provides a description for each psychometric measure derived from the various combinations.
FIG. 14 is an exemplary flow diagram illustrating an overview of the flow of data from a software application to the SDEP. At 1402 , a software application that may provide an interface to a user for interaction. The app may be designed to run on an HMD, or any other device capable of providing a VR/AR/MxR environment for user interaction. Information collected by the application software may be provided to a Software Development Kit (SDK) at 1404 . The SDK works with a group of software development tools to generate analytics and data about use of the application software. At 1406 the data is provided as session data from the SDK to the SDEP. At 1408 , session data is pre-processed at the SDEP, which may include organizing and sorting the data in preparation for analysis. At 1410 , stimulus and response data that has been pre-processed is generated and passed further for analysis and processing. At 1412 , data is analyzed and converted to performance indices or scores or other measures of perceivable information, such as VPI scores. At 1414 , the analyzed data is sent back to the SDK and/or application software in order to modify, personalize, or customize the user experience. In embodiments data is passed from 1402 , from application software through the chain of analysis, and back to the application software non-intrusively, in real time.
FIG. 15 illustrates an exemplary outline 1500 of a pre-processing part of the process flow ( 1408 , FIG. 14 ).
FIG. 16 is an exemplary representation 1600 of the programming language implementation of a data processing function responsible for taking in raw data (pre-processed), choosing and implementing the appropriate analysis, sending and receiving summary measures based on the analysis to temporary and long-term stores for estimates of ‘endurance’ measures and score normalization, respectively, and computing scores to be sent back to the application for display to the end user. In embodiments, the programming language used is Python. The figure shows application of several Python functions to FAMED data in order to derive VPI scores. The figure illustrates color-coded processes for each FAMED function. In an embodiment, FOV functions are in Red, Accuracy in Green, Multi-Tracking in Purple, Endurance in Orange, and Detection in Blue. In an embodiment, parallelograms represent variables; rounded rectangles represent functions; elements are color coded for user/session data, which are shown in yellow.
Referring to the figure, contents of a large red outline 1602 represent the processing function (va_process_data), which includes three main sections—a left section 1604 , a middle section 1606 and a right section 1608 . In an embodiment, left section 1604 takes in raw data and applies either Accuracy or Detection/Discrimination analysis functions to the data yielding a single measure summarizing the incoming data. That is sent to middle-level functions 1606 for measures of Field of View and Multi-Tracking as well as to an external store. That first external store, or cache, returns similar measures from the recent past to be used for measures of Endurance. The output from the middle-level functions 1606 are sent to another external store that accumulates measures in order to estimate central tendency (i.e. arithmetic mean) and variability (i.e. standard deviation) for normalization. Data from this second, external store are combined with the present measurements to be converted into Scores in the right-level section 1608 . The figure also illustrates a small sub-chart 1610 in the lower left of the figure to show the placement of analysis portion 1600 in the broader chain.
FIG. 17 provides a flowchart illustrating a method for modifying media, in accordance with an embodiment of the present specification. In an embodiment, the method is implemented within SDEP 118 described above in context of FIG. 1A and the various embodiments. A user is presented with a media, for example, VR/AR/MxR media. In embodiments, media is presented through an HMD or any other type of VR/AR/MxR media rendering device. While the user experiences the media, at 1702 , the SDEP receives vision data of the user. In an embodiment, the user is in accordance to user 102 of FIG. 1A . In an embodiment, vision data is received from various afferent and efferent data sources that are engaged within the media environment. At 1704 , the vision data is used to process the media and modify it. In embodiments, vision data is processed in real time. The media may be modified to enable the user to experience the media in an optimal manner. The media may be modified to improve the user experience in order to minimize VIMS or any other problems that may be induced otherwise based on the visual capacity of the user. In embodiments, the media is modified differently for different applications. For example, a media may be modified differently for users of games and differently for users who are potential customers, where the media is respectively presented through a game and through an advertisement. In other examples, the media may be presented differently for users of a specific application, and content experience. In an embodiment, the vision data is processed in real time to modify it. The vision data here is the vision model/profile/persona developed through both batch and real-time analysis and contextualizing of afferent and efferent data, along with autonomic data input. Alternatively, the vision data is stored and analyzed for modification, in batches. The different forms of processing media are described above in context of FIG. 2 . The VPI may also be used to process the media for its modification in accordance to various metrics. At 1706 , modified media is represented to the user. In an embodiment, modified media is presented to a group of users at the same time or at different times, where the users in the group may correspond to similar vision data. The resulting media may be in continuation to previously presented media, modified in accordance to certain metrics determined for the user.
FIG. 18 provides a flowchart illustrating a method for modifying media, in accordance with another embodiment of the present specification. In an embodiment, the method is implemented within SDEP 118 described above in context of FIG. 1A and the various embodiments. A user is presented with a media, for example, VR/AR/MxR media. In embodiments, media is presented through an HMD or any other type of VR/AR/MxR media rendering device. While the user experiences the media, at 1802 , the SDEP receives vision data of the user. In an embodiment, the user is in accordance to user 102 of FIG. 1A . In an embodiment, vision data is received from various afferent and efferent data sources that are engaged within the media environment. At 1804 , the vision data is used to identify metrics that may affect the visual experience of the user directly or indirectly. The metrics, as described above in context of the table of FIG. 13 , may aid in deconstructing psychometrics that impact user's experience. At 1806 , information derived from the metrics may be used to process the media and modify it. The media may be modified to enable the user to experience the media in an optimal manner. The media may be modified to improve the user experience in order to minimize VIMS or any other problems that may be induced otherwise based on the visual capacity of the user. In embodiments, the media is modified or personalized differently for different applications. For example, a media may be personalized differently for users of games and differently for users who are potential customers, where the media is respectively presented through a game and through an advertisement. In an embodiment, the vision data is processed in real time to modify it. Alternatively, the vision data is stored and analyzed for modification, in batches. The different forms of processing media are described above in context of FIG. 2 . At 1808 , modified media is represented to the user. In an embodiment, modified media is presented to a group of users at the same time or at different times, where the users in the group may correspond to similar vision data. The resulting media may be in continuation to previously presented media, modified in accordance to certain metrics determined for the user.
Examples of Use
Data generated by the SDEP in accordance with various embodiments of the present specification may be used in different forms. In embodiments, data output by the SDEP may be packaged differently for gamers, for advertisers, and others.
The sensory inputs determined and analyzed by the system may eventually drive work and play engagements. In embodiments, sensory information may be purchased from users and used to create sensory data exchanges after adding value to the data through platforms such as the SDEP. In embodiments of the present specification, the senses of individuals and potential consumers may be measured and monitored with the SDEP.
In embodiments, the SDEP allows for advantageously using data generated from technologies such as smart devices, wearables, eye-tracking tools, EEG systems, and virtual reality and augmented reality HMDs. For example, EEG bands may be used to track eye movement against electrodes in the brain as well as game-based applications designed to create vision benchmarks and, ultimately, help improve visual acuity over time.
In embodiments, data output by the SDEP may be packaged differently for medical use (visual acuity, eye strain, traumatic brain injury, and sports vision performance), for athletes/sports, and others. For example, applications include the ability to track the effects of digital eye strain over a period of time or to screen for traumatic brain injury in contact sports such as football by measuring key areas of the eye-brain connection.
In embodiments, systems and methods of the present specification are used to develop deep learning systems and used to model artificial neural networks. Artificial Neural Networks and its advanced forms, including Deep Learning, have varying applications such as and not limited to image/speech recognition, language processing, Customer Relationship Management (CRM), bioinformatics, facial expression recognition, among others. In embodiments, the SDEP uses feedback loops between efferent and afferent information to model the human sensory system. In further embodiments, the SDEP sources information from additional sensors to combine with the afferent and efferent sources of information.
Example Use 1: Modifying Media In Order To Improve A User's Comprehension Of Information In That Media
In one embodiment, a user's degree of comprehension is measured by testing knowledge and understanding, namely by presenting through a display a plurality of questions, receiving inputs from the user in response to those questions, and determining to what extent the user answered the questions correctly. A user's degree of comprehension may also be inferred from the user's behavior, including subtle changes in behavior, and from the user's autonomic and electrophysiological measures.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to improve that user's comprehension of information being communicated by that media. FIG. 19 illustrates a flow chart describing an exemplary process for modifying media in order to improve comprehension, in accordance with some embodiments of the present specification. At 1902 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken into account, either individually or in combination.
To probe a user's comprehension of some spatially defined process in media, the system applies measures of where the user is looking, measuring the proportion of time the user spends looking at relevant areas within the media vs. irrelevant areas within the media. The system can also measure the degree to which the user is focusing its attention on specific areas as compared to less focused sampling of the visual space.
In one embodiment, the system determines a Ratio of Relevant Fixation R Rel.Fix . defined as the ratio of fixation number, frequency or average duration in relevant areas of interest to irrelevant areas:





R

Rel
.
Fix
.


=



N

fixation

relevant



N

fixation

irrelevant



≈


f

fixation

relevant



f

fixation

irrelevant



≈



D
_


fixation

relevant




D
_


fixation

irrelevant








If the user is looking about randomly, and relevant and irrelevant areas are roughly equal in size, this ratio should be around 1. If the user is focused more on relevant areas, the ratio should be greater than 1, and the greater this ratio the more comprehension the system attributes to the user. The system determines if the user is comprehending, or not comprehending, media content based upon said ratio. If the ratio is below a predetermined threshold, then the system determines the user is not focused, is looking around randomly, and/or is not comprehending the media. If the ratio is above a predetermined threshold, then the system determines the user is focused, is not looking around randomly, and/or is comprehending the media.
In one embodiment, the system determines measures derived from saccade parameters showing more eye movements towards relevant areas compared to irrelevant areas. The system may determine if the user is comprehending, or not comprehending, media content in a VR/AR/MxR environment based upon saccadic movements.
With regard to saccade angle, the mean of the absolute angle relative to relevant regions |θ| saccade-relevant should be much less than 90° if the user is looking towards relevant areas more often, around 90° if the user is looking around randomly and greater than 90° if the user is generally looking away from relevant areas. If the user is looking more towards relevant areas, the angle would be less than 90°, and the system may determine that the user is focused and is able to comprehend the media. If the angle is near 90° or more, then the system may determine that the user is not focused, is looking around randomly, and/or has low comprehension or is not comprehending the media.
With regard to saccade magnitude, the mean magnitude component relative to relevant regions M saccade-relevant should be significantly positive if the user is looking towards relevant areas more often, around 0 if the user is looking around randomly, and significantly negative if the user is generally looking away from relevant areas. Here, a positive mean magnitude would imply a significant level of comprehension, whereas around 0, or a negative value would imply a low level of comprehension of media content in a VR/AR/MxR environment, by the user. In embodiments, the system uses this information to modify displayed media in the VR, AR and/or MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment, in order to improve that user's comprehension of information being communicated by that media
The Ratio of Relevant Fixation definition may also be expanded to include saccade parameters, although it may be assumed that these are generally equivalent to the fixation parameters:





R

Rel
.
Fix
.


=



N

saccade

relevant



N

saccade

irrelevant



≈


f

saccade

relevant



f

saccade

irrelevant








In another embodiment, the system determines a measure that exploits the fact that eye movements will frequently go back and forth between related words or objects in a scene.
The system defines Fixation Correlations C fixation between areas (A and B) known to be related, as a measure of comprehension:
C fixation =cor ( N fixation|A ,N fixation|B )≈ cor ( f fixation|A ,f fixation|B )≈ cor ( D fixation|A ,D fixation|B )

In one embodiment, the system defines
Saccade Correlations C saccade based on saccades with angles generally toward areas A and B (θ saccade-A →0 and θ saccade-B →0):
C saccade =cor ( N saccade|towards A ,N saccade|towards B )≈ cor ( f saccade|towards A ,f saccade|towards B )

The greater these kinds of correlations, the system determines that the more users are monitoring the behavior of two related objects in the scene suggesting greater comprehension. The system determines if the user is comprehending, or not comprehending, media content based upon the correlations. If the correlations are below a predetermined threshold, then the system determines the users are not focused, are looking around randomly, and/or are not comprehending the media. If the correlations are above a predetermined threshold, then the system determines the users are focused, are not looking around randomly, and/or are comprehending the media. It should be appreciated that the system may be engineered such that the reverse is true instead: If the correlations are above a predetermined threshold, then the system determines the users are not focused, are looking around randomly, and/or are not comprehending the media. If the correlations are below a predetermined threshold, then the system determines the users are focused, are not looking around randomly, and/or are comprehending the media.
Even without knowledge of the scene and what areas are relevant or should be correlated to signal comprehension, the system may assume that more or less focused attention within a scene is indicative of a degree of comprehension. In combination with more direct comprehension measures (i.e. questioning), a measure of focus can be used to take a simple correct/incorrect measure and assign to it some magnitude.
In some embodiments, comprehension is also gleaned from eye movement data of a listener compared to a speaker when both are viewing the same thing. When the eye movements of a listener are determined to be correlated to the eye movements of a speaker while explaining something going on in a shared visual scene, the system may determine that the user is able to comprehend. The system may attribute greater comprehension when the delay between the eye movements of the speaker and the corresponding eye movements of the listener is lower.
Correlation of the listener's eye movements may be calculated as:
C listening =cor ([ x,y,z] fixation|speaker ( t )[ x,y,z] fixation|listener ( t +τ))

using fixation position as a function of time for the speaker and listener with a delay of τ seconds. In embodiments, the above correlation peaks at around τ=2 s. The greater the correlation, the system determines that the more users are monitoring the behavior of two related objects in the scene, thereby suggesting greater comprehension. The system determines if the users are comprehending, or not comprehending, media content based upon the correlation. If the correlation is below a predetermined threshold, then the system determines the users are not focused, are looking around randomly, and/or are not comprehending the media. If the correlations are above a predetermined threshold, then the system determines the users are focused, are not looking around randomly, and/or are comprehending the media. It should be appreciated that the system may be engineered such that the reverse is true instead: If the correlations are above a predetermined threshold, then the system determines the users are not focused, are looking around randomly, and/or are not comprehending the media. If the correlations are below a predetermined threshold, then the system determines the users are focused, are not looking around randomly, and/or are comprehending the media.
In embodiments, an Area of Focus A focus is determined as the minimum area of the visual field (square degrees of visual angle) that contains some proportion p (e.g. 75%) of the fixations (N fixation ) or the total fixation duration (ΣD fixation ) for a given span of recording. This may be found algorithmically by estimating the smallest circular area containing all of a subset of fixations, defined by number or duration, and repeating for all possible subsets and keeping the smallest area among all subsets. In embodiments, the determined area may indicate the area that offers greater comprehension to users.
Looking Away
The action of users looking away (averting gaze) is correlated with cognitive load, indicative of a mechanism to limit the amount of detailed information coming into the visual system to free up resources. The system determines that the greater the amount of time users look away (ΣD fixation|away ) from the display, the more demanding could be the task; which in turn is determined to be evidence of greater comprehension (as looking away correlates with cognitive load as described above). In additional embodiments, if it is observed that the user looks more towards areas of less high-spatial-frequency contrast, the system may again determine that the given task is more demanding, leading to a need for the user to look away, and therefore evidence of greater comprehension. In embodiments, head movements associated with the action of looking away may be used in place of or in addition to the eye movements, to determine comprehension.
Reading
In embodiments, the system determines comprehension levels of a user by tracking the eyes during reading. The system has knowledge of the text being read, which is used to determine measures of comprehension. In embodiments, the measures are determined based on fixation durations on specific words, the number or frequency of regressive saccades (in an exemplary case, for English, leftward saccades on the same line/within the same sentence are determined; direction of regressive saccades may be different and measured accordingly for different languages) and fixation durations on specific sentence parts. The lower the durations of fixation and/or the greater the frequency of regressive saccades, the system determines that the greater is a user's comprehension. The system determines if the users are comprehending, or not comprehending, media content based upon the fixation durations. If the duration is above a predetermined threshold, then the system determines the users is facing difficulty in comprehending the media. If the durations are below a predetermined threshold, then the system determines the users are focused, and/or are comprehending the media.
It has also been shown that blink rates (f blink ) decrease during reading which also leads to an increase in variability in the time between blinks. In embodiments, the system determines a slowed blink rate relative to an established baseline (f blink significantly less than f blink ). The act of blinking slowly relative to a baseline may be determined by calculating a deviation from a mean. The greater the deviation from the mean, the system determines that greater is the comprehension of a user.
Electrophysiology
In embodiments, electrophysiological recordings yield measures of degree of comprehension within certain contexts. For example, using electroencephalographic event-related potentials (EEG-ERP) there may be seen increases in amplitudes of some cognitive potentials (N2, N400, P300, P600) for unexpected and/or incongruous words while reading. In embodiments, the system determines, in response to a significant amplitude magnitude increases in cognitive EEG potentials (N2, N44, P300, P600) resulting from infrequent, novel or unexpected stimuli, to be an indication of greater comprehension. In general, the system may conclude that the magnitude of amplitude changes compared to a baseline are proportional to a degree of comprehension. A positive change in amplitude may be attributed to greater comprehension while a lower or negative change may be attributed to lower ability to focus and/or comprehend.
The presence or absence of these amplitude changes, depending upon what's being read, aids the system to determine whether a user is correctly interpreting what they're reading. More generally, some EEG-ERP components can be used to measure cognitive load which may itself, within certain contexts, be an analog of comprehension. Galvanic skin response (GSR-ERP) can also be used, with increasing values indicating increasing cognitive load, and therefore greater comprehension. In embodiments, the system determines a significant increase in GSR-ERP as a signal of comprehension.
In embodiments, the system uses general electroencephalographic activity to measure comprehension. For example, increased activity in a beta/gamma frequency band (20-70 Hz) can be linked to various cognitive processes including associative learning. The system determines significant increases in energy of an EEG in beta and gamma frequency bands (≧16 Hz) to be a signal of increased comprehension.
Timing
Response times will generally be faster for correct responses when users know they are correct as compared to when they guessed correctly. Additionally, the system uses the relative timing of behavior to an introduction of key components of a scene or narrative, to measure comprehension. In embodiments, the system determines increases in comprehension when elements in a scene change from ambiguous to congruent, or a task/problem goes from unsolvable to solvable, at a given time. In embodiments, the system determines this information specifically in the moments after the new information is made available. The system may further select appropriate spans of time, based on the moments after the new information is made available, to analyze other measures described in various embodiments of the present specification.
Onset of Comprehension
The onset of comprehension can be revealed by state changes from various measures including those listed above for measuring the degree of comprehension. Measures of comprehension onset may not necessarily be as precise as reaction time data; instead of identifying the moment in time when comprehension begins, the following measures may indicate at what point in a sequence of stimulus and response events the user gains new understanding. For example, if relying on correct/incorrect responses the system uses the point in time when percentage of correct responses jumps from a low baseline to a higher level.
The system determines the onset of comprehension as the point in time where, when applicable, the percent of correct responses increases significantly. In an embodiment, onset of comprehension is determined using a t-test to compare percent correct responses from a second time frame relative to a first time frame or from the current N responses to the previous N responses, for a number N that is sufficiently large to be statistically significant.
Target Detection
In embodiments, the system determines an onset of comprehension when a user detect and/or selects a target where knowing a target's identity requires comprehension. Additionally, the system determines that the user is able to appropriately detect and/or select based on vocal or manual responses indicating identity of the target and/or pointing, gestures, facing or gaze directed at the target location. In embodiments, the system determines the onset of comprehension as the point when a target in a VR/AR/MxR media is correctly identified and or located. When applicable this should be at a rate or degree greater than that possible by chance. The system determines a rate of onset of comprehension greater that a specific threshold to be an indication of greater comprehension. On the other hand, the system may attribute a rate of onset of comprehension equal to or lower that the specific threshold to reduced or lack of comprehension. The system determines if the user is comprehending, or not comprehending, media content based upon the rate of onset of comprehension.
Fixation Duration
Initial sampling of a scene takes in all of the necessary information to solve a problem, but if users get stuck the duration of fixations increases as focus turns inward. Once a solution is discovered fixation durations drop as users resume normal scanning and verify their solution by checking the available information. Therefore, we can estimate the onset of comprehension by looking for a peak in fixation duration over time and finding any sudden decline thereafter. If the user is found to resume normal scanning that is associated with low fixation durations after an initial sampling that is associated with an increased fixation duration, the system determines such instances to be an indication of onset of comprehension. The instances are further combined with the information displayed in the corresponding VR/AR/MxR media, to determine onset of comprehension. If the instances are low or do not exist, then the system determines the user is not focused, is looking around randomly, and/or is not comprehending the media. If the instances exists and/or are high, then the system determines the user is focused, is not looking around randomly, and/or is comprehending the media. In embodiments the system determines the onset of comprehension as the end of a period of significantly longer fixation durations (D fixation ).
Pupil Dilation
In another embodiment, the system uses pupil dilation to detect the onset of comprehension. The magnitude and time to peak pupil dilation can be used to signal degree and onset of comprehension, respectively. The system may observe any significant dilation of the pupil, as compared to a few to several seconds prior, as a noteworthy event, in general. The system determines if the user is, at the onset, comprehending, or not comprehending, media content based upon the magnitude and time to peak pupil dilation.
In embodiments, the system determines a rapid and significant increase in pupil diameter (S pupi1 ) as the onset of comprehension. In embodiments, the context of the media rendered in the VR/AR/MxR environment is factored-in with the observations pertaining pupil dilation. An exemplary context is when the user is tasked with deriving novel meaning from a stimuli provided through the media.
Facial Cues
In embodiments, the system determines transient changes in facial expression as indications of the onset of states like confusion, worry, and concentration. In an exemplary embodiment, a user squinting is an indication of confusion, whereas releasing the squint is an indication that comprehension has occurred. In embodiments, the system determines a transition from partially to completely open eyes (significant increase in p both eyes open from a non-zero baseline) as the onset of comprehension in the appropriate contexts. The system determines if the user is comprehending, or not comprehending, media content based upon the transition from partially to completely open eyes.
Sudden Increase in Degree Measures
In various embodiments, the system uses one or a combination of the above-described measures to determine an onset of comprehension. The system samples the above-described measures for this purpose. In an example, if users go suddenly from a low, or baseline, degree of comprehension to a heightened degree of comprehension, the system can determine when comprehension began. In embodiments, the degree of comprehension is based on a combination of one or more of the measures described above to identify levels and onset of comprehension. Algorithmically this may be established by finding a time with the greatest, and also statistically significant, difference in degree of comprehension measures before and after. The system determines if the user is comprehending, or not comprehending, media content based upon the sudden differences in the degree measures. A sudden difference in a parameter is one which is greater than a predefined standard deviation of historical values for that parameter. When the a measure passes a predefined standard deviation, the parameter is deemed to have suddenly changed, thereby indicating a change in comprehension state (either increased or decreased) based on the value range of the parameter.
Failure or Lack of Comprehension
In embodiments, the system determines some measures that indicate a failure or general lack of comprehension. An absence of expected changes in degree of comprehension or any signs of comprehension onset may indicate a failure of comprehension. The system may identify some new behavior that is determined to signal the onset of a frustration or a search. In an embodiment, an increase in body temperature and/or heart rate may signal frustration during a comprehension task.
In an embodiment, the system determines a significant increase in body temperature and/or heart rate in association with delayed response to a question of understanding, as a signal of a lack of comprehension. The system determines if the user is comprehending, or not comprehending media content based upon a predefined increase in body temperature and/or heart rate in association with delayed response to a question of understanding, as a signal of lack of comprehension.
Eye gaze positions that seem uncorrelated with comprehension, particularly non-specific search characterized by brief fixation durations (D_fixation significantly less than _D_fixation) and saccades sampling the entire task space with large jumps (M_saccade significantly greater than _M_saccade) may indicate a desperate search for some missing clue. In embodiments, the system attributes such instances to lack of comprehension.
In embodiments, the system determines random or un-focused search characterized by significantly brief fixation durations and significantly large saccade magnitudes as indicative of a lack of comprehension in appropriate contexts. The system determines if the user is comprehending, or not comprehending media content based upon the random or un-focused search characterized by significantly brief fixation durations and significantly large saccade magnitudes.
Other Correlations
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous comprehension measures like making correct choices from among multiple alternatives. In some examples, some of the measures described above are context specific and may be more or less robust or even signal the opposite of what is expected. However the ability of users to respond correctly at rates better than expected by chance can be taken as a sign of comprehension and understanding. The system can correlate all available measures and look for trends in comprehension. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to increase its comprehensibility, for the user and/or a group of users.
At 1904 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 1906 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect reduced comprehension by the user of the VR, AR, and/or MxR media:
1. Decrease in palpebral fissure height 2. Increased blink rate 3. Increased rate of change for blink rate 4. Increased ratio of partial blinks to full blinks 5. Decreased target relevancy for pupil initial and final position 6. Decreased target relevancy for gaze direction 7. Decreased target relevancy for gaze initial and final position 8. Decreased target relevancy for fixation initial and final position 9. Increased fixation duration rate of change 10. Decreased target relevancy for saccade initial and final position 11. Decreased target relevancy for saccade angle 12. Increased ratio of anti-saccade/pro-saccade 13. Increased inhibition of return 14. Increased screen distance 15. Decreased target relevant head direction 16. Decreased target relevant head fixation 17. Decreased target relevant limb movement 18. Shift in weight distribution 19. Decreased alpha/delta brain wave ratio 20. Increased alpha/theta brain wave ratio 21. Increased body temperature 22. Increased respiration rate 23. Decrease in comprehension 24. Low oxygen saturation 25. Increased heart rate 26. Changes in blood pressure 27. Increased vocalizations 28. Increased reaction time

The system may determine a user has an increased degree of comprehension of the media in a VR, AR, and/or MX environment based upon the following changes:
1. Increased rate of change for pupil size 2. Increased rate of convergence 3. Increased rate of divergence 4. Increased fixation rate 5. Increased fixation count 6. Increased saccade velocity 7. Increased saccade rate of change 8. Increased saccade count (number of saccades) 9. Increased smooth pursuit

Additionally, the system may conclude that a user is experiencing increased or decreased comprehension of media based upon the following changes in combination with a specific type of user task. Accordingly, the system analyzes both the data types listed below together with the specific type of task being engaged in to determine whether the user is increasing or decreasing his or her level of comprehension.
1. Increased fixation duration 2. Decreased saccade magnitude (distance of saccade)

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; and change in olfactory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to improve comprehension, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 1908 , the changes in the plurality of data determined over time may be used to determine a degree of change in comprehension levels of the user. The change in comprehension levels may indicate either enhanced comprehension or reduced comprehension.
At 1910 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) comprehension. In embodiments, the media may be modified to address all the changes in data that reflect reduced comprehension. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm an improvement in comprehension: increase in palpebral fissure height; decrease in blink rate; decrease in the blink rate's rate of change; decreased ratio of partial blinks to full blinks; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased target relevancy for fixation initial and final position; increased fixation duration where task requires; decreased rate of change for fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; decreased alpha/theta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; targeted facial expressions; and targeted gustatory processing.
In embodiments, a specific percentage or a range of improvement in comprehension may be defined. In embodiments, an additional value for data may be acquired at 1912 , in order to further determine change in data over time at 1914 , after the modifications have been executed at 1910 . At 1916 , a new degree/percentage/range of improvement in comprehension may be acquired. At 1918 , the system determines whether the improvement in comprehension is within the specified range or percentage. If it is determine that the improvement is insufficient, the system may loop back to step 1910 to further modify the media. Therefore, the media may be iteratively modified 1910 and comprehension may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
Example Use 2: Modifying Media in Order to Decrease a User's Experience of Fatigue from Information in that Media
In one embodiment, a user's experience of fatigue and/or a severity of fatigue is measured by observing visible signs of fatigue as well as physiological measures that indicate fatigue. A user's degree of fatigue may also be inferred from the user's behavior, including subtle changes in behavior, and from the user's autonomic and electrophysiological measures. Depending on the information source, measures of fatigue may inform that the user is fatigued or that the user is becoming fatigued. Some behaviors like yawning, nodding off, or closed eyes and certain electrophysiological patterns can signal with little ambiguity that a user is fatigued, even at the beginning of a session of data recording. In embodiments, a baseline for comparison is determined that accounts for individual variability in many other measures, in order to conclude whether a person is becoming fatigued. For example, a user's reaction time may be slow for a number of reasons that have little or nothing to do with fatigue. In the example, an observation that user's reaction times are becoming slower over a window of time, may signal fatigue.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease measures of fatigue from information being communicated by that media. FIG. 20 illustrates a flow chart describing an exemplary process for modifying media in order to decrease fatigue, in accordance with some embodiments of the present specification. At 2002 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken into account, either individually or in combination.
To probe a user's measure of fatigue, the system applies measures of changes in comprehension, engagement or other derived states based on these measures detailed previously. An important tool in disambiguating the overlap of these states with general fatigue is a consideration of time. Time of day is expected to influence performance and the measures of fatigue, and is taken into consideration when measures from varying times of day is available. Duration of a session, or more generally, the duration of a particular behavior or performance of a given task, is also considered while measuring fatigue. Therefore, depending on time of day and duration of task, the probability that a measure will signal fatigue increases.
Some measures of fatigue may generally be classified as direct measures of fatigue, and the measures that indicate a transition to a state of fatigue (transitional measures of fatigue).
Direct measures of fatigue may be measured independent of baseline comparisons. In some embodiments, these are behaviors and measures typically associated with sleepiness or transitional states between wakefulness and sleep. Examples of direct measures may include visible signs of fatigue and physiological measures of fatigue.
Visible Signs
Visible signs of fatigue or sleepiness may largely be measured by imaging of the face and head. Video eye trackers may be used to obtain these measures, and EOG recording may also capture some behaviors.
Nodding of the head, notably with a slow downward and rapid upward pattern are likewise potential signals of fatigue. In an embodiment, head nodding with a slow downward movement, followed by rapid upward movement is considered indicative of fatigue.
Closed or partially closed eyes, especially for extended periods, can be yet another sign of fatigue. Prolonged periods of (mostly) closed eyes can be considered indicative of fatigue. For example, when the proportion of time that the eyes are at least 50% open is less than 75% (P eyes open (where P both eyes open ≧50%)<0.75), the system considers a user to be fatigued.
In embodiments, ocular fatigue is correlated with dry eyes. An abnormally low tear-break-up-time can be a signal of fatigue. In some embodiments, special imaging methods are used to measure ocular fatigue. The system considers significant signs of dry eye (such as low tear-break-up-time) as indicative of ocular fatigue.
In embodiments, yawning or other pronounced and discrete respiration is indicative of fatigue. Yawning or other isolated, deep inhalation of air can signal a fatigued state, and may be noted both for time and rate of occurrence.
One visible sign of transition to fatigue is determined through eye movements. In an embodiment, the system determines decrease in saccade velocity and magnitude, and decrease in frequency of fixations, to be a sign of slow eye movements, and therefore a sign of an onset of or increase in fatigue.
Also, in an embodiment, transitions to shorter and higher frequency of blinks is considered as an indication of fatigue onset. In this condition, user's eyes begin to close, partially or completely, and blinking goes from the normal pattern to a series of small, fast rhythmic blinks.
In another embodiment, sudden vertical eye movements is considered as indicative of fatigue.
A user transitioning to a state of fatigue may display a depth of gaze that drifts out towards infinity (zero convergence) and eye movements that may no longer track moving stimuli or may not respond to the appearance of stimuli. Therefore, in another embodiment, a 3-D depth of gaze towards infinity for extended periods is considered as indicative of fatigue.
Physiological Measures
Physiological sensors may be used to determine physiological indication of fatigue.
In an embodiment, significant decreases in heart rate and/or body temperature is associated with sleep, and is considered as indication of fatigue when the user displays these signs when awake.
In an embodiment, increased energy in low frequency EEG signals (for example, slow-wave sleep patterns) are interpreted as a signal of fatigue. For example, a trade-off where low frequency (<10 Hz) EEG energy increases and high frequency (≧10 Hz) EEG energy decreases is an indication of fatigue.
Transitions to fatigue may be determined from changes in behavior and other states over time, based on a significant deviation from an established baseline. Transitional measures of fatigue may be observed through visible signs as well as through behavioral measures.
Behavioral Measures
An increase in time taken by the user to react to stimuli may be considered indicative of fatigue. Additionally, measures of precision and timing in user responses to degrade, may be proportional to a level of fatigue.
In some embodiments, reductions in ‘performance’ metrics over extended periods of activity is considered as indicative of fatigue.
In some cases, user's vigilance decreases, leading to increasing lapse rates in responding to stimuli. In these cases, decreasing proportion of responding, in appropriate contexts, is considered as indicative of fatigue.
In one embodiment, the system determines significant reductions in comprehension, engagement and other excitatory states in the context of prolonged activity as signals of fatigue. In an embodiment, distractibility increases with decrease in comprehension and/or engagement, also signaling user's disengagement from the media experience. A prolonged duration of time may be defined based on the nature of the activity, but may generally range from tens of minutes to hours.
Other Correlations
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, physiological and/or visible measures, the less ambiguous measures of fatigue like sleepiness, and ability to characterize behavior after prolonged periods and at certain times of day. The system can correlate all available measures and look for trends in fatigue. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to reduce fatigue, for the user and/or a group of users.
At 2004 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2006 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increased level of fatigue experienced by the user of the VR, AR, and/or MxR media:
1. Decrease in palpebral fissure rate of change 2. Low distance palpebral fissure resting state 3. Low distance palpebral fissure active state 4. Increased ratio of partial blinks to full blinks 5. Decreased target relevancy for pupil initial and final position 6. Decreased target relevancy for gaze direction 7. Decreased target relevancy for gaze initial and final position 8. Increased rate of divergence 9. Decreased relevancy for fixation initial and final position 10. Increased fixation duration 11. Decreased target relevancy for saccade initial and final position 12. Decreased target relevancy for saccade angle 13. Decreased saccade magnitude (distance of saccade) 14. Increased ratio of anti-saccade/pro-saccade 15. Increased smooth pursuit 16. Increased screen distance 17. Decreased target relevant head direction 18. Decreased target relevant head fixation 19. Decreased target relevant limb movement 20. Decreased alpha/delta brain wave ratio 21. Low oxygen saturation 22. Changes in blood pressure 23. Increased reaction time

The system may determine a user is experiencing reduced levels of fatigue while interacting with the media in a VR, AR, and/or MxR environment based upon the following changes:
1. Increased blink rate 2. Increased rate of change for blink rate 3. Increased rate of change for pupil size 4. Increased rate of convergence 5. Increased fixation duration rate of change 6. Increased fixation rate 7. Increased fixation count 8. Increased inhibition of return 9. Increased saccade velocity 10. Increased saccade rate of change 11. Increased saccade count (number of saccades) 12. Shift in weight distribution 13. Increased alpha/theta brain wave ratio 14. Increased body temperature 15. Increased respiration rate 16. Increased heart rate 17. Increased vocalizations

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease fatigue, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2008 , the changes in the plurality of data determined over time may be used to determine a degree of change in levels of fatigue of the user. The change in fatigue levels may indicate either enhanced fatigue or reduced fatigue.
At 2010 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) fatigue. In embodiments, the media may be modified to address all the changes in data that reflect increased fatigue. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in fatigue levels: increase in palpebral fissure height; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation rate; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude; decreased ratio of anti-saccade/pro-saccade; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decreased shift in weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal blood pressure; target relevant facial expressions; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of increase in fatigue may be defined. In embodiments, an additional value for data may be acquired at 2012 , in order to further determine change in data over time at 2014 , after the modifications have been executed at 2010 . At 2016 , a new degree/percentage/range of increase in levels of fatigue may be acquired. At 2018 , the system determines whether the increase in levels of fatigue is within the specified range or percentage. If it is determine that the increase is greater that the specified range, the system may loop back to step 2010 to further modify the media. Therefore, the media may be iteratively modified 2010 and levels of fatigue may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 3: Modifying Media in Order to Improve a User's Engagement with Information in that Media
In one embodiment, a user's degree of engagement is derived from a number of measures. User's engagement may be determined as a binary state—whether or not the user is engaging. The binary engagement of the user with a particular application or task can be directly measured by their responding, or lack thereof, to events. Further, measures of comprehension and fatigue detailed above can be used to indicate such engagement as a prerequisite of comprehension and/or fatigue. Behavior oriented away from an application, device and/or task towards other things in the environment (i.e. engagement with something else) can also signal whether user is engaged with the application, device and/or task. A user's level of engagement can be derived by observing the user's focused attention, as opposed to divided attention, and measures of time-on-task. Users' behaviors that enhance their perception of stimuli may also indicate enhanced engagement. The behaviors that indicate an enhancement in perception of stimuli, may include leaning in, slowing blink rate, eye gaze vergence signaling focus at the appropriate depth of field, among others.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment, in order to improve that user's engagement with information being communicated by that media. FIG. 21 illustrates a flow chart describing an exemplary process for modifying media in order to improve engagement, in accordance with some embodiments of the present specification. At 2102 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken into account, either individually or in combination.
To probe a user's engagement with some spatially defined process in media, the system applies measures of where the user is looking, measuring the proportion of time the user spends looking at relevant areas within the media vs. irrelevant areas within the media.
In one embodiment, any measure signaling significant comprehension, or onset of comprehension, as is used to determine a signal of engagement with a task. Measures of comprehension can be used to indicate such engagement as a prerequisite of comprehension.
Engagement as a Binary State
Assigning a binary state, as a function of time, as to whether or not a user is engaged with an application may depend on less ambiguous cues. Measures (or cues) that indicate user engagement in a binary state may include the following:
1. Discrete, Conscious Responding
In an embodiment, a response rate of less than 100%, or another lower, baseline rate of responding, is determined to be an indication of an unengaged state. The response rate may depend on a context, such as but not limited to a situation where a brief duration of time is allowed for response that users may miss. In some embodiments, a user may be expected to respond through a manual interaction such as a mouse click or a screen touch. The system notes whether the user responded at any point in time when the user was expected to do so. If the user fails to respond then it is likely that the user is not engaged with the application.
In some embodiments, rapid changes in performance are noted by the system through a total percentage of correct responses provided by the users. The rapid changes in performance may signal engagement (with increase in rapid changes in performance) or disengagement (with decrease in rapid changes in performance). The system may exclude other causes for such performance changes, including but not limited to—little to no change in difficulty, and discounting learning/training effects for performance improvements. Therefore, a significant upward or downward deviation from average percent of correct responding is considered signaling engagement or disengagement, respectively. The system may determine if the user is engaging, not engaging, from media content in a VR/AR/MxR environment based upon presence or absence of responsive behavior.
2. Distraction
In an embodiment, distracted behavior can signal disengagement, or a shift of engagement away from one thing and towards another. Device inputs not related to the task at hand indicate onset and, potentially, duration of disengagement. Orienting of head or eyes away from an application, device or task likewise may indicate disengagement. Returning to the application, device or task may signal re-engagement. The system determines interactions away from a particular task or stimulus as indicating lack of engagement, or disengagement. The system determines disengagement due to distraction through measures of user attention based on body and eye tracking measures indicating that user is oriented away from the media. The system may determine if the user is engaging, not engaging, or disengaging, from media content in a VR/AR/MxR environment based upon distracted behavior.
Level of Engagement
A user's level of engagement further measures along a continuous dimension the degree to which users are engaged with an application, device or task to the exclusion of anything else. This relative measure may be normalized to the range of values recorded. In an example, fixation duration on the current item compared to the distribution of fixation durations is used as relative measure. Alternatively, in the presence of a distracting stimuli, the ratio of time or effort spent on one item versus another, is used to determine a level of engagement.
1. Time-on-Task
In one embodiment, user interactions are measured with more than one application or task. In embodiments, in such cases, the level of engagement with any application or task is taken as the ratio of time spent interacting with it compared to time spent not interacting with it. Therefore, the system may determine relative time-on-task as the proportion of time spent performing a task or processing a stimulus compared to the time not spent performing the task or processing the stimulus. Engagement is proportional to the value of relative time-on-tasks. The system determines greater engagement with tasks or applications where the user spends relatively greater time performing them. The system may determine if the user is engaging, not engaging, or disengaging, from media content in a VR/AR/MxR environment based upon relative time-on-tasks.
In an embodiment, users switch between applications or tasks, and responses are recorded for each application or task. In this case, the system uses the number and/or duration of interactions with each application/task to determine the level of engagement with them. Therefore, the system determines the ratio of interactions among available tasks as indicative of time-on-task for each as a relative measure of engagement with each task.
In an embodiment, the system performs eye tracking. In embodiments, the ratio of fixation count and/or duration between an application, device or task, and anything outside of it is used as a measure of level of engagement. Therefore, the system determines the ratio of fixation count and/or duration among stimuli and/or visual regions as indicative of time-on-task as a relative measure of engagement with each stimulus or visual region.
2. Enhancing Perception
Some behaviors allow users to better perceive stimuli and can signal their level of engagement with them. In some embodiments, face and/or body tracking is used by the system to measure when and the extent to which users lean towards a display or, while held, bring the display closer to their face. In an embodiment, the system determines significant shortening of the distance between a visual stimulus and the user's eyes as an indication of onset of engagement, and a proportional deviation from a baseline as an indication of level of engagement. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment based upon significant shortening of the distance between a visual stimulus and the user's eyes.
In an embodiment, tracking of gaze direction of both eyes is used to measure the extent that users are converging their gaze position at the appropriate depth to view stimuli (this is on top of whether gaze is in the appropriate direction). In an embodiment, the system determines adjustment of 3-D gaze position towards the appropriate depth (here considered separately from direction of gaze) to view a stimulus as a signal of engagement with that stimulus. The system may determine a level of user's engagement with stimuli in media content in a VR/AR/MxR environment based upon adjustment of 3-D gaze position towards the appropriate depth to view the stimuli.
In some embodiments, a relative measure of engagement is indicated when user maintains more rigid or steady fixation on a stimulus, which can aid in spotting subtle changes. In an embodiment, the system determines rigid fixation in the context of monitoring for subtle changes or motion, or the precise onset of any change or motion, as indicative of engagement. The system may determine a level of user's engagement with stimuli in media content in a VR/AR/MxR environment based upon rigid fixation in the context of monitoring for subtle changes or motion.
Greater sampling near and around a stimulus may indicate increasing engagement as a user studies the details of the stimulus. In an embodiment, the system determines a level of engagement based on an Area of Focus (also described in context of Tomprehension′), where the area of focus is correlated with the spatial extent of the stimulus in question. The system may determine a level of user's engagement with stimuli in media content in a VR/AR/MxR environment based upon the user's area of focus within the media.
In an embodiment, the system determines a blink rate that is significantly less than a baseline, as indicative of engagement with an ongoing task. Alternatively, the system determines given eye gaze position estimation of the user within a fixated region, to be an indication of a level of engagement with the ongoing task. A decrease in blink rate can indicate increasing level of engagement. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment based upon the user's blink rate and/or eye gaze position.
Sometimes a user may hold breathing to reduce body motion in order to focus on the media, for monitoring for subtle changes in stimuli within the media. In an embodiment, the system determines reduced or held respiration in the context of monitoring as indicative of engagement. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on user's breathing while monitoring for changes in a stimuli within the media.
3. Preference
A user may prefer some object(s) over the other when two or more alternatives are presented within a media. Even with only one object of interest, given appropriate sources of data, the system may determine following measures in the context of comparing the object of interest with everywhere else in the user's immediate environment, to derive a level of engagement.
In addition to generally looking more at objects for which users have preference, some other eye tracking measures can be used to estimate preference and, by extension, engagement. In an embodiment, the system predicts that, just before making a choice, a user's last fixation is on the item of choice. Therefore, the system determines that when a choice is made by a user, the duration of last fixation on the selected stimulus of choice, is defined as proportional to level of engagement of the user with that choice and with the selected task. The choice, in this case, is tied to a following selection, and eliminates cases of the instance of ‘choice’ that itself does not indicate a preference (for example, the user may choose to continue without an explicit selection). Alternatively, at the beginning of the decision making process, the duration of the first fixation is correlated with the ultimate selection. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on the duration of first fixation on any stimulus when a choice is made by the user, where the duration is determined to be proportional to level of engagement with the selected task.
In embodiments, broader patterns of eye gaze can reveal choices before they are made, and patterns of eye gaze can influence choice even when stimuli are not present. Also, in embodiments the system uses preference for features within objects to predict preference for novel objects with similar features.
In addition to gaze direction, other measures of preference/engagement may be made based on other eye tracking data. In an embodiment, the system determines pupil dilation will increase during decision making in favor of a task of choice, to measure user's level of engagement. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on the pupil dilation observed from the eye tracking data, while making a decision.
Another measure of preference/engagement may be derived from blinking. While it has been discussed that blinking is inhibited when the user is engaged with visual stimuli from very early in development, the system may also determine increased blinking, along with fewer fixations on task-relevant areas, to be associated with disengagement. The disengagement may also be measured by observing subsequent errors, post the significant increase in blinking. The system may determine a level of user's disengagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on a significant increase in blinking of the eyes.
In addition to event-related signals mentioned previously in the context of comprehension that may indicate attention to stimuli, the system may determine some more generalized measures that can indicate decision making and/or choice. Such measures can be assumed to be proportional to engagement in certain contexts. In an embodiment, the system determines increased bilateral phase synchrony of EEG activity during choice tasks as indicative of increased level of engagement with the task. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on electrophysiological measurements such as EEG.
In addition to EEG, other physiological and autonomic measures may be used by the system to determine a level of engagement. In an embodiment, the system determines an increased level of engagement to be proportional to an increase in heart rate. Similar changes in blood pressure, oxygen saturation, and respiration rate may be used by the system, along with changes in skin conductance (GSR). Therefore, in embodiments, the system determines an increase in autonomic arousal as indicative of increasing engagement, and decreases in arousal as disengagement. The system may determine a level of user's engagement with media content in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment based on changes in autonomic arousal.
Other Correlations
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous engagement signs like consistent interaction, proportionately high time-on-task, and perceptually enhancing behaviors. In some examples, some of the measures described above are context specific and may be more or less robust or even signal the opposite of what is expected. Measures with significant correlations with the less ambiguous signals of engagement may therefore become less ambiguous themselves and become new ways of identifying engagement. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to increase its engagement factor, for the user and/or a group of users.
At 2104 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2106 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increased engagement of the user with the VR, AR, and/or MxR media:
1. Decreased palpebral fissure rate of change 2. Increased rate of change for blink rate 3. Increased rate of change for pupil size 4. Increased rate of convergence 5. Increased rate of divergence 6. Increased fixation duration rate of change 7. Increased fixation rate 8. Increased fixation count 9. Increased inhibition of return 10. Increased saccade velocity 11. Increased saccade rate of change 12. Increased saccade count (number of saccades) 13. Increased smooth pursuit 14. Increased alpha/theta brain wave ratio

The system may determine a user has reduced level of engagement with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Low distance palpebral fissure resting state 2. Low distance palpebral fissure active state 3. Increased blink rate 4. Increased ratio of partial blinks to full blinks 5. Decreased target relevancy for pupil initial and final position 6. Decreased target relevancy for gaze direction 7. Decreased target relevancy for gaze initial and final position 8. Decreased relevancy for fixation initial and final position 9. Reduced fixation duration 10. Decreased target relevancy for saccade initial and final position 11. Decreased target relevancy for saccade angle 12. Decreased saccade magnitude (distance of saccade), depending on the task 13. Increased ratio of anti-saccade/pro-saccade 14. Increased screen distance 15. Decreased target relevant head direction 16. Decreased target relevant head fixation 17. Decreased target relevant limb movement 18. Shift in weight distribution 19. Decreased alpha/delta brain wave ratio 20. Increased body temperature 21. Increased respiration rate 22. Low oxygen saturation 23. Increased heart rate 24. Low blood pressure 25. Increased vocalizations

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to improve engagement, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2108 , the changes in the plurality of data determined over time may be used to determine a degree of change in engagement levels of the user. The change in engagement levels may indicate either enhanced engagement or reduced engagement.
At 2110 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) engagement. In embodiments, the media may be modified to address all the changes in data that reflect decrease in engagement. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm an improvement in comprehension: increase in palpebral fissure height; decrease in blink rate; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration depending on task; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude based on task; decreased ratio of anti-saccade/pro-saccade; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of improvement in engagement may be defined. In embodiments, an additional value for data may be acquired at 2112 , in order to further determine change in data over time at 2114 , after the modifications have been executed at 2110 . At 2116 , a new degree/percentage/range of improvement in engagement may be acquired. At 2118 , the system determines whether the improvement in engagement is within the specified range or percentage. If it is determine that the improvement is insufficient, the system may loop back to step 2110 to further modify the media. Therefore, the media may be iteratively modified 2110 and engagement may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 4: Modifying Media in Order to Improve a User's Overall Performance while Interacting with that Media
In an embodiment, user's overall performance is measured, and a media is modified in order to improve the performance of the user. The performance of a user may be determined in the form of user's vision performance, ability to comprehend, engagement levels, fatigue, and various other parameters, in combination, which directly or indirectly affect the overall performance of the user while interacting with the media including media in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment.
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, may be processed to determine the overall performance of the user. The data may indicate a level of performance of assessed during user's interaction with the media. The data may be further utilized to modify VR/AR/MxR media for the user in order to optimize overall performance, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media or conventional laptop, mobile phone, desktop or tablet computing media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to improve that user's overall performance while interacting with that media. FIG. 22 illustrates a flow chart describing an exemplary process for modifying media in order to improve overall performance, in accordance with some embodiments of the present specification. At 2202 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids.
Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken into account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous performance measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in overall performance. Accordingly, the media presented in a VR/AR/MxR environment or a conventional laptop, mobile phone, desktop or tablet computing environment is modified, in order to improve or optimize performance, for the user and/or a group of users.
At 2204 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2206 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect overall improvement of the user's performance while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Increased Rate of Change for Pupil Size 3. Increased Rate of Convergence 4. Increased Rate of Divergence 5. Increased Fixation Duration Rate of Change 6. Increased Fixation Rate 7. Increased Fixation Count 8. Increased Saccade Velocity 9. Increased Saccade Count (Number of Saccades) 10. Increased Smooth Pursuit

The system may determine a user has a reduced overall performance levels while interacting with the media in a VR, AR, and/or MxR environment based upon one or more of the following changes:
1. Low Distance Palpebral Fissure Resting State 2. Low Distance Palpebral Fissure Active State 3. Increased Blink Rate 4. Increased Rate of Change for Blink Rate 5. Increased Ratio of Partial Blinks to Full Blinks 6. Decreased Target Relevancy for Pupil Initial and Final Position 7. Decreased Target Relevancy for Gaze Direction 8. Decreased Target Relevancy for Gaze Initial and Final Position 9. Decreased Relevancy for Fixation Initial and Final Position 10. Changes in Fixation Duration, based on the context 11. Decreased Target Relevancy for Saccade Initial and Final Position 12. Decreased Target Relevancy for Saccade Angle 13. Decreased Saccade Magnitude (Distance of Saccade) 14. Increased Ratio of Anti-Saccade/Pro-Saccade 15. Increased Inhibition of Return 16. Increased Saccade Rate of Change 17. Increased Screen Distance 18. Decreased Target Relevant Head Direction 19. Decreased Target Relevant Head Fixation 20. Decreased Target Relevant Limb Movement 21. Shift in Weight Distribution 22. Decreased Alpha/Delta Brain Wave ratio 23. Increased Alpha/Theta Brain Wave ratio 24. Increased Body Temperature 25. Increased Respiration Rate 26. Low Oxygen Saturation 27. Increased Heart Rate 28. Low Blood Pressure 29. Increased Vocalizations 30. Increased Reaction Time

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; and change in olfactory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to improve overall user performance, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2208 , the changes in the plurality of data determined over time may be used to determine a degree of change in overall performance levels of the user. The change in overall performance levels may indicate either enhanced performance or reduced performance.
At 2210 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) performance. In embodiments, the media may be modified to address all the changes in data that reflect reduced performance. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm an improvement in comprehension increase in palpebral fissure height; decrease in blink rate; decreased rate of change for blink rate; increased target relevant pupil initial and final position; increased target relevant gaze direction; increased target relevant gaze initial and final position; increased target relevant fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade rate of change; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of improvement in overall performance may be defined. In embodiments, an additional value for data may be acquired at 2212 , in order to further determine change in data over time at 2214 , after the modifications have been executed at 2210 . At 2216 , a new degree/percentage/range of improvement in overall performance may be acquired. At 2218 , the system determines whether the improvement in overall performance is within the specified range or percentage. If it is determine that the improvement is insufficient, the system may loop back to step 2210 to further modify the media. Therefore, the media may be iteratively modified 2210 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 5: Modifying Media in Order to Decrease Symptoms Associated with Visually-Induced Motion Sickness Secondary to Visual-Vestibular Mismatch
In an embodiment, user's symptoms of Visually-Induced Motion Sickness (VIMS) secondary to visual-vestibular mismatch, is measured, and media is modified in order to decrease the symptoms for the user.
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, may be processed to determine the symptoms of VIMS, experienced by the user. The data may indicate a level of symptoms shown during or after user's interaction with the media. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the VIMS symptoms, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease user's symptoms of VIMS secondary to visual-vestibular mismatch, during or after interaction with that media. FIG. 23 illustrates a flow chart describing an exemplary process for modifying media in order to decrease symptoms of VIMS secondary to visual-vestibular mismatch, in accordance with some embodiments of the present specification. At 2302 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken into account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous symptomatic measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in overall display of VIMS symptoms. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease the VIMS symptoms secondary to visual-vestibular mismatch, for the user and/or a group of users.
At 2304 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2306 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in VIMS symptoms while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Ratio of Partial Blinks to Full Blinks 5. Decreased Target Relevancy for Pupil Initial and Final Position 6. Decreased Target Relevancy for Gaze Direction 7. Decreased Target Relevancy for Gaze Initial and Final Position 8. Decreased Relevancy for Fixation Initial and Final Position 9. Changes in Fixation Duration or Increased Convergence Duration 10. Decreased Target Relevancy for Saccade Angle 11. Decreased Saccade Magnitude (Distance of Saccade) 12. Increased Ratio of Anti-Saccade/Pro-Saccade 13. Increased Inhibition of Return 14. Increased Smooth Pursuit 15. Increased Screen Distance 16. Decreased Target Relevant Head Direction 17. Decreased Target Relevant Head Fixation 18. Decreased Target Relevant Limb Movement 19. Shift in Weight Distribution 20. Decreased Alpha/Delta Brain Wave ratio 21. Increased Body Temperature 22. Increased Respiration Rate 23. Low Oxygen Saturation 24. Increased Heart Rate 25. Changes in Blood Pressure 26. Decreased Reaction Time

The system may determine decrease in VIMS symptoms for a user while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Blink Rate 2. Increased Rate of Change for Blink Rate 3. Increased Rate of Change for Pupil Size 4. Increased Rate of Convergence 5. Increased Rate of Divergence 6. Increased Fixation Duration Rate of Change 7. Increased Fixation Rate 8. Increased Fixation Count 9. Decreased Target Relevancy for Saccade Initial and Final Position 10. Increased Saccade Velocity 11. Increased Saccade Rate of Change 12. Increased Saccade Count (Number of Saccades) 13. Increased Alpha/Theta Brain Wave ratio 14. Increased Vocalizations

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease user's symptoms of VIMS, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2308 , the changes in the plurality of data determined over time may be used to determine a degree of change in user's VIMS symptoms. The change in VIMS symptoms may indicate either reduced symptoms or enhanced symptoms.
At 2310 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) symptoms. In embodiments, the media may be modified to address all the changes in data that reflect increase in VIMS symptoms. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in VIMS symptoms: increased palpebral fissure height; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch, may be defined. In embodiments, an additional value for data may be acquired at 2312 , in order to further determine change in data over time at 2314 , after the modifications have been executed at 2310 . At 2316 , a new degree/percentage/range of decrease in symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch may be acquired. At 2318 , the system determines whether the decrease in symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch is within the specified range or percentage. If it is determine that the decrease is insufficient, the system may loop back to step 2310 to further modify the media. Therefore, the media may be iteratively modified 2310 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 7: Modifying Media in Order to Decrease Symptoms Associated with Post Traumatic Stress Disorder (PTSD)
Post-Traumatic Stress Disorder (PTSD) is a condition that is developed by an individual after they are exposed to a traumatic event. In embodiments of the present specification, the SDEP allows for collection of biometric information from hardware and software sources, utilizes machine and deep learning techniques, combined with image processing and machine learning to understand how multiple sensory and physiologic inputs and outputs affect both visual and human behavior, in conditions such as PTSD. Through these learnings, one may understand a person at a deeply intimate neurophysiological state. The learnings may be utilized to modify media in order to address a user's symptoms associated with PTSD. In further embodiments, the learnings are used to modulate light stimuli through HMDs in order to allow/enable perceiving information by the human body through neurophysiologic+/−electronic stimulation, such as through neuro-programming. In embodiments of the present specification, the communication is modulated through neurophysiologic+/−electronic stimulation, through the use of direct, reflected, diffracted, refracted light/sound, with both amplitude, depth, area, and frequency. Additionally, the communication is modulated through neurophysiologic+/−electronic stimulation+/−chemical stimulation, through the use of direct, indirect touch/taste/smell, with both amplitude, frequency, depth and area.
In embodiments of the present specification, the direction, location, amplitude, frequency, depth, pattern, and combination of these light fields, based on the same principles as stated with bio-electronic implants, allow for stimulation of certain visual and accessor visual channels of the retino-geniculo-cortical system, allowing in part for activation and encoding of different aspects of vision, including but not limited to stereopsis (depth), color, contrast, size discrimination, object/face recognition, border detection, oculomotor function, pupillary function, field of view, visual memory, for neuroplasticity by bypassing injured channels in favor of intact channels and/or developing new channels, and for neuro-programming for therapeutic approaches for the neural, cardiac, auditory, olfactory, tactile, gustatory, muscular, endocrine (hormone regulation—e.g. retinal ganglion cell subtype stimulation for circadian rhythm reset), metabolic, immune, psychology/psychiatric systems.
Embodiments of the present specification are also applicable to different types of vision implants, such as and not limited to PRIMA, IRIS I and IRIS II, and other vision implants that may be fixed under or outside the retina.
In an embodiment, an individual with PTSD interfaces with the system to understand how multiple sensory and physiologic inputs and outputs affect both visual and human behavior of the individual. In the example, SDEP database may be utilized to develop a benchmark for PTSD, including increased saccadic eye movements, pupillary dilation, color sensitivity to longer wavelengths of color/red RGB, increased heart rate, increased basal body temperature, auditory sensitivity to elevated intensity levels in binaural states, and increased sensitivity to patterns in images/videos/scenes with high RGB, decreased background/foreground luminance with multiple object recognition requirements. In an example, increase in anti-saccadic error with minimal pupillary reactivity and increased sensitivity to blue light (RGB of 0,0,1), with increased heart rate >100 bpm, and basal body temperature greater than 98.6 degrees F., may be benchmarked as PTSD related anxiety. Based on the data, presentation of therapeutics through the use of titrated visual and non-visual stimuli through the SDEP, real-time dynamic exchange of stimuli (RDES) can stimulate the neurophysiologic retina via communication with a bio-electronic implant, via communication through an HMD.
The therapeutic effect of such stimulation may be measured against the benchmark for decreased/normalization of saccadic eye movements, pupil reaction, color sensitivity, heart rate, basal body temperature, auditory sensitivity, and image sensitivity.
In an embodiment, user's symptoms of PTSD, is measured, and media is modified in order to decrease the symptoms for the user.
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, may be processed to determine the symptoms of PTSD, experienced by the user. The data may indicate a level of symptoms shown during or after user's interaction with the media. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the PTSD symptoms, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease user's symptoms of PTSD, during interaction with that media. FIG. 24 illustrates a flow chart describing an exemplary process for modifying media in order to decrease symptoms of PTSD, in accordance with some embodiments of the present specification. At 2402 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous symptomatic measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in overall display of PTSD symptoms. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease the PTSD symptoms, for the user and/or a group of users.
At 2404 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2406 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in occurrences of PTSD symptoms while interacting with the VR, AR, and/or MxR media:
1. Increased Blink Rate 2. Increased Rate of Change for Blink Rate 3. Increased Ratio of Partial Blinks to Full Blinks 4. Increased Rate of Change for Pupil Size 5. Decreased Target Relevancy for Pupil Initial and Final Position 6. Decreased Target Relevancy for Gaze Direction 7. Decreased Target Relevancy for Gaze Initial and Final Position 8. Decreased Relevancy for Fixation Initial and Final Position 9. Increased Fixation Duration Rate of Change 10. Decreased Target Relevancy for Saccade Initial and Final Position 11. Decreased Target Relevancy for Saccade Angle 12. Decreased Saccade Magnitude (Distance of Saccade) 13. Increased Ratio of Anti-Saccade/Pro-Saccade 14. Increased Saccade Velocity 15. Increased Saccade Rate of Change 16. Increased Saccade Count (Number of Saccades) 17. Increased Screen Distance 18. Decreased Target Relevant Head Direction 19. Decreased Target Relevant Head Fixation 20. Decreased Target Relevant Limb Movement 21. Shift in Weight Distribution 22. Increased Alpha/Theta Brain Wave ratio 23. Increased Body Temperature 24. Increased Respiration Rate 25. Increased Heart Rate 26. Increased Vocalizations 27. Increased Reaction Time

The system may determine decrease in occurrences of PTSD symptoms for a user while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Rate of Convergence 5. Increased Rate of Divergence 6. Increased Fixation Duration 7. Increased Fixation Rate 8. Increased Fixation Count 9. Increased Smooth Pursuit 10. Decreased Alpha/Delta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased inhibition of return; low oxygen saturation; low blood pressure; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease occurrences of user's symptoms of PTSD, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2408 , the changes in the plurality of data determined over time may be used to determine a degree of change in occurrences of user's PTSD symptoms. The change in occurrences of user's PTSD symptoms may indicate either reduced occurrences or enhanced occurrences.
At 2410 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) occurrences of PTSD symptoms. In embodiments, the media may be modified to address all the changes in data that reflect increase in occurrences of PTSD symptoms. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in PTSD symptoms: increased palpebral fissure height; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in symptoms associated with PTSD, may be defined. In embodiments, an additional value for data may be acquired at 2412 , in order to further determine change in data over time at 2414 , after the modifications have been executed at 2410 . At 2416 , a new degree/percentage/range of decrease in symptoms associated with PTSD may be acquired. At 2418 , the system determines whether the decrease in symptoms associated with PTSD is within the specified range or percentage. If it is determine that the decrease is insufficient, the system may loop back to step 2410 to further modify the media. Therefore, the media may be iteratively modified 2410 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 8: Modifying Media in Order to Decrease Double Vision Related to Accommodative Dysfunction
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine an extent of double vision related to accommodative dysfunction, experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the double vision related to accommodative dysfunction, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease user's double vision related to accommodative dysfunction, during interaction with that media. FIG. 25 illustrates a flow chart describing an exemplary process for modifying media in order to decrease user's double vision, in accordance with some embodiments of the present specification. At 2502 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous symptomatic measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in user's experience of double vision related to accommodative dysfunction. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease user's double vision, for the user and/or a group of users.
At 2504 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2506 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in double vision while interacting with the VR, AR, and/or MxR media:
1. Increased Blink Rate 2. Increased Rate of Change for Blink Rate 3. Increased Ratio of Partial Blinks to Full Blinks 4. Increased Rate of Change for Pupil Size 5. Decreased Target Relevancy for Pupil Initial and Final Position 6. Decreased Target Relevancy for Gaze Direction 7. Decreased Target Relevancy for Gaze Initial and Final Position 8. Increased Rate of Convergence 9. Decreased Relevancy for Fixation Initial and Final Position 10. Increased Fixation Duration 11. Increased Fixation Duration Rate of Change 12. Increased Fixation Rate 13. Increased Fixation Count 14. Decreased Target Relevancy for Saccade Initial and Final Position 15. Decreased Target Relevancy for Saccade Angle 16. Decreased Saccade Magnitude (Distance of Saccade) 17. Increased Ratio of Anti-Saccade/Pro-Saccade 18. Increased Inhibition of Return 19. Increased Saccade Velocity 20. Increased Saccade Rate of Change 21. Increased Saccade Count (Number of Saccades) 22. Decreased Target Relevant Head Direction 23. Decreased Target Relevant Head Fixation 24. Decreased Target Relevant Limb Movement 25. Shift in Weight Distribution 26. Decreased Alpha/Delta Brain Wave ratio 27. Increased Alpha/Theta Brain Wave ratio 28. Increased Body Temperature 29. Increased Respiration Rate 30. Low Oxygen Saturation 31. Increased Heart Rate 32. Low Blood Pressure 33. Increased Reaction Time

The system may determine decrease in double vision for a user while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Rate of Divergence 5. Increased Smooth Pursuit 6. Increased Screen Distance

It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease double vision, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2508 , the changes in the plurality of data determined over time may be used to determine a degree of change in user's double vision. The change in double vision may indicate either reduced double vision or enhanced double vision, related to accommodative dysfunction.
At 2510 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) double vision. In embodiments, the media may be modified to address all the changes in data that reflect increase in double vision related to accommodative dysfunction. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in double vision: decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of convergence; increased relevancy for fixation initial and final position; decreased fixation duration; decreased fixation duration rate of change; decreased fixation rate; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; ocular re-alignment or improvement in alignment and coordinated ocular motility; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in symptoms associated with double vision, may be defined. In embodiments, an additional value for data may be acquired at 2512 , in order to further determine change in data over time at 2514 , after the modifications have been executed at 2510 . At 2516 , a new degree/percentage/range of decrease in symptoms associated with double vision may be acquired. At 2518 , the system determines whether the decrease in double vision related to accommodative dysfunction is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 2510 to further modify the media. Therefore, the media may be iteratively modified 2510 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 9: Modifying Media in Order to Decrease Vection Due to Unintended Peripheral Field Stimulation
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine an extent of vection due to unintended peripheral field stimulation, experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the vection, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease user's experience of vection due to unintended peripheral field stimulation, during interaction with that media. FIG. 26 illustrates a flow chart describing an exemplary process for modifying media in order to decrease vection, in accordance with some embodiments of the present specification. At 2602 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in user's experience of vection due to unintended peripheral field vision. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease user's vection, for the user and/or a group of users.
At 2604 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2606 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in vection while interacting with the VR, AR, and/or MxR media:
1. Increased Rate of Change for Blink Rate 2. Increased Ratio of Partial Blinks to Full Blinks 3. Increased Rate of Change for Pupil Size 4. Decreased Target Relevancy for Pupil Initial and Final Position 5. Decreased Target Relevancy for Gaze Direction 6. Decreased Target Relevancy for Gaze Initial and Final Position 7. Increased Rate of Convergence 8. Decreased Relevancy for Fixation Initial and Final Position 9. Increased Fixation Duration 10. Increased Fixation Duration Rate of Change 11. Decreased Target Relevancy for Saccade Initial and Final Position 12. Decreased Target Relevancy for Saccade Angle 13. Decreased Saccade Magnitude (Distance of Saccade) 14. Increased Ratio of Anti-Saccade/Pro-Saccade 15. Increased Inhibition of Return 16. Increased Saccade Velocity 17. Increased Saccade Rate of Change 18. Increased Smooth Pursuit 19. Decreased Target Relevant Head Direction 20. Decreased Target Relevant Head Fixation 21. Decreased Target Relevant Limb Movement 22. Shift in Weight Distribution 23. Low Oxygen Saturation 24. Increased Heart Rate 25. Low Blood Pressure 26. Increased Reaction Time

The system may determine decrease in vection for a user while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Divergence 6. Increased Fixation Rate 7. Increased Fixation Count 8. Increased Saccade Count (Number of Saccades) 9. Increased Screen Distance 10. Decreased Alpha/Delta Brain Wave ratio 11. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased body temperature; increased respiration rate; increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease vection, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2608 , the changes in the plurality of data determined over time may be used to determine a degree of change in user's vection. The change in vection may indicate either reduced vection or enhanced vection, due to unintended peripheral field vision.
At 2610 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) vection. In embodiments, the media may be modified to address all the changes in data that reflect increase in vection due to unintended peripheral field vision. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in vection: decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of convergence; increased relevancy for fixation initial and final position; decreased fixation duration; decreased fixation duration rate of change; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased smooth pursuit; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in vection due to unintended peripheral field stimulation, may be defined. In embodiments, an additional value for data may be acquired at 2612 , in order to further determine change in data over time at 2614 , after the modifications have been executed at 2610 . At 2616 , a new degree/percentage/range of decrease in vection due to unintended peripheral field stimulation may be acquired. At 2618 , the system determines whether the decrease in vection is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 2610 to further modify the media. Therefore, the media may be iteratively modified 2610 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 10: Modifying Media in Order to Decrease Hormonal Dysregulation Arising from Excessive Blue Light Exposure
Blue light exposure has been shown to impact health. Elongated exposure to the waves transmitted through screen devices can disrupt circadian rhythm and impact health in various ways, including an impact on the hormones. The effect of blue light is believed to cause a decrease in the bodies' production of melatonin. Prolonged exposure to blue light is also believed to negatively impact ocular health.
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine an extent of hormonal dysregulation arising from excessive blue light exposure, experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the hormonal dysregulation, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease user's hormonal dysregulation arising from excessive blue light exposure, during interaction with that media. FIG. 27 illustrates a flow chart describing an exemplary process for modifying media in order to decrease hormonal dysregulation, in accordance with some embodiments of the present specification. At 2702 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in user's hormonal dysregulation arising from excessive blue light exposure. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease hormonal dysregulation, for the user and/or a group of users.
At 2704 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2706 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in hormonal dysregulation while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Ratio of Partial Blinks to Full Blinks 5. Decreased Target Relevancy for Pupil Initial and Final Position 6. Decreased Target Relevancy for Gaze Direction 7. Decreased Target Relevancy for Gaze Initial and Final Position 8. Increased Rate of Divergence 9. Decreased Relevancy for Fixation Initial and Final Position 10. Increased Fixation Duration 11. Decreased Target Relevancy for Saccade Initial and Final Position 12. Decreased Target Relevancy for Saccade Angle 13. Decreased Saccade Magnitude (Distance of Saccade) 14. Increased Ratio of Anti-Saccade/Pro-Saccade 15. Increased Inhibition of Return 16. Increased Smooth Pursuit 17. Increased Screen Distance 18. Decreased Target Relevant Head Direction 19. Decreased Target Relevant Head Fixation 20. Decreased Target Relevant Limb Movement 21. Shift in Weight Distribution 22. Decreased Alpha/Delta Brain Wave ratio 23. Increased Body Temperature 24. Increased Respiration Rate 25. Low Oxygen Saturation 26. Increased Heart Rate 27. Low Blood Pressure 28. Increased Reaction Time

The system may determine decrease in hormonal dysregulation for a user while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Blink Rate 2. Increased Rate of Change for Blink Rate 3. Increased Rate of Change for Pupil Size 4. Increased Rate of Convergence 5. Increased Fixation Duration Rate of Change 6. Increased Fixation Rate 7. Increased Fixation Count 8. Increased Saccade Velocity 9. Increased Saccade Rate of Change 10. Increased Saccade Count (Number of Saccades) 11. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; and change in olfactory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease hormonal dysregulation, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2708 , the changes in the plurality of data determined over time may be used to determine a degree of change in user's hormonal dysregulation. The change in hormonal dysregulation may indicate either reduced hormonal dysregulation or enhanced hormonal dysregulation, arising from excessive blue light exposure.
At 2710 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) hormonal dysregulation. In embodiments, the media may be modified to address all the changes in data that reflect increase in hormonal dysregulation arising from excessive blue light exposure. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in hormonal dysregulation: increased palpebral fissure height; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in hormonal dysregulation arising from excessive blue light exposure, may be defined. In embodiments, an additional value for data may be acquired at 2712 , in order to further determine change in data over time at 2714 , after the modifications have been executed at 2710 . At 2716 , a new degree/percentage/range of decrease in hormonal dysregulation arising from excessive blue light exposure may be acquired. At 2718 , the system determines whether the decrease in hormonal dysregulation is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 2710 to further modify the media. Therefore, the media may be iteratively modified 2710 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 11: Modifying Media in Order to Decrease Potential Photo-Toxicity from Over-Exposure to Screen Displays
Prolonged exposure to screen displays is believed to increase the potential for phototoxicity for a user. In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the potential of phototoxicity, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the potential of phototoxicity, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease potential phototoxicity from over-exposure to screen displays, during interaction with that media. FIG. 28 illustrates a flow chart describing an exemplary process for modifying media in order to decrease the potential for phototoxicity, in accordance with some embodiments of the present specification. At 2802 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in phototoxicity from over-exposure to screen displays. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease phototoxicity, for the user and/or a group of users.
At 2804 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2806 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in phototoxicity while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Divergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration 14. Increased Fixation Duration Rate of Change 15. Decreased Target Relevancy for Saccade Initial and Final Position 16. Decreased Target Relevancy for Saccade Angle 17. Decreased Saccade Magnitude (Distance of Saccade) 18. Increased Ratio of Anti-Saccade/Pro-Saccade 19. Increased Inhibition of Return 20. Increased Saccade Velocity 21. Increased Saccade Rate of Change 22. Increased Smooth Pursuit 23. Increased Screen Distance 24. Decreased Target Relevant Head Direction 25. Decreased Target Relevant Head Fixation 26. Decreased Target Relevant Limb Movement 27. Shift in Weight Distribution 28. Increased Alpha/Theta Brain Wave ratio 29. Increased Body Temperature 30. Increased Respiration Rate 31. Increased Heart Rate 32. Low Blood Pressure 33. Increased Reaction Time

The system may determine decrease in phototoxicity while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Increased Fixation Rate 3. Increased Fixation Count 4. Increased Saccade Count (Number of Saccades) 5. Decreased Alpha/Delta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: low oxygen saturation; increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease phototoxicity, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2808 , the changes in the plurality of data determined over time may be used to determine a degree of change in phototoxicity. The change in phototoxicity may indicate either reduced phototoxicity or enhanced phototoxicity, from over-exposure to screen displays.
At 2810 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) phototoxicity. In embodiments, the media may be modified to address all the changes in data that reflect increase in phototoxicity from over-exposure to screen displays. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in phototoxicity: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration; decreased fixation duration rate of change; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in phototoxicity from over-exposure to screen displays, may be defined. In embodiments, an additional value for data may be acquired at 2812 , in order to further determine change in data over time at 2814 , after the modifications have been executed at 2810 . At 2816 , a new degree/percentage/range of decrease in phototoxicity from over-exposure to screen displays may be acquired. At 2818 , the system determines whether the decrease in phototoxicity is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 2810 to further modify the media. Therefore, the media may be iteratively modified 2810 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 12: Modifying Media in Order to Decrease Nausea and/or Stomach Discomfort
Prolonged exposure to screen displays may result in nausea and/or stomach discomfort. In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of nausea and/or stomach discomfort, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the nausea and/or stomach discomfort, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease nausea and/or stomach discomfort, during interaction with that media. FIG. 29 illustrates a flow chart describing an exemplary process for modifying media in order to decrease nausea and/or stomach discomfort, in accordance with some embodiments of the present specification. At 2902 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in nausea and/or stomach discomfort. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease nausea and/or stomach discomfort, for the user and/or a group of users.
At 2904 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 2906 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in nausea and/or stomach discomfort while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Ratio of Partial Blinks to Full Blinks 5. Decreased Target Relevancy for Pupil Initial and Final Position 6. Decreased Target Relevancy for Gaze Direction 7. Decreased Target Relevancy for Gaze Initial and Final Position 8. Increased Rate of Divergence 9. Decreased Relevancy for Fixation Initial and Final Position 10. Increased Fixation Duration 11. Decreased Target Relevancy for Saccade Initial and Final Position 12. Decreased Target Relevancy for Saccade Angle 13. Decreased Saccade Magnitude (Distance of Saccade) 14. Increased Ratio of Anti-Saccade/Pro-Saccade 15. Increased Inhibition of Return 16. Increased Smooth Pursuit 17. Increased Screen Distance 18. Decreased Target Relevant Head Direction 19. Decreased Target Relevant Head Fixation 20. Decreased Target Relevant Limb Movement 21. Shift in Weight Distribution 22. Increased Body Temperature 23. Increased Respiration Rate 24. Low Oxygen Saturation 25. Increased Heart Rate 26. Low Blood Pressure 27. Increased Vocalizations 28. Increased Reaction Time

The system may determine decrease in nausea and/or stomach discomfort while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Blink Rate 2. Increased Rate of Change for Blink Rate 3. Increased Rate of Change for Pupil Size 4. Increased Rate of Convergence 5. Increased Fixation Duration Rate of Change 6. Increased Fixation Rate 7. Increased Fixation Count 8. Increased Saccade Velocity 9. Increased Saccade Rate of Change 10. Increased Saccade Count (Number of Saccades) 11. Decreased Alpha/Delta Brain Wave ratio 12. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease nausea and/or stomach discomfort, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 2908 , the changes in the plurality of data determined over time may be used to determine a degree of change in nausea and/or stomach discomfort. The change in nausea and/or stomach discomfort may indicate either reduced nausea and/or stomach discomfort or enhanced nausea and/or stomach discomfort.
At 2910 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) nausea and/or stomach discomfort. In embodiments, the media may be modified to address all the changes in data that reflect increase in nausea and/or stomach discomfort. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in nausea and/or stomach discomfort: increased palpebral fissure height; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in nausea and/or stomach discomfort, may be defined. In embodiments, an additional value for data may be acquired at 2912 , in order to further determine change in data over time at 2914 , after the modifications have been executed at 2910 . At 2916 , a new degree/percentage/range of decrease in nausea and/or stomach discomfort may be acquired. At 2918 , the system determines whether the decrease in nausea and/or stomach discomfort is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 2910 to further modify the media. Therefore, the media may be iteratively modified 2910 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 13: Modifying Media in Order to Decrease Visual Discomfort
Prolonged exposure to screen displays may result in visual discomfort, including at least one of eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, or aching around the eyes. In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of visual discomfort, including at least one of eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, or aching around the eyes, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the visual discomfort, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease visual discomfort, during interaction with that media. FIG. 30 illustrates a flow chart describing an exemplary process for modifying media in order to decrease the visual discomfort, in accordance with some embodiments of the present specification. At 3002 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in visual discomfort. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease visual discomfort, for the user and/or a group of users.
At 3004 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3006 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in visual discomfort while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Divergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration 14. Increased Fixation Duration Rate of Change 15. Increased Fixation Rate 16. Increased Fixation Count 17. Decreased Target Relevancy for Saccade Initial and Final Position 18. Decreased Target Relevancy for Saccade Angle 19. Decreased Saccade Magnitude (Distance of Saccade) 20. Increased Ratio of Anti-Saccade/Pro-Saccade 21. Increased Inhibition of Return 22. Increased Saccade Velocity 23. Increased Saccade Rate of Change 24. Increased Saccade Count (Number of Saccades) 25. Increased Smooth Pursuit 26. Increased Screen Distance 27. Decreased Target Relevant Head Direction 28. Decreased Target Relevant Head Fixation 29. Decreased Target Relevant Limb Movement 30. Shift in Weight Distribution 31. Increased Body Temperature 32. Increased Respiration Rate 33. Low Oxygen Saturation 34. Increased Heart Rate 35. Low Blood Pressure 36. Increased Vocalizations 37. Increased Reaction Time

The system may determine decrease in visual discomfort while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Decreased Alpha/Delta Brain Wave ratio 3. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease visual discomfort, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3008 , the changes in the plurality of data determined over time may be used to determine a degree of change in visual discomfort. The change in visual discomfort may indicate either reduced visual discomfort or enhanced visual discomfort.
At 3010 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) visual discomfort. In embodiments, the media may be modified to address all the changes in data that reflect increase in visual discomfort. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in visual discomfort: increased palpebral fissure rate of change; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; increased target relevancy for saccade initial and final position; decreased fixation duration; decreased fixation duration rate of change; decreased fixation rate; decreased fixation count; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased smooth pursuit; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in visual discomfort, may be defined. In embodiments, an additional value for data may be acquired at 3012 , in order to further determine change in data over time at 3014 , after the modifications have been executed at 3010 . At 3016 , a new degree/percentage/range of decrease in visual discomfort may be acquired. At 3018 , the system determines whether the decrease in visual discomfort is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3010 to further modify the media. Therefore, the media may be iteratively modified 3010 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 14: Modifying Media in Order to Decrease Disorientation and Postural Instability
Prolonged exposure to screen displays may result in disorientation and postural instability. In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of disorientation and postural instability, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the disorientation and postural instability, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease disorientation and postural instability, during interaction with that media. FIG. 31 illustrates a flow chart describing an exemplary process for modifying media in order to decrease disorientation and postural instability, in accordance with some embodiments of the present specification. At 3102 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in disorientation and postural instability. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease disorientation and postural instability, for the user and/or a group of users.
At 3104 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3106 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in disorientation and postural instability while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Divergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration Rate of Change 14. Increased Fixation Rate 15. Increased Fixation Count 16. Decreased Target Relevancy for Saccade Initial and Final Position 17. Decreased Target Relevancy for Saccade Angle 18. Decreased Saccade Magnitude (Distance of Saccade) 19. Increased Ratio of Anti-Saccade/Pro-Saccade 20. Increased Inhibition of Return 21. Increased Saccade Velocity 22. Increased Saccade Rate of Change 23. Increased Saccade Count (Number of Saccades) 24. Decreased Target Relevant Head Direction 25. Decreased Target Relevant Head Fixation 26. Decreased Target Relevant Limb Movement 27. Shift in Weight Distribution 28. Increased Body Temperature 29. Increased Respiration Rate 30. Low Oxygen Saturation 31. Increased Heart Rate 32. Low Blood Pressure 33. Increased Vocalizations Increased Reaction Time

The system may determine decrease in disorientation and postural instability while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Increased Fixation Duration 3. Increased Smooth Pursuit 4. Increased Screen Distance 5. Decreased Alpha/Delta Brain Wave ratio 6. Increased Alpha/Theta Brain Wave ratio 7. Stable Weight Distribution

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease disorientation and postural instability, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3108 , the changes in the plurality of data determined over time may be used to determine a degree of change in disorientation and postural instability. The change in disorientation and postural instability may indicate either reduced disorientation and postural instability or enhanced disorientation and postural instability.
At 3110 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) disorientation and postural instability. In embodiments, the media may be modified to address all the changes in data that reflect increase in disorientation and postural instability. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in disorientation and postural instability: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; decreased fixation rate; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in disorientation and postural instability, may be defined. In embodiments, an additional value for data may be acquired at 3112 , in order to further determine change in data over time at 3114 , after the modifications have been executed at 3110 . At 3116 , a new degree/percentage/range of decrease in disorientation and postural instability may be acquired. At 3118 , the system determines whether the decrease in disorientation and postural instability is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3110 to further modify the media. Therefore, the media may be iteratively modified 3110 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 15: Modifying Media in Order to Decrease Headaches and Difficulties in Focusing In an embodiment, data collected from the user, such as by HMDs, or any other
VR/AR/MxR system, is processed to determine the extent of headaches and difficulties in focusing, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the headaches and difficulties in focusing, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease headaches and difficulties in focusing, during interaction with that media. FIG. 32 illustrates a flow chart describing an exemplary process for modifying media in order to decrease headaches and difficulties in focusing, in accordance with some embodiments of the present specification. At 3202 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in headaches and difficulties in focusing. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease headaches and difficulties in focusing, for the user and/or a group of users.
At 3204 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3206 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in headaches and difficulties in focusing while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Divergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration Rate of Change 14. Increased Fixation Rate 15. Increased Fixation Count 16. Decreased Target Relevancy for Saccade Initial and Final Position 17. Decreased Target Relevancy for Saccade Angle 18. Decreased Saccade Magnitude (Distance of Saccade) 19. Increased Ratio of Anti-Saccade/Pro-Saccade 20. Increased Inhibition of Return 21. Increased Saccade Velocity 22. Increased Saccade Rate of Change 23. Increased Saccade Count (Number of Saccades) 24. Increased Screen Distance 25. Decreased Target Relevant Head Direction 26. Decreased Target Relevant Head Fixation 27. Decreased Target Relevant Limb Movement 28. Shift in Weight Distribution 29. Decreased Alpha/Delta Brain Wave ratio 30. Increased Body Temperature 31. Increased Respiration Rate 32. Low Oxygen Saturation 33. Increased Heart Rate 34. Changes in Blood Pressure 35. Increased Vocalizations 36. Increased Reaction Time

The system may determine decrease in headaches and difficulties in focusing while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Increased Fixation Duration 3. Increased Smooth Pursuit 4. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease headaches and difficulties in focusing, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3208 , the changes in the plurality of data determined over time may be used to determine a degree of change in headaches and difficulties in focusing. The change in headaches and difficulties in focusing may indicate either reduced headaches and difficulties in focusing or enhanced headaches and difficulties in focusing.
At 3210 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) headaches and difficulties in focusing. In embodiments, the media may be modified to address all the changes in data that reflect increase in headaches and difficulties in focusing. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in headaches and difficulties in focusing: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; decreased fixation rate; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in headaches and difficulties in focusing, may be defined. In embodiments, an additional value for data may be acquired at 3212 , in order to further determine change in data over time at 3214 , after the modifications have been executed at 3210 . At 3216 , a new degree/percentage/range of decrease in headaches and difficulties in focusing may be acquired. At 3218 , the system determines whether the decrease in headaches and difficulties in focusing is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3210 to further modify the media. Therefore, the media may be iteratively modified 3210 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 16: Modifying Media in Order to Decrease Blurred Vision and Myopia
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of blurred vision and myopia, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the blurred vision and myopia, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease blurred vision and myopia, during interaction with that media. FIG. 33 illustrates a flow chart describing an exemplary process for modifying media in order to decrease blurred vision and myopia, in accordance with some embodiments of the present specification. At 3302 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in blurred vision and myopia. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease blurred vision and myopia, for the user and/or a group of users.
At 3304 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3306 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in blurred vision and myopia while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Convergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration Rate of Change 14. Increased Fixation Rate 15. Increased Fixation Count 16. Decreased Target Relevancy for Saccade Initial and Final Position 17. Decreased Target Relevancy for Saccade Angle 18. Decreased Saccade Magnitude (Distance of Saccade) 19. Increased Ratio of Anti-Saccade/Pro-Saccade 20. Increased Inhibition of Return 21. Increased Saccade Velocity 22. Increased Saccade Rate of Change 23. Increased Saccade Count (Number of Saccades) 24. Increased Screen Distance 25. Decreased Target Relevant Head Direction 26. Decreased Target Relevant Head Fixation 27. Decreased Target Relevant Limb Movement 28. Shift in Weight Distribution 29. Decreased Alpha/Delta Brain Wave ratio 30. Increased Body Temperature 31. Increased Respiration Rate 32. Low Oxygen Saturation 33. Increased Heart Rate 34. Low Blood Pressure 35. Increased Reaction Time

The system may determine decrease in blurred vision and/or myopia while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Divergence 2. Increased Fixation Duration 3. Increased Smooth Pursuit 4. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease blurred vision and/or myopia, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3308 , the changes in the plurality of data determined over time may be used to determine a degree of change in blurred vision and/or myopia. The change in blurred vision and/or myopia may indicate either reduced blurred vision and/or myopia or enhanced blurred vision and/or myopia.
At 3310 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) blurred vision and/or myopia. In embodiments, the media may be modified to address all the changes in data that reflect increase in blurred vision and/or myopia. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in blurred vision and/or myopia: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of convergence; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; decreased fixation rate; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in blurred vision and/or myopia, may be defined. In embodiments, an additional value for data may be acquired at 3312 , in order to further determine change in data over time at 3314 , after the modifications have been executed at 3310 . At 3316 , a new degree/percentage/range of decrease in blurred vision and/or myopia may be acquired. At 3318 , the system determines whether the decrease in blurred vision and/or myopia is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3310 to further modify the media. Therefore, the media may be iteratively modified 3310 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 17: Modifying Media in Order to Decrease Heterophoria
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of heterophoria, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the heterophoria, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease heterophoria, during interaction with that media. FIG. 34 illustrates a flow chart describing an exemplary process for modifying media in order to decrease heterophoria, in accordance with some embodiments of the present specification. At 3402 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in heterophoria. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease heterophoria, for the user and/or a group of users.
At 3404 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3406 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in heterophoria while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Increased Rate of Divergence 12. Decreased Relevancy for Fixation Initial and Final Position 13. Increased Fixation Duration Rate of Change 14. Increased Fixation Count 15. Decreased Target Relevancy for Saccade Initial and Final Position 16. Decreased Target Relevancy for Saccade Angle 17. Decreased Saccade Magnitude (Distance of Saccade) 18. Increased Ratio of Anti-Saccade/Pro-Saccade 19. Increased Inhibition of Return 20. Increased Saccade Velocity 21. Increased Saccade Rate of Change 22. Increased Saccade Count (Number of Saccades) 23. Increased Screen Distance 24. Decreased Target Relevant Head Direction 25. Decreased Target Relevant Head Fixation 26. Decreased Target Relevant Limb Movement 27. Shift in Weight Distribution 28. Decreased Alpha/Delta Brain Wave ratio 29. Low Oxygen Saturation 30. Low Blood Pressure 31. Increased Reaction Time

The system may determine decrease in heterophoria while interacting with the media in a VR, AR, and/or MX environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Increased Fixation Duration 3. Increased Fixation Rate 4. Increased Smooth Pursuit 5. Increased Alpha/Theta Brain Wave ratio 6. Increased Ocular Alignment

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased body temperature; increased respiration rate; increased heart rate; increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease heterophoria, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3408 , the changes in the plurality of data determined over time may be used to determine a degree of change in heterophoria. The change in heterophoria may indicate either reduced heterophoria or enhanced heterophoria.
At 3410 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) heterophoria. In embodiments, the media may be modified to address all the changes in data that reflect increase in heterophoria. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in heterophoria: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; decreased fixation count; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in heterophoria, may be defined. In embodiments, an additional value for data may be acquired at 3412 , in order to further determine change in data over time at 3414 , after the modifications have been executed at 3410 . At 3416 , a new degree/percentage/range of decrease in heterophoria may be acquired. At 3418 , the system determines whether the decrease in heterophoria is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3410 to further modify the media. Therefore, the media may be iteratively modified 3410 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 18: Modifying Media In Order To Decrease Fixation Disparity
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of fixation disparity, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the fixation disparity, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease fixation disparity, during interaction with that media. FIG. 35 illustrates a flow chart describing an exemplary process for modifying media in order to decrease fixation disparity, in accordance with some embodiments of the present specification. At 3502 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in fixation disparity. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease fixation disparity, for the user and/or a group of users.
At 3504 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3506 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in fixation disparity while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Decreased Relevancy for Fixation Initial and Final Position 12. Increased Fixation Duration Rate of Change 13. Decreased Target Relevancy for Saccade Initial and Final Position 14. Decreased Target Relevancy for Saccade Angle 15. Decreased Saccade Magnitude (Distance of Saccade) 16. Increased Ratio of Anti-Saccade/Pro-Saccade 17. Increased Inhibition of Return 18. Increased Saccade Velocity 19. Increased Saccade Rate of Change 20. Increased Saccade Count (Number of Saccades) 21. Increased Screen Distance 22. Decreased Target Relevant Head Direction 23. Decreased Target Relevant Head Fixation 24. Decreased Target Relevant Limb Movement 25. Shift in Weight Distribution 26. Decreased Alpha/Delta Brain Wave ratio 27. Low Oxygen Saturation 28. Low Blood Pressure 29. Increased Reaction Time

The system may determine decrease in fixation disparity while interacting with the media in a VR, AR, and/or MxR environment based upon the following changes:
1. Increased Rate of Convergence 2. Increased Rate of Divergence 3. Increased Fixation Duration 4. Increased Fixation Rate 5. Increased Fixation Count 6. Increased Smooth Pursuit 7. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased body temperature; increased respiration rate; increased heart rate; increased vocalizations; change in facial expression (may be dependent on specific expression); change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease fixation disparity, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3508 , the changes in the plurality of data determined over time may be used to determine a degree of change in fixation disparity. The change in fixation disparity may indicate either reduced fixation disparity or enhanced fixation disparity.
At 3510 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) fixation disparity. In embodiments, the media may be modified to address all the changes in data that reflect increase in fixation disparity. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in fixation disparity: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in fixation disparity, may be defined. In embodiments, an additional value for data may be acquired at 3512 , in order to further determine change in data over time at 3514 , after the modifications have been executed at 3510 . At 3516 , a new degree/percentage/range of decrease in fixation disparity may be acquired. At 3518 , the system determines whether the decrease in fixation disparity is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3510 to further modify the media. Therefore, the media may be iteratively modified 3510 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 19: Modifying Media in Order to Decrease Vergence-Accommodation Disorder
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of vergence-accommodation disorder, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the vergence-accommodation disorder, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease vergence-accommodation disorder, during interaction with that media. FIG. 36 illustrates a flow chart describing an exemplary process for modifying media in order to decrease vergence-accommodation disorder, in accordance with some embodiments of the present specification. At 3602 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in vergence-accommodation disorder. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease vergence-accommodation disorder, for the user and/or a group of users.
At 3604 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3606 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in vergence-accommodation disorder while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Increased Rate of Change for Pupil Size 8. Decreased Target Relevancy for Pupil Initial and Final Position 9. Decreased Target Relevancy for Gaze Direction 10. Decreased Target Relevancy for Gaze Initial and Final Position 11. Decreased Relevancy for Fixation Initial and Final Position 12. Increased Fixation Duration Rate of Change 13. Decreased Target Relevancy for Saccade Initial and Final Position 14. Decreased Target Relevancy for Saccade Angle 15. Decreased Saccade Magnitude (Distance of Saccade) 16. Increased Ratio of Anti-Saccade/Pro-Saccade 17. Increased Inhibition of Return 18. Increased Saccade Velocity 19. Increased Saccade Rate of Change 20. Increased Saccade Count (Number of Saccades) 21. Increased Screen Distance 22. Decreased Target Relevant Head Direction 23. Decreased Target Relevant Head Fixation 24. Decreased Target Relevant Limb Movement 25. Shift in Weight Distribution 26. Decreased Alpha/Delta Brain Wave ratio

The system may determine decrease in vergence-accommodation disorder while interacting with the media in a VR, AR, and/or MxR environment based upon one or more of the following changes:
1. Increased Rate of Convergence 2. Increased Rate of Divergence 3. Increased Fixation Duration 4. Increased Fixation Rate 5. Increased Fixation Count 6. Increased Smooth Pursuit 7. Increased Alpha/Theta Brain Wave ratio

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased body temperature; increased respiration rate; low oxygen saturation; increased heart rate; low blood pressure; increased vocalizations; change in facial expression (may be dependent on specific expression); increased reaction time; change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease vergence-accommodated disorder, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3608 , the changes in the plurality of data determined over time may be used to determine a degree of change in vergence-accommodated disorder. The change in vergence-accommodated disorder may indicate either reduced vergence-accommodated disorder or enhanced vergence-accommodated disorder.
At 3610 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) vergence-accommodated disorder. In embodiments, the media may be modified to address all the changes in data that reflect increase in vergence-accommodated disorder. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data) 10. Increase use of longer viewing distances when possible 11. Match simulated distance with focal distance more closely 12. Move objects in and out of depth at a slower pace 13. Make existing object conflicts less salient

One or more of the following indicators may be observed to affirm a decrease in vergence-accommodated disorder: increased palpebral fissure rate of change; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; decreased rate of change for pupil size; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration rate of change; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade velocity; decreased saccade rate of change; decreased saccade count; decreased screen distance; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in vergence-accommodate disorder, may be defined. In embodiments, an additional value for data may be acquired at 3612 , in order to further determine change in data over time at 3614 , after the modifications have been executed at 3610 . At 3616 , a new degree/percentage/range of decrease in vergence-accommodate disorder may be acquired. At 3618 , the system determines whether the decrease in vergence-accommodate disorder is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3610 to further modify the media. Therefore, the media may be iteratively modified 3610 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 20: Modifying Media in Order to Increase Positive Emotion
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of positive emotion, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to increase the positive emotion, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to increase positive emotion, during interaction with that media. FIG. 37 illustrates a flow chart describing an exemplary process for modifying media in order to increase positive emotion, in accordance with some embodiments of the present specification. At 3702 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in positive emotion. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to increase positive emotion, for the user and/or a group of users.
At 3704 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3706 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect decrease in positive emotion while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Blink Rate 5. Increased Rate of Change for Blink Rate 6. Increased Ratio of Partial Blinks to Full Blinks 7. Decreased Target Relevancy for Pupil Initial and Final Position 8. Decreased Target Relevancy for Gaze Direction 9. Decreased Target Relevancy for Gaze Initial and Final Position 10. Decreased Relevancy for Fixation Initial and Final Position 11. Increased Fixation Duration 12. Increased Fixation Duration Rate of Change 13. Decreased Target Relevancy for Saccade Initial and Final Position 14. Decreased Target Relevancy for Saccade Angle 15. Decreased Saccade Magnitude (Distance of Saccade) 16. Increased Ratio of Anti-Saccade/Pro-Saccade 17. Increased Inhibition of Return 18. Increased Saccade Rate of Change 19. Increased Smooth Pursuit 20. Decreased Target Relevant Head Direction 21. Decreased Target Relevant Head Fixation 22. Decreased Target Relevant Limb Movement 23. Shift in Weight Distribution 24. Decreased Alpha/Delta Brain Wave ratio 25. Increased Body Temperature 26. Increased Respiration Rate 27. Low Oxygen Saturation 28. Increased Heart Rate 29. Low Blood Pressure 30. Increased Reaction Time

The system may determine increase in positive emotion while interacting with the media in a VR, AR, and/or MxR environment based upon one or more of the following changes:
1. Increased Rate of Change for Pupil Size 2. Increased Fixation Rate 3. Increased Fixation Count 4. Increased Saccade Velocity 5. Increased Saccade Count (Number of Saccades) 6. Increased Screen Distance 7. Increased Alpha/Theta Brain Wave ratio 8. Increased Vocalizations

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to: increased rate of convergence; increased rate of divergence; change in facial expression (may be dependent on specific expression); increased reaction time; change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to increase positive emotion, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3708 , the changes in the plurality of data determined over time may be used to determine a degree of change in positive emotion. The change in positive emotion may indicate either reduced positive emotion or enhanced positive emotion.
At 3710 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) positive emotion. In embodiments, the media may be modified to address all the changes in data that reflect decrease in positive emotion. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm an increase in positive emotion: increased palpebral fissure height; decreased blink rate; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; increased relevancy for fixation initial and final position; decreased fixation duration; decreased fixation duration rate of change; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased saccade rate of change; decreased smooth pursuit; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of increase in positive emotion, may be defined. In embodiments, an additional value for data may be acquired at 3712 , in order to further determine change in data over time at 3714 , after the modifications have been executed at 3710 . At 3716 , a new degree/percentage/range of increase in positive emotion may be acquired. At 3718 , the system determines whether the increase in positive emotion is within the specified range or percentage. If it is determined that the increase is insufficient, the system may loop back to step 3710 to further modify the media. Therefore, the media may be iteratively modified 3710 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 21: Modifying Media in Order to Decrease Negative Emotion
In an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of negative emotion, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user in order to decrease the negative emotion, such as but not limited to by minimizing visual, or any other discomfort arising from the media experience. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to the user for modifying displayed media in a VR, AR and/or MxR environment in order to decrease negative emotion, during interaction with that media. FIG. 38 illustrates a flow chart describing an exemplary process for modifying media in order to decrease negative emotion, in accordance with some embodiments of the present specification. At 3802 , a first value for a plurality of data, as further described below, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback; 8. A device to perform electroencephalography; 9. A device to perform electrocardiography; 10. A device to perform electromyography; 11. A device to perform electrooculography; 12. A device to perform electroretinography; and 13. One or more sensors configured to measure Galvanic Skin Response.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in negative emotion. Accordingly, the media presented in a VR/AR/MxR environment is modified, in order to decrease negative emotion, for the user and/or a group of users.
At 3804 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3806 , the first and second values of data are used to determine one or more changes in the plurality of data over time. In embodiments of the current use case scenario, one or more of the following changes, tracked and recorded by the hardware and software of the system, may reflect increase in negative emotion while interacting with the VR, AR, and/or MxR media:
1. Decreased Palpebral Fissure Rate of Change 2. Low Distance Palpebral Fissure Resting State 3. Low Distance Palpebral Fissure Active State 4. Increased Rate of Change for Blink Rate 5. Increased Ratio of Partial Blinks to Full Blinks 6. Decreased Target Relevancy for Pupil Initial and Final Position 7. Decreased Target Relevancy for Gaze Direction 8. Decreased Target Relevancy for Gaze Initial and Final Position 9. Increased Rate of Divergence 10. Decreased Relevancy for Fixation Initial and Final Position 11. Increased Fixation Duration 12. Decreased Target Relevancy for Saccade Initial and Final Position 13. Decreased Target Relevancy for Saccade Angle 14. Decreased Saccade Magnitude (Distance of Saccade) 15. Increased Ratio of Anti-Saccade/Pro-Saccade 16. Increased Inhibition of Return 17. Increased Smooth Pursuit 18. Decreased Target Relevant Head Direction 19. Decreased Target Relevant Head Fixation 20. Decreased Target Relevant Limb Movement 21. Shift in Weight Distribution 22. Decreased Alpha/Delta Brain Wave ratio 23. Increased Alpha/Theta Brain Wave ratio 24. Increased Body Temperature 25. Increased Respiration Rate 26. Low Oxygen Saturation 27. Increased Heart Rate 28. High Blood Pressure 29. Increased Reaction Time

The system may determine decrease in negative emotion while interacting with the media in a VR, AR, and/or MxR environment based upon one or more of the following changes:
1. Increased Blink Rate 2. Increased Rate of Change for Pupil Size 3. Increased Rate of Convergence 4. Increased Fixation Duration Rate of Change 5. Increased Fixation Rate 6. Increased Fixation Count 7. Increased Saccade Velocity 8. Increased Saccade Rate of Change 9. Increased Saccade Count (Number of Saccades) 10. Increased Screen Distance 11. Increased Vocalizations

Other changes may also be recorded and can be interpreted in different ways. These may include, but are not limited to change in facial expression (may be dependent on specific expression); increased reaction time; change in gustatory processing; change in olfactory processing; and change in auditory processing.
It should be noted that while the above-stated lists of data acquisition components, types of data, and changes in data may be used to determine variables needed to decrease negative emotion, these lists are not exhaustive and may include other data acquisition components, types of data, and changes in data.
At 3808 , the changes in the plurality of data determined over time may be used to determine a degree of change in negative emotion. The change in negative emotion may indicate either reduced negative emotion or enhanced negative emotion.
At 3810 , media rendered to the user may be modified on the basis of the degree of reduced (or enhanced) negative emotion. In embodiments, the media may be modified to address all the changes in data that reflect increase in negative emotion. In embodiments, a combination of one or more of the following modifications may be performed:
1. Increasing a contrast of the media 2. Making an object of interest that is displayed in the media larger in size 3. Increasing a brightness of the media 4. Increasing an amount of an object of interest displayed in the media shown in a central field of view and decreasing said object of interest in a peripheral field of view 5. Changing a focal point of content displayed in the media to a more central location 6. Removing objects from a field of view and measuring if a user recognizes said removal 7. Increasing an amount of color in said media 8. Increasing a degree of shade in objects shown in said media 9. Changing RGB values of said media based upon external data (demographic or trending data)

One or more of the following indicators may be observed to affirm a decrease in negative emotion: increased palpebral fissure height; decreased rate of change for blink rate; decreased ratio of partial blinks to full blinks; increased target relevancy for pupil initial and final position; increased target relevancy for gaze direction; increased target relevancy for gaze initial and final position; decreased rate of divergence; increased relevancy for fixation initial and final position; decreased fixation duration; increased target relevancy for saccade initial and final position; increased target relevancy for saccade angle; increased saccade magnitude (task relevant); decreased ratio of anti-saccade/pro-saccade; decreased inhibition of return; decreased smooth pursuit; increased target relevant head direction; increased target relevant head fixation; increased target relevant limb movement; decrease in shifts of weight distribution; increased alpha/delta brain wave ratio; normal body temperature; normal respiration rate; 90-100% oxygen saturation; normal heart rate; normal blood pressure; task relevant vocalizations; task relevant facial expressions; decreased reaction time; task relevant gustatory processing; task relevant olfactory processing; and task relevant auditory processing.
In embodiments, a specific percentage or a range of decrease in negative emotion, may be defined. In embodiments, an additional value for data may be acquired at 3812 , in order to further determine change in data over time at 3814 , after the modifications have been executed at 3810 . At 3816 , a new degree/percentage/range of decrease in negative emotion may be acquired. At 3818 , the system determines whether the decrease in negative emotion is within the specified range or percentage. If it is determined that the decrease is insufficient, the system may loop back to step 3810 to further modify the media. Therefore, the media may be iteratively modified 3810 and overall performance may be measured, until a percentage of improvement of anywhere from 1% to 10000%, or any increment therein, is achieved.
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation.
The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
Example Use 22: Modifying Media Resulting from Micro-Transactions
The sensory inputs determined and analyzed by the system may eventually drive work and play engagements. In embodiments, sensory information may be purchased from users and used to create sensory data exchanges after adding value to the data through platforms such as the SDEP. In embodiments of the present specification, the senses of individuals and potential consumers may be measured and monitored with the SDEP. The SDEP may provide data analysis regarding trends on user-behavior based on sensory data, user data, context of environment and location. Embodiments of SDEP may use machine learning and deep learning techniques to develop predictive recommendations in real-time and to allow the ability for a company to use a real-time dynamic change to the content/advertisement to personalize the experience to the consumer.
In embodiments, a user interfacing with an HMD or a similar device is monitored. The user may be offered an option to share their psychometric/sensory/biometric data, which may be further used to better understand and customize the user's experience in terms of type of content and other show suggestions. Assuming that the user opts to share the data, in an embodiment, during the interfacing, the SDEP determines the user to have a first sensory state. In an embodiment, the sensory states include a first blink rate, a first degree of pupil dilation, a first degree of saccadic movement, any other eye movement, inter-palpebral fissure distance, facial expression, and/or one or more other parameters such as the ones discussed in the following User Case Scenarios. Additionally, image processing of the content presented to the user at a certain rate (frames/per second) may deconstruct the content in to core psychometric raw data including color (RGB) values, contrast, size and location of objects. Further, smart devices such as a fitness monitoring band or a watch may obtain and provide data pertaining heart rate and basal body temperature; smart clothing may provide data pertaining respiratory rate and motion of body and limbs; smart shoes may provide data pertaining weight/pressure distribution. Similarly, there may be other sources that provide various psychometric/sensory/biometric data of the user. Examples of combinations of measures include comprehension levels, fatigue levels, engagement levels, among others. Furthermore, the SDEP determines a second sensory state, wherein the second sensory state indicates changes in the measured psychometric/sensory/biometric data relative to the first state. The psychometric/sensory/biometric data measured by the SDEP may be combined with characteristics of the visual data rendered to the user in the same duration. The combination may be further used to change a set of visuals and/or characteristics of the visuals in order to receive a desired psychometric/sensory/biometric result, indicating greater engagement, from the same user or across groups of users that have a similar profile as the user discussed in this embodiment. The SDEP thus utilizes each vision metric and subsequent weight of each vision metric to develop a conversion metric. Further, conversion metrics may be developed by the SDEP with additional value with a successful gesture toward a desired product. The user profiles may be grouped according to demographics, or any other parameters.
In an example, a user watching a sports event and an in-ad display using an HMD, is shown a branded shirt that is a certain shade of the color blue (RGB value, luminance level). The user may have elected to share advanced data settings in his HMD system. During the initial 1 minute period—SDEP noted meta-data trends of red at a certain RGB value was trending in males within user's demographic. The trend may be AB tested in real time with the user during the ad display by changing the Blue shirt to Red, and further changing the Red to a specific shade of color Red. The specific shade of color Red could be personalized to the user based on the personal trends noted of the user that were shared through the settings enabled in his HMD. The personal trends noted by the SDEP, through the user's HMD may include quantifiable metrics for user engagement, such as but not limited to decreased blink rate, decrease saccadic movements, including anti-saccadic error prior to full fixation of vision, dilation of pupil from steady state, movement of head in relationship to where ad is placed in VR/AR/MxR environment, with increase in heart rate, temperature and movement toward the object.
In embodiments, the SDEP may interface with the user to provide regular (periodic) updates to a separate entity, such as a third party or the content provider, about the psychometric/sensory/biometric user data shared with the SDEP. The proportion of psychometric/biometric/sensory data shared and duration of this share, may be used as a basis for a micro-transaction or series of micro-transactions that occur between the user and the separate entity. In embodiments, the SDEP provides a platform to enable such micro-transactions with the user. In some embodiments, user's revenue share may be proportional to the amount of the psychometric/sensory/biometric data shared regularly with the SDEP.
Therefore, in an embodiment, data collected from the user, such as by HMDs, or any other VR/AR/MxR system, is processed to determine the extent of data optionally shared by the user for the knowledge of a separate entity, such as a third party or the source of the content, which could be experienced by the user. The data may be further utilized to modify VR/AR/MxR media for the user. Additionally, the extent and duration of shared data may be utilized to transact with the user. In one embodiment, a transaction is in the form of a financial reward, where the amount of the reward is proportional to the extent and duration of shared data. In an embodiment, media is modified in real time for the user. In another embodiment, data is saved and used to modify presentation of VR/AR/MxR media to subsequent users with a similar data, or subsequently to the user.
More specifically, the present specification describes methods, systems and software that are provided to enable micro-transactions with the user, involving rewards in exchange of psychometric/sensory/biometric data, while also modifying displayed media in a VR, AR and/or MxR environment, during interaction with that media. FIG. 39 illustrates a flow chart describing an exemplary process for modifying media while enabling a micro-transaction, in accordance with some embodiments of the present specification. At 3902 , a first value for a plurality of data, such as psychometric/sensory/biometric data of the user, is acquired. In embodiments, data is acquired by using at least one camera configured to acquire eye movement data (rapid scanning and/or saccadic movement), blink rate data, fixation data, pupillary diameter, palpebral (eyelid) fissure distance between the eyelids. Additionally, the VR, AR, and/or MxR device can include one or more of the following sensors incorporated therein:
1. One or more sensors configured to detect basal body temperature, heart rate, body movement, body rotation, body direction, and/or body velocity; 2. One or more sensors configured to measure limb movement, limb rotation, limb direction, and/or limb velocity; 3. One or more sensors configured to measure pulse rate and/or blood oxygenation; 4. One or more sensors configured to measure auditory processing; 5. One or more sensors configured to measure gustatory and olfactory processing; 6. One or more sensors to measure pressure; 7. At least one input device such as a traditional keyboard and mouse and or any other form of controller to collect manual user feedback.

In embodiments, the data acquired by a combination of these devices may include data pertaining to one or more of: palpebral fissure (including its rate of change, initial state, final state, and dynamic changes); blink rate (including its rate of change and/or a ratio of partial blinks to full blinks); pupil size (including its rate of change, an initial state, a final state, and dynamic changes); pupil position (including its initial position, a final position); gaze direction; gaze position (including an initial position and a final position); vergence (including convergence vs divergence based on rate, duration, and/or dynamic change); fixation position (including its an initial position, a final position); fixation duration (including a rate of change); fixation rate; fixation count; saccade position (including its rate of change, an initial position, and a final position); saccade angle (including it relevancy towards target); saccade magnitude (including its distance, anti-saccade or pro-saccade); pro-saccade (including its rate vs. anti-saccade); anti-saccade (including its rate vs. pro-saccade); inhibition of return (including presence and/or magnitude); saccade velocity (including magnitude, direction and/or relevancy towards target); saccade rate, including a saccade count, pursuit eye movements (including their initiation, duration, and/or direction); screen distance (including its rate of change, initial position, and/or final position); head direction (including its rate of change, initial position, and/or final position); head fixation (including its rate of change, initial position, and/or final position); limb tracking (including its rate of change, initial position, and/or final position); weight distribution (including its rate of change, initial distribution, and/or final distribution); frequency domain (Fourier) analysis; electroencephalography output; frequency bands; electrocardiography output; electromyography output; electrooculography output; electroretinography output; galvanic skin response; body temperature (including its rate of change, initial temperature, and/or final temperature); respiration rate (including its rate of change, initial rate, and/or final rate); oxygen saturation; heart rate (including its rate of change, initial heart rate, and/or final heart rate); blood pressure; vocalizations (including its pitch, loudness, and/or semantics); inferred efferent responses; respiration; facial expression (including micro-expressions); olfactory processing; gustatory processing; and auditory processing. Each data type may hold a weight when taken in to account, either individually or in combination.
In embodiments, the system uses machine learning to be able to discover new correlations between behavioral, electrophysiological and/or autonomic measures, and the less ambiguous measures. In some examples, some of the measures described above are context specific. The system can correlate all available measures and look for trends in negative emotion. Accordingly, the media presented in a VR/AR/MxR environment is modified, for the user and/or a group of users.
At 3904 , a second value for the plurality of data, described above, is acquired. In embodiments, the first value and the second value are of the same data types, including the data types described above. At 3906 , the first and second values of data are used to determine one or more changes in the plurality of data over time. Different types of data changes may be recorded and can be interpreted in different ways.
At 3908 , the determined changes in the plurality of data determined over time may be stored in a database. The psychometric/sensory/biometric data measured by the SDEP may be combined with characteristics of the visual data rendered to the user in the same duration. The database may be maintained by the SDEP and/or a separate entity such as a third party, a company, or the content-provider of the content presented to the user. The data is further processed in accordance with the various embodiments described in the present specification, to model user behaviour and modify the media. The combination may be further used to change a set of visuals and/or characteristics of the visuals in order to receive a desired psychometric/sensory/biometric result, indicating greater engagement, from the same user or across groups of users that have a similar profile as the user discussed in this embodiment. The SDEP thus utilizes each vision metric and subsequent weight of each vision metric to develop a conversion metric. Further, conversion metrics may be developed by the SDEP with additional value with a successful gesture toward a desired product. The user profiles may be grouped according to demographics, or any other parameters.
At 3910 , the quantity and duration of changes in data determined over time may be used to reward the user. The reward may be provided by the separate entity in lieu of the user opting to share their psychometric/sensory/biometric data.
Weighing Sources of Information
In one embodiment, the system applies a numerical weight or preference to one or more of the above-described measures based on their statistical significance for a given application, based on their relevance and/or based on their degree of ambiguity in interpretation.
In one embodiment, the system first determines a media context within which the above listed data is collected. A context may include a type of application, such as a puzzle game, action game, movie, advertisement, strategy game, social network, or other form of media application. Context appropriate measures that may signal a particular state preference is given to those measures with the fewest alternative interpretations. It is also preferable to favor more general (less specific) states when interpreting measures. For example, an increase in heart rate suggests at least a heightened state of arousal, if not an increase in comprehension.
In another embodiment, a particular condition or state is determined independent of any other condition or state. The condition or state may include fatigue, engagement, performance, comprehension, symptoms associated with visually-induced motion sickness secondary to visual-vestibular mismatch, symptoms associated with post-traumatic stress disorder, double vision related to accommodative dysfunction, vection due to unintended peripheral field stimulation, vergence-accommodation disorders, fixation disparity, blurred vision and myopia, headaches, difficulties in focusing, disorientation, postural instability, visual discomfort, eyestrain, dry eye, eye tearing, foreign body sensation, feeling of pressure in the eyes, aching around the eyes, nausea, stomach discomfort, potential phototoxicity from overexposure to screen displays, and hormonal dysregulation arising from excessive blue light exposure. In another embodiment, a particular condition or state is determined in correlation with any other state since the states are potentially correlated in certain scenarios. For example, in certain applications, comprehension requires engagement. However, in other applications, engagement may not, necessarily, require comprehension. As fatigue increases, engagement and comprehension will likely decrease. Engagement and comprehension may also decrease without increasing fatigue if users simply become disinterested. Accordingly, the measuring of these states should be done independently and in parallel, followed by considerations of the interaction of those measures.
The system preferably arithmetically weights the various measures based upon context and a predefined hierarchy of measures, wherein the first tier of measures has a greater weight than the second tier of measures, which has a greater weight than the third tier of measures. The measures are categorized into tiers based on their degree of ambiguity and relevance to any given contextual situation. It would be apparent to one skilled in the art that the following measures are exemplary, and not exhaustive. Other measures and/or combinations of measures may be used in different tiers.
1. First Tier




a. Eye tracking measures of comprehension: The may include a combination of measures of comprehension such as relevant fixation (R_(Rel. Fix.)); mean of the absolute angle relative to relevant regions ( |θ| saccade-relevant ) mean magnitude; component relative to relevant regions ( M saccade-relevent ); fixation correlations (C fixation ); saccade correlations (C saccade ); Correlation of the listener's eye movements; an area of focus (A focus ). It may also include a level of engagement based on the area of focus (A focus ) where the area of focus is significantly correlated with the spatial extent of the stimulus in question.
b. A significant amplitude magnitude increases in cognitive EEG potentials (N2, N44, P300, P600) resulting from infrequent, novel or unexpected stimuli
c. A transition from partially to completely open eyes (significant increase in p both eyes open from a non-zero baseline)
d. Random or un-focused search characterized by significantly brief fixation durations and significantly large saccade magnitudes
e. Combination of measures of engagement such as response rate of less than 100% (or some lower, baseline rate of responding, depending on context); and measure of fatigue such as reductions in ‘performance’ metrics over extended periods of activity and decreasing proportion of responding, in appropriate contexts
f. Interactions away from a particular task or stimulus as indicating lack of or disengagement
g. Other measures of engagement including relative time-on-task as the proportion of time spent performing a task or processing a stimulus compared to not; the ratio of interactions among available tasks as indicative of time-on-task for each as a relative measure of engagement with each task; and the ratio of fixation count and/or duration among stimuli and/or visual regions as indicative of time-on-task as a relative measure of engagement with each stimulus or visual region
h. Combination of measures of engagement and fatigue such as significant shortening of the distance between a visual stimulus and the user's eyes as an indication of engagement onset, and the proportional deviation from baseline as an indication of level of engagement; and yawning or other pronounced and discrete respiration
i. Measures of fatigue such as prolonged periods of (mostly) closed eyes; and sudden vertical eye movements




2. Second Tier




a. Combination of measures such as a slowed blink rate relative to an established baseline (f blink significantly less than f blink ) and a blink rate significantly less than baseline. Also, combination of measures of increased blink rate, such as significant increase in blink rate and transitions to shorter and more frequency blinks.
b. Onset of comprehension as the point in time where, when applicable, the percent of correct responses increases significantly
c. Onset of comprehension as the point when a target in a VR/AR/MxR media is correctly identified and or located
d. Combination of measures related to the onset of comprehension as the end of a period of significantly longer fixation durations (D fixation ); the duration of last fixation on the selected stimulus; and when a choice is made, the duration of first fixation on any stimulus
e. Combination of measures such as a rapid and significant increase in pupil diameter (S pupil ), and significant pupil dilation in the context of a choice task
f. A significant upward or downward deviation from average percent correct responding as signaling engagement or disengagement, respectively
g. Adjustment of 3D gaze position towards the appropriate depth (here considered separately from direction of gaze) to view a stimulus as a signal of engagement with that stimulus; and 3D depth of gaze towards infinity for extended periods as indicative of fatigue
h. Rigid fixation in the context of monitoring for subtle changes or motion, or the precise onset of any change or motion, as indicative of engagement; and reduced or held respiration in the context of monitoring
i. Changes in eye movement patterns characterized by reduced saccade magnitude and fixation frequency




3. Third Tier




a. Significant increase in GSR-ERP
b. significant increases in energy of an EEG in beta and gamma frequency bands 16 Hz); increased bilateral phase synchrony of EEG activity during choice tasks; and tradeoff where low frequency (<10 Hz) EEG energy increases and high frequency (≧10 Hz) EEG energy decreases
c. A significant increase in body temperature and/or heart rate in association with delayed response to a question of understanding. Also, measures related to increase in autonomic arousal as indicative of increasing engagement, and decreases in arousal as disengagement; and significant decreases in heart rate and/or body temperature as indicative of fatigue
d. Any measure signaling significant comprehension, or onset of comprehension; and significant reductions in comprehension, engagement and other excitatory states in the context of prolonged activity
e. Significant signs of dry eye (e.g. low tear-break-up-time) as indicative of ocular fatigue




The system further tracks any and all correlations of different states, such as correlations between engagement, comprehension and fatigue, to determine timing relationships between states based on certain contexts. These correlations may be immediate or with some temporal delay (e.g. engagement reduction is followed after some period of time by fatigue increase). With the embodiments of the present specification, any and all correlations may be found whether they seem intuitive or not.
For any significant correlations that are found, the system models the interactions of the comprising measures based on a predefined algorithm that fits the recorded data. For example, direct measures such as user's ability to detect, discriminate, accuracy for position, accuracy for time, and others, are required across various application. Indirect measures such as and not limited to fatigue and endurance are also monitored across various applications. However, a gaming application may find measures of user's visual attention, ability to multi-track, and others, to be of greater significance to determine rewards/points. Meanwhile, a user's ability to pay more attention to a specific product or color on a screen may lead to an advertising application to lay greater significance towards related measures.
The above examples are merely illustrative of the many applications of the system of present invention. Although only a few embodiments of the present invention have been described herein, it should be understood that the present invention might be embodied in many other specific forms without departing from the spirit or scope of the invention. Therefore, the present examples and embodiments are to be considered as illustrative and not restrictive, and the invention may be modified within the scope of the appended claims.