The lower end of luminance levels in the real world is at 0 cd/m2, which is the absence of any photons (see Fig. 15.1A). The top of the luminance range is open-ended but one of the everyday objects with the highest luminance levels is the sun disk, with approximately 1.6 × 109 cd/m2 (Halstead, 1993).
Figure 15.1. Real-world luminance levels and the high-level functionality of the HVS.
A human can perceive approximately 14 log10 units,3 by converting light incident on the eye into nerve impulses using photoreceptors (see Fig. 15.1B). These photoreceptors can be structurally and functionally divided into two broad categories, which are known as rods and cones, each having a different visual function (Fairchild, 2013; Hubel, 1995). Rod photoreceptors are extremely sensitive to light to facilitate vision in dark environments such as at night. The dynamic range over which the rods can operate ranges from 10−6 to 10 cd/m2. This includes the “scotopic” range when cones are inactive, and the “mesopic” range when rods and cones are both active. The cone photoreceptors are less sensitive than the rods and operate under daylight conditions, forming photopic vision (in these luminance ranges, the rods are effectively saturated). The cones can be stimulated by light ranging from luminance levels of approximately 0.03 to 108 cd/m2 (Ferwerda et al., 1996; Reinhard et al., 2010). The mesopic range extends from about 0.03 to 3 cd/m2, above which the rods are inactive, and the range above 3 cd/m2 is referred to as “photopic.”
This extensive dynamic range facilitated by the photoreceptors allows us to see objects under faint starlight as well as scenery lit by the midday sun, under which some scene elements can reach luminance levels of millions of candelas per square meter. However, at any moment of time, human vision is able to operate over only a fraction of this enormous range. To reach the full 14 log10 units of dynamic range (approximately 46.5 f-stops), the HVS shifts this dynamic range subset to an appropriate light sensitivity using various mechanical, photochemical, and neuronal adaptive processes (Ferwerda, 2001), so that under any lighting conditions the effectiveness of human vision is maximized.
Three sensitivity-regulating mechanisms are thought to be present in cones that facilitate this process — namely, response compression, cellular adaptation, and pigment bleaching (Valeton and van Norren, 1983). The first mechanism accounts for nonlinear range compression, therefore leading to the instantaneous nonlinearity, which is called the “simultaneous dynamic range” or “steady-state dynamic range” (SSDR). The latter two mechanisms are considered true adaptation mechanisms (Baylor et al., 1974). These true adaptation mechanisms can be classified into light adaptation and chromatic adaptation (Fairchild, 2013),4 which enable the HVS to adjust to a wide range of illumination conditions. The concept of light adaptation is shown in Fig. 15.1C.
“Light adaptation” refers to the ability of the HVS to adjust its visual sensitivity to the prevailing level of illumination so that it is capable of creating a meaningful visual response (eg, in a dark room or on a sunny day). It achieves this by changing the sensitivity of the photoreceptors as illustrated in Fig. 15.2A. Although light and dark adaptation seem to belong to the same adaptation mechanism, there are differences reflected by the time course of these two processes, which is on the order of 5 min for complete light adaptation. The time course for dark adaptation is twofold, leveling out for the cones after 10 min and reaching full dark adaptation after 30 min (Boynton and Whitten, 1970; Davson, 1990). Nevertheless, the adaptation to small changes of environment luminance levels occurs relatively fast (hundreds of milliseconds to seconds) and impact the perceived dynamic range in typical viewing environments.
Figure 15.2. The concepts of light (and dark) adaptation where the gain of all photoreceptors is adjusted together (A) and chromatic adaptation where the sensitivity of each cone photoreceptor L, M, and S is adjusted individually (B).
Chromatic adaptation is another of the major adaptation processes (Fairchild, 2013). It uses physiological mechanisms similar to those used by light adaptation, but is capable of adjusting the sensitivity of each cone individually, which is illustrated in Fig. 15.2B (the peak of each of the responses can change individually). This process enables the HVS to adapt to various types of illumination so that, for example, white objects retain their white appearance.
Another light-controlling aspect is pupil dilation, which — similarly to an aperture of a camera — controls the amount of light entering the eye (Watson and Yellot, 2012). The effect of pupil dilation is small (approximately 1/16 times) compared with the much more impactful overall adaptation processes (approximately a million times), so it is often ignored in most engineering models of vision.5 However, secondary effects, such as increased depth of focus and less glare with the decreased pupil size, are relevant for 3D displays and HDR displays, respectively.
The luminance distribution is the pattern of luminance in a space bounded by surfaces [40] and is often simplified to luminance ratios. The luminance distribution is measured using HDR cameras (i.e. luminance distribution measurement devices) [59,60]. The pixel values, after some transformations, represent the luminance values. The HDR technology is essential as it allows to capture the luminance ranges occurring in real scenarios [59]. Fisheye lenses are used to capture the entire luminance distribution of a room as experienced from the camera position; therefore, it is advisable to measure from the viewers' position. Theoretically, the luminance distribution can also be measured by a (spot) luminance meter, but this is an imprecise and tedious process subject to major and rapid changes in the luminous conditions.
For ad hoc measurements, the luminance distribution is measured from the seating position at a height of 1.2 m, representing the view from the eye. As potential users in the room are not constantly looking at the same direction, some extreme situations need to be measured as well [16].
Continuous measurements of the luminance distribution are problematic because the respective space is occupied by the users. Two strategies can be distinguished to measure the luminance distribution while a space is occupied. For lab studies, two identical rooms located directly besides each other can be used [61,62]. In the first room, the participant is seated; in the second room, the appropriate measurement devices are set-up. This strategy is not feasible for field studies; so in field studies the measurement devices needs to be placed at a suboptimal position. Then, the measurement device is placed at a position as close as possible to the optimal position. Fan et al. [63] provide a methodology to determine this suboptimal position; in their study this position was rotated 30° at a distance of 0.3–0.5 meters from the optimal view point.
With the advent of HDR capture, transport, and display devices, a range of fundamental questions have appeared regarding the design of those imaging pipeline elements. For instance, viewer preferences regarding overall luminance levels as well as contrast were determined previously (Seetzen et al., 2006), as were design criteria for displays used under low ambient light levels (Mantiuk et al., 2009). Further, recent studies have revealed viewer preferences for HDR imagery displayed on HDR displays (Akyüz et al., 2007), and have assessed video viewing preferences for HDR displays under different ambient illuminations (Rempel et al., 2009).
Especially in the context of display devices, it is important to know how high the dynamic range of a display should be. An implicit goal is that a good display should be able to accommodate a dynamic range that is somewhat larger than the simultaneous dynamic range afforded by human vision, possibly also taking into account the average room illumination as well as short-term temporal adaptive effects. More advanced engineering approaches would try to identify a peak dynamic range so that further physical improvements do not result in diminishing returns.
15.2.1 Real-World Luminance and the HVS
The lower end of luminance levels in the real world is at 0 cd/m2, which is the absence of any photons (see Fig. 15.1A). The top of the luminance range is open-ended but one of the everyday objects with the highest luminance levels is the sun disk, with approximately 1.6 × 109 cd/m2 (Halstead, 1993).
Figure 15.1. Real-world luminance levels and the high-level functionality of the HVS.
A human can perceive approximately 14 log10 units,3 by converting light incident on the eye into nerve impulses using photoreceptors (see Fig. 15.1B). These photoreceptors can be structurally and functionally divided into two broad categories, which are known as rods and cones, each having a different visual function (Fairchild, 2013; Hubel, 1995). Rod photoreceptors are extremely sensitive to light to facilitate vision in dark environments such as at night. The dynamic range over which the rods can operate ranges from 10−6 to 10 cd/m2. This includes the “scotopic” range when cones are inactive, and the “mesopic” range when rods and cones are both active. The cone photoreceptors are less sensitive than the rods and operate under daylight conditions, forming photopic vision (in these luminance ranges, the rods are effectively saturated). The cones can be stimulated by light ranging from luminance levels of approximately 0.03 to 108 cd/m2 (Ferwerda et al., 1996; Reinhard et al., 2010). The mesopic range extends from about 0.03 to 3 cd/m2, above which the rods are inactive, and the range above 3 cd/m2 is referred to as “photopic.”
This extensive dynamic range facilitated by the photoreceptors allows us to see objects under faint starlight as well as scenery lit by the midday sun, under which some scene elements can reach luminance levels of millions of candelas per square meter. However, at any moment of time, human vision is able to operate over only a fraction of this enormous range. To reach the full 14 log10 units of dynamic range (approximately 46.5 f-stops), the HVS shifts this dynamic range subset to an appropriate light sensitivity using various mechanical, photochemical, and neuronal adaptive processes (Ferwerda, 2001), so that under any lighting conditions the effectiveness of human vision is maximized.
Three sensitivity-regulating mechanisms are thought to be present in cones that facilitate this process — namely, response compression, cellular adaptation, and pigment bleaching (Valeton and van Norren, 1983). The first mechanism accounts for nonlinear range compression, therefore leading to the instantaneous nonlinearity, which is called the “simultaneous dynamic range” or “steady-state dynamic range” (SSDR). The latter two mechanisms are considered true adaptation mechanisms (Baylor et al., 1974). These true adaptation mechanisms can be classified into light adaptation and chromatic adaptation (Fairchild, 2013),4 which enable the HVS to adjust to a wide range of illumination conditions. The concept of light adaptation is shown in Fig. 15.1C.
“Light adaptation” refers to the ability of the HVS to adjust its visual sensitivity to the prevailing level of illumination so that it is capable of creating a meaningful visual response (eg, in a dark room or on a sunny day). It achieves this by changing the sensitivity of the photoreceptors as illustrated in Fig. 15.2A. Although light and dark adaptation seem to belong to the same adaptation mechanism, there are differences reflected by the time course of these two processes, which is on the order of 5 min for complete light adaptation. The time course for dark adaptation is twofold, leveling out for the cones after 10 min and reaching full dark adaptation after 30 min (Boynton and Whitten, 1970; Davson, 1990). Nevertheless, the adaptation to small changes of environment luminance levels occurs relatively fast (hundreds of milliseconds to seconds) and impact the perceived dynamic range in typical viewing environments.
Figure 15.2. The concepts of light (and dark) adaptation where the gain of all photoreceptors is adjusted together (A) and chromatic adaptation where the sensitivity of each cone photoreceptor L, M, and S is adjusted individually (B).
Chromatic adaptation is another of the major adaptation processes (Fairchild, 2013). It uses physiological mechanisms similar to those used by light adaptation, but is capable of adjusting the sensitivity of each cone individually, which is illustrated in Fig. 15.2B (the peak of each of the responses can change individually). This process enables the HVS to adapt to various types of illumination so that, for example, white objects retain their white appearance.
Another light-controlling aspect is pupil dilation, which — similarly to an aperture of a camera — controls the amount of light entering the eye (Watson and Yellot, 2012). The effect of pupil dilation is small (approximately 1/16 times) compared with the much more impactful overall adaptation processes (approximately a million times), so it is often ignored in most engineering models of vision.5 However, secondary effects, such as increased depth of focus and less glare with the decreased pupil size, are relevant for 3D displays and HDR displays, respectively.
15.2.2 Steady-State Dynamic Range
Vision science has investigated the SSDR of the HVS through flash stimulus experiments, photoreceptor signal-to-noise ratio and bleaching assessment, and detection psychophysics studies (Baylor et al., 1974; Davson, 1990; Kunkel and Reinhard, 2010; Valeton and van Norren, 1983). The term “steady state” defines a very brief temporal interval that is usually much less than 500 ms, because there are some relatively fast components of light adaptation.
The earliest impact on the SSDR is caused by the optical properties of the eye: before light is transduced into nerve impulses by the photoreceptors in the retina and particularly the fovea, it enters the HVS at the cornea and then continues to be transported through several optical elements of the eye, such as the aqueous humor, lens, and vitreous humor. This involves changes of refractive indices and transmissivity, which in turn leads to absorption and scattering processes (Hubel, 1995; Wandell, 1995).
On the photoreceptor level, the nonlinear response function of a cone cell can be described by the Naka-Rushton equation (Naka and Rushton, 1966; Peirce, 2007), which was developed from measurements in fish retinas. It was, in turn, modeled on the basis of the Michaelis-Menten equation (Michaelis and Menten, 1913) for enzyme kinetics, giving rise to a sigmoidal response function for the photoreceptors, which can be described by
where V is the signal response, is the maximum signal response, L is the input luminance, σ is the semisaturation constant, and the exponent n influences the steepness of the slope of the sigmoidal function.6 It was later found to also describe the behavior of many other animal photoreceptors, including those of monkeys and humans. Later work (Normann and Baxter, 1983) elaborated on the parameter σ in the denominator on the basis of the eye’s point spread function and eye movements, and used the model to fit the results of various disk detection psychophysical experiments. One consequence was increased understanding that the state of light adaptation varies locally on the retina, albeit below the resolution of the cone array.
The SSDR has been found to be between 3 and 4 OoM from flicker and flash detection experiments using disk stimuli (Valeton and van Norren, 1983). Using a more rigorous psychophysical study using Gabor gratings, which are considered to be the most detectable stimulus (Watson et al., 1983), and using a 1/f noise field for an interstimulus interval to aid in accommodation to the display screen surface as well as matching image statistics, Kunkel and Reinhard (2010) identified an SSDR of 3.6 OoM for stimuli presented for 200 ms (onset to offset). Fig. 15.3 shows the test points used to establish the SSDR, as well as an image of the stimulus and background. The basic assumption is that the contrast of the Gabor grating forms a constant luminance interval. When this interval is shifted away from the semisaturation constant (toward lighter or darker on the x-axis), it remains constant on a log luminance axis (ΔL1 = ΔL2). However, the relative cone response interval (y-axis) decreases because of the sigmoidal nature of the function, leading to ΔV1 > ΔV2. The dynamic range threshold lies at the point where the HVS cannot distinguish the amplitude changes of the Gabor pattern.
Figure 15.3. Useful detectability of image features (here Gabor grating). (A) Detectability based on cone response; (B) Achromatic Gabor Grating with a background 1/f noise pattern. Note that the contrast of the grating is amplified for illustration. After Kunkel and Reinhard (2010).
We have now established that the SSDR defines the typical lower dynamic range boundary of the HVS, while adaptive processes can extend the perceivable dynamic range from 10−6 to 108, albeit not at the same time.
However, another outcome of Kunkel and Reinhard’s (2010) experiment is that the SSDR also depends on the stimulus duration, which is illustrated in Fig. 15.4. The shortest duration can be understood as determining the physiological SDDR, and the increase in SSDR (ie, the lower luminance goes lower, and the higher luminance goes higher, giving an increased perceptible dynamic range) with longer durations indicates that rapid temporal light adaptation occurs in the visual system.
Figure 15.4. The dynamic range increases as a function of the display duration of a stimulus. After Kunkel and Reinhard (2010).
Sometimes, the SSDR is misinterpreted as being what is needed for the presentation of a single image, but this ignores the fact that the retina can have varying adaptation states (local adaptation) as a function of position, and hence as a function of image region being viewed. This means that the eye angling to focus on a certain image region will cause different parts of the retina to be in different image-dependent light adaptation states. Or, as one uses eye movements to focus on different parts of the image, the adaptation will change despite the image being static. For the case of video, even more variations of light adaption can occur, as the scenes’ mean levels can change substantially, and the retina will adapt accordingly. Therefore, the necessary dynamic range is somewhere in between the SSDR and long-term-adaptation dynamic range.
Our application of interest is in media viewing, as opposed to medical or scientific visualization applications. While this can encompass content ranging from serious news and documentaries to cinematic movies, we refer to this as the “entertainment dynamic range” (EDR). Further, these applications require a certain level of cost-consciousness that is less pressing in scientific display applications. Video and moving picture applications that use EDR are unlikely to allow for full adaptation processes because of cost and most likely do not require them to occur either. Thus, use of a 14 log10 dynamic range that encompasses complete long-term light adaptation is unrealistic. Nevertheless, they do have strong temporal aspects where the adaptation luminance fluctuates.
To illustrate the necessity of EDR being larger than SSDR, Fig. 15.5 compares common viewing situations (single or multiple viewers) with theater or home viewing settings that address adaptation. In this example, the still image shown can be regarded as a frame in a movie sequence. The dark alley dominates the image area, yet has a bright region in the distance as the alley opens up to a wider street. Depending on where a viewer might look, the overall adaptation level will be different. In Fig. 15.5A, showing a single viewer, the viewer may look from interest area 1 to area 2 in the course of the scene, and that viewer’s light adaptation would change accordingly from a lower state to a higher state of light adaptation. Thus, the use of the SSDR would underestimate the need for the dynamic range of this image, because some temporal light adaptation would occur, thereby expanding the needed dynamic range. One could theoretically design a system with an eye tracker to determine where the viewer is looking at any point in time, and estimate the light adaptation level, combining the tracker position with the known image pixel values. The viewer’s resulting SSDR could theoretically be determined from this.
Figure 15.5. The SSDR is not enough in real-world applications such as with EDR. (A) Single viewers; (B) Multiple viewers.
However, in the image in Fig. 15.5B, multiple viewers are shown, each looking at different regions. Different light adaptation states will occur for each viewer, and thus a single SSDR cannot be used. Even the use of eye trackers will not allow the use of a single SSDR in this case. Having more viewers would compound this problem, of course, and is a common scenario. Therefore, a usable dynamic range for moving/temporal imaging applications (eg, EDR) lies between the SSDR and the long-term dynamic range that the HVS is able to perceive via adaptive processes.
One approach to determine the upper end of the EDR would be to identify light levels where phototoxicity occurs (the general concepts are well summarized in Youssef et al. (2011). A specific engineering study of phototoxicity (Pattanaik et al., 1998) was made to investigate the blue light hazard of white LEDs. The latter is a problem because the visual system does not have a strong “look away” reflex to the short-wavelength spectral regions of white LEDs (because they consist of a strong blue LED with a yellow phosphor). The results show that effects of phototoxicity occur at 160,000 cd/m2 and higher. Rather than actual damage, another factor to consider for the upper end is discomfort, which is usually understood to begin at 30,000 cd/m2 (Halstead, 1993). Well-known snow blindness lies in between these two ranges. Rather than use criteria based on damage or discomfort, another approach is to base the upper luminance level on preferences related to quality factors, which would be lower still than the levels associated with discomfort.7
In consideration of the lower luminance end, there are no damage and likely no discomfort issues. However, the lower luminance threshold would be influenced by the noise floor of the cone photoreceptors when leaving the mesopic range due to dark adaptation or if the media content would facilitate adaptation to those lower luminance levels (eg, a longer very low key scene). Time required to adapt to those lower levels would actually occur in the media. The plot in Fig. 15.6 shows that around 6–8 min of dark adaptation is required to engage the rods, from a higher luminance starting point (Riggs, 1971).
Figure 15.6. The time course of dark adaptation. After Riggs (1971).
15.2.3 Still Image Studies
The vast majority of studies to determine the needed, or appreciated, dynamic range have been for still images. A large body of work has studied viewer preferences for lightness and contrast that seem to be useful for assessment of the HDR parameters. The most recent of these studies (Choi et al., 2008; Shyu et al., 2008) are exemplary as they studied these parameters within a fixed dynamic range window. That is, they studied image-processing changes on fixed displays, not the actual display parameters. Unfortunately, these do not address the question of display range. Further, it is possible to study the dynamic range without invoking image contrast, which can be considered a side effect.
A key article worth mentioning is that by Kishimoto et al. (2010), who concluded there is lower quality with the increase of brightness, maximizing at around 250 cd/m2, thus concluding there was no need for HDR displays at all. The study carefully avoided the problems of studying brightness preferences within a fixed display dynamic range by using the then-new technique of global backlight modulation to shift the range further than previously possible. However, the native panel contrast in their study was around 1000:1 to 2000:1. Consequently, with increasing maximum luminance, the black level rose, so the study’s results are for neither brightness nor range, but are more for the maximum acceptable black level.
Another key study assessed both dynamic range for capture and display using synthetic psychophysical stimuli as opposed to natural imagery (McCann and Rizzi, 2011). Rather than a digital display, a light table and neutral density filters were used in that study to allow a wider dynamic range than the capability of the then-current displays. It used lightness estimates for wedges of different luminance levels arranged in a circle. These wedge-dissected circles were then presented at different luminance levels, and observer lightness8 estimates were used to determine the needed range for display, which was concluded to be surprisingly low at approximately 2.0 log10 luminance. These low ranges were attributed to the optical flare in the eye, and interesting concepts about the cube root behavior of L* as neural compensation for light flare were developed. However, the low number does not match practitioners’ experience, both for still images and especially for video. The most common explanation for their low dynamic range was that the area of the wedges, in particular the brightest ones, was much larger than the brightest regions in HDR display, and thus caused a much higher level of optical flare in the eye, reducing the visibility of dark detail much more than occurs in many types of practical imagery. Another explanation is that their use of lightness estimates did not allow the amplitude resolution needed to assess detectable differences in shadow detail. For video, it was clear their approach did not allow for scene-to-scene light adaptation changes.
In the first study to use an actual HDR display (Seetzen et al., 2006) to study the effects of varying dynamic range intervals (also called “contrast ratio,” CR), the interactions between maximum luminance and contrast were studied. This resulted in viewer estimates of image quality (in this case preference, as opposed to naturalness).
These data are important because they show the problem of black level rising with increases in brightness when the contrast is held constant. One can see this in Fig. 15.7 by comparing the curve for the 2500:1 contrast (black curve) with the curve for a 10,000:1 contrast. For the lower CR, quality is lost above 800 cd/m2, which is due to the black level visibly rising (becoming dark gray) with increasing maximum luminance. The black level rises because of the low CR. On the other hand, the results for the 10,000:1 CR show no loss of quality even up to the maximum luminance studied (6400 cd/m2). With this high contrast, the black level still rises, but it is one quarter the level of that for the 2500:1 CR, so it essentially remains black enough that it is not a limit to the quality.
Figure 15.7. Preferred image quality based on interactions between maximum luminance and contrast. After Seetzen et al. (2006).
Some other key studies for relevant HDR parameters are those focusing on black level (Eda et al., 2008, 2009; Murdoch and Heynderickx, 2012), on local backlight resolution (Lagendijk and Hammer, 2010), and on contrast and brightness preferences as a function of image and viewer preferences (Yoshida and Seidel, 2005). All of these studies have particular problems in determining the dynamic range for moving imagery. Simultaneous contrast has a strong effect on pushing a black level to appear perceptually black, but should not be relied on to occur in all content. For example, dark low-contrast scenes should still be perceived as achieving perceptual black, as opposed to just dark gray. Similar consequences should occur for low-contrast high-luminance scenes. To truly study the needed dynamic range for video, one needs to remove the side effects of image signal-to-noise ratio limits, display dynamic range limits, display and image bit-depth limits, and the perceptual side effects of the Stevens effect and the Hunt effect.
15.2.4 Dynamic Range Study Relevant for Moving Imagery
Extending still image studies to include motion and temporal components in general adds several new challenges for both experimental design and the analysis of the results. As with the design of any psychophysical study, it is important to reduce the degrees of freedom in the assessment to avoid the results being biased by effects other than the ones specified. Daly et al. (2013) listed design guidelines for psychophysical studies using moving imagery on HDR displays as follows:
•
Try to remove all image signal limitations,
•
Try to remove all display limitations,
•
Try to remove perceptual side effects.
Examples of image signal limits include highlight clipping, black crushing, bit-depth constraints (eg, contouring), and noise. Those problems usually appear as a combination of each other potentially amplifying their impact. For example, the dynamic range reported as being preferred can be biased because of the impact of spatial or temporal noise. Increasing the dynamic range of a signal magnifies its noise, which otherwise would be below the perceptual threshold — for example, in standard dynamic range (SDR) content. Thus, the noise becomes the reason that further dynamic range increases are not preferred. To avoid this, the imagery used as test stimuli were essentially noise-free HDR captures created by the merging of a closely spaced series of exposures resulting in linear luminance, floating-point HDR images.
Examples of display limitations include the actual maximum luminance and minimum luminance of the test display, which should be out of the range of the expected preference. One way to achieve this is to allow the display to reach the levels of discomfort. Another example is the local contrast limits, such as when dual modulation with reduced backlight resolution is used. These were largely removed as explained in the articles.
The test images were specifically designed to assess the viewer preferences without the usual perceptual conflicts of simultaneous contrast, the Stevens effect, the Hunt effect, contrast/sharpness, and contrast/distortion interactions as well as common unintended signal distortions of clipping and tone scale shape changes. The Stevens effect and simultaneous contrast effects were ameliorated by use of low-contrast images. While simultaneous contrast is a useful perceptual effect that can be taken advantage of in the composition of an image/and or scene to create the perception of darker black levels and brighter white levels, you do not want to limit the creative process to require it to use this effect. That is, dark scenes of low contrast should also achieve a perceptual black level on the display, as well as scenes of high contrast, and likewise for bright low contrast appearing perceptually white. The images used were color images, to avoid unique preference possibilities with a black-and-white aesthetic, but the color saturation of the objects and the scene was extremely low, thus eliminating the Hunt effect, which could bias preferences at higher luminances as a result of increased colorfulness (which may or may not be preferred). As a result, the images are testing worst-case conditions for the display. Further, the parameters of black level and maximum diffuse white (sometimes called “reference white”) were dissected into separate tests. Contrast was not tested explicitly, as contrast involves content rendering design issues, not display range capability. Note that moving imagery has analogies to audio with its strong dependence on time. In audio, dynamic range is not assessed at a given instant; it is assessed across time (Huber and Runstein, 2009). The same concepts led to study of the maximum and minimum luminances separately, as opposed to contrast in an intraframe contrast framework. The images were dissected into diffuse reflective and highlight regions for the third stage of the experiment.
The dynamic of each tested reflectance image was approximately 1 OoM, as shown in Fig. 15.8. In the perceptual test the images were adjusted by their being shifted linearly on a log luminance axis. The viewer’s task was to pick which of two was preferred. The participants were asked for their own preferences, and not for an estimate of naturalness, even though some may have used those personal criteria. To avoid strong light adaptation shifting, the method of adjustment was avoided, which would result in light adaptation toward the values the viewer was adjusting. Instead, the two-alternative forced choice method was applied with interstimulus intervals having luminance levels and timing set to match known media image statistics, such as average scene cut durations.
Figure 15.8. Illustration of an image having 1.0 OoM (between the 5th and 95th percentile criteria), and being shifted in the log-linear domain in steps of 0.2 OoM. This example is for an image testing for the black level bound of the dynamic range. (A) Range limiting; (B) Shift by constant intervals. After Daly et al. (2013) and Farrell et al. (2014).
For the upper bound of the dynamic range, the upper bounds of diffuse reflective objects and highlights were assessed independently. The motivations for this include perceptual issues to be discussed in Section 15.4 as well as typical hardware limitations, where the maximum luminance achievable for large area white is generally less than the maximum achievable for small regions (“large area white” vs “peak white” in display terminology). The connection between these image content aspects and the display limitations is that in most cases highlights are generally small regions, and only the diffuse reflective regions would fill the entire image frame.
Still images were used, but the study was designed to be video cognizant. That is, the timing of the stimuli and interstimuli intervals matches the 2–5 s shot-cut statistics typical of commercial video and movies. This aspect of media acts to put temporal dampening on the longer-term light adaptations as described previously. Also, keep in mind that video content may contain scenes that are essentially still images with little or no moving content.
The first stage of this experiment determined preferences for minimum and maximum luminance for the diffuse reflective region of images. The results were not fit well by parametric Gaussian statistics, so they were presented as cumulative distributions. The approximate mean value results of these two aspects of the dynamic range were used to set up the second stage of the experiment, which probed individual highlight luminance preferences. Here, the method of adjustment9 approach was applied on segmented highlights (with a blending function at the highlight boundary), while the rest of the image luminances were held fixed, as shown in Fig. 15.9. The adjustment increments were 0.33 OoM. Adaptation drift toward the adjustments would be minor because the highlights occupied a small image area.
Figure 15.9. Highlight preference study via method of adjustment on segmented images. The highlight luminances on the white spoon (A) are remapped in equal intervals while the reflective part of the spoon remains constant. This leads to the scanline plot in (B).
The segmentation was done by hand, while a blending function was used at the highlight boundaries. The luminance profiles within the highlights were preserved but modulated through luminance domain multiplication, which is also shown in Fig. 15.9. For this stage of the study, images from scenes with full color were used, and the highlights were both specular and emissive. For the specular highlights, the color was generally white, although for metallic reflections, some of the object color is retained in the highlights. For the emissive highlights, various colors occurred in the test data set of approximately 24 images.
The study was done for a direct view display (small screen) and for a cinema display (big screen) (Farrell et al., 2014). In both cases, the dynamic range study was across the full range of spatial frequencies, as opposed to being restricted to the lower frequencies by use of a lower-resolution backlight modulation layer (Seetzen et al., 2006). The direct view display had a range of 0.0045– 20,000 cd/m2, while the cinema display had a range of 0.002–2000 cd/m2.
Fig. 15.10 shows the combined results for black level, maximum diffuse white, and highlight luminance levels for the smaller display and the cinema display. The smaller screen is represented by the dashed curve, while the cinema results are shown as the solid curve. Cumulative curves show how more and more viewers’ preferences can be met by a display’s range if this range is increased (going darker for the two black level curves on the left, and going lighter for the maximum diffuse white and highlights curves on the right). While both display types had the same field of view, there were differences in the preferences, where darker black levels were preferred for the cinema versus the small display. Differences between the small-screen and large-screen results for the maximum diffuse white and highlights were smaller.
Figure 15.10. Preferred dynamic ranges for the smaller direct view display and the cinema. DR, dynamic range.
Because the displays had different dynamic ranges, the images were allowed to be pushed to luminance ranges where some clipping would occur (eg, for a very small number of pixels) if the viewer chose to do so. These values are indicated by the inclusion of a double-dash segment in the curves, and via the shaded beige regions. While technically the display could not really display those values (eg, the highlights would be clipped after 2000 cd/m2 for the cinema case), the viewers adjusted the images to allow such clipping. This is likely because there was a perceived value in having the border regions of the highlights increase even though the central regions of the highlights were indeed clipped. For the most conservative use of these data, the results in the corresponding shaded regions (cinema, small display) should not be used.
Fig. 15.10 can be used to design a system with various criteria. While it is common to set design specifications around the mean viewer response, it is clear from Fig. 15.10 that the choice of such an approach leaves 50% of viewers satisfied with the display’s capability, but 50% of the viewers’ preferences could not be met. A basic consequence is that the dynamic range of a higher-capability display (ie, with a larger dynamic range) can be adjusted downward, but the dynamic range of a lower-capability display cannot be adjusted upward.
The results require far higher capability than can be achieved with today’s commercial displays. Nevertheless, they provide guidance when designing and manufacturing products that exceed the visual system’s capability and preferences. Some products can already achieve the lower luminances for black levels, even for the most demanding viewers (approximately 0.005 cd/m2) for the small screen, for example. Rather, it is the capability at the upper end that is difficult to achieve. To satisfy the preferences of 90% of the viewers, a maximum diffuse white of 4000 cd/m2 is needed, and an even higher luminance of approximately 20,000 cd/m2 is required for highlights. At the time of this writing, the higher-quality “HDR” consumer TVs are just exceeding the approximately 1000 cd/m2 level.
Despite the current limits of today’s displays, we can use the data now to design a new HDR signal format. Such a design could be future-proof toward new display advances. That is, if we design a signal format for the most critical viewers at the 90% distribution level, such a signal format would be impervious to new display technologies because the limits of the signal format are determined from the preferences. And those are limited solely by perception, and not display hardware limits. Such a signal format design would initially be overspecified for today’s technology, but would have the headroom to allow for display improvement over time. This is important, because it has already been shown that today’s signal format limits (SDR, 100 cd/m2, Rec. 709 gamut, 8 bits) prevents advancement in displays from being realized with existing signals. It is possible to create great demonstrations with new display technologies, but once they are connected to the existing SDR ecosystems, those advantages disappear for the signal formats that consumers can actually purchase. So with those issues in mind, the next section will address the design of a future-proof HDR signal format that takes into account the results in Fig. 15.10.
A large volume of research exists that addresses the mapping, compression, expansion, and other types of manipulation of the extended luminance range afforded by high dynamic range technologies. However, most of this research focuses exclusively on the luminance dimension, ignoring the chromatic information in images. This is particularly problematic given the parallel push toward extending not only the luminance range but also the chromatic range. In this chapter, we look at solutions that combine the ideas of high dynamic range imaging with color management approaches in different contexts.
16.3.2 High Luminance Conditions and Visual Discomfort
The most important distinction between HDR and LDR is with respect to the luminance range (which in turn leads to HDR). Traditional LDR has a white point which depends on the maximum displayable luminance and contrast ratio (both are usually insufficient to accurately render real-world scenes considering typical LDR displays). Moreover with LDR, the pixel values are typically gamma encoded and perceptually uniform. As a result, a change in the pixel values can be directly related to the change in visual perception. However, with HDR there is more flexibility and one can accurately represent the real luminance (generally up to an unknown scale factor). Consequently, there is no fixed white point in HDR that can correspond to the maximum luminance (as it can vary from scene to scene). There is only brighter (or darker) scene intensity. Of course, in practice, we still need to define a white point for rendering content on HDR displays (because of hardware limitations) but it is typically much higher than in the LDR case. Therefore, HDR viewing will involve higher levels of brightness, in general. Because human vision is sensitive to the luminance ratio (rather than absolute luminance), changes in the luminance may not necessarily lead to the same change in visual perception of HDR. High luminance can also be a source of visual discomfort for observers and should be carefully tackled. Because the HDR display luminance is relatively much higher (eg, the SIM2 Solar HDR display1 has a maximum displayable luminance of 4000 cd/m2) than that of conventional display devices, much higher background illumination is needed to reduce the visual discomfort of observers. Improper settings can result in glare, leading to maladapted viewing conditions. For HDR, there are currently no standard illumination settings. Recommendation ITU-R BT.500-13 recommends the illumination should be approximately 15% of the peak display luminance. With an HDR display, this means approximately 600 cd/m2. But given the reduced sensitivity of the human eye at high luminance, values typically in the range of 150–200 cd/m2 were found to be adequate in our tests (Narwaria et al., 2014d).
The simultaneous dynamic range of the human visual system (HVS) covers a range of approximately four orders of magnitude (Kunkel and Reinhard, 2010). Overall, the HVS can adapt to light conditions with a dynamic range (range of luminance levels) of approximately 14 orders of magnitude (Hood and Finkelstein, 1986; Ferwerda et al., 1996). This range includes the so-called photopic and scotopic vision ranges.
In contrast to the wide range of luminance adaptation exhibited by the HVS, most existing conventional capture and display devices can accommodate a dynamic range of only between two and three orders of magnitude. This is often referred to as “low dynamic range” (LDR). High dynamic range (HDR) imaging has been demonstrated to increase the immersiveness of the viewing experience by capturing a luminance range more compatible with the scene and displaying visual information that covers the full visible (instantaneous) luminance range of the HVS with a larger color gamut (Reinhard et al., 2010). This, however, comes at the cost of much larger storage and transmission bandwidth requirements owing the increased bit depth that is used by most HDR formats to represent this information. Efficient HDR image and video compression algorithms are hence needed that can produce manageable bit rates for a given target perceptual quality. At the time of writing, standardization of compression for wide color gamut and HDR video is still an ongoing process (Sullivan et al., 2007; Winken et al., 2007; Wu et al., 2008; Zhang et al., 2013b; Duenas et al., 2014).
This chapter offers an introduction to the topic of HDR video compression. We review some of the latest HDR image and video coding methods, looking at both layered (backward-compatible/residual-based) and high-bit-depth (native) HDR compression approaches. In the process, we also examine certain HDR-related aspects of the HVS. This chapter assumes a working knowledge of image and video compression. For further details on this aspect, the reader is refereed to Bull (2014).
19.2.2 The Added Value of HDR Displays in Specific Medical Applications
The display’s weaknesses in terms of dynamic range and maximum luminance may be unacceptable in medical applications where a large number of luminance levels and fine details with very small luminance differences need to be discriminated. The display of a high-quality diagnostic image depends on at least three factors: (i) the capability of the detector to provide a large number of levels (bit depth of the acquired data set), (ii) a sophisticated mapping that converts source data into suitable driving levels, and (iii) the visualization of each of them as a distinctly perceived luminance value on the display screen.
A joint and harmonized advance in the whole image chain (acquisition, ie, sensors, image processing/mapping, and image visualization, ie, display) is needed. With regard to the visualization part, even if source data are represented by a large number of bits, the detectability of distinct luminance levels associated with each grayscale step depends on the luminance range of the device. Indeed, if the grayscale steps are too closely spaced, some may fall below the threshold of human perception.
In consumer applications, dynamic range compression or tone mapping techniques allow HDR images to be visualized on conventional displays (Ashikhmin, 2002; Meylan and Suesstrunk, 2006; Mantiuk et al., 2006), but their application on medical images poses serious concerns because the photometric distortion that is intrinsically introduced by the processing can cause the loss of clinically relevant details (Guarnieri et al., 2008b). In the medical field window-and-level adjustment techniques are used, at the price of a long analysis time and with the risk of missing details in the search phase (Yeganeh et al., 2012). Alternatively, an approach was proposed still based on the standard eight-bit gray level resolution but supported by eye-tracking techniques that dynamically process the display image by optimizing the luminance and contrast of the inspected area (Cheng and Badano, 2010).
Recent advances in the display industry have contributed to the development of HDR LCDs with extended luminance range, capable of effectively generating perceivable scales that extend to beyond 14 bits and of reproducing the original image with no theoretical distortion and no loss of information with respect to grayscale values. They will allow radiologists and physicians to perceive subtle and medically significant details in images where there is a large variation of the dynamic range in the data. Various technologies for HDR displays have been proposed (see Part IV of this book and Section 19.4). Regardless of the technological option adopted for the display, much attention has to be devoted to the implementation of the solution because an increased luminance range may come at a price with respect to other image quality parameters, such as increased veiling glare, optical crosstalk, and visual adaptation.
The range of the human eye spans from 10−6 to 108 cd/m2. Standard consumer devices are capable of displaying only a small fraction of the luminance range that humans observe in nature. Typical consumer television sets and computer screens use an input signal with eight bits per color channel, and operate with a peak luminance in the range of about 80–500 cd/m2. Thus, these standard consumer devices are capable of displaying only a small fraction of the high luminance range that humans observe in nature. To create more naturalistic representations of digital images and videos, several technologies have been developed to provide displays with high dynamic range (HDR) capability. The first HDR displays were intended for professional use, but there are also HDR televisions being launched in the high-end consumer market.
Compared with conventional low dynamic range displays, HDR displays have higher peak luminance, higher contrast ratio, and more accurate representation of colors (10–16 bits per color channel). There is no standard definition of HDR displays, but practical displays marketed with HDR capability usually achieve a peak luminance of at least 2000 cd/m2 and a contrast ratio of 10,000:1. However, reported contrast ratios should be treated with some caution, because different methods may have been used for contrast measurements.
HDR displays and projectors based on several different technologies have been developed. At the time of writing, most of the practical HDR displays use liquid crystal display (LCD) technology. LCDs can have high peak intensity, because of the bright and power-efficient light-emitting diodes (LEDs) used as backlight and which are available at a reasonable cost. The main disadvantage of LCDs is the limited local contrast: peak intensity can easily be enhanced by use of brighter backlight, but this will also raise the black level locally, because liquid crystals experience backlight leakage. Better local contrast can be achieved by use of plasma technology, but plasma displays have other disadvantages, such as lower power efficiency, lower peak intensity, and image retention (ghost imaging). Because of these disadvantages, plasma technology has not proven to be a competitive alternative to LCDs in the HDR display market segment.
A promising new HDR display technology uses organic LEDs (OLEDs) as pixels (Forrest, 2003). Since each OLED pixel is a light-emitting unit, backlight is not needed, and a high local contrast ratio can be achieved. OLEDs can also reach high peak intensity at a low power. In terms of production cost and lifetime, OLED technology is not yet competitive with LCDs for large displays, but in the long run, OLED displays are expected to dominate the HDR display market.
In this chapter, our main focus is on HDR display technology based on LCDs using LED backlight, the most prominent type at the moment. To improve local contrast in LCDs, local backlight dimming is an essential technique, allowing different backlight segments to be dimmed separately according to the image content to be displayed (Seetzen et al., 2004). First, we introduce the physical characterization of LCDs with LED backlight, including essential concepts for modeling LCDs, such as leakage, clipping, and basic distortion measures. Then, we present the models in more detail and discuss algorithms based on the models for optimization of the image contrast via local backlight dimming. We also extend our discussion to spatiotemporal characteristics observed in video signals and 3D images, and finally, present methods and results on subjective and objective image and video quality modeling and assessment of LED backlight displays.
The human visual system (HVS) is sensitive to contrast in luminance ranging from a 10−6cd/m2 (objects viewed under illumination from the stars) to 108cd/m2 (objects viewed on a bright sunny day) [1]. However, momentary dynamic range is limited to 4 orders of magnitude. In this range of luminance, a human can see details and at the same time, experiences lower luminance as noise and higher luminance as over-saturated areas. To extend the dynamic range, the HVS has the ability to adapt to changes in the ambient luminance and move the “detailed vision” window along the luminance range of a scene. Interestingly, it mainly adapts to an area covering approximately 2–4 degrees of the viewing angle around the gaze direction [2, 3]. Other areas of the scene, observed not in foveal but in para-foveal, and peripheral regions, have significantly less impact on the adaptation level, although, a human frequently changes gaze direction (even a hundred times per second) and tries to adapt to different regions. As the process of the luminance adaptation is slower than changes of gaze direction, the HVS is permanently in the maladaptation state, in which the adaptation luminance is changing toward a target value, but never reaches this value, because in the meantime, the target is changed.
We dynamically reproduce the process of maladaptation using the scene data stored in high dynamic range (HDR) video. We use an eye tracker to capture the gaze direction of a human observer. Then, we compute the temporal adaptation luminance and use its value to display the HDR image on the low dynamic range (LDR) display. The global tone mapping parameterized by the temporal adaptation can be applied to HDR frames to obtain perceptually correct and plausible tone compression results. For every pixel in the HDR frame, we use the same compression curve which changes over time, following the new values of the adaptation luminance. This technique, maintaining the advantages of the global compression, like lack of video artefacts, halos, or perceptual plausibility of visualization, and simultaneously retains visibility of details in the high dynamic range regions. Advanced local tone mapping algorithms apply different compression curves for every pixel in an HDR image. These techniques preserve visibility of details in the output LDR image, although the excess of details can be distracting for an observer and interpreted as unnatural. The GDTMO technique reveals all the details in the HDR image and at the same time, retains noisy and over-saturated areas at the peripheral region of vision.
We call this approach a gaze-dependent tone mapping operator (GDTMO). This concept was first proposed in Mantiuk and Markowski [4]. In this work, we extend the application from static images to the HDR video. We also introduce a new model of local adaptation proposed in Vangorp et al. [2] and modify the compression curve.
In Section 2 we present an overview of the eye tracking technologies and their limitations related to the computer imaging applications. In Section 3 we explain details of GDTMO and introduce its real time implementation. Section 4 presents results of the tests of this approach and discusses its limitations. We conclude the chapter and suggest directions for further work in the last section.
The range of colours that can be captured or reproduced by an imaging device is called the colour gamut of the device (not to be confused with gamma, the Greek letter used in equations to describe the contrast of a display). This gamut is always smaller than the range of possible colours that can be encountered in a scene, for two reasons.
First, the dynamic range of scene luminances may be very large – 100 000 : 1 or even greater – as in the contrast between shadows and specular (i.e. mirror-like) highlights. While some HDR displays are beginning to approach this luminance range, ordinary CRTs have a range of about 1000 : 1. Even this range is reduced by flare from ambient lighting or reflection from the observer or clothing. The dynamic range of reflection prints is lower still – of the order of 100 : 1.
Second, scenes may include colours of high chroma at any wavelength, originating with fluorescent objects, light sources, or other high chroma objects. When these scenes are captured by a camera with red, green, and blue primaries, or reproduced on a printer using cyan, magenta, and yellow primaries, some compression of these colours must take place; that is, original colours that are close together may be reproduced as the same colour.
In traditional silver halide photographic systems, gamut mapping came about through the natural compression of tone scale in the toe and shoulder of the reproduction curve of each primary; see Fig. 8.7. Digital imaging systems like those incorporating ICC colour management typically take a somewhat different approach. The boundary of the gamut of a device such as a printer is modelled mathematically in a colour space like CIELAB; colours outside this boundary (i.e. unprintable) are then explicitly mapped to colours within the boundary (i.e. printable). Such mapping requires tradeoffs; the mapped colour may have the same hue, chroma, or lightness as the original, but not all three.
8.7. Tone scale compression in photographic imaging.
Most ICC profile-building software varies the tradeoffs for the three intents for printers. Maintaining hue is of foremost importance for the perceptual intent, while chroma may be most important for the saturation intent. A technical problem arises; colours mapped along lines of constant hue angle in CIELAB do not maintain visually-constant hue, especially in the blue region. Makers of profile-building software have made adjustments to solve this problem or handled critical portions of gamut mapping in other colour spaces.
We use cookies which are necessary to make our site work. We may also use additional cookies to analyse, improve and personalise our content and your digital experience. For more information, see our Cookie Policy and the list of Google Ad-Tech Vendors.
You may choose not to allow some types of cookies. However, blocking some types may impact your experience of our site and the services we are able to offer. See the different category headings below to find out more or change your settings.
Manage Consent Preferences
Strictly Necessary Cookies
Always active
These cookies are necessary for the website to function and cannot be switched off in our systems. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, logging in or filling in forms. You can set your browser to block or alert you about these cookies, but some parts of the site will not then work.
Performance Cookies
These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site.
Targeting Cookies
These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. If you do not allow these cookies, you will experience less targeted advertising.
