Identifying Lines and Interpreting Vertical Jumps in Eye Tracking Studies of Reading Text and Code

Eye trackers are increasingly being used by researchers in software engineering to gain insights into what developers—or experiment participants—are actually doing and thinking [3, 19, 33, 42, 47, 48]. A recurring result is that code, unlike natural language texts, is not read in a largely linear fashion. Rather, participants seem to repeatedly scan the code, and to jump from one place to another.

When analyzing reading behavior, it is important to correctly identify exactly where in the code the participants are looking at each instant. We set out to perform a methodological study of how accurately this can be done. Specifically, we were interested in the tradeoff between realism (trying to be as close as possible to normal working conditions, with many coding elements crowded on the screen) and accuracy (which can be improved by using unrealistically large fonts and spacing). We did this by designing experiments that require participants to focus on only one specific line of text. As we know exactly where they are supposed to be looking, we can analyze the distribution of gazes around this target and calculate the probability of misidentifying it.

Interestingly, these simple experiments led to reading patterns similar to those observed for code. In particular, the participants often jumped from the target line to the question specifying what they were supposed to look for, and back again. In other words, their reading appeared to be inefficient, repeatedly focusing on things that they had just focused on previously (similarly to the repeated scanning sometimes seen in code reading [13, 20, 53]). This suggests that the patterns observed when people perform a task can be inherently different from the patterns they use when reading. When reading, one needs to go over the whole text, which is presented in the order in which it should be read. So reading text is largely linear in “story order.” But to perform a given task that is different from just reading the information in the text, other orders may be used. In particular, the order may reflect the need to orient oneself in the text, and to search for the parts that are relevant for the task. In our case, this sometimes included repeated verification of the line involved in the task and the precise details of the task. We also observe that participants probably use their peripheral vision to home in on targets with visual cues.

We therefore suggest “task order” reading, specifically including orientation and search activities, to generalize the “story order” and “execution order” previously suggested by Busjahn et al. [8]. In addition, we suggest that simple experiments such as ours can be embedded into larger experimental settings, both as a simple recalibration to compensate for drift during an experiment, and to provide objective exclusion criteria for participants who achieve low accuracy.

Eye tracking has been used in reading research for many years. Extremely detailed information about reading is now known, including the distribution of saccade lengths (the distance the eye moves from one fixation to the next), the perceptual span (how much can be seen in a single fixation, typically about eight letters), and the prevalence of regressions (looking back at previously read text, around 10% to 15% of saccades) [6, 39, 40]. Importantly, such attributes of eye movement change to reflect the difficulty of the text being read [41].

Eye tracking was first used to study code reading by Crosby and Stelovsky [13]. One of their findings was that reading code often involved multiple scans of the code. In addition, they found wide differences between participants—for example, employing different numbers of scans. Both findings indicate a departure from reading in a largely linear order, as one may expect for, say, a newspaper story. Subsequent studies have also found that reading code is different from reading text [5, 10, 43]. For one thing, reading code is slower, perhaps due to its complexity. In addition, different code elements receive different levels of attention, depending on their function and location [36]: for example, more emphasis is given to identifiers relative to keywords [9].

Significant research has been focused on the issue of reading order. Regressions in reading natural language text typically involve the previously fixated word, and rarely go to previous lines [6]. But practically all studies of code reading have noticed vertical scans of the code as identified by Crosby and Stelovsky [13]. According to Uwano et al. [53], such scans are a prerequisite for efficient finding of locations of interest in the code. Busjahn et al. [8] found that the reading order also depends on experience: reading by experts is less linear than reading by novices. This result was replicated by Peitek et al. [35]. In addition to scans, a whole catalog of reading patterns has been identified, including jumps back to previously read parts of the code, and jumps forward to preview what is coming [10, 20]. Sharafi et al. [46] show that different reading patterns are used in distinct phases of performing a task, such as when looking for relevant code locations, learning about the code, and editing it.

It is important to note that in computer program code, different lines often have completely different functions: a variable declaration, an assignment, a branching instruction, a function header, and so on. Identifying exactly which line is being read is therefore extremely important to understand what the reader is doing. The ability to distinguish between reading of adjacent lines in eye tracking studies depends on the interaction of four important factors (Figure 1) [32]:

In addition, measurements always include noise. Data from reading 25-line pages of text, which were shown line-by-line to obtain a ground truth for the assignment of fixations to lines, indicates that the combined view paths exhibit erratic behavior and many overlaps [7].

To improve tracking accuracy, experimental recommendations typically suggest that the font used be “large enough” to provide good discrimination, say around a size of 18 pt [47]; one recent study on the parsing of URLs even used a font size of 64 (!) [38]. But even 18 pt is larger than what developers usually use in routine work. This reduces the realism of the study and also limits the amount of code that can be displayed in one screen.

If the font and spacing are not large enough, studies will exhibit data quality problems [16]. These sometimes cause uncertain assignments of fixations to lines, leading to apparent fluctuations between adjacent lines (e.g., see [5, 20, 53]). The detrimental effects of such problems have been recognized previously also in the context of reading natural language texts, and several studies have specifically considered the issue of assigning fixations to lines [7, 11, 30, 49, 58]. However, the developed heuristics are based, at least to some degree, on the assumption that the text consists of lines that are read sequentially. This assumption is often not valid in the context of reading code.

Our initial goal was to study this problem more systematically. We set out to design experiments in which we can measure the vertical distribution of gaze points, as measured by an eye tracker, around a known target. This would enable a precise calculation of the probability to misinterpret the line being looked at as a function of the font size and line spacing. But the results also included unanticipated reading patterns with vertical jumps. These allow us to make some observations on why reading code may be different from reading normal text.

As noted previously, under normal viewing conditions, the vertical resolution may not be adequate to correctly and consistently identify the line being read. This obviously depends on the font size and line spacing. Our experiments were designed to systematically study this effect in practice. The approach was to design tasks that require experiment participants to focus on one specific line in the text. We then collect data on how they find this line, and on how focused they appear to be, given the experimental apparatus and the parameters of displaying the text. We do this in a real experimental setting, as opposed to using an artificial eye [16], as our goal is to characterize what can be achieved in practice and not to assess an eye tracking apparatus.

The collected data required some cleaning and adjustments before it could be used. All our analysis is based on the average of data from the left and right eyes [17]. As noted previously, fixations were computed by the tracker software. We observed no difference between the color options, so the following presentation is based on both together and we do not distinguish between them.

We plotted the gaze and fixation data relative to the lines of text, using time for the horizontal axis and the screen Y coordinate as the vertical axis. In this display the screen X coordinate is not shown, which matches our focus on the vertical dimension. An example is shown in Figure 4.

Initial observations revealed that some data was extremely noisy, and no period of focus on the target line could be identified (Figure 5). Both authors independently reviewed the graphs depicting each participant/text combination and judged which were unusable. There was complete agreement on excluding one participant altogether. Apparently this participant had narrow eyes, so the pupils were partly obscured and the tracker could not identify them reliably; perhaps she was squinting. In addition, six specific texts of other participants were also excluded. Although this exclusion rate is high, it is not unprecedented, and much higher exclusion rates of up to 60% have been reported in eye tracking studies [32].

The heart of the experiment is to measure the distribution of gazes when the participants are supposed to be focusing on the target line. But participants also spend time reading the question and finding the target line. To identify the time period when they are focused on the target line, we recruited an external judge. This judge had previously participated in the experiment, so she had firsthand experience with it and understood the meaning of focusing on the target line. The first author and this judge independently viewed the graphs of all texts for all participants, and noted when they thought the participant focused on the target line. In most cases, they agreed to within 3 seconds, which we consider to be an immaterial difference. There were 11 cases of larger disagreements; in these cases, the second author acted as arbiter.

Given the identification of periods in which it was believed that the participants were focused on the target line, it was obvious that many of the observations suffered from systematic bias. For most participants, the fixations appeared to be located above the target line, sometimes by a considerable margin. For a few, the fixations appeared below the line. We note that this is not unusual, and such bias has also been noticed in the context of developing the iTrace tool for eye tracking when using an integrated development environment [45]. Palmer and Sharif [34] have suggested a methodology for the automatic correction of such a bias, based on creating clusters of fixations, and trying to adjust them so that they fit likely areas of interest in the stimuli. Other algorithms have also been proposed [11, 49, 58].

As we know exactly where the participants are supposed to be looking, we can use a simple and effective strategy. For each displayed text, we find the median height of all the fixations in the focus period and compute the deviation of this height from the center of the target line—similar in principle to the task-embedded calibration suggested by Pi and Shi [37]. We then average these deviations for all the texts displayed to each participant. This produces a participant-specific correction that can be applied to all texts. An example is shown in Figure 6.

We elected to make the correction per participant because it was obvious that different participants had different biases, which could be the result of their individual physiology or posture. The alternative would be to make a separate correction for each text, but it is less obvious why there could be different biases for different texts displayed one after the other using the same device in the same session.

After correcting the systematic bias, we can finally study the distribution of fixations around the target line.

Our main goal from the outset was to characterize the distribution of fixations around the target line. We want to know, during the periods when the experiment’s participants are focused on the target lines, what fraction of the fixations indeed identify the target line.

As we set out to characterize the accuracy of eye tracking in the vertical dimension, a major threat is whether our setup is optimal and representative. One specific concern is that we use a low-cost eye tracker. Although this tracker is apparently not inferior to more expensive models [50], it is still desirable to replicate the experiment with a range of trackers and experimental conditions.

A related concern is dealing with drift and the need for recalibration. Our experiments were not very long, so we decided to settle for correction for systematic bias at the level of individual experimental participants. But this issue too deserves additional research, including the option of using the knowledge of the target line to perform the adjustment per experiment.

Some specific elements in the experiments also suffer from threats. For example, when we study the distribution of the focus around the target line (Section 5.2), there is a risk that some of the fixations represent intentional glances at other locations in the text. We excluded glances at the question using the technical criterion of being below the bottom line of text, but there is no easy way to identify other such glances. So the results concerning the fraction of “indicative” fixations may not correctly reflect all of what the participants were doing.

Another major threat is posed by the need to identify the focus period, when the participants are looking at the target line. We tried to mitigate this by devising a protocol involving both authors and an external judge. However, the judgment of when the focus period starts and ends is still subjective.

Finally, our experimental materials were based on regular text and not code, and on completely synthetic tasks. Using text to learn about code has been done before (e.g., by Marter et al. [28]). In our case, it is justified by the fact that using normal text and such tasks is actually appropriate for the study of the distribution of gazes around a target line. And it enabled us to make observations and theorize about the causes of reading patterns. However, focusing on lines of code may be different due to variability in length and indentation; text, in contradistinction, tends to have a “block shape” with lines that are all similar to each other. In addition, our conjectures regarding reading patterns need to be validated in the context of code comprehension, using carefully designed experiments with more realistic tasks, and also by soliciting intent from the participants.

In addition, contemporary code development is done using integrated development environments, which affects the way code is accessed. This adds a whole new level of complexity to code reading studies [45], similar to the difference between online and physical reading of news [15]. This is a major undertaking that we leave to future work.

Eye tracking studies of code reading are promoted as a window to the mind of the reader (e.g., [3, 13]). They are thought to be useful for identifying mental models used for code comprehension—and specifically whether a top-down or bottom-up approach is being used [1, 3, 9]. Our work indicates that such interpretations should be done with care. We note that analyzing what readers are doing requires a precise identification of the lines they are focusing on, because adjacent code lines may have completely different functions. But this is difficult to achieve with reasonably sized fonts. Moreover, we observe that reading patterns may reflect human behaviors such as the need to reassure yourself in what you are doing, and may not necessarily reflect more significant cognitive processes.

In practical terms, this article makes two immediate contributions. The first is to show that conventional settings of font sizes and line spacing lead to very small angles between adjacent lines, which do not allow for adequate identification of which line is being read. A font size of at least 20 pt and a wide spacing are apparently needed to guarantee that most fixations are assigned correctly, at least for our setup. However, additional research is needed to derive recommendations that are suitable for different eye tracking devices. The second is to devise a simple experiment that can be embedded into larger experimental frameworks. This can be used for two purposes: to augment the conventional calibration procedure and correct accumulated bias, and to provide objective exclusion criteria for participants.

In future work, we intend to design experiments on actual code reading. These experiments will employ different tasks, which require different levels of comprehension [14], and use code with different levels of readability—such as employing variables with concise abbreviated names or long verbose names. Such experiments will expose whether the reading patterns change depending on the task and the code characteristics. We will also debrief the participants after they perform the task, and ask them to explicitly explain their reading pattern as reflected in the scanpath generated by the eye tracker. Independently, we will also analyze the scanpath data to identify different phases of perception, such as exploratory scanning vs. focused study [54, 55]. Such analysis can use the K coefficient of Krejtz et al. [22, 23], which is based on the ratio of fixation durations to saccade lengths.