Enrique Medina Ripoll; José Laparra Hernández; Néstor Arroyo Gómez; José S. Solaz Sanahuja; Noelia Rodríguez Ibáñez*; Sergio Veleff**; José Gerpe Blanco**; Elisa Signes i Pérez

Instituto de Biomecánica (IBV)
*FICOSA
**INTEKIO

25% of traffic accidents in the European Union are directly related to driving under conditions of drowsiness or fatigue; accidents due to these causes are clearly a major social and economic problem. Technological contraptions already exist, that try to prevent them by detecting various factors relating to the drowsiness of the driver. However, these technologies frequently yield false positive results, mainly due to noise caused by environmental and emotional factors. Together with the companies FICOSA and INTEKIO, the Instituto de Biomecánica (IBV) is participating in the development of a new system based on cameras in order to resolve the problems mentioned above.

INTRODUCTION

Apart from its important human and social impact, traffic accidents have a very high cost in economic terms. It is estimated that the cost of each death due to a traffic accident amounts to approximately one million euros, while each injured person represents an expenditure of between 23,000 and 143,000 euros. In the EU, the total annual amount due to traffic accidents exceeds 100 billion euros. Given that one of every four accidents is directly related to driver fatigue or drowsiness, it is estimated that the economic cost of accidents due to these causes ranges between 10 and 24 billion euros each year. Today, technology exists that seeks to address this problem. However, research in this field is still in its infancy and there are myriad technical and scientific approaches yet to be developed that may make significant improvements to today’s drowsiness detection systems.

THE CURRENT SITUATION OF ON-BOARD SYSTEMS FOR THE DETECTION OF DROWSINESS

The systems that have been developed to date in the field of driver drowsiness detection follow one of the following approaches, according to the variables on which they are based:

• Analysis of the behavior of the vehicle. These were the first systems to appear on the market [1]. They record variables such as any deviation from the limits of the lane, the use of the accelerator and the brake or the angle of the steering wheel to detect changes in the driving pattern that may be related to driver fatigue or drowsiness.
• Analysis of the physiological parameters of the driver. These detect signs of drowsiness or fatigue in the driver based on measurements of his or her brain activity, heart rate, respiratory rate [2]–[4], galvanic skin response, [5], [6] eye movement and the degree of opening-closing of the eyelids [7].

Although the techniques that analyze the behavior of the vehicle significantly reduce accidents [1], strictly speaking they cannot be considered as detectors of drowsiness since the patterns they detect can be attributed easily to other causes such as distractions, bad driving habits or simply to a sporty driving style. On the contrary, systems based on the analysis of the physiological parameters of the driver process first-hand information on the condition of that driver. However, these parameters are sometimes difficult to interpret and are often distorted by the influence of other physical or emotional states.

On the other hand, the technology that is used to measure the physiological parameters of the driver has traditionally been very invasive (in direct contact with the driver) [8]. Very invasive systems are admissible in a laboratory environment but not in a real life setting. Therefore, the development of methods for measuring physiological variables in a non-invasive way is a priority, and the systems that are most accepted in the industry are those that are based on image analysis. A good example of this is the PERCLOS system, used quite successfully to analyze the behavior of the eyes of the driver. However, these systems still have a wide margin for improvement, since they are sensitive to lighting and to the use of glasses or contact lenses. For all these reasons, the trend is increasingly to work on detecting drowsiness through other parameters such as the respiratory rate, a method that has been shown to be effective in research carried out by IBV into automobile industry applications. [4].

OBJECTIVE

The objective of this research, carried out within the framework of the "Development of an On-board Camera System for the Detection of Drowsiness" project (co-financed by the CDTI and certificated as an IBEROEKA Project under code number IDI-20150255), is to obtain information that makes it possible to design an on-board system based on image analysis to detect drowsiness at the wheel. The system will record images of unintentional movements of the driver’s chest due to breathing. These images will be processed by an image processing algorithm in order to obtain the respiratory rate and to associate any changes therein with the state of fatigue or drowsiness of the driver.

MATERIAL AND METHODS

In this part of the research, IBV has so far carried out two experiments that are described below.

Experiment 1: Comparison of KINECT with the plethysmography band (respiratory monitoring)

This comparison was made in order to validate KINECT as a non-invasive system based on images with which to measure the respiratory rate. To perform this validation, we were able to call on a sample of subjects without respiratory problems. The images obtained by KINECT were compared with data recorded by a plethysmography band, a highly validated system used to measure chest and abdomen movements (Figure 1). With these results we were able to determine that KINECT was a valid system for measuring the rate of respiration. This validation allowed us to use KINECT as the standard measuring system in Experiment 2.

Figure 1. Depth map registered by KINECT (left) an and actual image of the subject (right).

Experiment 2. Phase 1: Testing the cameras under different conditions

In the first phase of Experiment 2, tests were performed under different conditions, in order to select the best position in which to place the cameras so that they might register the movement of the driver’s chest. We analyzed two camera models designed specifically for automobile applications: PAC16 and FRCAM (Table 1, Figure 2). In order to find the best position in which to place the cameras, we performed tests taking into account different environmental and subject-related factors according to a factorial design of experiments that helped to reduce the tests that we needed to perform (Table 2).

Figure 2. Representation of the positions of the cameras that we tested.

Figure 3. Cameras used in the tests. Camera 1: PAC16 PoC (right) and  Camera 2: FRCAM (left).

Experiment 2. Phase 2: Tests with users in the driving simulator

Once we determined the optimal position for the cameras in the previous phase, we attempted to assess their use as a system for measuring the respiratory rate and to obtain data with which to design the drowsiness detection algorithm. To do so, tests were carried out with users in the IBV automotive laboratory, under controlled lighting, thermal and acoustic conditions. Each subject participated in two sessions: one under normal sleep conditions and the other under conditions of sleep deprivation (without their having slept for at least 24 hours). In the two sessions, we asked the subjects to drive in a virtual scenario for 1 hour 40 minutes. The virtual scenario consisted of a road with mild traffic, smooth curves and a night-time ambience. During the simulation we recorded the subject’s chest movement using the different cameras described above: Camera 1, Camera 2 and KINECT.

 Image analysis and design of processing algorithms

The recordings we obtained were analyzed to design the respiratory rate detection algorithm. The intended operation of the algorithm will be based on the sequential execution of five image processing modules:

• Image enhancement module. The overall contrast is increased by equalizing the histogram, in order to address problems caused by uncontrolled lighting conditions.
• Filter module to isolate noise and to improve image stabilization. In this way, the movement of the car does not affect the detection of the movement of the chest. A multimodal analysis is also performed of the luminance values of the pixels in order to eliminate peaks of light that may be produced by public lighting.
• Motion detection module. This uses techniques that are based on the differentiation of the framework in order to quantify the level of movement. The result is a segmentation of the image in the different regions in which movement is detected.
• Motion detection by segments module. This analyzes the motion detected in each segment of the ones that were defined above, discarding non-regular movements. Semi-regular movement signals with a high correlation are averaged out to produce unique motion signal.
• Module for estimating the respiratory rate. A frequency analysis is made of the signal of the movement using the Fourier transform.

RESULTS

The most relevant results obtained in each part of the experiment that we have carried out to date, may be summed up as:

The validity of Kinect as a respiration measurement system

Figure 4 presents an example of the respiratory signal acquired with KINECT and the reference signal measured simultaneously with the plethysmography band. As can be seen, the peaks of both signals are perfectly synchronized.

Figure 4. Representation of the signals obtained.

Table 2 presents the average respiratory rate for each subject according to both systems of measurement; only slight differences can be appreciated. On the other hand, the analysis of the intraclass correlation coefficient (ICC) shows that both signals present a correlation greater than 0.9 (Table 3). Finally, the reliability of KINECT is demonstrated statistically when comparing the respiratory rate results of both measurement systems by calculating the Cronbach Alpha, which returns a parameter greater than 0.99. (Table 4).

ADAPTING THE CAMERAS TO THE CAR INTERIOR

With regard to the qualitative analysis of the images obtained by each camera in the different test conditions, these results stand out:

Figures 5a and 5b present images that were recorded by camera 1 and 2 respectively under laboratory condition #10 (see Table 1), before and after processing. Camera 2 (FRCAM) detects movement in dark conditions better than camera 1 (PAC16 PoC). This feature is important if one takes into account that a state of drowsiness is much more likely during the night.

Figure 5. Preliminary results of the qualitative analysis.

• Placing cameras close to the chest facilitates the detection of movement. In this way, the image is much more detailed and sharp, patterns and folds appear so that breathing movements produce greater image variation. However, placing cameras in a lateral position (see Figure 2 positions C and D) raises a problem in that they may interfere with the driver’s arm. Depending on how the driver's seat is adjusted (height and displacement) and the natural driving position of the subject, his or her arm may cause interference when focusing on the chest or abdomen (see Figure 5c).
• Recordings from camera 2 conducted under conditions of high illumination with white clothing presented a high level of saturation (see Figure 5d; laboratory condition #9, Table 1). The saturated parts of the image did not provide significant information to help detect movement.

CONCLUSIONS

• The experiments demonstrate that it is possible to measure a driver’s respiratory rate by using images obtained with a single camera.
• The results of the experiments show that it is possible to measure the movement of the chest due to breathing, in a non-invasive way, by analyzing depth maps obtained with low cost infrared cameras (KINECT), with results equivalent to those of an invasive plethysmography band, with both systems returning similar measurements.
• This solution offers significant advantages because it makes it possible to measure the respiratory rate of a driver in a completely non-invasive way. Furthermore, the lack of contact with the body avoids problems of wear and tear and durability.
• It has been shown that the video signal works well at the optimum position, under difficult light conditions (adequate lighting, darkness and bursts of light), thereby solving the main problem presented by the technique that we employed, none other than the false positives that can appear in movement detection, due to non-controlled exterior lighting conditions. When driving at night on roads and streets, public lighting produces periodic changes in luminance on sub-regions of the image. However, the use of adaptive filters as in this instance, can help cancel mechanical signals, if appropriate references are available as to the source of the noise, as has been demonstrated in previous research studies. [4]

ACKNOWLEDGEMENTS

To the companies participating in this initiative: FICOSA and INTEKIO. 
The "Development of an On-board Camera System for the Detection of Drowsiness" project (SOMNOADAS) has received funding from the CDTI (IDI-20150255) and has been certified as an IBEROEKA Project (IB14-775).

BIBLIOGRAPHY

[1]  J. S. Hickman, F. Guo, M. C. Camden, R. J. Hanowski, A. Medina, and J. E. Mabry, “Efficacy of roll stability control and lane departure warning systems using carrier-collected data,” J. Safety Res., vol. 52, pp. 59–63, 2015.

[2]  N. Rodríguez-Ibáñez, M. García-González, M. Fernández-Chimeno, J. Ramos-Castro, and others, “Drowsiness detection by thoracic effort signal analysis in real driving environments,” in Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, 2011, pp. 6055–6058.

[3]  J. Santamaria and K. H. Chiappa, “The EEG of drowsiness in normal adults.,” J. Clin. Neurophysiol., vol. 4, no. 4, pp. 327–382, 1987.

[4]  J. Solaz, H. de Rosario, P. Gameiro, and D. Bande, “Drowsiness and Fatigue Sensing System Based on Driver’s Physiological Signals,” in Transport Research Arena (TRA) 5th Conference: Transport Solutions from Research to Deployment, 2014.

[5]  A. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting driver drowsiness based on sensors: a review,” Sensors, vol. 12, no. 12, pp. 16937–16953, 2012.

[6]  W. Boucsein and W. Ottmann, “Psychophysiological stress effects from the combination of night-shift work and noise,” Biol. Psychol., vol. 42, no. 3, pp. 301–322, 1996.

[7] L. De Gennaro, M. Ferrara, F. Ferlazzo, and M. Bertini, “Slow eye movements and EEG power spectra during wake-sleep transition,” Clin. Neurophysiol., vol. 111, no. 12, pp. 2107–2115, 2000.

[8]  C. Nishimura and J. Nagumo, “Feedback control of the level of arousal using skin potential level as an index,” Ergonomics, vol. 28, no. 6, pp. 905–913, 1985.