Li Feifei’s team used “environmental intelligence” to illuminate the dark medical space to the top of “Nature”

After eight years, Li Feifei’s team has released another major scientific research achievement! On September 9, an academic paper jointly signed by Professor Li Feifei of the Department of Computer Science at Stanford University, Professor Arnold Milstein of Stanford Medical School, and his doctoral student Albert Haque appeared in the top academic journal “Nature”.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 1

The paper is titled “Using environmental intelligence to illuminate the dark space of medical care”,Aims to achieve environmental intelligence through the combination of artificial intelligence and non-contact sensors to potentially improve the physical execution of healthcare services.

After the thesis was included in “Nature”, Li Feifei revealed in the circle of friends that this thesis is composed of two generations of doctoral students, more than 10 undergraduate/master/doctoral/postdoctoral, and nearly 10 Stanford Medical School doctors and professors. After 8 years of hard work Built.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 2

At the same time, she also said,

“AI Sensors” (artificial intelligence sensors) will definitely have a profound impact on patient treatment, elderly care and medical services, and our research is only a small step.

When Li Feifei joined the AI ​​medical wave, he focused his attention on “AI Sensors”. In her view, the advancement of smart sensors, AI algorithms and related technologies has brought new possibilities for smart medicine.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 3

Previously, she had conducted in-depth cooperation with Professor Arnold Milstein and tried to introduce smart sensors in the ICU scene to improve the security threats that medical staff might bring due to system inefficiency, high cost, and shortage of staff. This solution is in Several hospitals have conducted tests.

Similarly, this scientific research result is still centered on smart sensors, but its solutions are for the physical space of hospitals and daily life to achieve a wider range ofEnvironmental intelligence

based on”Environmental intelligence“AI medical solutions

The so-called environmental intelligence is an electronic space that can sensitively respond and feedback to human existence through machine learning and non-contact sensors.It has great application value for hospitals and daily life spaces.

For example, in the hospital field, more efficient clinical work processes can be achieved to improve patient safety in intensive care units and operating rooms; in daily life, the management of patients with chronic diseases can be improved by understanding daily behaviors, and the independence of the elderly can be improved.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 4

Specifically, machine learning and sensors can use computers to assist in understanding medical activities and to supplement existing clinical decision support systems. Passive and non-contact sensors can form a kind of environmental intelligence after being embedded in the environment, which can perceive people’s activities and adapt to their continuous health needs. As shown below:

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 5

  • Depth Sensor: Measures the distance of the target object.

  • Temperature sensor (Thermal Sensor): measures the surface temperature.

  • Radio Sensor: Estimate distance and speed.

  • Acoustic Sensor: Measures the sound waves formed by air pressure.

Similar to modern intelligent driving systems, this kind of environmental intelligence can help clinicians and home nurses improve their body movements, which is a key step in modern medical care. Clinical physical action support has achieved better manufacturing, safer autonomous vehicles, and smarter sports and entertainment, and physical space can also transform the rapid flow of biomedicine into error-free healthcare services.

Of course, similar to other technologies, large-scale conversion to clinical applications must overcome challenges such as severe clinical verification, appropriate data privacy, and model transparency.

In the paper, the researchers verified their research algorithms and feasibility and effectiveness through several exemplary clinical use cases and patient results, and further discussed broader social and ethical factors, including issues of privacy, fairness, and transparency. .

AI sensors help multi-dimensional medical space

In 2018, about 7.4% of Americans needed overnight medical staff. In the same year, the National Health Service (NHS) of the United Kingdom reported 17 million hospitalized cases. The problems of medical staff being overworked, understaffed, and limited resources have become quite serious.

Li Feifei’s research team believes that environmental intelligence can play an important role in alleviating the pressure of clinical services, improving the quality and effectiveness of medical services, and can be applied to multiple medical spaces.

  • Intensive Care Unit (ICU)

One use case where environmental intelligence plays a role is computer-assisted patient movement monitoring.

According to statistics, in the United States, intensive care units cost 108 billion U.S. dollars each year, accounting for 13% of total hospital expenses. In critically ill patients, neuromuscular damage may lead to a two-fold increase in the one-year mortality rate and a 30% increase in hospitalization costs. Mobilizing patients to receive environmental intelligence monitoring as soon as possible can reduce the relative incidence by 40%.

At present, face-to-face assessment methods have limitations such as high cost, observer bias, and human error. Non-contact environmental sensors can effectively solve the above problems and provide continuous and accurate patient motion data.

In a pioneering study, researchers installed Ambient Sensors in the ICU room and collected 362 hours of data from eight patients.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 6

Compared with the manual examination by three doctors, the machine learning algorithm divides the patient’s movement into bed activities, out-of-bed activities and walking activities, and its accuracy reaches 87%.

In addition, in another larger test, the researchers installed depth sensors in eight ICU wards, and their algorithm trained a convolutional neural network on 379 videos and divided mobility activities into four categories.

When verified on a sample data set of 184 videos, the algorithm showed 87% sensitivity and 89% specificity.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 7

  • Operating room

Use environmental camera detection algorithms to reduce the frequency of surgical accidents.

Globally, there will be more than 230 million operations each year, of which 14% of patients will have medical accidents. If there is a faster and more effective surgical feedback system, the probability of occurrence can be significantly reduced to 50%.

Environmental cameras are a good way. In a prostatectomy, the researchers trained a convolutional neural network through video to track the needle drive during the operation. The result was an accuracy of 92% compared with the operation of 12 surgeons.

In addition, in a cholecystectomy operation, the researchers used ten methods of resection video to reconstruct the movement trajectory of the instrument during the operation, which can reach the level of an expert surgeon.

Importantly, in the operating room, environmental intelligence is not limited to Endoscopic Videos, but can also be used for item counting. For example, to monitor surgical instruments to prevent them from accidentally staying in the patient’s body, or for computing personnel to track the body parts of the surgical members through a ceiling-mounted camera, the error can be as low as 5cm.

  • Other medical spaces

During or after each patient’s visit, the doctor must keep a record. According to statistics, clinicians spend 35% of their time on sorting out medical documents, which has resulted in shortened medical time for patients and increased management costs.To deal with this problem, Ambient Microphones is an effective solution.

In a study, researchers collected 90,000 conversations between patients and doctors, and conducted deep learning training on the 14,000 hours of outpatient audio generated. As a result, the algorithm showed that the accuracy of word transcription reached 80%. And in terms of clinical practicability, a medical staff found that detecting the microphone on the glasses reduced the time to record documents from 2 hours to 15 minutes, and the time spent with the patient doubled.

From a management perspective, environmental intelligence can also improve activity-based costing. Currently, employee observations, interviews, and electronic health records are used to correlate clinical activities with costs. As mentioned before, environmental intelligence can automatically identify clinical activities, count medical staff, and estimate the duration of activities. However, there is still a lack of environmental intelligence for data verification in cost accounting.

Intelligent solutions for independent living of the elderly

The trend of global population aging is increasing year by year. According to statistics, by 2050, the world’s population over 65 will increase from 700 million to 1.5 billion.

In the absence of children’s care, the daily management of the elderly living alone, including bathing/dressing/eating, chronic disease management, and physical rehabilitation are all very important.

  • Daily self-management

Providing timely clinical care through environmental intelligence can double the ability of daily living and reduce the annual mortality rate. The traditional daily life management method is completed by self-reporting/caregiver’s manual scoring, and there are often subjective biases and untimely measurement problems.

The non-contact environmental sensor can detect the activities of the elderly in a larger range, and can also detect some more subtle clinical data such as heart rate, blood sugar level and respiratory rate.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 8In a study, researchers installed depth sensors and temperature sensors in elderly bedrooms and observed 1,690 activities in a month, including 231 cases of caregiver assistance. The results showed that the accuracy of the convolutional neural network in detection assistance reached 86%.

In a different study, the researchers collected 10-day videos from the homes of elderly people and got similar results. For example, the microphone detects shower and toilet activities, and the accuracy rates are 93% and 91%, respectively.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 9

However, these studies are only test results in a small number of environments, and the space of daily life is highly variable, so there are still certain challenges in wide application. In addition, privacy is also an important issue. If this technology is applied to daily life spaces, its development and verification of privacy security systems are essential.

In addition, another application for the elderly to live independently is Fall Detection.According to statistics, about 29% of community residents fall at least once a year, and if they lie on the ground for more than one hour after a fall, their death rate will increase by 5 times.

For decades, researchers have developed fall detection systems with wearable devices and non-contact environmental sensors. After testing, it was found that the accuracy rate of the wearable device to detect a fall was 96%, and the accuracy rate of the environmental sensor was 97%.

When the two are used in combination, the fall detection accuracy of the depth sensor increases from 90% to 98%, which indicates that there is a potential synergy effect between the non-contact sensor and the wearable sensor.

In addition, the researchers used the depth sensor in 16 elderly apartments for a two-year trial test. The results showed that the sensor produced a false alarm once a month, and the fall detection rate was 98%. At the same time, environmental sensors can provide real-time e-mail alerts to nursing staff in assisted living communities. Compared with other elderly data, its real-time intervention significantly slowed the functional decline of 86 elderly people.

  • Chronic disease management

Gait Analysis is an important tool for diagnosis and measurement of treatment effects in physical rehabilitation and chronic disease management.

Experimental studies have shown that using Accelerometers to estimate the clinical standards (6 minutes and 102 steps) of 30 patients with chronic lung disease has an average error rate of 6%, and the wearable device attached to the body also brings patients inconvenient. In contrast, non-contact sensors can continuously measure gait, improve fidelity, and create interactive home rehabilitation programs.

A study used depth sensors to measure the gait patterns of 9 Parkinson’s disease patients. The study found that the depth sensor can track the vertical movement of the knee with an error of only 4 cm.

In another study, the researchers used depth sensors to make a sports game for patients with cerebral palsy. After 24 weeks of testing, the balance and gait of patients who used the game improved by 18%. And if the microphone is used in combination with a wearable sensor, its gait detection can be increased from 3% to 7%.

  • Mental health

Mental diseases such as depression, anxiety and bipolar disorder affect 43 million adults in the United States and 165 million in the European Union. It is estimated that 56% of adults with mental illness do not seek treatment due to financial problems or access barriers.

Environmental sensors can provide continuous and economical symptom screening methods for detection. In one study, researchers collected audio, video, and in-depth data from 69 individuals in 30-minute semi-structured clinical interviews. Using the speech and upper body movements of the patients in these data, the machine learning algorithm detected 46 patients with schizophrenia with a positive predictive value of 95% and a sensitivity of 84%.

Simultaneously,Environmental sensors can further provide cheaper and higher quality solutions for their psychotherapy. In one study, researchers used microphones and speech recognition algorithms to transcribe and evaluate the methods of heart healers from 200 data sets (both 20-minute interviews). The accuracy rate is 82%.

In general, although through real-time monitoring and feedback, environmental intelligence can reduce accidental clinical errors of medical staff, help patients achieve disease screening and diagnosis, and assist the elderly to improve their ability to take care of themselves in daily life, but its technology is in real-world scenarios and applications, and There are still many challenges and opportunities for wider applications.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 10

Mainly exists at two levels:

  • Recognizing human behavior in complex scenes: Research needs to be conducted across multiple fields of machine intelligence, such as visual tracking, human pose estimation, and human-object interaction models.

  • Dealing with big data and rare events in the clinical environment: This requires new machine learning methods to be able to model rare events and process the big data to be developed.

AI+Medical, privacy protection is the top priority

With the continuous development of artificial intelligence technology, data privacy has increasingly become a sensitive topic. Li Feifei’s team stated that it developed this technology with privacy and security in mind, not only in terms of the technology itself, but also in the continuous participation of all stakeholders in the development process.

The figure shows some existing and emerging privacy protection technologies. One method is to de-identify the data by deleting the individual’s identity. Another method is data minimization, which minimizes data capture, transmission, and personnel bycatch. When a ward is empty, the environmental system detection may be suspended, but even if the data is cancelled, the individual can be re-identified. Super-resolution technology can partially reverse the effects of facial blur and dimensionality reduction technology, making it possible to realize re-identification. This indicates that the data should be kept on the device to reduce the risk of unauthorized access and re-identification.

Li Feifei's team used "environmental intelligence" to illuminate the dark medical space to the top of "Nature" 11

In addition, some healthcare organizations still share patient information with third parties such as data agencies. To alleviate this situation, patients should take the initiative to ask healthcare providers to adopt privacy protection measures. In addition, clinicians and technicians must collaborate with key stakeholders (for example, patients, family members, or caregivers), legal experts, and decision makers to develop a governance framework for the environmental system.

In addition to privacy, the Li Feifei team also considered three other aspects of the trustworthiness of artificial intelligence, including fairness, transparency, and research ethics. However, they said that solving the above four types of factors requires close cooperation between experts in the fields of medicine, computer science, law, ethics, and public policy.

For more content, please refer to the paper: