50 cases from the “Case Records of Massachusetts General Hospital” were selected from a total of 61 cases from 2004/2005. Each case had a documented ‘correct’ diagnosis. Case histories (history, physical examination findings and laboratory test results, but not data from tables and figures) were pasted en bloc as natural text data entry. Findings were compared to the recommended strategy of entering discrete key findings.

Using whole text entry, the correct diagnoses was suggested in 37 of the 50 cases (74%). Using key findings entered manually, the correct diagnosis was suggested in 48 of the 50 cases (96%). The 2 missed diagnoses (progressive multifocal encephalopathy, and nephrogenic fibrosing dermopathy) were not included in the database at the time of the study.

Resident medical officers extracted key clinical and laboratory findings on the basis of admission notes and results of investigations carried out within 30min of admission for patients admitted to a pediatric intensive care unit in a metropolitan hospital in India. The list of diagnoses generated by the Isabel DRS after submission of these terms was collected. The outcome measure studied was the presence of the final diagnosis in the list generated by the Isabel DRS.

200 subjects (boys 111, girls 89, aged 28 days-12 years) were analyzed. Congenital heart disease, respiratory tract infections, meningitis, tetanus and septicemia were the most frequently encountered diagnoses. The Isabel DRS was accurate in 80.5% of the cases.

Results were published in the Indian Pediatrics Journal in November 2005.

Bavdekar SB, Pawar M. Evaluation of an Internet delivered pediatric diagnosis support system (ISABEL) in a tertiary care center in India. Indian Pediatr. 2005 Nov;42(11):1086-91.

Medical students were randomly assigned to either the Isabel group (in which the students used Isabel in addition to their standard resources to generate differential diagnoses for patients) or a control group. Quantitative assessment of benefit was examined by using a standardised post-rotation test given to all students. User feedback and qualitative information was collected by means of a questionnaire and interviews.

43 students were randomised (22 Isabel group, 21 control). 15 students in the Isabel group completed the trial. 12/15 students found the Isabel DRS to be more useful than standard resources; 10/15 students reported that Isabel often or always provided an additional diagnosis not initially considered. All students interviewed agreed that the use of web-based tools should be incorporated into medical education after formal evaluation of such tools. There was no difference in post-rotation test scores between the two groups (ISABEL 73 vs. control 74, p = 0.67), perhaps due to small numbers.

These results were presented at the 15th Annual Pediatric Critical Care Colloquium, NYC in October 2004.

Maffei FA, Nazarian EB, Ramnarayan P, Thomas NJ, Rubenstein JS. Use of a Web-based Tool to Enhance Medical Student Learning in the Pediatric Intensive Care Unit and Inpatient Wards. Ped Crit Care Med. 6(1):109.

Time period: August–December 2000

99 hypothetical case scenarios, and clinical data from 100 real patients, were used to test the performance of the Isabel pediatric DRS. The ‘correct’ or final diagnosis was known for all cases. Hypothetical cases were provided by 12 different pediatricians, and clinical data from real patients were collected from emergency departments at 4 NHS sites. Isabel suggested the ‘correct’ or final diagnosis in 91% of the hypothetical cases, and 95% of the real cases.

A two-person expert panel also provided an ‘optimal’ set of 2-3 diagnoses, for each real case, that juniors would have needed to work up in order to ensure safe decision making. In 73% cases, Isabel displayed all such diagnoses.

Results of this scientific study were peer-reviewed and published in the Archives of Disease in Childhood.

Ramnarayan P, Tomlinson A, Rao A, Coren M, Winrow A, Britto J. ISABEL: a web-based differential diagnostic aid for pediatrics: results from an initial performance evaluation. Arch Dis Child. 2003 May;88(5):408-13

Data were collected from 594 patients (53.4% of screened attendances). Mean age was 49.4 years (95% CI 47.7-51.1) and most patients had significant past illnesses. The majority were assessed first by junior doctors (70%). 266/594 (44.6%) were admitted to hospital. Overall, the diagnostic system displayed the final discharge diagnosis in 95% of inpatients and 90% of 'must-not-miss' diagnoses suggested by the expert panel. The discharge diagnosis appeared within the first ten suggestions in 78% of cases.

Padmanabhan Ramnarayan, Natalie Cronje, Ruth Brown, Rupert Negus, Bill Coode, Philip Moss, Taj Hassan, Wayne Hamer, Joseph Britto. Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emergency Medicine Journal 2007;24:619-624; doi:10.1136/emj.2006.044107

Since Isabel depends on users providing free text input, and relies on them to effectively process the advice suggested to make changes to their decisions, it is important to establish the impact of Isabel’s advice on ordinary clinicians. Even though Isabel may be extremely accurate, it is possible that users may not benefit from its use, due to poor quality input, or lack of confidence in the advice offered. It was important that extraneous factors such as lack of access to the Internet did not affect the study. Hence, it was conducted in an experimental setting.

Due to the presence of a customised thesaurus that converts most medical abbreviations, slang and inaccurate terminology provided by users into appropriate medical terms, Isabel provides users the ability to express clinical features in free natural language text. At the back-end, Isabel’s powerful natural language processing software also extracts key ideas from medical text accurately. These facilities allow users to perform rapid searches. Studies of previous diagnostic systems suggested that users took 20-40 minutes to enter clinical data using a controlled vocabulary.

Berner ES. Diagnostic decision support systems: how to determine the gold standard? J Am Med Inform Assoc. 2003 Nov-Dec;10(6):608-10

The impact of the Isabel DRS on decision-making in acute general pediatrics prompted an examination of the potential utility of the system in a critical care setting. In this setting, where diagnostic decision making was not a large component of practice, it was unclear whether Isabel would be of any benefit.

Although Isabel was primarily developed for use in a secondary care setting, there has been considerable interest in the use of the DRS in primary care. One study that assessed the potential utility of Isabel in primary care, conducted by Dr Claire Scott, examined 1000 consecutive cases of negligence claims.