Hostname: page-component-7c8c6479df-5xszh Total loading time: 0 Render date: 2024-03-27T06:11:31.939Z Has data issue: false hasContentIssue false

Paramedics assessing Elders at Risk for Independence Loss (PERIL): Derivation, Reliability and Comparative Effectiveness of a Clinical Prediction Rule

Published online by Cambridge University Press:  18 March 2016

Jacques S. Lee*
Affiliation:
Sunnybrook Research Institute, Toronto, ON Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON
P. Richard Verbeek
Affiliation:
Sunnybrook Research Institute, Toronto, ON Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON
Michael J. Schull
Affiliation:
Sunnybrook Research Institute, Toronto, ON Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON
Lisa Calder
Affiliation:
Department of Emergency Medicine, University of Ottawa, Ottawa, ON The Ottawa Hospital Research Institute, Ottawa, ON
Ian G. Stiell
Affiliation:
Department of Emergency Medicine, University of Ottawa, Ottawa, ON The Ottawa Hospital Research Institute, Ottawa, ON
John Trickett
Affiliation:
The Ottawa Hospital Research Institute, Ottawa, ON
Laurie J. Morrison
Affiliation:
Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON Rescu, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, ON
Michael Nolan
Affiliation:
County of Renfrew Paramedic Services, Pembroke, ON
Brian H. Rowe
Affiliation:
Department of Emergency Medicine, University of Alberta, Edmonton, AB School of Public Health, University of Alberta, Edmonton, AB
Sunil Sookram
Affiliation:
Department of Emergency Medicine, University of Alberta, Edmonton, AB
David Ryan
Affiliation:
Regional Geriatric Program of Toronto, Toronto, ON
Alex Kiss
Affiliation:
Sunnybrook Research Institute, Toronto, ON
Gary Naglie
Affiliation:
Department of Medicine and Rotman Research Institute, Baycrest Health Sciences, Toronto, ON Department of Medicine and Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON
*
Correspondence to: Dr. Jacques S. Lee, Sunnybrook Health Sciences Centre, BG-04, 2075 Bayview Avenue, Toronto, Ontario, M4N 3M5; Email: jacques.lee@sunnybrook.ca

Abstract

Objectives

We conducted a program of research to derive and test the reliability of a clinical prediction rule to identify high-risk older adults using paramedics’ observations.

Methods

We developed the Paramedics assessing Elders at Risk of Independence Loss (PERIL) checklist of 43 yes or no questions, including the Identifying Seniors at Risk (ISAR) tool items. We trained 1,185 paramedics from three Ontario services to use this checklist, and assessed inter-observer reliability in a convenience sample. The primary outcome, return to the ED, hospitalization, or death within one month was assessed using provincial databases. We derived a prediction rule using multivariable logistic regression.

Results

We enrolled 1,065 subjects, of which 764 (71.7%) had complete data. Inter-observer reliability was good or excellent for 40/43 questions. We derived a four-item rule: 1) “Problems in the home contributing to adverse outcomes?” (OR 1.43); 2) “Called 911 in the last 30 days?” (OR 1.72); 3) male (OR 1.38) and 4) lacks social support (OR 1.4). The PERIL rule performed better than a proxy measure of clinical judgment (AUC 0.62 vs. 0.56, p=0.02) and adherence was better for PERIL than for ISAR.

Conclusions

The four-item PERIL rule has good inter-observer reliability and adherence, and had advantages compared to a proxy measure of clinical judgment. The ISAR is an acceptable alternative, but adherence may be lower. If future research validates the PERIL rule, it could be used by emergency physicians and paramedic services to target preventative interventions for seniors identified as high-risk.

Résumé

Objectifs

Les auteurs ont mené un programme de recherche afin de dériver une règle de prévision clinique et d’en vérifier la fiabilité, règle qui permettrait de reconnaître les personnes âgées fortement prédisposées à la perte d’autonomie à l’aide des observations des ambulanciers paramédicaux.

Méthode

Les auteurs ont d’abord élaboré une liste de vérification, la Paramedics assessing Elders at Risk of Independence Loss (PERIL), qui comprenait 43 questions à répondre par oui ou par non, dont les éléments de l’outil Identifying Seniors at Risk (ISAR). Par la suite, 1185 ambulanciers paramédicaux provenant de trois services en Ontario ont été formés pour utiliser la liste de vérification, après quoi a été évaluée la fidélité interobservateurs dans un échantillon de commodité. Le principal critère d’évaluation, soit le retour au service des urgences, l’hospitalisation ou la mort en l’espace d’un mois, a été évalué à l’aide de bases de données provinciales. Enfin, les auteurs ont procédé à la dérivation d’une règle de prévision clinique à l’aide de modèles de régression logistique à plusieurs variables.

Résultats

Sur 1065 participants, 764 (71,7 %) ont fourni des données complètes. La fidélité interobservateurs était bonne ou excellente dans 40 questions sur 43. Par la suite a été dérivée une règle en quatre points : 1) Y a-t-il des problèmes à la maison qui contribuent aux résultats défavorables? (risque relatif approché [RRA] : 1,43); 2) La personne a-t-elle composé le 911 au cours des 30 derniers jours? (RRA : 1,72); 3) S’agit-il d’un homme? (RRA : 1,38); et 4) Le soutien social est-il déficient? (RRA : 1,4). La règle PERIL a donné de meilleurs résultats qu’une mesure de substitution du jugement clinique (surface sous la courbe : 0,62 contre 0,56; p=0,02) et l’observance du questionnaire était plus élevée pour la règle PERIL que pour l’outil ISAR.

Conclusions

La règle PERIL en quatre points offre une bonne fidélité interobservateurs et une bonne observance, et elle présente des avantages comparativement à la mesure de substitution du jugement clinique. L’outil ISAR est une solution de rechange acceptable, mais l’observance peut être plus faible que pour la règle PERIL. Si la validité de la règle PERIL était confirmée plus tard dans la recherche, elle pourrait être utilisée par les médecins d’urgence et les services ambulanciers paramédicaux pour cibler des interventions préventives chez les personnes âgées considérées comme fortement prédisposées à la perte d’autonomie.

Type
Original Research
Copyright
Copyright © Canadian Association of Emergency Physicians 2016 

Introduction

Many older adults are at an increased risk for preventable events such as falls, delirium, and functional decline.Reference Fried, Ferrucci and Darer 1 One in five seniors lack the support they need to function daily.Reference Desai, Lentzner and Weeks 2 Gurley et al. chillingly captured the potential severity of the problem: those found helpless or dead accounted for 3.7% of calls to emergency medical services (EMS).Reference Gurley, Lum and Sande 3 This population-based study showed that the rate of being “found down” increased dramatically with age, from 3/1,000 person years in the 60–64 age group, to 27/1,000 person years among patients 85 and older.Reference Shah, Glushak and Karrison 4 More than 35% of those found down had been incapacitated for over 12 hours; those found down after 72 hours had a mortality of 67% during their hospitalization.Reference Gurley, Lum and Sande 3 Clearly, there are isolated vulnerable older people living in the community with unmet needs who could benefit from a safety net.Reference Desai, Lentzner and Weeks 2

Due to social isolation and a lack of support and timely access to primary care, such frail older adults frequently depend on EMS to function as their safety net.Reference Shah, Glushak and Karrison 4 This may explain why, despite their increased disease burden, the proportion of non-urgent calls increases with age and is twice as common among patients 65 years and older compared to younger adults.Reference Clark and FitzGerald 5 Up to one-third of ambulance calls among older adults are for assistance with mobility (e.g., being unable to rise from the toilet or after a non-injurious fall).Reference Shah, Glushak and Karrison 4 Older adults are more likely to call 911 on multiple occasions and to decline transportation to hospital after calling 911, compared to younger adults.Reference Marks, Daniel and Afolabi 6 Repeat 911 calls account for 18% to 40% of ambulance calls among people 65 years of age or older.Reference Weiss, Ernst and Miller 7 Thus, an older adult calling 911 for a seemingly minor problem may be a “sentinel” event, signaling a diminished ability to continue functioning independently.Reference Bernstein 8 However, interventions to prevent adverse outcomes such as falls,Reference Tinetti, Baker and McAvay 9 hospitalization,Reference McCusker, Jacobs and Dendukuri 10 and functional declineReference McCusker, Jacobs and Dendukuri 10 have not been shown to be either effective or economical when applied broadly to all older adults.Reference McCusker, Jacobs and Dendukuri 10 - Reference Wilber and Gerson 12 Interventions must be targeted at high-risk individuals to be effective.Reference Miller, Lewis and Nork 13 , 14 The Identifying Seniors at Risk (ISAR) tool was developed to identify high-risk older patients in the ED, but has never been validated for use by paramedics.Reference McCusker, Bellavance and Cardin 15

Given their frequent encounters in the homes of older adults, and the substantial proportion of older EMS patients who are not transported to ED, paramedics may be uniquely positioned to identify those at risk for adverse outcomes.Reference Gerson, Schelble and Wilson 16 This potential has increasingly been recognized by the creation of “community paramedicine” programs,Reference Ball 17 - Reference Weiss, Chong and Ong 20 but paramedics have historically received little specialized geriatric training to help them identify older adults at risk for adverse outcomes.Reference Wilber and Gerson 12 Thus we sought to develop the “Paramedics assessing Elders at Risk for Independence Loss” (PERIL) clinical prediction rule to permit paramedics to identify high-risk older adults.

Our specific objectives were: 1) to test the reliability of 43 candidate questions for a clinical prediction rule; 2) to derive the PERIL clinical prediction rule, a checklist for use by paramedics to assess whether an older adult is at high risk for repeat ED visits, hospitalization or death in the month following an EMS encounter; 3) to compare our prediction rule to a proxy measure of paramedic global assessment of risk; and 4) to compare our prediction rule to the existing ISAR prediction tool when employed by paramedics in the prehospital setting.

Methods

Study design

This program of research included the development of potential variables, reliability testing, and the derivation of the PERIL clinical prediction rule following methodological standards for decision rules.Reference Stiell 21 A pool of 43 candidate predictor variables was selected based on three published systematic reviews.Reference Aminzadeh and Dalziel 11 , Reference Grief 22 , Reference McCusker, Karp and Cardin 23 Our aim was to ensure that the rule was “clinically sensible,” i.e., all required data elements would be available to paramedics and feasible to collect during the EMS encounter.Reference Beveridge, Ducharme and Janes 26 Development of the data collection form was guided by an expert panel composed of a clinical epidemiology and decision rule expert, two geriatricians, and a paramedic with research experience, as well as paramedic focus groups (see Figure 1). All items were framed as simple yes or no questions. One question was included to act as a proxy measure of paramedics’ global assessment of risk for adverse outcomes: “Given the current home situation, are there any problems that would prevent this client from being safely discharged home from the ED, or contribute to recurrent EMS/Emergency use?” We recognized that when completing our proxy variable of clinical judgment, paramedics in the study would have the benefit of the training they received and the completion of the 42 other variables at the same time. This meant that our proxy measure of clinical judgment would be a “conservative” comparison standard for the performance of the PERIL rule relative to truly unaided clinical judgment by untrained paramedics. We also included all questions from the Identifying Seniors at Risk (ISAR), and trained paramedics in how to use the ISAR questions.Reference McCusker, Bellavance and Cardin 24

Figure 1 PERIL (Paramedics assessing Elders at Risk for Independence Loss) data collection form.

We then developed a curriculum to educate paramedics about the study rationale, objectives, and use of the PERIL data form.Reference Lee, Verbeek and Morrison 25 We used existing one-hour EMS continuing educational sessions to train 1,185 paramedics, including advanced and basic care providers. Paramedics received a didactic lecture about the presentation of common geriatric syndromes and signs of reduced function evident in the home setting. Paramedics then viewed a video-taped simulated encounter between a paramedic crew and an older patient, and recorded any problems they observed. Finally, we conducted a prospective observational cohort study to derive the PERIL clinical prediction rule. We assessed inter-rater reliability for all candidate predictor variables in a convenience sub-sample when two paramedics were able to complete the PERIL data for the same patient.Reference Lee, Verbeek and Morrison 25

Study participants

Eligible patients of three Canadian EMS systems, including Toronto, Ottawa, and Renfrew County, Ontario, were 65 years of age and older and assessed in their home by a paramedic responding to a 911 call. Recruitment occurred from January 1, 2005 to March 15, 2008. We excluded patients who were critically ill, as determined by the Canadian Triage Assessment ScaleReference Beveridge, Ducharme and Janes 26 (CTAS category 1); who lived in a nursing home, as determined by a list of such institutions for each community; or who were attended to in a public place rather than in the their home, because our hypothesis was that paramedics could observe clues of declining function in the patient’s home.

Methods of measurement

Baseline measures

Baseline demographic measures included patients’ age and sex. Information abstracted from the paramedic Ambulance Call Record (ACR) included the reason for the 911 call, chronic medical conditions, regular prescription medications, vital signs, and whether the patient was transported to an ED or not.

PERIL data-collection form

The standardized PERIL checklist comprised of 38 dichotomous variables (see Figure 1) that paramedics observed when attending to an older adult in the patient’s home. The five ISAR items were based on asking the patient questions instead of direct observations, for a total of 43 items (See Figure 1).

Outcome measures

Our primary composite measure of adverse outcomes was based on previous studies looking at adverse outcomes in older ED patients.Reference McCusker, Bellavance and Cardin 24 , Reference McCusker, Bellavance and Cardin 27 , Reference Meldon, Mion and Palmer 28 It included any repeat visit to the ED, hospitalization, or death within 30 days of the initial EMS encounter. We excluded adverse outcomes that were related to the initial EMS call, i.e., occurred on the same day as the initial EMS call with no ED discharge occurring prior to the adverse event. Patients’ data were linked to provincial administrative databases, including the National Ambulatory Care Reporting System, the Discharge Abstract Database, and the Registered Persons Database. We used deterministic matching for 79% of the cohort where Ontario Health Insurance Plan (OHIP) numbers were available and probabilistic matching based on name and date of birth for the rest of the cohort. These datasets were linked using unique, encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences (ICES).

Statistical analysis

Sample size calculation

We estimated that a sample size of 1,000 older adults would provide sufficient power to examine a minimum of 35 predictive variablesReference Peduzzi, Concato and Kemper 29 and detect odds ratios of 1.6 or greater, assuming a power of 80%, alpha level of 0.05, and adjusting for an r-squared of 0.4 between the independent variables in the model, and a primary outcome rate of 35% observed in our pilot data.Reference Peduzzi, Concato and Kemper 29 , Reference Hsieh, Bolock and Larsen 30

Model derivation

We chose logistic regression modeling over recursive partitioning to derive and maximize the specificity of the PERIL prediction rule,Reference Cook and Goldman 31 given the limited resources available for frail older adults. A highly sensitive rule with low specificity would identify too many “false positive” individuals who would consume the limited available resources without actually requiring them. We used Harrell’s technique to build logistic regression models.Reference Harrell 32 In summary, only clinically relevant predictor variables were chosen and variables were not screened based on univariate association with the primary outcome to protect against type-I error. We refined the list by excluding predictors that lack variables with greater than 95% of observations in a single category, or variables with poor inter-rater agreement (κ<.60). We used correlation matrix analysis to assess for high multicollinearity (VIF > 2.5), and if variables showed high multicollinearity, we chose the most clinical relevant variable, or ran bootstrap simulations for sets of equally clinical relevant variables with high multicollinearity.Reference Efron and Tibshirani 33 We then built candidate predictive models using backward stepwise regression and selected the final model based on predictive ability, ease of use, and parsimony (i.e., fewest number of variables). Receiver Operating Curve (ROC) of the final PERIL prediction rule, paramedic clinical judgment, and the ISAR tool were also reported and areas under the curve (AUC) were compared.Reference Lee, Schwindt and Langevin 34

Ethics

The Sunnybrook and Ottawa Hospital Research Ethics Boards approved the study. As a minimal risk observational study, the requirement for written informed consent was waived.

Results

Characteristics of study subjects

Paramedics screened 1,418 older adults, and 1,065 (75.1%) were eligible. All 1,065 eligible subjects were linked with provincial databases to acquire patient outcomes and 764 (71.7%) had complete data allowing logistic-regression (see Figure 2). There was no difference in adverse outcome rates between patients with and without complete data (269/764 or 35.2%, 95% CI 31.8% to 38.7%, for subjects with complete data, vs. 372/1,065 or 34.9% , 95% CI 32.1% to 37.9%, among subjects missing at least one variable). Their mean age was 80.9 years (interquartile range 75.3 to 86.3 years), 297 (38.9%) were male and 54 (7%) were not transported to hospital (see Table 1).

Figure 2 PERIL derivation study flow chart.

Table 1 Demographic characteristics of eligible subjects in the cohort used to derive the PERIL decision rule

Note: CI=confidence interval; CTAS=Canadian Triage and Acuity Scale.

Primary outcome

A total of 269 individuals experienced 291 adverse outcomes, for a primary outcome rate of 35.2% (269/764). The three components of the primary outcome broke down as 169/764 (22.1%) older adults who were hospitalized, 89/764 (11.6%) who had a subsequent ED visit, and 33/764 who died after the initial EMS encounter (4.8%). All adverse events occurred within 30 days of the initial call to 911. Of the total 291 composite adverse outcomes, 18 older adults experienced two events and two experienced three events.

Selection of variables

Five variables were answered “yes” in less than 5% of patients, and were excluded from analysis: home too cold, chronic renal failure, agitation, delusions, and suspected elder abuse. Given the extremely low frequency and their weak univariate association with our primary outcome, they did not influence the final model in a sensitivity analysis. Two sets of variables showed multicollinearity: a) “previous 911 use” and “previous ED use”; and b) the three variables “lacks help needed to bathe/shower,” “lacks help needed to get to toilet,” and “lacks help needed to clean living quarters.” For the first sets of collinear variables, we selected “previous 911 use,” given our focus on prehospital screening. For the second set of collinear variables, “lacks help needed to bathe/shower” was selected because it showed the strongest association with the outcome in bootstrap simulation analysis in the absence of any clear superiority clinically. Thus a total of 38 of the original 43 variables were entered into the logistic regression model (see Table 2).

Table 2 Inter-observer reliability, adherence and univariate association with adverse outcomes of 38 potentially predictive variables included in the logistic regression analysis

Reliability and adherence

There was “good” (kappa ≥0.6) to “excellent” (kappa ≥0.8) reliability for 40/43 of the questions.Reference Landis and Koch 35 Among the 38 questions retained for logistic regression, reliability was “good” to “excellent” for 37/38 questions and “substantial” for one question (history of chronic medical conditions—see Table 2). Thus no questions were excluded on the basis of poor reliability or kappa <0.6. Paramedic adherence in completing individual questions is also listed in Table 2. In addition, rates of missing data among the 38 questions ranged from 0 to 328/1065 (30.8%). The ISAR variables that required paramedics to question the patient directly were missing in 27% of cases or more, whereas variables based on paramedic observations had high adherence rates (see Table 2).

Model derivation

Four significant independent predictors remained after model selection: 1) 911 use in the past 30 days; 2) “Given the current home situation, are there any problems that would prevent this client from being safely discharged home from the ED, or contribute to recurrent EMS/Emergency use?”; 3) overall lack of social support, defined as “Patient has unmet needs for social support that, in your opinion, will contribute to recurrent EMS/Emergency use”; and 4) male sex (see Table 3).

Table 3 The Paramedics assessing Elders at Risk for Independence (PERIL) Rule

Note: ED=Emergency Department; EMS=emergency medical services.

The four-item PERIL prediction rule was significantly associated with the composite adverse primary outcome measure, and the Hosmer-Lemeshow statistic failed to show any lack of fit (χ2=2.51, df=7, p=0.93). The PERIL prediction rule had an AUC of 0.62, compared to the AUC of 0.57 for clinical judgment as captured by the single proxy variable of paramedic overall risk assessment (p=0.02). Since the odds ratios for all variables in the final predictive model were approximately equal, we constructed the PERIL prediction rule, with one point assigned for each positive question resulting in a maximum of four points, with a range of one to four points. (See Table 4). The sensitivity and specificity for all cut-off values are listed in Table 5.

Table 4 Predictive performance of the Paramedics assessing Elders at Risk for Independence (PERIL) clinical prediction rule

Note: CI=confidence interval; ED=emergency department; EMS=emergency medical services.

Table 5 Comparing adverse outcomes, sensitivity and specificity of the Paramedics assessing Elders at Risk for Independence Loss (PERIL) compared to single global risk assessment variable

Note: ED=Emergency Department; EMS=emergency medical services.

Finally, we compared the predictive performance of the ISAR to the four-item PERIL rule in a subsample of 633 subjects with complete ISAR variables. The AUC for the ISAR was similar to the PERIL tool in this subsample (0.57 vs. 0.61 respectively, p=0.28).

Discussion

This large multi-centred observational study demonstrated the feasibility of Canadian paramedics identifying older adults at high risk of future adverse events using a clinically sensible prehospital rule, i.e., one that uses information that can be reasonably collected by paramedics at the time of the 911 call. We found that paramedics could reliably observe data needed for the PERIL rule when attending to older patients; inter-observer reliability was substantial to excellent for all variables considered for the model.

This study adds to the evidence supporting the feasibility of community paramedicine, a model of care whereby paramedics apply their training and skills in “non-traditional,” preventative, community-based roles.Reference Mason, Knowles and Colwell 19 - Reference McCusker, Bellavance and Cardin 24 Gerson et al. first studied screening by paramedics for clues of unmet needs in the home environment of older adults.Reference Dixon, Mason and Knowles 18 In 121/124 cases where paramedics identified older adults as having unmet needs, this was confirmed during a home-visit by a “gold-standard” geriatric assessor from a community agency. Gerson et al. also reported that 91% of the paramedics supported the screening of vulnerable elders and wished to continue doing so.

Paramedic ability to identify high-risk older adults using unaided clinical judgment had not been studied, so we assessed whether a single proxy variable assessing paramedic global assessment of risk would be sufficient to identify high-risk older adults. Addition of the three other PERIL questions had superior predictive ability compared to our single proxy variable of “unaided” clinical judgement (see Table 5). An additional advantage of the four-question prediction rule, over the single proxy measure of paramedic judgment alone, is that it allows the use of different cut-off levels tailored to the needs of the paramedic service (see Table 5). The specificity achieved with the single proxy variable of clinical judgement was 41.5%, whereas specificity was improved to 83.8% when a cut-off of three was chosen for the PERIL rule (see Table 5). High-volume systems that need to limit the number of referrals made may choose a more specific threshold. In low-volume rural settings, a more sensitive threshold can be chosen. It should also be noted that participating paramedics had undergone 60 minutes of training on assessing the 42 other risk factors for adverse outcomes, and this may have improved the predictive ability of our proxy variable compared to truly “unaided” clinical judgement among untrained paramedics.

McCusker et al. were the first group that followed methodological standards for developing an ED-based clinical prediction rule targeting older adults. The five-item ISAR tool was originally designed to identify ED patients over 70 at high risk for adverse events within six months of the original ED visit. The adverse events observed included functional decline in 54%, death in 35%, and nursing-home admission in 10.8%. The ISAR showed an AUC of 0.71, sensitivity of 72%, and specificity of 58% for this composite outcome.Reference Harrell 32 The data from the original study were subsequently used to assess the ability of the ISAR to predict return to the ED within 30 days; the predictive performance for early return to ED was substantially lower (AUC 0.63).Reference McCusker, Bellavance and Cardin 27 Our literature search revealed that our study is the first to examine use of the ISAR by paramedics.

While there was no statistically significant difference in the predictive performance of ISAR compared to the PERIL rule, the completion of the ISAR questions was lower—all five of the ISAR questions were missing in more than 27% of eligible patients. In contrast, paramedics completed the three observation-based PERIL items nearly 100% of the time. Thus, paramedics provided sufficient data to compute a meaningful score based on three items of the PERIL score nearly 100% of the time.

Our composite outcome was chosen based on existing ED screening rules, and was intended to represent adverse outcomes that might be prevented by appropriate preventative management. There is some debate as to whether return ED visits should be considered as an adverse outcome, as McCusker has shown that most ED visits by older people are appropriate.Reference McCusker, Karp and Cardin 23 One approach is to look at whether repeat ED visits are for the same condition. However, it is problematic to use administrative databases such as NACRS to capture whether a return visit is for the same problem. For example, a patient with non-cardiac chest pain could have several ICD-10 discharge codes, even if they returned with a similar complaint. However, 169/269 (62.8%) of adverse outcomes in this study were subsequent hospitalization, and only 89/269 (33.1%) were for return to the ED.

This study has several limitations. First, we were unable to collect data on patients paramedics failed to screen due to problems with paramedic system information systems at the time of the study, and thus we have no data on the potential number of patients that were not screened. There was a further loss of 28.3% of eligible subjects due to missing variables. Clearly, our data show that paramedics are more likely to complete observation-based items, as opposed to items that require paramedics to ask patients additional questions that were outside their usual workflow. The only variable in the final PERIL prediction rule that was collected directly from the patient was “previous 911 use,” which was asked of the patients in 71.8% of cases. The recent adoption of electronic patient care records by many paramedic and EMS services will allow for the automated collection of previous 911 use in the future, and improve the accuracy and capture rate of this variable.

The predictive performance of the PERIL prediction rule was only moderate, with an area under the curve of 0.62, and low sensitivity. However the PERIL rule was designed to maximize specificity over sensitivity, to avoid patients falsely identified as being at high risk. We chose this strategy because in most Canadian health care systems, there is more demand for community health care services than supply, and the current standard for referral to such services is unaided clinical judgement. Multiple studies have shown that ED staff have difficulty identifying older adults at high risk for adverse outcomes using clinical judgement alone.Reference Aminzadeh and Dalziel 11 , Reference McNamara, Rousseau and Sanders 36 , Reference Singal, Hedges and Rousseau 37 Our data confirm that paramedics are also challenged to identify high-risk older individuals with their clinical judgement alone, even when aided by training and completion of 42 other variables to assess risk at the same time.

In summary, this study demonstrated the feasibility, high reliability, and adherence of trained paramedics with completing a 43-item assessment of older patients to identify those at high risk for adverse outcomes within one month of an EMS encounter. The PERIL prediction rule has advantages compared to a proxy measure of paramedic clinical judgement, including better predictive performance and the ability to set different thresholds tailored to the needs of the paramedic service. While the ISAR is an acceptable alternative in terms of predictive ability, PERIL is a simpler rule that had better adherence. Further research should validate the PERIL predication rule in an independent population of older patients, validate the four-item version of PERIL, and explore methods to optimize screening of older adults at high risk for adverse outcomes. If validated, future research should explore whether emergency physicians’ and paramedic services’ use of the PERIL prediction rule can be used to target high-risk individuals for referrals to preventative community-based interventions (e.g., primary care, home care, community paramedicine teams, or geriatric outreach teams), and whether such referrals reduce adverse outcomes.

Acknowledgments

JSL, RPV, MJS, LM, IS, DR, AK and GN conceived the study idea, and designed the trial. JSL, RPV, MJS, LC, IS, JT, LM, MN, BHR, SS, DR, AK and GN helped obtain research funding. JSL, LC, MN, BHR, and SS undertook recruitment of participating centers supervised the conduct of trial and data collection. JSL conducted training, executed recruitment of patients, data collection and managed the data, including quality control. MS, GN, IS, RPV and AK provided statistical advice on study design and analyzed the data. MS, LM, DR, AK, and GN provided advice on analysis. GN is the senior author and acted as academic mentor to JSL. JSL drafted the manuscript, and all authors contributed substantially to its revision and provided final approval. JSL takes responsibilities for the paper as a whole.

Competing Interests: The authors acknowledge the Canadian Institutes of Health Research, Institute of Aging, who supported this work. Dr. Rowe is supported as a Tier I Canada Research Chair in Evidence-based Emergency Medicine by the Canadian Institutes of Health Research (CIHR) through the Government of Canada (Ottawa, ON). Dr. Naglie is supported by the George, Margaret and Gary Hunt Family Chair in Geriatric Medicine, University of Toronto. Dr. Morrison holds the Robert and Dorothy Pitts Chair in Acute Care and Emergency Medicine, Li Ka Shing Knowledge Institute, St Michael’s Hospital and the University of Toronto. She also receives salary support from the National Institute of Health, USA for studies related to cardiac arrest and trauma.

References

1. Fried, LP, Ferrucci, L, Darer, J, et al. Untangling the concepts of disability, frailty, and comorbidity: implications for improved targeting and care. J Gerontol A Biol Sci Med Sci 2004;59(3):255-263.CrossRefGoogle ScholarPubMed
2. Desai, MM, Lentzner, HR, Weeks, JD. Unmet need for personal assistance with activities of daily living among older adults. Gerontologist 2001;41(1):82-88.CrossRefGoogle ScholarPubMed
3. Gurley, R, Lum, N, Sande, M, et al. Persons found in their homes helpless or dead. N Engl J Med 1996;27(334(26)):1710-1716.CrossRefGoogle Scholar
4. Shah, MN, Glushak, C, Karrison, TG, et al. Predictors of emergency medical services utilization by elders. Acad Emerg Med 2003;10(1):52-58.CrossRefGoogle ScholarPubMed
5. Clark, MJ, FitzGerald, G. Older people’s use of ambulance services: a population based analysis. J Accid Emerg Med 1999;16(2):108-111.Google Scholar
6. Marks, PJ, Daniel, TD, Afolabi, O, et al. Emergency (999) calls to the ambulance service that do not result in the patient being transported to hospital: an epidemiological study. Emerg Med J 2002;19(5):449-452.CrossRefGoogle Scholar
7. Weiss, SJ, Ernst, AA, Miller, P, et al. Repeat EMS transports among elderly emergency department patients. Prehosp Emerg Care 2002;6(1):6-10.Google Scholar
8. Bernstein, E. Repeat visits by elder emergency department patients: sentinel events. Acad Emerg Med 1997;4(6):538-539.CrossRefGoogle ScholarPubMed
9. Tinetti, ME, Baker, DI, McAvay, G, et al. A multifactorial intervention to reduce the risk of falling among elderly people living in the community. N Engl J Med 1994;331(13):821-827.CrossRefGoogle ScholarPubMed
10. McCusker, J, Jacobs, P, Dendukuri, N, et al. Cost-effectiveness of a brief two-stage emergency department intervention for high-risk elders: results of a quasi-randomized controlled trial. Ann Emerg Med 2003;41(1):45-56.CrossRefGoogle ScholarPubMed
11. Aminzadeh, F, Dalziel, WB. Older adults in the emergency department: a systematic review of patterns of use, adverse outcomes, and effectiveness of interventions. Ann Emerg Med 2002;39(3):238-247.Google Scholar
12. Wilber, ST, Gerson, LW. A research agenda for geriatric emergency medicine. Acad Emerg Med 2003;10(3):251-260.Google Scholar
13. Miller, DK, Lewis, LM, Nork, MJ, et al. Controlled trial of a geriatric case-finding and liaison service in an emergency department. J Am Geriatr Soc 1996;44(5):513-520.CrossRefGoogle ScholarPubMed
14. IICE ed. Challenges and Innovations in the Care of Older Emergency Department Patients. Montreal, Quebec, Canada: International Interdisciplinary Conference on Emergencies; 2005.Google Scholar
15. McCusker, J, Bellavance, F, Cardin, S, et al. Screening for geriatric problems in the emergency department: reliability and validity. Identification of Seniors at Risk (ISAR) Steering Committee. Acad Emerg Med 1998;5(9):883-893.Google Scholar
16. Gerson, LW, Schelble, DT, Wilson, JE. Using paramedics to identify at-risk elderly. Ann Emerg Med 1992;21(6):688-691.CrossRefGoogle ScholarPubMed
17. Ball, L. Setting the scene for the paramedic in primary care: a review of the literature. Emerg Med J 2005;22(12):896-900.CrossRefGoogle ScholarPubMed
18. Dixon, S, Mason, S, Knowles, E, et al. Is it cost effective to introduce paramedic practitioners for older people to the ambulance service? Results of a cluster randomised controlled trial. Emerg Med J 2009;26(6):446-451.Google Scholar
19. Mason, SM, Knowles, EL, Colwell, B. A randomized controlled trial to evaluate the effectiveness of community paramedic practitioners managing older people calling emergency medical services with minor acute conditions [Abstract]. Acad Emerg Med 2005;12(Suppl 1):134-135.CrossRefGoogle Scholar
20. Weiss, SJ, Chong, R, Ong, M, et al. Emergency medical services screening of elderly falls in the home. Prehosp Emerg Care 2003;7(1):79-84.Google Scholar
21. Stiell, IG. Clinical decision rules in the emergency department. CMAJ 2000;163(11):1465-1466.Google Scholar
22. Grief, CL. Patterns of ED use and perceptions of the elderly regarding their emergency care: a synthesis of recent research. J Emerg Nurs 2003;29(2):122-126.Google Scholar
23. McCusker, J, Karp, I, Cardin, S, et al. Determinants of emergency department visits by older adults: a systematic review. Acad Emerg Med 2003;10(12):1362-1370.Google Scholar
24. McCusker, J, Bellavance, F, Cardin, S, et al. Detection of older people at increased risk of adverse health outcomes after an emergency visit: the ISAR screening tool. J Am Geriatr Soc 1999;47(10):1229-1237.Google Scholar
25. Lee, J, Verbeek, R, Morrison, L, et al. PERIL - Paramedics and Elders at Risk of Independence Loss: Agreement between paramedics in both simulated and live patient encounters. Can J Ger 2008;11(1):56 [Abstract].Google Scholar
26. Beveridge, R, Ducharme, J, Janes, L, et al. Reliability of the Canadian emergency department triage and acuity scale: interrater agreement. Ann Emerg Med 1999;34(2):155-159.CrossRefGoogle ScholarPubMed
27. McCusker, J, Bellavance, F, Cardin, S, et al. Prediction of hospital utilization among elderly patients during the 6 months after an emergency department visit. Ann Emerg Med 2000;36(5):438-445.Google ScholarPubMed
28. Meldon, SW, Mion, LC, Palmer, RM, et al. A brief risk-stratification tool to predict repeat emergency department visits and hospitalizations in older patients discharged from the emergency department. Acad Emerg Med 2003;10(3):224-232.Google Scholar
29. Peduzzi, P, Concato, J, Kemper, E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49(12):1373-1379.Google Scholar
30. Hsieh, FY, Bolock, DA, Larsen, MD. A simple method of sample size calculation for linear and logistic regression. Stat Med 1998;17(14):1623-1634.Google Scholar
31. Cook, EF, Goldman, L. Empiric comparison of mulitvariate analytic techniques: Advantages and disadvantages of recursive paritioning analysis. J Chronic Dis 1984;37(9-10):721-731.CrossRefGoogle Scholar
32. Harrell, F. Regression Modeling Strategies. New York: Springer-Verlag; 2002.Google Scholar
33. Efron, B, Tibshirani, RJ. An Introduction to the Bootstrap. New York: Chapman & Hall; 1993.Google Scholar
34. Lee, JS, Schwindt, G, Langevin, M, et al. Validation of the triage risk stratification tool to identify older persons at risk for hospital admission and returning to the emergency department. J Am Geriatr Soc 2008;56(11):2112-2117.Google Scholar
35. Landis, JR, Koch, GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-174.Google Scholar
36. McNamara, RM, Rousseau, E, Sanders, AB. Geriatric emergency medicine: a survey of practicing emergency physicians. Ann Emerg Med 1992;21(7):796-801.Google Scholar
37. Singal, BM, Hedges, JR, Rousseau, EW, et al. Geriatric patient emergency visits. Part I: Comparison of visits by geriatric and younger patients. Ann Emerg Med 1992;21(7):802-807.Google Scholar
Figure 0

Figure 1 PERIL (Paramedics assessing Elders at Risk for Independence Loss) data collection form.

Figure 1

Figure 2 PERIL derivation study flow chart.

Figure 2

Table 1 Demographic characteristics of eligible subjects in the cohort used to derive the PERIL decision rule

Figure 3

Table 2 Inter-observer reliability, adherence and univariate association with adverse outcomes of 38 potentially predictive variables included in the logistic regression analysis

Figure 4

Table 3 The Paramedics assessing Elders at Risk for Independence (PERIL) Rule

Figure 5

Table 4 Predictive performance of the Paramedics assessing Elders at Risk for Independence (PERIL) clinical prediction rule

Figure 6

Table 5 Comparing adverse outcomes, sensitivity and specificity of the Paramedics assessing Elders at Risk for Independence Loss (PERIL) compared to single global risk assessment variable