Quality and Outcome Indicators for Acute Healthcare Services

© Commonwealth of AustraliaISBN 0 644 37481 XFirst published March 1997This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no partymay be reproduced by any process without prior written permission from the Australian GovernmentPublishing Service. Requests and enquiries concerning reproduction and rights should be addressedto the Manager, Commonwealth Informaiton Services, Australian Government Publishing Service,GPO Box 84, Canberra ACT 2601Responsibiltiy for the content of this report remains with the authors. This report does not necessarilyreflect the views of the Commonwealth Department of Health and Family Services.Cover design by the Publications and Design (Public Affairs, Parliamentary and Access Branch)Commonwealth Department of Health and Family ServicesProduced by the Australian Government Publishing Service

2ContinuityDischarge PlanningCoordinationTechnical Proficiency -Appropriateness -These dimensions of care are reflected variably in currently available indicators. They include concepts ofquality that overlap (that is, there is some redundancy in this classification) and they are notcomprehensive (that is, there are important quality dimensions - such as equity and benevolence - whichare inadequately represented. These dimensions of care are conceptually complex and poorly translatedinto quantitative indicators. Consequently no feasible indicators of these dimensions were identified).We have proposed a number of attributes of quality of care indicators which provide a basis for assessingand deciding the potential utility of any putative indicator for a nationally consistent quality and outcomeindicator set. These attributes are:ATTRIBUTEReliabilityValidityENCOMPASSINGInternal consistencyTest/re-test stabilityInterrater reliabilityFace validityContent validityConstruct validityPredictive validitySensitivitySpecificityPredictive valueResponsiveness -Interpretability -SignificanceBurdenVulnerability toundesired effectsRelevanceCollection impostGamingDistortion of healthservice deliveryUtility -Availability ofalternate formsAmenity toindependent corroborationLanguage adaptedLiteracy adaptedCulturally adaptedAuditData SourceIt is frequently stated that the science of healthcare quality indicators is in its infancy. Certainly we foundthat good data on the attributes of proposed indicators and specific information on their developmental

3rigour were frequently missing. Some indicators have no basis in science. Nonetheless, we foundsufficient evidence supporting quality of care indicators to suggest that an appropriate growth anddevelopment analogy would place quality indicators in early childhood, rather than infancy, with severalquality and outcome indicator programs having passed significant developmental milestones. Most existingquality and outcome indicators are imperfect. In the majority of indicators we examined there wasinsufficient information available on indicator attributes to allow us to draw firm conclusions aboutindicator performance against all of our chosen attributes. We see the current generation of indicators asstepping stones to future better indicators. It will only be by their application in the health sector thatindicators will improve. Provided all those using current indicators are aware of their limitations andappropriately cautious in their interpretation, no harm will come from use of less than perfect performancemeasures. Existing - albeit imperfect - indicators do provide useful pointers to quality of care. If resultsof indicator data occasionally differ significantly from user expectations it may be necessary for thoseusing these tools to consider revising their understanding of the role of indicators in monitoring qualityrather than discarding these indicators and persisting with attempts to find “perfect” indicators.This report contains insights derived from an examination of how we and other western healthcare systemshave applied quality of care indicators. These experiences contain valuable lessons for those embarking onlarge scale quality monitoring in healthcare - particularly on how best to manage the introduction ofquality and outcome indicators to maximise their value as accountability tools, instruments guidingconsumer choice and facilitators of provider efforts to improve quality. One recurring imperativeidentified by our review is the requirement for a collaborative approach to indicator development and thesubsequent interpretation and dissemination of quality and outcome indicator data. Such collaborationmust embrace purchasers, providers and consumers of healthcare to see quality and outcome indicatorssuccessfully achieve their multiple goals.Quality of care indicators are constructed from data obtained from administrative databases, abstractedfrom medical records, reported by patients in response to structured survey or by some combination ofthese methods. Administrative databases are a relatively complete data source, although there aresignificant risks of inaccuracy if data are used for purposes other than those originally intended. Patientclassification and categorisation within existing coding systems mask patient differences of potentialimportance for quality and outcome indicator construction. In contrast to the low cost of indicator datafrom administrative databases, medical record abstraction is expensive. The information present in recordsis inconsistent and frequently incomplete. Occasionally, the added expense of collection is justified by theclinical detail obtained by record review, although such abstraction cannot be a basis for core nationalindicators. Patient surveys are uniquely capable of providing valuable insights into the processes andoutcomes of care and care acceptability. Although expensive, as indicator data sources go, patient surveyswill be essential for a worthwhile national quality monitoring program embracing consumer perspectiveson care delivery.The development of explicit guidelines for the care of particular clinical conditions has been a majorachievement of several international and national expert groups over recent years. The National Healthand Medical Research Council (NHMRC) has undertaken initiatives to develop guidelines for national use,including areas targeted as national health priority areas. Clinical practice guidelines can readily beadapted to generate quality of care indicators for specific conditions or interventions.Following a review of existing indicators available for the identified key dimensions of quality of care inthe acute healthcare sector, and assessment of their relative strengths and weaknesses, we have made aseries of recommendations regarding the potential development of a quality and outcome indicator programnationally.

41.1 STRUCTURE OF A NATIONAL INDICATOR PROGRAM1.1.1 National quality and outcome indicators be developed as two complementary sets - a coreindicator set intended for long-term continuous collection and a series of indicator modulesfocused on periodic collection for a finite duration at a defined frequency.1.1.2 Core indicators focus on aspects of care that are common across conditions.1.1.3 Indicator modules be targeted to specific conditions, diseases, diagnoses or interventionsand include a balanced portfolio of clinical indicators, health status, acceptability and costindicators. It is likely that most benefit for facility-level quality improvement in acutehealthcare will flow from energy devoted to such targeted quality of care indicator modules.1.1.4 Where possible, the development of indicator modules should commence in identifiednational health priority areas.1.1.5 Core indicator sets be based upon data from administrative databases or patient surveys.1.1.6 Indicator modules include hybrid data collection - including medical record sampling,review and data abstraction - to establish the cost benefit of more sophisticated indicatorconstruction methodologies in a national quality monitoring context.1.1.7 Indicator development programs be cooperative - involving all relevant agencies - andmultidisciplinary.1.1.8 Appropriate sampling technique be used in indicator data acquisition wherever possible toincrease the complexity of data available for a given cost.1.1.9 A resource be developed to assist providers with indicator data collection, analysis,feedback and interpretation and with the potential to provide independent confirmation ofdata integrity.1.1.10 Indicators developed for national use include comprehensive operational definitions,preferably encoded within suitable software packages, to enhance data reliability.1.2 SUGGESTED INDICATORS FOR ADOPTION IN AUSTRALIAThere are no indicators with unequivocal and universal support as “gold standard” means ofmonitoring healthcare quality. Comparative studies of indicator performance in acute healthcare arerare and do not provide a basis for the unquestioned support of any individual quality of careindicator or indicator set in an Australian context. From the information available on currentquality indicators we have selected a reasonable, basic set of indicators which we believe havesufficient evidence to support their probable utility. This evidence includes knowledge that thepremise underpinning the indicator is sound, that data collection is feasible (given current orreasonably anticipated future health services information systems) and that the indicator would havepractical utility. The absence of compelling evidence to guide our choice of suggested indicatorsmeans that ultimately, those indicators recommended receive that recommendation based on anassessment by us of their overall worth - following our in-depth review of available information onquality indicators. A balance has been sought between the size of the initial indicator set foradoption and the desire for comprehensive coverage of the quality of acute healthcare services.Over time a larger indicator library can be developed based upon local research, development andexperience and a close linkage of our national indicator programs to international initiatives inquality indicators for acute healthcare.

51.2.1 Access- Elective surgery waiting times- Emergency Department waiting times- Waiting times in Emergency Department prior to emergency admission- Patient-based reports of elective surgery, Emergency Department and OutpatientDepartment waiting times and the acceptability of these waiting times to patients1.2.2 Efficiency- Cost per casemix adjusted separation1.2.3 Safety- Indicators of adherence to best practice guidelines for the processes of care or variationsin observed to predicted outcomes should be incorporated in indicator modules targetingparticular clinical circumstances. In view of the importance of cardiovascular disease,modules addressing interventions in patients with ischaemic heart disease (angioplasty,CABG, angiography and AMI) should be included in any initial module sets.1.2.4 Effectiveness- Generic health status (SF36 or SF12) be included in the core indicator set, examiningchange in health status about an acute episode of care.- Health status measures, either generic or condition-specific, be incorporated intoindicator modules targeting specific clinical circumstances.- Mortality rates, stratified with data available within administrative databases, becollected for key clinical conditions and compared to risk-adjusted mortality indicesdeveloped within indicator modules. Such analyses will establish the cost-benefit ofmore sophisticated risk-adjustment modelling in the context of a national qualitymonitoring program (commencing with angiography, CABG, angioplasty and AMI).- Unplanned readmission after inpatient treatment for asthma (paediatric and adultgroups).- Low and very low birthweight rates for infants.1.2.5 Continuity- Patient-based assessments using relevant modules from the Picker CommonwealthSurvey instrument.1.2.6 Acceptability- A national survey on the acceptability of care be based upon components of the PickerCommonwealth, Hospital Corporation of America and Royal College of Surgeonsinstruments.1.2.7 Technical Proficiency- Avoid putative generic indicators of facility-wide technical proficiency- Promote risk-adjusted technical proficiency indicators developed within indicatormodules addressing targeted clinical circumstances.1.2.8 Appropriateness- Use relative utilisation rates of targeted interventions as proxies for appropriateness.

6These recommended indicators are summarised on Page 8.The complexity underlying the process of trialing putative quality indicators should not beunderestimated. Even apparently simple indicators require in-depth analysis of competingoperational definitions and generation of an agreed, detailed operational definition (preferablysupported by written materials with computer software expansion, clarification and problem-solvingguides). This agreed operational definition is then to be rigorously and uniformly applied in thefield. In the case of modular indicator sets, project teams with a range of relevant skills andexperience should build balanced sets of indicators from existing available indicators or develop denovo indicators to address relevant performance measurement needs. Both classes of indicatorsshould be trialed in a collaborative, cooperative atmosphere and involve provider, purchaserregulator and consumer representation in the project teams. Pilot projects across interestedrepresentative facilities should precede any more widespread implementation to ensure that thenational indicator set, when launched, is valuable to all concerned and credible. Engaging existingdrivers of quality indicators in the trialing process (e.g. the States and Territories and TheAustralian Council on Healthcare Standards Care Evaluation Program (ACHS CEP)) will beessential to ensure that common operational definitions for quality indicators are arrived at. Thiswill avoid the creation of competing, overlapping data collection requirements of healthcarefacilities - which could lead to inefficiencies, the disenfranchisement of providers and a drasticreduction in the reliability of indicator data.1.3 FUTURE DIRECTIONS FOR INDICATOR DEVELOPMENT1.3.1 Dimensions of care with particular needs for indicator development:- Unmet access needs- Allocative and technical efficiency- Acceptability- Continuity- Appropriateness- Indicator modules for vulnerable patient populations1.3.2 Quality of care and outcomes indicators for healthcare developed by multidisciplinarygroups should focus on producing balanced portfolios of indicators targeting particularconditions, diseases or interventions and reflecting many key dimensions of healthcarequality and the perspectives of purchasers, providers and patients.1.3.3 Most of the significant health concerns of Australians relate to chronic illnesses, where theperformance of the integrated healthcare system is far more important than the performanceof its isolated components. In the long term, quality and outcome indicator programs shouldaddress integrated healthcare system performance rather than retaining an acute healthcaresector focus. Measurement of the quality and outcomes offered by the care continuum willthen inform the success of overall service delivery.1.3.4 Australia should develop centres of excellence pursuing quality of care indicator researchand seek their inclusion in the WHO Quality Assurance Collaboration.Changes in the science of quality of care indicators and the science of medicine will require that qualityand outcome indicator programs are dynamic, iterative programs attuned to advances in academic and

7clinical knowledge. Few indicators should therefore be built around mandating changes in routineminimum data set collections. It is unlikely that we will ever see single indicators or small indicator setsthat work for all parties, at all times and for all purposes. Above all else, effective use of quality andoutcome indicators requires knowledge of their limitations and intelligence, good-will and common sense intheir applications. Programs to “perfect” indicators must be complemented by educational campaigns toimprove the understanding of their use by all with an interest in monitoring healthcare quality.

8Indicators Recommended for Trialing in a Core National Set• Access- Elective surgery waiting times- Emergency Department waiting times- Waiting times in Emergency Department prior to Emergency admission- Patient-based reports of elective surgery, Emergency Department andOutpatient Department waiting times and the acceptability of these waitingtimes to patients• Efficiency- Cost per casemix adjusted separation• Effectiveness- Generic health status (SF36 or SF12)• Continuity- Patient-based assessments using relevant modules from the PickerCommonwealth survey instrument• Acceptability- A national survey based upon components of the Picker Commonwealth,Hospital Corporation of America and Royal College of Surgeons instruments• Appropriateness- Relative utilisation rates of targeted proceduresIndicators Recommended for Trialing in Indicator Modules at a Defined Frequencyand for a Finite DurationProjects should be promoted to develop indicator modules targeting particular clinicalcircumstances. These balanced sets of quality of care indicators should include a variety ofperspectives on the quality of care chosen from relevant dimensions of care such as:AccessEfficiencySafetyEffectivenessTechnicalProficiencyAppropriateness- Waiting times- Cost- Adherence to best practice guidelines- Observed to expected outcome ratios- Condition-specific health status or health-related quality of lifemeasures- Stratified mortality rates- Unplanned readmission rates (asthma)- Low and very low birthweight infants (obstetric care)- Risk-adjusted technical proficiency indicators- Condition/procedure-specific relative utilisation rates

92. INTRODUCTION2.1 The Aim of the ProjectThe National Hospital Outcomes Program (NHOP) commissioned this Project to critically review thestatus of Australian and overseas knowledge regarding the development and use of quality of care andhealth outcome indicators in acute care services. It specifically sought to identify those performanceindicators that contribute to improvements in the quality and outcomes of care to inform the developmentof a set of nationally consistent quality of care and health outcome indicators for acute healthcare servicesin Australia.The Project wished to:• Identify key issues surrounding the use of quality and outcome indicators in a national context• Identify key dimensions of quality of care for indicator development and implementation• Assess the usefulness of existing indicators for application in the Australian context• Advise on future directions for quality and outcome indicator developmentsThe research was to include commentary on hospital access, technical and allocative efficiency,effectiveness, appropriateness, nursing, change in health status, discharge planning and patient safetyindicators at a minimum. Health concerns of vulnerable patient groups, such as Aboriginals and TorresStrait Islander peoples, were to be addressed in the context of indicator utilisation. The Projectspecifically excluded replication of work undertaken within complementary projects.The Commonwealth, States and Territories in recent years have been putting substantial efforts intoimproving the cost efficiency of public hospitals. Health consumers are greatly concerned about thequality and outcomes of care, particularly in an environment of programs pursuing efficiencyimprovements. The Commonwealth, States and Territories are concerned that quality of care and healthoutcomes are at least maintained - and preferably improve - whilst efficiency gains are progressing. InSchedule I of the 1993-1998 Medicare Agreements, the Commonwealth, States and Territories agreed tothe development of national quality of care and health outcome indicators for hospitals. It is believedessential that mechanisms be developed to monitor quality of hospital care so that consumers andgovernments can be sure of the value of delivered care (i.e. the ratio of quality to cost) and providers ofcare are stimulated to pursue quality improvement within acute healthcare programs.The NHOP seeks to develop and implement performance indicators and determine standards of quality ofcare for Australian hospitals. It is a three year program to develop and trial workable indicators that canbe used to promote improvements in the quality of care and health outcomes in Australian hospitals. TheProject is to identify a manageable set of indicators that would primarily serve the accountabilityrequirements of a large purchaser of acute care services. This Project does not seek identification ofmicroindicators that might be relevant in the internal management of quality by acute care providers. TheProject is the first stage in the programmed development of a nationally consistent quality and outcomeindicator set and will provide a basis for future work to develop, refine and test indicator sets suitable fornational application.Data for generation of quality and outcome indicators has either to be specifically created or obtained fromdata sources developed for other purposes. There has been a strong trend to manipulate existing datasources to produce quality indicators at low cost. As is discussed in detail later in this report (Section 5.4)there are considerable difficulties in the use of data for purposes other than those originally intended bydata gatherers. Indicators based solely around data from administrative databases will always be ofsomewhat limited utility. The longterm success of quality and outcome indicator programs will requirethat necessary data are identified and subsequent steps taken to obtain such data (be it routine collectionfrom all or sampling strategies targeting patient population subsets) rather than persevering with strategies

10attempting to build credible indicators entirely from existing computer database contents. At presentAustralian healthcare providers collect various data items - with no uniform definitions applicable to manyapparently similar data (such as elective surgery waiting times). As requisite data points for qualitymonitoring are identified, national uniform data definitions must be incorporated into the national datadictionary and mechanisms enacted to audit compliance with definition application in the field. Such aprospective approach to quality indicator data collection will undoubtedly increase the cost of indicatorcollection - but the consequent increase in the value of indicator data flowing from the prospectivedesignation and collection of reliable, relevant data will be more than commensurate.2.2 The Aim of this ReportIn this final report we convey, in a summary format, our opinions and recommendations regarding qualityof care and health outcome indicators for acute healthcare services. We seek to do this in an accessibleway, hence the report on these complex 1-1136 and at times controversial matters is relatively succinct.Those interested in more detailed analyses and commentaries will find these in the attached appendices tothis report or within the bibliography listing of information sources.Quality and outcome indicators are of interest to many customer groups. These include:• Patients and their families• Healthcare professionals• Purchasers of healthcare• Regulators of healthcare• Health system employees• CommunitiesOccasionally, the needs and expectations of these groups regarding quality and outcome indicators do notalign, and sometimes these differences are substantial. Our report repeatedly places commentary andrecommendations in a context of the intended use of derived indicator data. This report encompassesdiscussion of the history of performance indicators in acute healthcare, recommendations on keydimensions of quality of care requiring monitoring and the identification of pertinent issues aroundintroduction of a nationally consistent quality and outcome indicator set. It seeks to point out gapsbetween existing indicators and those desired by potential users. Knowledge of these gaps permitsprioritisation of subsequent research and development efforts.3. BACKGROUND3.1 History of Indicator DevelopmentHistorically, the responsibility for healthcare quality was thought to reside only with the medical andclinical care professions (the so-called “professional model” of accountability) 297 . Consumers andpurchasers assumed that relatively uniform quality of care existed across any given healthcare deliverysystem and that investigation and treatment approaches were relatively standard within that system of care.Several decades ago there was a move to make provider facilities accountable for care delivered withinthose facilities, which saw an increased emphasis on facility-level peer-review of performance regardingquality of care. Facility peer-review is essentially a strengthening of the professional model ofaccountability which typically fails to provide objective evidence on performance for external review.Concerns that existed regarding significant variations in quality of care and the nature of care deliveredwithin the acute healthcare sector coincided with interest in improving acute care efficiency and restrainingtotal expenditure on acute healthcare in several western communities 16-28 . Whilst the relative balancebetween consumerism and cost-control varied between communities, all have demonstrated a progressiveinterest in objective quality evaluation in healthcare. Consumers, purchasers, regulatory agencies,

11governments and providers increasingly seek to compare provider performance to address accountabilityrequirements, promote quality improvement and inform consumer decisions.Quality of care and outcome indicators first appeared almost a century ago with proposals to monitor theimpact of hospital care by pioneers such as Florence Nightingale in the UK and Ernest Codman in theUSA. These proposals were not widely acceptable to healthcare professionals and the mechanics of largescale data collection were impractical, hence they were not embraced. Indicators resurfaced in the early1970s when the potential for improving quality by using measurement was rediscovered by socialscientists. In North America, groups that sought to systematically monitor acute healthcare quality(Professional Standards Review Organisations) debuted in 1973 and began applying crude qualityindicators to the hospital sector. In the 1980s the availability of improved data management systems, theperceived impact of efficiency strategies such as prospective payment systems and Deming’s QualityImprovement principles rejuvenated interest in quality monitoring and saw a move from indicator usage forquality maintenance (quality assurance) to use for quality upgrading (quality improvement).In the mid 1980s programs were launched to evolve complete sets of quality and outcome indicators,rather than use of individual indicators on an ad-hoc basis. These indicator sets were designed to be partof comprehensive quality monitoring programs in hospitals (e.g. The Maryland Hospital AssociationQuality Indicator Project; 1985 55,304,335 . The Joint Commission on Accreditation of HealthcareOrganisations Indicator Measurement System; 1986 14 . ACHS CEP Clinical Indicator Program;1989 367,507,508 ). More recently the focus in healthcare monitoring has moved from acute healthcare qualityassessments, specific for an episode of care, to assessment of the performance of integrated healthcaresystem performance (e.g. The Consortium Research on Indicators of System Performance; 1990 300-303 .The Health Employers Data and Information Set; 1989 307 . The Clinical Accountability and SystemsPerformance Evaluation Quality and Outcome Indicator Set; 1996) embracing review of the continuum ofcare. This move to a broader focus on healthcare provision is driven by the recognition that:• Chronic illnesses, in particular, do not have clearly defined start/stop points for performanceassessment based on a single episode of care.• The quality of integration and coordination of care is as important as the quality of componentservices.• Performance indicators should reflect whether required care was delivered to communities requiringcare, not simply how well care was delivered to hospitalised individuals who have successfullyaccessed care.Initial quality indicators tended to address healthcare structures (such as the level of equipment and staff).Later, indicator efforts were directed at process measures (such as if care was delivered as planned) whenprocess management was recognised as the key to successful quality improvement 255 . In the late 1980s astrong emphasis on outcome indicators emerged - driven by Paul Elwood 649 and others - requiringsystematic evidence of the end result of healthcare interventions (such as the duration or quality of lifeachieved).Original indicator sets tended to focus on narrow aspects of healthcare delivery (such as technicalproficiency or outcomes as perceived by providers of care) 320,321 . More recently, indicator sets haveemphasised the importance of a balanced portfolio of indicators, encompassing clinical outcomes, theacceptability of care, functional health outcomes, critical process measures and cost of care delivery (suchas Hospitals Corporation of America’s “Value Compass” Indicator Sets) 492,930 .Typically, indicator development programs commence with an emphasis on hospital episode of carereview, because of the availability of data systems in hospital practice and an assumption that hospitalbasedepisodes of care represent critical moments in individual patients’ overall healthcare. Such hospitalbasedindicator measures should only represent the beginning of an integrated approach to comprehensivequality of care and outcome monitoring within a healthcare system.

12SUMMARY OF TRENDS IN INDICATOR USE• Autonomy → Peer Review → Facility Review → External Review• Structure → Process → Outcome Focus• Individual Indicators → Indicator Sets• Random Indicator Development → Balanced Sets of Indicators• Quality Control → Quality Assurance → Quality Improvement Context3.2 TerminologyThis field of health services research is replete with examples of individuals using the identical term torepresent quite different underlying concepts and different terms to reflect a common theme. No universallexicon exists. There has been a relatively recent growth in interest in quality and outcome indicators andthe appropriation of words from common usage to serve as descriptors for very specific quality andoutcome concepts. We provide brief working definitions we have used for important terms appearing inthis report. (A more detailed glossary of some examples of what others believe to be the various preferreddefinitions of terms is contained in Appendix 1).Structure:comprises the characteristics of care or resources compiled todeliver care to the patient. It includes the physical facilities, the staffand the licensing and credentialling of healthcare providers andselected patient characteristics 297 .Process:refers to the actual delivery of care. The series of linked, often (butnot necessarily) sequential steps that convert an input into anoutput, cause some set of outcomes to occur, generate usefulknowledge or add value 315 .Outcome:the significant result or end product of care delivery, such asimproved survival, functional health status or quality of life 304 .Health Outcome:a change in the health of an individual or a group of people orpopulation which is attributable to an intervention or series ofinterventions 352,504 .Health Intervention:any action which is intended to improve someone’s health (orreduce the rate at which it deteriorates), whether the action is aimedat health promotion, disease prevention, early diagnosis, a clinicalintervention, counselling or social service support, educational orpreventive measures, a change in administrative or budgetaryresponsibilities, regulations relating to safety, the relief of poverty,better housing or whatever 352,504 .Measures: seek to directly quantify quality of care or health outcomes 352,504 .Indicators:statistics or other units of information which reflect, directly orindirectly, the performance of the healthcare system in maintainingor increasing the well-being of its target population 352,504 .Generic Measures orare measures or indicators that can be applied to individuals in anyIndicators:health condition 352,504 .Clinically SpecificIndicators or Measures:Attribution:Association:are indicators or measures that relate to specific clinical conditionsor measures of function that may have particular significance forparticular conditions 352,504 .a health outcome is attributable to an intervention if the interventionhas been shown in a rigorous scientific way to cause the change inhealth status 352,504 .a health outcome is associated with an intervention if the change inhealth status generally occurs following the intervention but has notbeen demonstrated through rigorous scientific study to cause thechange 352,504 .

13Health Status:an integrated indicator of health (i.e. well-being), typicallyincorporating biological function, physical and mental health, socialand role functioning 505 .Health-related Quality of that component of quality of life related to the sense of health (i.e.Life:well-being) of the individual concerned 764 .Customer:the recipient of a service within healthcare or anyone who hasexpectations regarding healthcare delivery 339 .Consumer: the population of potential customers 339 .The differential application of “indicator” versus “measure” and “quality/outcome” versus “performance”was particularly noted in our review of the literature. Terms such as “quality indicator” and “qualitymeasure” or “outcome indicator” and “outcome performance indicator” are effectively usedinterchangeably by some - yet are regarded as substantially different by others.If indicators are identifiable markers which describe or relate to quality of care and measures arequantifiable dimensions of objects or functions then clearly there is a continuum between an identifiabledescriptor related to quality (i.e. an indicator) and a quantified marker of quality (i.e. a measure). It is amatter of opinion as to when an “accurate description” merges into “quantification” 333 .We have elected to use the term “indicator” for all instruments that estimate the extent to which qualityhealthcare services are delivered, believing the overall status of quality evaluation in healthcare warrantsuse of a term implying less precision in the inferential relationship between evaluation and quality of care333,341 . We acknowledge that all “quality indicators” are strictly-speaking “quality performance indicators”337 . They provide reflections on the care setting (structure) and on things providers do (processes) orachieve (outcomes) rather than judgements based upon these reflections. For brevity’s sake, and indeference to common usage, we have frequently applied the term “quality indicator” without reference tothe “performance” qualifier.4. METHODOLOGY FOR THE PROJECT4.1 Literature ReviewAll consultant members contributed to an initial bibliographic database search and handsearch of relevantdata sources. One month into the six month consultancy the Centre for Health Program Evaluationmembers assumed primary responsibility for further literature review of indicators of access and efficiencywhile The Alfred Healthcare Group and Department of Epidemiology and Preventive Medicine, MonashUniversity, members continued the broad literature search for other identified key dimensions of quality ofcare (see 5.1 below). Further details of the literature search methodologies are contained in Appendix 2.In common with other researchers in this field, we found the literature search to be a relatively inefficientand ineffective tool for accessing the body of knowledge of healthcare quality indicator applications320,321,340 . This reflects poorly developed key-word identifiers for research in indicators (limiting thesuccess of computer-based data searches) and the spread of relevant literature across a broad range ofjournals (rendering hand searches of journals relatively inefficient). Additionally, much of the detailedknowledge surrounding quality and outcome indicators is not published in conventional forms - but iscontained in commissioned consultancy reports, purchaser or regulatory bodies internal workingdocuments or within healthcare providers’ information systems. At Project’s end, the computer-basedbibliographic search and the hand-search strategies produced relatively equivalent numbers of relevantcitations.

144.2 Expert ConsultationThe Project team (Appendix 3) allocated responsibilities for canvassing expert opinion to complement theknowledge of quality of care and health outcome indicators obtained from the literature review in thefollowing way:• The Centre for Health Program Evaluation consulted with a number of States and Territories relevantto access and efficiency indicators.• The Alfred Healthcare Group members consulted with the Commonwealth on related researchprograms and overseas consultation with acknowledged leaders in indicator development andimplementation. This included site visits to centres of excellence in the United States and the UnitedKingdom (Appendix 4).• The Department of Epidemiology and Preventive Medicine consulted with academic research groupspursuing relevant research within Australia.• The Australasian Association for Quality in Healthcare consulted with State and Territorygovernments to ascertain the status of relevant indicator developments and experience withinAustralia.Taken together, these contacts with experts working in the field were extraordinarily productive, both inproviding textural details and balanced opinion frequently absent from published reports and in helpingidentify additional sources of relevant information and mechanisms to access these sources. These expertconsultations were crucial in obtaining the operational details of many indicators and indicator sets - detailswhich frequently informed our decisions regarding the ultimate utility of proposed indicators. Additionaldetails concerning these site visits are included in Appendix 5. We acknowledge the generosity of those whoshared their valuable time and accumulated wisdom with us over the six months of the consultancy andcontributed to our decision-making processes.5. RESULTS FROM THE PROJECT5.1 Key Dimensions of QualityThe definition of quality in healthcare remains a challenge 12 . As for all abstract concepts it is possibleto define healthcare quality equally correctly from a variety of perspectives. We believe no singleperspective or resultant definition to be inherently superior to others. Use of several definitions isboth possible and legitimate. We consider the following definitions contribute to a broadunderstanding of what is meant by quality in healthcare.• The degree of excellence. OXFORD DICTIONARY• How closely the result of a medical service approaches the fundamental objectives ofprolonging life, relieving distress, restoring function and preventing disability. PAULLEMBCKE• Achieving and producing health and satisfaction. AVEDIS DONABEDIAN• Consistently meeting or exceeding informed customers’ expectations. W.EDWARDS DEMING• The degree to which health services for individuals and populations increase the likelihood ofdesired health outcomes and are consistent with current professional knowledge. INSTITUTEOF MEDICINE

15Whilst there are multiple dimensions to healthcare quality, only some of these are currently amenable toquantification. Following a preliminary review of the available literature we agreed with our SteeringCommittee to address indicators in the following care dimensions:AccessDIMENSIONTimelinessENCOMPASSINGEfficiencyTechnical EfficiencyAllocative EfficiencySafety -EffectivenessEfficacyAcceptabilityConsumer PerceptionCustomer PerceptionSatisfactionRelevanceCultural AppropriatenessConsumer Involvement in Health ServicesContinuityDischarge PlanningCoordinationTechnical Proficiency -Appropriateness -The working definitions for these designated dimensions of care are:Access:Efficiency:The capacity of individuals to obtain the same quality of service.Maximising benefits (or outcomes) for a given cost:♦♦Technical efficiency: the degree to which the least cost combination ofresource inputs occur in production of a particular service.Allocative efficiency: the degree to which maximum benefit (oroutcomes) are obtained from available resources.Safety:Effectiveness:Acceptability:Continuity:TechnicalProficiency:The extent to which potential risks were avoided and inadvertent harmminimised in care delivery processes.The degree to which an intervention produces measurable increases in survivalor improved quality of life (or improved outcomes) when applied in routinepractice.The degree to which the service meets or exceeds the expectations of informedcustomers and consumers.The extent to which an individual episode of care is coordinated and integratedinto overall care provision.The extent to which the performance of interventions by healthcareprofessionals is consistent with contemporary standards and knowledge ofskills relevant to that intervention.

16Appropriateness:The extent to which potential benefits of an intervention exceed the risksinvolved.Some identified dimensions of quality (such as equity) are only partially represented in the above schema(access reflects some aspects of equity). Others (such as benevolence) see aspects represented withinseveral of our proposed dimensions of quality of care, but are still inadequately represented overall. Wechose not to include care dimensions for the purpose of this Project when initial review failed to identifyany feasible indicators likely to be applicable in any national indicator program. These oversights do notindicate that dimensions not incorporated into this review are less crucial than those featuring - rather theseidentified dimensions of quality of care are a reasonable starting point from which to begin building acomprehensive quality and outcome indicator set for acute healthcare services. This final choice ofdimensions of quality of care encompasses aspects of care relevant to patients, providers and purchasers.There is minimal overlap between individual dimensions and the coverage of aspects of quality iscomprehensive. Because of the importance of patient safety in acute healthcare, we have chosen toemphasise safety by its designation as a discrete dimension of care, despite considerable content overlapwith matters analysed in the dimensions of effectiveness, technical proficiency and appropriateness.5.2 Criteria for Assessing IndicatorsHealthcare quality indicators need to be judged against criteria which indicate whether they are likely tofulfil their intended purposes. We chose the following assessment criteria:♦Reliability:The degree to which an indicator is free from random error, is reproducible (or stable) over time andshows interrater agreement at one point in time.This includes the concepts of:• Internal consistency• Test/retest stability• Interrater reliabilityReliability will be largely dependent on the adequacy of the operational definition for the indicatorand the rigour of data collection, data analysis and data audit.♦Validity:Given the quality monitoring purpose for which it is intended, do inferences regarding quality ofcare based upon the indicator accurately reflect the quality of care delivery?Validity is a matter of degree and must be judged with an understanding of the intended applicationof an indicator (that is, is it valid for its intended purpose?).Judgements of validity are based upon review of:• Face validity: Does the indicator appear to relate to quality of care?• Content validity: How closely does the indicator relate to quality of care and how well arerelevant aspects of care quality covered by the indicator?• Construct validity: What relation does the indicator have to other measures of quality?• Predictive validity: How well does an indicator of good/poor care predict that good/poorcare was delivered?

17Unfortunately criterion validity (the degree to which an indicator is related to widely accepted validmeasures of quality) could not be included in our assessments as criterion measures for quality areunavailable in acute healthcare.Within validity determinations lies knowledge of quality indicator sensitivity, its “true positive” rateand specificity, its “true negative” rate.♦♦♦♦♦♦♦♦Responsiveness:How does the indicator change as quality of care changes? Is the indicator capable of detecting thesorts of differences in quality of care typically experienced in acute healthcare services?Interpretability:Does the indicator make sense? Does it communicate a consistent message to those who use it?Significance:Does the indicator reflect aspects of care that matter to users of the indicator and are relevant incurrent healthcare contexts?Burden:How difficult or costly is indicator data collection and indicator construction?Utility:Has the indicator been proven to be of value when used in acute healthcare (either foraccountability, directing consumer decisions or quality improvement)?Vulnerability to Undesired Effects:What is the likelihood that use of the indicator would create perverse incentives for healthcareproviders (such as to corrupt indicator data or alter healthcare provision in undesirable ways)?Availability of Alternate Forms:Can the indicator be altered to allow its use in different target populations (e.g. those requiringlanguage or cultural adaptations)?Amenity to Independent Corroboration: Can indicator data be confirmed by others?These attributes of indicators were culled from several proposed assessment criteria panels 320,321,324 . Theyprovide a realistic basis upon which to judge indicator suitability for application in an Australian context.In the majority of cases examined we found insufficient information upon indicator performance to enableus to make appropriate firm judgements regarding indicator performance against these desired attributes.Final judgements on the utility of indicators involved assessments of all available information regardingindicators and the potential for local collection of requisite information.5.3 Current Status of Indicator Development5.3.1 AustraliaThe major program for acute healthcare quality indicators underway in Australia is the ACHSCEP and specialist medical colleges conjoint development of clinical indicators 367,507,508 .Although this work was specifically excluded from our terms of reference, it is such a dominantforce in indicator development that we believe some brief comments on its achievements thus farare appropriate. In 1996 there are seven indicator sets in use - with 16 clinical indicator setsplanned to be available for use within ACHS accreditation programs in 1997 (Appendix 6).These indicators have been cooperative developments between the ACHS and relevant medicalspecialty colleges and strongly reflect the perspectives of providers, with a heavy emphasis ontechnical proficiency. Although drawing upon overseas indicator developments, operational

18definitions for ACHS CEP indicators are sufficiently different from international definitions torender direct comparisons impossible. Their focus upon clinician involvement in the derivationof indicator data means that they frequently cannot be simply collected from routineadministrative databases and hence may not be easily applied as core indicators at a nationallevel. At present indicators are not risk-adjusted using patient level data, but are stratified on thebasis of provider-facility characteristics. Throughout their development there has been relativelylittle data made available on the methodologic rigour of indicators - with little current dataavailable on their reliability, validity and responsiveness. These are not necessarily criticisms ofthe AHCS CEP nor the contributions of specialist colleges to this clinical indicator program.Rather, they reflect the context within which the ACHS program has progressed and, inparticular, its stated aims to promote facility quality improvement and engage clinicians inmeasurement endeavours. It does mean that ACHS clinical indicators are not immediatelysuitable for transfer to a national quality of care and health outcome indicator program withoutclose scrutiny of their suitability for that purpose and subsequent formal trials of theirapplication in such a context.Numerous other indicator initiatives have occurred within facilities, benchmarking projects, bestpractice programs and the Health Departments of States and Territories. These initiatives havelargely not attempted to generate the sort of high level indicators which would be of value in anational monitoring program. Rather, they looked at indicators suitable for the specific purposeat hand. We found quality of care issues dealt with by a broad range of interested parties(including a variety of departments and sections in governmental and regulatory bodies) and itproved difficult to access comprehensive information on work in progress or recently completedindicator development work in the timeframe of the Project. Additionally, it appeared that someagencies were ambivalent about contributing to a developmental program for a nationallyconsistent set of quality and outcome indicators and may have tempered their contributions toour research in proportion to this ambivalence.Some examples of identified quality indicator applications include:Commonwealth The Commonwealth Department of Health and Family Services supports theACHS CEP clinical indicator program (see above). Within the Health Services OutcomesBranch it has groups looking at primarily patient safety, health models, consumer perspectiveson health, health outcomes policy, workplace change promotion and healthcare quality andoutcomes. Recent key initiatives include the funding of a study addressing the reliability, validityand risk-adjustment issues around four so-called “hospital-wide quality of care indicators” (i.e.wound infection, nosocomial bacteraemia, unplanned return to theatre and emergencyreadmission).The Health Models Development Section supports an Ambulatory Care Data Working Groupwhich has recently proposed performance indicators for Ambulatory Care with a potentialrelevance for the acute healthcare sector. These are:♦♦♦♦♦♦Waiting times in Emergency Departments - based upon The Australian College ofEmergency Medicines five triage categories.Waiting times for admission to hospital from the Emergency Department.Cost per outpatient separation.Patient satisfaction.Waiting times for outpatients appointments.Waiting times in outpatients per attendance.At present these indicators have conceptual definitions, with operational definitions andsubsequent field trials pending.

19The National Demonstration Hospital Program has identified performance indicators whichrequire development at three levels (program, project and hospital). To date, some 20 programlevel performance indicators have been advanced. They have crude operational definitionsprovided, which would be inadequate for routine applications. These indicators do not covermany of the significant dimensions of care quality (e.g. effectiveness, technical proficiency,safety) being restricted to access and efficiency indicators (as is the purpose of the program).They are:♦ Percentage of Category 1 patients waiting more than specified days.♦ Percentage of Category 2 patients waiting more than appropriate time.♦ What is the average excess wait (days) for patients classed as Category 1?♦ What is the average excess wait (days) for patients classed as Category 2?♦ By how many does the total waiting list number exceed appropriate levels in each category?♦ Trend in elective admissions from the waiting list this month.♦ Does the percentage change in all admissions exceed the percentage change in electiveadmissions?♦ Does the percentage change in admissions exceed the percentage change in annual funding?♦ Change in unplanned readmissions within 28 days.♦ Percentage of elective patients surveyed for satisfaction.♦ Level of overall elective patient satisfaction with total admission.♦ Is the waiting list at least equal to a month’s elective admissions?♦ Surgical patients treated per sessional hour per month.♦ Is operating room utilisation improving?♦ Percentage elective admissions on planned day of surgery.♦ Percentage of same-day surgical admissions.♦ Percentage of electives cancelled on planned day of surgery.♦ Patients treated per available bed per month.♦ Months since last audit based on direct patient survey.The Health Service Outcomes Branch, through the Quality and Outcomes Section, are funding atrial of an Integrated Quality Management Model which will involve, amongst a range ofactivities, the application of indicator sets within provider facilities. These will be developedaround the principle of the “Value Compass” (of Hospitals Corporation of America) 492 and willinvolve balanced indicator sets for specific clinical conditions incorporating indicators of clinicaloutcomes, functional health status, patient perception and cost. The actual indicator sets to beapplied are currently under development.The Commonwealth is engaged in developing numerous additional indicators, many of which arein areas outside acute health services. A complete list is included in Appendix 6.The Australian Institute of Health and Welfare play a pivotal role in the collection and analysisof current performance indicator data. They have been instrumental in establishing the interimpopulation norms for the SF-36, a functional health status questionnaire designed to measuregeneric aspects of health. They also have produced the First National Report on Health SectorPerformance Indicators for the National Health Ministers’ Benchmarking Working Group. Thisinaugural report contains indicators relevant to quality as follows:♦♦♦♦♦♦Cost per casemix adjusted separation.Cost of treatment per outpatient.Inpatient average length of stay for top 20 AN-DRGs.Rate of emergency patient readmissions within 28 days.Rate of unplanned return to operating room.Patient satisfaction.

20♦♦♦♦♦♦Proportion of facilities accredited with ACHS.Waiting times for elective surgery.Accident and emergency waiting times.Outpatient waiting times.Variations in intervention rates.Separations per 1000 population.The authors of this report emphasise that consistent national data is currently unavailable formany of these indicators and caution - appropriately - that they be interpreted in light of thesemethodological deficiencies. These caveats notwithstanding, it is likely that this initiative of TheAustralian Health Ministers’ Conference marks the first step on a journey to improvedperformance indicators of increased relevance and utility.Victoria The Department of Health and Human Services (then Health and Community Services)completed a comprehensive review of quality activities in acute healthcare in 1995 - the resultsof which are contained in “A New Framework for Quality in Victoria’s Public Hospitals: FinalReport” 357,358 . This identifies several major quality and outcome indicator initiatives acrossVictoria sponsored by The Department of Health and Human Services:♦♦♦♦♦Monitoring of unplanned hospital readmissions.Standard survey of patient satisfaction (based upon the Picker Commonwealth Instrument).A requirement for hospital quality plans.Subsidisation for participation in the ACHS hospital accreditation program.Monitoring of Emergency Department performance.Additional identified indicators included:• Percentage of private separations/facility.• Percentage of private bed days/facility.• Average available beds.• Average length of stay (by ANDRG).• Non-admitted patient occasions of service.• Separations/facility.• Same-day medical separations as percentage of all separations.• Non-same day emergency separations as percentage of all separations.• Non-same day elective separations as a percentage of all separations.• Nursing home type bed days and separations.• Regional separations as percentage of all separations.• Outside region non-tertiary separations as a percentage of all non-tertiary separations.• Non-tertiary separations from tertiary hospitals as percentage of all network non-tertiaryhospitals.• Tertiary separations from tertiary hospitals as percentage of all network tertiary hospitals.• Transfers between network campuses as a percentage of total separations.• Transfers to/from other networks as a percentage of total separations.South Australia Activities initiated through the Casemix and Clinical Costing Unit of the SouthAustralian Health Commission include:♦♦♦♦♦Emergency patient readmissions.Returns to theatre.Nosocomial infections.Health status assessment (SF36).Patient Satisfaction.

21♦♦♦♦♦♦Waiting times for elective surgery.Available bed days: Daily average.Cost per casemix adjusted separation.Cost per casemix treatment per patient.Inpatient average length of stay by DRG.Proportion of facilities accredited by ACHS.The South Australian Health Commission has also undertaken considerable research anddevelopment on utilisation review. This management tool particularly addresses issues aroundsite of care (such as appropriateness of admission) or intensity of care (such as numbers ofinvestigations per case). We regard utilisation review as an important facility managementinstrument with little application in any national quality and outcome indicator programaddressing acute hospital care and have not included review of this, or other utilisation reviewprotocols, within our review.Australian Capital Territory The Department of Health and Community Care identifiedevaluation of the SF36 health status survey as a relevant activity for quality and outcomeindicator development, occurring under the auspices of the Care Continuum and HealthOutcomes Project. Additional developmental work is currently underway in:• Ambulatory care.• Palliative care.• Breast cancer.• Hospital-based cancer care.• Discharge planning.• Asthma.• Injury.• Orthopaedic rehabilitation domiciliary service.• Cardiac disease.Western Australia The Health Department of Western Australia indicated that a number ofperformance measures had been applied in the past, although existing purchaser/providercontracts have not had any quality indicators reported. A sample of past performance measuresincluded:♦ Average cost per bed day.♦ Average cost per admission.♦ Average cost per non-inpatient occasion of service.♦ Average length of stay per admission.♦ Median wait time from hospital waiting list.♦ Percentage of clients satisfied with their hospital stay.♦ Overall satisfaction index.♦ Percentage of hospitals operating quality assurance programs.♦ Percentage of hospitals accredited.♦ Hospital acquired wound infection.♦ Apgar score of four or less at five minutes for newborns whose birthweight was at least 500grams.♦ Geographic variation in admission rates per 1000 population.Tasmania Under a “Health Goals and Targets for Tasmania” program, a process to identify keyhealth issues was commenced in May 1992. Currently there are no state-wide indicatorinitiatives, although Regional Business Plans have an agreed set of performance indicators -including some quality indicators - against which they report. These indicators include:

22♦ Emergency Department waiting times by triage code.♦ Outpatient waiting times by specialist area.♦ ACHS CEP hospital-wide medical indicators (nosocomial infection, unplanned return totheatre, post-operative pulmonary embolism and unplanned readmission).Northern Territory No state-wide indicator collection is currently occurring. Major providerfacilities (e.g. Royal Darwin Hospital) are collecting ACHS CEP hospital wide medicalindicators, including:♦ Unplanned return to the operating room.♦ Nosocomial infection.♦ Unplanned readmissions.Queensland Queensland Health have data available on:♦ Client satisfaction with Accident and Emergency Department care delivery.♦ Separation rates.♦ Cost of care delivery.They support pursuit of ACHS accreditation (and hence indirectly promote use of ACHS CEPclinical indicators). They noted a current inability to track and report elective surgery accesstimes because of limitations to existing information systems. They identify ongoing projectsaround waiting lists, commercial key performance indicators, client services standards and bestpractice implementation.New South Wales The New South Wales Health Department has been reviewing the utility of anumber of ACHS hospital-wide medical indicators in quality assessment (including unplannedreadmission, unplanned return to theatre) and have an extensive developmental programaddressing issues surrounding the identification of nosocomial infection. In addition, routinemonitoring of elective surgery queuing times and emergency department waiting times isoccurring. Pilot patient satisfaction surveys have been performed across NSW hospitals,including an emergency department survey.Specific indicators currently applied by New South Wales Health include:♦ Unplanned readmission to hospital or inpatient health service within 28 days of dischargefrom an inpatient health service where the original admission had been for an electiveprocedure.♦ Patients assessed in Outpatient Department within 1 month of referral.♦ Patient satisfaction with their access to information.♦ GP satisfaction with their access to information.♦ “Urgency 1 and 2” patients waiting longer than 1 month.♦ “Urgency 3” patients waiting longer than 6 months.♦ “Urgency 3” patients waiting longer than 12 months.♦ “Urgency 3” elective surgical patients given definite dates of admission with 12 weeks ofplanned procedure.♦ Elective surgical patients attending preadmission process.♦ Elective patient satisfaction with their waiting time for inpatient care.♦ Elective patients having their planned admission date brought forward.♦ Elective patients admitted on day of procedure (selected).♦ Patients having an unplanned return to operating room.♦ Patients having a procedure as a “same day” patient (selected).♦ Patients discharged on anticipated date.

23♦ Patients informed of their discharge date on admission.♦ Relative stay index for 5 elective surgical procedures.♦ Elective patients delayed (not self deferred).♦ Waiting time from Emergency Department arrival to be seen by medical officer (by triagecategory).♦ Emergency Department treatment time: from time seen by medical officer to admission toinpatient bed.It should be noted that existing State and Territory programs have paid little attention toascertaining the reliability and validity of indicator data provided to central indicator monitoringprograms. Typically, indicator data are used to aid policy implementation and/or to satisfyrudimentary accountability requirements. The only research and development activitiesidentified by the Project team primarily directed at acute care quality and outcome indicators isthe nosocomial infection indicator work underway within NSW 360 . Overall States andTerritories are tending to promote ACHS-CEP hospital-wide medical indicators, rather thandeveloping de novo indicator programs, although the operational definitions for these indicatorsare frequently modified by individual States and Territories to suit the unique requirements oftheir data handling systems. As a result, even the information available on apparently similarACHS CEP-based indicators may not be comparable between States and Territories.The State and Territory indicators have largely been selected from those in common usagenationally and internationally. Typically, they are readily available from routine data sources.Those indicators requiring additional data collection have frequently not yet been introduced intoroutine monitoring programs. Many comments on the lack of knowledge of the reliability andvalidity were made. Concerns were expressed frequently about lack of indicator risk adjustment.Although some limited evidence of the application of indicators in local quality improvementefforts were reported, typically little was known of the utility of indicator data. Often thoseindicators in use for longer periods had more caveats and concerns regarding their usefulness -perhaps reflecting accumulated knowledge of their limitations with continued application (forexample, many reported uncertainty regarding the value of the unplanned readmission indicator).5.3.2 North America The past two decades have seen an explosion of activity in quality andoutcome indicator development and implementation in North America. The range of indicatorsapplied and the multitude of agencies developing and implementing indicator programs makes acomprehensive review of these activities a daunting task. We have elected to present briefsummaries of programs that illustrate seminal principles in indicator development, novelindicator strategies or competing views on indicator applications. Additional detail on NorthAmerican quality indicators is contained in Appendix 6.NATIONAL PROGRAMS/MULTISTATE PROGRAMS OF NOTE IN USA1. Joint Commission on Accreditation of Health Care Organisations (JCAHO)Indicator Measurement System (IM System)2. National Committee for Quality Assurance (NCQA)Health Employer Data and Information Set (HEDIS)3. Maryland Hospitals Association (MHA) Quality Indicator Project (QIP)4. Consortium Research on Indicators of System Performance (CRISP)

24JCAHO IM System: This quality and outcome indicator program has been under developmentsince 1985 14,138,144 . As at 1996 it consists of 33 indicators covering obstetrics, perioperativecare, cardiovascular, trauma, oncology, infection control, medication use and health carenetwork performance. In 1997, additional indicators for depressive disorders, long term care andhome infusion therapy are planned for introduction.The JCAHO launched the IM System with a view to making it the pre-eminent indicator programin the USA - the use of which would eventually be mandatory for providers wishing to seekJCAHO accreditation 142 . There was considerable industry opposition to a clinical performancemonitoring system developed by a regulatory body. This opposition and the desire by JCAHO tomandate IM System use broadly across the USA resulting in a rigorous developmental programwhich has actively sought information regarding the reliability, validity and relevance ofindicators.The IM System has detailed operational definitions available for all indicators 142 (see examplefor nosocomial infection: Appendix 13) which help ensure data reliability. An extensiveresearch program, based around an initial (Alpha) testing for face validity and indicatorfeasibility is followed by a more exhaustive second phase (Beta testing) examination addressingreliability, validity and relevance issues. Whilst some tens of facilities participate in Alphatesting, surveys indicator data collection and on-site visits for several hundred facilities occur inBeta testing - providing firm data upon which to base judgements on indicator attributes.The IM System is based around a common software program which guides accurate datacollection and includes audit-checks for data reliability encoded in the software. A substantialresearch and development group and service infrastructure support indicator application byfacilities. Despite these features, it has enjoyed only modest penetration in the US market -reflecting competing indicator systems, concerns that the IM System is still developmental andresistance because of its origins in a regulatory body. In 1996 JCAHO elected to discontinue theplanned mandating of IM Systems for the purposes of JCAHO accreditation, instead opting toreview and approve any indicator that fulfilled their criteria for methodological rigour. A call forindicators was made, inviting submission of any existing indicators or indicator sets to JCAHOfor review by their expert panel. At completion of this review process, the JCAHO will publisha compendium of quality and outcome indicators which are deemed sufficiently wellcharacterised to meet their methodological rigour requirements and hence be approved for use infacilities seeking JCAHO accreditation. This compendium will offer a valuable entree to thestate-of-the-art in indicator science in North America (it is scheduled for release in the NorthernHemisphere Autumn 1996).The JCAHO IM System provides facilities with risk-adjusted 144 indicator data based uponpatient, not hospital, characteristics. Individual risk adjustment methodologies are designed foreach indicator - typically using their database to generate “predicted” indicator rates with whichfacilities can compare their actual rates per indicator.NCQA: HEDIS HEDIS was designed to permit employers to judge the value of the healthcarethey are buying for their employees and to make health plans accountable for their performance307,313 . The NCQA - an accreditation body that focuses on health plans rather than hospitals andis the major player in this market - has developed the HEDIS measure set in three iterations (thusfar), i.e. HEDIS 1.0, HEDIS 2.0 and HEDIS 2.5. The HEDIS measures were a response to theproliferation of competing indicator systems amongst health plans which were typically notproducing comparable data. HEDIS was seen as providing a common language for guidingchoice of health plans. The measures were chosen by a Performance Assessment Committee(PAC), consisting of employers, NCQA and technical expertise representatives. To date, no dataare available on indicator/measure attributes, such as reliability or validity, although it is planned

25to collect such data for the HEDIS 3.0 measures currently under development. HEDIS 2.5includes more than 60 measures across the broad areas of:♦♦♦♦Quality of care.Member access and satisfaction.Membership and utilisation.Finance.The particular measures within each of these major areas were chosen on the basis of theirperceived relevance and value to the employer community, the reasonable ability of health plansto provide the requested data and their potential impact on improving health care delivery. Themeasures are a mix of population-based performance and episode of care performance indicators(see Appendix 6). Those specific for acute episodes of care could feasibly be adapted for use inacute healthcare in Australia. At present HEDIS data, although frequently stratified, are notrisk-adjusted. Relevant indicators in HEDIS 3.0 are proposed to be risk-adjusted using indicatorspecificmodels developed around patient level data. NCQA has also had a “Call for Indicators”in 1996, seeking indicators suitable for inclusion in HEDIS 3.0.MHA: QIP The oldest of the multifacility indicator monitoring programs, the MHA QIPprovides comparable indicator data on 10 inpatient quality and outcome indicators and 5ambulatory care indicators to over 1000 participating facilities in North America, Japan, Europeand the United Kingdom (Appendix 6). It is the largest single indicator monitoring program inexistence anywhere in acute healthcare. The Project seeks to educate participants in the use ofquality indicator data for quality improvement. The MHA QIP guarantees confidentiality for allparticipant facilities. It does not risk-adjust data, primarily because of a belief that it isunnecessary as indicators are intended only for facility quality endeavours and not comparativejudgement by others. Indicators have available data on reliability, validity and relevance -collected over their 10 years of experience with indicator use. The software provided to facilitatedata collection contains data reliability checks. There are several documented case-studiesavailable which demonstrate favourable strategies developed in response to initial indicator datareview (such as a progressive reduction in absolute rates and the extent of variation betweencentres for Caesarean section in New Hampshire).CRISP This is a voluntary collaborative program administered from The Henry Ford HealthSystem in Detroit. Eighteen integrated healthcare systems participate in a research programwhich aims to identify preferred indicators of healthcare delivery performance. Indicators aredivided by category:♦♦♦♦♦♦♦♦♦♦♦Population health.Community benefit.Quality of care.Episode prevention.Episode characteristics.Satisfaction.Efficiency.Capacity.Financial performance.Research.Education.At present, indicators lie in two tiers, the first representing indicators of established reliability,validity and utility and the second tier a group of developmental indicators 300,301,302,303 (seeAppendix 6 for further detail). CRISP uses risk adjustment to increase the comparability ofperformance indicators across the systems. The risk adjustment models are developed for each

26indicator and based on patient level data, typically demographic descriptors (e.g. age, sex,race/ethnicity, educational attainment). CRISP maintains all indicator data as confidentialcommunications - arguing that, at present, the data lie within a research program, are notsufficiently well characterised for further release and failure to protect participant confidentialitywould compromise the ultimate development of credible indicators suitable for public release.There are several other noteworthy examples of quality and outcome indicator applications inNorth America. They include:CONQUEST Computer needs-oriented quality measurement evaluation system (CONQUEST)is a system of interlocking databases summarising information on approximately 1200 clinicalperformance measures used by public and private sector organisations to examine technicalquality of clinical care. Developed as a collaboration between the Centre for Quality of CareResearch and Education at the Harvard School of Public Health (in Boston) and the Centre forHealth Policy Studies (in Colombia) it is essentially a typology of clinical performance measureswhich allows identification of measures on the basis of characteristics or properties of thesemeasures. This software is based upon a research project commissioned by the Agency forHealth Care Policy and Research (AHCPR) in 1994. It can be used to search for measuresunder:• Name.• Rigour of development.• Organisation type developed for.• Intended use.• Extent of use.• Practicality.• Clinical event.• Age of patient.• Care needs.• Care setting.• Process/outcome.• Data source.• Sampling frame.• Stratification method.• Time window.• Allowance for patient factors in scoring.• Comparative standards.• Display of results.Its widespread availability (anticipated for mid-1996) to health service researchers should speedaccess to information on existing indicators and indicator sets.FACCT: The newly established Foundation for Accountability plans to endorse qualityindicators, advocate their widespread use by providers and promote consumer use of endorsedindicators. They have developed criteria for assessing performance indicators for use byconsumers (Guidebook for Performance Measurement) and are likely to contribute significantlyto the adoption of indicators by health plans and acute care users across America.Veterans’ Affairs: The US Department of Veterans Affairs has an active research programencompassing a range of studies on the use of process and outcome indicators in healthcare, therelationships between selected process measures and observed outcomes, the reliability andvalidity of particular indicators and the application of patient-based measures of health status,process and outcome in performance assessment. They have refined risk-adjustment

27methodologies for use in veterans populations and developed instruments to facilitate technicalproficiency assessments. The VA perform much of the detailed methodological researchnecessary to ensure that indicators are appropriately used for quality monitoring in their acuteand integrated healthcare systems.Cleveland Health Quality Choice: This exemplary program was the first healthcare marketreform plan of its kind in the USA which brought business, providers, physicians and expertisein quality measurement together in a voluntary collaborative effort to measure and improve thequality and affordability of healthcare services. It commenced in 1989. It has developed its ownindicators to measure and compare the quality of selected services at participating hospitals inthe Greater Cleveland area. It evaluates four service areas:• Surgery: e.g. Total hip replacement• General Medicine: e.g. Pneumonia treatment• Intensive Care: e.g. Respiratory failure• Obstetrics and Gynaecology e.g. ChildbirthIt reports utilisation, in-hospital mortality, length of stay and patient satisfaction across a rangeof high volume or high cost interventions (see detail: Appendix 6). All data is risk-adjusted,using the Acute Physiology and Chronic Health Evaluation (APACHE III) System of adjustmentfor intensive care patients and a customised, patient-level risk adjustment system for noncriticallyill populations. The quality indicator data is first validated by a stringent reviewprocess involving independent experts, the CHQC and providers. Following the confirmation ofdata reliability and validity it is released publicly. The initial development and confirmation ofdata credibility prior to public indicator data release required a three year cooperative program.Semi-annual consumer reports are now published which provide indicator data and guidancewith interpretation. A summary format is available to the public. Because of the complexity ofthe complete set of data and the sensitivity associated with the interpretation of the 300 page pluscomprehensive report, it is mandatory to attend a half day training workshop in appropriate useof the data before being provided access to the comprehensive report. This program,components of which has been replicated in various guises across North America, contains anumber of cardinal features which have determined its success, including rigorous developmentof credible measures and the cooperative development program involving providers andpurchasers.New York and Pennsylvania States: These States collect, calculate and publish information onquality of care obtained from mandated hospital discharge data sets. They report risk-adjustedmortality for Coronary Artery Bypass Grafting (CABG), major morbidity, length of stay,charges and physician-specific CABG mortality. These have been controversial programs, attimes appearing confrontational. There is no evidence that consumers’ or providers’ behaviourshave changed favourably as a consequence of public release of this data. Initial reports ofimproved patient outcomes with CABG proved to be due to conscious or unconsciousmanipulation of contributed data to favourably realign risk-adjusted mortality prediction. Thecosts of wide-scale data collection for the risk-adjustment methodologies used - and for systemsto audit data reliability - are considerable.HCFA’s Mortality Analyses: In 1986 the Health Care Financing Administration (HCFA)released via the media risk-adjusted overall mortality data for US hospitals treating Medicarepatients. For several subsequent years, HCFA continued to release these outcomes data asputative quality indicators, despite concerns about the accuracy of administrative databasesunderpinning indicator construction and the detail of the risk-adjustment models used to predictwhether quality of care fell within likely acceptable bounds. Considerable energy and expensewere devoted to identifying the deficiencies of this attempt at public accountability. EventuallyHCFA itself ceased publication of such indicator data when internal review failed to demonstrate

28that mortality categorisation of quality of care aligned with independent determinations of carequality (i.e. the approach was invalid). In many respects this experiment is a model for how notto introduce putative quality of care and health outcome indicators.HCFA’s Health Care Quality Improvement Initiative: As part of a new focus on qualityimprovement, HCFA’s HCQII is seeking to improve the mainstream of care, rather than relyingon identifying isolated apparent error. The initiative seeks to get doctors and hospitals involvedin the analysis of treatment patterns and promote improvement through adherence to bestpractice processes. The initiatives see HCFA, doctors, hospitals and specialty societies worktogether to develop indicators of quality of care. HCFA then facilitate data collection, analysisand confidential feedback to providers and promote use of the data for internal qualityimprovement and benchmarking activities.An example of this program, focused on care of patients with myocardial infarction, saw 26proposed quality of care indicators refined to 12 final quality indicators (see Appendix 6 fordetail). These indicators reflect the provider focus of these activities. They are largely measuresof technical proficiency based upon acknowledged guidelines for the management of acutemyocardial infarction. The approach taken by HCQII has much to recommend it and reflects therealisation by HCFA that its previous unilateral approach to quality monitoring, informingconsumers and accountability (see HCFA Mortality Analysis - above) was unsuccessful.Health Care Report Cards: We have identified three dozen or so Health Care Report Cards,released to the public in the USA in recent years (predominantly post 1993). They contain avariety of putative quality and outcome indicators (see Appendix 6). Typically, there is no detailas to the operational definition for presented indicators nor for data reliability or validity. Giventhe “for profit” status of many groups generating these self-reports of performance, it issignificant that none of the reports contain data independently audited for accuracy. It is open todebate how valuable the contained information is to healthcare consumers - as the targetaudience for these initial attempts at consumer reports was frequently healthcare purchasers,rather than the direct recipients of service delivery.5.3.3 United KingdomThe predominant focus of activities in the UK has been the reinforcing and standardisation ofClinical Audit and an aligning of indicators of quality and outcomes of care with the NHSPatient Charter. The UK has been at the forefront of the evolution of so-called “Evidence-BasedMedicine” and are currently active in designing quality and outcome indicators within thiscontext. There has been relatively little in the way of comprehensive national quality andoutcome indicator programs to date - although a report from the Clinical Accountability andSystem Performance Evaluation (CASPE) Research Group to the Department of Health(anticipated for the European summer, 1996) will identify quality and outcome indicators for tencommon health conditions (including asthma, diabetes, heart failure, bladder outlet obstruction,childbirth, back pain, hypertension, ischaemic heart disease and menorrhagia) that span theinterests and perspectives of patients, providers and purchasers of health. Many of theseindicators will have data on reliability and validity based upon field-testing over the past threeyears. It is unclear at present what use the Department of Health will make of these indicators.They are targeted at assessment of an integrated healthcare system, rather than hospitals, but arelikely to include indicators of relevance to acute healthcare.The National Health System (NHS) so-called “League Tables” have been published for severalyears - encompassing a range of indicator data including waiting times and overall mortality.The data is not risk-adjusted and is not audited. There is no evidence that it has materiallyaltered provider behaviour with regard to care delivery. Several academic departments andresearch consortia are active in the development, characterisation and promotion of health statusand quality of life assessment instruments (e.g. The London Handicap Scale). Whilstcontributing much to the characterisation of such instruments, there is little evidence of attempts

29to systematically introduce these outcome measures into current practice on a wide scale. Recentchanges in funding of acute healthcare, with enhancement of the purchaser/provider split (socalled),are seeing increasing sophistication in contractual arrangements between those fundingcare and those delivering care. Based upon the examples made available to us, there was littleevidence of sophistication in quality performance indicators within these contracts with mostemphasis on volume and cost. Those indicators used were “conventional” (e.g. patientsatisfaction; wound infection). Accreditation of acute hospitals is a relatively recent concept inthe UK. The King’s Fund Organisational Audit Unit is the predominant force in accreditation.They currently adopt a standards-based approach, not dissimilar to the ACHS prior toimplementation of the Charter for Change. There are no requirements for the routine applicationof quality of care indicators for the purposes of accreditation.In the absence of a systematic national approach to quality and outcome indicators, the NHS hasfunded a regional trial of the feasibility of UK hospitals’ participation in the MHA QIP. Thistrial of participation, coordinated from Newcastle University, has been judged a success by thoseinvolved. It is currently being expanded to involve more facilities. It provides an interestingmodel for “instant” access to an established quality indicator program, providing comparativedata for quality improvement at a relatively modest cost. Such an approach may prove attractiveto individual hospitals or hospital groups in Australia, although the confidentiality requirementsof MHA QIP would preclude its use as a substitute for a national monitoring program.5.3.4 EuropeMost European countries are involved in active, large-scale quality monitoring programs. Themajority are based on cooperative, quality improvement focused activities (such as the WorldHealth Organisational confidential indicator feedback programs in diabetes care, hypertensionand dental care). In large part, these programs are similar to initiatives described for NorthAmerica and are using indicators that are conceptually similar, although operational definitionstend to be unique as they are tailored for local practice patterns and information systems.Exemplary programs identified include:♦♦♦WHO collaborative demonstration project: This has established quality and outcomeindicators for common conditions (e.g. diabetes mellitus and dental caries) and fed-backconfidential reports to individual clinicians on their own performance vis-a-vis theseindicators (e.g. retinopathy screening for diabetes or caries rates for dental care). Linked toeducational programs based around benchmark processes of care (identified through theseprograms), these programs have seen a progressive and substantive improvement in auditedoutcomes.Establishing Quality Improvement in Healthcare in Spain: Over the past 10 years theMinistry of Healthcare and Consumer Affairs in Spain has been promoting a program tointroduce routine performance measurement and the utilisation of measured performance forimprovement. This program has focused on ensuring that data collected to construct qualityindicators is reliable and continues to work to develop appropriate risk-adjustment modelsbased on patient-level data. Initial confidential data collection and validation was followedby collaborative public release of conventional indicator data (e.g. mortality; queuingtimes).Norway’s Contract for Quality: Norway has established explicit access targets forhealthcare providers based upon designated medical or surgical conditions. These targetsstate access standards (e.g. waiting times for elective surgery or ambulatory specialtyreview by condition) but do not as yet stipulate outcome targets.

305.4 Data Sources for Indicator ConstructionThere are essentially four means of obtaining data for indicator construction:a) Administrative Databasesb) Medical Recordsc) Patient Surveysd) Hybrid Methods (i.e. more than one of the above).5.4.1 Administrative DatabasesLarge databases established for other purposes (e.g. billing; risk management; utilisation review;costing) are increasingly able to be linked together and/or applied to quite different purposes,such as quality indicator construction. There are inherent dangers in using data obtained for onepurpose in a different context, largely centred on data accuracy. Typically, data items perceivedto be of little direct relevance to the primary intent of data collection (e.g. comorbidity recordedin some billing databases) are poorly collected and hence inaccurate. These inaccuracies tend toresolve over time when it is appreciated that these data have an important secondary purpose -albeit of little relevance to the primary purpose of data collection.More serious limitations of administrative database are their lack of clinical detail relevant toquality indicator construction. This absence of significant clinical detail (e.g. history andexamination findings, investigation results and a full intervention profile) severely limit theirusefulness for construction of clinically relevant indicators. Common coding systems (such asAN-DRG and ICD9-CM) frequently group conditions and interventions which do not logicallybelong together for process or outcome analyses - and it is not possible to “unbundle” thesedisparate groups after their numerical codes have been allocated. Administrative databases tendto offer more complete data at considerably less expense than alternative data sources. Untilcomprehensive, compatible clinical databases are developed around computerised medicalrecords, indicators derived from administrative databases will remain attractive, low-cost optionsfor those implementing large-scale quality and outcome monitoring programs.5.4.2 Medical RecordsExisting paper medical records are a rich source of narrative detail on an individual episode ofcare. They are, however, frequently incomplete data sources and inherently of variable contentand quality. The need for trained data abstracters makes data derived from medical recordsexpensive. Differences between the content of different medical record systems may hamperbroad-based monitoring programs - although knowledge of the need to identify particularinformation within the medical record for subsequent quality monitoring efforts would be likelyto drive standardisation of entry of these data points nationally.5.4.3 Patient SurveysPatient surveys provide vital information about the acceptability of care delivery, reports ofexperiences with the processes of care and health status and quality of life, patients’ healthbehaviours and intervention-specific outcomes. In recent years it has been established thatpatients provide accurate and reliable estimates of all these aspects of care - with the onlydemonstrated limitation being in reports of resource utilisation (where they consistentlyunderestimate utilisation rates in most reported survey comparisons). Although relativelyexpensive to administer, patient surveys provide invaluable information that cannot be obtainedfrom any other source. Traditional written surveys are increasingly complemented or replacedby verbal (usually telephone) surveys or computer-aided surveys. There is also a trend to theamalgamation of components of specific-purpose surveys to create omnibus measures,generating data on several different domains of quality of care. These might include:

31• Patient details (e.g. age/sex/self reported diagnosis)• Description of processes of care (e.g. waiting times, communication, dischargeplanning)• Functional health status (e.g. SF36) or health related quality of life• Acceptability of care (e.g. satisfaction)• Outcomes of care (e.g. visual acuity after cataract extraction).Hybrid methods combine the advantages, and the disadvantages, of the above methods. Inparticular, they sum the cost burdens, and typically should be avoided unless the clinicalcircumstance demands the complexity for credible analysis, thereby justifying the added cost.Many indicators would more readily be derived if requisite data were mandated in universalminimum data set requirements. However, many quality and outcome indicators may only berelevant for a relatively short period of time, as advances in medical care result in newperformance indicators, supplanting previous indicators (such as in heart attack, where time tothrombolytic therapy has replaced use of anticoagulant drugs). Most indicators should be builtaround samples of care. Only when indicators are deemed robust, useful and likely to provedurable in the longterm, should there be consideration of revising minimum data sets solely forindicator construction. Although much useful information exists within Australian minimumdatasets at present, quality and outcome indicators should not be restricted to all conceivablepermutations and combinations of existing data within administrative databases. Usefulindicators will require extending data collection into medical record abstraction and patientsurveys, with very occasional need to modify the uniform data set, most probably to obtainrelevant risk-adjustment information. Existing administrative coding systems (AN-DRG andICD classifications) may require alterations to enable easy quality and outcome indicatorconstruction.Future revisions of these categorisation systems should seek input from those engaged in qualitymonitoring within hospitals to maximise any potential for routine indicator construction withminimisation of duplication of effort.5.5 Examples of Indicators by Dimensions of QualityThis section of the results discusses a range of the specific indicators identified by us over the course ofthis Project. As already indicated, there are literally several thousand indicators in use (or proposed foruse) worldwide. It is not practical - and we believe it undesirable - to discuss each of these severalthousand indicators serially, with comments on their indicator attributes. Rather, we will discuss measuresthat have some promise as future indicators in a nationally consistent quality and outcome indicator setand will illustrate, by example, indicators which failed to impress our reviewers - highlighting the specificcriteria with which substantive deficiencies were identified. More detailed discussion of each dimension isprovided within Appendices 11 and 12.5.5.1 Access♦Waiting TimesMany quality monitoring programs use the concept of waiting time as the principal index ofaccess to care. Hence, waiting times are recorded for:: elective surgery...* by urgency category* by procedure* by specialty

32: outpatient appointments* by urgency category* by specialty: emergency departments* by urgency category: emergency admission: outpatient waiting time at episode of careThe use of any of these queuing indicators in initial national indicator sets would requirethat adequate data sources exist or could reasonably be anticipated being brought intoexistence over a short timeframe. Waiting times are typically recorded by providers andvarying operational definitions (e.g. of waiting time, urgency categorisation) render manyindicators of apparently similar concepts non-comparable.There has been a move to standardise these operational definitions and to monitor queuingtime from the decision to deliver care to actual care delivery (sometimes called “decision toincision” time) to avoid gaming of queuing indicators and allow independent audit of actualperformance.There have also been moves to stratify conditions in some relationship to need forhealthcare. This is either on the basis of perceived global urgency (as is common practice,in various forms, in Australia currently) or specific diagnostic groupings (as is the case inseveral Scandinavian countries currently). Global urgency measures leave the decisionregarding the need for care with clinicians providing care. Narrow diagnosticcategorisation, although superficially attractive, would still require clinical judgement abouturgency and could provide incentives to shift diagnostic classification to suit queuingtargets. The necessity to retain urgency categorisation and add detailed categorisation wouldmake queuing data management difficult. Consensus on target waiting times within eachcategory and between categories may be difficult to obtain in provider communities, assingle procedures such as coronary artery disease requiring CABG may be eitherurgent/semi-urgent or non-urgent depending on the clinical circumstance on the patient.The use of patient-reported waiting times and judgements on the acceptability of reportedwaiting times are commonplace in North America and Europe. They accurately reflectindependently observed waiting times and are not subject to gaming by providers.Satisfaction with waiting times from the consumer perspective is relevant - and of particularvalue when interpreted in the light of the actual wait time. Satisfaction is largely dependentupon consumer expectations - which are moulded by numerous patient, provider andsocietal influences. Patients are more likely to report the total waiting time (i.e. time fromdecision to treat to receipt of treatment) than provider-based indicators such as those fromdecision time to the time of allocation of a booking for the procedure.Refinements of waiting time indicators include:• Clearance time: the time that would be required to clear the current waiting listusing current hospital elective surgery throughput estimates.• Proportion of patients admitted after waiting inappropriately.(See also discussion on Commonwealth access and efficiency indicators - within 5.3.1 -for additional refinements proposed by the National Demonstration Hospitals Program)

33An indirect indicator of access - patient emergency department walkouts after triage - hasbeen proposed as a useful indicator of emergency care access. Although influenced bywaiting time, several patient-related factors strongly impact upon absolute walkout rates(e.g. presence of psychiatric illness and/or substance abuse). Although a useful facilitylevelindicator for monitoring trends over time, we cannot recommend it for nationalapplication in the absence of suitable risk-adjustment for casemix. The experience of theMaryland Hospitals Association with this indicator supports our view - with widelydiscrepant “walkout” rates between inner urban and other community facilities which havebeen relatively fixed for given facilities with particular geographic and casemix profilesover lengthy time intervals.♦Preventable Admissions/Avoidable DeathsThese are excellent indicators of the adequacy of primary care although at best indirectindicators of access to appropriate acute care services. They have a real role in assessingquality in an integrated healthcare system. However, the inability to dissect failures inprimary care delivery from any index of acute care access render them of limited value injudging quality of acute care services. Comparisons between regions or facilities wouldalso require knowledge of the relative prevalence of the index conditions in the servicedpopulation. As future indicators of overall health sector performance they are stronglysupported, but they are not of value as acute healthcare quality indicators.♦Condition Specific UtilisationThere have been attempts at inferring access to care based upon the relative utilisation ofapparently clinically related interventions/conditions. Although offering useful insights onclinical care patterns, these are of limited value as access indicators.e.g.Access to acute hospital care for ischaemic heart disease measured by the ratio ofIHD admission to IHD death.COMMENT: This would be confounded by severity of illness and quality of care andwould tell little of access but much about the appropriateness of accessed care,although desired indicator rates are unknown.e.g.Access to CABG measured by rates of CABG compared with rates of angiography.COMMENT: This utilisation statistic would infer more about clinical practicepatterns regarding the appropriateness of these interventions than access.♦Geographic Access MeasuresAlthough some complex models exist to define geographic access to care, these are usuallyderived to justify policy decisions regarding relocation of services or guide future planningof service profiles. Most studies of consumers identify time to service delivery as crucial -rather than distance - with a hierarchy of acceptable times based upon the level ofsophistication and complexity of care required. Assumptions about transportation modelsinherent in most geographic access models (e.g. public transport versus private vehicle;motor vehicle versus air ambulance) further limit their practical value as access measures inacute healthcare.

34♦Stratified Utilisation DataKnowledge of the age, sex, ethnic background and social class of those receiving acute careservices, coupled with knowledge of disease prevalence in these population strata, enableinferences about relative access of care. Such analyses are particularly valuable as pointersto groups who have needs for care but fail to access care. Linking of such utilisation rateindicator data to population health needs data (such as from the Australian Bureau ofStatistics Health Status and Health-Related Behaviour Survey) would be a mechanism bywhich indicator data could assist decision-making on access.Utilisation (e.g. discharge rate per 1000 population; average length of stay) can becompared within populations without knowledge of prevailing disease patterns as indices ofattained/achieved access. Such measures in an Australian context probably tell more aboutappropriateness of care than access, as rationing of particular interventions is rarelybelieved to be preventing receipt of appropriate care.♦Unmet NeedThese survey methodologies seek out patient reports of conditions warranting acute care(e.g. chest pain) and establish what care was sought and provided. The survey methods areexpensive and only applicable on larger population samples. They do, however, produceaccurate statistics on how successfully potential needs for care are matched by patterns ofexisting care delivery.♦Ambulance Bypass5.5.2 EfficiencyThis indicator has been applied in some jurisdictions as an index of access to emergencyservices. Our majority opinion is that this indicator has too many limitations to warrantinclusion in a national indicator program. It reflects access to services for a very smallpatient population judged to be not critically ill (i.e. patients using ambulance transport tohospital deemed non-time-critical by ambulance officers) and is amenable to gaming (as thedecision to declare an emergency department on bypass is arbitrary and based solely on thesubjective judgement of an emergency physician).Technical Efficiency♦Cost/Activity RatiosThis most common technique for measuring the technical efficiency of hospitals essentiallyanalyses the ratio of cost to some index of activity (i.e. quantity of delivered care). Theseindicators include:• Cost per Casemix adjusted separation• Cost of treatment per outpatient episode• Comparative average length of stay (ALOS).Whilst conceptually simple, costing data that is accurate, reliable and comparable is notreadily available in most healthcare systems - and is certainly currently not available withinAustralian hospitals. It would appear preferable to commence with simple measures ofefficiency and ensure that data used to construct these are accurate, reliable and comparablebefore progressing to more complex measures. Data on ALOS are more readily availableand potentially reliable. Their use as efficiency measures assumes that DRG outputs areequal in health outcome terms and that close relationships exist between cost and ALOS -

35assumptions that may not be reasonable given condition and severity differences withinadministrative database patient populations based on AN-DRG and ICD9-CMclassifications and differences in inpatient bed utilisation patterns such as those betweenurban and rural providers. Data on cost-per-casemix adjusted separation are currentlyprobably less accurate and considerably less comparable than ALOS data. Use of thisindicator will undoubtedly motivate improvements to data quality. At present we believecost data on outpatient episodes of care is likely to be so inaccurate (by virtue of the verylimited application of information technology in outpatient costing) as to warrant avoidingrecommending its widespread use as a quality indicator in the short-term.♦Cost Frontier Estimation (or Production Frontier Estimation)These complex techniques, such as Data Envelopment Analysis, take account of themultiple inputs (such as labour, use of capital) and outputs (such as inpatient and outpatientservices, teaching, research) that are identified in acute healthcare. They involve modellingof inputs and outputs to generate weighted sums of inputs and outputs. The results of thismodelling can only be used to compare similar types of healthcare providers 368-402 . Thearbitrary allocation of weights to given inputs or outputs allow skewing of any modeltowards/against a significantly different service provision pattern (for example, if hospitalsdo not provide some services, weighting of these outputs appropriately in the model couldresult in apparently efficient or inefficient performance by comparison with a facilityproviding that service). Like all sophisticated mathematical models they depend heavily onthe precision of the manipulated original data points 403-404 . As indicated above, theaccuracy and comparability of currently available input and output data in Australianhealthcare would indicate that cost frontier estimates (such as Data Envelopment Analysis)are not realistic options for efficiency monitoring nationally at the present time.Allocative Efficiency♦The Oregon ExperimentThis well-publicised experiment in priority setting in healthcare produced a QualityAdjusted Life Year (QALY) league table based upon comparative cost-utility analyses.Although the approach is generally considered to have merit, it is compromised by the grosssimplifications in the costing of health interventions, the limited scope of services coveredby the model and the way QALYs were measured 407-411 .The requirement for marginal analysis as distinct from average analysis was ignored in theexperiment. Rankings of health interventions were based on average cost-utility ofinterventions, without due recognition that at the margin, cost-effectiveness will vary withpatient population, size of program and specific program characteristics.♦Program Budgeting with Marginal Analysis (PBMA)The PBMA approach has been developed by the Health Economics Research Unit inAberdeen and is being trialed by the NSW Health Department, through the Centre forHealth Economics Research and Education 350,406 .The model focuses on the marginal levels of health gains and health losses associated withparticular healthcare interventions. In these terms priority is given to expanding programswhere the resultant health gains would be greatest, with a reduction in health services orprograms where the losses are minimal. With some modifications this approach has thepotential to be applied to the acute health sector.

36PBMA requires health service providers or funders to classify their services into programstreams before evaluating where the gains and losses might lie.The use of this technique for management and decision-making purposes must be supportedby consistent and reliable evidence on both the likely costs and outcomes of varioustreatments or interventions to enable priorities to be set with confidence. This model can bemore readily applied to a single sector of the healthcare system (such as the hospital sector)than the other allocative efficiency models currently being tested in Australia.♦Purchaser-Provider SplitThe “Purchaser-Provider” and “Managed Competition” models of health service fundingand delivery have been designed to promote both allocative and technical efficiency. In bothof these models there is a separation of the purchasers and the providers of healthcare, with,in the pure model, the purchaser being responsible for the total health care of a communityor constituency, receiving commensurate funds for the purchase of services on behalf oftheir community 348,405 . A small number of countries have moved towards such a healthfunding system, including Britain and New Zealand. The possible relevance to Australiahas been extensively debated in Australia 412,413 . Importantly, any “purchaser-provider”system requires advice to the purchaser, on how to select between competing healthinterventions. Logically, purchasers would need quality indicators to guide their purchasingchoice. Evidence available from existing programs suggests that initial decisions werealmost exclusively based upon cost and an assurance of quality. Such programs arecurrently seeking realistic indicators of quality of care which they could subsequently use tosupplant “cost-only” decision-making. Our enquiries found little evidence of sophisticationin their quality indicator utilisation to date. These models of health service funding anddelivery require, and are well placed, to make use of a rigorous approach to priority setting.A number of health economists in Australia have recommended the purchaser/providerframework as a means for achieving optimal resource allocation for health promotion, incombination with the adoption of a rigorous approach to priority setting (e.g. using modelssuch as the disease-based framework for priority setting by Segal and Richardson 348 ).♦Guidelines for Listing of Drugs on the Pharmaceutical Benefits Schedule (PBS)In Australia in 1993 the Commonwealth Government developed in its health planningframework the systematic use of economic analysis to inform pharmaceutical resourceallocation decisions, before the listing of drugs on the Pharmaceutical Benefits Schedule(PBS). Economic appraisal (cost-minimisation analysis, cost-effectiveness or cost-utilityanalysis), must be undertaken by pharmaceutical companies in support of the listings ofnew drugs on the PBS. Guidelines for the industry on preparation of submissions to thePBS have been developed, the most recent being updated in November 1995 414 . Theeconomic analyses are submitted to the Pharmaceutical Benefits Advisory Committee(PBAC) for evaluation prior to approval for listing on the PBS.Comparative cost-effectiveness (or cost-utility analysis) provides the framework withinwhich the inclusion or exclusion of new drugs to the PBS is made. However, this approach,like many of the models for priority setting does have limitations. In this example, practicalproblems associated with the definition of evidence and the limited choice of comparatorslimit its usefulness as a model for priority setting.The PBS Guidelines provide for the comparison of drug and non-drug approaches tomanagement, and thus potentially extend priority setting beyond pharmaceuticals. If thePBAC were to seek economic analysis of classes of drugs, and a comparison with non-drug

37approaches to management, this would extend substantially the scope of interventionscovered.♦Macro Economic Evaluation Model (MEEM)The MEEM approach is being developed by the Health Economics Unit of the Centre forHealth Program Evaluation at Monash University 415 . This approach is designed for diseaseimpact costing, and resource allocation in health promotion and illness prevention. TheMEEM is essentially the construction of a ranking index from a systematic analysis ofavailable databases allowing the evolution of a broad-based framework for priority setting.Its component parts (disease impact assessments, life expectancy analysis, and projectappraisal) can also be used for descriptive analysis, including issues of equity and needsassessment. The underlying rationale of MEEM is that judgements about priorities forillness prevention and health promotion should be guided by information on: the publichealth significance of health problems (measured by a range of indicators, including cost ofillness); the theoretical preventability (efficacy) and practical preventability (effectiveness)of the health problems; and the relative cost-effectiveness (efficiency) of individualpreventive measures aimed at achieving the potential for prevention.The MEEM model has limited applicability to the acute sector, as its primary focus is onillness prevention and health promotion. However, the disease costing component of themodel does provide a very comprehensive estimation of the annual hospital cost of treatingparticular diseases (based on ICD-9 classifications) in Australia. This may be of someinterest to the Commonwealth in helping to gain an understanding of the public hospitalexpenditure on particular disease categories.♦Disease-based Framework for Priority SettingThe Centre for Health Program Evaluation has developed a further approach to resourcepriority setting, designed to cover the entire health sector, based on the principles ofallocative efficiency, with disease groupings as the context for analysis 348,405 . The model iscurrently being implemented in relation to non-insulin dependent diabetes mellitus, withsupport from the Public Health Section of the Department of Health and CommunityServices, Victoria 402 .While the theoretical prerequisites for allocative efficiency can be defined, implementationof the above models is problematical. The measurement of marginal costs and benefits isdifficult and the magnitude of the research task for health sector-wide allocative efficiencyis daunting. The aim of the models of allocative efficiency is to link healthcare inputs(dollars) to healthcare outcomes (such as quality adjusted life years - QALYS) in such away that different services and treatments can be compared on a common basis - using forexample a measure of outcome, the QALY, as a generic outcome indicator. Priority settingthen allows decision makers to choose between healthcare services which produce thegreatest outcome (QALY) per unit of input (dollars). A recent review of outputmeasurement for resource allocation decisions in healthcare discusses the uses andlimitation of such an approach 416 .There are a number of practical problems with these approaches and their applicability tothe hospital sector. Firstly, most of the models require a large investment of resources formeasuring and analysing the costs and outcomes associated with the full range of allpossible healthcare interventions. Secondly, most of the models take a societal perspective(that is, they are concerned with the costs and outcomes of all members of the community -patients, funders etc) and do not confine their measurement to one sector of healthcare (e.g.

385.5.3 Safetya hospital). In addition, restricting the outcome measure to a single health delivery systemmay generate distortions and cost shifting.A further issue is that little attention has been paid to developing a single outcome measure(or measures) to capture the extent to which hospitals or healthcare systems achieveallocative efficiency.♦Incident Reporting SystemsThe reporting, categorisation and subsequent utilisation of information on incidents (that is,events which had the potential to produce injury or did in fact produce injury) in safetypromotion within hospitals has enormous potential for improving safety margins in acutehealthcare 42-47,50,73-75,114,129,134,135,150-153 . Anonymous incident reporting offers a relatively lowcost, sensitive index of safety performance with qualitative detail which can direct safetyimprovement endeavours 9,10 . The expansion of the successful anaesthesia/intensive careincident monitoring program across acute care services (in an initiative of The AustralianPatient Safety Foundation funded by the Commonwealth) should provide importantconfirmation of the utility of such incident monitoring approaches. The use of externallyreported adverse event or incident reports which rely upon the self-identification of error -which include identifying information - cannot be supported as a quality of care indicator,as such identified incident reporting systems would create perverse incentives for providersto either not detect or report incidents. The bulk of evidence suggests that the majority ofincidents are currently unreported in conventional hospital practice 42,47,50,73,74,111 . It wouldbe undesirable to create any indicator of performance which magnified this incidentawareness blindspot, by creating an incentive to under-report incidents to meet externallyapplied standards or thresholds of incident reporting.♦Adverse Event MonitoringVarious Adverse Event (that is, reports of injury as a consequence of a medical interventionor failure to intervene appropriately) monitoring systems have been proposed, based eitherupon administrative databases or medical record review. These events must be clearly dueto the intervention rather than primary illness or be preventable if good care was given andpatient preferences regarding risk taking must be known before tradeoffs between risks fordifferent outcomes may occur.Administrative database software systems use algorithms to flag adverse outcomes whichmay reflect quality of care problems 54,89,93,99,137,150,151 . Because existing common codingsystems (e.g. AN-DRG and ICD9-CM) do not reliably indicate when an event occurredduring an episode of care, such computer-based screens identify a mixture of events presentat the time of hospitalisation and events occurring during hospitalisation. They are thuscurrently relatively crude indicators of adverse events due to hospitalisation. Futureimprovements in coding for administrative databases (such as stipulation of pre-existent andhospital-related events by placing events in time sequence and improved codes forintervention complications) and improvements in software algorithms to detect and weight arange of potential intervention-related adverse events may see future algorithms foradministrative database safety screens fulfil their potential as patient safety indicators.Medical record review - seeking evidence of adverse events - may be undertaken at randomor on records identified following an initial “screen” for features suggestive of an adverseevent (such as the presence of an unplanned return to the operating room) 1,47,50,63,69,114,135,145 .Such preliminary screens may involve use of manual record review for criteria flagging

39records for expert review or use of computer-based screens of administrative database codesto detect records warranting focused review. Reviewed records are assessed using eitherexplicit criteria for quality of care or implicit (expert opinion) determine to review if anadverse event has occurred and if patient safety has been compromised unacceptably. Suchadverse event reviews have been associated over time with a decline in detected adverseevents, reinforcing the application of medical record-based adverse event screening forsafety improvement 185 .Record review for adverse events and quality of care analysis had been the major tool usedby the Peer Review Organisations (PROs) in the USA until recently. PROs are independentcontractors to US government purchasers whose charter is quality of care oversight. Formany years, implicit review of medical records (identified for review by specificoccurrences), formed the basis of PROs quality assessments. However, analysis of theinterrater reliability of implicit review in these circumstances suggests that there was poorconcordance between independent reviews. In addition, provider defensiveness againstexternal review of the processes of care has limited the improvement utility of PRO adverseevent review. Both these factors have led to a decision to broaden quality PRO initiativesbeyond these medical record reviews (e.g. HCFA’s HQII: see above) and, many StatePRO’s have discontinued such external record reviews because of poor cost-effectiveness127,294 .5.5.4 EffectivenessConceptually, any external quality and outcome indicator program based on adverse eventdetection in the medical record has a potential to modify the nature of that record. Routineexternal adverse event detection programs could create incentives for non-recording ofmatters relevant to adverse events. Thus, whilst internal promotions of adverse eventscrutiny based on medical record review as one component of facility-based patient safetyinitiatives are applauded - they cannot be advocated as useful in any national quality andoutcome monitoring scheme. Promoting patient safety will require a layering of strategies,including the development and local implementation of practice guidelines, facilityaccreditation, staff credentialling and hospital-based improvement programs to detect,interpret and respond to safety concerns. Safety is the dimension of quality of care wheremost responsibility must rest with providers. External review indicators risk masking safetyproblems, rather than promoting remedies. Accountability for facility-wide safety must beprovided by accreditation programs which incorporate a focus on safety issues rather thanexternal generic safety indicator programs.Targeted safety indicators can be constructed for specific conditions or interventions muchmore successfully than generic indicators of safety across facilities. Such indicators shouldbe included in modular indicator sets relevant to diagnoses/conditions/interventions ofinterest (see 7.1 below) and should be reported to reviewing bodies which include relevantprofessional colleges and specialist societies to help guarantee full and frank eventreporting. These indicators typically relate adherence to best practice processes of care(such as time to delivery of thrombolytic therapy in patients with acute myocardialinfarction) or report adverse outcomes (such as procedure specific complications or death).As the risks of intervention are often proportional to the severity of patient illness - and notinfrequently to the anticipated benefits of the intervention - such indicators requireadjustment for patient factors (such as severity of illness and treatment preferences) beforethey can reasonably be used to monitor safety performance.Indicators of effectiveness are essentially either of outcomes of care or outcome-proxies (whichare output measures or process measures known to be related to subsequent health outcomes).Outcomes can be defined either by providers or patients.

40♦Provider-Assessed OutcomesMortality: Acute care systems typically look at intrahospital mortality as one indicator ofhealth outcome 7,35,38,85,86,92,131,153,208 . When possible, linkage of data systems to enablereporting of mortality within longer timeframes (30 days/60 days) perhaps provides a moreaccurate indication of the impact of an episode of care on overall health and avoids biascreated by premature discharge or interfacility transfer 92,116,119,131 . The interpretation ofmortality requires knowledge of the anticipated outcome in the absence of the interventionand of the best achievable outcome with that intervention. For use as a quality of careindicator, it is important that variations in care quality translate into mortality variations. Inmany situations there are deficits in our knowledge of process-outcome links - andcurrently, mortality rates are not widely accepted as credible quality indicators 131,133,138,156 .They are, however, easy to measure and intuitively attractive, if the issues of adjusting forillness severity can be resolved (see 6.2 below). Global mortality indices provide riskadjustmenthurdles that may well be insuperable given the inadequacies of current modellingand the costs of collecting the necessary clinical detail for analysis 156 . Mortality withinstratified population groups (e.g. diagnosis or intervention) could be readily collected forsome conditions. There is the potential for relative mortality performance to be a crediblequality of care indicator if it proves that population strata contain a relatively uniform mixof case-severity. This would minimise the need for collection of additional risk-adjustmentdata 13,138,141 .Morbidity: There have been a number of models which aggregate intrahospital morbidity tocontribute to an index of outcome. These have been described under the Patient Safetydimension (see 5.5.3 above).Clinical Outcomes: Various clinician/provider-assessed indices of physical andpsychological function fall into a range of measures frequently labelled as “clinicaloutcomes” 31,779,1005 . These are often laboratory, medical imaging or physiological measureswhich are used as proxy measures for increased longevity or improvement in quality of lifeby clinicians.For example:• Range of joint movement at a defined time after total joint replacement.• Coronary artery patency after coronary angioplasty.• Reduction in tumour size in cancer therapy.• Fall in white blood cell count in treatment of leukaemia.• Exercise tolerance after CABG.The relationships between these measures and genuine outcomes (i.e. survival and quality oflife) are variable and sometimes tenuous 55,146,148,149 . These proxy measures are, however,easily quantified and provide useful short-term feedback to clinicians regarding the apparentsuccess of interventions. They are more or less specific forinterventions/diagnoses/conditions and should form part of balanced indicator setsdeveloped for key targeted clinical circumstances (see 7.1 below). At present, mostindicators are built around immediate and short-term measures which capture changes invariables related to health status within an episode of care. Mechanisms need to bedeveloped to routinely record clinical outcomes at relevant more remote time points if thetargeted condition warrants such longer-term indicators of clinical outcomes (for example,the 12 month infection rate for total joint replacement or 12 month coronary artery patencyrates following coronary artery angioplasty) 148,149 .

41♦Patient Assessed OutcomesPatient-based reports of outcomes of healthcare: These have a number of strengths andlimitations. Obtaining comparable information directly from patients (typically bystructured survey) enables the analysis of short-term and long-term outcomes of care fromthe patient’s perspective 207,291,306,310,320,321,1061,1071 . These outcomes may include reactions tothe intervention itself (pain, swelling, incapacity and so on) as well as longer term changesin symptoms and function. Detailed information can be obtained on the extent to whichspecific functional impairments or symptoms changed for the better or worse aftersurgery/interventions. Patient-reported data are relevant, but not all-inclusive 291 . Theycomplement clinical observations and tests while providing comprehensive and reliable dataon perceptual dimensions of health outcomes. Patient-reported outcomes are particularlyappropriate when the indication for an intervention is primarily improving function orrelieving symptoms. Patient-reported outcomes tend to integrate all processes of care andtheir consequences. They may suffer from incorrect attribution of outcomes tointerventions, but over a range of conditions they have proven largely reliable and valid (bycomparison with other parallel outcome assessments) 291,306,1061,1071 . Survey instrumentsexist for a variety of interventions (such as cataract surgery) and chronic conditions (suchas hypertension, arthritis, asthma and bronchitis, low back pain and diabetes) and more areunder development by the AHCPR-funded Patient Outcome Research Teams (PORTS)317,491,494for:• Hip fracture repair and osteoarthritis.• Prevention of low birth-weight infants.• Benign prostatic hypertrophy and localised prostate cancer.• Pneumonia.• Back pain.• Biliary tract disease.• Ischaemic heart disease.• Schizophrenia.• Secondary and tertiary stroke prevention.• Acute myocardial infarction.• Cataract.• Childbirth.• Diabetes.Health status measures: are standardised instruments to quantify functional health statusfrom patient reports 662,649,652,780,806,899,1071,1104,1105,1106,1110,1130 . They typically divide healthfunction into domains (such as physical, emotional, psychological, social and role functions)and are either generic (that is, applicable to all) or specific for a targeted population(condition/diagnosis/intervention/age group). The science underpinning validation of theseindicators of health status has advanced significantly over the past decade 1104-1106 . Manyhealth status instruments are now well characterised, with detail of their performance in arange of populations and particular health states known. They are responsive tointerventions aimed at improving healthcare and the significance of measured changes infunctional status is increasingly understood 1011,1048,1063,1110,1130,1134 . Interventions for acutedepression, heart valve replacement, treatment for chronic arthritis, total knee replacement,total hip replacement and cardiac revascularisation in ischaemic heart disease are amongstthe growing list of conditions in which the delivery of medical services have been shown toimpact positively on health status as measured by the Short-Form 36 (SF36) instrument572,576,665,669,798,809,863,931,951,968,11041106 . This health status measure has the most data available onits use in acute healthcare, both within Australia and internationally. The Short Form 36(SF36) is a generic measure of health status that can be used to assess health changefollowing an acute episode of care 1106 . Recent demonstrations that health status prior to an

425.5.5 Acceptabilityepisode of care is reliably reported retrospectively by patients would enable single surveyspost episode of care to detect the impact of care 192 . The availability of shorter surveyinstruments such as the SF36 related Short Form 12 (SF12), will facilitate the incorporationof health status quantification into broader, omnibus surveys of patients’ experiences ofhealth and healthcare 490 . The Medical Outcomes Trust in Boston has established aScientific Advisory Committee to review and endorse health status instruments. It is likelythat instruments endorsed by this body will increasingly become the health status instrumentstandards within healthcare quality and outcomes indicator programs.Health-related Quality of Life (HRQOL): These are instruments which encompass aspectsof functional health status and patients’ reports of the impact of their health on theirenjoyment of life 659,663,670,671,1027,1085 . Rather than representing a discrete measurementmethodology distinct from health status, the two approaches offer a continuum, withfunctional health status more focused on self-reports of what one can or cannot do andHRQOL more reflecting assessments of how health impacts the enjoyment of life. A largeliterature base on HRQOL instruments exists206 . There are both generic anddisease/condition specific HRQOL instruments. There is consensus that they are of value inhealth services research, although it would be our view that current HRQOL instrumentsare insufficiently characterised to recommend their widespread application in large scaleacute healthcare outcome indicator programs. Further developments in HRQOLinstruments may well allow such applications in the near future.This report was not to address many of the complex issues surrounding acceptability of caregiven that a major report on the role of patient satisfaction surveys in a national approach tohospital quality management has recently been completed for the Commonwealth which canvasesthe issues around the reporting of consumers’ experiences and assessments of quality of care 188 .Patient satisfaction is known to be associated with better results from healthcare interventions(such as better compliance with preventative care/medications) and improved functional healthstatus 198 . Although there is still much to learn about how best to obtain feedback from patientsabout their needs and wants and how best to use this knowledge to drive improvement processes,the current state of knowledge allows application of proven instruments whilst furtherdevelopmental programs proceed. Provider generated instruments such as those of the HospitalCorporation of America (HCA) or the Royal College of Surgeons (RCS), and patient-focusedinstruments such as the Picker-Commonwealth survey, could easily be adapted to provide initialreference data on the level of acceptability of a range of processes and outcomes of acutehealthcare. The Picker-Commonwealth instrument has been applied in the USA, Canada, theUK, Australia and Europe - and thus offers an additional benefit of potential internationalcomparisons of healthcare acceptability.5.5.6 Continuity♦Indicators of ProcessAt a facility level, it is common to seek evidence that appropriate discharge planning andcare integration steps have been instituted as part of quality monitoring. External reportingof compliance with such process measures would be unreliable as a quality of care indicatoras such self-reports of compliance with discharge planning processes would be open togaming and not amenable to independent audit for confirmation of data integrity.

43♦Indicators of OutcomeThe success of endeavours to ensure a care continuum may be judged by surveying thosemost intimately involved in care processes. Patients and their families are the principalexperts on care continuity, with primary care physicians and other community-basedhealthcare providers secondary resources 261,337 . Surveys of these participant groups willreadily inform how well integrated an acute episode of care appeared to those involved andprovide descriptive commentary on the success of processes in place to optimise continuityof care. Modules on discharge planning and communication regarding ongoing care exist inseveral existing survey instruments 188 . Continuity indicators are prime examples of theusefulness of patient reports on care processes and outcomes in quality of care indicatorsets.5.5.7 Technical Proficiency♦Observed versus Predicted Outcomes AnalysisMany indicators of technical proficiency are based upon some permutation of observed topredicted outcome ratios. Predicted outcomes are based upon historical norms, bestpractice targets, arbitrary standards or models of anticipated median outcomes built onrelevant patient level data on existing clinical practice profiles. Observed outcomes arecompared to predicted, and where significant variation is identified it is inferred that thesevariations are related to variations in technical proficiency 6,7,13,16,83,85,93,131,132,133,156 .For example:• Myocardial infarction rates following coronary angioplasty• Stroke rates follow carotid endarterectomy• Anastomotic dehiscence rates following bowel anastomosesAll comparisons of observed-to-predicted outcomes are critically dependent on knowledgeof a link between technical proficiency and outcome and the validity of the estimatedfrequency of the predicted outcome of interest. In both these domains the current science ofquality monitoring has significant limitations (see especially 6.2 below). Unequivocalinferences about quality/technical proficiency can rarely be made on the basis ofobserved/predicted outcomes data, although they provide a practical stepping-stone toimproved proficiency measurement 140,143 .♦Compliance with Guidelines for CareThere is a growing body of guidelines for care delivery in particular conditions. Potentiallythese stipulate what should be done, how it should be done, when it should be done andindicate the likely outcomes if good care (as defined by the guideline) is delivered.Indicators of technical proficiency targeted to particular conditions/interventions can bedeveloped based around these guidelines. There are now comprehensive methodologiesavailable to help translate clinical practice guidelines into instruments to evaluate quality ofcare 189 . The HCFA Cooperative Cardiovascular Project, within its HQII program,provides an example of how existing care guidelines can be converted into quality processand outcome indicators (see Appendix 6 for additional detail) 127 . As many guidelines arebased upon evidence - or at worse expert consensus - they form an important foundation forquality indicator development, hastening indicator development and reducing the risk ofgenerating indicators of limited relevance.

445.5.8 Appropriateness♦Case-by-Case Comparison with Expert CliniciansTraditional approaches to appropriateness evaluation see comparison of each individualepisode of care to expert opinion on appropriate care - typically by reference to consensuscriteria on what constitutes appropriate care (e.g. the clinical circumstances in whichinterventions are warranted given knowledge of the relative risks and benefits) 20,21,25,39-41,52,67 . This clinician perspective is subject to criticism if doubt exists about the evidencesupporting applied criteria for appropriateness - although within given providercommunities there is a reasonable concordance of opinion on the appropriateness of targetedinterventions21,39-41,67 . These case-by-case reviews are sufficiently imperfect andimpractical for use in any national indicator program as to be discounted for this purpose 53 .♦Inferences Based on Variation in UtilisationA more practical approach to large scale monitoring of appropriateness of care is theanalysis of comparative utilisation statistics 22-24,33,37,92,110,112 . Relative rates of application ofa particular intervention provide information suggesting potential overuse (if rates areinexplicably high) or problems with access to care (if rates are inexplicably low). The useof utilisation rates as proxies for appropriateness is dependent upon the assumption thatpotential access is relatively uniform - as would be the case for many medical interventionswithin urban Australia. Data stratification (such as by age, sex or ethnicity) allowsidentification of areas of potential inappropriate intervention or access difficulty inparticular vulnerable subgroups. Utilisation best reflects appropriateness when theprevalence of the relevant condition in the serviced population is known and can form thedenominator in the utilisation rate equation (that is, a rate of intervention/1000 in populationat risk of needing intervention) 116,1003 . Although imperfect appropriateness indicators, suchutilisation rates are available at low cost and would provide interesting comparativeinformation for Australian populations.N.B.It should be noted that “appropriateness” is used in several ways in health servicesresearch. Its use as a dimension of quality in this report is restricted to the assessment ofnet benefit versus risk (that is, was an intervention worth doing). It does not encompassthe site of the intervention (which we would see primarily as an efficiency issue), nor theappropriateness from the patient’s perspective (which we have encompassed within thedimensions of acceptability and effectiveness).5.5.9 Other DimensionsThe original Project brief asked for specific comment on health status, discharge planning andnursing indicators. Health status measures are tools used to monitor effectiveness - and arediscussed within this context (see 5.5.4 above). Discharge planning constitutes one aspect ofcoordination of the continuum of care and is addressed within this context (see 5.5.6 above).After reviewing an extensive body of information on indicators of nursing 171-178,209-261,285-290,323 ,we elected to treat all dimensions of care as indicators of integrated care delivery by all relevantclinical professionals. We thus include nursing contributions to care and those of other involvedhealthcare professionals within indicators for all identified care dimensions (see Appendix 7 foradditional detail).

456. Discussion6.1 Rationale for a Nationally Consistent Quality and Outcome Indicator ProgramNational indicator programs would seek to provide:• Accountability by providers.• Information to guide consumer decisions.• Incentives for quality improvement.They would achieve these aims without imposing an unacceptable collection burden on providers, atreasonable cost to the community and without creating incentives for undesirable behaviour by providers(such as gaming of data or inappropriate skewing of service delivery profiles). An ability to comparequality and outcomes data, confident that the same indicator instrument was being applied in a consistentfashion across facilities, should stimulate national benchmarking activities. As indicators are expensive todevelop, a move to a nationally consistent set could conceivably reduce the net cost of indicatordevelopment - by allowing the routine application of well-characterised instruments broadly acrossAustralia. Inevitably, national indicators will lead to a prioritisation of quality improvement efforts - ashas occurred elsewhere following widespread application of specific quality of care indicators. Thisinfluence on provider behaviours could be used to direct their attention to national health goals and targets.National indicator sets should encompass aspects of quality of healthcare services of relevance to patients,providers and purchasers of care. Such a balanced, comprehensive approach to monitoring dimensions ofhealthcare quality is necessary to avoid unintended and undesired consequences which might arise fromskewing of a system’s performance to produced favourable results in narrow segments of monitoredservice delivery. We believe indicators spanning the dimensions of care identified as pivotal by this report(i.e. access, efficiency, safety, effectiveness, acceptability, continuity, technical proficiency andappropriateness) would provide a firm basis for a successful, coordinated national indicator program witha balance between comprehensive coverage and feasibility.The cost of indicator collection would be substantial - but would constitute a small proportion of overallhealth expenditure. Judgements on the value of indicator monitoring are varied - but input frompurchasers, providers and consumers must be integrated before final value decisions are reached. Anyintroduction of national quality of care indicators must be accompanied by parallel projects to demonstratethe performance of the indicators themselves against reasonable expectations regarding indicatorattributes. This current review has highlighted that many indicators have been developed and applied inthe workplace with little or no evidence of attributes such as reliability or validity and no systematicprocesses to monitor responsiveness, burden, interpretability or utility. Throughout the western worldthere is a progressive move to large-scale monitoring of healthcare - in part because of a need to monitorthe healthcare industry because of its importance economically and in part a desire to inform consumersand drive improvement efforts. Within Australia, many States are moving to collect quality and outcomeindicator data. There would be little sense in collecting disparate data at a State and Territory level, ifagreement on indicator sets would allow national comparison at a minimal marginal cost. Whatever formfinal national indicator programs adopt, systematic tracking of the attributes of individual indicators andindicator sets will better characterise quality indicators and support decision-making regarding qualityindicator applications. Future reviews similar to this would then find evidence available upon which tobase firm judgements on the value of indicator programs and analyse attributes of their component parts.One of the enduring debates around quality and outcome indicator collection concerns the compatibility ofindicators for internal quality improvement, external accountability functions and informing consumerchoice. There is no consensus on the ability of a single indicator to serve these functions. Some argue thatthese functions are inherently incompatible, with accountability programs inherently “unsafe”environments for healthcare providers, in contrast to the “safe” environment required for successful qualityimprovement programs. Whilst agreeing with the need for both activities, these experts opine that externalindicator programs will never enhance local quality enhancements - and at worse may detract from them.

46Others argue that it is the behaviour of those participating in indicator programs which determines theircongruence for accountability and quality improvement purposes. It will be essential that education (ofproviders and those receiving indicator data) and a collaborative development of credible indicatorsunderpins any national indicator implementation in Australia. Although there are stark examples offailures in indicator applications (such as HCFA - and others - release of inadequately risk-adjustedfacility mortality data), there are equally examples of successful dual-usage indicator programs (such asthe Cleveland Health Quality Choice), suggesting that the optimism of those supporting compatibility ofthese uses of some selected quality of care indicators is justified.The introduction of a nationally consistent quality and outcome indicator program would undoubtedly beassociated with changes in practice, as have occurred elsewhere 12,304,339 . The rationale for these changes inpractice and their relationship to indicator usage per se have never been systematically analysed. Whilst itwould be hoped that the availability of indicator data and the ability to benchmark performance motivatedquality improvement effects which resulted in improved processes and outcomes (such as a reduction inCaesarean section rates or readmission rates) these changes may have resulted from a halo effect of qualitymonitoring or indeed have occurred because of unrelated changes in health service delivery. In the leastdesirable circumstance, quality indicators may drive data manipulation exercises such that apparentimprovements merely represent the quality monitoring equivalent of “creative accounting”. As isemphasised throughout this report, the likely health service delivery consequences of a national qualityindicator program will be inextricably linked to the context of indicator development and use.6.2 Risk AdjustmentAt present we believe that none of the available severity adjustment systems reviewed are clearly superiorto others. There are a number of recent enlightening reviews of this subject 34,35,71,131,133,141,144,154,156,778 .They agree to the value of adjusting indicator data for patient characteristics so that the process oroutcome indicators better reflect the quality of delivered care and not factors unrelated to care quality (socalled“confounding factors”). Ideal risk adjustment systems incorporate a range of modifiers related tothe patient’s suitability for application of care processes (such as allergy to recommend treatment options)and their intrinsic vulnerability for a given outcome (such as the specific condition afflicting the patient,the severity of that disease, the presence of other medical conditions and patient’s overall physical,psychological, emotional and social resources). Following ideal risk adjustment, any remaining differencesin indicator rates would directly reflect differences only in the quality of care provided.Ideal risk adjustment methodologies do not currently exist. There is evidence that the more informationavailable for risk adjustment, the better the resulting model - although the gain in predictive abilities ofrisk-adjustment models (their ultimate validation) is not directly proportional to the quantity of data pointsprovided for modelling 293,329 . The principal question posed is therefore whether the gain in comparativeprecision warrants the costs of data collection and risk adjustment. Providers typically are not responsiveto unadjusted indicator data 141 . Credibility with providers is an important motivation for risk adjustment.It is crucial that the limitations of adjusted or unadjusted indicator data are known to all who use the dataand that they act on the data accordingly. Serious problems arise if crude or incompletely risk-adjustedindicator data are acted upon as if they mean precisely what they may seem to mean (for example, ifoverall hospital mortality rates are reported as quality indicators and specialist cancer hospitals providingpalliative care services with high mortality rates are sanctioned).More sophisticated risk-adjustment models not only use more factors for adjustment, they also attempt tocompensate for known problems with data (for example, they seek information from medical recordsrather than administrative databases to ensure accurate patient profiling) 156 . Better models avoid maskingdifferences in process or outcome that could potentially reflect differences in quality of care (for example,if ethnic minorities systematically received poorer quality of care - adjusting for ethnicity would concealthat aspect of relative performance) 320,324 .

47An alternative to patient-level risk adjustment is facility-level adjustment 293 . We believe this approachrisks the masking of systematic differences in care delivery between facilities and is not desirable. Nomajor respected risk-adjustment system studied included such facility-level adjustment in its modelling.Stratification of patient populations provides a relatively simple means of attempting to correct for patientleveldifferences impacting on process or outcome indicator data. The premise underpinning stratificationis that the identified groups (strata) are either homogeneous or that they may be treated as such forcomparative purposes because any patient-level differences average out across the groups compared.These strata/groups may reflect primary diagnosis, disease severity, age or sex, or some combination ofsuch factors. When used successfully stratification allows indicator comparisons at low cost - providedthe data for stratification are routinely available in administrative databases.Stratification may be the only practical approach to addressing the monitoring of quality of care and healthoutcomes in vulnerable patient groups, such as Aboriginal and Torres Strait Islander peoples. Providedinformation permitting stratification is readily available within data sets (such as indicators ofAboriginality or postcodes as a proxy for socio-economic class) it would be possible to reviewcomparative indicator data in those deemed potentially vulnerable. Indicator rate comparisons wouldsuggest areas of inequity in service delivery warranting further scrutiny.Quality indicators have been subjected to various levels of risk adjustment. Any single indicator reviewedfor this report has been presented in formats ranging from unadjusted facility or practitioner-level data todata reflecting risk-adjustment for 30 or more potential confounding variables. Expert opinion differs onthe utility of risk-adjustment and in particular its cost-effectiveness. There is no evidence to categoricallydecide the value of risk adjusting clinical indicators - although there is consensus that risk adjustmentimproves provider acceptance of indicator data and significantly alters apparent comparative performanceof up to 30% of individual facilities. Each indicator requires the development of a unique risk adjustmentmodel based upon local experience of indicator and confounder data. Risk adjustment models do notsuccessfully “transport” from one healthcare environment to another - presumably because many apparentconfounders are in fact proxy measures of patient characteristics which act as valid proxies only in adefined patient population. The need to develop risk adjustment models for those indicators in the nationalindicator set will depend on the intended use of indicator data. It is probable that raw and adjusted orstratified indicator data will be required for most of the proposed indicators. The development of locallyrelevant risk adjustment models should form an integral part of the construction of detailed operationaldefinitions for each indicator to be trialled as part of a putative national indicator set. Guidance on datacollection requirements necessary to adjust for confounding influences will initially be based around expertlocal opinion and the experience of others in risk-adjusting quality indicators in healthcare103,106,118,144,156,302,328,342 .6.3 Characteristics of a Successful National Quality and Outcome Indicator ProgramWe believe a successful national indicator program will require several key characteristics. It will be:• Comprehensive: A sufficient breadth of indicators will be used to guarantee that a range ofdimensions of care for common, relevant conditions are encompassed within the program.• Collaborative: The program will evolve with the active participation of all interested parties pursuinga common goal of attaining a credible indicator set which can satisfy accountability requirements,guide consumer-choice and promote quality improvement.• Consumer-focused: Indicator programs must embrace consumer involvement in the identification ofareas of care delivery warranting monitoring and in the design, implementation, interpretation andfeedback on indicators of quality of care and outcomes.• Current: National indicator sets will need to be updated regularly to keep abreast of developments inthe science of quality of care monitoring and the science of medical care delivery.

48• Cost-efficient: Wherever practical, indicators should utilise existing data for indicator construction,making incremental changes to routine databases as the needs for additional data points are identifiedby appropriate research and field trials.Such a national indicator program would then be credible to all concerned and be of value in guidingpurchasing decisions, integral to facility quality improvement programs and inform consumers’ healthcaredecisions. Achievement of these desired goals within the framework of a national indicator program willrequire more than agreement to a nationally consistent indicator set. Assistance with data collection, theinterpretation of statistical analyses, benchmarking, the design and implementation of improvementprojects and coordination of collaborative projects with other facilities will require the availability ofresources beyond those currently seen in most Australian healthcare facilities. To ensure that informationsharing on acute healthcare performance enhances local quality improvement efforts, provideraccountability and consumer choice, it will be necessary to expand the community of professionalscommitted to quality and working collaboratively with hospitals to translate the potential benefits ofquality and outcome monitoring into achieved gains.The provision of this human resource and logistical support is an absolute requirement for the success ofthe national indicator program. A number of options exist for how this resource might be provided.Existing government, professional or regulatory agencies could be delegated responsibility for some or allof these functions. Alternatively, such functions could be concentrated within a group focusing on qualityissues in healthcare (such as those centres of excellence proposed in the final report of the Taskforce onQuality in Australian Healthcare). The use of some existing agencies to provide the program support mayraise concerns about potential for apparent conflicts of interest. Stand-alone, quality-focused agenciesmight be deemed an unwanted additional complexity and expense. A USA model worthy of review in thissetting is the HCFA Quality Improvement Initiative. HCFA contracts out these support functions for thisquality monitoring project, which are provided by dedicated groups (typically set up by existing agenciesas independent entities with the purpose of enhancing improvement efforts).7. RECOMMENDATIONS7.1 Structure of a National Indicator ProgramThe development of a nationally consistent set of quality of care and health outcome indicators will be aniterative process, with more in common with a research and development program than with the workplaceimplementation of an established technology. If it is agreed that a variety of interests are best served bythe availability nationally of credible comparative information on the performance of acute healthcareservices, there is the need to begin with the best available feasible indicators and progressively improveand supplement these, rather than deferring all action because of the absence of perfect indicators.We recommend:7.1.1 National quality and outcome indicators be developed as two complementary indicator sets:A core indicator set, intended for continuing collection and a series of indicator modulescontaining indicator sets intended for collection at a defined frequency for a finite duration.7.1.2 The core indicator set should focus on generic aspects of care - including access, efficiencyand acceptability of care.7.1.3 The indicator modules be targeted to specific conditions, diseases, diagnoses or interventionsand include a balanced array of indicators encompassing clinical indicators, health statusindicators, acceptability indicators and efficiency/cost indicators.

49Examples of the type of indicators contained within such condition-specific modules areprovided at the end of this section. It is emphasised that these are listed to explain theconcept of indicator modules and that formal development of indicator modules for trialingrequires expert local input (see 7.1.5).7.1.4 The core indicator set be based around information derived from administrative databases(because of cost efficiency) or patient surveys (because of the high value of patient-basedinformation).7.1.5 The indicator modules targeting a particular clinical circumstance be based on hybrid datacollection and include in their function a formal assessment of the contribution of additionaldata points obtained from medical record data abstraction to the utility of indicator data.7.1.6 Indicator modules targeting particular clinical circumstances be developed as cooperativeactivities between government, regulatory bodies, providers, specialist and professionalcolleges and societies, consumers and those with expertise in indicator application. Theseprograms could be based at centres with expertise in data handling (such as AIHW) butshould include active participation of professional groups who will enhance the peercredibility of quality monitoring and help ensure that accurate data are transmitted foranalysis.7.1.7 Wherever possible sample techniques be used to obtain detailed representative data ratherthan attempting routine data collection from all episodes of care. This will increase thecomplexity of available data whilst controlling costs.7.1.8 As part of a national indicator development program a resource be developed to assistfacilities with indicator data collection and interpretation and provide independentconfirmation of data integrity.7.1.9 Indicator programs should develop comprehensive operational definitions for indicators toenhance data reliability, preferably including software programs with inbuilt data reliabilitychecks.Examples of Condition-SpecificQuality and Outcome Indicator ModulesCoronary Artery Bypass Grafting4-8 indicators chosen from:• Unplanned return to operating room• Inhospital mortality• 5 year mortality• Change in health status (at 6 or 12 months)• Secondary prevention strategies• Health related quality of life at 3 months• Understanding by patient of indication for CABG• Average length of stay• Satisfaction with inpatient episode of care• Other process or outcome indicators

50Myocardial Infarction4-8 indicators chosen from:• Compliance with Best Practice Guideline for Care- Time to revascularisation attempt (either thrombolysis or angioplasty)- Inhospital survival- Health status at 3 months- Administration of appropriate medicationsü Beta Blockersü Aspirinü ACE Inhibitors• Myocardial function at 12 months• Survival at 5 years• Risk factor education success• Risk factor intervention rates• Return to previous activity index• Satisfaction with inpatient episode of care• Length of stay• Cost of care• Other process or outcome indicators7.2 Suggested Indicators for Trial in AustraliaIt is a difficult judgement to support particular quality and outcome indicators without detailed knowledgeof the intended use of these indicators. As we have repeatedly stated, the determination of the adequacy ofan indicator is not categorical (that is, indicators are not either adequate or inadequate, good or bad),rather, indicator validity and utility lie upon a continuum with final adjudication on their relative adequacyrequiring knowledge of the context for performance measurement. Our recommendations assume acollaborative, iterative, developmental program for progressing towards a nationally consistent set ofquality of care and health outcome indicators.We have found little information available on indicator attributes which would guide dogmatic choices ofindicators for our recommendations. The majority of putative quality and outcome indicators have no dataon their performance. The available data would suggest that many indicators are of some value - with fewstanding out as examples demanding inclusion in quality monitoring programs. The recommendationsbelow are based upon our judgements regarding indicators of potential value, cognisant of existing andfeasible short-term data availability. Our assessments of these indicators against the indicator attributeprofile is included in Appendix 6 - which also encompasses notation of identified indicators which havesufficient information available to suggest that they also could be adapted for use in an Australian contextand examples of indicators with significant shortcomings when judged against these criteria.♦Access7.2.1 Existing elective surgery waiting times be refined to a common operational definition whichencompasses a uniform categorisation of urgency. Waiting time should be the time from thedecision to intervene to intervention, to reduce incentives for provider manipulation of data.Derivative measures, such as clearance times, help with interpretation of waiting/queuingtimes and should be continued.7.2.2 Following successful implementation of an indicator of uniform waiting time for electivesurgery, consideration be given to stratification of queuing time by specialty in addition to

51urgency for core use in indicator sets or by intervention/procedure within modular setstargeting particular clinical circumstances.7.2.3 Emergency Department waiting times - stratified by triage category.7.2.4 Waiting times in Emergency Department prior to inpatient admission.7.2.5 Patient surveys be undertaken to derive reported elective surgery waiting times, EmergencyDepartment waiting times, time in Emergency Department awaiting admission, outpatientwaiting times (for appointment) and outpatient waiting time for particular attendances.This information could be stratified by patient-reported urgency of care and the specialtyproviding service and the acceptability of reported queuing times recorded.♦Efficiency7.2.6 The continued development of cost/casemix adjusted separation be supported to resolveoperational definition and reliability issues before proceeding to consider more complex,high level technical efficiency indicators.7.2.7 No indicators of allocative efficiency are currently suitable for trialing.♦Safety7.2.8 No indicators of patient safety are currently suitable for trialing in the core indicator set.Progression of implementation of comprehensive, anonymous incident reporting systemsare seen as the optimal national strategy for monitoring safety in hospitals. All reviewedgeneric safety indicators were considered either conceptually flawed or liable to createundesirable incentives to bias the identification and reporting of safety concerns.7.2.9 Targeted safety indicators should be included in relevant modules of indicators focusedupon defined clinical circumstance. These will typically reflect adherence to best practiceguidelines or some variation on observed to predicted adverse outcome monitoring andcould best be initially developed for interventions for ischaemic heart disease(angiography, CABG, angioplasty and AMI).♦Effectiveness7.2.10 Generic health status measures (the SF36 or SF12) be part of omnibus surveys (see 5.4.3)administered to recent recipients of care and these health status reports be compared toself-reported health on admission as indicators of care effectiveness for use in the coreindicator set.7.2.11 Health status measures (SF36 or SF12 as generic measures or other validated conditionspecific measures relevant to the targeted clinical circumstances) be included in thebalanced indicator sets developed for indicator modules addressing particular conditions,diagnoses, diseases or interventions.7.2.12 Mortality rates for selected clinical conditions, diseases, procedures or interventions becollected - stratified by readily available administrative data - and apparent quality of careperformance be compared to more sophisticated health outcome indicators based uponrisk-adjusted mortality developed within targeted modules of indicators addressing theseprocedures. This would provide information as to how significantly complex riskadjustmentalters apparent comparative performance and will guide judgements on the

52long term utility, in an Australian acute healthcare context, of these approaches tooutcomes monitoring.7.2.13 Unplanned readmission following index admission for asthma - stratified by age (0-19 and20-49) provides a valid indicator of the effectiveness of the overall asthma care plan andits revision or reinforcement during the initial hospital admission. Unplanned readmissionis, we believe, likely to be restricted in use in future to such condition specific applicationsrather than continuing as a generic hospital-wide medical indicator - although finaldecisions on its generic utility must await the results of studies underway under theauspices of the National Hospital Outcomes Program.7.2.14 Low (less than 2500 grams) and very low (less than 1500 grams) birthweight rates bemonitored. Although difficulties with attribution of outcome to facility-level quality ofcare are encountered with this indicator, at a population level it provides oversight of theadequacy of maternity care and examination of stratified results may yield evidence ofcomposite healthcare system quality problems (including aspects of access andeffectiveness). Hospitals providing childbirth care have a crucial role to play in thepromotion or direct provision of adequate maternal antenatal care and in advocacy forsocial support for pregnant women. Thus, whilst this indicator reflects much more thanfacility-level performance during an inpatient episode of care, it is not an unreasonableindex of the broader success of healthcare providers in the management of antenatal care.♦Continuity7.2.15 Patient-based assessment: Surveys of patients’ perceptions of care should includeassessments of the success of discharge planning and integration of care based upon themodules currently within the Picker Commonwealth survey. In circumstances wherepatients themselves are unable to provide such feedback, the information should beobtained from their carers.♦Acceptability7.2.16 Acceptability: We believe that a national survey program based upon random sampling ofrecent acute care patients using instruments built upon well validated surveys - such as thePicker Commonwealth survey, the Hospitals Corporation of America survey and theRoyal College of Surgeons surveys - is the appropriate action in the short term.7.2.17 Needs: A national program to better identify the needs of Australian consumers, inparticular vulnerable subgroups such as Aboriginals and Torres Strait Islanders, shouldbe implemented. This information should be the basis for refinements of the nationalsatisfaction survey instrument.7.2.18 Contemporaneous reporting of process and outcome information: Whenever practicable,surveys of acceptability should be linked to simultaneous patient reporting of theirperceptions of processes and outcomes of care.♦Technical Proficiency7.2.19 Targeted technical proficiency indicators: The major thrust for indicators of technicalproficiency should be within the modular indicator sets focusing upon specific clinicalconditions, diagnoses, diseases or interventions. These should be appropriately riskadjustedprocess indicators built upon accepted care guidelines or outcome indicators builtupon observed to predicted morbidity or mortality ratios (where the frequency of themonitored outcome is sufficiently high to avoid masking of quality of care issues by

53anticipated random variation). Initial modules should be built around cardiovasculardisease interventions. Later modules could examine conditions such as the care ofdepression, the management of newly diagnosed cancer or cataract extraction - asexamples of common conditions with a serious health impact, which have effectivetreatments and performance indicators available.♦Appropriateness7.2.20 Case by case analysis of appropriateness of care: The limitations of this analytictechnique, its cost and its influence on a collaborative development of external indicatorprograms render it unsuitable for a national indicator program. Such individualappropriateness reviews should be encouraged within provider facilities.7.2.21 Utilisation as a proxy appropriateness indicator: It would be valuable to analysepopulation- based differences in interventions believed to have significant unexplainedvariation in utilisation as indirect indicators of appropriateness of care. Where reliabledata on the population-based incidence of primary disease processes is available, suchutilisation data would prove more valuable in judgements of apparent comparativeappropriateness and might direct more focused review of a representative sample ofindividual cases for analysis of individual case appropriateness. Conditions to considerfor such utilisation review include:• Cardiac catheterisation.• Coronary artery bypass grafting.• Angioplasty (PTCA).• Cholecystectomy.• Hysterectomy.• Laminectomy.• Caesarean Section (primary).• Vaginal birth after primary Caesarean Section.• Prostatectomy.7.3 Future Directions in Indicator Development7.3.1 Dimensions of care quality needing indicator developmentWhilst an argument can be made that all areas of quality and outcome indicators requiresubstantial research and development, we believe there to be a particular need for indicators thatreflect:♦♦Unmet access: Most existing hospital access indicators deal with characteristics of thosewho have succeeded in achieving access to our acute healthcare services. Betterinformation is required on consumer health needs. This requires information about thosewho fail to access healthcare to better inform judgements on equity of healthcare services.Such indicators will require population-based surveys seeking evidence of likely need andthe health system’s response to that need.Allocative efficiency: Much needs to be done to refine instruments for allocativeefficiency that more accurately reflect the balance between consumers’ needs and evidencethat services can meet those needs and offered care influence health outcomes - rather thanmodels mirroring directions of health policy. This will require improved information onconsumer needs and an explicit linkage of this information to evidence of existing capacityto address these needs (including evidence of effectiveness such as that provided by theCochrane Centre Collaborations).

54♦♦♦Acceptability: There is a particular need for better instruments to monitor the culturalappropriateness of care and the perceptions of vulnerable subgroups receiving care (suchas Aboriginals and Torres Strait Islanders).Continuity: Feasibility studies addressing the methodologies for obtaining relevantfeedback from representative, community-based healthcare professionals on the success ofintegration of acute healthcare services into the care continuum should be supported.Appropriateness: Research should be pursued into methods for linking patient-basedperceptions of the outcomes of care and preferences regarding the weighting of indicationsfor care delivery (based upon patient perception of need for care) to provider assessmentsof the risks and benefits of care delivery. Initial studies should focus on commonconditions with evidence that patient election regarding care is a significant issue (e.g.treatment for benign prostatic hypertrophy and primary breast cancer).There are important dimensions of quality of care which are inadequately represented in ourproposed framework for performance measurement. Equity is only partly encompassed by accessmeasures, benevolence is incompletely represented in indicators of effectiveness and acceptabilityand the ethics of care delivery essentially unrepresented. These, and other, care dimensions areimportant. If it were feasible to develop quality indicators that better reflect these more complexand value-laden aspects of healthcare it would clearly be desirable. It is our opinion that thelikelihood of developing credible, quantitative indices of these more complex dimensions of care islow and that for the foreseeable future most research and development efforts in quality indicatordevelopment in acute healthcare would usefully be focused within the framework offered by ourdimensions of care quality.7.3.2 Indicator Development Strategies:♦We strongly recommend that future national indicator development be addressed bymultidisciplinary groups and be directed towards specific clinical circumstances with aview to the evolution of balanced sets of indicators (i.e. reflecting clinical indicators,patient functional health outcomes and HRQOL, the acceptability of care and cost of care)rather than exhaustive sets of indicators addressing a relatively narrow range ofdimensions of the quality of care delivery processes or achieved health outcomes.Australian quality and outcome indicator development should be more closely linked tointernational developments in health services research. Wherever possible, we should seekto build upon the knowledge established by others in much larger population bases andadapt putative indicators for local utilisation trials rather than building all quality of careindicators “from the ground up”. Linking our acute healthcare indicators more closely tointernational indicators would allow for more international comparisons - which webelieve will be increasingly relevant in health services as in benchmarking performance inother major national industries. The trialing of such derivative indicators would alsoallow our intellectual and fiscal resources to concentrate upon indicators reflectingnuances of healthcare delivery that are uniquely Australian and may speed thedevelopment of indicators of relevance to health issues or consumer groups currentlypoorly served by existing quality of care and health outcome indicators.♦We strongly recommend that work be supported to develop an indicator module whichspecifically addresses the health concerns of vulnerable patient groups, in particularAboriginal and Torres Strait Islander peoples. Together with stratification of indicatordata, this approach would strengthen knowledge of current gaps in service delivery andassist in monitoring of the impact of strategies to enhance care of vulnerable subgroups.

55♦Australia should seek to identify centres of excellence to link into the WHO QualityAssurance Collaboration Centre, to promote interfaces between clinical practice andacademic developments in quality of care indicators.7.3.3 Integrated Health System Performance Assessment:♦♦National quality and outcome indicators should increasingly focus on integrated healthservice delivery rather than the performance of individual sectors such as acute care.Most of the major health concerns of Australians relate to chronic illnesses, where theperformance of the integrated system of care is far more important than the performanceof its isolated components. Patients turn to the healthcare services seeking adequate careand hoping for good overall outcomes. With chronic illnesses (such as hypertension,ischaemic heart disease, diabetes, arthritis and cancer) the overall success of interventionsmust sum the inputs of health promotion, preventative care, early detection programs andepisodes of care in community and hospital sectors. Analyses of performance must judgewhether appropriate care has been delivered (e.g. education about risk reduction,screening for early disease, risk factor identification and therapeutic intervention) notwhere that care was delivered. Many of the best indicators of quality of care are suchglobal measures of performance (such as have patients been advised about relevantlifestyle changes, has blood pressure or cholesterol-level been checked, has adequacy ofdiabetes control and diabetes self-education been implemented, have necessary vaccinesbeen administered in a timely fashion).Any move to an integrated performance monitoring would require both the adoption of aunique identifier which respected the privacy and confidentiality needs of patients and thecooperation of the myriad of agencies - both at government and regulatory levels - whichoversight our compartmentalised health system. Such a move towards a more holisticperformance appraisal may seem impractical given current realities, but the common goalof improving the value of healthcare, by raising the quality of care and lowering ormaintaining costs, will not be realised until such a global perspective on health servicesperformance is adopted.

56BIBLIOGRAPHYNote: *Represents works regarded by us as key in informing our decision-making for the purposes of thisreport.1. Brennan T, Leape L, Laird N et al. The Harvard Medical Practice Study. New England Journal of Medicine 324: 370-384,1991 *2. Metropolitan Hospitals Planning Board - Phase 2: Discussion Paper: Criteria to Assess the Performance of Networks.Metropolitan Hospitals Planning Board Report. Victorian Government Printing Office. 19953. Scally G. Dealing with duffers. The Lancet 346: 720, 19954. Stiell I, Wells G, Laupacis A et al. Multicentre trial to introduce the Ottawa Ankle Rules for use of radiography in acuteankle injuries. British Medical Journal 311: 594-597, 19955. Castella X, Artigas A, Bion J et al. The European/North American Severity Study Group. Critical Care Medicine 23:1327-1335, 19956. Rapoport J, Teres D, Barnett R et al. A comparison of Intensive Care Unit utilisation in Alberta and WesternMassachusetts. Critical Care Medicine 23: 1336-1346, 1995 *7. McGarvey RN, Harper JJ. Pneumonia mortality reduction and quality improvement in a community hospital. QualityReview Bulletin 19: 124-130, 19938. Berwick D. Eleven worthy aims for clinical leadership of health system reform. Journal of the American MedicalAssociation 272: 797-802, 1994 *9. Symposium on the Australian Incident Monitoring Study. Anaesthesia and Intensive Care 21: 501-595, 1993 *10. Leape LL, Bates DW, Cullen DJ. Systems analysis of adverse drug events. JAMA 274: 35-43, 199511. Leape LL. Error in medicine. JAMA 272: 1851-1857, 1994 *12. Headrick L and Neuhauser D. Quality health care. JAMA 273: 1718-1720, 1995 (R), JAMA 271: 1711-1712, 199313. Goldman RL, Thomas TL. Using mortality rates as a screening tool: The experience of the Department of VeteransAffairs. Joint Commission Journal on Quality Improvement 20 (9): 511-521, 1994 *14. Braff JB, Way BB, Steadman HJ. Incident Reporting: Evaluation of New York’s pilot incident logging system. QualityReview Bulletin 12 (3): 90-98, 198615. Department of Health. Review of Guidance on Doctors' Performance: Maintaining Medical Excellence. London. DOH.1995. AND Editorial Lancet 346: 431, 199516. Anderson GM, Newhouse JP, Roos LL. Hospital care for elderly patients with diseases of the circulatory system: Acomparison of hospital use in the USA and Canada. NEJM, 321: 1443-1448, 198917. Simbourg DW. DRG Creep: A new hospital-acquired disease. NEJM 304: 1602-1604, 198118. Stern RS, Epstein AM. Institutional responses to prospective payment based on diagnosis-related groups: Implications forcost, quality and access. NEJM 312: 621-627, 198519. Ginsburg PB, Carter GM. Medicare case-mix index increase. Health Care Finance Review 7: 51-65, 198620. Winslow CM, Kosecoff JB, Chassin M, Kanousee D, Brook RH. The appropriateness of performing coronary artery bypasssurgery. JAMA 260: 505-509, 198821. Brook RH, Kosecoff JB, Park RE, Chassin MR, Winslow CM, Hampton JR. Diagnosis and treatment of coronary disease:comparison of doctors' attitudes in the USA and UK. Lancet 750-753, 1988 *22. Wennberg JE, Gittelsohn A. Small-area variations in health care delivery. Science 182: 1102-1108, 1972

5723. Wennberg JE, Gittelsohn A. Variations in medical care among small areas. Scientific American 46: 120-134, 198224. Chassin MR, Brook RH, Park PE et al. Variations in the use of medical and surgical services by the Medicare population.NEJM 314: 285-290, 198625. Leape LL, Park RE, Solomon DH, Chassin MR, Kosecoff JJ, Brook RH. Does Inappropriate Use Explain Small-areaVariations in the Use of Health Care Services? JAMA 263: 669-672, 199026. Burns H. Interventions vary (thrombolytics in AMI, haemorrhoidectomy, prostatectomy, hysterectomy, D&C, tonsillectomy,CABG) as do outcomes (cancer). Healthcover 5: 36-39, 199527. Classen DC, Evans S, Pestotnik SL, Horn SD, Menlove RL, Burke JP. The timing of prophylactic administration ofantibiotics and the risk of surgical wound infection. NEJM 326: 281-286, 199228. Rapoport J, Teres D, Lemeshow S, Gehlbach S. A method for assessing the clinical performance and cost-effectiveness ofintensive care units: A Multi-Centre Inception Cohort Study. Critical Care Medicine 22: 1385-1391, 199429. Jollis JG, Ancukiewicz M, DeLong E, et al. Discordance of databases designed for claims payment versus clinicalinformation systems. Implications for outcomes research. Annals of Internal Medicine 119: 844-850, 199330. Miller M, Miller L, Fireman B, Black S. Variation in practice for discretionary admissions. Impact on estimates of qualityof hospital care. JAMA 271: 1493-1498, 199431. Orchard C. Comparing healthcare outcomes. BMJ 308: 1493-1496, 199432. Comorbidities, complications and coding bias. Does the number of diagnosis codes matter in predicting in-hospitalmortality? JAMA 267: 2197-2203, 1992 *33. Personal Communication. Dr Marie Miller. Department of Quality and Utilisation. Kaiser Permanente Medical CareProgram. 1814 Franklin St., Oakland CA 94612, 199634. Concato J, Horwitz R, Feinstein AR et al. Problems of comorbidity in mortality after prostatectomy. JAMA 267: 1077-1082, 1992 *35. Budois RW, Rogers WH, Moxley JH, Draper D, Brook RH. Hospital inpatient mortality - is it a predictor of quality?NEJM 317: 1674-1680, 1987 *36. Marc R, de Leval M et al. Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switchoperations. Journal of Thoracic and Cardiovascular Surgery 107: 914-924, 199437. Brook RH and Lohr K. Monitoring quality of care in the Medicare program. Two proposed systems. JAMA 258: 3138-3141, 198738. Greenfield S, Aronow H, Elashoff R, Watanabe D. Flaws in mortality data. The hazards of ignoring comorbid disease.JAMA 260: 2253-2255, 1987 *39. Leape L, Hillborne L, Park R et al. The appropriateness of use of coronary artery bypass graft surgery in New York State.JAMA 269: 753-760, 1993 *40. Hillborne L, Leape L, Bernstein S et al. The appropriateness of use of PCTCA in New York State. JAMA 269: 761-765,1993 *41. Bernstein S, Hillborn L, Leape L, et al. The appropriateness of use of coronary angiography in New York State. JAMA269: 766-769, 1993 *42. Bates D, Cullen D, Laird N et al. Incidence of adverse drug events and potential adverse drug events. Implications forprevention. JAMA 274: 29-34, 1995 *43. Cooper JB, Newbower RS, Kitz RJ. An analysis of major errors and equipment failures in anaesthesia management:Considerations for prevention and detection. Anaesthesiology 60: 34-42, 198444. Gaba DM, Maxwell M, DeAnda A. Anaesthetic mishaps: Breaking the chain of accident evolution. Anaesthesiology 66:670-676, 198745. Bogner MS Ed. Human error in medicine. Hillsdale NJ. Erlbaum 1994

5846. Davies JM, Webb RK. Adverse events in anaesthesia: The wrong drug. Canadian Journal of Anaesthesia 41: 83-86,199447. O'Neil AC, Peterson LA, Cook EF et al. Physician reporting compared with medical record review to identify adversemedical events. Annals of Internal Medicine 119: 370-376, 1993 *48. Rubenstein L, Kahn K, Reinisch E et al. Changes in quality of care for 5 diseases measured by implicit review, 1981 to1986. JAMA 264: 1974-1979, 199049. Williams S, Bryan P, Schlup M. Quality control: An application of the cusum. BMJ 304: 1359-1361, 199250. Brennan T, Localio R, Leape L et al. Identification of adverse events occurring during hospitalisation. A cross-sectionalstudy of litigation, quality assurance and medical records at two teaching hospitals. Annals of Internal Medicine 112: 221-226, 1990 *51. Knaus W, Draper E, Wagner D, Zimmerman J. An evaluation of outcome from intensive care in major medical centres.Annals of Internal Medicine 104: 410-418, 198652. Caplan R, Posner K, Cheney F. Effects of outcome on physician judgements of appropriateness of care. JAMA 265: 1957-1960. Anaesthesia B2-AN, 1991 *53. Watching the doctor-watchers. How well do PRO methods detect hospital care quality problems. JAMA 267: 2349-2354,199254. Panniers TL, Newlander J. The adverse patient occurrences inventory: Validity, reliability and implications. QualityReview Bulletin, September 198655. Kazandjian VA. Relating outcomes to processes of care: The Maryland Hospital Association's Quality Indicator Project.Joint Commission Journal on Quality Improvement 530-538, November 199356. Kazandjian VA, Lawthers J, Cernak C and Pipesh F. Quality and accountability. Joint Commission Journal on QualityImprovement 20: 7, 199457. Burstin HR, Lipsitz SR, Brennan TA. Socioeconomic status and risk for substandard medical care. JAMA 268: 2383-2387,199258. Chassain MR. Improving the quality of healthcare. NEJM 335: 1060-1062,199659. Weissman JS, Gatsonis C, Epstein AM. Rates of avoidable hospitalization by insurance status in Massachusetts andMaryland. JAMA 268: 2388-2394, 199260. Blumenthal D, Epstein A. The role of physicians in the future of quality management. NEJM 335: 1328-1331, 199661. Brennan TA, Hebert LE, Laird NM et al. Hospital characteristics associated with adverse events and substandard care.JAMA 265: 3265-3269, 199162. Couch NP, Tilney NL, Rayner AA, Moore FD. The high cost of low-frequency events. The anatomy and economics ofsurgical mishap. NEJM 304: 634-637, 198163. Steel K, Gertman PM, Crescenzi C, Anderson J. Iatrogenic illness on a general medical service at a University TeachingHospital. NEJM 304: 638-642, 198164. Roessner J. The healthiest place in America. Quality Connection 2: 10-11, 199365. Hiatt HH, Barnes BA, Brennan TA et al. A study of medical injury and medical malpractice. NEJM 321: 480-484, 198966. McDonald C. Protocol-based computer reminders, the quality of care and the non-perfectability of man. NEJM 295: 1351-1355, 197667. Phelps C E. The methodologic foundations of studies of the appropriateness of medical care. NEJM329: 1241-1245, 199368. Tannenbaum S. What physicians know. NEJM 329: 1268-1271, 199369. Vincent CA. Research into medical accidents: A case of negligence? BMJ 299: 1150-1153, 1989 *

5970. Nadzam DM. Infection control indicators in critical care settings. Heart & Lung 21 (5): 477-481, 199271. Schafer J et al. Outcome prediction models on admission in a medical intensive care unit: Do they predict individualoutcome? Critical Care Medicine 18 (10): 1111-1117, 199072. Nadzam DM et al. Data-driven performance improvement in health care: The Joint Commission's Indicator MeasurementSystem (IM System). Joint Commission Journal on Quality Improvement 19 (11): 492-500, November 199373. Hart GK, Baldwin I, Gutteridge G, Ford J. Adverse incident reporting in intensive care. Anaesthesia and Intensive Care 22:556-561, 199474. Giraud T et al. Iatrogenic complications in adult intensive care units: A prospective two-centre study. Critical CareMedicine 21: 40-51, 199375. Allnut MF. Human factors in accidents. British Journal of Anaesthesia 856-864, 1987 *76. Luft HS, Bunker JP, Enthoven AC. Should operations be regionalised? The empirical relationship between surgicalvolume and mortality. NEJM 301: 1364-1369, 197977. Kimmel SE, Berlin JE and Laskey WK. The relationship between coronary angioplasty procedure volume and majorcomplications. JAMA 274: 1137-1142, 1995 **78. Lee A, Bishop G, Hillman KM, Daffun K. The medical emergency team. Anaesthesia and Intensive Care 23: 183-186,199579. Quigley P, Mathis A and Nodhturft V. Improving clinical documentation quality. Journal of Nursing Care Quality 8 (4):66-73, 199480. Hartz AJ et al . Assessing providers of coronary revascularization: A method for peer review organisations. AmericanJournal of Public Health 82 (12): 1631-1640, 199281. Winograd CH et al. Screening for frailty: Criteria and predictors of outcomes. Journal of the American Geriatrics Society39: 778-784, 199182. Ferguson RP et al. Serum albumin and prealbumin as predictors of clinical outcomes of hospitalized elderly nursing homeresidents. Journal of the American Geriatrics Society 41: 545-549, 199383. Iezzoni L, Ash A, Coffman G, Moskowitz M. Predicting in-hospital mortality. A comparison of severity measurementapproaches. Medical Care 30: 347-359, 1992 *84. Holloway JJ, Vanderburg Medendrop S, Bromberg J. Risk factors for early readmissions among veterans. Health ServicesResearch 25: 213-237, 199085. DesHarnis S, McMahon L, Woblewski R. Measuring outcomes of hospital care using multiple risk-adjusted indices. HealthServices Research 26: 425-445, 1991 *86. Bradbury RC, Stearns FE, Steen P. Interhospital variations in admission-severity-adjusted hospital morbidity and mortality.Health Services Research 26: 408-424, 199187. Nettleman MD and Nelson AP. Adverse occurrences during hospitalisation on a general medical service. ClinicalPerformance and Quality Health Care 2: 67-72, 199488. Mark BA, Burleson DL. Measurement of patient outcomes: Data availability and consistency across hospitals. Journal ofNursing Administration 25: 52-59, 199589. Bates DW, O'Neil AC et al. Potential identifiability and preventability of adverse events using information systems.Journal of the American Medical Informatics Association 1: 404-411, 199490. Hammond P, Harris AL, Das SK, Wyatt JC. Safety and decision support in oncology. Methods of Information in Medicine33: 371-381, 199491. Happ BA. The effect of point of care technology on the quality of patient care. Proceedings - The Annual Symposium onComputer Applications in Medical Care 183-187, 199392. Roos NP. Using administrative data from Manitoba, Canada to study treatment outcomes: developing control groups and

60adjusting for case severity. Social Science and Medicine 28: 109-113, 1989 *93. Silber JH, Williams SV, Krakauer H, Schwartz JS. Hospital and patient characteristics associated with death after surgery.A study of adverse occurrences and failure to rescue. Medical Care 30: 615-629, 199294. Conducting medical effectiveness research: A report from the Inter-PORT Work Groups. (Various Authors). Medical Care32 (Suppl): JS1-JS110, 1994 *95. Anderson GF, Alonso J, Kohn LT, Black C. Analyzing health outcomes through international comparisons. Medical Care32: 526-534, 199496. Young WW, Kohler S, Kowalski J. PMC patient severity scale: Derivation and validation. Health Services Research 29:367-390, 199497. Iezzoni L, Heeren T, Foley S et al. Chronic conditions and risk of in-hospital deaths. Health Services Research 29: 435-460, 199498. Aiken L, Smith HL, Lake ET. Lower Medicare mortality among a set of hospitals known for good nursing care. MedicalCare 32: 771-787, 199499. Iezzoni LI, Daley J, Heeren T et al. Identifying complications of care using administrative data. Medical Care 32: 700-715, 1994 *100. Romano P, Mark D. Bias in the coding of hospital discharge data and its implications for quality assessment. MedicalCare 32: 81-90, 1994101. Roos LL, Mustard CA, Nicol JP et al. Registries and administrative data: Organisation and accuracy. Medical Care 31:201-212, 1993102. Best WR and Cowper DC. The ratio of observed-to-expected mortality as a quality of care indicator in non-surgical VApatients. Medical Care 32: 390-400, 1994103. Des Harnais SI, McMahan LF, Wroblewsk RT and Hogan AT. Measuring hospital performance. The development andvalidation of risk-adjusted indices of mortality, readmissions and complications. Medical Care 28: 1127-1141, 1990 *104. Green LA and Becker MP. Physician decision making and variation in hospital admission rates for suspected acute cardiacischaemia. A tale of two towns. Medical Care 32: 1086-1097, 1994105. Selker HP, Griffith JL and D'Agostino RB. A time-insensitive predictive instrument for acute hospital mortality due tocongestive heart failure: Development, testing and use for comparing hospitals: A multicentre study. Medical Care 32:1040-1052, 1994106. Hartz AJ, Guse C, Sigmann P et al. Severity of illness measures derived from the Uniform Clinical Data Set (UCDS).Medical Care 32: 881-901, 1994107. Friede A, Taylor W, Wadelman L. On-line access to a cost-benefit/cost-effectiveness analysis bibliography via CDCWONDER Medical Care 31: JS12-JS18,1993108. Brennan TA, Localio RT, Laird NL. Reliability and validity of judgements concerning adverse events suffered byhospitalised patients. Medical Care 27: 1148-1158, 1989 *109. Peterson LA, Brennan TA, O'Neill AC et al. Does housestaff discontinuity of care increase the risk for preventable adverseevents? Annals of Internal Medicine 121: 866-872, 1994110. Hartz AJ, Kuhn EM, Kayser KL, et al. Assessing providers of coronary revascularisation - a method for peer revieworganisations. American Journal of Public Health 82: 1631-1640, 1992111. Sutton J, Standen P, Wallace A. Unreported accidents to patients in hospital. Nursing Times 90: 46-49, 1994112. Krakauer H, Bailey BC, Cooper H, Yu WK, Skellan KJ, Kattakkuzhy G. The systematic assessment of variations inmedical practices and their outcomes. Public Health Reports 110: 2-12, 1995113. Lindley CM, Tully MP, Paramsothy V, Tallis RC. Inappropriate medication is a major cause of adverse drug reactions inelderly patients. Age and Aging 21: 294-300, 1992

61114. Wolff AM. Limited adverse occurrence screening. A medical quality control system for medium sized hospitals. MJA 156:449-452, 1992 *115. Clayton DG, Barker L, Runciman WB. Evaluation of safety procedures in anaesthesia and intensive care. Anaesthesia andIntensive Care 21: 670-672, 1993 *116. Roos NP, Black CD, Roos LL, et al. A population-based approach to monitoring adverse outcomes of medical care.Medical Care 33: 127-138, 1995 *117. Hardwick RH, Saltrese-Taylor A, Collins D. Need to measure outcome after discharge in surgical audit. Quality in HealthCare 1: 165-167, 1992118. Naessens JM, Leibson CL, Krishan I, Ballard DJ. Contribution of a measure of disease complexity (COMPLEX) to aprediction of outcome and charges among hospitalised patients. Mayo Clinic Proceedings 67: 1140-1149, 1992119. Riley G, Lubitz J, Gornick M, Mentech R, Eggers P, McBean M. Medicare beneficiaries: Adverse outcomes afterhospitalisation for eight procedures. Medical Care 31: 921-949, 1993120. Hand R, Pointek F, Klemka-Walden L, Inczauskis D. Use of statistical control charts to assess outcomes of medical care:Pneumonia in Medicare patients. American Journal of the Medical Sciences 307: 329-334, 1994121. Ashton CM, Kuykendall DH, Johnson ML, et al. A method of developing and weighting explicit process of care criteria forquality assessment. Medical Care 32: 755-770, 1994122. Knaus W, et al. APACHE-II: A severity of disease classification system. Critical Care Medicine 13 (10): 818-829, 1985123. Lemeshow S, Teres D, et al. A comparison of methods to predict mortality of intensive care unit patients. Critical CareMedicine 15 (8): 715-722, 1987124. Brannen A L, Godfrey L V, Goetter W E. Prediction of outcome from critical illness: A comparison of clinical judgementwith a prediction rule. Archives of Internal Medicine 149, 1989125. Lemeshow S, et al. Refining intensive care unit outcome prediction by using changing probabilities of mortality. CriticalCare Medicine 16 (5): 470-477, 1988126. Segal HE, Rummel L, Wu B. The utility of PRO data on surgical volume: The example of carotid endarterectomy. QualityReview Bulletin 19: 152-157, 1993127. Jencks S F. Measuring quality of care under Medicare and Medicaid. Health Care Financing Review 16: 39-54, 1995 *128. Randall D, Cebul. Quality improvement, casemix and preventive procedures: When should the playing field be leveled forcarotid endarterectomy? Quality Review Bulletin 19: 150-151, 1993129. Leape LL, Lawthers AG, Brennan TA, Johnson WG. Preventing medical injury. Quality Review Bulletin 19: 144-149, 1993130. Thomas JW and Holloway JJ. Investigating early readmission as an indicator for quality of care studies. Medical Care 29(4): 377-394, April 1991131. Dubois RW, et al. SPECIAL REPORT: Hospital inpatient mortality: Is it a predictor of quality? NEJM 317 (26): 1674-1679, 1987 *132. Chang RWS, et al. Use of APACHE II severity of disease classification to identify intensive care unit patients who wouldnot benefit from total parenteral nutrition. Lancet : 1483-1486, 28 June 1986133. Iezzoni LI, et al. Comorbidities, complications and coding bias - Does the number of diagnosis codes matter in predictingin-hospital mortality? JAMA 267 (16): 2197-2203, 22/29 April 1992 *134. Oulton R. Use of incident report data in a system-wide quality assurance/risk management program. Quality ReviewBulletin 10: 583-587, December 1994135. Hannan EL, et al. A methodology for targeting hospital cases for quality of care record reviews. American Journal ofPublic Health 79 (4): 430-436, April 1989136. Bion JF, et al. Sickness scoring and response to treatment as predictors of outcome from critical illness. Intensive CareMedicine 14: 167-172, 1988

62137. Westbrook JJ, Rushworth RL, Rob MI. Evaluating health care: What can hospital seperation data tell us about thecomplications of hospital care? Journal of Quality in Clinical Practice 14: 157-166, 1994138. Communication: Deborah M Nadzam PhD RN, Director, Indicator Measurement, JCAHO139. DesHarnais SI, Simpson KN. Indices for monitoring hospital outcomes in developed countries. Health Policy 21: 1-15,1992140. Health care reform "report cards" are useful but significant issues need to be addressed. GAO. Report to the Chairman,Committee of Labour and Human Resources, US Senate, Sept 1994. US GAO Health, Education and Human ServicesDivision **141. Kazandjian VA. Outcomes research and accountability: the race to adjust performance data for risk of public disclosure.Vice President for Research, Maryland Hospital Association, Feb 1995 (submitted for publication) *142. Joint Commission on Accreditation of Healthcare Organisations. IMSystem General Information. 24 February 1995143. Kazandjian V, et al. Balancing science and practice in Indicator development: The Maryland Hospital Association QualityIndicator (QI) Project. International Journal for Quality in Health Care 7: 39-46, 1995144. Joint Commission Risk Adjustment Methodology. Joint Commission on Accreditation of Healthcare Organisations. 24February 1995145. Bates D, et al. Evaluation of screening criteria for adverse events in medical patients. Medical Care 33: 452-462, 1995146. Hammermeister K et al. A literature review of evidence for process and outcome interrelationships. Medical Care 33: 055-056, 1995 *147. Hofer TP and Hayward RA. Can early readmission rates accurately detect poor quality hospitals? Medical Care 33: 234-245, 1995 *148. Wray P et al. Selecting disease-outcome pairs for monitoring the quality of hospital care. Medical Care 33: 75-89, 1995 *149. Hammermeister et al. Why it is important to demonstrate linkages between outcomes of care and processes and structuresof care. Medical Care 33: 055-056, 1995 *150. Classen D et al. Computerised surveillance of adverse drug events in hospital patients. JAMA 266: 2847-2851, 1991151. Evans RS et al. Prevention of adverse drug events through computer surveillance. Proceedings - The Annual Symposium onComputer Applications in Medical Care 437-441, 1992152. Westbrook JJ, Rushworth RL, Rob MI. Evaluating health care: What can hospital seperation data tell us about thecomplications of hospital care? Journal of Quality in Clinical Practice 14: 157-166, 1994153. Wolff AM. A review of methods used for medical quality assurance in hospitals: Advantages and disadvantages. Journalof Quality in Clinical Practice 14: 85-97, 1994154. Localio AR and Hamory BH. A report card for report cards. Annals of Internal Medicine 123: 802-803, 1995 *155. Sibritt DW. Readmissions of day-only patients in NSW acute hospitals. Journal of Quality in Clinical Practice 14: 31-38,1994156. Iezzoni LI et al. Predicting who dies depends on how severity is measured: Implications for evaluating patient outcomes.Annals of Internal Medicine 123: 763-770, 1995 **157. Clement D, et al. Access and outcomes of elderly patients enrolled in managed care. JAMA 271: 1487-1492, 1994158. Access of Medicaid recipients to outpatient care. NEJM 330: 1426-1430, 1994159. Majeed FA et al. Monitoring and promoting equity in primary and secondary care. BMJ 308: 1426-1429, 1994160. Friedman E. Bibliography of "Money isn't everything - Non-financial barriers to access". JAMA 271: 1535-1538, 1994161. Jost T. Health System Reform. Forward or backward with quality insight. JAMA 271: 1508-1511, 1994

63162. Freeman H and Corey CR. Insurance status and access to health services among poor persons. Health Services Research 28:531-541, 1994163. Economic appraisal and health technology policy. Social Science and Medicine 38: 1591-1678, 1994164. Lairsen DR et al. Equity of healthcare in Australia. Social Science and Medicine 41: 475-482, 1995 *165. Carr-Hill R A. Efficiency and equity implications of the Health Care Reforms. Social Science and Medicine 39: 1189-1201, 1994166. Vagero D. Equity and efficiency in health reform. A European view. Social Science and Medicine 39: 1203-1210, 1994167. Bashshur RL et al. Beyond the uninsured: Problems in access to care. Medical Care 32: 409-419, 1994168. Britton S. A hospital-wide outcome study. Australian Clinical Review 11, 132-135, 1991169. Stipek CE. Grouping systems enhance outcomes analysis. Topics in Health Information Management 15 (4): 14-25, 1995170. Bruster S, Jarman B et al. National survey of hospital patients. BMJ 309: 1542-1546, 1994171. North American Nursing Diagnosis Association (NANDA). Nursing Diagnosis: Definition and Classification 1995-1996.Philadelphia PA NANDA 1994172. Martin K. Omaha systems. Classification systems for describing nursing practice. Kansas City, MO, American NursesAssociation, 1990173. Moorhead S et al. Nursing interventions classification (NIC). A comparison with the Omaha System and the Home CareClassification. Journal of Nursing Administration 23: 23-29, 1993174. Delaney C, Moorhead S. The nursing minimum data set, standardised language and health care quality. Journal ofNursing Care Quality 10: 16-30, 1995175. Marek KD. Outcome measurement in nursing. Journal of Nursing Quality Assurance 4 (1): 1-9, 1989176. Rantz MJ. Quality measurement in nursing: Where are we now? Journal of Nursing Care Quality 9 (2): 1-7, 1995177. McDaniel C and Nash J. Compendium of instruments measuring patient satisfaction with nursing care. Quality ReviewBulletin 16: 182-188, 1990178. Coenen A and Schoneman D. The nursing minimum data set: Use in the quality process. Journal of Nursing Care Quality10: 9-15, 1995179. Khuri SF, Daley J et al. The National Veterans Administration Surgical Risk Study: risk adjustment for the comparativeassessment of the quality of care. Journal of the American College of Surgeons 180: 519-531, 1995180. O'Leary A. Patient satisfaction as a measure of quality in the care of the elderly. BMJ 1 (9): 470-472, 1992181. Ewing H et al. Developing clinical indicators for cholecystectomy. Australian and New Zealand Journal of Surgery 63:181-185, 1993182. Ruffin M. Developing and using a data repository for quality improvement: The genesis of IRIS. Joint CommissionJournal on Quality Improvement 21 (10): 512-520, October 1995183. Cullen D et al. The incident reporting system does not detect adverse drug events: A problem for quality improvement.Joint Commission Journal on Quality Improvement 21 (10): 541-548, October 1995184. Schiff G. Red man syndrome and the red button system: Where next for inpatient adverse drug reaction reporting? JointCommission Journal on Quality Improvement 21 (10): 549-552, October 1995185. AM Wolff. Limited adverse occurrence screening: using medical record review to reduce hospital adverse patient events.MJA 164: 458-461, 1996186. Morris A, Clemmer T and Pestotnik S. Personal Communications.

64187. James B. Bibliographic material on computer assisted decision-making. Vice President of Medical Research andContinuing Medical Education. IHC Institute for Health Care Delivery Research. Utah, USA.188. Draper M and Hill S. The role of patient satisfaction surveys in a national approach to hospital quality management.Australian Government Publishing Service, Oct 1995189. Agency for Health Care Policy and Research. Using clinical practice guidelines to evaluate quality of care. AHCPR Pub95-0045, March 1995190. Blumenthal D. Quality of care - what is it? NEJM 335: 891-4, 1996191. American Pain Society Quality of Care Committee. Quality improvement guidelines for the treatment of acute pain andcancer pain. JAMA 274: 1874-1880, 1995192. Guadagnoli E, Cleary PD. How consistent is patient-reported preadmission health status when collected duringhospitalisation and posthospitalisation. Medical Care 33: 106-112, 1995 *193. Ayonin JZ, Guadagnoli E, Cleary PD. Physical and psychosocial functioning of women and men after coronary artery bypassgrafting. JAMA 274: 1767-1770, 1995 *194. LaCombe MA. What is it patients want? American Journal of Medicine 99: 558-589, 1995195. Donabedian A. Explorations in quality assessment and monitoring. Vol 3. The methods and findings of qualityassessment and monitoring: an illustrated analysis. Ann Arbor, Mich: Health Administration Press, 1995196. Iezzoni LI, Restuccia JD, Shwartz M et al. The utility of severity of illness information in assessing the quality of hospitalcare: the role of the clinical trajectory. Medical Care 30: 438-444, 1992197. Weingarten SR et al. A study of patient satisfaction and adherence to preventive care practice guidelines. AmericanJournal of Medicine 99: 590-596, 1995198. Cleary PD, McNeil BJ. Patient satisfaction as an indicator of quality of care. Inquiry 25: 25-36, 1988 *199. McFadden ER et al. Protocol therapy for acute asthma: Therapeutic benefits and cost savings. American Journal ofMedicine 99: 651-661, 1995200. Cleary PD, McNeil BJ. Patient satisfaction as an indicator of quality of care. Inquiry 25: 25-36, 1988201. Jennifer Daley, Lisa Iezzoni and Shukri Khuri. Complication rates as a measure of Quality of Care (letter) JAMA 274:1674-1675, 1995. Comment on Silber JH et al. Evaluation of the complication rate as a measure of quality of care. JAMA274: 317-323, 1995202. Gottlieb TW. Quality assurance in a long term care facility. Quality Review Bulletin 10: 51-54, 1984203. Emanuel EJ and Emanuel LL. What is accountability in healthcare? Annals of Internal Medicine 124: 229-239 1996 andEmanuel LL. A professional response to demands for accountability. Practical recommendations regarding ethical aspectsof patient care. Annals of Internal Medicine 124: 240-249, 1996204. Bion JF et al. Validation of a prognostic score in critically ill patients undergoing transport. BMJ 291: 432-434, 1985205. Iezzoni LI. Severity of illness measures. Comments and Caveats. Medical Care 28: 757-761, 1990 *206. Quality of life bibliography and indexes. Medical Care (Suppl) 28: 12: DS1-DS77 *207. Patient judgements of hospital quality: Report of a pilot study. Meicald Care (Suppl) 28: 9: S1-S56, 1990208. Thomas JW, Holloway JJ and Guire KE. Validating risk-adjusted mortality as an indicator for quality of care Inquiry 30(1): 6-22, Spring 1993209. Rantz M. Quality measurement in nursing: Where are we now? Journal of Nursing Care Quality 9 (2): 1-7, 1995 *210. Arford P and Allred C. Value = Quality + Cost. Journal of Nursing Administration. 25 (9): 64-69, 1995211. Griffiths P. Progress in measuring nursing outcomes. Journal of Advanced Nursing 21: 1092-1100, 1995

65212. Ryan M et al. Measuring patient satisfaction: A case study. Journal of Nursing Care Quality 9 (2): 44-54, 1985213. Redfern S, et al The reliability and validity of quality assessment measures in nursing. Journal of Clinical Nursing 1: 47-51, 1992214. Brooten D and Naylor M. Nurses' effects on changing patient outcomes. Journal of Nursing Scholarship 27 (2): 95-99,1995215. Slater P. The past and future of nursing research in Australia. The Australian Journal of Advanced Nursing 1 (2): 43-54,1984216. Nadzam P. The Agenda for Change: Update on indicator development and possible implications for the nursing profession.Journal of Nursing Care Quality 5 (2): 18-22, 1991217. Rudolph B and Hill C. The components of hospital quality: A nursing perspective. Journal of Nursing Care Quality 9 (1):57-65, 1994218. Higgins M, et al. Assessing the outcomes of nursing care. Journal of Advanced Nursing 17: 561-568, 1992219. Valentine K. Comprehensive assessment of caring and its relationship to outcome measures. Journal of Nursing QualityAssurance 5 (2): 59-68, 1991220. Lang D and Dorman Marek K. The policy and politics of patient outcomes. Journal of Nursing Quality Assurance 5 (2): 7-12, 1991221. Stewart B and Archbold P. Focus on psychometrics - nursing intervention studies require outcome measures that aresensitive to change: Part 2. Research in Nursing and Health 16: 77-81, 1983222. Stewart B and Archbold P. Focus on psychometrics - nursing intervention studies require outcome measures that areSensitive to Change: Part 1. Research in Nursing and Health 15: 477-481, 1992223. Hegyvary S. Issues in outcomes research. Journal of Nursing Quality Assurance 5 (2): 1-6, 1991224. Schroeder P. From the Editor. Journal of Nursing Quality Assurance 5 (1): vi, 1991225. Schroeder P. From the Editor. Journal of Nursing Quality Assurance 5 (2): viii, 1991226. Schroeder P. From the Editor. Journal of Nursing Quality Assurance 6 (3): viii, 1992227. Schroeder P. From the Editor. Journal of Nursing Quality Assurance 8 (2): vii, 1994228. Schroeder P. From the Editor. Journal of Nursing Quality Assurance 9 (3): viii, 1995229. Schroeder P. From the Editor. Journal of Nursing Quality Assurance10 (1): viii, 1995230. Bond S and Thomas L. Measuring Patients' Satisfaction with Nursing Care. Journal of Advanced Nursing 17: 52-63, 1992231. Libreri F. An Acute Pain Service: A Quality Assurance Survey of Nurses and Doctors. Australian Journal of AdvancedNursing 12 (4): 33-38, 1995232. Pratt R, et al. The Effects of all RN and RN-EN Staffing on the Quality and Cost of Patient Care. Australian Journal ofAdvanced Nursing 10 (3): 27-38, 1993233. Hanson R et al. Patient Dissatisfaction in a Paediatric Accident and Emergency Department. Journal of Quality ClinicalPractice 14: 137-143, 1994234. Directory of Nursing Research in Australia 1991-1993, Royal College of Nursing Australia, Seventh Edition, Print MintSouth Melb, Vic, 1994235. Roberts K. A snapshot of australian nursing scholarship 1993-1994. Collegian 13 (1): 4-10, 1996236. Senior S, et al. Continuous quality improvement in a day ward. Journal of Quality in Clinical Practice 15: 177-181, 1995237. Webster J. Quality assurance: Measures of maternal knowledge and satisfaction. Australian Clinical Review 13: 119-126,1993

66238. Donnells M, et al . Improving care through a medication administration process action team. Journal of Nursing CareQuality 9 (3): 38-44, 1995239. Crowley B, et al . Clinical Indicators: A tool for improving pain management documentation. Journal of Nursing CareQuality 6 (1): 40-46, 1991240. Bechtel G, et al. A continuous quality improvement approach to medication administration. Journal of Nursing CareQuality 7 (3): 28-34, 1993241. Ruckstuhl M, et al. Patient falls: An outcome indicator. Journal of Nursing Care Quality 6 (1): 25-29, 1991242. Davis B and Bush H. Developing effective measurement tools: A case study of the Consumer Emergency Care SatisfactionScale. Journal of Nursing Care Quality 9 (2): 26-35, 1995243. Cooper M. Can a zero defects philosophy be applied to drug errors? Journal of Advanced Nursing 21: 487-491, 1995244. Andersen Y. European nurses take lead in quality assurance. International Nursing Review 41 (1): 13-16, 1994245. Pelkonen M. Nursing quality assurance in Finland. International Nursing Review 41 (1): 23-26, 1994246. Giebing H. Nursing quality management in the Netherlands. International Nursing Review 41 (1): 17-21, 1994247. Guva M Collaboration: The pivotal point for quality patient outcome. Journal of Nursing Care Quality 9 (3): 53-58, 1995248. Windle P. Comit - improving patient outcomes. Nursing Management 26 (9): 64 AA, 64 DD, 64 FF, 64 HH, 1995249. Mark B and Burleson D. Measurement of patient outcomes - Data availability and consistency across hospitals. Journal ofNursing Administration 25 (4): 52-59, 1995250. Davis S and Adams-Greenly M. Integrating patient satisfaction with a quality improvement program. Journal of NursingAdministration 24 (12): 28-31, 1994251. Pearson A. Determining quality in a unit where nursing is the primary intervention. Journal of Advanced Nursing 14: 269-273, 1989252. Abramowitz S, et al. Analysing patient satisfaction: A multi-analytic approach. Quality Review Bulletin 122-130, 1987253. Stratmann W, et al. Patient satisfaction surveys and multicollinearity. Quality Management in Health Care 1-12, 1994254. Rosenthal M, et al. Developing methods and research instruments for assessing physical and psychosocial outcomes ofpatient care as a function of nursing staff mix. The Australian Journal of Advanced Nursing 8 (3): 34-41, 1991255. Donabedian A. Quality assessment and assurance: Unity of purpose, diversity of means. Inquiry 25: 173-192, 1988256. Marek K. Outcome measurement in nursing. Journal of Nursing Quality Assurance 4 (1): 1-9, 1989257. Lower M and Burton S. Measuring the impact of nursing interventions - The challenges of the 1990s. Journal of NursingQuality Assurance 4 (1): 27-34, 1989258. Bond S and Thomas. Issues in measuring outcomes of nursing. Journal of Advanced Nursing 16: 1492-1502, 1991259. Bieniek J. Medication administration program - Caulfield General Medical Centre, Unpublished Paper, 1993260. Bieniek J. Establishing the relationship between possible falls risk factors and the incidents of patient falls Unpublishedpaper, 1993261. Rosenthal M, et al. Developing research methods and instruments for assessing patient outcomes, 1991262. Goldman R. The reliability of peer assessments of quality of care. JAMA 276 (7): 958-960, 1992 *263. Morgan CJ, Branthwaite MA. Severity scoring in intensive care. BMJ 292: 1546, 14 June 1986264. Edwards RT and Barlow J. Rationing health care by waiting list: An extra-welfarist perspective. Discussion paper 114.Centre for Health Economics, University of York, UK, 1994

67265. Money RC, Fine DJ and Loree SW. Comparing the allocative efficiencies of hospitals. International Journal ofManagement Science 18 (1): 71-83, 1990266. Richardson J, Segal L, Carter R, Catford J, Galbally R and Johnson S. Prioritising and financing health promotion inAustralia. Research Report 4. Centre for Health Program Evaluation, Melbourne, 1995267. Sherman HD. Hospital efficiency measurement and evaluation: Empirical test of a new technique. Medical Care 22 (10):922-38, 1984268. Grosskopf S and Valdmanis V. Measuring hospital performance: A non-parametric approach. Journal of HealthEconomics 6: 89-107, 1987269. Ehreth J. The development and evaluation of hospital performance measures for policy analysis. Medical Care 32 (6): 568-87, 1994270. Hao L and Pegals CC. Evaluating relative efficiencies of Veterans Affairs Medical Centres using data envelopment, ratioand multiple regression analysis. Journal of Medical Systems 18 (2): 55-67, 1994271. Register CA and Bruning ER. Profit and technical efficiency in the production of hospital care. Southern EconomicJournal 53: 899-913, 1986272. Salinas-Jimenez J and Smith P. Data Envelopment Analysis applied to quality in primary health care. Discussion Paper124. Centre for Health Economics. York University, UK, 1994273. Mooney G, Gerard K, Donaldson C and Farrar L. Priority setting in purchasing: Some practical guidelines. ResearchPaper No.6. The Health Economics Research Unit, University of Aberdeen, 1992274. Ozcan YA, Luke RD and Haksever C. Ownership and technical efficiency across hospital types. Medical Care, 30 (9):781-794, 1992275. Ozcan YA and Luke D. A national study of efficiency of hospitals in urban markets. Health Services Research, 27 (6):719-739, 1993276. Ozcan YA and Bannick RR. Trends in Department of Defence hospital efficiency. Journal of Medical Systems, 18 (2): 69-83, 1994277. Grosskopf S and Valdmanis V. Evaluating hospital performance with case-mix adjusted outputs. Medical Care 31 (6):525-532, 1993278. Finkler MD and Wirtschafter DD. Cost effectiveness and data envelopment analysis. Health Care Management Review 18(3): 81-88, 1993 *279. Rosko MD. Measuring technical efficiency in healthcare organisations. Journal of Medical Systems 14 (5): 307-322, 1990280. Ran S, Lowe CN and Harder G. A quantitative approach to quality improvement and resource allocation. Journal ofQuality in Clinical Practice, 15: 11-16, 1995281. Daly PM. Surgical waiting lists in Victorian hospitals. MJA 155, 19 August 1991282. Eddy DM. Clinical decision making: From theory to practice, "Rationing resources while improving quality". JAMA 272(10), 4 September 1994 *283. Managed Health Care Professionals present strategies for managing cost and quality. (Meeting Update). Quality ReviewBulletin, November 1991284. Pella J, Cook A. Neonates and DRGs: Resolving one weakness. Australian Casemix Bulletin, 1 (2), June 1989285. Kelly, K, et al. The Medical Outcomes Study: A nursing perspective. Journal of Professional Nursing 10(4): 209-216,1994286. Gillette, B., Jenko, M. Major clinical functions: A unifying framework for measuring outcomes. Journal Nursing CareQuality 6 (1): 20-24, 1991287. Eriksen, L. Patient satisfaction: An indicator of nursing care quality. Nursing Management, pp 31-35, 1987

68288. Megivern, K., Halm, M. Jones, G. Measuring patient satisfaction as an outcome of nursing care. Journal of Nursing CareQuality 6 (4): 9-24,1992289. Ambler Peters D. Outcomes: The mainstay of a framework for quality care. Journal of Nursing Care Quality 10 (1): 61-69,1995290. Podgorny K. Developing nursing focused quality indicators: A professional challenge. Journal of Nursing Care Quality 6(1) 47-52, 1991291. CATARACT SURGERY Patient-Reported Data on appropriateness and outcomesUnited States General Accounting Office (Report to Congressional Requesters), April 1993 *292. BONE MARROW TRANSPLANTATION International comparisons of availability and appropriateness of use. United States GeneralAccounting Office (Report to the Chairman, Committee on Labor and Human Resources, U.S. Senate), March 1994293. HEALTH CARE REFORM "Report Cards" are useful but significant issues need to be addressed. United States General AccountingOffice (Report to the Chairman, Committee on Labor and Human Resources, U.S. Senate), September 1994 **294. MEDICARE Increased HMO oversight could improve quality and access to care. United State General Accounting Office (Report tothe Special Committee on Aging, U.S. Senate), August 1995295. HEALTH CARE Employers and individual consumers want additional information on quality. United States General AccountingOffice (Report to the Ranking Minority Member, Committee on Labor and Human Resources, U. S. Senate), September 1995296. Goldberg HI and Cummings MA (Co-Editors). Conducting medical effectiveness research: A report from the Inter-PORT work groups.Medical Care (Suppl) 32 (7) , July 1994 *297. Neuhauser D (Editor). Quality and accountability in practice: measuring, managing, and making it all work in a reformed health caresystem. Medical Care (Suppl) 33 (1), 1995298. Gelb D, Safran D, Rogers W, Lieberman W, Kosinski M. The primary care assessment survey: A tool for improvement, overview of apractice-based performance assessment. The Health Institute New England Medical Center, Boston MA, November 1995299. Kazis LE et al. A modified version of a short form health survey for assessing health record quality of life in veterans: The VeteransHealth Study. Bedford Health Services Research and Development Field Program, Department of Veterans Affairs, Bedford MA,January 1996300. Nerenz DR, Zajac BM, Rosman HS and Zuckerman H. Consortium Research on Indicators of System Performance (CRISP) Report ofPhase I. Center for Health System Studies, Henry Ford Health System, Detroit, Michigan, October 23, 1992301. Nerenz DR, Zajac BM, Rosman HS and Zuckerman H. Consortium Research on Indicators of System Performance (CRISP) Phase IIProposal. Centre for Health System Studies, Henry Ford Health System, Detroit, Michigan, November 20, 1992302. Nerenz DR et al. Consortium Research on Indicators of System Performance (CRISP) Phase III Proposal. Centre for Health SystemStudies, Henry Ford Health System, Detroit, Michigan, December 1994303. Neslund-Dudas C, (Project Coordinator). Consortium Research on Indicators of System Performance (CRISP), Report Sampler.Centre for Health System Studies, Henry Ford Health System, Detroit, Michigan, May 1995304. Kazandjian VH, Lawthers J, Cernak CM and Pipesh FC. Relating outcomes to processes of care: The Maryland Hospital Association'sQuality Indicator Project (QI Project). The Journal of the Joint Commission on Accreditation of Healthcare Organisations pp 530 -538, November 1993305. Medical affairs issues report, The Group Health Association of America (GHAA) and the American Managed Care and ReviewAssociation (AMCRA). GHAA/AMCRA Washington DC, November/December 1995306. Primary care assessment survey. The Health Institute, New England Medical Center, Boston MA, January 1996307. Measuring quality: Guide for presenting HEDIS 2.0 to employers, Group Health Association of America, Inc., 1995 *308. Health care report Cards: Profiles of all major report cards, performance reports, shopping guides, and consumer satisfaction surveys(Special Report), Accountability News for Health Care Managers, Atlantic Information Services, Inc. Washington DC, 1995309. Health care costs in Massachusetts, Executive Summary, Massachusetts Rate Setting Commission, Boston MA, December 1992

69310. Griswold PR, Chairman and Freedman LI, Commissioner. Health care quality and the importance of outcomes measurement . AReport of the Massachusetts Rate Setting Commission to the Joint Committee on Health Care, Boston MA, December 1993 *311. Griswold PR, Chairman, Freedman LI, Commissioner and Randle ML, Commissioner. Preventable hospitalization in Massachusetts.Massachusetts Rate Setting Commission, Boston MA, January 1994312. Weinstein BE, Chairman, Freedman LI, Commissioner and Randle ML, Commissioner. Improving primary care: Using preventablehospitalization as an approach. Massachusetts Rate Setting Commission, Boston MA, April 1995313. Weinstein BE, Chairman, Freedman LI, Commissioner and Randle ML, Commissioner. Monitoring the acute care hospital industry.Massachusetts Rate Setting Commission, Boston MA, May 1995314. Weinstein BE, Chairman, Freedman LI, Commissioner and Randle MI. Commissioner. Comparing hospital costs: Continuing thedialogue for improvement. Massachusetts Rate Setting Commission, Boston MA., July 1995315. Abbott L and James B. Involving physicians in health outcomes assessment. Medical Outcomes Trust Bulletin 4 (2): 1, March 1996316. Damiano AM. The Sickness Impact Profile: Part 1. Medical Outcomes Trust Bulletin 4 (2): 2, March 1996317. Greene RJ. Status of the Patient Outcomes Research Teams (PORTs). Medical Outcomes Trust Bulletin 4 (2): 3, March 1996318. Reverby S. Stealing the golden eggs: Ernest Amory Codman and the science and management of medicine. Bulletin of the History ofMedicine 55: 156-171, 1981319. DesHarnais SI and Simpson KN. Indices for monitoring hospital outcomes in developed countries. Health Policy 21: 1-15, 1992320. Understanding and choosing clinical performance measures for quality improvement: Development of a typology, Report for theDepartment of Health and Human Services, Public Health Service, AHCPR, Rockville, MD. Centre for Health Policy Studies,Columbia, MD, 31 January, 1995 **321. Development of a typology of clinical performance measures for quality improvement. Attachments. Center for Quality of CareResearch and Education, Harvard School of Public Health, Boston MA and Center for Health Policy Studies, Columbia, MD, 12August 1994 **322. Patients Charter Annual Report 1994 - 1995, Bedfordshire Health Consumer Affairs Department, Bedfordshire UK323. Marek KD. Outcome measurement in nursing. Journal of Nursing Quality Assurance 4 (1): 1- 9, 1989324. The Quality Indicator Study Group. An approach to the evaluation of quality indicators of the outcome of care in hospitalized patients,with a focus on nosocomial infection indicators. Infection Control and Hospital Epidemiology. 16 (5): 308 -316, May 1995 **325. Lavis JN and Anderson GM. Appropriateness in health care delivery: Definitions, measurement and policy implications. CanadianMedical Association Journal 154 (3): 321- 328, February 1996326. Fahey PP and Gibberd RW. Monitoring post-operative pulmonary embolisms in Australia. Variation between hospitals. InternationalJournal for Quality in Healthcare [In Press]327. Rontal R et al. Applications for risk-adjusted outcome measures. Quality Assurance in Health Care 13 (4): 283 - 292, 1991328. Fahey P. Clinical indicators, casemix & quality: Issues via example. NSW Health Services Research Group. Department of Statistics,University of Newcastle, Australia, May 1994329. Quality of Measurement or Quality of Medicine? (Editorial), JAMA 273 (19): 1537 - 1538, 17 May 1995 *330. Variation in DRG-based pulmonary embolism rates between NSW acute care hospitals, Paper presented at The Sixth NationalCasemix Conference, August 1994 Hobart, Tasmania. Paul Fahey331. Defining performance of organisations. Joint Commission Journal on Quality Improvement 19 (7): 215 - 221, July 1993332. Hopkins A et al, What do we mean by appropriate health care? Quality in Health Care 2: 117 - 123, 1993333. Health Policy Newsletter. Jefferson Medical College, Thomas Jefferson University, January 1996334. Kazandjian VA and Sternberg EL. Indicators of performance or the search for the best pointer dog. The Epidemiology of Quality, 1995

70335. Quality Indicator Project Users Guide. Maryland Hospital Association, Lutherville, MD, 1995336. Goldman RL and Thomas TL. Using mortality rates as a screening tool: The experience of the Department of Veterans Affairs. JointCommission Journal on Quality Improvement 20 (9): 511 - 521, 1994337. Nerenz DR and Zajac BM. Indicators of performance for vertically integrated health systems (Ray Woodham visiting fellowshipprogram project summary report). Center for Health System Studies, Henry Ford Health System, 1994 *338. Cullen DJ et al. The incident reporting system does not detect adverse drug events: A problem for quality improvement. JointCommission Journal on Quality Improvement 21 (10): 541 -552, October 1995339. Kazandjian VA. The 1996 Medical Outcomes & Guidelines Sourcebook, A Progress Report and Resource Guide on MedicalOutcomes Research and Practice Guidelines: Developments, Data, and Documentation. Maryland Hospital Association, LuthervilleMD340. Kazandjian V, Wood P and Lawthers J. Balancing science and practice in indicator development: The Maryland Hospital AssociationQuality Indicator (QI) Project. International Journal for Quality in Health Care 7 (1): 39 - 46, 1995341. Measuring the outcomes of medical care. Papers based on a conference held in September 1989 organised by the Royal College ofPhysicians and the King's Fund Centre for Health Services Development. Edited by Anthony Hopkins and David Costain, AntonyRowe Ltd., Wiltshire.342. Romano PS et al. The California Hospital Outcomes Project: Using administrative data to compare hospital performance. JointCommission Journal on Quality Improvement 21 (12): 668 - 682, December 1995343. Health Outcomes Bulletin, Australian Health Outcomes Clearing House (AHOCH), Australian Institute of Health and Welfare(AIHW). Number 4 Summer 1995344. Victorian Department of Health and Community Services. A New Framework for Quality in Victoria's Public Hospitals, Final Report,Volume 1, November 1995345. Drummond MF. Principles of Economic Appraisal in Health Care, Oxford Medical Services, 1988346. Drummond MF, Stoddart GL and Torrance GW. Methods for the Economic Evaluation of Health Care Programs, Oxford MedicalPublications, 1988347. Donaldson C and Magnussen J. Economic Framework for Allocative Efficiency in the Health Sector. The Australian EconomicReview: 89-98, 1994348. Segal L and Richardson J. Efficiency in resource allocation. National Centre for Health Program Evaluation, Working Paper No. 34,March 1994349. Scotton RB and Owens HJ. Case payments in Australian Hospitals: issues and options. Public Sector Management Institute, MonashUniversity, August 1990350. Mooney G, Gerard K, Donaldson C and Farrar L. Priority setting in purchasing: some practical guidelines, Research Paper 6, HealthEconomics Research Unit, University of Aberdeen, UK, 1992351. National Health Ministers’ Benchmarking Working Group. First national report on health sector performance indicators: publichospitals - the state of play, Australian Institute of Health and Welfare, Canberra, 1996 *352. Australian Institute of Health and Welfare (AIHW). National Health Data Dictionary, Version 4.0, Canberra, 1995353. Ehreth J. The development and evaluation of hospital performance measures for policy analysis. Medical Care 32 (6 Suppl): s68-s87,1994354. Rosko MD. Measuring technical efficiency in healthcare organisations. Journal of Medical Systems 14 (5): 307-322, October 1990355. Metropolitan Hospitals Planning Board Victoria. Developing Melbourne’s hospital network. Phase 1 Report, 1995356. Metropolitan Hospitals Planning Board Victoria 1995. Criteria to assess the performance of networks. Phase 2 Report, 1995357. Victorian Department of Health and Community Services. A new framework for quality in Victoria’s public hospitals. Final report.Volume 1, 1995

71358. Victorian Department of Health and Community Services. A new framework for quality in Victoria’s public hospitals. Final report.Volume 2, 1995359. Victorian Department of Health and Community Services. Hospital service report, December 1995360. NSW Health. Annual report 1994-95, 1995361. NSW Health. Caring for health - equity, efficiency and quality. NSW Government’s Economic Statement for Health, 1995362. Woolhandler S and Himmelstein DU. The deteriorating administrative efficiency of the US health care system. NEJM 324: 1253-8,1991363. Michael R and Piper K. Flinders Medical Centre non-inpatient costing study. Australian Health Review. 14 (2): 127-36, 1991364. Lagaida R and Hindle D. A casemix classification system for hospital-based ambulatory services. A report to the National AmbulatoryCasemix Project. State Health Publication No.IC 92-89. New South Wales Department of Health, 1992.365. Keith AR, Pirkus JE, et al. Delivery of primary care in hospital and community settings in Australia. Quality Assurance in HealthCare 5 (2): 131-41, 1993366. Jackson T, Sevil P, Tate R and Collard K. The development of relative resource weights for non-admitted patients. National Centrefor Health Program Evaluation, Monash University, May 1991367. The Australian Council on Healthcare Standards (ACHS) Care Evaluation Program. Hospital-wide medical indicators: a users’manual. ACHS Sydney, July 1991 *368. Zuckerman S, Hadley J and Iezzoni L. Measuring hospital efficiency with frontier cost functions. Journal of Health Economics 13 (3):255-280, 1994 *369. Vitaliano DF and Toren M. Cost and efficiency in nursing homes: a stochastic frontier approach. Journal of Health Economics 13 (3):281-300, 1994370. Kooreman P. Nursing home care in The Netherlands: a nonparametric efficiency analysis. Journal of Health Economics 13 (3): 301-316, 1994371. Newhouse JP. Frontier estimation: How useful a tool for health economics. Journal of Health Economics 13 (3): 323-328,1994 *372. A Dor. Non-minimum cost functions and the stochastic frontier: on applications to health care providers. Journal of HealthEconomics 13 (3): 329-334, 1994373. Hadley J and Zuckerman S. The role of efficiency measurement in hospital rate setting. Journal of Health Economics 13 (3):335-340, 1994374. Vitaliano DF and Toren M. Frontier analysis: a reply to Skinner, Dor and Newhouse. Journal of Health Economics 13 (3):341-344, 1994375. Kooreman P. Data envelopment analysis and parametric frontier estimation: complementary tools. Journal of HealthEconomics 13 (3): 345-346, 1994376. Eckerman S. Data envelopment analysis. Australian Institute of Health and Welfare, unpublished report obtained by request377. Charnes A, Cooper W and Rhodes E. Measuring the efficiency of decision making units. European Journal of OperationalResearch 2 (6): 429-444, 1978378. Salinas-Jimanez J and Smith P. Data envelopment analysis applied to quality in primary health care. Discussion Paper 124,Centre for Health Economics, York Univeristy, UK 1994379. Bessant A, Bessant W, Kennington J and Reagan B. An application of mathematical programming to assess productivity inthe Houston independent school district. Management Science 28: 1355-67, 1982380. Charnes A, Cooper WW and Rhodes E. Evaluating program performance and management efficiency: an application of dataenvelopment analysis to program follow through. Management Science 27: 668-97, 1981

72381. Lewin AY, Morey RC and Cook TJ. Evaluating administrative efficiency of courts. Omega 10 (4): 401-11, 1982382. Fare R and Primont D. Efficiency measures for multi-plant firms. Operation Research Letters 3: 257-60, 1984383. Banker RD. Estimating the most productive scale size using data envelopment analysis. European Journal of OperationsResearch 17: 35-44, 1984384. Fare R, Grosskopf S and Pasurka C. The effect of environmental regulations on the efficiency of electric utilities: 1969 versus1975. Applied Economics 21: 225-32, 1989385. Charnes A, Clarke CT, Cooper WW and Golany B. A developmental study of data envelopment analysis in measuring theefficiency of maintenance units in the US air force. Annals of Operations Research 2: 95-112, 1985386. Byrnes P, Fare R and Grosskopf S. Measuring productive efficiency: an application to Illinois Strip Mines. ManagementScience 30: 671-81, 1984387. Byres P and Fare R. Surface mining of coal: efficiency of US interior mines. Applied Economics 19: 1665-73, 1987388. Whiteman J and Pearson K. Benchmarking telecommunications using data envelopment analysis. Economic papers 12 (3):97-105, 1993389. Sherman HD. Hospital efficiency measurement and evaluation: empirical test of a new technique. Medical Care 22 (10): a22-a38, 1984390. Burgess JF and Wislon PW. Technical efficiency in Veterans Administration Hospitals in The Measurement of ProductiveEfficiency, eds HO Fried, CAK Lovell and SS Schmidt, Oxford University Press, Oxford 1993391. Grosskopf S and Valdmanis V. Evaluating hospital performance with casemix adjusted outputs. Medical Care 31 (6): 525-532, 1993392. Grosskopf S and Valdmanis V. Measuring hospital performance: a non-parametric approach. Journal of Health Economics6: 89-107, 1987393. Hao L and Pegals CC. Evaluating relative efficiencies of Veterans’ Affairs medical centres using data envelopment, ratio andmultiple regression analysis. Journal of Medical Systems 18 (2): 55-67, 1994394. Morey RC, Fire DJ and Loree SW. Comparing the allocative efficiencies of hospitals. International Journal of ManagementScience 18 (1): 71-83, 1990395. Ozcan YA and Luke D. A national study of efficiency of hospitals in urban markets. Health Services Research 27 (6): 719-739, 1993396. Ozcan YA, Luke RD and Haksever C. Ownership and technical efficiency across hospital types. Medical Care 30 (9): 781-794, 1992397. Ozcan YA and Bannick RR. Trends in Department of Defence hospital efficiency. Journal of Medical Systems 18 (2): 69-83,1994398. Register CA and Bruning ER. Profit and technical efficiency in the production of hospital care. Southern Economic Journal535: 899-913, 1986399. Sexton TR, Leiken AM, Nolan AH, Liss L, Hogan A and Silkman RH. Evaluating managerial efficiency of VeteransAdministration Medical Centres using data envelopment analysis. Medical Care 27 (12): 1175-1188, 1989400. Valdmanis V. Sensitivity analysis for DEA models - an empirical analysis using public versus NFP hospitals. Journal ofPublic Economics 48: 185-205, 1992401. Finkler MD and Wirtschafter DD. Cost effectiveness and data envelopment analysis. Health Care Management Review 18(3): 81-88, 1993402. Granley JA and Cubbin JS. Public Sector Efficiency Measurement: Applications of Data Envelopment Analysis, Amsterdam,North Holland, 1992403. Thrall RM. Classifications transitions under expansion of inputs and outputs in DEA. Managerial and Decision Economics10 (2): 159-162, 1990

73404. Rosenstein AH. Cost-effective health care: tools for analysis. Health Care Management Review 19 (2): 53-61, 1994405. Richardson J, Segal, Carter R, Catford J, Galbally R and Johnson S. Prioritising and financing health promotion in Australia,Research Report 4. Centre for Health Program Evaluation, Melbourne, 1995 *406. Mooney G. Program budgeting: an aid to planning and priority setting in healthcare. Effective Health Care 2 (2): 65-68,1994407. Eddy DM. What’s going on in Oregon. Journal of the American Medical Association 266 (3): 417-420, 1991 *408. Kaplan RM. Value judgement in the Oregan medicaid experiment. Medical Care 32 (10): 975-988, 1994 *409. Nord E. An alternative to QALYs: the saved young life equivalent (SAVE). BMJ 15 (2):124-134, 1992410. Hall J and Haas M. The rationing of health care: should Oregon be transported to Australia? Australian Journal of PublicHealth 16 (4): 340-349, 1993411. Street A and Richardson J. The value of health care: what can we learn from Oregon? Australian Health Review 15 (2): 124-134, 1992 *412. Scotton RB 1993. Restructuring the financing of health care. Paper presented for the Office of the Economic PlanningAdvisory Council Seminar Investing in Health Care - A Challenge for the Future, Canberra, October 1993413. Street A. Purchaser/provider and managed competition: importing chaos. National Centre for Health Program Evaluation.Working Paper No. 36. April 1994414. Commonwealth Department of Human Services and Health. Guidelines for the pharmaceutical industry on preparation ofsubmissions to the pharmaceutical benefits advisory committee: including major submissions involving economic analysis.Australian Government Publishing Service, Canberra. November 1995415. Carter R. A macro approach to economic appraisal in the health sector. Australian Economic Review 2nd quarter: 105-112,1994 *416. Drummond MF. Output measurement for resource allocation decisions in health care. Oxford Review of Economic Policy 5(1): 59-74, 1993417. Morey RC, Fine DJ, Loree SW et al. The trade-off between hospital cost and quality of care: an exploratory empiricalanalysis. Medical Care 30 (8): 677-698, 1992418. Evans R. Canada: the real issues. Journal of Health Politics, Policy and Law 17 (4): 738-762, 1992419. Kenney G. How access to long-term care affects homehealth transfers. Journal of Health Politics, Policy and Law 18 (4):937-965, 1993420. Ford E, Cooper R. Racial/ethnic differences in health care utilisation of cardiovascular procedures. Health Service Research30 (1): 237-250, 1995421. Madhok R, Green S. Orthopaedic outpatient referral guidelines: experiences from an English health district. InternationalJournal for Quality in Health Care 6 (1): 73-76, 1994422. Wells K, Rogers W, Davis L et al. Quality of care for depressed elderly pre-post prospective payment system: differences inresponse across settings. Medical Care 32 (3): 257-276, 1994423. Arrington B, Haddock C. Who really profits from not-for-profit hospitals? Health Services Research 25 (2): 291-304, 1990424. Green J, Dale J. Primary care in accident and emergency and general practice: a comparison. Social Science and Medicine35 (8): 987-995, 1992425. Dranove D, White W and Wu L. Segmentation in local hospital markets. Medical Care 31 (1): 52-64, 1993426. Fitzgerald J, Freund D, Hughett B, et al. Influence of organisational components on the delivery of asthma care. MedicalCare 31 (3): MS61-MS73, 1993

74427. Beland F, et al. Socio-spatial perspectives on the utilisation of emergency hospital services in two urban territories in Quebec.Social Science and Medicine 30 (1): 53-66, 1990428. de Marco P, Dain C, Lockwood T, et al. How valuable is feedback of information on hospital referral patterns? BMJ 307:1465-1466, 1993429. Azeem Majeed F, et al. Monitoring and promoting equity in primary and secondary care. BMJ 308: 1426-1429, 1994430. Bailey J, Black M, Wilkin D. Specialist outreach clinics in general practice. BMJ 308: 1083-1086, 1994431. Smith T. Waiting times: monitoring the total post-referral wait. BMJ 309: 593-596, 1994432. Pollock A. Where should health services go: local authorities versus the NHS? BMJ 310: 1530-1539, 1995433. Every N, Larson E, Litwin P, et al. The association between on-site cardiac catheterisation facilities and the use of coronaryangiography after acute myocardial infarction. NEJM 329: 546-551, 1993434. Chirboga D et al. A community wide perspective of gender differences and temporal trends in the use of diagnostic andrevascularisation procedures for acute myocardial infarction. American Journal Cardiology 71: 268-273, 1993435. Muus K, Luktke R, Gibbens B. Community perceptions of rural hospital closure. Journal of Community Health 20 (1): 65-73, 1995436. Mirvis D, et al. Variation in utilisation of cardiac procedures in the Department of Veterans Affairs health care system: effectof race. Journal Americal College Cardiology 24: 1297-304, 1994437. Shea S et al. Predisposing factors for severe uncontrolled hypertension in an inner city minority population. NEJM 327: 776-781, 1992438. Lloyd P, McCarthy W, Nolan J. The beds to the west scheme: the politics and outcomes of the 1982 NSW hospitalrationalisation program. MJA 153: 486-491, 1990439. Naylor CD, Levington C, Baigrie R. Adapting to waiting lists for coronary revascularisation: do Canadian specialists agree onwhich patients come first? Chest 101 (3): 715-722, 1992440. Hanson R, Clifton-Smith B, Fasher B. Patient dissatisfaction in a paediatric emergency department. Journal of Quality inClinical Practice 14: 137-143, 1994441. Gittelsohn A, Powe N. Small area variations in health care delivery in Maryland. Health Services Research 30 (2): 295-317,1995442. McCelland A. In Fair Health? Equity and the Health System. National Health Strategy Background Paper No. 3.Commonwealth of Australia, 1991443. Harvey R. Making it Better: strategies for improving the effectiveness and quality of health services in Australia.Background paper No. 8. National Health Strategy. Commonwealth of Australia, 1991,444. Hospital services in Australia: access and financing. National Health Strategy Issues Paper No. 2. Commonwealth ofAustralia, 1991445. Health Department, Victoria. Equitable Regional Funding for Acute Hospitals: a discussion paper. Victorian Department ofHealth, 1986446. Mays L. Prioritising Elective Patients, AIHW Working paper 1994447. Mays L. and Gillett S. Uses of waiting list information, AIHW 1994448. Gillett S, Katauskas E. Waiting lists - a look at the literature, AIHW 1993449. Gillett S. and Mays L. Interpreting waiting list information, AIHW 1994450. Richardson B. Doctor supply utilisation and health outcomes: an economic analysis ASHSA University NSW School HealthAdministration, 1994

75451. Newton J and Goldacre M. How many patients are admitted in districts other than their own and why? Journal of PublicHealth Medicine 16 (2): 159-164, 1994452. Papadimitriou D, Mathur M and Hill D. A survey of rural road fatalities. Australian and New Zealand Journal of Surgery64: 479-483, 1994453. Keife C et al. Is cost a barrier to screening mammography for low income women receiving medicare benefits? Archives ofInternal Medicine 154: 1217-1224, 1994454. Skolnick A. Are there too many US transplantation centres? JAMA 271: 1062-3, 1994455. Blustein J. High technology cardiac procedures - the impact of service availability on service use in New York State. JAMA270: 344-349, 1993456. Wright R. Community oriented primary care. JAMA 269: 2544-2547, 1993457. Boyle F, Bell D. Breast cancer treatment outside teaching hospitals. Australian and New Zealand Journal of Surgery 22:309-310, 1994458. Yedidia M . The impact of social factors on the content of care. Archives of Internal Medicine 152: 595-600, 1992459. Del Mar C et al. Shared antenatal care in Brisbane. Australian and New Zealand Journal of Obstetrics and Gynaecology31: 305, 1991460. Weissman J et al. Delayed access to health care: risk factors, reasons and consequences. Annals of Internal Medicine 114:325-31, 1991461. Kellerman A. Too sick to wait. JAMA 266: 1123-24, 1991462. Nesbitt T, et al. Access to obstetric care in rural areas : effect on birth outcomes. American Journal of Public Health 80: 814,1990463. Stewart J et al. Major complications of coronary arteriography: the place of cardiac surgery. British Heart Journal 63: 74-77, 1993464. Thakker Y et al. Peadiatric inpatient utilisation in a district general hospital. Archives of Disease in Childhood 70: 488-92,1994465. Brownell M, Roos N. Variation in length of stay as a measure of efficiency in Manitoba hospitals. Canadian MedicalAssociation Journal 152: 675, 1995 *466. French JA et al. Effect of information about waiting lists on referral patients of general practitioners. British Journal ofGeneral Practice 40: 186-189, 1990467. Naylor CD et al. Waiting for coronary revascularisation in Toronto: 2 years’ experience with a regional referral office.Canadian Medical Association Journal 149: 955, 1993468. Mahon A et al. Factors that influence general practitioners’ choice of hospital when referring patients for elective hospital.British Journal of Practice 43: 272-6, 1993469. Nunez D. Inappropriate outpatient referrals increasing? British Journal of Clinical Practice 47: 73, 1993470. Ho E et al. Ontario patients’ acceptance of waiting times for knee replacements. Journal of Rheumatology 21: 2101, 1994*471. Baume P, Wolk J. Difficulties in admitting patients to metropolitan public hospitals: the view of medical practitioners. MJA163: 401, 1995472. Baume P. Waiting time for non-urgent specialist appointments. MJA 162: 648, 1995473. Duckett S. Casemix funding in Victoria; the first year. MJA 162: 650, 1995474. American Association of Family Physicians - Position statement on access to health care. American Family Physician 45:2797, 1992475. Bengston A et al. The epidemiology of a coronary waiting list. Journal of Internal Medicine 235: 263, 1994 *

76476. Roberts M et al. Time delays to lytic therapy and outcomes in 100 consecutive patients. European Heart Journal 15: 594-601, 1994477. Dos Santos L et al. Paediatric emergency department walkouts. Paediatric Emergency Care 10: 76, 1994478. Brown S et al. Management of minor head injuries in the accident and emergency department: the effect of an observationward. Journal of Accident and Emeregency Medicine 11: 144-148, 1994479. Tennyson Williams P. Generalist physicians in nonmetropolitan counties in Ohio. Archives of Family Medicine 3: 425-428,1994480. Allen D, Kamradt J. Relationship of infant mortality to the availability of obstetrical care in Indiana. Journal on FamilyPractice 33: 609-612, 1991481. Charlton J et al. Geographical variation in mortality from conditions amenable to medical intervention in England and Wales.Lancet 691-694, 1983482. Parchman M. Primary care physicians and avoidable hospitalisations. Journal on Family Practice 39: 123-128, 1994483. Metropolitan Hospitals Planning Board. Criteria to assess the performance of networks 1995484. National Health Ministers’ Benchmarking Working Group. First National Report on Health Sector Performance IndicatorsAIHW, Commonwealth of Australia, February 1996485. Victorian Department of Health and Community Services. Hospital Services Report, October 1995486. Montalto M, Dunt D. Delivery of traditional hospital services to patients at home. MJA 159 (4): 263-265, 1993487. Monalto M. Using referral letters to measure quality and performance in general practice. Journal of Quality in ClinicalPractice 15: 45-50, 1995488. Montalto M . Hospital in the home units: early issues for GP involvement. Australian Family Physician 24 (5): 797-801,May 1995489. Pirkis J, Montalto M. General practitioners in hospitals. Australian Family Physician 24 (6): 1026-29, 1995490. Short Form 12 (SF-12). A users’ manual. Medical Outcomes Trust, Boston, 1996 *491. Outcomes and PORTs [Editorial]. Lancet 340: 1439, 1992 *492. Developing a patient measurement system for the future: An interview with Eugene C. Nelson, DSc, MPH. Joint Commission Journalon Quality Improvement 19: 368-373, 1993.493. David Nash on the future of health care. Outcomes Measurement & Management 2-3, Jan/Feb 1993494. PORT-IIs told - if the usual study models won't work, create new ones. Outcomes Measurement & Management 5-6,November/December 1993.495. Clinical trials and clinical practice. Lancet 342: 877-878, 1993496. [Special issue on health care outcomes]. Critical Public Health 4 (4): 1993497. Limitations of cost-utility research. Outcomes Measurement & Management 6-7, July/August 1993 *498. Patient satisfaction, quality of care, and outcomes. Outcomes Measurement & Management 4-5, July/August 1993499. Strengthening a health outcomes approach for Victoria, 1993500. The management of hypertension: a consensus statement. MJA (Suppl): 1-16, 1994501. Disinterring the doctor's mistakes. Canberra Times 23 Mar, 1994502. Abstracts of papers, Canberra: International Network on Health Expectancy, 1994

77503. Improving survey measurement: health status transitions in a longitudinal list sample design. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 1994504. Australian Health Ministers’ Advisory Council. Sunshine Statement, 1993 (UnPub)505. Draft proposal to establish nursing home medical advisory panels towards securing high quality outcomes for elderly Australiansrequiring hostel or nursing home accommodation, 1991. (UnPub)506. Aaronson, NK, Meyerowtiz BE, Bard M and et al. Quality of Life research in oncology: past achievements and future priorities.Cancer 67: 839-843, 1991507. ACHS, Charter for Change, Zetland, NSW: Australian Council on Healthcare Standards, 1994508. ACHS Care Evaluation Program, Developing clinical indicators - the ACHS Care Evaluation Program, (UnPub)509. Aharony L. and Strasser S. Patient satisfaction: What we know about and what we still need to explore. Medical Care Review 50: 49-79, 1993 *510. AHCPR, Medical Treatment Effectiveness Program (MEDTEP) active research projects as of September 30, 1990. AHCPR ProgramNote, Rockville, MD:Agency for Health Care Policy and Research, 1990511. Alonso JA, Anto JP, Gonzalez M and et al. Measurement of general health status of non-oxygen-dependent chronic obstructivepulmonary disease patients. Medical Care 30 (Suppl): MS125-MS135, 1992512. Amadio PC. Outcomes measurements [Editorial]. The Journal of Bone and Joint Surgery 75-A: 1583-1584, 1993513. Anderson GF, Alonso J, Kohn LT and Black C. Analysing health outcomes through international comparisons. Medical Care 32: 526-534, 1994514. Anderson JJ and Chernoff MC. Sensitivity to change of rheumatoid arthritis clinical trial outcome measures. Journal of Rheumotology20 (3): 535-537, 1993515. Davies AR, Ware JE Jr. Involving consumers in quality of care assessment. Health Affairs (Millwood) 7 (1): 33-48, 1988516. Anderson JP, Kaplan RM, Berry CC et al. Interday reliability of function assessment for a health status measure: the quality of wellbeingscale. Medical Care 27 (11): 1076-1083, 1989517. Anderson J StC, Sullivan F and Usherwood TP. The medical outcomes study instrument (MOSI) - use of a new health status measurein Britain. Journal on Family Practice 7 (20): 5-218, 1990 *518. Andrews G, Hall. Goldstein G and et al. The economic costs of schizophrenia. Archives of General Psychiatry 42: 537-543, June 1985519. Anon, Information needs for health care quality assurance and outcomes, (UnPub). Personal communication: J. Ware Jnr520. Anon, Quality of life questionnaire, 1991. (UnPub). Personal communication: J. Ware Jnr521. Anon, Cross design synthesis: a new strategy for studying medical outcomes? Lancet 340: 944-946, 17 October 1992522. Avison ER, Verrier M and Wood-Dauphinee S. Rehabilitation outcome measures: Where do we stand? Canadian Journal of PublicHealth 83 (Suppl): S4-S6, 1992523. Ayanian JZ. Race, class, and the quality of medical care. JAMA 271: 1207-8, 1994 *524. Bailit HL. Use of effectiveness research in managed care plans. In: Effectiveness and outcomes in health care: Proceedings of aninvitational conference by the Institute of Medicine, edited by Heithoff, K. A. and Lohr, K. N. Washington, DC: National AcademyPress, 1990525. Baker R. Avedis Donabedian: an interview. Quality in Health Care 2: 40-46, 1993526. Balaban DJ, Sagi PC, Goldfarb NI and et al, Weights for scoring the quality of well-being instrumental among rheumatoid arthritics; acomparison to general population weights. Medical Care 24 (11): 973-980, November 1986527. Banta D. Developing outcome standards for quality assurance activities. Quality Assurance in Health Care 4 (1): 25-32, 1992

78528. Banta HD, Engel GL and Schersten T. Volume and outcome of organ transplantation. International Journal of Technology Assessmentin Health Care 8 (3): 490-505, 1992529. Bardsley M. and Cole J. Measured steps to outcomes. Health Services Journal 18-20, 17 October 1991530. Bardsley MJ and Coles JM. Practical experiences in auditing patient outcomes. Quality in Health Care 1: 124-130, 1992531. Barendregt JJ and Bonneux L. Changes in incidence and survival of cardiovascular disease and their impact on disease prevalence andhealth expectancy. Paper presented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February1994532. Barnum H. Evaluating healthy days of life gained from health projects. Social Science and Medicine 24 (10): 833-841, 1987533. Bartlett J and Cohen J. Building an accountable, improvable delivery system. Administration and Policy in Mental Health 21: 51-58,1993534. Battista RN. Discussion: Morris and Sherwood's "Quality of Life of Cancer Patients at Different Stages in the Disease Trajectory".Journal of Chronic Diseases 40 (6): 555-556, 1987535. Baue AE. Breast-conservation operations for treatment of cancer of the breast. JAMA 271: 1204-1205, 1994536. Baum FE and Cooke RD. Community health needs assessment: use of the Nottingham Health Profile in an Australian study. MJA 150:581-590, 1989537. Baume P. Of ends and means [letter]. MJA 155: 142-4, 1991538. Baume P. Goals, financing and outcomes: The pursuit of better health in Australia, 1991. (UnPub)539. Bellamy T, Newton JS, LeBaron NM and Horner RH. Quality of life and lifestyle outcomes: a challenge for residential programs. In:Quality of life: perspectives and issues, edited by Schalock RL and Begab MJ, Washington, DC: American Ass in Mental Retardation,1990540. Bendsten P and Hornquist JO. Severity of rheumatoid arthritis, function and quality of life: sub-group comparisons. Clinical andExperimental Rheumatology 11: 495-502, 1993541. Berg AO. Variations among family physician's management strategies for lower urinary tract infection in women: A report from theWashington Family Physicians Collaborative Research Network. Journal of the American Board of Family Practice 4: 327-330, 1991542. Berg KO, Wood-Dauphinee SL, Williams JI and Maki B. Measuring balance in the elderly: Validation of an instrument. CanadianJournal of Public Health 83: S7-S11, 1992543. Bergner M. Measurement of health status. Medical Care 23: 696-704, 1985544. Bergner M. Quality of life, health status, and clinical research. Medical Care 27 (Suppl): S148-S156, 1989545. Bergner M, Barry MJ, Bowman MA, Doyle MA, Guess HA and Nutting PA. Where do we go from here? Opportunities for applyinghealth status assessment measures in clinical settings. Medical Care 30 (Suppl): MS219-230, 1992546. Bergner M, Bobbitt RA, Carter WB and et al. The sickness impact profile: development and final revision of a health status measure.Medical Care 19: 787-805, 1981 *547. Bergner M, Bobbitt RA, Carter WB and Gilson BS. The Sickness Impact Profile: Development and final revision of a health statusmeasure. Medical Care 19: 787-805, 1981548. Bergner M, Bowman M, Doyle MA and et al. Where do we go from here? General audience discussion. Medical Care 30 (Suppl):MS231-9, 1992549. Berwick D M. Health services research and quality of care. Assignments for the 1990's. Medical Care 27 (8): 763-771, August 1989550. Berwick DM, Murphy JM, Goldman PA and et al. Performance fo a five-item mental health screening test. Medical Care 29: 169-176,1991551. Berwick DM and Weinstein MC. What do patients value? Willingness to pay for ultrasound in normal pregnancy. Medical Care 23: 7July: 881-893, 1985

79552. Best J. The matter of outcome. MJA 148: 650-652, 1988553. Bevan G. Quality-adjusted life years (qalys); an exposition and critique, 1988. (UnPub)554. Bion J. Outcomes in intensive care. BMJ 307: 953-954, 1993555. Blake RL and Vandiver TA. The reliability and validity of a ten-item measure of functional status. Journal of Family Practice 23 (5):455-459, 1986556. Blim RD. Strategies for improving and expanding the application of health status measures in clinical settings [Discussion]. MedicalCare 30 (Suppl.): MS196-7, 1992557. Bloom BS. Does it work? The outcomes of medical interventions. International Journal of Technology Assessment in Health Care 6:326-332, 1990558. Boers M and Tugwell P. The validity of pooled outcome measures (indices) in rheumatoid arthritis clinical trials. Journal ofRheumatology 20 (3): 568-574, 1993559. Bolduc M. For a conceptual model that better reflects the environment, 1992 pp1-20 (UnPub)560. Bombardier C, Ware J, Russell IJ and et al. Auranofin therapy and quality of life in patients with rheumatoid arthritis. Results of amulti-center trial. American Journal of Medicine 81: 4 October: 565-578, 1986561. Bond S and Thomas L. Input on outcomes. Nursing Times 88 (38): 50-52, 16-22 September 1992562. Bone M, Bebbington AC and Nicolaas G. Policy relevance and comparability problems of health expectancy indicators. Paperpresented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994563. Bone MR. International efforts to measure health expectancy. Journal of Epidemiology and Community Health 46: 555-558, 1992564. Borowitz M and Sheldon T. Controlling health care: from economic incentives to micro-clinical regulation. Health Economics 2: 201-204, 1994565. Borthwick-Duffy SA. Quality of life persons with severe or profound mental retardation. In: Quality of life: perspectives and issues,edited by Schalock RL and Begab MJ, Washington, DC: American Ass on Mental Retardation, 1990566. Boshuizen HC and van der Water HPA. Socio-economic differences in health expectancy in the Netherlands. Paper presented to the 7thMeeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994567. Bowling A. The conceptualization of functioning, health and quality of life. In: Measuring Health: A Review of Quality of LifeMeasurement Scales, Philadelphia: Open University Press: pp 1-11, 1991568. Bowling A. Measuring health: a review of quality of life measurement scales, Buckingham, England and Bristol Pa: Open UniversityPress: pp 1-11, 1991 *569. Boyd NF, Selby PJ, Sutherland HJ and et al. Measurement of the clinical status of patients with breast cancer: evidence for the validityof self assessment with linear analogue scales. Journal of Clinical Epidemiology 41 (3): 243-250, 1987570. Brazier J. The SF-36 health survey questionnaire: A tool for economists. Health Economics 2: 213-215, 1993 *571. Brazier J, Jones N and Kind P. A comparison of two health-status measures: Euroqol meets SG-36, 1993. (UnPub)572. Brazier JE, Harper R, Jones NMB and et al. Validating the SF-36 health survey questionnaire: new outcome measures in primarycare. BMJ 305: 160-164, 1992 *573. Brazier JE, Harper R, Joens NMB, O'Cathain A, Thomas KJ, Usherwood T and Westlake L. Validating the SF-36 health surveyquestionnaire: new outcome measure for primary care. BMJ 305: July 18: 160-164, 1992574. Brekke JS. An examination of the relationships among three outcome scales in schizophrenia. The Journal of Nervous and MentalDisease 180 (3): 162-167, 1993575. Cairns J and Schackley P. Sometimes sensitive, seldom specific: A review of the economics of screening. Health Economics 2: 43-53,1993

80576. Callahan CM, Drake BG, Heck DA and Dittus RS. Patient outcomes following tricompartmental total knee replacement. JAMA 271:1349-57, 1994577. Callahan D. Ethics and priority setting in Oregon. Health Affairs 78-87, Summer 1991578. Calman KC. Quality of life in cancer patients - an hypothesis. Journal of Medical Ethics 10: 124-127, 1984579. Cangialose CB. Outcomes research [letter]. NEJM 330: 435, 1994580. Carey RG, Rondinelli RD and Murphy JR. Functional Outcome Predictors. Archives of Physical Medicine and Rehabilitation 73:April: 400-401, 1992581. Carnes BA and Olshanky SJ. Evolutionary perspectives on human senescence. Population Development Review 1994582. Carr-Hill RA. The measurement of patient satisfaction. Journal of Public Health Medicine 14: 236-249, 1992583. Carr-Hill RA and Morris J. Current practice in obtaining the "Q" in QALYs: a cautionary note. BMJ 303 (6804): 699-701, 21September 1991 *584. Carter R. An overview of health-related quality of life measures - conceptual origins, applications and usefulness. (UnPub)585. Carter WB, Bobbitt RA, Bergner M. and et al. Validation of an interval scaling: the sickness impact profile. Health Services Research11: 516-528, 1976586. CASPE-HKS Ltd, Using occurrence screening in hospital-wide clinical audit, 1991. (UnPub)587. Cella DF. Overcoming difficulties in demonstrating health outcome benefits. Journal of Parenteral and Enteral Nutrition 16 (6Suppl): 1065-1115, Nov-Dec 1992588. Chambers LW, Haight M, Norman G and et al. Sensitivity to change and the effect of mode of administration on health statusmeasurement. Medical Care 25: 470-480, 1987589. Chambers LW, MacDonald LA, Tugwell P and et al. The McMaster Health Index Questionnaire as a measure of quality of life forpatients with rheumatoid disease. Journal of Rheumatology 9: 780-784, 1982590. Chambers LW, Sackett DL, Goldsmith CH and et al. Development and application of an index of social function. Health ServicesResearch 11: 430-441, 1976591. Chapman S. QALYs: a personal experience [letter]. Australian Journal of Public Health 17: 397, 1993592. Conn VS, Taylor SG and Casey B. Cardiac rehabilitation program participation and outcomes after myocardial infarction.Rehabilitation Nursing 17 (2): 58-62, Mar-Apr 1992593. Connelly JE, Philbrick JT, Smith GR, Kaiser DL and Wymer A. Health perceptions of primary care patients and the influence onhealth care utilization. Medical Care 27 (Suppl): S99-S109, 1989594. Conroy J and Feinstein CS. Measuring quality of life: Where have we been? Where are we going? In: Quality of life: perspectives andissues, edited by Schalock RL and Begab MJ, Washington, DC: American Ass on Mental Retardation, 1990595. Cook DJ, Guyatt GH, Juniper E and et al. Interviewer versus self-administered questionnaires in developing a disease-specific, healthrelatedquality of life instrument for asthma. Journal of Clinical Epidemiology 46: 529-34, 1993596. Coombs HC. The quality of life and its assessment, 1977. (UnPub)597. Cotton P. Doctors are asked: Does it work? JAMA 270: 153-8, 1993598. Cox DR, Fitzpatrick R, Fletcher AE, Gore SM, Spiegelhalter J and Jones DR. Quality of life assessment: can we keep it simple? J RStatist Soc 155: Part 3: 353-393, 1992599. Cribb A. Quality of life - a response to KC Calman. Journal of Medical Ethics 11: 142-145, 1985600. Crockett A, Moss J and Alpers J. The continuing impact of home oxygen therapy for respiratory patients on a hospital budget.Australian Health Review 15: 259-68, 1992

81601. Crockett AJ, Alpers JH and Moss JR. Home oxygen therapy: An audit of survival. Australian and New Zealand Journal of Medicine21: 217-21, 1991602. Culyer AJ. The morality of efficiency in health care: Some uncomfortable implications. Health Economics 1: 7-18, 1992603. Culyer AJ. Need, values and health status measurement. In: Economic Aspects of Health Services edited by Culyer AJ and Wright KG,pp 9-31, London: M. Roberston, 1978604. Cummins RA. The Comprehensive Quality of Life Scale - Intellectual Disability: An instrument under development. Australia andNew Zealand Journal of Developmental Disabilities 17: 259-64, 1991605. Cummins RA. Comprehensive Quality of Life Scale for Adults, 4th edition (ComQol-4), 1993 (UnPub)606. Cummins RA. Health promotion and the Comprehensive Quality of Life Scale. Health Promotion Journal of Australia 3: 46-47, 1994607. Cummins RA and Baxter C. A case for the inclusion of subjective quality of life data in service-delivery evaluations. In: Evaluation:Does it make a difference? Conference proceedings, Brisbane Hilton Hotel, 7-9 July, 1993, Brisbane: Australasian Evaluation Society,1993608. Cummins RA and Baxter C. Choice of outcome measures in service evaluations for people with an intellectual disability. EvaluationJournal of Australasia 1994 (In Press)609. Cummins RA, McCabe MP, Romeo Y and Gullone E. The Comprehensive Quality of Life Scale (ComQol): Instrument developmentand psychometric evaluation on college staff and students. Educational and Psychological Measurement (In Press)610. D'Arcy C, Holman J, Donovan RJ and Corti B. Evaluating projects funded by the Western Australian Health Promotion Foundation: asystematic approach. Health Promotion International 8: 199-208, 1993611. Dagher M and Lloyd RJ. Managing negative outcome by reducing variances in the emergency department. Quality Review Bulletin 15-21, January 1991612. Daniels N. Is the Oregon Rationing Plan fair? JAMA 265: 2232-2235, 1991613. Darke S. The measurement of outcome in opiate treatment evaluation studies. Drug and Alcohol Review 11: 169-174, 1992614. Das V. Cultural, social and behavioural determinants of health; what is the evidence? 1989 (UnPub)615. Dearing G. Measuring and managing outcomes improve quality and reduce costs. Outcomes Measurement & Management 3-5,May/June 1994616. DeFriese G. Measuring the effectiveness of medical interventions: new expectations of health services research [Editorial] HSR: HealthServices Research: 691-695, 1990617. Dempsey DT. Outcomes research in laparoscopic surgery: The National Laparoscopic Surgery Register. Journal of LaparoendoscopicSurgery 3: 375-378, 1993618. DesHarnais S, McMahon LF, Jr. and Wroblewski R. Measuring outcomes of hospital care using multiple risk-adjusted indexes.Health Services Research 26 (4): 425-445, Oct 1991619. DesHarnais SI, Chesney JD, Wroblewski RT and et al. The risk-adjusted mortality index. A new measure of hospital performance.Medical Care 26 (12): 1129-1148, 1988620. DesHarnais SI and Simpson KN. Indices for monitoring hospital outcomes in developed countries. Health Policy 21: 1-15, 1992621. DeVivo MJ, Rutt RD, Black KJ and et al. Trends in spinal cord injury demographics and treatment outcomes between 1973 and 1986.Archives of Physical Medicine and Rehabilitation 73 (5): 424-430, May 1992622. Deyo RA and Carter WB. Strategies for improving and expanding the application of health status measures in clinical settings.Medical Care 30 (Suppl): MS176-MS186, 1992623. Deyo RA, Cherkin D and Conrad D. The back pain outcome assessment team. Health Services Research 25: 5 December: 1990624. Deyo RA and Patrick DL. Barriers to the use of health status measures in clinical investigation, patient care, and policy research.Medical Care 27 (Suppl): S254-S268, 1989

82625. Dickey B and Azeni H. Impact of Managed Care on Mental Health Services. Health Affairs 197-204, Fall 1992626. Dixon J and Welch HG. Priority setting: lessons from oregon. Lancet 337 (8746): 891-894, 13 April 1991627. Donabedian A. Basic approaches to assessment: Structure, process and outcome. In: The definition of quality and approaches to itsassessment. Volume 1, Explorations in quality assessment and monitoring, Ann Arbor: Health Administration Press 9: 79-128, 1980628. Donabedian A. Quality assessment and assurance: unity of purpose, diversity of means. Inquiry 25: 173-192, Spring 1988629. Donabedian A. The role of outcomes in quality assessment and assurance. Quality Review Bulletin 356-60, November 1992630. Donabedian A. The quality of care: How can it be assessed? JAMA 260: 1743-8, 1988 *631. Donaldson C, Atkinson A, Bond J and et al. QALYs and long term care for elderly people in the UK: scales for assessment of qualityof life. Age Ageing 17: 379-387, 1988632. Donaldson C, Atkinson A, Bond J and et al. QALYs and long-term care for elderly people in the UK:Scales for assessment of quality oflife. International Journal of Social Psychiatry 34 (4): 243-247, Winter 1988633. Donovan JL, Frankel SJ and Eyles JD. Assessing the need for health status measures. Journal of Epidemiology and Community Health47: 158-162, 1993634. Donovan K, Sanson-Fisher RW and Redman S. Measuring quality of life in cancer patients. Journal of Clinical Oncology 7: 7 July:959-968, 1989635. Dougherty CJ. Setting health care priorities. Hastings Center Report 21 (3): 1-10, 1991636. Dowie J. A short and slightly impolite paper about health status and health service outcome measurement, 1991 (UnPub)637. Dowie J. Equity, efficiency and ethics. In: The ethics of allocating health resources. Report of a conference at Westmead Hospital,February 15-16, 1991, Sydney: NSW Health, 1991638. Dowie J. The PIQALI: the appropriate way to deal with 'process utility' in health service evaluation, 1993. (UnPub)639. Dowie J. 'Process utility' can seriously damage your health service evaluation but the generic measure of benefit should include 'serviceoutcomes', 1993. (UnPub)640. Dowie JA. Valuing the benefits of health improvement. Australian Economic Papers 21-41, June 1970641. Drummond M, Brandt A, Luce B and et al. Standardizing economic evaluation methodologies in health care: practice, problems andpotential. International Journal of Technology Assessment in Health Care 1991642. Drummond M and O'Brien B. Clinical importance, statistical significance and the assessment of economics and quality-of-lifeoutcomes. Health Economics 2: 205-212, 1993643. Drummond MF. Discussion: Torrance's "Utility approach to measuring health-related quality of life." Journal of Chronic Diseases 40(6): 601-603, 1987644. Dugdale P. The NSW Health Outcomes Initiative in Context, 1992. (UnPub)645. Duncan PG, Cohen MM, Tweed WA and et al. The Canadian four-centre study of anaesthetic outcomes: III. Are anaestheticcomplications predictable in day surgical practice? Canadian Journal of Anaesthesia 39 (5): 440-448, 1992646. Early Breast Cancer Trialists' Collaborative Group, Systematic treatment of early breast cancer by hormonal, cytotoxic, or immunetherapy. Lancet 339: 1-14-71-85, 1992647. Edgerton RB. Quality of life from a longitudinal research perspective. In: Quality of Life: Perspectives and Issues, edited by SchalockRL and Begab MJ. Washington, DC: American Ass on Mental Retardation, 1990648. Ellis BH, Jr, McCan I, Price G and et al. The New Mexico treatment outcome study: evaluating the utility of existing informationsystems. Journal of Health Care for the Poor and Underserved 3 (1): 138-150, Summer 1992649. Ellwood PM. Shattuck Lecture - outcomes management. A technology of patient experience. NEJM 318 (23): 1549-1556, June 9, 1988

83650. Endicott J, Nee J, Harrison W and Blumenthal R. Quality of Life Enjoyment and Satisfaction Questionnaire: A new measure.Psychopharmacology Bulletin 29: 321-6, 1993651. Engel W, Freund DA, Stein JS and Fletcher RH. The treatment of patients with asthma by specialists and generalists. Medical Care27: 306-12, 1989.652. Epstein AM. The outcomes movement: will it get us where we want to go? NEJM 323: 266-269, 1990653. Epstein AM, Hall JA, Tognetti J, Son LH and Conant L. Using proxies to evaluate quality of life. Medical Care 27 (3 Suppl): S91-S98,1989654. Erickson P, Kendall EA, Anderson JP and et al. Using composite health status measures to assess the nation's health. Medical Care 27(3 Suppl): S66-S77, 1989655. Essink-Bot M, Stouthard MEA and Bonsel GJ. Generalizability of valuations on health states collected with the EuroQolc-Questionnaire. Health Economics 2: 237-246, 1993656. Essink-Bot ML, Bonsel GJ and Van der Maas PJ. Valuation of health states by the general public: feasibility of a standardizedmeasurement procedure. Social Science and Medicine 31 (11): 1201-1206, 1990657. Evans I. The challenge of breast cancer [Lancet Conference]. Lancet 343: 1085-6, 1994658. Evans RG. Tension, compression and shear: directions, stresses and outcomes of health care cost control. Journal of Health Politics,Politics, Policy and Law 15 (1): 101-128, Spring 1990659. Faden R and Leplege A. Assessing quality of life. Medical Care 30 (Suppl): MS166-MS175, 1992660. Falloon IRH. Early intervention for first episodes of schizophrenia: A preliminary exploration. Psychiatry 55: 4-15, 1992661. Fallowfield LJ, Hall A, Maguire GP and Baum M. Psychological outcomes of different treatment policies in women with early breastcancer outside of clinical trial. BMJ 301: 575-80, 1990662. Fanshel S and Bush JW. A health status index and its application to health services outcomes. Operations Research 18: 1021-1066,1970663. Feeny DH and Torrance GW. Incorporating utility-based quality-of-life assessment measures in clinical trials: Two examples. MedicalCare 27 (Suppl): S190-S204, 1989664. Feinstein AR. Benefits and obstacles for development of health status assessment measures in clinical settings. Medical Care 30:MS50-MS56, 1992665. Ferrans CE and Powers MJ. Quality of life of hemodialysis patients. ANNA Journal 20: 575-81, 1993666. Ferraris VA and Propp ME. Outcome in critical care patients: a multivariate study. Critical Care Medicine 20 (7): 967-976, July 1992667. Fisher B, Costantino J, Redmond C and et al. Lumpectomy compared with lumpectomy and radiation therapy for the treatment ofintraductal breast cancer. NEJM 328: 1581-1586, 1993668. Fisher ES and Wennberg JE. Administrative data in effectiveness studies: the prostatectomy assessment. In: Effectiveness andOutcomes in Health Care, edited by Heithoff KA and Lohr KN. Washington, DC: National Academy Press, 1990669. Fitzpatrick R. Patient satisfaction and quality of life measures. In: Outcome measures in orthopaedics, edited by Pynsent PB, FairbankJCT and Carr A. Oxford: Butterworth, 1993670. Fitzpatrick R, Fletcher A, Gore S. and et al. Quality of life measures in health care. I: applications and issues in assessment. BMJ 305(6861): 1074-1077, 31 October 1992671. Fletcher A, Jones D, Cox D and et al. Quality of life measures in health care II: design, analysis and interpretation. BMJ 305 (6862):1145-1148, 7 November 1992672. Fletcher AE and Bulpitt CJ. Measurement of quality of life in clinical trials of therapy. Cardiology 75 (Suppl) 1: 41-52, 1988673. Flood AB, Lorence DP, Ding J, McPherson K and Black NA. The role of expectations in patient's reports of post-operative outcomesand improvement following therapy. Medical Care 31: 1043-56, 1993

84674. Fowler FJ, Wennberg JE, Timothy RP and et al. Symptom status and quality of life following prostatectomy. Journal of the AmericanMedical Record Association 259: 3018-3022, 1988675. Fox DM and Leichter HM. Rationing care in Oregon: The new accountability. Health Affairs 7-27, Summer 1991676. Fox DM and Leichter HM. The ups and downs or Oregon's rationing plan. Health Affairs 66-70, Summer 1993677. Frater A. Meeting report: Health Outcomes. Health Services Journal conference, London, November 1992. Health Services Journal 69678. Frater A and Costain D. Any better? Outcome measures in medical audit. BMJ 304 (6826): 519-520, 29 February 1992679. Freund D and Harvey R. Making it better. MJA 155: 583-4, 1991680. Freund DA, Dittus RS, Fitzgerald J and et al. Assessing and improving outcomes: total knee replacement. Health Services Research 25(5): 723-726, December 1990681. Freund DA, Stein J, Hurley R, Engel W, Woomert A and Lee B. The Kansas City Asthma Care Project: specialty differences in thecost of treating asthma. Annals of Allergy 60: 3-7, 1988682. Freund O. National survey of total knee replacement, 1991 (UnPub)683. Fried BJ, Boers M and Baker PRA. A method for achieving consensus on rheumatoid arthritis outcome measures: the OMERACTConference Process. Journal of Rheumatology 20 (3): 548-551, 1993684. Fries JF. The hierarchy of outcome assessment. Journal of Rheumatology 20 (3): 546-547, 1993685. Fries JF and Ramey DR. Platonic outcomes. Journal of Rheumatology 20 (3): 415-417, 1993686. Froberg DG and Kane RL. Methodology for measuring health-state preferences - II: scaling methods. Journal of ClinicalEpidemiology 42 (5): 459-471, 1989687. Froberg DG and Kane RL. Methodology for measuring health-state preferences - III: population and context effects. Journal ofClinical Epidemiology 42 (6): 585-592, 1989688. Froberg DG and Kane RL. Methodology for measuring health-state preferences - I: measurement strategies. Journal of ClinicalEpidemiology 42 (4): 345-354, 1989689. Froberg DG and Kane RL. Methodology for measuring health-state preferences - IV: progress and a research agenda. Journal ofClinical Epidemiology 42 (7): 675-685, 1989690. Frommer M, Rubin G and Lyle D. The NSW Health Outcomes Program. NSW Public Health Bulletin 3: 136-137691. Frostick SP and Hunter JB. Complications. In: Outcome measures in orthopaedics, edited by Pynsent PB, Fairbank JCT and Carr A.Oxford: Butterworths, 1993692. Furlong W, Feeny D, Torrance GW and et al. Guide to design and development of health-state utility instrumentation, 1990. (UnPub)693. Gafni A. The quality of QALYs (quality-adjusted-life-years):do QALYs measure what they at least intend to measure? Health Policy13: 81-83, 1989694. Gafni A and Birch S. Equity considerations in utility-based measures of health outcomes in economic appraisals: an adjustmentalgorithm, 1990 (UnPub)695. Ganiats TG, Palinksa LA and Kaplan RM. Comparison of Quality of Well-Being Scale and Functional Status Index in patients withatrial fibrillation. Medical Care 30: 958-964, 1992696. Garratt A, Ruta DA, Abdalla MI, Buckingham JK and Russell IT. The SF-36 health survey questionnaire: an outcome measure suitablefor use within the NHS? BMJ 306: 1440-1444, 1993697. Garratt AM, Macdonald LM, Ruta DA and et al. Towards measurement of outcome for patients with varicose veins. Quality in HealthCare 2: 5-10, 1993698. Gaut B. Ethics, distribution and Oregon. MJA 159 (4): 280-281, 16 August 1993699. Geigle R and Jones SB. Outcomes measurement: A report from the front. Inquiry 27: 7-13, 1990

85700. Gelber RD and Gelman RS. A quality of life oriented endpoint for comparing therapies. Biometrics 45: September: 781-795, 1989701. Gerard K, Dobson M and Hall J. Framing and labelling effects in health descriptions: Quality adjusted life years for treatment of breastcancer. Journal of Clinical Epidemiology 46: 77-84, 1993702. Gerard K and Mooney G. QALY league tables: Handle with care. Health Economics 2: 59-64, 1993703. Gerrity MS, Gaylord S and Williams ME. Short versions of the Timed manual Performance Test. Medical Care 31: 617-628, 1993704. Gillam L. Resource allocation: the unavoidable debate (Part 1). Bioethics News 11: 49-57, 1992705. Goldberg HI, Pantell RH and Weber JR. Finale Panel: Reactions, reflections and predictions. Medical Care 30 (Suppl): MS283-MS293, 1992706. Goldberg J, Eisen SA, True WR and Henderson WG. Health effects of military service: Lessons learned from the Vietnam experience.AEP 2: 841-853, 1992707. Golden WE. Health status measurement: implementation strategies. Medical Care 30 (Suppl): MS187-MS195, 1992708. Goldsmith CH, Boers M, Bombardier C and et al. Criteria for clinically important changes in outcomes: development, scoring andevaluation of rheumatoid arthritis patient and trial profiles. Journal of Rheumotology 20 (3): 561-565, 1993709. Golenski JD and Thompson SM. A history of Oregon's basic health services act: an insider's account. Quality Review Bulletin 144-149,May 1991710. Gomez CR. Outcomes of carotid endarterectomy. Annals of Internal Medicine 114 (8): 703-704, 15 April 1991711. Goodwin A. The NSW health outcomes initiative. NSW Public Health Bulletin 3: 25-26, 1992712. Gornick M, Lubitz J and Riley GUS. Initiatives and approaches for outcomes and effectiveness research. Health Policy 17: 209-225,1991713. Gough I and Thomas T. Need satisfaction and welfare outcomes: Theory and explanations. Social Policy and Administration 28: 33-56, 1994714. Gouveia W A. Measuring and managing patient outcomes. American Journal of Hospital Pharmacy 49 (9): 2157-2158, September1992715. Graetz B. Health consequences of employment and unemployment: longitudinal evidence for young men and women. Social Scienceand Medicine 36 (6): 715-724, 1993716. Greco P and Eisenberg JM. Changing physicians’ practices. NEJM 329: 1271-1274, 1993717. Greenfield S. The state of outcome research: are we on target? NEJM 320: 1142-1143, 1989 *718. Greenfield S and Nelson EC. Recent developments and future issues in the use of health status assessment measures in clinicalsettings. Medical Care 30 (Suppl): MS23-MS41, 1992719. Greenfield S, Nelson EC, Zubkoff M and et al. Variations in resource utilization among medical specialties and systems of care.Results from the Medical Outcomes Study. Journal of the American Research Association 267 (12): 1624-1630, 25 March 1992720. Greenough G and Fraser RD. Assessment of outcome in patients with low-back pain. Spine 17 (1): 36-41, 1992721. Grembowski D, Patrick D, Diehr P, Durham M, Beresford S, Kay E and Hecht J. Self-efficacy and health behaviour among olderadults. Journal of Health and Social Behaviour 34: 89-104, 1993722. Grill, R, Mainini F, Penna A and et al. Inappropriate Halsted mastectomy and patient volume in Italian hospitals. American Journal ofPublic Health 83: 1762-4, 1993723. Grisso J. Making comparisons. Lancet 342: 157-60, 1993724. Gross PF. Health indicator conceptualization, measurement and data collection: a state-of-the-art review, 1977. (UnPub)

86725. Grotvedt L and Viksand G. Life expectancy without disease and disability in Norway. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 1994726. Gudex C. QALYs and their use by the Health Service, 1986. (UnPub)727. Gudex C and Kind P. The QALY toolkit, 1987. (UnPub)728. Guillemin F, Bonbardier C and Beaton D. Cross-cultural adaption of health-related quality of life measures: Literature review andproposed guidelines. Journal of Clinical Epidemiology 46: 1417-1432, 1993729. Gulliford MC. Evaluating prognostic factors: Implications for measurement of health care outcome. Journal of Epidemiology andCommunity Health 46: 323-326, 1992730. Gulliford MC, Barton JR and Bourne HM. Selection for oesophagectomy and postoperative outcome in a defined population. Qualityin Health Care 2:17-20, 1993731. Guyatt GH, Eagle DJ, Sachett B and et al. Measuring quality of life in the frail elderly. Journal of Clinical Epidemiology 46: 1433-1444, 1993732. Guyatt GH, Feeny DH and Patrick DL. Measuring health-related quality of life. Annals of Internal Medicine 118: 622-629, 1993733. Guyatt GH, Sander JO, Veldhuyzen van Zanten SJO and et al. Measuring quality of life in clinical trials: a taxonomy and review.Canadian Medical Association Journal 140 (12): 1441-1448, 15 June 1989734. Haas M and Hall J. The Oregon Plan. NSW Public Health Bulletin 3: 5-1, 1992735. Hadorn DC. The Oregon priority-setting exercise. Hastings Center Report 21 (3): 11-16, 1991736. Haines A and Feder G. Guidance on guidelines. Writing them is easier than making them work. BMJ 305 (6857): 785-786, 3 October1992737. Hall J. Quality vs. quantity. NSW Public Health Bulletin 2: 115-116, 1994738. Hall J, Hall N, Fisher E and et al. Measuring outcomes of general practice, (UnPub)739. Hall J and Masters G. Measuring outcomes of health services: a review of some available measures. Community Health Studies 10 (2):147-155, 1986740. Hall J, Shiell A and CHERE, Health Outcomes: A Health Economics Perspective, Sydney: Centre for Health Economics Research andEvaluation, 1993 *741. Hall JA, Epstein AM and McNeil BJ. Multidimensionality of health status in an elderly population: Construct validity of ameasurement battery. Medical Care 27 (Suppl): S168-S177, 1989742. Hall JA, Milburn MA and Epstein AM. A causal model of health status and satisfaction with medical care. Medical Care 31: 84-94,1993743. Hall W, Andrews G and Goldstein G. The costs of Schizophrenia. Australian and New Zealand Journal of Psychiatry 19: 3-5, 1985.744. Hall W, Goldstein G, Andrews G and et al. Estimating the economic costs of Schizophrenia. Schizophrenia Bulletin 11 (4): 598-610,1985745. Hankey GJ, Dennis MS, Slattery JM and Warlow CP. Why is the outcome of transient ischaemic attacks different in different groups ofpatients? BMJ 306: 1107-1111, 1993746. Harada N, Sofaer S and Kominski G. Functional status outcomes in rehabilitation. Medical Care 31: 345-57, 1993747. Harris J. QALYfing the value of life. Journal of Medical Ethics 13: 117-123, 1987748. Harris L and Richardson J. Pressure for health care reform. MJA 160: 463-5, 1994749. Harris MD. Outcomes of care from patient's perspectives. Home Health Care 10 (3): 64-68, 1992750. Harrison RI, Glenn DC, Niesche FW and et al. Surgical management of breast cancer: Experience of the Central Sydney Area HealthService Breast X-ray Programme, 1988 - 1991. MJA 160: 617-20, 1994

87751. Hart PM and Wearing AJ. Problems of stability and change in quality of life research: Implications for a longitudinal model ofpersonality, health and well-being. Paper presented at the First Conference of the Australian Centre for Social Research, Launceston,January 25-27, 1994752. Harwood RH, Gompertz PH and Ebrahim S. Outcome measures should be relevant. BMJ 304 (6831): 917-918, 4 April 1992753. Hasnain R and Garland M. Health care in common, 1990. (UnPub)754. Hays LB. The Health Care Financing Administration and the Effectiveness Initiative. In: Effectiveness and Outcomes in Health Care,edited by Heithoff KA and Lohr KN. Washington, DC: National Academy Press, 1990755. Hays RD, Sherbourne CD and Mazel RM. The Rand 36-Item Health Survey 1.0. Health Economics 2: 217-227, 1993756. Hayward MD, Crimmins EM and Friedman S. Cognitive functioning changes among the chronically impaired elderly in the UnitedStates. Paper presented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994757. Heal LW and Sigelman CK. Methodological issues in measuring the quality of life of individuals with mental retardation. In: Qualityof life: perspectives and issues, edited by Schalock RL and Begab MJ. Washington, DC: American Ass on Mental Retardation, 1990758. Health Services Research Group, Outcomes and the management of health care. Canadian Medical Association Journal 147: 1775-80,1992759. Heap MJ, Munglani R, Klinck JR and Males AG. Elderly patients' preferences concerning life-support treatment. Anaesthesia 48:1027-1033, 1993760. Heathcote CR and McDermid IM. Projections of cohort life expectancy based on weighted least squares method. Paper presented tothe 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994761. Heithoff KA, Lohr KN and Rettig RA. Genesis of the Effectiveness Initiative and IOM's role. In: Effectiveness and outcomes in healthcare: Proceedings of an invitational conference by the Institute of Medicine, edited by Heithoff KA. and Lohr KN. Washington, DC:National Academy Press, 1990762. Higgins M, McCaughan D, Griffiths M and et al. Assessing the outcomes of nursing care. Journal of Advanced Nursing 17: 561-568,1992763. Hirst C, Furnival CM, Fielding GD and Porter AJ. The detection of early breast cancer: three-year results from a diagnostic breastclinic. MJA147: 328-30, 1987764. Hollandsworth JG. Evaluating the impact of medical treatment on the quality of life: a 5 year update. Social Science and Medicine 26(4): 425-434, 1988765. Holzemer WL, Bakken Henry S, Stewart A and Janson-Bjerklie S. The HIV quality audit marker (HIV-QAM): an outcome measurefor hospitalized AIDS patients. Quality of Life Research 2: 99-107, 1993766. Hopkins A. Measuring the quality of medical care, London: Royal College of Physicians, 1990 *767. Hopkins A, Fitzpatrick R, Foster A and et al. What do we mean by appropriate health care? Quality in Health Care 2: 117-123, 1993768. Horton R. Errors admitted over falsified US cancer data. Lancet 343: 1029, 1994769. Horvath DG. The ethics of medical resource allocation, Canberra: Australian Health Ethics Committee, 1993770. Houston MC. New insights and new approaches for the treatment of essential hypertension: selection of therapy based on coronaryheart disease risk factor analysis, hemodynamic profiles, quality of life and subsets of hypertension. American Heart Journal 117: 4April: 911-950, 1989771. Hugh TB, Chen FC, Hugh TJ and et al. Laparoscopic cholecystectomy. A prospective study of outcome in 100 unselected patients.MJA 156 (5): 318-320, 2 March 1992772. Human rights and Equal Opportunity Commission, Human rights and mental illness: Report of the National Inquiry into the HumanRights of People with Mental Illness, Canberra: AGPS, 1993773. Hume AL. Applying quality of life data in practice. Considerations for antihypertensive therapy. Journal of Family Practice 28 (4):403-411, 1989

88774. Hunt SM, McEwen J and McKenna SP. Measuring Health Status, London:Croom Helm, 1986775. Hunt SM, McKenna SP, McEwen J and et al. The Nottingham Health Profile: subjective health status and medical consultations.Social Science and Medicine 15A: 221-229, 1981776. Hurley SF. Screening: the need for a population register. MJA 153: 310-311, 1990777. Hyland ME. The validity of health assessments: Revolving some recent differences. Journal of Clinical Epidemiology 46: 1019-1023,1993778. Iezzoni LI. Monitoring quality of care: What do we need to know? Inquiry 30: 112-114, 1993779. IOM Core Committee, Promise and limitations of effectiveness and outcomes research. In: Effectiveness and outcomes in health care:Proceedings of an invitational conference by the Institute of Medicine, edited by Heithoff KA and Lohr KN. Washington, DC: NationalAcademy Press, 1990780. Irwig L. An approach to evaluating health outcomes. Public Health Bulletin 4 (12): 135-136, 1993781. Irwig L, Tosteson ANA, Gastonis C and et al. Guidelines for meta-analyses evaluating diagnostic tests. Annals of Internal Medicine120: 667-676, 1994782. Jadad AR and McQuay HJ. The measurement of pain. In: Outcome measures in orthopaedics, edited by Pynsent PB, Fairbank JCT andCarr A. Oxford: Butterworth, 1993783. Jagger C and Clarke M. Annual screening of the elderly in the UK: a data source for health life expectancy calculations? Paperpresented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994784. Jencks SF. Issues in the use of large data bases for effectiveness research, In: Effectiveness and outcomes in health care, edited byHeithoff KA and Lohr KN. Washington, DC: National Academy Press, 1990 *785. Jenkins CD, Jono RT, Stanton BA and et al. The measurement of health-related quality of life: major dimensions identified by factoranalysis. Social Science and Medicine 31 (8): 925-931, 1990786. Jenkinson C. Why are we weighting? A critical examination of the use of item weights in a health status measure. Social Science andMedicine 32 (12): 1413-1416, 1991787. Jenkinson C, Coulter A, and Wright L. Short form 36(SF-36) health survey questionnaire: normative data for adults of working age.BMJ 306: 1437-1440, 1993788. Jenkinson C, Fitzpatrick R and Argyle M. The Nottingham Health Profile: an analysis of its sensitivity in differentiating illness groups.Social Science and Medicine 27: 1411-1414, 1988789. Johnson EW. Outcome measures.....a Chimera? American Journal of Physical Medicine and Rehabilitation 71 (4): 201, 1992790. Johnson J, Weissman MM and Klerman GL. Service utilization and social morbidity associated with depressive symptoms in thecommunity. JAMA 267: 1478-83, 1992791. Johnson SH. Nursing outcomes: focusing on results. Dimensions of Critical Care Nursing 11 (1): 4-5, January-February 1992792. Johnston ME, Langton KB, Haynes RB and Mathieu A. Effects of computer-based clinical decision support systems on clinicianperformance and patient outcome. Annals of Internal Medicine 120 (2): 135-142, 1994793. Johnston RC, Fitzgerald RH, Harris WH, Muller ME and Sledge CB. Clinical and radiographic evaluation of total hip replacement.The Journal of Bone and Joint Surgery 72-A: 161-168, 1990794. Juniper EF, Guyatt GH, Willan A and Griffith LE. Determining a minimal important change in a disease-specific quality of lifequestionnaire. Journal of Clinical Epidemiology 47: 81-87, 1994795. Kahn CN. III Policy implications of outcomes research. International Journal of Technology Assessment in Health Care 6: 295-296,1990796. Kane RL and Lurie N. Appropriate effectiveness: a tale of carts and horses. Quality Review Bulletin 322-326, October 1992

89797. Kanouse DE and Jacoby I. When does information change practitioners' behaviour? International Journal of Technology Assessmentin Health Care 4: 27-33, 1988798. Kantz ME, Harris W, Levitsky K, Ware JE and Ross Davies A. Methods for assessing condition-specific and generic functional statusoutcomes after total knee replacement. Medical Care 30 (Suppl): MS240-MS252, 1992799. Kaplan RM. Health outcome models for policy analysis. Health Psychology 8 (6): 723-735, 1989800. Kaplan RM. Behaviour as the central outcome in health care. American Psychologist 45 (11): 1211-1220, November 1990801. Kaplan RM. Quality of life assessment for health resource allocation. Paper presented to the Harkness Health Conference, Canberra, 8-9 December, 1993802. Kaplan RM and Anderson JP. A general health policy model: update and applications. Health Services Research 23: 2 June: 203-235,1988803. Kaplan RM. and Anderson JP. The general health policy model: an integrated approach. In: Quality of Life Assessments in ClinicalTrials, edited by Spilker B. New York: Raven Press Ltd, pp 131-149, 1990804. Kaplan RM, and Anderson JP. A general health policy model: applications of new public health indicators in studies of aging. In:Reducing Frailty and Falls in Older Persons, edited by Weindruch R, Hadley E, Ory M and et al, pp 385-411, 1991805. Kaplan RM, Anderson JP, Wu AW. and et al. The quality of well-being scale. Applications in AIDS, Cystic Fibrosis and Arthritis.Medical Care 27 (3) (Suppl): S27-S43, March 1989806. Kaplan RM, Bush JW and Berry CC. Health status: types of validity and the index of well-being. Health Services Research 11: 478-507, Winter 1976807. Kaplan RM and Ernst JA. Do category rating scales produce biased preference weights for a health index? Medical Care 21 (2): 193-207, February 1983808. Kaplan RM and Hartwell SL. Differential effects of social support and social network on physiological and social outcomes in men andwomen with type II diabetes mellitus. Health Psychology 6 (5): 387-398, 1987809. Kaplan SH, Greenfield S and Ware JE. Assessing the effects of physician-patient interactions on the outcomes of chronic disease.Medical Care 27 (Suppl): S110-S127, 1989810. Kassirer JP. The quality of care and the quality of measuring it. NEJM 329 (17): 1263-1265, 1993 *811. Kaufmann MA, Buchmann B, Scheidegger D and et al. Severe head injury: should expected outcome influence resuscitation and firstdaydecisions? Resuscitation 23: 199-206, 1992812. Kazis LE, Anderson JJ and Meenan RF. Effect sizes for interpreting changes in health status. Medical Care 27 (Suppl): S178-S189,1989813. Keeler EB, Manning WG and Wells KB. The demand for episodes of mental health services. Journal of Health Economics 7: 369-392, 1988814. Kerridge R, Glasziou PP and Hillman KM. The use of 'Quality-adjusted life years' (QALYs) to evaluate outcome and economicjustification of treatment in intensive care, 1992. (UnPub)815. Kettlewell MGW and et al. Surgical review in the 80's: measuring surgical outcomes, (UnPub)816. Kind P. The designs and construction of quality of life measures, (UnPub)817. Kind O and Gudex CM. Measuring health status in the community: a comparison of methods. Journal of Epidemiology andCommunity Health 48: 86-91, 1994818. Kind P, Roser R and Williams A. Valuation of Quality of Life: Some psychometric evidence. In: The Value of Life and Safety, edited byJones-Lee MW. North-Holland Publishing Company, pp 159-170, 1982819. Kirwan JR. Outcome measures in rheumatoid arthritis clinical trials: assessing improvement. Journal of Rheumatology 20 (3): 543-545, 1993820. Kitzhaber JA. The Oregon Basic Health Services Act, 1990. (UnPub)

90821. Kitzhaber JA. Summary: the health services prioritization process, 1990 (UnPub)822. Kitzbaher JA. A healthier approach to health care. Issues in Science and Technology 7 (2): 59-65, 1991823. Kitzhaber JA. Constructive debate in a real world. Health Management Quarterly First Quarter: 1991824. Kitzhaber JA. Prioritising health services in an era of limits: the Oregon experience. BMJ 307: 373-377, 1993 *825. Mant J. and Hicks N. Health status measurement and assessment of medical care. International Journal for Quality in Health Care 8:107-110, 1996826. Klapow JC, Slater MA, Patterson TL, Doctor JN, Atkinson JH and Garfin SR. An empirical evaluation of multidimensional clinicaloutcome in chronic low back pain patients. Pain 55: 107-118, 1993827. Klein R. Warning signals from Oregon. BMJ 304: 1457-1458, 1992828. Klerman G, Olfson M, Leon A and Weissman MM. Measuring the need for mental health care. Health Affairs 24-33, Fall 1992829. Klingman D, Pine PL and Simon J. Outcomes of surgery under Medicaid. Health Care Financing Review 11: 1-16, 1990830. Knippenberg FCEV and de Haes JCJM. Measuring the quality of life of cancer patients: psychometric properties of instruments.Journal of Clinical Epidemiology 41 (11): 1043-1053, 1988831. Koska MT. Outcomes research: hospitals face confidentiality concerns. Hospitals 66 (1): 32-34, 5 January 1992832. Krefting L, Warren S and Grace M. Measuring long-term outcome after traumatic brain injury. Canadian Journal of Public Health 83(Suppl): S64-S68, 1992833. Kuhse H and Singer P. The quality/quantity-of-life distinction and its moral importance for nurses. International Journal of NursingStudies 26 (3): 203-212, 1989834. Kurtin PS, Ross Davies A, Meyer KB, DeGiacomo JM and Kantz ME. Patient-based health status measures in outpatient dialysis.Medical Care 30 (Suppl): MS136-MS149, 1992835. LaPuma J and Lawlor EF. Quality-adjusted life-years: Ethical implications for physicians and policy makers. Journal of the AmericanMedical Record Association 263: 21 June 6: 2917-2921, 1990836. Laffel G and Blumenthal D. The case for using industrial quality management science in health care organisations. JAMA 262: 2869-2873, 1989 *837. Laffel GL, Barnett AI, Finkelstein S and Kaye MP. The relation between experience and outcome in heart transplantation. NEJM 327(17): 1220-1225, October 1992 *838. Landry FJ, Parker JM and Phillips YY. Outcome of cardiopulmonary resuscitation in the intensive care setting. Archives of InternalMedicine 152 (11): 2305-2308, November 1992839. Lansky D, Butler JBV and Waller FT. Using health status measures in the hospital setting: from acute care to "outcomes management".Medical Care 30: MS57-73, 1992 *840. Ledesert B, Ritchie K and Touchon J. Disability due to dementia. Paper presented to the 7th Meeting of the International Network onHealth Expectancy, Canberra, 23-25 February, 1994841. Lee PP and Javitt JC. Measuring the benefit and value of services. Archives of Ophthalmology 112: 32, 1994842. Leeder S. Valuable health:what do we want, and how do we get it? Australian Journal of Public Health 16: 6-14, 1992 *843. Leeder S. The recent Medicare agreement, goals and targets, and health outcomes. Australian Journal of Public Health 17: 87-88,1993844. Leeder S and Lazarus R. Clinical applications of epidemiology. NSW Public Health Bulletin 3: 1-2, 1992845. Lehman AF, Postrado LT, Roth D and et al. Continuity of care and client outcomes in the Robert Wood Johnson Foundation Programon chronic mental illness. Milbank Quarterly 72: 105-22, 1994

91846. Leigh JP. Gender, firm size, industry and estimates of the value-of-life. Journal of Health Economics 6: 255-273, 1987847. Leighton Read J, Quinn RJ and Hoefer MA. Measuring overall health: an evaluation of the three important approaches. Journal ofChronic Diseases 40: 7S-21S, 1987848. Lemay P. Quality of life - measuring outcomes of pharmaceutical management. Summary of workshop proceedings. Canadian Journalof Public Health 83 (3): s5-s16, May-June 1992849. Lena HF and London B. The political and economic determinants of health outcomes: a cross-national analysis. International Journalof Health Services 23: 585-602, 1993850. Leu RE. Economic evaluation of new drug therapies in terms of improved life quality. Social Science and Medicine 21 (10): 1153-1161, 1985851. Levine MN, Gafni A, Markham B and MacFarlane D. A bedside decision instrument to elicit a patient's preference concerningadjuvant chemotherapy for breast cancer. Annals of Internal Medicine 117: 53-8, 1992852. Lewis CC, Pantell RH and Kieckhefer GM. Assessment of children's health status: Field test of new approaches. Medical Care 27(Suppl): S54-S65, 1989853. Lewis G and Wilkinson G. Another British disease? A recent increase in the prevalance of psychiatric morbidity. Journal ofEpidemiology and Community Health 47: 358-361, 1993854. Li SC, Ioannides-Demos LL, Spicer WJ and et al. Prospective audit of aminoglycoside usage in a general hospital with assessments ofclinical processes and adverse clinical outcomes. MJA 151 (4): 224-232, 21 August 1989855. Lichter PR. Outcomes research and its future impact on physician payment reform. Ophthalmology 99 (5): 649-650, May 1992856. Lichter PR. Different providers and differing error rates in health care outcomes: cataract co-management at what price?Ophthalmology 100 (4): 445-446, 1993857. Lightfoot RW. Relevance of outcomes research to the practice of rheumatology. Bulletin on the Rheumatic Diseases 41 (8): 1-3, 1992858. Lim LL, Valenti LA, Knapp JC, Dobson AJ, Plotnikoff R and et al. A self-administered quality of life questionnaire after acutemyocardial infarction. Journal of Clinical Epidemiology 46: 1249-56, 1993859. Lipscomb J. Time preference for health in cost-effectiveness analysis. Medical Care 27: S232-S253, 1989860. Lipton HL and Bird JA. Drug utilization review in ambulatory settings: State of the science and directions for outcomes research.Medical Care 31: 1069-82, 1993861. Lohr KN. Outcome measurement: concepts and questions. Inquiry 25: 37-50, Spring 1988 *862. Lohr KN. Advances in health status assessment: overview of the conference. Medical Care 27 (3) (Suppl): S1-S11, March 1989 *863. Lohr KN. Applications of health status assessment measures in clinical practice. Medical Care 30 (Suppl): MS1-MS14, 1992 *864. Lomas J, Anderson GM, Domnick-Pierre K and et al. Do practice guidelines guide practice? The effect of a consensus statement on thepractice of physicians. NEJM 321 (19): 1306-1311, 9 November 1989 *865. Long AF, Dixon P, Hall R and et al. The outcomes agenda: contribution of the UK clearing house on health outcomes. Quality inHealth Care 2: 49-52, 1993866. Loome F and McKenzie L. The use of QALYs in health care decision making. Social Science and Medicine 28 (4): 299-308, 1989867. Lord SR and Sinnett PF. Femoral neck fractures: admissions, bed use, outcome and projections. MJA 145: 493-496, 1986868. Lovell DJ. Newer functional outcome measurements in juvenile rheumatoid arthritis: a progress report. Journal of Rheumotology 33:(Suppl): 28-31, April 1992869. Ludke RL, Booth B and Lewis-Beck JA. Relationship between early readmission and hospital quality of care indicators. Inquiry 30: 95-103, 1993870. MacKeigan LD and Pathak DS. Overview of health-related quality-of-life measures. American Journal of Hospital Pharmacy 49 (9):2236-2245, September 1992

92871. MacLean DS. Outcome and cost of family physician's care: Pilot study of three diagnosis-related groups in elderly inpatients. Journal ofthe American Board of Family Practice 6: 588-93, 1993872. Maeland JG and Havik OE. After the myocardial infarction. A medical and psychological study with special emphasis on perceivedillness. Scandinavian Journal of Rehabilitation Medicine (Suppl) 22: 1-87, 1989873. Maklan C. MEDical Treatment Effectiveness Program Poem (MEDTEPP). Milbank Quarterly 68: 170, 1990874. Malycha P. Treatment options for breast cancer. Australian Family Physician 22: 35-39, 1993875. Manchester D. Neuroleptics, learning disability, and the community: some history and mystery. BMJ 307: 184-7, 1993876. Manton KG. Epidemiological , demographic and social correlates of disability among the elderly. Milbank Quarterly 67 (Suppl) 2 Pt 1:13-58, 1989877. Martin CM and Douglas RM. Getting "value for money": measuring the quality and outcome of general practice care. MJA 159 (4):253-256, 16 August 1993878. Martin KS, Scheet NJ and Stegman MR. Home Health clients: Characteristics, outcomes of care, and nursing interventions. AmericanJournal of Public Health 83: 1730-1734, 1993879. Marwick C. Federal Agency focuses on outcomes research. JAMA 270: 164-165, 1993880. Mata GV, Fernandez, RR, Carmona AG and et al. Factors related to quality of life 12 months after discharge from an intensive careunit. Critical Care Medicine 20 (9): 1257-1262, 1992881. Mathers C. Health expectancies in Australia 1993: Preliminary results. Paper presented to the 7th Meeting of the International Networkon Health Expectancy, Canberra, 23-25 February, 1994882. Mathers C and Robine J. Health expectancy indicators: recommendations for terminology. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 1994883. McCallum J, Shadbolt B and Wang D. Self-rated health and survival: a 7-year follow-up study of Australian elderly, 1992. (UnPub)884. Connelly JE, Philbrick JT, Smith GR. Health perceptions of primary care patients and their influence on health care utilisation.Medical Care 27 (3 Suppl): S99-S109, 1989885. McCarthy M. Data fraud in US breast cancer trial [letter]. Lancet 343: 908, 1994886. McCarthy PR, Gelber S and Dugger DE. Outcome measurement to outcome management: the critical step. Administration and Policyin Mental Health 21: 59-68, 1993887. McClellan M and Brook RH. Appropriateness of care. A comparison of global and outcome methods to set standards. Medical Care 30(7): 565-586, July 1992 *888. McColl M and Skinner HA. Measuring psychological outcomes following rehabilitation. Canadian Journal of Public Health 83(Suppl): S12-S18, 1992889. McCormick K. Research: areas of outcome research for nursing. Journal of Professional Nursings 8 (2): 77, March- April 1992890. McCorvey E, Wright JT, McKenney JM and et al. Does antihypertensive therapy influence quality of life? Clinical Pharmacology andTherapeutics 8 (5): 359-364, May 1989891. McCulloch D. Can we measure 'output'? Quality-adjusted life years, health indices and occupational therapy. British Journal ofOccupational Therapy 54 (6): 219-221, June 1991892. McGorry PD. Paradigm failure in functional psychosis: Review and implications. Australian and New Zealand Journal of Psychiatry25: 43-55, 1991893. McHorney CA, Ware JE and Raczek AE. The MOS 36-item Short-Form health survey (SF-36): II. Psychometric and clinical tests ofvalidity in measuring physical and mental health constructs. Medical Care 31: 247-263, 1993894. McHorney CA, Ware JE, Rodgers W and et al. The validity and relative precision of MOS short-and long-form health status scales andDartmouth COOP charts: results from the Medical Outcomes Study. Medical Care 30 (Suppl): MS253-MS265, 1992

93895. McLean AJ, Harrison PM, Byrne AJ and et al. The choice of ulcer healing agent influences duodenal ulcer relapse rate and long-termclinical outcome. Australian and New Zealand Journal of Medicine 15: 367-374, 1985896. McLean AJ, Ioannides-Demos LL, Horne MK and et al. Impact of a pharmacokinetics consultation service on clinical outcomes in anambulatory care epilepsy clinic. American Journal of Hospital Pharmacy 45: 1549-1551, July 1988897. McMahon RP, Proschan M, Geller NL, Stone PH and Sopko G. Sample size calculation for clinical trials in which entry criteria andoutcomes are counts of events. Statistics in Medicine 13: 859-70, 1994898. McNeil BJ, Weichselbaum R and Pauker SG. Speech and survival: Tradeoffs between quality and quantity of life in laryngeal cancer.NEJM 305: 982-987, 1981899. Meenan RF and Pincus T. Editorial. The status of patient status measures. Journal of Rheumatology 14 (3): 411-414, 1987900. Megivern K, Halm MA and Jones G. Measuring patient satisfaction as an outcome of nursing care. Journal of Nursing Care Quality 6(4): 9-24, July 1992901. Mendeloff JM and Kaplan RM. Are large differences in "Lifesaving" costs justified? A psychometric study of the relative value placedon preventing deaths. Risk Analysis 9 (3): 349-363, 1989902. Merigan TC and Byar DP. Sounding Board. NEJM 323 (19): 1341-1348, 1990903. Michel TH. Outcome assessment in cardiac rehabilitation. International Journal of Technology Assessment in Health Care 8 (1): 76-84, 1992904. Micossi P, Carbone M, Stancanelli G and Fortino A. Measuring products of healthcare systems [letter]. Lancet 341: 1566-1567, 1993905. Mills I. Outcome measures - getting there. Health Services Journal 16: 822, July 1987906. Minaire P. Disease, illness and health: theoretical models of the disablement process. Bulletin of the World Health Organisation 70(3): 373-379, 1992907. Mitchell JB. The role of large data bases in effectiveness research. In: Effectiveness and Outcomes in Health Care, edited by HeithoffKA and Lohr LN. Washington, DC: National Academy Press, 1990908. Moinpour C.McM Feigl P, Metch B and et al. Quality of life end points in cancer clinical trials: review and recommendations. Journalof the National Cancer Institute 81 (7): 484-495, 1989909. Molzahn AE. Should quality of life measures be used to assess quality of care? ANNA Journal 19 (1): 88, February 1992910. Montegue J. Outcomes on the upswing. Hospital and Health Networks Jan 20: 56, 1994911. Moon M. The economic situation of older Americans: emerging wealth and continuing hardship. Pediatriatric Annals 17 (12): Dec:762-764, 1988912. Mor V. Cancer patients' quality of life over the disease course: lessons from the real world. Journal of Chronic Diseases 40 (6): 535-544, 1987913. Morris JN and Sherwood S. Quality of life of cancer patients at different stages of the disease trajectory. Journal of Chronic Diseases40 (6): 545-553, 1987914. Morris PLP and Jones B. Life satisfaction across treatment methods for patients with end-stage renal failure. MJA 150 (8): 428-432, 17April 1989915. Morrison GC and Gyldmark M. Appraising the use of contingent valuation. Health Economics 1: 233-243, 1992916. Mosteller F, Ware JE and Levine S. Final Panel: Comments on the Conference on Advances in Health Status Assessment. MedicalCare 27 (Suppl): S282-S294, 1989917. Moulds RFW. Measurement of cost-effectiveness of drug therapy: a review of the treatment of hypertension. MJA 153 (Suppl): 54-56,6 August 1990918. Muir Gray JA. Two classes of creativity - improving systematic reviews. Journal of Epidemiology and Community Health 48: 4-5,1994

94919. Mulley AG. Assessing patients' utilities. Can the ends justify the means? Medical Care 27 (3) (Suppl): S269-S281, March 1989920. Mulley AG. Applying effectiveness and outcomes research to clinical practice. In: Effectiveness and outcomes in health care:Proceedings of an invitational conference by the Institute of Medicine, edited by Heithoff KA and Lohr KN. Washington, DC: NationalAcademy Press, 1990921. Mulrow CD, Aguilar C and Endicott JE. Quality of life changes and hearing impairment. A randomized trial. Annals of InternalMedicine 113: 188-194, 1990922. Mulrow CD, Gerety MB, Kanten D, Cornell JE, DeNino LA and et al. A randomised trial of physical rehabilitation for very frailnursing home residents. JAMA 271: 519-524, 1994923. Murphy DJ and Cluff LE. Support: study to understand prognoses and preferences for outcomes and risks of treatments. Journal ofClinical Epidemiology 43:1990924. Murray C. Quantifying the burden of disease. The technical basis for disability adjusted life years. 1993 (Unpub)925. Murray LS, Teasdale GM, Murray GD and et al. Does prediction of outcome alter patient management? Lancet 341: 1487-91, 1993926. Mutafova M, Maleshkov C and Tonkova S. Disability-free life expectancy: a pilot investigation. Paper presented to the 7th Meeting ofthe International Network on Health Expectancy, Canberra, 23-25 February, 1994927. Myers A. The clinical Swiss Army Knife: empirical evidence on the validity of IADL functional status measures. Medical Care 30(Suppl): MS96-MS111, 1992928. Najman J and Levine S. Evaluating the impact of medical care and technologies on the quality of life: A review and critique. SocialScience and Medicine 15F: 107-15, 1981929. Nelson E and Berwick DM. The measurement of health status in clinical practice. Medical Care 27 (Suppl): S77-S90, 1989 *930. Nelson EC. Using outcome measures to improve care delivered by physicians and hospitals. In: Effectiveness and outcomes in healthcare: Proceedings of an invitational conference by the Institute of Medicine, edited by Heithoff KA and Lohr KN. Washington, DC:National Academy Press, 1990931. Nerenz DR, Repasky DP, Whitehouse FW and Kahkonen DM. Ongoing assessment of health status in patients with diabetes mellitus.Medical Care 30 (Suppl): MS112-MS124, 1992932. Nord E. EuroQol: health related quality of life measurement. Valuations of health states by the general public in Norway. HealthPolicy 18: 25-36, 1991933. Nord E. An alternative to QALYs: the saved young life equivalent (SAVE). BMJ 305: 875-877, 1992934. Nord E. Methods for quality adjustment of life years. Social Science and Medicine 34 (5): 559-569, 1992935. Nord E. Social evaluation of health care versus personal evaluation of health states. International Journal of Assessment in HealthCare 9: 463-478, 1993936. Nord E. Unjustified use of the Quality of Well-Being scale in priority setting in Oregon. Health Policy 24: 45-53, 1993937. Nord E. The QALY: A measure of social value rather than individual utility? Health Economics 3: 89-93, 1994938. NSW Health Department, Health care at any cost? 1991 (UnPub)939. NSW Health Department, Getting it right: Focusing on the outcomes of health services and programs, Sydney: NSW Health, 1994940. Nusselder WJ, van der Velden J, van Sonsbeek JLA and et al. The effect of elimination of selected chronic diseases on the disabilityfreelife expectancy: compression or expansion of morbidity? Preliminary results. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 1994941. Olsen J. Time preferences for health gains: An empirical investigation. Health Economics 2: 257-265, 1993942. Olshanky SJ and Carnes BA. Demographic perspectives on human senescence. Paper presented to the 7th Meeting of the InternationalNetwork on Health Expectancy, Canberra, 23-25 February, 1994

95943. Olweny CLM. Quality of life in cancer care. MJA 158: March 15: 429-432, 1993944. Orenstein DM, Pattishall EN, Nixon PA and et al. Quality of well-being before and after antibiotic treatment of pulmonaryexacerbation in patients with cystic fibrosis. Chest 98: 1081-1084, 1990945. Orentlicher D. Rationing and the Americans with Disabilities Act. JAMA 271: 308-14, 1994946. Overy S. Government encouraging doctors to change focus. Australian Doctor 22 April: 25, 1994947. Oxner RBG, Simmonds NJ, Gertner DJ, Nightingdale JMD and Burnham WR. Controlled trial of endoscopic injection treatment forbleeding from peptic ulcers with visible vessels. Lancet 339: 966-968, 1992948. Pailthorpe CA. Outcomes in trauma. In: Outcome measures in orthopaedics, edited by Pynsent PB, Fairbank JCT and Carr A. Oxford:Butterworth, 1993949. Palmer RM, Saywell RM, Zollinger T and et al. The impact of prospective payment system on the treatment of hip fractures in theelderly. Archives of Internal Medicine 149: 2237-22341, 1989950. Panisset M, Roudier M, Saxton J and Boller F. Severe Impairment Battery: A neuropsychological test for severely demented patients.Archives of Neurology 51: 41-45, 1994951. Parkerson GR, Connis RT, Broadhead WE, Patrick DL, Taylor TR and Tse C. Disease-specific versus generic measurement of healthrelatedquality of life in insulin-dependent diabetic patients. Medical Care 31: 629-639, 1993952. Parkerson GR, Deyo RA, Golden WE and et al, Strategies for improving and expanding the application of health status measures inclinical settings. Medical Care 30: MS210-MS218, 1992953. Parmenter TR. A study of the quality of life of people with severe physical disabilities, 1988 (UnPub)954. Parsonage M and Neuburger H. Draft. Discounting and QALYs, 1991 (UnPub)955. Parsonage M and Neuburger H. Discounting and health benefits. Health Economics 1: 71-79, 1992956. Pashos CL and McNeil BJ. Consequences of variation in treatment for acute myocardial infarction. Health Services Research 25 (5):717-722, December 1990957. Patrick DL. Assessing health-related quality of life outcomes. In: Effectiveness and Outcomes in Health Care, edited by Heithoff KAand Lohr KN. Washington, DC: National Academy Press, 1990958. Patrick DL. Strategies for improving and expanding the application of health status measures in clinical settings [Discussion]. MedicalCare 30 (Suppl): MS198-MS199, 1992959. Patrick DL and Bergner M. Measurement of health status in the 1990s. Annual Review of Public Health 11: 165-83, 1990960. Patrick DL and Deyo RA. Generic and disease-specific measures in assessing health status and quality of life. Medical Care 27(Suppl): S217-S232, 1989961. Petrou S and Renton A. The QALY: A guide for the public health physician. Public Health 107: 327-336, 1993962. Phelps CE. Death and taxes. An opportunity for substitution. Journal of Health Economics 7: 1-24, 1988963. Pill R and Stott NCH. Development of a measure of potential health behaviour: a salience of lifestyle index. Social Science andMedicine 24 (2): 125-134, 1987964. Piper MC. Rehabilitation outcome measures: Where do we go from here? Canadian Journal of Public Health 83: S69-S70, 1992965. Pollack VE, Pesce A and Kant KS. Original investigations. Continuous quality improvement in chronic disease: a computerisedmedical record enables description of a severity index to evaluate outcomes in end-stage renal disease. American Journal of KidneyDiseases XIX: 6 June: 514-522, 1992966. Prescott M. Outcomes solutions may impede access to health care. Outcomes Measurement & Management 4-5, May/June1994967. Pryce-Jones M. Assessing the quality of discharge procedures for elderly people. Health Services Management 23-26, June 1992

96968. Radford PJ. General outcome measures. In: Outcome Measures in Orthopaedics, edited by Pynsent PB, Fairbank JCT and Carr A.Oxford: Butterworth, 1993969. Rantucci MJ and Segal HJ. Over-the-counter medication. Outcome and effectiveness of patient counselling. Journal of Social andAdministrative Pharmacy 3 (3): 81-91, 1986970. Raphael B. Psychiatry 'at the coal-face'. Australian and New Zealand Journal of Psychiatry 20: 316-322, 1986971. Raskin IE and Maklan CW. Medical treatment effectiveness research: A view from inside the Agency for Health Care Policy andResearch. Evaluation and the Health Professional 161-186, June 1991972. Rawles J, Light J and Watt M. Loss of quality adjusted days as a trial endpoint: effect of early thrombolytic treatment in suspectedmyocardial infarction. Journal of Epidemiology and Community Health 47: 377-381, 1993973. Regier DA, Hirschfield RMA and Goodwin FK. The NIMH Depression Awareness, Recognition, and Treatment Program: Structure,aims and scientific basis. American Journal of Psychiatry 145: 1351-1357, 1988974. Reinertsen JL. Outcomes management and continuous quality improvement: the compass and the rudder. Quality Review Bulletin 19(1): 5-7, January 1993975. Reisine ST. The impact of dental conditions on social functioning and the quality of life. Annual Review of Public Health 9: 1-19, 1988976. Relman AS. Assessment and accountability: The third revolution in medical care. NEJM 319: 1220-2, 1988 *977. Reverby S. Stealing the golden eggs: Ernest Amory Codman and the science and management of medicine. Bulletin of the History ofMedicine 55: 156-171, 1981978. Reves Network on Health Expectancy and the Disability Process, Introduction to the Network, 1991. (UnPub)979. Revicki DA. Relationship between health utility and psychometric health status measures. Medical Care 30 (Suppl): MS274-282, 1992980. Richardson J. Cost utility analyses in health care: present status and future issues - Working Paper No. 8 In: Research Into HealthCare: Designs, Dilemmas and Disciplines, edited by Daly J, McDonald I and Willis E. Sage Publications, 1989981. Richardson J. Cost utility analyses: what should be measured - utility, value or healthy year equivalents? 1989 (UnPub)982. Richardson J. Economic analyses in health care: theory and practice, 1990 (UnPub)983. Richardson J. Economic assessment of health care: theory and practice. Australian Economic Review 1991984. Richardson J, Schwartz S and Glasziou PP. QALYs for resource allocation: a reply to Burrows and Brown [letter]. Australian Journalof Public Health 17: 394-6, 1993985. Rickard MT, Lee W, Read JW and et al. Breast cancer diagnosis by screening mammography: early results of the Central Sydney AreaHealth Service Breast X-ray Programme. MJA 154: 126-131, 1991986. Ries AL, Kaplan RM and Blumberg E. Use of factor analysis to consolidate multiple outcome measures in chronic obstructivepulmonary disease. Journal of Clinical Epidemiology 44 (6): 497-503, 1991987. Ringdal GI and Ringdal K. Testing the EORTC Quality of Life Questionnaire on cancer patients with heterogeneous diagnoses.Quality of Life Research 2: 129-140, 1993988. Ritchie K. International comparisons of dementia-free life expectancy: a critical review of the results obtained. Paper presented to the7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994989. Rob M. Current health indicators. NSW Public Health Bulletin 2: 7990. Roberts MM, Alexander FE, Anderson TJ and et al. Edinburgh trial of screening for breast cancer: mortality at seven years. Lancet335: 241-246, 1990991. Roberts RS. Pooled outcome measures in arthritis: the pros and cons. Journal of Rheumatology 20 (3): 566-567, 1993992. Robine J. Disability-free life expectancy trends in France: 1981-1991, international comparison. Paper presented to the 7th Meeting ofthe International Network on Health Expectancy, Canberra, 23-25 February, 1994

97993. Robine J, Brouard N, Jagger C, Ritchie K and van de Water H. Harmonization of health expectancy in Europe. Paper presented to the7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994994. Robine J and Mormiche P. L'esperance de vie sans incapacite augmente. Paper presented to the 7th Meeting of the InternationalNetwork on Health Expectancy, Canberra, 23-25 February, 1994995. Robinson R. Cost-effectiveness analysis. BMJ 307: 793-795, 1993996. Rochon PA, Gurwitz JH, Simms RW and et al. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in thetreatment of arthritis. Archives of Internal Medicine 154: 157-163, 1994997. Rockland LH. A review of supportive psychotherapy, 1986-1992. Hospital and Community Psychiatry 44: 1053-1060, 1993998. Roelands M and van Oyen H. Mental and social variable as operationalisation of health in the calculation of health expectancy. Paperpresented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 1994999. Roelands M, van Oyen H and Baro F. Dementia free life expectancy in Belgium. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 19941000. Roizen MF, Coalson D, Hayward RSA and et al. Can patients use an automated questionnaire to define their current health status?Medical Care 30 (Suppl): MS74-MS84, 19921001. Romieu I and Robine J. World atlas on health expectancy calculations. Paper presented to the 7th Meeting of the International Networkon Health Expectancy, Canberra, 23-25 February, 19941002. Rontal R, Kiess MJ, DesHarnais SI and et al. Applications for risk-adjusted outcome measures. Quality Assurance in Health Care 3(4): 283-292, 19911003. Roos LL, Fisher ES, Brazauskas R and et al. Health and surgical outcomes in Canada and the United States. Health Affairs 56-72,Summer 1992 *1004. Roper WL, Winkenwerder W, Hackbarth GM and et al. Effectiveness in health care: an initiative to evaluate and improve medicalpractice. NEJM 319 (18) November 3: 1197-1202, 19881005. Ross Davies A, Doyle MAT, Lansky D, Stevic MO and Doyle JB. Outcomes assessment in clinical settings: A consensus statement onprinciples and best practices in project management. Joint Commission Journal on Quality Improvement 6-16, January 19941006. Rosser R. Issues of measurement in the design of health indicators: a review, 1979. (UnPub)1007. Rosser R, Allison R, Butler C et al. The index of health-related quality of life (IHQL): a new tool for audit and cost-per-QALY analysis,(UnPub)1008. Rosser R, Allison R, Butler C and et al. The index of health-related quality of life (IHQL): a new tool for audit and cost-per QALYanalysis. In: Quality of Life Assessment: Key Issues in the 1990s, edited by Walker SR and Rosser RM. Dordrecht, Netherlands:Kluwer Academic Publishers, 19931009. Rosser R and Kind P. A scale of valuations of states of illness: is there a social concensus? International Journal of Epidemiology 7(4): 347-357, 19781010. Rost K, Smith G R, Burnam MA and Burns BJ. Measuring the outcomes of care for mental health problems: The case of depressivedisorders. Medical Care 30 (Suppl): MS266-MS273, 19921011. Rothman ML, Hedrick S and Inui T. The Sickness Impact Profile as a measure of the health status of noncognitively impaired nursinghome residents. Medical Care 28 (Suppl): S157-S167, 19891012. Rothman ML and Revicki DA. Issues in the measurement of health status in asthma research. Medical Care 31: MS82-MS96, 19931013. Rubenstein LV, Calkins, DR, Greenfield S and et al. Health status assessment for elderly patients. Report of the Society of GeneralInternal Medicine Task Force on Health Assessment. Journal of the American Geriatrics Society 37: 562-569, 19891014. Rubin G, Frommer M, Morey S and Leeder S. On the right track. NSW Public Health Bulletin 2: 1-2, 19911015. Ruta DA, Garratt AM and Russell IT. A new approach to the measurement of quality of life: the patient generated index (PGT), 1992(UnPub)

981016. Rutstein DD, Berenberg W, Chalmers TC and et al. Measuring the quality of medical care (second revision of tables May 1980): aclinical method. NEJM 294: March 11: 582-588, 19761017. Sackett DL, Chambers LW, Macpherson AS and et al. The development and application of indices of health: general methods and asummary of results. American Journal of Public Health 67: 423-428, 19771018. Sackett DL and Torrance GW. The utility of different health states as perceived by the general public. Journal of Chronic Diseases 31:697-704, 19781019. Sacks NPM and Baum M. Primary management of carcinoma of the breast. Lancet 342: 1402-1408, 19931020. Sage WM. Outcomes research [letter]. NEJM 330: 434-435, 19941021. Salive ME, Mayfield JA and Weissman NW. Patient outcomes research teams and the agency for health care policy and research.Health Services Research 25 (5): 697-708, December 19901022. Sansoni J. Evaluation and management of entrepreneurial education activities [draft], (UnPub)1023. Saravay SM, Steinberg MD, Weinschel B, Pollack S and Alovis N. Psychological comorbidity and length of stay in the generalhospital. American Journal of Psychiatry 148: 324-329, 19911024. Sarna L. Women with lung cancer: impact on quality of life. Quality of Life Research 2: 13-22, 19931025. Saunders CM and Baum M. Quality of life during treatment for cancer. British Journal of Hospital Medicine 48 (2): 119-123, 19921026. Sawyer M G, Sarris A, Baghurst PA, Cornish CA and Kalucy RS. The prevalence of emotional and behaviour disorders and patterns ofservice utilisation in children and adolescents. Australian and New Zealand Journal of Psychiatry 24: 323-330, 19901027. Schalock RL. Where do we go from here? In: Quality of Life: Perspectives and Issues, edited by Schalock RL and Begab MJ.Washington, DC: American Association on Mental Retardation, 19901028. Schalock RL. Attempts to conceptualize and measure quality of life. In: Quality of Life: Perspectives and Issues, edited by Schalock RLand Begab MJ. Washington, DC: American Association on Mental Retardation, 19901029. Schieber GJ, Poullier J and Greenwald LM. Health spending, delivery, and outcomes in OECD countries. Health Affairs 120-129,Winter 19921030. Schilling RF, El-Bassel N, Gilbert L and Glassman M. Predictors of changes in sexual behaviour among on methadone. AmericanJournal of Drug and Alcohol Abuse 19: 409-422, 19931031. Schoenbaum SC. An attempt to manage variation in obstetrical practice. In: Effectiveness and Outcomes in Health Care: Proceedingsof an Invitational Conference by the Institute of Medicine, edited by Heithoff KA and Lohr KN. Washington, DC: National AcademyPress, 19901032. Schoenbaum SC. Editorial: Toward fewer procedures and better outcomes. JAMA 269 (6): 794-896, 10 February 19931033. Schwartz JS and Lurie N. Assessment of medical outcomes. New opportunities for achieving a long sought-after objective.International Journal of Assessment in Health Care 6: 333-339, 19901034. Schwartz S. Preferences for health outcomes: Factors affecting health state utilities, 1994. (UnPub)1035. Schwartz S, Richardson J and Glasziou P. Quality-adjusted life years: origins, measurements, applications, objections. AustralianJournal of Public Health 17: 272-278, 19931036. Scottish Cancer Trials Breast Group, and ICRF Breast Unit, Adjuvant ovarian ablation versus CMF chemotherapy in premenopausalwomen with pathological stage II breast carcinoma: the Scottish trial. Lancet 341: 1293-1298, 19931037. Secretary of State for health, The health of the nation: a strategy for health in England, 1992. (UnPub)1038. Seedhouse D. Core health services: a fiction? New Zealand Medical Journal 961 (Suppl): 8, 19931039. Shah S, Vanclay F and Cooper B. Stroke rehabilitation: Australian patient profile and functional outcome. Journal of ClinicalEpidemiology 44: 21-28, 1991

991040. Shah SK, Vanclay F and Cooper B. Improving the sensitivity of the Barthel index for stroke rehabilitation. Journal of ClinicalEpidemiology 42 (8): 703-709, 19891041. Shapiro E and Roos NP. Elderly non-users of health care services: their characteristics and their health outcomes, (UnPub), 19951042. Sheldon TA and Borowitz M. Changing the measure of quality in the NHS: from purchasing activity to purchasing protocols. Quality inHealth Care 2: 149-150, 19931043. Shepherd SL, Hovell MF, Harwood IR and et al. A comparative study of the psychosocial assets of adults with cystic fibrosis and theirhealthy peers. Chest 97: 1310-1316, 19901044. Shern DL, Wilson NZ, Coen AS and et al. Client outcomes II. Longitudinal client data from the Colorado Treatment Outcome Study.Milbank Q 72: 123-148, 19941045. Showstack J, Katz P, Amend W and et al. The effect of cyclosporine on the use of hospital resources for kidney transplantation. NEJM321 (16): 1086-1092, 19 October 19891046. Silver GA. Outcomes measures. Lancet 337: 1088-1089, 19911047. Simes RJ and Glasziou PP. Meta-analysis and quality of evidence in the economic evaluation of drug trials. Pharmaco Economics 1:282-292, 19921048. Singleton N and Turner A. SF 36 is suitable for elderly patients [letter]. BMJ 307: 126-127, 19931049. Siu AL, McGlynn EA, Morgenstern H and et al. Choosing quality of care measures based on the expected impact of improved care onhealth. Health Services Research 27 (5): 619-650, December 19921050. Skolnick AA. Joint Commission will collect, publicise outcomes. JAMA 270: 165-171, 19931051. Smith D. Does therapy work? Sydney Morning Herald 1 June 19931052. Smith L. Reflections on the next stage of the epidemiological transition. Paper presented to the 7th Meeting of the InternationalNetwork on Health Expectancy, Canberra, 23-25 February, 19941053. Smith P. Performance indicators: are they worth it? Audit 74-78, 19871054. Smith R and Dobson M. Measuring utility values for QALYs: Two methodological issues. Health Economics 2: 349-355, 19931055. Snow CA & Associates Report. Page 7:"It's not blank." Opinions about and suggested changes for a "quality of life" questionnaire, 1991(UnPub)1056. Sox HC. Effectiveness research and changing physician practice patterns. In: Effectiveness and Outcomes in Health Care: Proceedingsof an Invitational Conference by the Institute of Medicine, edited by Heithoff KA and Lohr KN. Washington, DC: National AcademyPress, 19901057. Spiegelhalter DJ, Gore S and Fitzpatrick R. Quality of life measures in health care.III: resource allocation. BMJ 305 (6863): 1205-1209, 14 November 19921058. Stalfelt A and Wadman B. Assessing quality of life in leukaemia: Presentation of an instrument for assessing quality of life in patientswith blood malignancies. Quality Assurance in Health Care 5: 201-211, 19931059. Stedman T, Haire M and Welham J. Quality of life and tertiary prevention for the chronically mentally ill, 1991 (UnPub)1060. Stein R, Gortmaker SL, Perrin EC and et al. Severity of illness: concepts and measurements. Lancet December 26: 1506-1509, 19871061. Steinberg EP, Bergner M, Sommer A and et al. Variations in cataract management: patient and economic outcomes. Health ServicesResearch 25 (5): 727-731, December 19901062. Stewart AL, Hays RD and Ware JE. The MOS Short-Form General Health Survey: reliability and validity in a patient population.Medical Care 26: 724-732, 19881063. Stewart AL, Sherbourne CD, Hays RD, Wells KB, Nelson EC, Kamberg CJ, Rogers WH, Berry SH and Ware JE. Summary anddiscussion of MOD measures. In: Measuring Functioning and Well-Being. The Medical Outcomes Study Approach, edited by StewartAL and Ware JE,Jr. Durham: Duke University Press, pp 345-403, 1992

1001064. Stockbridge H, Hardy RI and Glueck CJ. Public cholesterol screening: motivation for participation, follow-up outcome, self-knowledgeand coronary heart disease risk factor intervention. Journal of Laboratory and Clinical Medicine 114 (2): 142-151, 19891065. Studenski S and Woods Duncan P. Measuring rehabilitation outcomes. Geriatric Rehabilitation 9: 823-830, 19931066. Swain SM. In situ or localised breast cancer: How much treatment is needed? NEJM 328: 1633-1634, 19931067. Szabadi E. Physical exercise and mental health. BMJ 296: 5 March: 659-660, 19881068. Tancredi LR and Bovbjerg RR. Advancing the epidemiology of injury and methods of quality control: ACEs as an outcomes-basedsystem for quality improvement. Quality Review Bulletin 201-209, June 19921069. Tanenbaum SJ. What physicians know. NEJM 329: 1268-1271, 1993 *1070. Tanenbaum SJ. Outcomes research [letter]. NEJM 330: 435, 19941071. Tarlov AR, Ware JE, Greenfield S and et al. The Medical Outcomes Study: an application of methods for monitoring the results ofmedical care. Journal of the American Medical Record Association 262: 925-930, 1989 *1072. Tatchell M. Measuring hospital output: a review of the service mix and case mix approaches. Social Science and Medicine 17 (13):871-883, 19831073. Temkin NR, Dikmen S, Machamer J and McLean A. General versus disease-specific measures. Medical Care 27 (Suppl): S44-S53,19891074. Testa MA and Simonson DC. Management of hypertension and nephropathy in diabetes. Measuring quality of life in hypertensivepatients with diabetes. Postgraduate Medical Journal 64 (Suppl) 3: 50-58, 19881075. The Essendon Breast X-ray Program Collaborative Group, A mammographic screening pilot project in Victoria 1988-1990. MJA 157:670-673, 19921076. The Lancet, Breast cancer: clearing trails in the forest without losing our way [Editorial]. Lancet 343: 1049-1050, 19941077. Thier SO. Forces motivating the use of health status assessment measures in clinical settings and related clinical research. MedicalCare 30 (Suppl): MS15-MS22, 19921078. Thomas S, Steven I, Browning C and et al. Focus groups in health research: a methodological review. Annual Review of Health andSocial Sciences 2: 7-20, 19921079. Thompson JF, McGhan WF, Ruffalo RL and et al. Long-term care. Clinical pharmacists prescribing drug therapy in a geriatric setting:outcome of a trial. Journal of the American Geriatrics Society 32: 2 February: 154-159, 19841080. Toevs CD, Kaplan RM and Atkins CJ. The costs and effects of behavioural programs in chronic obstructive pulmonary disease.Medical Care 22:12 December: 1088-1101, 19841081. Tollefson GD and Holman SL. Analysis of the Hamilton Depression rating Scale factors from a double-blind, placebo-controlled trial offluoxetine in geriatric major depression. International Clinical Psychopharmacology 8: 253-259, 19931082. Tollefson GD, Souetre E, Thomander L and Potvin JH. Comorbid anxious signs and symptoms in major depression: impact onfunctional work capacity and comparative treatment outcomes. International Clinical Psychopharmacology 8: 281-293, 19931083. Torrance GW. Social preferences for health states: an empirical evaluation of three measurement techniques. Socio-economicPlanning Sciences 10: 129-136, 19761084. Torrance GW. Measurement of health stat utilities for economic appraisal. A review. Journal of Health Economics 5: 1-30, 19861085. Torrance GW. Utility approach to measuring health-related quality of life. Journal of Chronic Diseases 40 (6): 593-600, 19871086. Torrance GW, Boyle MH and Horwood SP. Application of multi-attribute utility theory to measure social preferences for health states,1982. (UnPub)1087. Tu EJ and Chen K. Recent changes in active life expectancy in Taiwan. Paper presented to the 7th Meeting of the InternationalNetwork on Health Expectancy, Canberra, 23-25 February, 1994

1011088. Tugwell P and Boers M. OMERACT Conference on outcome measures in rheumatoid arthritis clinical trials: introduction. Journal ofRheumatology 20 (3): 528-530, 19931089. Turner JA, Ersek M, Herron L and et al. Patient outcomes after lumbar spinal fusions. JAMA 268 (7): 907-911, 19 August 19921090. Tuteur PG. Strategies for improving and expanding the application of health status measures in clinical settings. Medical Care 30(Suppl): MS202-MS204, 19921091. UK Clearing House, Outcomes briefing, 19931092. US Department of Health and Human Services, Capping Medicaid prescription drugs may backfire. Research Activities 148:December:1-7, 19911093. US Department of Health and Human Services, Progress of research on outcomes of health care services and procedures, 19911094. Valkonen T, Sihvonen A and Lahelma E. Disability-free life expectancy by level od education in Finland. Paper presented to the 7thMeeting of the International Network on Health Expectancy, Canberra, 23-25 February, 19941095. van den Berg Jeths A, Ruwaard D and Kramers PGN. Healthy life expectancy in the Dutch document "Public health status andforecasts". Paper presented to the 7th Meeting of the International Network on Health Expectancy, Canberra, 23-25 February, 19941096. van den Brink W, Leenstra A, Ormel J and van de Willige G. Mental health intervention programs in primary care: their scientificbasis. Journal of Affective Disorders 21: 273-284, 19911097. van Oyen H, Tafforeau J and Roelands M. Regional inequities in health expectancy in Belgium (draft). Paper presented to the 7thMeeting of the International Network on Health Expectancy, Canberra, 23-25 February, 19941098. Vanclay F. Functional outcome measures in stroke rehabilitation. Stroke 22: 105-108, 19911099. Verbrugge LM and Balaban DJ. Patterns of change in disability and well-being. Medical Care 27 (Suppl ): S128-S147, 19891100. Verbrugge LM, Lepkowski JM and Imanaka Y. Comorbidity and its impact on disability. Milbank Quarterly 67: 450-484, 19891101. Veronesi U, Luini A, Del Vecchio M and et al, Radiotherapy after breast-preserving surgery in women with localized cancer of thebreast. NEJM 328: 1587-1591, 19931102. Von Kroff M, Ustum TB, Ormel J, Kaplan I and Sartorius N. Self-report of disability: Reliability and validity in an internationalprimary care study, 1993. (UnPub)1103. Ward M, King L, Rubin G and Stewart G. Setting a new agenda. NSW Public Health Bulletin 2: 5-6, 19911104. Ware J. Measuring patient function and well-being: Some lessons from the Medical Outcomes Study. In: Effectiveness and Outcomesin Health Care, edited by Heithoff KA and Lohr KN. Washington, DC: National Academy Press, 1990 *1105. Ware JE. Standards for validating health measures: definition and content. Journal of Chronic Diseases 40: 473-480, 1987 *1106. Ware JE. Comments on the use of health status assessment in clinical settings. Medical Care 30 (Suppl): MS205-MS209, 1992 **1107. Ware JE and Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Medical Care 26: 393-402, 1988*1108. Ware JE and Sherbourne CD. The MOS 36-Item Shortform Health Survey (SF-36): conceptual framework and item selection. MedicalCare 30: 473-483, 1992 **1109. Ware JE,Jr, Sherbourne CD and Davies AR. 16. Developing and testing the MOS 20-item short-form health survey: a generalpopulation application. In: Measuring Functioning and Well-Being. The Medical Outcomes Study Approach, edited by Stewart AL andWare JE, Jr. Durham: Duke University Press, pp 277-303, 19921110. Wasson J, Keller A, Rubenstein L and et al. Benefits and obstacles of health status assessment in ambulatory settings. Medical Care 30(Suppl): MS42-MS49, 19921111. Watchel T, Piette J, Mor V and et al. Quality of life in persons with human immunodeficiency virus infection: measurement by theMedical Outcomes Study Instrument. Annals of Internal Medicine 116: 129-137, 19921112. Wennberg JA. AHCPR and the strategy for health care reform. Health Affairs 67-71, Winter 1992

1021113. Wennberg JE. Outcomes research, cost containment and the fear of health care rationing. NEJM 323 (17): 1202-1204, 25 October19901114. Wennberg JE. Outcomes research, patient preference, and the primary care physician. Journal of the American Board of FamilyPractice 4: 365-367, 1991 *1115. Wennberg JE, Barnes BA and Zubkoff M. Professional uncertainty and the problem of supplier-induced demand. Social Science andMedicine 16: 811-824, 19821116. Wennberg JE, Bunker JP and Barnes B. The need for assessing the outcome of common medical practices. Annual Review of PublicHealth 1: 277-295, 1980 *1117. Westbrook J, Frommer M. and Rubin G. What is the difference between quality assurance and health outcomes? Public HealthBulletin 4 (12): 133-134, 19931118. Westlie L, Fauchald P, Talseth T, Jakobsen A and Flatmark A. Quality of life in Norwegian kidney donors. Nephrology, Dialysis,Transplantation 8: 1146-1150, 19931119. Whiteford H. A national mental health policy for Australia. MJA 157: 510-511, 19921120. Whiteford H. Future directions for mental health services in Australia. Australian Journal of Public Health 16: 350-353, 19921121. Whiteneck GG, Charlifue SW, Gerhart KA and et al. Quantifying handicap: a new measure of long-term rehabilitation outcomes.Archives of Physical Medicine and Rehabilitation 73 (6): 519-526, June 19921122. WHOQOL Group, Study protocol for the World Health Organisation project to develop a Quality of Life assessment instrument(WHOQOL). Quality of Life Research 2: 153-159, 19931123. Whynes DK and Neilson AR. Convergent validity to two measures of quality of life. Health Economics 2: 229-235, 19931124. Wilkins R, Chen J and Ng E. Changes in health expectancy in Canada from 1986 to 1991. Paper presented to the 7th Meeting of theInternational Network on Health Expectancy, Canberra, 23-25 February, 19941125. Williams A. The cost-effectiveness approach to the treatment of angina. In: The Management of Angina Pectoris, edited by PattersonD. Castle House Publications pp 131-141, 19871126. Williams A. 200 years is not enough (UnPub), 19891127. Williams GH. Beyond blood pressure control. Effect of antihypertensive therapy on quality of life. American Journal of Hypertension1 (4): Part 2: 363S-365S, October 19881128. Wilson AA, Hartnett M and Ferrari P. Outcome measurement from the functional status perspective. Home Health Care 10:3: 32-46,19921129. Wilson D, Wakefield M and Taylor A. The South Australian Health Omnibus Survey. Informatics in Health Care 3: 65-68, 19941130. Wilson RW and Drury TF. Interpreting trends in illness and disability: health statistics and health status. Annual Review of PublicHealth 5: 83-196, 19841131. Wirtschafter DD, Jones KR and Thomas JT. Using health care outcomes to improve patient care in the NICU. Joint CommissionJournal on Quality Improvement 20: 57-65, 19941132. Woolf SH. Practice guidelines, a new reality in medicine. Archives of Internal Medicine 152: 946-952, 19921133. Worcester MC, Hare DL, Oliver RG, Reid MA and Goble AJ. Early programmes of high and low intensity exercise and quality of lifeafter acute myocardial infarction. BMJ 307: 1244-1247, 19931134. Wu AW, Rubin HR, Mathews WC and et al. A health status questionnaire using 30 items from the Medical Outcomes Study:preliminary validation in persons with early HIV infection. Medical Care 29: 786-798, 19911135. Yates DW, Woodford M and Hollis S. Preliminary analysis of the care of injured patients in 33 British hospitals: first report of theUnited Kingdom major trauma outcome study. BMJ 305 (6856): 737-740, 26 September 19921136. Zander K. Focusing on patient outcome: case management in the 90's. Dimensions of Critical Care Nursing 11 (3): 127-129, May-June 1992

Quality and Outcome Indicators for Acute Healthcare Services

Create successful ePaper yourself

Delete template?

Save as template?