See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/242915900 Bibliometric performance measures Article in Scientometrics · July 1996 DOI: 10.1007/BF02129596 CITATIONS READS 200 692 2 authors: Francis Narin Kimberly Hamilton Retired President, CHI Research, Inc The Patent Board 96 PUBLICATIONS 10,802 CITATIONS 9 PUBLICATIONS 1,736 CITATIONS SEE PROFILE SEE PROFILE Some of the authors of this publication are also working on these related projects: "UTILIZATION OF BASIC SCIENTIFIC RESEARCH AS EVIDENCED BY US PATENT CITATIONS" Coauthors Carpenter and Narin, Research Management Magazine of IRI, XXIII-2, March 1980. View project All content following this page was uploaded by Francis Narin on 18 February 2015. The user has requested enhancement of the downloaded file. Jointly published by Elsev&r Science Ltd, Oxford and Akad~miai Kiad6, Budapest Scientometrics, Vol. 36, No. 3 (1996) 293-310 BIBLIOMETRIC PERFORMANCE MEASURES F. NARIN, KIMBERLY S. HAMILTON CHI Computer Horizons Inc., 10 White Horse Pike, Haddon Heights, NJ 08035 (USA) r (Received May 24, 1996) Three different types of bibliometrics - literature bibliometrics, patent bibliometrics, and linkage bibliometric can all be used to address various government performance and results questions. Applications of these three bibliometric types will be described within the framework of Weinberg's internal and external criteria, whether the work being done is good science, efficiently and effectively done, and whether it is important science from a technological viewpoint. Within all bibliometrics the fundamental assumption is that the frequency with which a set of papers or patents is cited is a measure of the impact or influence of the set of papers. The literature bibliometric indicators are counts of publications and citations received in the scientific literature and various derived indicators including such phenomena as cross-sectoral citation, coauthorship and concentration within influential journals. One basic observation of literature bibliometrics, which carries over to patent bibliometrics, is that of highly skewed distributions - with a relatively small number of high-impact patents and papers, and large numbers of patents and papers of minimal impact. The key measure is whether an agency is producing or supporting highly cited papers and patents. The final set of data are in the area of linkage bibliometrics, looking at citations from patents to scientific papers. These are particularly relevant to the external criteria, in that it is quite obvious that institutions and supporting agencies whose papers are highly cited in patents are making measurable contributions to a nation's technological progress. Introduction It seems quite appropriate for b i b l i o m e t r i c techniques to play a m a j o r role in the application o f the U . S . G P R A ( G o v e r n m e n t P e r f o r m a n c e and Results Act) to scientific research. M a n y b i b l i o m e t r i c techniques w e r e d e v e l o p e d with the evaluation o f research as a primary goal, especially e v a l u a t i o n o f research departments, international aggregates, for which there are virtually no institutions, and other quantitative alternatives.1 In this paper we will discuss three different categories o f b i b l i o m e t r i c analysis: literature b i b l i o m e t r i c s , patent bibliometrics, and linkage bibliometrics, all o f which may be used in establishing quantitative indicators o f both the outputs and the outcomes o f g o v e r n m e n t p r o g r a m s . 0138- 9130/96/US $ 15.00 Copyright 9 1996 Akaddmiai Kiadr, Budapest All rights reserved F. NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES The drive to develop techniques for evaluating research programs arose in the first years of financial stress following the post World War II period of unlimited science growth. As a result of the development of radar, atomic energy and other technological wonders in the war years, the essential contribution of science to technology became apparent, and of technology to national policy, and the broad support of science became a national goal. Inevitably, of course, the growth in support had to level off, and the scientific community had to face the stressful prospect of limited funding. The challenge of assuring that the output of publicly supported science was worthy of the dollars being devoted to it was attacked in two elegant papers published by Alvin Weinberg in the early 1960's. 2,3 Weinberg set out to systematize the criteria for scientific choices. Weinberg categorized the criteria as internal and extemal. The internal criteria deal with the question "Is it good science?" whereas the external criteria deal more explicitly with what kind of impact that particular area of science was having. It is very important to note that he felt that these two sets of criteria, internal and external, should be viewed separately, a point that seems to be lost in much of the current discussion of program evaluation. Weinberg is an eloquent writer and we will basically summarize what he said in his own words, by excerpting sections from his 1963 paper. "As science grows, its demands on our society's resources grow. It seems inevitable that science's demands will eventually be limited by what society can allocate to it. We shall then have to make choices. These choices are of two kinds. We shall have to choose among different, often incommensurable, fields of science -between, for example, high-energy physics and oceanography or between molecular biology and science of metals. We shall also have to choose among the different institutions that receive support for science from the government - among university, governmental laboratories and industry. The first choice I call scientific choice; the second, institutional choice. My purpose is to suggest criteria for making scientific choices - to formulate a scale of values which might help establish priorities among scientific fields whose only common characteristic is that they all derive support from the government." "Internal criteria are generated within the scientific field itself and answer the question: How well is the science done? External criteria are generated outside the scientific field and answer the question: Why pursue this particular science? Though both are important, I think the external criteria are the more important." "Two internal criteria can be easily identified: (1) Is the field ready for exploitation? 294 Scientometrics36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES (2) Are the scientists in the field really competent?" "Three external criteria can be recognized: (1) technological merit (2) scientific merit, and (3) social merit." The first is fairly obvious: once we have decided, one way or another, that a certain technological end is worthwhile, we must support the scientific research necessary to achieve that end." "The criteria of scientific merit and social merit are much more difficult: scientific merit because we have given little thought to defining scientific merit in the broadest sense, social merit because it is difficult to define the values of our society." "Relevance to neighboring fields of science is, therefore, a valid measure of the scientific merit of a field of basic science." "... that field has the most scientific merit which contributes most heavily to and illuminates most brightly its neighboring scientific disciplines." The imperative for justifying scientific investment also led to the first TRACES study (Technology in Retrospect and Critical Events in Science) which tried to link, in a relatively systematic way, the development of important innovations to the science base. 4 The initiation of the Science Indicators Report Series by the National Science Board in 19725 was another step in responding to the need for quantitative output and outcomes indicators. In the subsequent sections of this paper we will show how the three types of bibliometrics, literature, patent and linkage, can all be used to address two categories of GPR-related questions: is the work being done by an agency good science, efficiently and effectively done, and is it important from a technological viewpoint, in essence Weinberg's internal and external criteria. We will discuss both some of the basic indicators that can be developed, and some of their limitations. However, at the outset we want to make an extremely important point about the skewness of virtually all bibliometric distribution, and how that must be carefully taken into account in any analysis and interpretation of bibliometric data. 6 The fundamental point is that virtually every distribution of scientific productivity or impact is highly skewed, with a small number of highly productive scientists, hot areas, research institutions, etc., and a relatively large number of institutions and producers of much smaller impact. The distributions are more often logarithmic than linear, with difference factors of 10 to 100 between the most productive entities and the least productive entities, in any scientific or technological distribution. Scientometrics 36 (1996) 295 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES For example, the influence of individual scientific journals - where influences is defined as the weighted average number of citations per paper in a journal - is distributed logarithmically, over a range of more than 100 between the most influential, most cited journals in a subfield and the least influential, least cited journals in a subfield. 7 Similarly, recent data has shown that the number of patents being produced by individual inventors, even within a single company, is logarithmically distributed, just as the original Lotka's Law showed for scientific papers. 8,9 The reason this is very important GPRA related work is that for one individual, or even a small group, a very large amount of the ultimate impact on society may be centered around a few relatively rare, extremely high impact discoveries, papers, inventions, etc. Further, in highly skewed distributions the means and the medians can be quite different, and one may often want to do statistical testing in the logarithm of the distribution, rather than in the distribution itself, in order to have something even approaching a normal curve. Therefore, in all of the subsequent discussions of publication and citation rates, it must be kept in mind that comparisons should always take skewness into account, and that differences of five-to-ten percent in impact between two research institutions are almost certainly insignificant in a realm where the differences between two research institutions are often factors of five-to-ten or more. Within all of the bibliometric analyses, of course, there are a series of basic tenants that are widely and almost universally accepted. The tenants include: 1. Counts of the number of papers, and the numbers of patents, provide basic indicators of the amount of scientific and technological productivity. 2. Counts of citations to these papers and patents, and from patents to papers, provide indicators both of the quality (impact) of research, and of the linkage between basic and applied research, between one subfield and another, and between technology and science. 3. Counts of coauthorships, and especially international coauthorships, are an indicator of quality, and that scientists who cooperate with their colleagues in other institutions and overseas are more likely to be doing quality research than those that are relatively isolated. 4. There are natural differences in citation and publication pattems across research and technology, unrelated to quality of the science, and one must be extremely careful in doing comparisons of any bibliometric parameters without fully adjusting for these differences: for example, between the citation parameters in a relatively lightly citing subfield such as acoustics and a very heavily citing subfield such as biochemistry and molecular biology. 296 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES Literature indicators for GPRA The bibliometric indicators which have been in use for the longest time are, of course, literature indicators: indicators of the scientific performance of organizations, agencies and countries based on counts of publications and citations in the scientific literature. These have been used most extensively in the United States in various studies done in the 1970's and 1980's for the National Institutes of Health 1~ and, of course, in the pioneering work at the National Science Foundation in the development of the Science Indicators series, starting with Science Indicators 1972, ongoing to Science and Engineering Indicators-1996, which has just been published. Amongst the literature indicators which are relevant to Weinberg's internal criteria are the number of publications, publications per dollar, the number of citations received by the papers, as well as various influence and impact surrogates for citation rate. In addition to those basic productivity and impact measures, another measure applicable to the internal criteria of whether an agency is producing good science is coauthorship, since it is widely accepted that having papers coauthored with one's colleagues at prestigious universities and research institutes, and with colleagues overseas, are indicators of acceptance within the scientific community. The number of papers produced by a given research institution is certainly a measure of its output, and a very basic one for GPRA. In most bibliometric studies of the physical, biological and mathematical science this is confined to articles, notes and reviews in refereed research journals, such as those covered in the Science Citation Index (SCI). This measure has particular significance when it is placed within the context of an organization's budget. Figure 1 shows such a placement, adapted from a paper published in 1976, graphing the number of biomedical papers published by the top 122 U.S. universities, compared to the amount of funds these schools received from NIH three years before. 11 The correlation, of course, is extremely high, approximately r =: 0.95, indicating that virtually all variation in the number of biomedical papers produced by those institutions is fully accounted for by the funding received: large and small universities produce essentially the same number of papers on a per dollar of funding basis. In general, we have never found that the size of an institution is of any significance in measurement of either the productivity or the quality of its research output. Productivity and quality vary widely, but are not primarily driven by organizational size. Small, highly specialized institutions can produce papers of just as high quality papers per funding increment as large, well known institutions. All bibliometric indicators of the quality of research produced are normally based on either direct citation counts, or various citation surrogates such as journal influence Scientometrics 36 (1996) 297 F. NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES or journal impact factor, and all are subject to one extremely important constraint: the data must be normalized for differences in field, subfield, and sometimes specialty parameters. Citation densities, that is the number of references per paper, the number of times a paper is cited, and time lags all vary widely from one field to another, and one subfield to another, and sometimes even within a subfield by specialty area. As a general rule, citation densities are highest in very hot and fast areas of science such as molecular biology and genetics, and much lower and slower in some of the more traditional areas of astronomy, descriptive biology, mathematics, and so forth. Citation patterns are also quite light in most of the engineering fields, and very different in the Social Sciences. 800 700 600 ll* S s 500 _~ 4oo g 3oo s 9 < 200 ~ 9 ..'". ," 100 0 99 9 D t, . "-eeq9 b s9 * ! I i i i I, i I 2000 4000 6000 8000 10000 12000 14000 16000 Average NIH Constant 1967 Dollars (thousands), 1965-1969 Fig. 1. Publication output v e r s u s NIH funds for 122 universities Perhaps the most fundamental challenge facing any evaluation of the impact of an institution's programs or publications is to properly normalize and adjust for field and subfield differences. This point is illustrated by Fig. 2, which shows the average citations per paper received in a five year window for papers in biomedical subfields, where subfield is, in this case, CHI's division of approximately 1,300 biomedical journals into 49 different subfields. Obviously, the factor of 4 or more between the citation frequency of papers in the most highly cited subfield biochemistry and molecular biology compared to the ~r;D,,tnmatrie~ ~6 (1996) F, NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES least highly cited specific biomedical subfield pharmacy indicates that citation studies of biomedical publication performance must be very carefully adjusted for differences in the subfield. For example, a highly prestigious paper in a lightly citing subfield may be less frequently cited than an average paper in a highly citing subfield, and it is simply not valid to directly compare citation counts in one of the lightly citing clinical subfields with citation counts in the much more highly citing basic subfields. 5 Year Citing Window) o o Fig. 2. Cumulativecites per paper to 1986 Clinical Medicine & Biomedical Research papers from 1986-91 papers by subfield One type of normalization sometimes done, with which we totally disagree, is to divide a papers' citation count by the average citation count for papers in the specific journal in which the paper appeared. This has the perverse effect of penalizing a research institution that encourages its scientists to publish in the most prestigious and best journals in its field. Scientometrics 36 (1996) 299 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES The correct way to do citation normalization is on a subfield basis, dividing citation counts by subfield analysis. In highly precise studies one should further adjust for the citation frequencies within specialized areas. Coauth0rship rates may be used to determine another internal indicator of performance, based on the generally accepted notion that scientists who coauthor with people outside their institutions, and especially with their colleagues overseas, are the scientists who are the most widely accepted in their community. A particularly interesting demonstration of this is contained in a study done for the European Community, which showed that Community papers that were coauthored with scientists outside the home country were more than twice as highly cited as those that were authored at a single institution within a single country; papers coauthored within one country were cited intermediate between the single institution papers and the internationally coauthored papers. 12 Coauthorship rates should always, of course, be adjusted for the dynamic changes that are occurring in coauthorship, and for the size of countries. International coauthorship has been increasing steadily in the United States and everywhere else, and is quite field dependent, in complicated ways. For example, although institutional coauthorship rates are generally higher in the field of Clinical Medicine than in the field of Biomedical Research, international coauthorship is higher in Biomedical Research than in Clinical Medicine. It is extremely important that one adjust coauthorship data for fields and subfield differences and, of course, year, since coauthorship rates have been increasing monotonically for more than 20 years. The final internal criteria for bibliometric performance to be discussed here relates to the concept of research level. CHI Research has classified each of some 3,000 journals covered in the Science Citation Index (SCI) on a research level scale ranging from 1 (most applied journals) to 4 (most basic journals). This scale allows one to characterize the degree of basicness of a research program, or the publication of a research institution. The four research levels and prototype journals are: Level 1 Applied Technology (Clinical Observation in Biomedicine) Engineering-Technological Science (Clinical Mix in Biomedicine) J Iron & Steel Inst J Am Med Assn J Nuc Sci & Tech Proc IEEE New Eng J Med Level 3 Applied Research (Clinical Investigation in Biomedicine) J Appl Phys Cancer Res Level 4 Basic Scientific Research Phys Rev J Am Ch Soc J Biol Chem Level 2 300 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES A particularly telling application of this is illustrated in Fig. 3, redrawn from a paper published almost 20 years ago, which shows the average research level for the publications of the 95 largest U.S. hospitals, medical schools, clinics, and research universities. Also shown there is the average research level for papers published by the U.S. National Institutes of Health (NIH) Intramural program. The figure quite graphically illustrates the usefulness of this indicator: the average level of the papers from the hospitals, clinics and medical schools is, for every single one, more clinical than the average level of the papers from the major universities. Furthermore, the papers of the NIH Intramural program lie right in the middle of the distribution: the NIH Intramural program is more clinical than any university program, more basic than any hospital or medical school program, and, in fact, very beautifully spans the two constituencies which the intramural program at NIH serves so well. The construction of indicators relevant to Weinberg's external criteria within the scientific literature are normally based on cross-citing patterns, either citations to a basic subfield from an applied subfield, or citations to university and other public science from industrial, private science. An illustration of this kind of indicator is shown in Fig. 4 which shows the trend in citation from industry papers to university sector papers in all fields. It is quite clear that industrial science continues to heavily use the more basic papers produced at the universities, and the universities have, if anything, a growing role in producing the science base of industrial technology. There are, of course, various technical problems which impact any bibliometric evaluation. One of the most important problems, especially for program evaluation, is that of constructing the basic program bibliography. For a support program where there are many different grantee institutions, agency records are almost always quite incomplete, and usually in quite non-standard form. Just establishing the journal, volume, page and year for the bibliography of the papers supported in a typical U.S. government agency program is a massive job, and often takes many person months of unification, yet such a standard form is absolutely imperative for any kind of analysis, before you can even establish the field and subfield of the papers, or whether the publications are, in fact, papers in refereed journals, in journals covered by the SCI, and so forth and so on. Even the construction of an institutional bibliography is not trivial. Papers from Harvard University contain literally hundreds of different variations of the name Harvard University, and finding all of them in the literature is a substantial challenge. In recent years the Institute for Scientific Information (ISI) has done some standardization of these names in the SCI, and that is a help in using that data. However, many universities contain hospitals and research labs that have names other Scientometrics 36 (1996) 301 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES than that of the university, and those addresses often do not explicitly mention the university name, and so that a substantial amount of research is necessary to obtain a decent bibliography. In creating the U.S. Science Indicators over the years, CHI has created a thesaurus of over 600,000 different names which are associated with 3,500 major U.S. research institutions, and it. takes a substantial amount of time, effort and care to construct an institutions bibliography from the many names under which its scientists will publish. MEDICAL SCHOOLS, HOSPITALS, AND CLINICS ] UNIVERSITIES 15 E 0 vltn i 21o (1 = Most Clinical) 2's 37oI NIH INTRAMURAL 3'5 ,'o (4 = Mwst Basic) Average Level Fig. 3. Average level for biomedical papers of NIH and the 95 largest U.S. universities, medical schools, hospitals and clinics. Source: The Intramural Role of the NIH as a Biomedical Research Institute, F. NARIN, S. B. KEITH Federation Proceedings, Vol. 37, No. 8, June 1978 Another important factor that must be carefully normalized for are the time effects. Specifically, of course, citations accumulate over time, and as it was mentioned earlier, citation data must be over comparable time periods, and within the same subfield or area of science. However, in addition to that, the time patterns of citation are far from uniform across country, and any valid evaluative indicator must use a fixed window and be appropriately normalized. To illustrate this Fig. 5 shows a phenomenon we call the "Citation Time Anomaly", namely the anomaly in the percent of references from a country's papers to earlier papers for the same country. The 302 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRICPERFORMANCE MEASURES ordinate is the percent of the references from each country's papers which are to the same country's papers, published in the prior year indicated on the abscissa. 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 i i : = ', [ t Industry Sector University Sector 1985 Cixl~ 1981-83 Federal Gov Sector 1986 Ctt.'~ 1982-84 ~] 1997 Cili,~l 1983-88 [ ] 1989 CainQ 1984-86 tr---I C~ 1989 Crting 1985-87 [ [~ 1990 Cklng 1986-88 [ ] 1991 Citing 1997-89 FFRDC Sector FFRDC = Federarly Funded Research & Development Cenler Fig. 4. Percent of industry paper cites to various sectors' papers 1981 journal set Clearly the effects are extremely strong, with some 52 percent of the German physics references to German papers in the same year, only 40 percent in the first year down to 21 percent for papers published ten years earlier. Similarly in U.S. Clinical Medicine, 83 percent of the essentially immediate references are to U.S. papers, whereas after ten years only percent are to U.S. papers. This effect is quite general, and shows just as strongly for almost any country and for almost any field or subfield, and obviously could have major effects on an evaluation. Specifically, if one is evaluating the short time citation rate to non-U.S, papers in a database that is as heavily U.S. as the Science Citation Index is, then the short term Scientometrics 36 (1996) 303 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES indicators will give far lower citation ratios to the non-U.S, papers than long term data, since many of the quick U.S. references are to U.S. papers. This, of course, would effect any short term citation indicator substitute, such as the impact factor, and argues strongly for being very careful to adjust for the citation window in any kind of cross national Citation analysis. For comparison of institutions within one country this is not as important, but for cross national comparisons it is quite a striking, important phenomenon. Percent of Cites frc~m U . S . P a p e c s t o U . S . C l i n i c a l M e d i c i n e Papers 85 - - 1 9 9 3 8o -- -- -- 1991 ........ 70 __ 1989 - 65 line represen~ o n e d t i n o year . . . . 1987 ...... 1988 N g m b e r e~ Ymlrs P l k x t o C ~ l g Year Pmcent of C#es h'om W. Gmmm P~oem to W. Gmman 1993 - -........ 30 1991 1989 [ [ 20 10 0 Each line represents o n e chino yee,r ~ ~ J i i i t i i i t i N u m b e r o f Years Pdor t o CRIng Yelw Fig. 5. "Citation time anomaly" 304 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES Linkage indicators for GPRA A series of very direct indicators of the impact that an agency's science is having on industrial technology may be developed from linkage statistics, by characterizing the citations from patented industrial technology to the agency's scientific research papers. The analysis of linkage from technology to science has been underway for a number of years, and is becoming a crucially important means of demonstrating the importance of science via Weinberg's external criteria. This type of analysis of industrial impact is now being used at a number of research agencies. 13-15 In a paper published in 1992 we summarized the status of linkage knowledge at that time. 16 When a typical U.S. patent is issued it contains on its front page seven or eight references to earlier U.S. patents, one reference to a foreign patent, and 1.3 non patent references (NPR), which are reference to material other than earlier U.S, or foreign patents. Of these non patent references approximately 43 percent or a total of 60,000 per year in recent years are to scientific research papers in the central core of 3500 or so journals covered by the Science Citation Index. There are a number of exceedingly important aspects of this general relationship from a GPRA viewpoint. First, technology-to-science linkage is increasing rapidly, with a two to three-fold increase in citations from patents to papers over the last six years, as shown in Fig. 6, which summarizes the citations from U.S. patents to U.S. authored papers in two 11 year citation windows. The number of papers cited has more than doubled, the number of citations to these papers has increased similarly, and the number of support sources on the cited papers has more than tripled. This rapidly increasing citation is to rather basic, mainstream science. The citation is heavily to Level 3 and 4 Applied and Basic research papers; in the Biomedical and Chemical areas predominately to the Level 4, very basic journals; in the electrical and computer industry more mixed, with a heavy citation to Level 3 Applied Physics journals. This citation is also quite rapid, with the age of the cited scientific papers almost as young as the age of the cited patents. 17 It is largely to public science, to scientific work done at universities or government laboratories, and supported by government agencies. The top 10 research support agencies cited in 1993 and 1994 U.S. patents include 6 NIH Institutes, the National Science Foundation, the American Cancer Society, the Department of Energy, and the Navy.. This data clearly demonstrates the crucial contribution that public science makes to patented industrial technology. Scientometrics 36 (1996) 305 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES # of Support Sourcas Cited * 80000 r ~ all cited papers 9 supported papers 70000 ; [~ unsupported papers i ] 60000 ~- 50000 40000 30000 i I i ! 20000 T~ 10000 1957/88 1993/94 citLng citing 1975-85 1981-91 1987/85 1993/94 citing citing 1975-85 1981-91 1997/85 1993/94 citing citing ~975-85 1981-91 9 Includes only those papers found in the librar'/ Fig. 6. Citations from U.S. patents to U.S.-authored SCI journal papers Patent indicators for GPRA The patents generated by various government laboratories, and by the grantees and contractors of government agencies, may also be used in performance assessments, just as papers are. As with papers, patents data may be used to construct indicators of internal and external performance for an agency, internal in that their existence, per se, is an indicator of contributions to technology, and external in that the impact of those patents on other technology, and ebpecially on U.S. private sector technology, is an indicator of the contribution of those technological inventions to society. 306 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES The most basic of the internal indicators are, of course, the numbers and distributions of patents across different technologies. As a general rule government agencies have been relatively steady in their patenting over the last few decades, whereas university patenting has been increasing quite rapidly. Figure 7, for example, shows the rate of patenting by the U.S. government agencies and U.S. universities . Clearly, 9niversities have been increasing their patenting at a very much more rapid rate than~tlae government agencies. One general point about university patenting in the U.S. is that it is rather heavily concentrated in biotechnology, one of the very, very hot areas of U.S. technology. Other properties of those patents, particularly their relatively high science linkage, also indicate that university patents are in leading edge areas of technology. 2500 US Academic 21XX) ~ 1513t3 i ///1 m | z 1000 \ \ \\ 500 0 1997 j ~ F ! i I I ' 1988 1989 1990 1991 1992 1993 1994 1995 Fig. 7. U.S. patenting trends As a measure of the quality of patents one can compute various citation indicators. The one we tend to use most is Current Impact Index (CII), which is, in essence, the citation rate for the last five years of an agency's patents, as cited by the most current Scientometrics 36 (1996) 307 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES year. For example, in computing CII for 1995 we will take all the agency's patents in each of the five years 1990-1994, note how many times each patent is cited, divide it by the expected citation rates for all patents in the same product group and year, and average those ratios, so that a CII equal to 1.0 is the expected value. Science Linkage (SL), the average number of times each patent references scientific materials, is another indicator of quality patents. For example, for the 5 years, 1991-1995, U.S. Navy patents have CII = 0.69 and SL = 1.09, while U.S. Dept. of Health and Human Services (largely NIH patents) have CII = 0.75 but SL = 8.10. While neither agencies' patents are highly cited (expected CII = 1.00) they are quite science-linked, especially HHS. Overall for all U.S. invented U.S. patents in the time period the SL = 0.99, so that both NIH and NRL are in relatively highly science linked areas with NIH, of course, far more heavily science-linked. Evaluating the efficiency of a government laboratory or program in producing patents is very problematic, and perhaps impossible. The R&D dollars associated with each patent produced in a large industrial laboratory tends to be in the one to five million dollar range, and we have observed similar ranges for some government laboratories. However, most government laboratories have fundamental missions far removed from the production of patentable technology, and in many cases the patents are the product of only certain parts of the labs. Therefore, comparisons of patents per R&D dollar are far more problematic than comparisons of papers per R&D dollar in the university environment, where publication is a central objective of the scientific process. There are a number of indicators of the external contribution that agency and laboratory patents make, both direct indicators in terms of licensing, and indirect in their contribution to the advances of technology in general. The indirect contributions can be tracked by looking at the pattern of citation to laboratories' patents, as is illustrated by Fig. 8, which shows citations from the top assignees citing to NRL patents, Another way of looking at the patent impact from an agency viewpoint is to look at the licensing information, at how many licenses the laboratory has obtained, and how much revenue is derived from those licenses. While we don't have detailed licensing revenue information, licensing activity is a reflection of the growing technology transfer activities of most U.S. labs and universities, and there is an interesting interaction that we have found between the bibliometric characteristics of patents and their licenses. In particular, we looked at the patents which had been licensed for most of the Navy laboratories, and found that there are two important bibliometric characteristics of licensed patents: first, these 308 Scientometrics 36 (1996) F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES patents are cited much more often than the other lab patents, and second, they tend to be much more science linked. This is illustrative of a rather interesting convergence of the internal and external criteria. Those patents which are most highly cited and most science linked, both internal indicators of the likely impact of that technology, are also the patents that tend to be most heavily licensed, and are therefore making the most ~lirect external contribution to the economy. I~ oro. .... '"J 16 SumJtomoElec t US Air .9 I,owa S~. Univ ]13_j AIcstel Alsthom CGE Philips i .... ..... ]/2"/ ;Si GE PLC Mitsubishi Elec Raytheon NASA Sumitorno Metals ,. [ ~1 ! ,~IL._ / .~ Northrup Grurnrnan 8P NRL 231 Patents ~ .. ./ / //' J/" /" "I'/ " ]j~/ 10 ToshibaNEC Fujitsu Fig. 8. Companies with 10 or more patents citing to 1985-89 NRL patents Which brings us to our final point. As was perceived at the very beginning of the modern age of science, the most important characteristic of research is the quality of the work. Highly cited, science dependent, quality patents, and the related basic scientific work, seem to have wide economic impact. This has also been Shown in the sample cases studied from the beginnings of modern bibliometric analysis, and now is Scientometrics 36 (1996) 309 F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES shown statistically in technology and in its interface with science: high impact, basic scientific research is the type of research directly driving our technological advancement. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. F. NARIN, "Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity", Report prepared for the National Science Foundation, Contract NSF C-627, NTIS Accession #PB252339/AS, (1976), 456pp. A.M. WEINBERG,Criteria for scientific choice II: The two cultures, Minerva, 3 (1964) 3-14. A.M. WEINBERG,Criteria for scientific choice", Minerva, 1 (1963) 159-171. IIT Research Institute, "Technology in Retrospect and Critical Events in Science (TRACES)". Report prepared for the National Science Foundation, Contract NSF-C535, (1968), 72pp. National Science Foundation, Science Indicators 1972, U.S. Government Printing Office, Reports of the National Science Board. P.O. SEGLEN,The skewness of science, Journal of the American Society for Information Science, 43 (9), (1992) 628-638. G. P1NSKI, F. NARIN, Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics, Information Processing and Management, 12 (5), (1976) 297-312. F. NARIN, A. BREITZMAN,Inventive Productivity, Research Policy, 24 (1995) 507-519. A.J. LOTKA, The frequency distribution of scientific productivity, Journal of the Washington Academy of Science, 16 (1926) 317-323. F. NARIN, H.H. GEE, "An Analysis of Research Publications Supported by NIH, 1973-1980", NIH Program Evaluation Report, U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health, (1986). J.D. FRAME, F. NARIN, NIH funding and biomedical publication output, Federation Proceedings, 35 (14) (1976) 2529-2532. F. NARIN, K. STEVENS, E. WHITLOW, Scientific cooperation in Europe and the citation of multinationally authored papers, Scientometrics, 21 (1991) 313-323. L.B. ELLWEIN, P. KROLL, F. NAR1N, Linkage between research sponsorship and patented eye-care technology, To be published (1996). J. ANDERSON, N. WILLIAMS, D. SEEMUNGAL, F. NARiN, D. OLIVASTRO, Human genetic technology: Exploring the links between science and innovation. To be published in Technology Analysis and Strategic Management, (1996). F. NARIN, "Linking Biomedical Research to Outcomes - The Role of Bibliometrics and Patent Analysis", Presented at The Economics Round Table, National Institutes of Health, (1995), 27pp. F. NARIN, D. OLIVASTRO, Status report - linkage between technology and science, Research Policy, 21 (3) (1992) 237-249. F. NARIN, E. NOMA, IS technology becoming science?, Scientometrics, 7 (1985) 369-381. 310 View publication stats Scientometrics 36 (1996)
