Bibliometric Performance Measures: Impact & Evaluation

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/242915900
Bibliometric performance measures
Article in Scientometrics · July 1996
DOI: 10.1007/BF02129596
CITATIONS
READS
200
692
2 authors:
Francis Narin
Kimberly Hamilton
Retired President, CHI Research, Inc
The Patent Board
96 PUBLICATIONS 10,802 CITATIONS
9 PUBLICATIONS 1,736 CITATIONS
SEE PROFILE
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
"UTILIZATION OF BASIC SCIENTIFIC RESEARCH AS EVIDENCED BY US PATENT CITATIONS" Coauthors Carpenter and Narin,
Research Management Magazine of IRI, XXIII-2, March 1980. View project
All content following this page was uploaded by Francis Narin on 18 February 2015.
The user has requested enhancement of the downloaded file.
Jointly published by Elsev&r Science Ltd, Oxford
and Akad~miai Kiad6, Budapest
Scientometrics,
Vol. 36, No. 3 (1996) 293-310
BIBLIOMETRIC PERFORMANCE MEASURES
F. NARIN, KIMBERLY S. HAMILTON
CHI Computer Horizons Inc., 10 White Horse Pike, Haddon Heights, NJ 08035 (USA)
r
(Received May 24, 1996)
Three different types of bibliometrics - literature bibliometrics, patent bibliometrics, and
linkage bibliometric can all be used to address various government performance and results
questions. Applications of these three bibliometric types will be described within the framework
of Weinberg's internal and external criteria, whether the work being done is good science,
efficiently and effectively done, and whether it is important science from a technological
viewpoint. Within all bibliometrics the fundamental assumption is that the frequency with which
a set of papers or patents is cited is a measure of the impact or influence of the set of papers.
The literature bibliometric indicators are counts of publications and citations received in the
scientific literature and various derived indicators including such phenomena as cross-sectoral
citation, coauthorship and concentration within influential journals. One basic observation of
literature bibliometrics, which carries over to patent bibliometrics, is that of highly skewed
distributions - with a relatively small number of high-impact patents and papers, and large
numbers of patents and papers of minimal impact. The key measure is whether an agency is
producing or supporting highly cited papers and patents. The final set of data are in the area of
linkage bibliometrics, looking at citations from patents to scientific papers. These are particularly
relevant to the external criteria, in that it is quite obvious that institutions and supporting
agencies whose papers are highly cited in patents are making measurable contributions to a
nation's technological progress.
Introduction
It seems quite appropriate for b i b l i o m e t r i c techniques to play a m a j o r role in the
application o f the U . S . G P R A ( G o v e r n m e n t P e r f o r m a n c e and Results Act) to scientific
research. M a n y b i b l i o m e t r i c techniques w e r e d e v e l o p e d with the evaluation o f research
as a primary goal, especially e v a l u a t i o n o f research departments,
international
aggregates,
for
which
there
are
virtually
no
institutions, and
other
quantitative
alternatives.1 In this paper we will discuss three different categories o f b i b l i o m e t r i c
analysis: literature b i b l i o m e t r i c s , patent bibliometrics, and linkage bibliometrics, all o f
which may be used in establishing quantitative indicators o f both the outputs and the
outcomes o f g o v e r n m e n t p r o g r a m s .
0138- 9130/96/US $ 15.00
Copyright 9 1996 Akaddmiai Kiadr, Budapest
All rights reserved
F. NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES
The drive to develop techniques for evaluating research programs arose in the first
years of financial stress following the post World War II period of unlimited science
growth. As a result of the development of radar, atomic energy and other technological
wonders in the war years, the essential contribution of science to technology became
apparent, and of technology to national policy, and the broad support of science
became a national goal. Inevitably, of course, the growth in support had to level off,
and the scientific community had to face the stressful prospect of limited funding.
The challenge of assuring that the output of publicly supported science was worthy
of the dollars being devoted to it was attacked in two elegant papers published by
Alvin Weinberg in the early 1960's. 2,3
Weinberg set out to systematize the criteria for scientific choices.
Weinberg categorized the criteria as internal and extemal. The internal criteria deal
with the question "Is it good science?" whereas the external criteria deal more
explicitly with what kind of impact that particular area of science was having.
It is very important to note that he felt that these two sets of criteria, internal and
external, should be viewed separately, a point that seems to be lost in much of the
current discussion of program evaluation.
Weinberg is an eloquent writer and we will basically summarize what he said in his
own words, by excerpting sections from his 1963 paper.
"As science grows, its demands on our society's resources grow. It seems
inevitable that science's demands will eventually be limited by what society can
allocate to it. We shall then have to make choices. These choices are of two kinds.
We shall have to choose among different, often incommensurable, fields of science
-between, for example, high-energy physics and oceanography or between
molecular biology and science of metals. We shall also have to choose among the
different institutions that receive support for science from the government - among
university, governmental laboratories and industry. The first choice I call scientific
choice; the second, institutional choice. My purpose is to suggest criteria for
making scientific choices - to formulate a scale of values which might help
establish priorities among scientific fields whose only common characteristic is
that they all derive support from the government."
"Internal criteria are generated within the scientific field itself and answer the
question: How well is the science done? External criteria are generated outside the
scientific field and answer the question: Why pursue this particular science?
Though both are important, I think the external criteria are the more important."
"Two internal criteria can be easily identified:
(1) Is the field ready for exploitation?
294
Scientometrics36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
(2) Are the scientists in the field really competent?"
"Three external criteria can be recognized:
(1) technological merit
(2) scientific merit, and
(3) social merit."
The first is fairly obvious: once we have decided, one way or another, that a
certain technological end is worthwhile, we must support the scientific research
necessary to achieve that end."
"The criteria of scientific merit and social merit are much more difficult: scientific
merit because we have given little thought to defining scientific merit in the
broadest sense, social merit because it is difficult to define the values of our
society."
"Relevance to neighboring fields of science is, therefore, a valid measure of the
scientific merit of a field of basic science."
"... that field has the most scientific merit which contributes most heavily to and
illuminates most brightly its neighboring scientific disciplines."
The imperative for justifying scientific investment also led to the first TRACES
study (Technology in Retrospect and Critical Events in Science) which tried to link, in
a relatively systematic way, the development of important innovations to the science
base. 4 The initiation of the Science Indicators Report Series by the National Science
Board in 19725 was another step in responding to the need for quantitative output and
outcomes indicators.
In the subsequent sections of this paper we will show how the three types of
bibliometrics, literature, patent and linkage, can all be used to address two categories
of GPR-related questions: is the work being done by an agency good science,
efficiently and effectively done, and is it important from a technological viewpoint, in
essence Weinberg's internal and external criteria.
We will discuss both some of the basic indicators that can be developed, and some
of their limitations. However, at the outset we want to make an extremely important
point about the skewness of virtually all bibliometric distribution, and how that must
be carefully taken into account in any analysis and interpretation of bibliometric data. 6
The fundamental point is that virtually every distribution of scientific productivity
or impact is highly skewed, with a small number of highly productive scientists, hot
areas, research institutions, etc., and a relatively large number of institutions and
producers of much smaller impact. The distributions are more often logarithmic than
linear, with difference factors of 10 to 100 between the most productive entities and
the least productive entities, in any scientific or technological distribution.
Scientometrics 36 (1996)
295
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
For example, the influence of individual scientific journals - where influences is
defined as the weighted average number of citations per paper in a journal - is distributed logarithmically, over a range of more than 100 between the most influential,
most cited journals in a subfield and the least influential, least cited journals in a
subfield. 7 Similarly, recent data has shown that the number of patents being produced
by individual inventors, even within a single company, is logarithmically distributed,
just as the original Lotka's Law showed for scientific papers. 8,9
The reason this is very important GPRA related work is that for one individual, or
even a small group, a very large amount of the ultimate impact on society may be
centered around a few relatively rare, extremely high impact discoveries, papers,
inventions, etc. Further, in highly skewed distributions the means and the medians can
be quite different, and one may often want to do statistical testing in the logarithm of
the distribution, rather than in the distribution itself, in order to have something even
approaching a normal curve. Therefore, in all of the subsequent discussions of publication and citation rates, it must be kept in mind that comparisons should always take
skewness into account, and that differences of five-to-ten percent in impact between
two research institutions are almost certainly insignificant in a realm where the
differences between two research institutions are often factors of five-to-ten or more.
Within all of the bibliometric analyses, of course, there are a series of basic tenants
that are widely and almost universally accepted. The tenants include:
1.
Counts of the number of papers, and the numbers of patents, provide basic
indicators of the amount of scientific and technological productivity.
2.
Counts of citations to these papers and patents, and from patents to papers,
provide indicators both of the quality (impact) of research, and of the linkage
between basic and applied research, between one subfield and another, and
between technology and science.
3.
Counts of coauthorships, and especially international coauthorships, are an
indicator of quality, and that scientists who cooperate with their colleagues in
other institutions and overseas are more likely to be doing quality research
than those that are relatively isolated.
4.
There are natural differences in citation and publication pattems across
research and technology, unrelated to quality of the science, and one must be
extremely careful in doing comparisons of any bibliometric parameters
without fully adjusting for these differences: for example, between the
citation parameters in a relatively lightly citing subfield such as acoustics and
a very heavily citing subfield such as biochemistry and molecular biology.
296
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
Literature indicators for GPRA
The bibliometric indicators which have been in use for the longest time are, of
course, literature indicators: indicators of the scientific performance of organizations,
agencies and countries based on counts of publications and citations in the scientific
literature. These have been used most extensively in the United States in various
studies done in the 1970's and 1980's for the National Institutes of Health 1~ and, of
course, in the pioneering work at the National Science Foundation in the development
of the Science Indicators series, starting with Science Indicators 1972, ongoing to
Science and Engineering Indicators-1996, which has just been published.
Amongst the literature indicators which are relevant to Weinberg's internal criteria
are the number of publications, publications per dollar, the number of citations
received by the papers, as well as various influence and impact surrogates for citation
rate. In addition to those basic productivity and impact measures, another measure
applicable to the internal criteria of whether an agency is producing good science is
coauthorship, since it is widely accepted that having papers coauthored with one's
colleagues at prestigious universities and research institutes, and with colleagues
overseas, are indicators of acceptance within the scientific community.
The number of papers produced by a given research institution is certainly a
measure of its output, and a very basic one for GPRA. In most bibliometric studies of
the physical, biological and mathematical science this is confined to articles, notes and
reviews in refereed research journals, such as those covered in the Science Citation
Index (SCI). This measure has particular significance when it is placed within the
context of an organization's budget. Figure 1 shows such a placement, adapted from a
paper published in 1976, graphing the number of biomedical papers published by the
top 122 U.S. universities, compared to the amount of funds these schools received
from NIH three years before. 11 The correlation, of course, is extremely high,
approximately r =: 0.95, indicating that virtually all variation in the number of
biomedical papers produced by those institutions is fully accounted for by the funding
received: large and small universities produce essentially the same number of papers on
a per dollar of funding basis. In general, we have never found that the size of an
institution is of any significance in measurement of either the productivity or the
quality of its research output. Productivity and quality vary widely, but are not
primarily driven by organizational size. Small, highly specialized institutions can
produce papers of just as high quality papers per funding increment as large, well
known institutions.
All bibliometric indicators of the quality of research produced are normally based
on either direct citation counts, or various citation surrogates such as journal influence
Scientometrics 36 (1996)
297
F. NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES
or journal impact factor, and all are subject to one extremely important constraint: the
data must be normalized for differences in field, subfield, and sometimes specialty
parameters. Citation densities, that is the number of references per paper, the number
of times a paper is cited, and time lags all vary widely from one field to another, and
one subfield to another, and sometimes even within a subfield by specialty area. As a
general rule, citation densities are highest in very hot and fast areas of science such as
molecular biology and genetics, and much lower and slower in some of the more
traditional areas of astronomy, descriptive biology, mathematics, and so forth. Citation
patterns are also quite light in most of the engineering fields, and very different in the
Social Sciences.
800
700
600
ll*
S
s
500
_~
4oo
g
3oo
s
9
<
200
~
9 ..'". ,"
100
0
99
9
D
t,
.
"-eeq9
b s9 *
!
I
i
i
i
I,
i
I
2000
4000
6000
8000
10000
12000
14000
16000
Average NIH Constant 1967 Dollars (thousands), 1965-1969
Fig. 1. Publication output v e r s u s NIH funds for 122 universities
Perhaps the most fundamental challenge facing any evaluation of the impact of an
institution's programs or publications is to properly normalize and adjust for field and
subfield differences.
This point is illustrated by Fig. 2, which shows the average citations per paper
received in a five year window for papers in biomedical subfields, where subfield is, in
this case, CHI's division of approximately 1,300 biomedical journals into 49 different
subfields. Obviously, the factor of 4 or more between the citation frequency of papers
in the most highly cited subfield biochemistry and molecular biology compared to the
~r;D,,tnmatrie~ ~6 (1996)
F, NARIN, K. S. HAMILTON:BIBLIOMETRICPERFORMANCEMEASURES
least highly cited specific biomedical subfield pharmacy indicates that citation studies
of biomedical publication performance must be very carefully adjusted for differences
in the subfield. For example, a highly prestigious paper in a lightly citing subfield may
be less frequently cited than an average paper in a highly citing subfield, and it is
simply not valid to directly compare citation counts in one of the lightly citing clinical
subfields with citation counts in the much more highly citing basic subfields.
5 Year Citing Window)
o
o
Fig. 2. Cumulativecites per paper to 1986 Clinical Medicine & Biomedical Research papers from 1986-91
papers by subfield
One type of normalization sometimes done, with which we totally disagree, is to
divide a papers' citation count by the average citation count for papers in the specific
journal in which the paper appeared. This has the perverse effect of penalizing a
research institution that encourages its scientists to publish in the most prestigious and
best journals in its field.
Scientometrics 36 (1996)
299
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
The correct way to do citation normalization is on a subfield basis, dividing citation
counts by subfield analysis. In highly precise studies one should further adjust for the
citation frequencies within specialized areas.
Coauth0rship rates may be used to determine another internal indicator of
performance, based on the generally accepted notion that scientists who coauthor with
people outside their institutions, and especially with their colleagues overseas, are the
scientists who are the most widely accepted in their community. A particularly
interesting demonstration of this is contained in a study done for the European
Community, which showed that Community papers that were coauthored with
scientists outside the home country were more than twice as highly cited as those that
were authored at a single institution within a single country; papers coauthored within
one country were cited intermediate between the single institution papers and the
internationally coauthored papers. 12
Coauthorship rates should always, of course, be adjusted for the dynamic changes
that are occurring in coauthorship, and for the size of countries. International
coauthorship has been increasing steadily in the United States and everywhere else, and
is quite field dependent, in complicated ways. For example, although institutional
coauthorship rates are generally higher in the field of Clinical Medicine than in the
field of Biomedical Research, international coauthorship is higher in Biomedical
Research than in Clinical Medicine. It is extremely important that one adjust
coauthorship data for fields and subfield differences and, of course, year, since
coauthorship rates have been increasing monotonically for more than 20 years.
The final internal criteria for bibliometric performance to be discussed here relates
to the concept of research level. CHI Research has classified each of some 3,000
journals covered in the Science Citation Index (SCI) on a research level scale ranging
from 1 (most applied journals) to 4 (most basic journals). This scale allows one to
characterize the degree of basicness of a research program, or the publication of a
research institution. The four research levels and prototype journals are:
Level 1
Applied Technology
(Clinical Observation in Biomedicine)
Engineering-Technological Science
(Clinical Mix in Biomedicine)
J Iron & Steel Inst
J Am Med Assn
J Nuc Sci & Tech
Proc IEEE
New Eng J Med
Level 3
Applied Research
(Clinical Investigation in Biomedicine)
J Appl Phys
Cancer Res
Level 4
Basic Scientific Research
Phys Rev
J Am Ch Soc
J Biol Chem
Level 2
300
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
A particularly telling application of this is illustrated in Fig. 3, redrawn from a
paper published almost 20 years ago, which shows the average research level for the
publications of the 95 largest U.S. hospitals, medical schools, clinics, and research
universities. Also shown there is the average research level for papers published by the
U.S. National Institutes of Health (NIH) Intramural program. The figure quite
graphically illustrates the usefulness of this indicator: the average level of the papers
from the hospitals, clinics and medical schools is, for every single one, more clinical
than the average level of the papers from the major universities. Furthermore, the
papers of the NIH Intramural program lie right in the middle of the distribution: the
NIH Intramural program is more clinical than any university program, more basic than
any hospital or medical school program, and, in fact, very beautifully spans the two
constituencies which the intramural program at NIH serves so well.
The construction of indicators relevant to Weinberg's external criteria within the
scientific literature are normally based on cross-citing patterns, either citations to a
basic subfield from an applied subfield, or citations to university and other public
science from industrial, private science. An illustration of this kind of indicator is
shown in Fig. 4 which shows the trend in citation from industry papers to university
sector papers in all fields. It is quite clear that industrial science continues to heavily
use the more basic papers produced at the universities, and the universities have, if
anything, a growing role in producing the science base of industrial technology.
There are, of course, various technical problems which impact any bibliometric
evaluation. One of the most important problems, especially for program evaluation, is
that of constructing the basic program bibliography. For a support program where
there are many different grantee institutions, agency records are almost always quite
incomplete, and usually in quite non-standard form. Just establishing the journal,
volume, page and year for the bibliography of the papers supported in a typical U.S.
government agency program is a massive job, and often takes many person months of
unification, yet such a standard form is absolutely imperative for any kind of analysis,
before you can even establish the field and subfield of the papers, or whether the
publications are, in fact, papers in refereed journals, in journals covered by the SCI,
and so forth and so on.
Even the construction of an institutional bibliography is not trivial. Papers from
Harvard University contain literally hundreds of different variations of the name
Harvard University, and finding all of them in the literature is a substantial challenge.
In recent years the Institute for Scientific Information (ISI) has done some
standardization of these names in the SCI, and that is a help in using that data.
However, many universities contain hospitals and research labs that have names other
Scientometrics 36 (1996)
301
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
than that of the university, and those addresses often do not explicitly mention the
university name, and so that a substantial amount of research is necessary to obtain a
decent bibliography. In creating the U.S. Science Indicators over the years, CHI has
created a thesaurus of over 600,000 different names which are associated with 3,500
major U.S. research institutions, and it. takes a substantial amount of time, effort and
care to construct an institutions bibliography from the many names under which its
scientists will publish.
MEDICAL SCHOOLS, HOSPITALS,
AND CLINICS
]
UNIVERSITIES
15
E
0
vltn
i
21o
(1 = Most Clinical)
2's
37oI
NIH
INTRAMURAL
3'5
,'o
(4 = Mwst Basic)
Average Level
Fig. 3. Average level for biomedical papers of NIH and the 95 largest U.S. universities, medical schools,
hospitals and clinics. Source: The Intramural Role of the NIH as a Biomedical Research Institute,
F. NARIN, S. B. KEITH Federation Proceedings, Vol. 37, No. 8, June 1978
Another important factor that must be carefully normalized for are the time effects.
Specifically, of course, citations accumulate over time, and as it was mentioned
earlier, citation data must be over comparable time periods, and within the same
subfield or area of science. However, in addition to that, the time patterns of citation
are far from uniform across country, and any valid evaluative indicator must use a
fixed window and be appropriately normalized. To illustrate this Fig. 5 shows a
phenomenon we call the "Citation Time Anomaly", namely the anomaly in the percent
of references from a country's papers to earlier papers for the same country. The
302
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRICPERFORMANCE MEASURES
ordinate is the percent of the references from each country's papers which are to the
same country's papers, published in the prior year indicated on the abscissa.
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
i
i
:
=
',
[
t
Industry Sector
University Sector
1985 Cixl~ 1981-83
Federal Gov Sector
1986 Ctt.'~ 1982-84
~] 1997 Cili,~l 1983-88
[ ] 1989 CainQ 1984-86
tr---I
C~ 1989 Crting 1985-87 [
[~ 1990 Cklng 1986-88
[ ] 1991 Citing 1997-89
FFRDC Sector
FFRDC = Federarly Funded Research & Development Cenler
Fig. 4. Percent of industry paper cites to various sectors' papers 1981 journal set
Clearly the effects are extremely strong, with some 52 percent of the German
physics references to German papers in the same year, only 40 percent in the first year
down to 21 percent for papers published ten years earlier. Similarly in U.S. Clinical
Medicine, 83 percent of the essentially immediate references are to U.S. papers,
whereas after ten years only percent are to U.S. papers. This effect is quite general,
and shows just as strongly for almost any country and for almost any field or subfield,
and obviously could have major effects on an evaluation.
Specifically, if one is evaluating the short time citation rate to non-U.S, papers in a
database that is as heavily U.S. as the Science Citation Index is, then the short term
Scientometrics 36 (1996)
303
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
indicators will give far lower citation ratios to the non-U.S, papers than long term
data, since many of the quick U.S. references are to U.S. papers. This, of course,
would effect any short term citation indicator substitute, such as the impact factor, and
argues strongly for being very careful to adjust for the citation window in any kind of
cross national Citation analysis.
For comparison of institutions within one country this is not as important, but for
cross national comparisons it is quite a striking, important phenomenon.
Percent
of Cites
frc~m U . S . P a p e c s t o U . S . C l i n i c a l M e d i c i n e
Papers
85
- - 1 9 9 3
8o
-- --
-- 1991
........
70
__
1989
-
65
line represen~ o n e d t i n o year
. . . .
1987
......
1988
N g m b e r e~ Ymlrs P l k x t o C ~ l g Year
Pmcent of C#es h'om W. Gmmm P~oem to W. Gmman
1993
- -........
30
1991
1989
[
[
20
10
0
Each line represents o n e chino yee,r
~
~
J
i
i
i
t
i
i
i
t
i
N u m b e r o f Years Pdor t o CRIng Yelw
Fig. 5. "Citation time anomaly"
304
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
Linkage indicators for GPRA
A series of very direct indicators of the impact that an agency's science is having
on industrial technology may be developed from linkage statistics, by characterizing
the citations from patented industrial technology to the agency's scientific research
papers. The analysis of linkage from technology to science has been underway for a
number of years, and is becoming a crucially important means of demonstrating the
importance of science via Weinberg's external criteria. This type of analysis of
industrial impact is now being used at a number of research agencies. 13-15 In a paper
published in 1992 we summarized the status of linkage knowledge at that time. 16
When a typical U.S. patent is issued it contains on its front page seven or eight
references to earlier U.S. patents, one reference to a foreign patent, and 1.3 non patent
references (NPR), which are reference to material other than earlier U.S, or foreign
patents. Of these non patent references approximately 43 percent or a total of 60,000
per year in recent years are to scientific research papers in the central core of 3500 or
so journals covered by the Science Citation Index.
There are a number of exceedingly important aspects of this general relationship
from a GPRA viewpoint. First, technology-to-science linkage is increasing rapidly,
with a two to three-fold increase in citations from patents to papers over the last six
years, as shown in Fig. 6, which summarizes the citations from U.S. patents to U.S.
authored papers in two 11 year citation windows. The number of papers cited has more
than doubled, the number of citations to these papers has increased similarly, and the
number of support sources on the cited papers has more than tripled.
This rapidly increasing citation is to rather basic, mainstream science. The citation
is heavily to Level 3 and 4 Applied and Basic research papers; in the Biomedical and
Chemical areas predominately to the Level 4, very basic journals; in the electrical and
computer industry more mixed, with a heavy citation to Level 3 Applied Physics
journals.
This citation is also quite rapid, with the age of the cited scientific papers almost as
young as the age of the cited patents. 17 It is largely to public science, to scientific
work done at universities or government laboratories, and supported by government
agencies. The top 10 research support agencies cited in 1993 and 1994 U.S. patents
include 6 NIH Institutes, the National Science Foundation, the American Cancer
Society, the Department of Energy, and the Navy.. This data clearly demonstrates the
crucial contribution that public science makes to patented industrial technology.
Scientometrics 36 (1996)
305
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
# of Support
Sourcas Cited *
80000
r ~ all cited papers
9 supported papers
70000
; [~ unsupported papers
i
]
60000 ~-
50000
40000
30000 i
I
i
!
20000 T~
10000
1957/88 1993/94
citLng
citing
1975-85 1981-91
1987/85 1993/94
citing
citing
1975-85 1981-91
1997/85 1993/94
citing
citing
~975-85 1981-91
9 Includes only those papers found in the librar'/
Fig. 6. Citations from U.S. patents to U.S.-authored SCI journal papers
Patent indicators for GPRA
The patents generated by various government laboratories, and by the grantees and
contractors of government agencies, may also be used in performance assessments, just
as papers are. As with papers, patents data may be used to construct indicators of
internal and external performance for an agency, internal in that their existence, per se,
is an indicator of contributions to technology, and external in that the impact of those
patents on other technology, and ebpecially on U.S. private sector technology, is an
indicator of the contribution of those technological inventions to society.
306
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
The most basic of the internal indicators are, of course, the numbers and
distributions of patents across different technologies. As a general rule government
agencies have been relatively steady in their patenting over the last few decades,
whereas university patenting has been increasing quite rapidly. Figure 7, for example,
shows the rate of patenting by the U.S. government agencies and U.S. universities .
Clearly, 9niversities have been increasing their patenting at a very much more rapid
rate than~tlae government agencies.
One general point about university patenting in the U.S. is that it is rather heavily
concentrated in biotechnology, one of the very, very hot areas of U.S. technology.
Other properties of those patents, particularly their relatively high science linkage, also
indicate that university patents are in leading edge areas of technology.
2500
US Academic
21XX)
~
1513t3
i
///1
m
|
z
1000
\ \
\\
500
0
1997
j
~
F
!
i
I
I
'
1988
1989
1990
1991
1992
1993
1994
1995
Fig. 7. U.S. patenting trends
As a measure of the quality of patents one can compute various citation indicators.
The one we tend to use most is Current Impact Index (CII), which is, in essence, the
citation rate for the last five years of an agency's patents, as cited by the most current
Scientometrics 36 (1996)
307
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
year. For example, in computing CII for 1995 we will take all the agency's patents in
each of the five years 1990-1994, note how many times each patent is cited, divide it
by the expected citation rates for all patents in the same product group and year, and
average those ratios, so that a CII equal to 1.0 is the expected value. Science Linkage
(SL), the average number of times each patent references scientific materials, is
another indicator of quality patents.
For example, for the 5 years, 1991-1995, U.S. Navy patents have CII = 0.69 and
SL = 1.09, while U.S. Dept. of Health and Human Services (largely NIH patents)
have CII = 0.75 but SL = 8.10.
While neither agencies' patents are highly cited (expected CII = 1.00) they are
quite science-linked, especially HHS. Overall for all U.S. invented U.S. patents in the
time period the SL = 0.99, so that both NIH and NRL are in relatively highly science
linked areas with NIH, of course, far more heavily science-linked.
Evaluating the efficiency of a government laboratory or program in producing
patents is very problematic, and perhaps impossible. The R&D dollars associated with
each patent produced in a large industrial laboratory tends to be in the one to five
million dollar range, and we have observed similar ranges for some government
laboratories. However, most government laboratories have fundamental missions far
removed from the production of patentable technology, and in many cases the patents
are the product of only certain parts of the labs. Therefore, comparisons of patents per
R&D dollar are far more problematic than comparisons of papers per R&D dollar in
the university environment, where publication is a central objective of the scientific
process.
There are a number of indicators of the external contribution that agency and laboratory patents make, both direct indicators in terms of licensing, and indirect in their
contribution to the advances of technology in general. The indirect contributions can
be tracked by looking at the pattern of citation to laboratories' patents, as is illustrated
by Fig. 8, which shows citations from the top assignees citing to NRL patents,
Another way of looking at the patent impact from an agency viewpoint is to look at
the licensing information, at how many licenses the laboratory has obtained, and how
much revenue is derived from those licenses.
While we don't have detailed licensing revenue information, licensing activity is a
reflection of the growing technology transfer activities of most U.S. labs and
universities, and there is an interesting interaction that we have found between the
bibliometric characteristics of patents and their licenses. In particular, we looked at the
patents which had been licensed for most of the Navy laboratories, and found that
there are two important bibliometric characteristics of licensed patents: first, these
308
Scientometrics 36 (1996)
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
patents are cited much more often than the other lab patents, and second, they tend to
be much more science linked. This is illustrative of a rather interesting convergence of
the internal and external criteria. Those patents which are most highly cited and most
science linked, both internal indicators of the likely impact of that technology, are also
the patents that tend to be most heavily licensed, and are therefore making the most
~lirect external contribution to the economy.
I~
oro.
.... '"J 16
SumJtomoElec
t US Air
.9
I,owa S~. Univ
]13_j
AIcstel Alsthom CGE
Philips
i
....
..... ]/2"/
;Si
GE PLC
Mitsubishi Elec
Raytheon
NASA
Sumitorno Metals
,.
[
~1
!
,~IL._
/
.~
Northrup Grurnrnan
8P
NRL
231 Patents
~
..
./
/
//'
J/"
/"
"I'/
"
]j~/
10
ToshibaNEC
Fujitsu
Fig. 8. Companies with 10 or more patents citing to 1985-89 NRL patents
Which brings us to our final point. As was perceived at the very beginning of the
modern age of science, the most important characteristic of research is the quality of
the work. Highly cited, science dependent, quality patents, and the related basic
scientific work, seem to have wide economic impact. This has also been Shown in the
sample cases studied from the beginnings of modern bibliometric analysis, and now is
Scientometrics 36 (1996)
309
F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES
shown statistically in technology and in its interface with science: high impact, basic
scientific research is the type of research directly driving our technological
advancement.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
F. NARIN, "Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of
Scientific Activity", Report prepared for the National Science Foundation, Contract NSF C-627, NTIS
Accession #PB252339/AS, (1976), 456pp.
A.M. WEINBERG,Criteria for scientific choice II: The two cultures, Minerva, 3 (1964) 3-14.
A.M. WEINBERG,Criteria for scientific choice", Minerva, 1 (1963) 159-171.
IIT Research Institute, "Technology in Retrospect and Critical Events in Science (TRACES)". Report
prepared for the National Science Foundation, Contract NSF-C535, (1968), 72pp.
National Science Foundation, Science Indicators 1972, U.S. Government Printing Office, Reports of
the National Science Board.
P.O. SEGLEN,The skewness of science, Journal of the American Society for Information Science, 43
(9), (1992) 628-638.
G. P1NSKI, F. NARIN, Citation influence for journal aggregates of scientific publications: theory, with
application to the literature of physics, Information Processing and Management, 12 (5), (1976)
297-312.
F. NARIN, A. BREITZMAN,Inventive Productivity, Research Policy, 24 (1995) 507-519.
A.J. LOTKA, The frequency distribution of scientific productivity, Journal of the Washington Academy
of Science, 16 (1926) 317-323.
F. NARIN, H.H. GEE, "An Analysis of Research Publications Supported by NIH, 1973-1980", NIH
Program Evaluation Report, U.S. Department of Health and Human Services, Public Health Service,
National Institutes of Health, (1986).
J.D. FRAME, F. NARIN, NIH funding and biomedical publication output, Federation Proceedings, 35
(14) (1976) 2529-2532.
F. NARIN, K. STEVENS, E. WHITLOW, Scientific cooperation in Europe and the citation of
multinationally authored papers, Scientometrics, 21 (1991) 313-323.
L.B. ELLWEIN, P. KROLL, F. NAR1N, Linkage between research sponsorship and patented eye-care
technology, To be published (1996).
J. ANDERSON, N. WILLIAMS, D. SEEMUNGAL, F. NARiN, D. OLIVASTRO, Human genetic technology:
Exploring the links between science and innovation. To be published in Technology Analysis and
Strategic Management, (1996).
F. NARIN, "Linking Biomedical Research to Outcomes - The Role of Bibliometrics and Patent
Analysis", Presented at The Economics Round Table, National Institutes of Health, (1995), 27pp.
F. NARIN, D. OLIVASTRO, Status report - linkage between technology and science, Research Policy, 21
(3) (1992) 237-249.
F. NARIN, E. NOMA, IS technology becoming science?, Scientometrics, 7 (1985) 369-381.
310
View publication stats
Scientometrics 36 (1996)