[MIMAS logo] ITAM Evaluation: The Impact of OpenURLs on End Users


Ken Eason1 and Susan Harker2

December 2003

1 The Bayswater Institute, London, W2 4RL
2 Department of Human Sciences, Loughborough University, LE11 3TU

Summary

In the past two years many universities and services have been implementing OpenURL technology that enables users to move `seamlessly' from one service to another. SFX technology is an example that allows users to follow up references and check whether they can access full text electronic versions of articles. This evaluation explores a number of implementations of SFX and reports user experiences and reactions to date. The results show that it is a service of great potential that would be highly valued by users. Responses to it have been favourable, particularly in those institutions that have implemented their own resolver and have access to a large number of electronic journals. Under these conditions users often find they can get to electronic full text in only a few keystrokes which they describe as magic. There are signs, however, that user responses are less favourable when the chances of success are much lower, e.g. in institutions without a local resolver and where access to electronic journals is limited. Under these circumstances users are often negative about the rate of success, the absence of early information, e.g. an abstract, the time that can be wasted and the experience of moving from service to service. Some recommendations are offered to ensure the high promise of this approach is sustained and that early and negative user experiences do not inhibit later take-up as the service improves.

1. Introduction

Within the broader ITAM (Implementing the DNER Technical Architecture at MIMAS) project the particular implementation that was likely to have a direct impact on end users was the adoption of OpenURL technology. A number of universities and services are implementing OpenURL technology which makes use of metadata to facilitate integration. It permits end users the experience of a `seamless' movement from the identification of an article in one service to obtaining a full text electronic copy from another service. There are currently about ten brands of OpenURL resolver available. This evaluation is primarily concerned with SFX because it is the brand that has been implemented in zetoc and in most of the universities for which data on user responses is available. Earlier evaluation work in zetoc (1) suggests that `joined up' services which enable them to go directly from a reference to electronic full text is a high priority for users. This report is an evaluation of how users have responded to early experiences of having SFX technology available to them for these purposes.

2. Methods

This evaluation is restricted in two ways. First, the provision of SFX and related technology is at an early stage. As a result it is not fully implemented in many institutions and the provisions vary widely from one institution to another. Some have hosted services that link to their own resources. Some use national services and, in most cases, this means that use of SFX leads to services that may have electronic full text but there is no guarantee the user will have access rights. These variations mean that user experience of SFX may differ widely and it will be necessary for the evaluation to take account of the nature of the service available to any specific end user.

Second, it has not been possible to mount a primary study of user responses to these developments. This report is based on secondary data collected from a number of universities. There is therefore some unevenness in the data that was collected. A trial was also undertaken to test the level of service available at one university.

The early stage of implementation of these services and the unevenness of the data mean that this evaluation is best interpreted as an early indication of some of the user issues attendant upon the adoption of this kind of technology. It is inevitably a somewhat opportunistic evaluation and cannot claim to have systematically captured the experiences and views of a representative sample of users. The evaluation is to be regarded as an exploratory study that serves to identify the research questions that need to be addressed in a more systematic survey with primary data from users. Such a survey will be possible when users have had more experience of services that make use of OpenURL technology.

3. The Sample

The sample from which data has been extracted is from two sources:-

These sources of data give the following coverage. The institutions are divided into those that have established a hosted resolver service, e.g. SFX and 1CATE, and those that rely on national services such as zetoc.

UniversityImplementation Views by users / usage data? Data from the zetoc evaluation (No. of users)
Hosted Resolver
LoughboroughMetalib/SFXFocus Groups11
Derby1CATE (Oct. 2002) No
Royal HollowaySFX (Aug 2001) MetaLib (Sep.2003) 4
East AngliaSFX (Jan 2002) Metalib (April 2002) No
Hosted by MIMAS/Default Resolver
BathSFX (Nov.2002)SurveyNo
BradfordSFX (July 2002) MetaLib (Aug. 2003) No
PlymouthSFX June 2002 7
ManchesterSFX Dec 2001 6
King's LondonSFX 9
Others
EdinburghLinkFinder Plus (Not yet in service) No
BangorWeb-bridge (Feb 2003) No
WestminsterOlive Project No
Zetoc Survey
Other universities and colleges etcZetoc enhancements  193

Table 1. Sources of Data

The data that is available makes possible a comparison between the user experience in institutions which have their own resolver service and those that rely on national services. There is a body of data about Loughborough University and as a result it has been the subject of a separate case study reported in Appendix A.

4. Results

4.1 The Need for the Service

The attendees at the Talking Shops were very positive about OpenURL technology such as SFX. They report that their user communities are seeking `join up', i.e. to be able to get from reference details to abstracts and full text in electronic form without leaving their workstations. Where librarians are able to report data from their users this contention is borne out. Stubbings (5) reports the findings of three focus groups in which the staff and PhD students of three departments were given hands-on experience of MetaLib (a portal which provides access to all the electronic resources of the university and a route to SFX). 70% thought it was an easier way of discovering what was available than using the existing library database. Bath University conducted a survey of 39 users. 21 reported trying the SFX button and 18 wanted the library to continue to provide the service. All of the libraries report a steady increase in the usage statistics for these services since they were introduced.

The need for the service is substantiated by the results of the second zetoc questionnaire survey. Asked to assess which of a number of challenges were most important for the electronic services community to resolve, users (n=146) gave the top rating to 'making it easier to get from search results to full text.' Other results show that the users in this case meant electronic full text. There can be little doubt that users would very much value a service that enables them to go from reference details to electronic full text of articles.

4.2 Service Delivery

OpenURL technology is new and has been gradually implemented over the past two years. As a consequence what is available to end users varies. It should be recognised that the services which use it are rapidly evolving in both form and availability. It is useful to distinguish four routes by which SFX in particular is being made available to users.

4.2.1. Some Universities have purchased SFX or related technologies and are using it to make their electronic resource collections available to their users. This means that when a user clicks on the SFX button to follow a particular reference they are often offered target services including a link to a full text version provided the university subscribes to such a resource or it is free. The users of this service should not see inaccessible links. In universities where there is a large collection of electronic journals this, in theory, means that users have access to the full text of thousands of journals (e.g. 5,600 at Royal Holloway, 4,000 at Bradford, 3,500 at Loughborough). However, to initiate the OpenURL source the library has to contact journal publishers and this takes time and resources and may still result in a failure to enable the automatic linking to the full text. There may therefore be gaps in the service the users experience.

4.2.2. An SFX service provided via a specific resource. For example, Web of Science provides an SFX link from its database. Zetoc provides an SFX service labelled as `More Information'. When an institution has provided the address to their OpenURL resolver to one of these source services, clicking on the Web of Science SFX button or the zetoc `More Information' link takes the user to their institution's OpenURL resolver. This gives them access to full text articles that are free or which their institution subscribes to. As in the first case above, they are not shown inaccessible links.

4.2.3. For users in UK academia whose institutions do not have an OpenURL resolver registered with zetoc the `More Information' link goes to a default OpenURL resolver, MDL LitLink. This identifies where a full text service (or other service) is available; it does not specify whether this service is available at the institution of the user. Consequently this service is given a `worth a try' tag because although a full text electronic version of an article may be available it may not be accessible by a particular user.

4.2.4. A Default Resolver Service provided by MIMAS, e.g. to Bradford, Bath, King's London and Manchester etc. In this case MIMAS sets up the SFX service for each University. This reduces the resources needed by each University to launch the service but is likely to be a coarser grained level of resolution with consequences for the quality of service to the end user. The default resolver service was set up as part of the ITAM project for the duration of the project only. Its purpose was to investigate the use of SFX, and also hosting SFX for several institutions. The Universities involved in this trial have now moved on to purchasing their own SFX resolvers, except for Bath and Manchester who have full SFX resolver services hosted by MIMAS

The different forms of delivery should mean that, whilst users in all universities and colleges could have some access to SFX and related forms of technology, it is likely to be experienced as a more effective service in some places than others. It is likely that the most effective services will be in universities with their own SFX provision (categories 1 and 2 above) and a rich collection of electronic journal resources. The less effective services will be experienced in those universities and colleges that have access to SFX via national resources and where they do not subscribe to large numbers of electronic journals. We might then predict that there will be a more positive reaction to these services from users in universities with hosted services than in those receiving national services. In addition it may be expected that users in those institutions with large holdings of electronic journals will respond more positively than those in institutions with limited electronic access rights.

4.3 Institutional Analysis of User Responses

The zetoc evaluation data provides a basis for comparing the effect of these services across different types of institution. It does not provide data to evaluate responses to all four of the routes identified in section 4.2. but it is possible to compare those universities with their own resolvers (4.1.1.) with those that do not. In addition it is possible to make some estimation of the effect of having a large or limited electronic journal collection. In table 2 the sample of 193 users has been divided into six types of institution, in terms of having specific implementations of SFX and likely access to electronic journals. The SFX Universities are those that have introduced hosted services (and have substantial electronic journal collections). The established universities do not have hosted services but may have substantial electronic journal collections. The new universities, research centres, FE and Other Colleges and NHS Trusts all have access to zetoc (and potentially other services with SFX) but are less likely to have large electronic journal collections.

InstitutionsUsers Zetoc ScoreZetoc Enhancements
 No.% Libs Overall ScoreUsageIntegr-ation BetterSameWorse Total
       No%No %No%No
SFX Universities34358.26.61.6 2693141328
Established Universities48708.97.9 1.025815161331
New Universities40607.87.00.8 145610401425
Research Centres18727.26.70.5 457343007
FE and Other Colleges101008.37.5 0.82293422297
NHS43616.45.80.68 4012600020
Total193627.86.81.079 67342954118

Table 2. A Comparison of User Responses Across Institutions

In addition to citing the number of users responding from each category of institution, the table notes the percentage of librarians within each group because librarians were shown in the full zetoc survey to be more active users than the general population of users. For some categories of institution in the sample, for example, users in the FE colleges, librarians dominate the sample. It is worthy of note that the sample from the SFX universities has the lowest percentage of librarians. The full zetoc study employed a usage score to indicate how active users were with the service. The overall zetoc usage score in Table 2 shows a high score across all categories, 7.8 compared with 6.0 in the full sample. This is probably correlated with the high percentage of librarians in this sample. The zetoc score is of significance because it shows that the sample is composed largely of `active' users compared with the `passive' users who were the majority in the full zetoc survey. The zetoc usage score has been sub-divided into a more specific usage score (use of the data base, alerts etc) and an integration score (getting to zetoc from other services e.g. Metalib, and exporting results to other services, e.g. via SFX or to Endnote etc.). In the present context it may be noted that the highest scores for integration are in the SFX and established universities where these opportunities are likely to be most available.

The columns `better', `same' and `worse' record the results of asking users to evaluate enhancements to zetoc, most of which were to provide OpenURL links between services. Not all users had tried these services but they were in greatest use in the university sector especially in the SFX universities. 93% of these users considered that the enhancements meant they received a better service. 81% of the users from established universities also felt a better service had resulted. The percentage of `better' drops quickly for the other categories and in most instances more than 50% of users feel the service is the `same' or `worse'. Although the numbers in the sample are small this does appear to indicate that there is real experience of a different quality of service across the different kinds of institution. The differences are in the direction predicted and seem to indicate that the changes made to the zetoc service have increased the differences in experience, i.e. it gives those in universities with rich electronic resources a better service whilst potentially increasing the frustration of those where there are limited resources. This conclusion is important but is based on rather weak data and needs to be substantiated by further research.

In order to learn more about the experiences of users in universities where SFX is most fully implemented, Loughborough University was used as a case study. Loughborough implemented SFX and Metalib in 2002 and its users have access to the full text of over 3,500 journals. Appendix A reports the history of this development at Loughborough and reports the experience of 11 users. Appendix B reports a small sample of trial uses of the service undertaken to reproduce the kind of results that users were reporting.

Of the 11 users in the Loughborough sample, 8 had used Metalib and 4 SFX. Those who used SFX were very pleased with it when it worked but disappointed by the limited degree of success and the time it took before they knew whether they would be successful or not. Those who had not used SFX were interested to hear about it and expected to follow it up. There were a number of comments that it was not obvious what SFX was and that there are so many new developments that it is difficult to find the time to check them all out. This sample consisted of a fairly active group of users who knew whether they could obtain full text for the journals they used regularly and who knew which target to visit to get them. They felt the SFX route would be most useful when they were exploring unfamiliar journals and that, at the moment, the success rate was not high enough to spend a lot of time on the service. A number of people commented that they really needed early access to an abstract in order to check whether it would be worthwhile to continue the search to see if full text was obtainable.

The trials reported in Appendix B involved searches on two topics carried out using SFX by two routes, via MetaLib and via the zetoc website. At the time of the search the Loughborough resolver was not accessible by the zetoc website (it has since been connected). The trial therefore was a comparison of two of the categories of service identified in section 4.2 as (4.1.1) where the user is informed only of those targets that are available as full text in the institution and (4.1.3) where they are informed of all sources of full text regardless of the access rights in the institution. The search topics were chosen to take the searcher to unfamiliar journals. The overall success rate in reaching full text was 19% and in reaching abstracts was 44%. The success rate in reaching full text was the same for each mode of access. The difference between the routes was that the user searched more pages when going via the zetoc website (because of searching a number of targets some of which were not accessible in the institution). However, more abstracts were accessed via the website. The searches visited 159 pages in total, 60 via MetaLib and 99 via the zetoc website.

This pattern of use and success rate seems to be typical of the results obtained by users in the survey who used these two modes to search for articles in journals that were unfamiliar to them. Users reported much higher levels of success when they sought full text from a journal they knew was available in one of the services to which their university subscribed. The 19% level of success achieved in these trials which looked at unknown journals was probably not sufficiently high to persuade most users to continue their use of the service, especially via the website which involves more potentially fruitless searching. It is difficult to assess from this data whether users would continue using the host resolver route (4.1.1) or the route to the local host resolver via zetoc (4.1.2) at this level of success. They would not achieve a higher level of success but would waste less time because they would be told more quickly whether a full text option was accessible.

4.5 Qualitative Data from Interviewees and Questionnaires

The users' experience of SFX and related technology was further explored by analysis of interviews and unstructured questionnaire comments from users across the universities in the sample. The following are quotes first from users in universities with hosted resolvers and secondly from users in other universities. In the interviews users were questioned specifically about their experiences of zetoc but they were also asked to describe their experiences of all the forms of OpenURL technology that they had encountered

4.5.1 Users in universities with their own resolvers

There is good evidence that the `joined-up service' is what people want. Here are some comments from users who are using SFX in universities that have their own resolvers.

When it gets you directly to a full text electronic publication it is magic. It saves all those trips to the library, photocopying, filling out ILL's etc.
A university teacher
I always use SFX now that I have learned what it does. It is great when you get straight to full text but often all I want is an abstract and I can quite often get that. SFX is not a good title though - I'm sure most people don't know what it does.
A postgraduate student

But sometimes the experiences are more frustrating than exciting

It can be very frustrating (using `More Information via zetoc) to keep clicking on all the links to get to the abstract or the full text electronically only to find at the end of it all that there is no abstract or the university does not have a subscription.
A university teacher
I often find it gets to a dead end, even when you know you have permission to have the full copy, and this can be very frustrating
A postgraduate student

Users are beginning to explore what they can do by using SFX through the local portal or by going directly to a target as opposed to the `More Information' route in zetoc

I have used the Zetoc ` More Information' facility occasionally in the past but I get to full text more frequently by going via the resources on Metalib.
A postgraduate student
I succeed in getting full text electronically in about 50% of cases. I know ScienceDirect is the route to important journals in my field. In areas I don't know I might use SFX or 'More Information' but it is less likely to be successful
A postgraduate student

4.5.2 Users in Established and New Universities

Users in other universities are also looking for a `joined up' service but tend to be more critical of what is available to them.

If I could just click on to the reference and link to a full copy of the article immediately it would be fine but I have to go through a complicated set of procedures to get access. I am not sure there are any benefits and I may switch it (zetoc) off.
Professor at a new university
My main requirement is to get to the abstract directly from the reference. It is often all I need. If I had to use a complex service only to be told I could not get a full text article to which I knew the college held rights I would be cross. My strategy is to store alerts and searches and then to check them with Medline or Web of Science because I know I can get abstracts at least. It avoids an awful lot of links that may not work.
Lecturer at an established university

Some users are pessimistic about whether it will ever be a help to them.

It would be an improvement if it were possible to go direct to electronic full text from reference details. But the library here is very poor on Politics journals so most things of interest to me we would not subscribe to so I would not get them anyway.
Professor of Politics in a new university

A lot of the comments are about the way in which people used particular resources that they knew were available to them. In some cases they could use SFX within a service. A frequent comment was that this was the way to avoid wasting time.

I use the SFX button virtually every time I use Web of Science and have always found it useful. I find it an absolutely essential complement to Web of Science
Lecturer in an established university
If I require the electronic full text of an article I go to the publisher's website rather than through the University web pages. We do not subscribe to those that are relevant to me.
Lecturer in a new university
I know how to reach full text versions of my main journals. The SFX route ought to be a useful alternate for accessing citations to otherwise unfindable research, but I haven't had much success with it so far.
Lecturer in an established university

One of the disconcerting things for users is the way they find themselves moving from service to service.

There are difficulties and frustration with the way the different services function, information they require and particularly how far they let you get before they declare you cannot have access to a full text electronic version
Research student in a new university
I was quite insulted by the messages I got saying I was not authorised to have full text. It felt as though I was doing something underhand. I'm not sure I'll use a service that treats me like this.
Professor in an established university

The way in which the target service tells users they cannot have access varies considerably. There are straightforward but not very informative messages:-

There are the technical messages

And there are those that might give offence

5. Discussion

There can be little doubt that SFX and related OpenURL technologies, which enable users to get `seamlessly' from reference details to full electronic texts, are potentially the answer to the dreams of many users. The evidence reported here reinforces this conclusion. If such technologies can be embedded in services effectively they will revolutionise the habits of many users. The problem at the present stage of development , as Ros Doig from Derby University suggested at the second Talking Shop, is to manage users' expectations. If they expect the service to always be `magic', and it is often unsuccessful and a consumer of time, they may become disillusioned. And they may, as they no doubt have with many other services, give up trying. In a world in which there is always a clamour of new services seeking the attention of users, it may be difficult to get such users back even when the service improves to a level that would make it worth their while. At the present stage of development of services it is important both to reach users and to avoid alienating them.

Although the results reported are, at best, patchy they suggest that the service in the universities with their own resolvers and access to large numbers of electronic journals, is approaching a level users find acceptable and worth persevering with. It seems that in universities without their own resolvers and limited access to electronic resources, only the really enthusiastic user will persevere with the service as it exists currently. However, the level of service a specific user can obtain is the result of several variables and is not easy to predict. Four variables are particularly important in determining the level of service:-

If all four of these conditions are satisfied, the chance of success is high. Any of them can causes the success rate to fall dramatically.

In addition to the factors that define the service there are factors about the user which will determine whether they are likely to make good use of the facilities that are available to them and whether they are likely to return to the service:-

There are signs in the sample that users are already beginning to work out their best strategy for maximising success and minimising time wasting. They tend to know, for example, whether their library subscribes to a service that can obtain full text for their mainstream journals. They may use SFX within these services to access full text. They are already beginning to define the SFX route provided by general services as a secondary resource to use for unfamiliar areas and to recognise that the success rate will be lower. Users seem to work out a `cost-benefit' strategy for themselves which seeks to maximise success and minimise failure and time wasting and they are taking on board the opportunities and limitations of SFX as currently provided in their location. The danger is that their strategy may exclude them from recognising when there is an improvement in the quality of the service.

An unknown that the evaluation has not been able to address is the attitude that will be taken to SFX by the `passive majority' of users. This survey reports the views of the `active minority' who are usually interested in new services and prepared to try them. Most users are, however, more passive, they stick with what they know and are late adopters of new services. It seems probable that the active users have a higher tolerance level of failure and of time wasting than their colleagues. We might predict therefore that, except in the SFX universities, the current levels of service are not yet satisfactory for most of the more passive users. It is interesting to note that most universities have chosen not to promote OpenURL technology with the fanfare that its huge promise might warrant. Perhaps this is a sensible recognition that it is a developing service and encouraging high expectations may be dangerous. There may be a case for arguing that at present the service should not be promoted widely because the `passive majority' are best kept away from it lest they have negative experiences and turn away from it.

6. Recommendations

6.1 Need for Further Research

The implementation of OpenURL technology is a very important step in the evolutionary development of electronic libraries and could have a major impact on users. If we are to maximise learning from this developmental period each implementation needs to be accompanied by evaluation of user experiences. There is also a need for systematic research across all institutions to assess the impact on different user types and disciplines and the impact of different ways of implementing the technology. At the centre of such evaluations should be a careful assessment of the tolerance levels of users - what rates of success in getting to full text are acceptable and with what amount of effort? Answering these questions is important to establish benchmarks for what services should seek to achieve. They are also useful benchmarks for determining implementation and publicity strategies, i.e. have we got to a level where we can afford to encourage everybody to use this service?

6.2 Recommendations for developments in the service

Although this evaluation is limited in scope, it is possible to propose some specific recommendations for the ways in which the services should evolve:-

References

Appendix A: Loughborough University: A Case Study of a Local Resolver Service

The information reported here is based on the study carried out by Stubbings (5) and data collected from the Loughborough users who responded as part of the zetoc surveys (1) (4).

Implementation Process

Loughborough created a large internal team of librarians and support staff to consider how to implement MetaLib and SFX (Stubbings 2003). The SFX service was launched in July 2002 closely followed by MetaLib in August 2003. Stubbings reports that both services took considerable effort to mount because of the need for co-operation with resource and content providers. For SFX for example "letters were sent to all suppliers asking if they were SFX-compliant. It was surprising how few knew anything about the technology and required additional data. Quite a few said they were not OpenURL- complaint but it would be something into which they would look." p.31

When MetaLib was launched it provided a single portal to the 170 databases to which the University subscribed but only 35% of them were cross-searchable. When SFX went live nearly all the library's e-journals (over 3,500) were accessible but only 35% of the databases to which the university subscribes had been SFX-enabled.

User Responses.

Stubbings reports that three focus groups were held at the time of launch and that there was a 100% positive response to MetaLib. 80% reported that it had made them aware of resources that they did not know were on the University subscription list.

Following the launch "Feedback has been very positive. Comments include "a very useful resource", "I did not know so many databases were available for my subject" and "I love playing with MetaLib". Stubbings comments that Ex Libris do not provide statistics on MetaLib use but the Library has seen an overall growth in use of databases of 10% and some have risen by up to 300%. Stubbings gives no data on the take up of SFX but she does note several difficulties in using the service; "for example, INSPEC was returning the journal details in the imprint field not in the source field and therefore the SFX service would not work correctly. Art Abstracts and EI Compendex did not always search as anticipated as they do not allow phrase searching through Z39.50." p.28. Her advice is "Test, test and test again".

Usage Data via the Zetoc Evaluation.

Four Loughborough users responded to the second zetoc questionnaire and there were seven in the interview study. A table of relevant results is given below.

User RoleNo. Zetoc Usage ScoreUse of MetaLib Use of SFX
Faculty57.842
Researchers313.330
PG Students311.312
Librarians0   
Total1110.48 (66%)4 (33%)
Total Sample1936.111 (6%)10 (5%)

Table 3. The Loughborough Sample

This sample, despite the fact that it contains no librarians, consists of an active group of users most of whom exhibited mature and stable ways of using zetoc. The overall usage score of 10.4 was substantially higher than the average and was the highest score of any institution in the sample. The users were unanimous in placing the need for a seamless way of achieving electronic full text as the major benefit they were seeking from electronic libraries. The proportion using Metalib and SFX is much higher than for the sample as a whole (however, these facilities were not available in most other institutions).

The following conclusions are drawn primarily from the interview records:

Appendix B: Trials with SFX

In order to examine the reasons for the reports of users who have tried SFX a number of trials were undertaken. These were undertaken at Loughborough University which has implemented MetaLib and SFX and has full text access to over 3,500 electronic journals. It is therefore one of the University environments in which use of SFX is likely to be at its most successful. The trial was set up to reproduce what happens when a user undertakes a search which leads them to seek full text for a set of articles in unfamiliar journals. This is the situation in which the user does not know whether the university has any access rights to the journals in question.

The specific aims of the trials were fourfold:-

  1. To establish the frequency of success of obtaining access to full electronic text and abstracts.
  2. To establish the number of pages users have to visit in order to be successful or unsuccessful in the search
  3. To compare the performance of using SFX via MetaLib (the route described in 4.1.1) with the `More Information' route in zetoc (as described in 4.1.3).
  4. To examine the messages that users receive as they move between services.

Two tests were undertaken:

  1. A search using a narrow search term `Nursing Decision Support Systems' in both the Zetoc website and MetaLib
  2. A search using a wide search term `empowerment' again in both the Zetoc website and in MetaLib

In both cases the intention was to select a sample of `hits' representing an array of different journals and other sources and to examine whether SFX could reach full text.

Trial One: Nursing Decision Support Systems

A search was made using the zetoc database via MetaLib and via the zetoc website using the search phrase `nursing decision support systems'. It yielded 16 hits in the zetoc database accessed through the website. It yielded 0 hits when the zetoc database was accessed as a resource in MetaLib. Metalib offers other database resources and widening the search to include the Computing and Information Science database led to 3 hits. There was no overlap between the articles identified in the two searches and therefore no direct comparison was possible, Three articles were used to explore the SFX/'More Information' links in both services with the results reported in the following table.

JournalNo. of Targets TargetResult No. of Pages
Metalib/SFX     
Art. Int. in Med1ScienceDirect Successful4
13th Sym on Comp. Ap. In Med. Care1LitLink Unsuccessful3
14th Sym on Comp. Ap. In Med Care1LitLink Unsuccessful3
   Total pages 10
Zetoc Website     
Nursing Inquiry5EBSCO Unsuccessful4
  Blackwell Synergy Unsuccessful1
  SwetsWise Unsuccessful1
  NCBI Abstract4
  OVID Unsuccessful3
J. of Op. Res. Soc3EBSCO Unsuccessful4
  SwetsWise Unsuccessful1
  Catchword Unsuccessful3
Nursing Admin. Quart.2NCBI Abstract7
  OVID Unsuccessful3
    Total Pages31

Table 4. SFX Search for `Nursing Decision Support Systems'

The search using Metalib was successful on one occasion (33% success) and overall, counting from the page including the reference on the list of hits, took 10 pages. Using the zetoc website produced a different set of hits, no successes in reaching full text, (but two abstracts) and took 31 pages because it produced more targets that might have an accessible full text version.

(NB Since this trial was undertaken, using SFX via the zetoc website at Loughborough has been changed and it now makes use of the local resolver service. This means that the user is informed of the local university holdings and the results are therefore the same as using the zetoc database via MetaLib. The original trial results are retained here because the multi-target results are representative of what users of zetoc would experience where there is no local resolver service).

Trial Two; Search Term `Empowerment'

Using Metalib the Zetoc database gave 4159 hits. Using the Zetoc website gave 3772 hits The same 10 references were selected for follow up from with the following results:

JournalNo. of Targets Target Result No. of Pages
Metalib/SFX     
Telematics and Information1ScienceDirect Successful5
J. of Gender Studies2Ingenta Select
MetaPress
Successful12
Social Policy & Society1Library stock Unsuccessful5
J.Nursing Care Qual No services Unsuccessful4
Qual. Management in Healthcare  No servicesUnsuccessful4
Adv. In Nursing Science No services Unsuccessful4
J. of Nursing Admin. No services Unsuccessful4
IT Education and Society No services Unsuccessful4
J. of Human Development No services Unsuccessful4
Communication No services Unsuccessful4
    Total pages50
Zetoc Website     
Telematics and Information4EBSCO
Swetswise
Catchword
Science Direct
Successful9
J.of Gender Studies3EBSCO
SwetWise
Catchword
Successful10
J. Nursing Care Qual2NCBI
OVID
Abstract7
Qual. Management in Healthcare1NCBI Abstract5
Adv. In Nursing Science2NCBI
OVID
Abstract7
Social Policy & Society2EBSCO
CrossRef
Abstract
Purchase Option
7
J. of Human Development3Ingenta
Catchword
Abstract
Purchase option
11
J. of Nursing Admin3EBSCO
SwetsWise
OVID
Unsuccessful7
IT Education and Society No services Unsuccessful3
Communication.2Swetwise Unsuccessful4
    Total Pages68

Table 5. SFX Search for `Empowerment'

It was possible to get to full text for 2 of the 10 articles and both routes were successful. The website route took more pages (68 compared with 50) because of the number of targets that could be checked. The totals are worst possible outcomes because they assume the user checks every possibility. If they get full text from the first target they would presumably go no further. It is notably that on seven occasions the website route obtained an abstract whilst this way of using Metalib did not locate abstracts. On two occasions it also offered the user an opportunity to purchase the full text of an article.

An overall summary of the results is provided in Table 6.

ServiceMetalib/Zetoc/SFX Website/Zetoc/SFX
 Trial OneTrial Two Trial OneTrial Two
 No% No%No %No%
Full Text133220 00220
Abstracts0000 267550
Unsuccessful267880 133330
No. of pages10 50  31 68 
Av. Pages per ref.3.3 5.0  10.1 6.8 

Table 6. Summary of SFX Search Results

Allowing for the duplication, the success rate for obtaining full text was 3 in 16 (19%). Both approaches had the same success rate obtaining full text for the same articles. Abstracts were obtained in 7 of 16 instances (44%). A total of 159 pages were visited to obtain 5 full text results (32 pages for each success); using MetaLib took a total of 60 pages to obtain 3 full text articles, using the zetoc website took 99 pages to obtain 2 full text articles and 7 abstracts.


The ITAM project was funded by the Joint Information Systems Committee (JISC) for the UK Higher and Further Education Councils as part of the JISC Services 5/99 DNER Programme.

2 February 2004, epub@manchester.ac.uk          [Go to MIMAS home page]          [Valid XHTML 1.0!]