Access to the feature articles. This page is intended for printing purposes. Note that the internal links to references will not work correctly.
Hans-Jörg Lieder gives an interim report on the MALVINE project.
Much work has been carried in the library community in recent years regarding the provision of digitized catalogue data about printed books. Most major libraries and archives offer at least part of their data in web-based OPACs. However access conditions are not as well-developed for modern manuscripts and letters [1]. Digital information about this type of material is still rather scarce and only few online catalogues are available.
Anyone wanting to improve this rather deplorable situation is confronted with a number of difficulties. Firstly, the obvious needs to be stated: manuscripts are, unlike books, unique objects. The cataloguing of manuscripts is therefore much more elaborate and time-consuming than the cataloguing of books and consequently a lot of the material in question is simply not fully catalogued yet. Another difficulty arises from the fact that handwritten material is owned or kept by a number of different institutions: public, university and national libraries, archives, a variety of specialized documentation or research centers and even museums. As these institutions evolve from different traditions and serve different purposes the practice of cataloguing and providing information about the holdings might differ to a considerable degree and national cataloguing rules, this in particular marks the difference to books, have not been available so far [2]. This results in a large array of differing rules and standards in use.
These circumstances are not only responsible for the apparent lack of available online data but also result in another undesired effect: when accessing the material, different search strategies need to be learned and adopted anew in each separate institution. Before even obtaining the required document, scholars and other users are thus confronted with a good deal of work, work that could be made partly redundant in future by MALVINE.
MALVINE, Manuscripts And Letters Via Integrated Networks In Europe, aims at improving online access to the multitude of European manuscript holdings. The basic idea of MALVINE is to build an electronic network of relevant institutions in Europe, a network which is independent of heterogeneous technical solutions and cataloguing traditions locally used by the data providers and which is accessible from all over the world using low cost personal computers and web technologies.
MALVINE addresses both the potential users of manuscript holdings, in the project's terminology called public users, and the staff working on the collections in the relevant institutions, referred to as expert users. A variety of kernel services for these two main user groups were developed or are being planned.
The major result for public users is a metadata based search engine for this specialized sector, thus providing a harmonized access to a large number of European collections of manuscripts. A multilingual user interface [3] for both a simple (see Figure 1) and an advanced search option (see Figure 2) presents the catalogue data as though belonging to a homogeneous unified database. The agreement to a common multilingual terminology and the use of network components, standards and protocols enhances state-of-the-art search and query mechanisms in different European languages and ensures interoperability of heterogeneous systems.
 Figure 1: MALVINE Simple Search Site (English Version) |
The relevant information is provided by one multi-site searching procedure. Compared to conventional card catalogue data, the search options in MALVINE are greatly enhanced. Next to the most obvious search criterion, the name of an author, a number of additional data entries are searchable: every person significantly related to a given manuscript - the writer of an addition to a text, a commentator, a photographer, an artist etc.; an item may be searched by its title, by time and place of its origin and even by its content. In the advanced search mode, more detailed information about a manuscript and related items like printed editions of, and literature about the manuscript may be searched and retrieved [4] (see Figure 3 for an example of a retrieved catalogue record). An online document ordering service, either of conventional microforms or of digitized images of the originals, is offered. In the near future public users will also have access to biographical data; this functionality is currently being implemented.
 Figure 2: MALVINE Advanced Search Site, German Version (part of site only) |
For expert users the availability of data relating to many different collections of manuscripts is also one of the most important service features of MALVINE. While at present a lot of unnecessary work in cataloguing and describing the material must be done, a network of institutions offers various possibilities of useful data creation and retrieval by future utilization of authority files [5]. Further services include the SGML/XML [6] based support of data migration. The processing of a number of administrative activities like accounting and creation of users statistics is planned.
Two major feasibility studies are integrated into MALVINE. The Feasibility Study: MALVINE Information Brokerage Platform [7] investigates the feasibility and the advantages of the introduction of electronic services for information brokering in the manuscript sector. Special attention is paid to a MALVINE architecture of services, ranging from search and retrieval and document delivery facilities to electronic commerce services like negotiation, ordering, accounting, payment and to legal services, such as the administration of copyright and authentication questions. Many of these possibilities depend on future developments in this sector and whether users of manuscripts will accept a more commercialized environment, since many services will only be available if paid for by the user. Current tendencies in comparable markets suggest that there will be a commercial potential in the manuscript sector to some extent. Users seem to be willing to pay for data if they know they can acquire it speedily and efficiently and if they are confident of its high quality. On a technical level, the system will be able to integrate additional, enhanced functions at a later stage by developing the required gateways and interfaces.
 Figure 3: Example of a detailed result display (Portuguese Version) |
Another accompanying study, the SGML Feasibility Study [8], mainly explores the possibility of SGML/XML based migration of locally held data. The concrete objective of this study was the production of a tool that converts between the various catalogue formats in use and additionally offers an SGML or XML encoding format (see Figure 4 for an example of converted data). The chosen DTD (Document Type Definition) is EAD (Encoded Archival Description) [9]. This tool is used whenever the need for a conversion of the local formats arises in MALVINE. In most cases local catalogues are being accessed by the search server online using the Z39.50 protocol. Whenever this is not possible the catalogue data can be converted into SGML using EAD as the DTD, the EAD version can then be converted into another database with Z39.50 abilities [10]. Additionally it has to be stated, that the choice of the actual DTD, although important, is not irreversible. Once the catalogue data is available in one DTD format, later changes into another DTD will not be too difficult to execute.
Figure 4: Example of an EAD encoded data record (Person name authority file record with indication of manuscript holdings, SBB) |
Apart from the above mentioned services and products, one of the important goals of MALVINE is the establishing of contacts and data exchange between the different users of the system. Such an exchange will prove to be universally beneficial. MALVINE therefore provides an electronic feedback function, allowing for flexible and convenient communication between respective users. In many cases scholars have more detailed and precise information about a particular item than the institution cataloguing it. They might, therefore, via user-friendly web-based communication channels, contribute to the quality improvement of the catalogue data. Further contacts are imaginable: publishers might easily be brought into contact with manuscript dealers, authors with scholars etc.
After extensive preliminary negotiations the European Union project MALVINE, scheduled for a duration of 30 months, started in July 1998 [11]. A number of well known and reputable institutions in various European countries have combined their resources to strive for the ambitious project aims in a joint effort, thus forming the MALVINE consortium and acting in a variety of functions:
When determining the data providing and testing project partners, particular care was taken to ensure a high degree of competence in the sector of cataloguing of manuscripts and letters. A number of participating institutions, those being high profile data providers, are therefore able to act as a kind of national focal point for the promotion and dissemination of the project results.
At the same time the availability of a large variety of test data was required, particularly during the development and demonstration phase of the project. It was also intended to consider the extendibility of the system into related areas not limited to manuscript material. Two partners in particular give the project this extra perspective: the National Museum of Denmark offers electronic data about museums objects, and the commercial publisher K.G. Saur offers biographical data.
The work plan of the project started with extensive surveys of the needs and wishes of potential future users of MALVINE. This included questioning of both a large variety of institutions owning handwritten material and numerous interested individuals being representatives of particular user groups, such as philologists, historians, art historians, philosophers, musicologists. Thus a hierarchical accumulation of functional requirements to the system could be determined. This information was the starting point for the development of the MALVINE metadata profile as the basis of the MALVINE search server and other software components. A first demonstrator version of the system is now available and currently being tested. The MALVINE consortium was able to motivate around 500 test users from all parts of the world. The main task for the immediate future will be the debugging of the MALVINE software and the improvement of the offered services, all in close cooperation with the test users. These were subscribed to a mailing list and are regularly updated on all changes concerning the MALVINE website. The test users' feedback is processed using online forms in order to minimize the required effort. Participation in the MALVINE test phase is still possible and may be of particular interest for those involved in other related EU projects [13].
The technical side of a project of the scope of MALVINE is obviously rather complex [14]. While this is not the place to elaborate on all technical details, it should be stressed that care has been taken to ensure the relative simplicity of the required hardware both on the side of the data providers and the end users.
Electronic formats used to store data about manuscripts and letters may vary from institution to institution and even more so from country to country. Catalogue data might be offered as part of a local OPAC, in a variety of data base formats and even in plain word-processing applications, even digitized images of the precious original documents will in future be available. The main focus and primary goal of the technical development work therefore consisted in achieving technical and semantic interoperability of different data bases containing different types of data and in performing a multi-server search based on existing standards.
The MALVINE software can be seen as a smart piece of portal software which can be easily integrated into other web based applications and solutions.
 Figure 5: Architecture of MALVINE |
The concept of MALVINE is based on a distributed architecture containing three main layers:
MALVINE uses state-of-the-art technology in all key components: the multilingual web user interface makes use of the latest web standards, including dynamic and static web pages, cascading style sheets (CCS), JavaScript, HTML, Servlets and Java Server Pages. The key component of MALVINE, the MALVINE application server, is using the Websphere application server. The software is completely written in Java.
The advantages of this solution are:
The MALVINE OPAC is based on a standard relational database structure. The OPAC translates the incoming Z39.50 requests and produces appropriate responses. The OPAC exchange formats are using XML / XSL which is smoothly integrated in the software.
The setting up of the OPAC database will be done through some services offered within the MALVINE toolbox. For the connection between the database and the MALVINE-Z39.50 target system a JDBC/SQL interface will be supported. For administration the OPAC database with the appropriate sets of data several facilities will be available (conversion, import, export and entry facility). The MALVINE exchange format is based on EAD / XML.
The benefits of the planned MALVINE system are hopefully apparent on a number of different levels: benefits for the public users are manifold and there are clear opportunities for the implementation of more effective work methods in libraries, archives and other institutions offering data about manuscripts. Also on the institutional level the technical development of MALVINE enhances the interoperability of different interfaces. It demonstrates that all institutions, be they large or small, can present their own special holdings to a broad public without having to give up either local cataloguing traditions or local technical solutions. All can thus contribute to the aims of the information society. The low costs involved allow for improved acceptance of MALVINE, differences in languages and in technical equipment are bridged. Even small institutions with low budgets are able to implement the MALVINE system. The cultural heritage of Europe as manifested in holdings of manuscripts and letters is presented as part of a landscape consisting of individual countries whose representations in various areas of cultural activity are unique, yet share a deep awareness of common or neighbouring sources.
),
in the librarian sector e.g. those for
country and language codes. It may be anticipated that the suggested use of
both authority files and applicable standards within a network of institutions
furthers the increased utilization and implementation of the same.
as a pdf file.
as a pdf file.
as a pdf file.
Hans-Jörg Lieder
Berlin State Library - Prussian Cultural Heritage
Department of Manuscripts
D-10772 Berlin
Germany
URL: <http://www.sbb.spk-berlin.de/>
or
http://www.malvine.org/
Email: hans-joerg.lieder@sbb.spk-berlin.de
Tel: +49-30-266-2249M
Hans-Jörg Lieder is deputy coordinator of the MALVINE project.
For citation purposes:
Hans-Jörg Lieder, "MALVINE – An Interim Report",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/malvine/>
John Shercliff reports on this web site creation project for children.
HERCULE was conceived as a project which would allow children to become creators of web material rather than simply consumers, learn skills in web page design and multimedia creation and research the culture and heritage of their local communities.
Look at the Hercule website at <http://www.hercule.org.uk/> [1]. The main theme of the new HERCULE website structure due in March/April 2000 will be an interactive TV screen as shown in Figure 1.
 Figure 1: Theme of the new HERCULE website - an interactive TV screen |
HERCULE further develops the DG XIII Telematics for Libraries Programme by bringing together librarians, teachers, arts workers and young people in a trans national project that exploits the use of multimedia and the World Wide Web to:
The objectives of HERCULE are to:
The partners are Walsall Public Libraries (UK), Quest Business Technology Centre, Walsall (UK), Dublin Corporation Public Libraries (Ireland) and Biblioteca da Camara Municipal de Oeiras (Portugal).
Initially the partners prepared for the involvement of children by setting up the technical framework for the project.
The first part of the content for the website was the catalogue of websites which is a categorized and annotated series of links to sites in the partner countries and world wide on the subjects of the environment and education. A process of evaluation for the sites by librarians, teachers and children was also developed.
The creation of material by children started in September 1999 with different approaches being taken in different schools and countries. The children taking part are in the 11-12 year old age range and are not of very high academic ability. IT, research and presentation skills have all been developing along with Project HERCULE.
The aim of the project is to have the web material completed and installed in the web site by July 2000. The site is to be extensively restructured in March, when the material produced by children will be systematically uploaded into the structure. At present examples of different approaches (not all of them 100% successful!) can be seen in the ideas section on the HERCULE website [2]. The earlier attempts will be archived as the project progresses.
As a pilot project we are experimenting with different ways of presenting material, providing resources to the children, and encouraging a response from children who are often not highly motivated. The end result will be a prototype project which can be used as a whole or in part by others who wish to encourage children to create multimedia web sites on similar topics and hopefully the ideas will continue to develop after the end of the HERCULE Project in December 2000.
HERCULE is not a technically complicated project. The simple requirements are for a web site and electronic communication between partners, libraries, schools and the children participating in the project.
The HERCULE web site was first created early in 1999 as a framework with very basic information. It is maintained on First Class Intranet Server, which gives us easy access to update and add more material to the site. The creation of web material and its uploading onto the website has been a learning process for librarians, teachers and children. The documentation of this process will be one of the benefits of the project and should help others who are faced with the same process of discovery.
Participation by librarians, teachers and children in the process is a key element. It is very much a hands on project by people who are not computer experts (or who were not at the beginning of the project). We have learnt a lot from the experts but have not relied on them to carry out all the work involved. The processes and structures we have evolved must be capable of being operated by everyone concerned.
First Class Intranet Server provides the communication service. The server was initially used for the interchange of management information between partners and increasingly for communication through e-mail and chatline. The real advantage of FCIS is in work with the children since it allows resources and children's work in progress to be accessible from school and library and allows easy interchange of completed work from the children's desktops to the live website. Only the administrators have access to the website files!
The catalogue of websites was produced in the first part of 1999 and installed on the HERCULE web site in September 1999 [3]. The list was compiled by librarians to provide easy links for children to information on the web for their curriculum studies into the environment. The subjects covered reflect the different needs of schools in the partner countries at the time. There are also global sites which cover the environment and also national education sites.
The catalogue was never intended to be a comprehensive list of all the sites available. To some extent the sites were recommended by librarians; poor sites were not included and the sites had been checked by librarians for their suitability for the children taking part in the project. A variety of different types of sites were included.
Evaluation sheets were developed by the Dublin partners who coordinated this work package, and used by all three partners with children to evaluate the sites. This provided some useful insights into what children expected from an information website and the factors which were likely to encourage further use of the sites or put children off from using them. In general it was found that children's attention was kept by websites with interactive features and eye-catching features and that speed of response was important. For further information see the Users Response report [4].
As a result of this it was decided to change the format of the catalogue of sites to include ten recommended sites from each country plus ten global sites and improve the presentation of the catalogue. This work is still in hand and will be included in the new look web site to be launched in March.
The catalogue of websites will be checked to ensure that links are still operating and new sites added for the remainder of the project. Criteria for inclusion of sites have been developed.
The production of content by the children is integral to HERCULE and the creation of a process and environment to encourage and facilitate children in the creation of their own web content will form the main "deliverable" for HERCULE which will be capable of exploitation by partners and others in future.
Planning meetings took place with school staff in June, July and August in all partner countries to agree methodologies and details required for the project to take place. It proved difficult to do any detailed planning until September because there were too many undecided factors affecting the classes involved. Dublin librarians held a meeting with the teacher involved in October to explain the project and the website.
 |
 |
| Figure 2 (a) Children working in Oeiras | (b) Children at Ballyfermot Library, Dublin |
The project was introduced to both schools in Walsall during trips to the library. The students were introduced to the concept of HERCULE. The topics to be included in the website were discussed and the HERCULE website and a few selected sites were looked at to give students some idea of what websites could look like. The sessions went well and the students were enthusiastic. The project was presented to the Portuguese children in September, some web sites were looked at and the topics to be covered were discussed. In Dublin the children were first introduced to producing content for the HERCULE website in October, although they had looked at some of the websites listed in the catalogue in July. These activities took place in Ballyfermot Library, with children from St. Gabriel's National School.
At the first session in each school in Walsall the students were organised in groups of 2, 3 or 4 to study and produce material on topics they chose. The larger groups have tended to split into pairs and some children have produced individual work within the group structure. The students chose their own topics such as transport, shops, the past, youth clubs, sport, etc.
Sessions for HERCULE have taken place using computers in local public libraries, exclusively in Oeiras and Dublin. Sessions in school have taken place in children's form classrooms, computer suites with and without internet access, and out in the community. Help has been provided by school staff with either IT or subject skills (or both), library staff from local libraries (librarians and supervisors), children from later years in school who have better IT skills and the Windows on our World co-ordinator. (Windows on our world is a similar project covering all the schools in Walsall but without any links to other countries).
In Dublin the children researched the local area and produced written work which was later reproduced on the web site. The focus of the work will be at the library as there is not as much access to computers in the school. The topics chosen were very similar to those in Walsall and Oeiras.
The children in Oeiras produced the icons for the menu of art forms to be used as the main structure of the HERCULE website [5]. Children and Librarians collaborated with the art designer to produce the Flash technology software for the menu.
The importance of planning the work, particularly information needs, quickly became clear. There was initially a lot of input on different topics from the children's own experience but the initial enthusiasm was slowed down a bit by lack of information. Planning where to get information from (books, interviews, photos, internet etc) was a necessary next step. The information skills of some of the children are not highly developed and this became another area for action. A simple quiz based on local information was used at Bloxwich Library to give some basic practice in finding information.
Some of the perceived information needs were not realistic and some children became disillusioned by their lack of progress. In retrospect it would have been better to provide a lot of varied information right at the beginning of the project. The lack of easily accessible material slowed the project down until it could be remedied. The use of First Class Server access for the children to material which had been uploaded by library staff helped.
Photographs provided a visual stimulus and an attractive way of presenting information. Some of these were taken using digital cameras by students on supervised trips into the community during school time, others were provided by library staff from local history collections and present day digital and scanned photos.
The printed sources for heritage information caused problems for many students due to the impenetrable prose in which many are written and the unattractive format of books published many years ago. Books of photographs with informative captions were the most popular ready-made sources of heritage information. There is a need for project staff to reformat information into easily digested chunks to make it more palatable to the less academic students.
Children in Oeiras are collecting information about the selected topics from books, magazines, Internet, CD-Roms etc. Help from librarians was essential as children's information skills are not highly developed. The children also had a bus tour of Oeiras as many of them were not familiar with the area. A tourist guide gave a commentary and provided documentation.
Some children found it difficult to come up with ideas to put on their web pages, even for topics they had chosen themselves. The original idea was to base the project on what children think about their communities in the present day and use that as a starting point for looking at the heritage aspect. Many of the children needed more stimulus to develop their ideas and a questionnaire "A day in the life of ..." was devised to help in this. The initial idea to "let the children's imagination fly" was not very easy to put into practice!
Photographs are a good way of presenting impressions of the community as it is now and carefully chosen old photos can give a vivid impression of life in the past, and its links to the present through buildings and roads that still exist. A series of "Then and Now" views is planned, with plenty of details of people and how they lived in the past and the present will encourage an interest.
 |
 |
| Figure 3: Bloxwich High Street (Walsall) in the 1900s | Bloxwich High Street in 1999 |
One problem with using books for the project is that many of the photographs chosen by children may have copyright restrictions on their reproduction.
The initial web pages produced by the students in Darlaston were made using Microsoft Word, which was subsequently saved as HTML for inclusion on the website. This had the advantage of using software that the children were familiar with and allowed quite sophisticated use of fonts, colour and layout. Some of the pages had to be modified using Microsoft FrontPage to allow them to appear on the web browser effectively. This was initially done by library staff, but the children will need to be aware of how it was done so that they can take the requirements of webpage display into account and do the conversion themselves.
Some of the Darlaston students used Microsoft FrontPage to create a homepage with links to the other pages. This worked well, especially as this was a particularly able group.
The students at TP Riley used Microsoft Powerpoint to create their initial web pages, as suggested by one of the teaching staff. This allowed very attractive pages to be created on the PC with animation and sound but these did not convert at all easily when saved as web pages, and many of the features did not work on Internet Explorer. There may be ways round this but the structure of the files and HTML created by Powerpoint make it difficult to modify using any other software (see [6]).
More work will need to be done to allow the students to understand the limitations of the different ways of working and the effect on the final web page. The importance of quick downloading of images and other files from the website will also have an effect. Web page design skills will develop as the project progresses. It is also important that children who are not particularly interested in this aspect should be allowed to create content which can be presented on the web page by others.
In Dublin Dreamweaver was used to create the Ballyfermot web pages using material produced by the children. Children will be able to use Microsoft Frontpage at Ballyfermot Library to create their own web pages (see [7]).
The First Class server now plays an important role in the project in Walsall. The server is set up at present so that each student and teacher has their own desktop, which they can log into from school or the local library. Included on the desktop is access to Email and "TP Riley Students" and "Darlaston Students" conferences which give read-only access to information (mainly photographs at present). These can be downloaded and included in students work.
First Class has been installed on eight computers at Ballyfermot Library and will be used by the children for HERCULE. It may be possible to also install FCIS at the school but at present most of the work on the project is likely to take place in the library.
Students can also upload their work to the server to save it on their desktops and this also provides a route by which it can be easily uploaded to the HERCULE website which is hosted on the First Class Server.
The HERCULE website files are only accessible to the administrator (at present the technical partners, Project Manager and staff in Dublin and Portugal) It is relatively easy to add new sections to the website and, as the content increases, the structure of the website will be developed to make slotting in new sections, and links to them, as easy as possible.
The eventual content of the website has yet to be finalised. A structure for the site has been created by computer staff and children in Oeiras after consultations with the other partners. This will become the main access to material produced by children on the web site and the updated version will be installed by the end of March.
The structure is a modified version of the first menu of art forms [5] and is based on seven main headings:
It is not clear, however, whether it will be possible or desirable to try to produce a comprehensive coverage of every aspect of each community's heritage, culture and environment.
The initial web pages in Walsall were kept deliberately simple so that results could be produced fairly rapidly. The children are now showing more interest in using more advance techniques such as animation, sound and video clips and more interactive ways of presenting information. The students are not really aware of the complications and skills needed to use these techniques but expect to be able to introduce them. We will look at ways of introducing these skills with the help of arts workers as the project progresses.
Over the next six months content will be developed, methods of presentation will be explored and the material will be uploaded into the website structure which will be in place by the end of March.
Communication between the children involved in the different partner countries has always been an exciting possibility for HERCULE. As the children are now getting to grips with the technology and beginning to have something to show each other on their websites it is intended to set up e-mail, chat sessions and possibly video conferencing sessions between them. The important thing here is to ensure that access is controlled so that we know that the children are talking to genuine participants in HERCULE. There is a lot of potential for children to gain insights into the differences and similarities in the lifestyle, culture and heritage of other countries of Europe.
The main priority for the partners for the rest of the duration of HERCULE, apart from the completion of the content, is the dissemination, promotion and evaluation of the project. HERCULE has tremendous exploitation potential and the development of successful ways of working with children to create web material will have applications in many similar projects. There is potential to use the HERCULE website to include material from other countries or other regions within Britain, Ireland and Portugal. A package of suggested ideas, materials and procedures will be produced which will enable schools, libraries or other organizations working with children to produce web material.
This has been an overview of the progress so far on HERCULE. The first annual report, with detail of work packages, deliverables and progress is available on the HERCULE web site [8].
A lot of new skills have been learned by everyone involved and working with children has been interesting and entertaining. There will be real benefits from the project in enabling other groups to recreate similar projects in future.
The benefits to children taking part in this and similar projects cannot be overemphasized. Although it is generally held that children are highly computer literate and some have access to computer facilities and the internet at home this is far from universal. Many children in less affluent parts of Britain, Portugal, Ireland and other countries of Europe have little access to computers or the internet. Libraries can provide this and schools can help to provide the skills required to use them. HERCULE has shown children that they can not only make use of the World Wide Web but that they can add their own material to it. For the citizens of the future this will be an increasingly important skill, the children of HERCULE are beginning to make an important step towards that future.
John Shercliff
Walsall MBC
Bloxwich Library
Elmore Row
Bloxwich
Walsall
WS3 2HR
United Kingdom
URL: <http://www.walsall.gov.uk/cultural_services/library/>
Email: <shercliffj@walsall.gov.uk>
Tel: +44 1922 710059
John Shercliff is Project Manager for HERCULE and Library Services Manager for Walsall Public Libraries and Information Services. He also acts as chair of Walsall Libraries Flexible Learning Centre Group, responsible for coordinating Walsall's free public access computer provision.
For citation purposes:
John Shercliff, "HERCULE: Heritage and Culture through Libraries in Europe",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/hercule/>
Klaus Reinhardt and Christiane Dorff report on the development of a "presentation toolkit" for re-use at national and international (library) events.
In June 1998 the contract for the EXPLOIT project was signed by the European Commission and the German Libraries Institute (DBI) in its role as project co-ordinator. EXPLOIT stands for Pan-European Exploitation of the Results of the Libraries Programme. It is an accompanying measure under the Libraries Programme which is part of the EU Telematics Applications Programme. The other main partners of the project are The British Council and UKOLN (UK Office for Library and Information Networking) at the University of Bath. The measure was planned for a lifetime of 24 months. It has recently been extended until the end of 2000.
The results of the approximately 100 projects and actions funded under the Libraries Programme are now becoming available. Exploitation and the encouragement of take-up among libraries within the member states is now a key issue. The existing and potential users include: libraries, publishers, book trade, industry, research associations and policy makers all over Europe.
There will be two main phases of the Accompanying Measure. Firstly a process of self-assessment by projects followed by analysis and clustering of results followed by a range of inter-linked activities including:
A number of problems are involved: although projects are responsible for the exploitation of their results, it has proven difficult for many project consortia to take a wider market perspective; awareness of and access to information on project results is not adequate at present; and there is no multi-project forum available to discuss (rather than just present) these results. The projects launched have been of a relatively small scale and the libraries scene in Europe is in general somewhat fragmented. There is a need for additional mechanisms which can increase the impact of the activities under the Telematics for Libraries Programme.
The development of effective mechanisms for the exploitation of emerging results from the Telematics for Libraries Programme across the EU has to become a major priority if the programme is to have the envisaged impact. Similarly, the transfer of know-how in this area to those CEE countries which have applied for accession to the EU is important if organisations in this sector are to catch up and apply European standards. The EXPLOIT Accompanying Measure will provide integrated pan-European activities designed to meet this need, building in features which are sustainable after the completion of the measure.
The aim of the presentation toolkit is to facilitate the demonstration of project results at national and international events. It points out the direction of the technological development of the libraries sector rather than presenting project results individually. Through its well-structured content and attractive appearance it gives professionals of the libraries sector the opportunity to inform audiences of the latest development in an easy, ready-made way.
First, the main areas of technological development had to be identified. In order to achieve that all projects had to be clustered into homogeneous groups. This was a lengthy process using several classification lists from the European Commission. In addition, it proved to be appropriate to classify the projects into (1) resources used, (2) used or developed technology and (3) targeted user group. The outcome of that process is the following list of nine clusters:
Every project can be represented in more than one cluster (three clusters at maximum), however, following the main content of the project, one cluster will be defined as "main cluster".
| No. | Project | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| 1 | AIDA, No. 2036 | 4 | |||||||||
| 2 | ARCA, No. 3039 | 4 | |||||||||
| 3 | BALTICSEAWEB, No. 4020 | 1 | 4 | ||||||||
| 4 | BAMBI, No. 3114 | 3 | |||||||||
| 5 | BIBDEL, No. 20784 | 4 | 5 | ||||||||
| 6 | BIBLINK, No. 4034 | 1 | 3 | ||||||||
| 7 | BIBLIOTECA, No. 2023 | 1 | |||||||||
| 8 | BORGES, No 3052 | 4 | |||||||||
| 9 | CAMILE, No. 4026 | 6 | |||||||||
| 10 | CANAL/LS, No. 3063 | 1 | 4 | ||||||||
| 11 | CANDLE, No. 5649 | 4 | |||||||||
| 12 | CANTATE, No. 2016 | 7 | |||||||||
| 13 | CASA, No. 4058 | 1 | |||||||||
| 14 | CASELIBRARY, No. 3130 | 4 | |||||||||
| 15 | CDBIB, (FP3 prep. action) | 1 | |||||||||
| 16 | CECUP, No. 5663 | 2 | |||||||||
| 17 | CHILIAS, No. 4086 | 3 | 5 | 8 | 9 | ||||||
| 18 | COBRA/COBRA+, No 4100 | 1 | 3 | ||||||||
| 19 | COPINET, No. 3033 | 2 | 4 | ||||||||
| 20 | DALI, No. 3132 | 4 | |||||||||
| 21 | DEBORA, No. 5608 | 3 | |||||||||
| 22 | DECIDE, No. 3062 | 6 | |||||||||
| 23 | DECIMAL, No. 3012 | 6 | |||||||||
| 24 | DECOMATE, No. 3078 | 1 | 2 | 3 | 4 | ||||||
| 25 | DECOMATE II, No. 5672 | 2 | 3 | 4 | |||||||
| 26 | DEDICATE, No. 5611 | 5 | |||||||||
| 27 | DELICAT, No. 4025 | 1 | |||||||||
| 28 | DERAL, No. 5660 | 5 | 8 | ||||||||
| 29 | DIEPER, No. 5632 | 3 | 4 | ||||||||
| 30 | ECUP, ECUP+, No. 4000 | 2 | |||||||||
| 31 | EDIL, No. 1042 | 4 | |||||||||
| 32 | EDILIBE I (FP3 prep. action) | 1 | |||||||||
| 33 | EDILIBE II, No. 1011 | 1 | |||||||||
| 34 | EDUCATE, No. 2003 | 5 | |||||||||
| 35 | EFILA, EFILA+, No. 4012 | 1 | 4 | ||||||||
| 36 | EFILA97, No. 5601 | 1 | 4 | ||||||||
| 37 | ELISE, No. 1008 | 3 | 4 | ||||||||
| 38 | ELISE II, No. 3005 | 3 | 4 | ||||||||
| 39 | ELITE, No. 4087 | 4 | |||||||||
| 40 | ELSA, No. 2006 | 3 | 4 | ||||||||
| 41 | ELVIL, No. 4056 | 3 | 4 | 5 | |||||||
| 42 | ELVIL2000, No. 5667 | 1 | 4 | 5 | |||||||
| 43 | EQLIPSE, No. 3019/3077 | 6 | |||||||||
| 44 | EQUINOX, No. 5634 | 6 | |||||||||
| 45 | EULER, No. 5609 | 1 | 3 | 4 | |||||||
| 46 | EURILIA, No. 2083 | 3 | 4 | ||||||||
| 47 | EUROPAGATE, No. 2062 | 1 | 4 | ||||||||
| 48 | EXCEL, No. 5650 | 1 | |||||||||
| 49 | EXLIB, No. 1037 | 9 | |||||||||
| 50 | EXPLOIT, No. 5603 | n.a. | |||||||||
| 51 | FACIT, No. 1044 | 1 | |||||||||
| 52 | FASTDOC, No. 2076 | 2 | 4 | ||||||||
| 53 | HARMONICA | 1 | 3 | 7 | |||||||
| 54 | HELEN, No. 1040 | 1 | |||||||||
| 55 | HERCULE, No. 5604 | 3 | 8 | 9 | |||||||
| 56 | HISTORIA, No. 3117 | 3 | |||||||||
| 57 | HYPERLIB, No. 1015 | 4 | |||||||||
| 58 | IFLA-EU, No. 5605 | n.a. | |||||||||
| 59 | ILIERS, No. 4039 | 4 | 5 | 8 | 9 | ||||||
| 60 | ILSES, No. 4050 | 4 | |||||||||
| 61 | IMPRESS, No. 4091 | 1 | |||||||||
| 62 | INCIPIT, No. 2031 | 1 | 3 | ||||||||
| 63 | ION (FP 3 prep. action) | 4 | |||||||||
| 64 | JUKEBOX, No. 1049 | 2 | 3 | 7 | |||||||
| 65 | LAURIN, No. 5629 | 2 | 3 | 4 | |||||||
| 66 | LIBECON/2000, No. 5656 | 6 | |||||||||
| 67 | LIBERATION, No. 4063 | 3 | 4 | 5 | |||||||
| 68 | LIBERATOR, No. 5646 | 4 | 8 | ||||||||
| 69 | LIRN, No. 2014 | 1 | |||||||||
| 70 | LISTED, No. 4111 | 2 | 5 | 8 | |||||||
| 71 | MALVINE, No. 5674 | 1 | 3 | 4 | |||||||
| 72 | MASTER, No. 5612 | 1 | 3 | 4 | |||||||
| 73 | MECANO, No. 2045 | 4 | |||||||||
| 74 | MINSTREL, No. 3091 | 6 | |||||||||
| 75 | MIRACLE, No. 5670 | 1 | 7 | 9 | |||||||
| 76 | MOBILE, No. 2050 | 4 | 8 | 9 | |||||||
| 77 | MORE, No. 1047 | 1 | |||||||||
| 78 | MUMLIB, No. 3025 | 2 | 3 | 4 | 8 | ||||||
| 79 | MURIEL, No. 3007 | 2 | 3 | 4 | 5 | ||||||
| 80 | NEDLIB, No. 5648 | 2 | 3 | 4 | |||||||
| 81 | OLUIT, No. 3121 | 3 | 4 | ||||||||
| 82 | ONE, No. 3099 / ONE II, No. 5643 | 4 | |||||||||
| 83 | PLAIL, No. 2081 | 5 | 8 | ||||||||
| 84 | PRIDE, No. 5264 | 4 | |||||||||
| 85 | PUBLICA, No. 4055/5628/5653 | 8 | |||||||||
| 86 | REACTIVE TELECOM, No. 3122 | 5 | 8 | ||||||||
| 87 | RIDDLE, No. 1038 | 1 | 4 | ||||||||
| 88 | SELF, No. 3020 | 6 | |||||||||
| 89 | SESAM, No. 3093 | 4 | |||||||||
| 90 | SOCKER, No. 1050 | 4 | |||||||||
| 91 | SPRINTEL, No. 2032 | 6 | 8 | ||||||||
| 92 | SR-TARGET/PARAGON, No. 3034 | 2 | 4 | 7 | |||||||
| 93 | TECUP, No. 5664 | 2 | 4 | ||||||||
| 94 | TESTLAB, No. 4003 | 3 | 4 | 9 | |||||||
| 95 | TOLIMAC, No. 4047 | 2 | 4 | 6 | |||||||
| 96 | TRANSLIB, No. 3038 | 1 | 4 | ||||||||
| 97 | UNIVERSE, No. 4032 | 4 | |||||||||
| 98 | USEMARCON, No. 2054 | 1 | |||||||||
| 99 | VAN EYCK, I&II No. 1054 | 3 | |||||||||
| 100 | VERITY, No. 5627 | 5 | 9 | ||||||||
| 101 | VILIB, No. 5618 | 1 | 4 | ||||||||
| Totals number of projects per cluster | 30 | 15 | 29 | 52 | 14 | 10 | 5 | 15 | 8 | 2 |
In Table 1 a highlighted number indicates the main cluster for the project.
The theme areas have been used as the basis for the presentation toolkit. For each of the nine areas an individual presentation module has been created using Microsoft PowerPoint.
Most of the presentation modules contain speaker's notes which provide additional information on the theme area. All slides have a consistent look-and-feel.
Some of the presentation modules have already been tested by EXPLOIT staff and external experts at events in Rome, Warsaw and Bordeaux. Feedback has been very positive so far from the speakers as well as from audiences. Recently all project co-ordinators were asked to validate the presentations from their point of view. Comments and suggestions have been collected, and the presentation toolkit has been revised respectively. We would like to thank everybody for their contributions.
The presentation toolkit is currently available in English only. However a German and a French version are being produced.
The revised version of the presentations are publicly available online and for download on the EXPLOIT web site [1], which is illustrated below.
 Figure 1: Accessing the Presentations From The EXPLOIT Web Site |
Clicking the yellow star at the left hand side allows to view the presentations in HTML format online. To download the PowerPoint 97 formatted files please click the underlined title on the right hand side.
There will be an ongoing upgrade process of the presentations until the end of the year 2000. Comments and ideas are welcome and will be considered in the next update. Updates will take place in larger time intervals (2-3 months).
As already mentioned these presentations are prepared for re-use of experts at national or international (library) events. Please feel free to download the presentations and use them as often as you like. You may want to add some slides (e.g. national results in the same area) to the existing presentations. However, the provided presentations themselves must not be changed without consultation with the author of this article. To prevent accidental changes the presentations are password protected.
Finally, an important request: Please keep us informed of any re-use of the presentations. When you contact us to help us keep our records up-to-date on use of the presentation toolkit, please provide the following information:
Thank you very much in advance!
Klaus Reinhardt and Christiane Dorff
German Libraries Institute Berlin
International Libraries Cooperation Unit
Kurt-Schumacher-Damm 12-16
13405 Berlin
Germany
Tel: +49-30-41034-468 / -165
Email: <reinhardt@dbi-berlin.de> /
<dorff@dbi-berlin.de>
For citation purposes:
Klaus Reinhardt and Christiane Dorff, "EXPLOIT: Presentations of EU Libraries Programme Results Available Online",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/exploit/>
Andrew Cox reveals how the CANDLE project is developing a system to greatly improve users' access to licensed networked resources, such as electronic journals.
The Cactus system being developed by the partners in the CANDLE project is a prototype of the management tools information services need in the age of networked electronic resources [1] [2].
Our current focus of development is in the area of authentication and authorisation. Cactus will simplify logging on for users by, in effect, remembering passwords for them. Users will be able to connect to secure resources from anywhere, not just on campus. On campus they will get the added benefit of seeing a customised set of information resources and applications that prioritizes what is relevant to them. The collection of data on who is using what will be improved, providing the knowledge upon which to rationalise subscriptions. The control features within Cactus will give managers the tools to scaleably manage user groups so that it is possible to move away from simple institution-wide licences to low cost licences for small groups for short periods.
South Bank University has just passed an important landmark. We now subscribe to more electronic journals than print ones. We are not alone. At least one library has now cancelled all its print journals, and only subscribes to electronic journals [3].
But the management of electronic journals, indeed any networked information resources raise many problems. The partners in the CANDLE project recognised the possibility of providing solutions to some of these problems by building on an existing system developed in the CaseLibrary project.
One of the key issues with networked resources is authentication and authorisation. Authentication is the process by which you establish an identity online. Authorisation is the process by which it is decided which resources this identity is eligible to use.
Passwords are a familiar approach. The problem is they are not very secure. People share passwords despite the strictures not to. And we all suffer from password overload. We have difficulty remembering all our different passwords. We either waste time going to the helpdesk to have forgotten passwords reset or we write them down, pretty much blowing any security they might have.
IP address checking is another approach. This works on the logic of determining your access rights on the basis of where your machine is. If your machine is on campus you must be a member of the University and so entitled to use a service the institution has subscribed to. It's a very attractive approach because it is transparent to the user, and relatively easy to administer. But increasingly people want to work from off campus.
In the UK we have the Athens system which offers a great simplification of password schemes, and with more and more services offering Athens authentication life has been made a lot easier for us [4]. But there are still lots of services that do not offer Athens authentication. Significant among these are the diverse range of locally generated web materials needing secured access. These resources must be recognised as being as much part of the same problem of proliferating passwords as licensed materials. And administering Athens if everyone is to have a password is still a headache.
How to accomplish authentication and authorisation in a way that is easy for users, efficient for administrators and secure has been the cause of considerable debate [5] and [6]. The benefits are clear.
Cactus should deliver many of these benefits.
CANDLE is a further enhancement of software a number of the partners developed in the CaseLibrary project [7]. CaseLibrary was essentially developing a system for managing resources on shared workstations. Within the system you can define and associate terminals, users, user groups and resources (including local information sources, applications like Word and printers and filestore). Thus you can say this group of users can use this application on these terminals. This user has this printer. The aim was to provide an efficient way to manage resources and measure actual usage.
Features of the system are:
The main thrust of development in CANDLE (1998-2000) is to expand the system to manage networked information resources.
The software has been migrated to NT and SQLServer and the CaseLibrary functions are now available for NT and Windows 95 or 98 clients.
The partners in the CANDLE project are:
Basically our approach to authentication/authorisation is to use an 'intelligent' proxy, which when a user wants to use a service checks whether the user is registered for that resource, and if they are steps in and handles authentication. Once logged on at the beginning of a session the user does not have to remember any other passwords.
The key steps followed to complete a user's request are:
To reduce load on the proxy we are using Proxy Autoconfiguration (.pac) files which only send http requests through Cactus if they are to a server with licensed material on it. The user's browser picks up the .pac file at the beginning of their session from a central URL, so changes for all users can be made by administrators.
Only on campus machines that have the cactus client installed will get the full benefit of a customised desk top. For off campus users there will simply be a web page where they will be challenged to authenticate themselves. This authentication exchange will get users onto the proxy, from which point the benefits of simplified authentication will be enabled.
The benefits of Cactus are:
We are aware some potential stumbling blocks with our approach.
There are other possible technical solutions to authentication problems such as digital certificates. But we believe the Cactus approach using a proxy is the best available at the moment:
An alpha version of the Cactus software is being installed and tested at the Consortium sites. We hope soon to have a beta version soon that can be trialed on a small scale, then rolled out through the consortium institutions and to three official test sites.
Any other institutions including from outside Higher Education will be welcome to trial the software. We are already talking to some half dozen institutions. If your are interested contact us.
We know that there is a demand for the functionality Cactus offers. It is obvious for example that public libraries, Internet cafes and museums offering access to information resources need something like our system. And we have many ideas about how the software could be further developed.
The UK Joint Information Systems Committee have given us funding to further develop the system to allow us to authenticate users with Athens. If we can store Athens passwords in Cactus we will not have the headache of distributing them to users or sorting out questions about lost passwords. Within the project we will be exploring how we can bring in a wider range of access restricted institutional information sources into the system.
Complimentary to systems developed by projects such as Pride, Decomate and Agora we see the Cactus system as a substantial building block for an integrated management system for the digital library.
Andrew Cox
Senior Researcher
LITC
103 Borough Road
London
SE1 0AA
UK
LITC URL:
<http://www.sbu.ac.uk/litc/>
Tel: +44 020 781 7058
Email: <coxam@sbu.ac.uk>
Andrew Cox is senior researcher at LITC, a research unit in Learning and Information Services, South Bank University. LITC are involved in a number projects funded by the EC (Candle, Easel, Pride) and JISC (Heron, NewsAgent, Candle-Athens). We publish specialist publications for the LIS community including Library Technology and Vine. We also do consultancy work, recent customers have included the British Library and the National Health Service.
For citation purposes:
Andrew Cox, "One Password For Everything",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/candle/>
PricewaterhouseCoopers was invited by the European Commission to assess the impact of the Libraries Programme under FP4. A total of 49 projects (including accompanying measures) were financially supported under this programme, receiving 29 Million Euro in total. This report [1] provides an integrated presentation of results and related findings.
The impact study maps the projects and their current and expected results against the programme objectives. This may serve as a baseline for future reference and evaluation.
It must be noted that many of the projects are not yet finished. A full assessment will only be possible once all the projects have been completed.
The study is based on a fact-finding phase, an opinion-finding phase and an analysis phase. A Study Committee consisting of three experts familiar with the programme was set up to assess the progress of the work and to advise on any related matters.
The analysis was based on interviews with key external experts, European Commission representatives, project co-ordinators, National Focal Points, Programme Managers (NORDINFO, eLIB, GLOBALINFO) and library associations (IFLA, EBLIDA).
An important effect of the Programme has been the further development of the human networks in the library world that were created under previous programmes. It helped establish relationships in Europe between libraries of various types, and with traditional and new partners. The role of projects in the area of standardisation is increasingly being recognised around the world.
Under the Programme, a wide variety of technical and service issues are being addressed, covering many library domains and specific professional domains. Although several promising products and services which form an interesting potential for future deployment are being developed, exploitation still requires strong attention in the Programme.
While the Programme succeeds in attracting participation from all Member States and several associated states, and many national representatives feel that it is having a positive impact on the national environment, the creation of synergies with national programmes could be improved.
The study presents some emerging issues for future considerations, addressing the content of the Programme, project management, synergies with national programmes and the approach to exploitation.
In conclusion, the Libraries Programme is widely acknowledged as having contributed – and continuing to contribute - to its goal: “The creation of a European Library space”. It does so by contributing to the technological developments and standards area in the library field, by improving co-operation amongst libraries in Europe and by improving library services to European citizens.
For citation purposes:
"Study on the Impact of the Telematics for Libraries FP4 Programme",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/impact/>
Neil Smith reports on how two Research & Technology Development (RTD) projects in the Digital Library arena - UNIverse and PRIDE – have been driving the development of marketplace products in two key areas of the digital library picture – ILL/document delivery and authentication and security.
Libraries are moving into the Digital Library age – an environment where technology is leading change and where libraries need to integrate a range of components in order to build their own landscape. New information resources, the internet and networking boom, the emergence of e-commerce and increasingly sophisticated users are all dictating the shape of the digital library of the future.
FDI [1] is recognised as an international supplier of information management solutions, enabling technology and associated services. FDI solutions portfolio consists of a range of standards-based component products that are designed to integrate with each other and with other components within the digital library landscape.
Since 1993 the company has been active in a number of leading edge Research & Technology Development (RTD) projects and implementation groups in wide area services and distributed information management. Fretwell-Downing is lead partner in a number of these undertakings including the DALI and UNIverse projects, funded under the Telematics for Libraries programme.
Involvement with these RTD projects has influenced FDI's development ethos and product set by enabling:
The UNIverse project [2] was a large-scale project based on the concept of a virtual union catalogue for libraries. As well as linking a number of catalogues, the project offered a series of advanced library services to both end-users and librarians. Because the project involved many different libraries and systems it was developed using emergent open standards. These included the ANSI Z39.50 standard to allow search and retrieval from remote databases, the ISO 10161 Inter Library Loan protocol for ordering documents and the GEDI Recommendation for electronic document delivery. The project built on the work of previous EC-funded projects, particularly DALI [3], and addressed many of the issues relating to service distribution and globalisation currently facing the library and information community.
The project delivered many 'technology firsts'. These included:
The outcomes of the UNIverse project are being maximised in three main ways - as commercial products; as technology transfer; and as the basis of information services. In accordance with the remit of the project, FDI have further developed two commercially available products - VDX and WebView - which have been developed and enhanced from components of the UNIverse system.
The Virtual Document eXchange (VDX) software brings together Search and Retrieve, Inter Library Loan and Electronic Document Delivery. VDX has been implemented in a range of libraries to meet a range of marketplace requirements including:
WebView provides an intelligent WWW / Z39.50 gateway based environment to allow libraries to present a single gateway or ‘portal’ onto multiple heterogeneous databases. It is incorporated as the end-user access component on all digital library products available from FDI.
WebView provides intelligent support for the search – locate – request – deliver paradigm for information delivery, with features including:
Further research and/or development work, which may result in other products arising from the project, is also being conducted by other partners. This includes:
Z39.50 Harvesting Agent – this software utility collects bibliographic catalogue-related metadata from publicly available catalogues and stores them in the explain-proxy database of a UNIverse type system. This is being developed by Kyros.
Java Web Interface – an end-user client interface based on the Java language. The Technical Knowledge Centre and Library of Denmark have undertaken the work-to-date on this product.
Character Set Software - a general Character Set translation tool that converts any character set from/to Unicode. Ex Libris undertook the work on this module.
The wide interest in the UNIverse type of architecture for library services provision has also led to further technology transfer projects. UNEX, led by University College Dublin and funded under the INCO-COPERNICUS programme, involves pilot service operation of the UNIverse system in the Russian Academy of Science Library in St. Petersburg. The project involves a series of dissemination workshops throughout the countries of the former Soviet Union.
The (ongoing) PRIDE [4] project seeks to implement and demonstrate the use of a combined directory service that has integrated support for both Patron Information (e.g. information about people and organisations) and Service Information (e.g. information needed to contact remote networked services). FDI are the lead technical partners in the consortium and have prime responsibility for delivery of the PRIDE demonstrator.
The PRIDE demonstrator comprises:
The PRIDE Directory, based on combination of X.500 and LDAP standards, underpins the demonstrator. Information which is of critical importance to all the players involved in the distributed system is held in a form which enables it to be accessed and replicated simply, using open standards. The information included in the directory includes:
Although the PRIDE project is not yet complete FDI are already researching and planning ways in which these directory based services can be integrated into the VDX product.
The projects described above are influencing the development of enabling library technology that is designed to meet the changing needs of libraries as they move towards the digital library age. The Digital Library environment consists of a number of components:
 Figure 1: The Digital Library Jigsaw |
In designing products for this environment, vendors need to take into account a range of generic factors which would apply to any networked information service. However, due to the tight relationship required between traditional library service provision and networked information services, many issues faced are specific to the library sector. This is a topical theme; a series of workshops were dedicated to this at the Online 1999 Conference.
This blend of specific and generic factors leads to a requirement for solutions developed from the integration of a large number of open system components. Some of these components (e.g. ILL, Library Automation) are library specific, whilst other components (e.g. Authentication, Encryption and e-payments) are generic information technology that require knowledgeable application to the library domain. Only by applying and integrating domain specific and generic technologies will the digital library evolve to be a coherent, integrated environment as opposed to a random amalgam of resources.
UNIverse and PRIDE have contributed to the development of understanding and the practical application of solutions that make up some of the key components of the digital library.
Within generic WWW terminology a ‘portal’ is typically an entry point allowing personalised searching of web indexes and sites. Within the digital library context portals are provided through sophisticated WWW/Z39.50 gateways that allow personalised cross-database and domain searching. UNIverse contributed to FDI's WebView product, which is designed to meet this need.
Delivery of any ‘document-like object’ e.g. books, journal articles, images, electronic publications, to institutions or users. Delivery mechanisms can vary from traditional postal delivery of physical items, through ‘scan on demand’ delivery of journal articles to full online delivery of digital material. VDX, which uses the experiences of UNIverse, including its integration with RLG's Ariel TM software, is designed with this mind.
Generic industry solutions exist to the problem of strong authentication of users. Within the digital library domain these are required to guarantee the authenticity of the user for rights management issues and to secure financial transactions across an open network. PRIDE is currently addressing these issues.
Traditional inter-library loans extends into the digital domain as a mechanism for resource sharing and transaction tracking (for both financial and copyright management), was one of the concepts behind UNIverse. The digital library environment provides interfaces for transactions between libraries/information suppliers as well as transactions with end users and ISO 10161 compliant ILL is at the core of the VDX product.
Increasingly there is a requirement for integration of network web-based resources into the digital library. Some of these resources carry embedded meta-data, such as Dublin Core. Information harvesting technologies such as web crawlers allow the automated construction of catalogues of web resources. By presenting these catalogues through Z39.50 interfaces they can be tightly integrated with traditional media. This a central part of the PRIDE demonstrator.
The traditional library service of SDI (Selective Dissemination of Information) can be applied across the heterogeneous and distributed data set managed by the digital library. This can be used to generate a more holistic ‘information push’ service. Again, this is being addressed as part of PRIDE.
Industry standard encryption technology is required for strong authentication, secure payments and digital watermarking. Another component of the PRIDE demonstrator.
EU and National level funded projects are driving lead edge research into new models for information management and delivery. In order to fully maximise the impact of these RTD projects, a key aspect of the remit is to deliver the results to the marketplace, in the form of practical solutions.
The marketplace products that are being designed by Fretwell-Downing Informatics, using the experiences of UNIverse and PRIDE, are component based and are designed to fit into an emerging component-based picture of the Digital Library environment.
As technological, environmental and commercial factors impact further on the development of the Digital Library vision, FDI will to continue to seek partnerships with others who share this vision, including suppliers, national bodies and user organisations. Fretwell-Downing Informatics will also continue to conduct innovative R&D which will, in turn, hopefully lead to new standards based products which will benefit the library community as a whole.
Neil Smith
Project Manager
Fretwell-Downing Informatics
The Portergate
257 Ecclesall Road
Sheffield
S11 8NX
URL: <http://www.fdgroup.com/fdi/>
Email: Neil.Smith@fdgroup.com
Neil Smith is Project Manager for Fretwell-Downing Informatics. His role includes responsibility for managing FDI's RTD programme and he was Project Manager for UNIverse project.
For citation purposes:
Neil Smith, "Inside The Digital Library",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/fd/>
This article provides an overview of Eurotext [1], a project funded for three years under the eLib Programme and currently making the transition to a subscription service. A collaborative project, Eurotext has sought to provide an accessible learning and teaching resource for those teaching or studying aspects of European policy and integration.
In Issue 3 of Exploit Interactive Elizabeth Graham outlined the background and rationale behind the UK Electronic Libraries Programme and drew attention to a number of the projects funded under its auspices [2]. The ‘On-Demand Publishing’ area of the eLib Programme sought to facilitate access to course materials and resources in electronic form whilst promoting the acceptance and use of these resources and services. Funded under this section of the eLib programme from 1996-99, Eurotext sought to provide a meaningful learning and teaching resource for those studying aspects of European integration and policy. The project is currently making the transition to a sustainable subscription service.
European Documentation Centres (EDCs) established by the European Commission in some university libraries have, for many years, been the principal support for those teaching or researching European policy. Receiving a single printed copy of all official publications of the European Union, EDCs provided a vital, but limited point of contact with the vast amounts of EU documentation available. Currently more than half UK universities do not have an EDC, creating a divide between resource rich and resource poor institutions. Indeed, even within universities with an EDC students frequently experience difficulty accessing and locating relevant documentation. Similarly, rising student numbers and the diversification of the student body (namely the increase in part-time and distance learners) in recent years has increased the strain on library resources and academic departments in terms of both resources and staff time. In response to issues such as these the 1993 Follet Report concluded that the “exploitation of IT is essential to create the effective library service of the future” [3]. More recently this theme has been reiterated in the UK Government's ‘e-University’ initiative through which Higher Education is urged to “use technology to spread excellence [4]. Therefore the need to create enhanced access to EU documentation, and to contribute to the creation of on-line learning and teaching resources follows a sustained commitment to ITC within education.
In March 1995 a consortium of three universities, the University of Hull, the University of Lincolnshire and Humberside and the University of Ulster submitted a successful bid to establish the Eurotext project. Coincidentally the European Commission had just launched its web site Europa [5] which provided basic information about the institutions and policies of the European Union. The aim of Eurotext was to facilitate the creation of a digitized collection of the key EU texts, selected by academics, which would be appropriate to the needs of students. Made available over the web as a national – and potentially international – service, Eurotext would meet the shortfalls in resources and encourage the use of electronic resources in teaching and learning. Accompanied by an academic introduction to each topic, the documents would be searchable by both keyword and bibliographical reference and would provide easy-to-use access to key relevant documentation. A collaborative venture, the project was built on the documentation collections, technical expertise and institutional commitment of the partners.
The processes of consultation, user testing, feedback and response are vital to the creation of user-centred resources. Such resources, created by and for the user, are essential in fostering the use of ICT and electronic media in teaching and learning. From its inception Eurotext has functioned as a path-finding initiative between librarians, academics and students. To this end all three groups have been directly involved in creation of the resource at various stages. Within the first two years of the project five groups of students, at sites across the UK, provided vital feedback on the design and creation of Eurotext's interface. Specifying features such as navigation method, structure, page layout and search capacity students, as potential users, were intrinsic to the creation of an accessible interface.
Similarly, the academic community played a major role throughout the project. From providing input into the original funding proposal to writing subject introductions UK academics have been involved in the development and enhancement of the service. In the project's second year interested UK academics were invited to join an ‘Academic Network’ to contribute to the selection of texts and to the production of subject introductions. Of the 55 academics who volunteered to participate, many were Jean Monnet professors of European integration or were teaching modules funded under the Jean Monnet Programme. The criteria for text selection was primarily those texts which the academic community viewed necessary for their educational purposes.
The European Commission's policy of encouraging the re-use of copyright materials enabled a reproduction licence for the duration of the project to be agreed. The major stipulation, a clause somewhat at odds with the accepted nature of copyright, was that texts from the Official Journal and COM Documents (the series most frequently used for Eurotext) could not be reproduced as facsimile copies in PDF format. In short, they could not look like the original. The implementation of a ‘Eurotext house style’ required the development of a cost effective OCR scanning operation. The cost per page has gradually been reduced to £1, while the rate of growth now averages 1,000 pages a month – all presented in a structured way by 28 subjects and searchable by keyword. In addition to this, Eurotext provides links to relevant EU sites, a help section in response to user feedback, and 21 academic introductions.
In the meantime, Europa, the free web site which the European Commission launched in February 1995, has grown at a rapid rate to about 1,600,000 pages and includes a wide range of current full-text documents as well as supporting information. It dwarfs Eurotext in size but its aim is to be comprehensive, whereas Eurotext has always offered only the texts thought relevant for study and teaching. Users of Europa suffer from an excessively high number of accesses (80,000,000 in January 1999) with the associated delays experienced when trying to access it after 11.00 a.m. Eurotext offers ease of access, a structured format, simple keyword searching and introductions covering, background, key issues and their implications for the UK – all very popular with students faced with a growing quantity of information available. It also leads students into relevant sections of Europa, thus encouraging independent learning.
Whilst a funded project Eurotext was offered as an open access resource for evaluation purposes. As such it was used throughout the UK both within library and teaching environments and was successfully accessed in over 30 other countries. Though successful in achieving its aims and objectives, Eurotext's labour-intensive development has only been possible due to institutional support and generous funding regimes. In order to continue the service from August 1999 Eurotext needs to generate income from subscriptions to supplement the on-going institutional support.
The shift to service status has in no way slowed the development of Eurotext. The content of the site has recently been doubled through the provision of searchable directed links to specific texts on Europa. A redesigned home-page, a site-map, expanded content pages and the inclusion of further subjects have brought a new dimension to the site.
Whilst the use of technology has facilitated an increase in access to materials, the sheer volume of resources, journals, data sets and statistics currently on offer over the Internet may swamp many users. Making a web-based resource visible to its appropriate potential user groups is not an entirely simple task. Through the organization of resources into hubs and directories the sea of information available on the web has been made somewhat more accessible. Eurotext is grateful to hubs for hosting us on their resource pages and particularly to the Joint Information Systems Commission ‘Resource Guide for the Social Sciences’ for including us in their workshop programme. A continuing effort to raise awareness of the service's existence has produced a marked rise in both national and international interest. Eurotext is currently supporting users both nationally and internationally, with recent requests received from Iceland and Australia as well as UK higher and further education.
The continuation of the Eurotext service is dependent on its ability to function as a sustainable subscription service. A restructuring of pricing now includes a 30 day free trial period as well as discounts for HE institutions in the UK, for EDCs and other European information specialists. A reduced subscription rate is also offered to individuals.
Over the last four years there has been a substantial change in learning and teaching practice and in the accepted methods of knowledge access within libraries. The eLib Programme contributed greatly to this and the staff of Eurotext feel privileged to have been part of it and to have been given the opportunity to develop a meaningful user-centred resource as a valuable service for the education community.
Freda Carroll
EDC Librarian & Eurotext Co-ordinator
Academic Services: Libraries
University of Hull
HULL
HU6 7RX
United Kingdom
Tel: +44 1482 465966
URL: <http://www.hull.ac.uk/lib/euro/euroinfo.htm>
and
<http://eurotext.ulst.ac.uk/>
Email: w.m.carroll@acs.hull.acc.uk
Liz Pearce
Project Assistant
Academic Services: Learning Development
University of Hull
HULL
HU6 7RX
United Kingdom
Tel: +44 1482 465966
URL: <http://www.ctls.hull.ac.uk/>
Email: e.h.pearce@acs.hull.ac.uk
For citation purposes:
Freda Carroll and Liz Pearce, "Eurotext: The Transition from Project to Service",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/eurotext/>
When new projects and services are launched they are often expected to provide regular performance indicators which seek to provide an objective description of their service. projects and service may also be expected to reach agreements with the funding body on the minimum levels of services to be provided (often referred to as Service Level Agreements or SLAs). In this article Brian Kelly looks at performance indicators and SLAs which are relevant to web sites.
Why have performance indicators? Aren't they very time-consuming to produce? Aren't they of interest only to "bureaucrats and bean-counters"?
The interest in performance indicators reflects a need to ensure that the funding (in many cases public funding) is being spent wisely. It can be true, though, that performance indicators can be very time-consuming for service providers to produce. And many service providers would point out that performance indicators can be misleading, and may require knowledge of the service in order to make valid interpretations.
It should be born in mind that service providers themselves will also benefit from monitoring service indicators. Indications of growth in the service will help to identify when extra capacity (such as CPU power, disk space, etc.) is required. The informaton on growth can also be used to promote a service. Knowledge of failures (e.g. system unavailability) are needed in order to fix the problem. Records of persistent failures will be needed in order to identify and correct more serious procedural or systematic problems.
An example of a body which has responsibility for monitoring the performance of national services against their Service Level Definitions (SLDs) is the Monitoring and Advisory Unit (MAU) [1]. One of the roles of the MAU is "Monitoring the performance of each aspect of each service against the Service Level Definitions (SLDs) contained in its MoU." Bodies such of the MAU will be interested in gaining a greater understanding of appropriate performance indicators for web sites, and the resource implications in collecting such information.
In this article we describe performance indicators which are relevant to projects and services which provide a web site as a significant deliverable. The article reviews a number of quantifiable performance indicators, discusses the validity of the indicators and describes the resource implications in collecting and analysing the data.
The first performance indicator to consider is web server statistics. Most readers will probably be familiar with web server statistics: they are normally included in annual reports (and always seem to show a healthy growth) and often can be found on web sites themselves.
Web statistics are produced by the web server software. The raw data will be produced by default - no additional configuration will be needed to produce the server's default set of usage data.
The server log file records information on requests (normally referred to as a "hit") for a resource on the web server. Information included in the server log file includes the name of the resource, the IP address (or domain name) of the user making the request, the name of the browser (more correctly, referred to as the "user agent") issuing the request, the size of the resource, date and time information and whether the request was successful or not (and an error code if it was not). In addition many servers will be configured to store additional information, such as the "referer" (sic) field, the URL of the page the user was viewing before clicking on a link to get to the resource.
An example of a server log file is shown in Figure 1.
#Software: Microsoft Internet Information Server 4.0 #Version: 1.0 #Date: 1999-12-25 00:00:21 #Fields: date time c-ip cs-username cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs(User-Agent) cs(Cookie) cs(Referer) 1999-12-25 00:00:21 194.237.174.119 - GET /issue1/jobs/Default.asp - 200 20407 AltaVista-Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 00:03:39 194.237.174.119 - GET /statistics/ExpIntHits1.asp - 200 10519 AltaVista-Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 00:26:54 209.67.247.158 - GET /robots.txt - 200 303 FAST-WebCrawler/2.0.9+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) - - 1999-12-25 00:32:47 194.237.174.119 - GET /issue2/default.asp - 200 5332 AltaVista-Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/bg.gif - 200 300 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit-lib.org/issue1/webtechs/ 1999-12-25 01:49:54 206.186.25.7 - GET /issue1/webtechs/Default.asp - 200 24659 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) - http://www.statslab.cam.ac.uk/%7Esret1/analog/webtechs.html 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/global_home_h.gif - 200 487 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit-lib.org/issue1/webtechs/ 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/global_search_disabled.gif - 200 534 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit-lib.org/issue1/webtechs/ 1999-12-25 01:49:56 206.186.25.7 - GET /resources/images/main/local_home01.gif - 200 663 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit-lib.org/issue1/webtechs/Figure 1: Sample Web Server Log File
Figure 1 shows the first few records of the server log file for the Exploit Interactive web magazine for Christmas Day, 1999. The first four lines are comments. The first line of data shows that at 21 seconds past midnight on Christmas Day a computer with the IP address 194.237.174.119 issued a GET request (the normal method for requesting a resource) for the resource http://www.exploit-lib.org/issue1/jobs/Default.asp (note the domain name is not included in the entry in the log file, since it is the same for all entries). The resource was 20,407 bytes and the resource was transferred successfully (a 200 error code). The resource was requested by the AltaVista-Intranet user agent (a robot which indexes web sites).
The first four records are from a web robot from AltaVista. However at 01:49 there is a request for http://www.exploit-lib.org/issue1/webtechs/Default.asp resource. This request is issued by a Mozilla/2.0 browser (the code for Netscape) and the user was following a link from http://www.statslab.cam.ac.uk/%7Esret1/analog/webtechs.html. It will be noticed that this request is accompanied by requests for a number of images (.gif files).
It is not too difficult to see how this raw data can be used to provide graphical displays showing growth in the numbers of hits, profiles of the browsers used to access the web site, etc. An example of a simple display of changes in the number of hits during 1999 for the University of Cambridge Statistical Laboratory web site is shown in Figure 1 (complete data is available at [2]).
month: reqs: pages: Gbytes: --------: -------: -------: -------: Jan 1999: 465,499: 203,206: 6.773:Feb 1999: 412,622: 192,120: 5.762:
Mar 1999: 512,918: 234,660: 6.816:
Apr 1999: 520,227: 239,278: 6.902:
The well-known saying about "lies, damm lies and statistics" can be updated to refer to "lies, damm lies and web statistics". In the interpretation of web statistics there are a number of caveats to be aware of.
If the numbers of hits shows a steady growth over an extended period, is this a clear indication of a growth in the popularity of the service? The answer, quite simply, is no. If the numbers of hits on your web site grows by, say 50% over a year, the number of visitors could actually be decreasing. On the other hand, the numbers of visitors could be growing at a far greater rate.
The numbers of hits received by a web site is influenced by several factors:
As a consequence of the points mentioned above usage summaries will overestimate the numbers of visitors who make use of the information of services provided by a web site. However it should be pointed out that there are other factors which will result in an underestimation of the numbers of visitors:
A number of web statistics analysis packages provided information on the the number of user sessions (visitors). Is this information more meaningful than "hits"? We must first define the term.
A user session can be defined as a series of requests from a unique IP address within a specified period of time (often 30 minutes). So a growth in the number of user sessions will not be affected by changes in the architecture of the web site (i.e. more images added). So are user sessions a more relevant indicator? The answer is yes, but user sessions can still be misleading. User sessions will still be distorted by robots, one-off visitors and caching. They will also be affected by multi-user machines, so that if a PC is used by several people, it will be regarded as the same user. More worryingly, institutional caches or firewalls, will be treated as a single user. So if you are pleased to notice a growth in the average time spent by users at your web site, this could be the result of many one-off visitors who are behind the same firewall or cache who are accessing your web site indepentently of each other.
Jeff Goldberg from the University of Cranfield Computer Centre has written a document entitled "On Interpreting Web Statistics" which argues that web usage statistics are (worse than) meaningless [6]. Although this document is now quite old, we have seen that web statistics can be misleading. Does this mean that we should forget about web statistics as a performance indicator?
I would argue that the answer to this is no. Although, as Susan Haigh and Janette Megarity point out in their report on "Measuring Web Site Usage: Log File Analysis" "log file analysis is perhaps best viewed as an art disguised as a science." [7] web statistics do provide valuable information. However it may be necessary to carry out data mining, in order to detect patterns which may be hidden from simple analyses.
A wide range of web statistical analysis packages are available, including free packages such as Analog [8], Analog's companion package Report Magic [9], Webaliser [10] and aWebVisit [11]. Licensed products include WebTrends [12] and Accrue's HitList [13]. A more complete listing is available at Internet Product Watch [14] and Yahoo! [15].
An alternative approach to using web statistical analysis packages is to make use of externally-hosted statistical analysis services. Services such as NedStat [16], SiteMeter [17], SuperStats [18] and Stats4All [19]. As described in a recent Ariadne article [20], there appears to be a growing market for a range of externally-hosted services. They have the advantage of being easy to set up. In many cases they are free, and are funded by advertising, although there may be a licensed version which is free of advertising.
A review of the NedStat and SiteMeter services is given elsewhere in this issue of Exploit Interactive [21].
What are the resource implications in providing meaningful summaries of web usage statistics?
Initially the web server must be configured appropriately. For example, what information should be recorded? Should IP addresses be resolved (so that domain names such as bath.ac.uk will be stored in the log files) or, in order to maximise the performance of the web server, should the resolution take place when the statistics are being analysed?
It may be necessary to develop automated processes for managing server log files. Server log files can be very large. The log files for Exploit Interactive, which are kept on a daily basis, vary in size from 15 Kb to 3,844 Kb. The large size of log files will have implications for disk storage and the processing power of the computer which will carry out the analysis.
It is desirable that automated processes are implemented for analysing server log files. This will normally require use of a server system (often a Unix or NT server). Recent versions of log analysis software will enable automated analyses to be initiated from within the application. However this not be possible in entry level packages.
Although several analysis packages are freely-available, the requirements for data-mining or automation may necessitate the purchase of an expensive package, or significant software development / system configuration effort if a solution based on free tools is required. It should also be pointed out that a powerful computer may be needed to analyse log files.
As well as indicators of access to a web site, it is also desirable to provide indicators of access problems. This may include information on broken links and server availability statistics.
We've all come across 404 error messages which indicate a broken link. So we know how irritating they are. It is desirable that web services should minimise the numbers of broken links. This should include internal links and links to external resources.
Link checkers can be used to detect broken links. Many authoring packages will provide link-checking capabilities. In addition there are a number of dedicated link checking applications available, some of which are listed at Yahoo! [22].
It should be noted that simple link-checking software will typically only report on simple hypertext links (the <A HREF=".."> element) and inline images (the <IMG SRC=".."> element). In the current ganeration of web sites there are likely to be several other types of links, including links to style sheet files, links to images within style sheets, links in JavaScript, links in HTML FORMs, etc. If link-checkng software which can report on such links is not available it will be necessary to analyse the server log files in order to detect user requests for unavailable resources.
Information on the server availability is important. The server may be managed by an the service's technical staff or by a central IT department. Procedures for detecting when the server is unavailable and systematically recording the down time may be available. It this is not the case third-party services are available which can monitor services and provide automated notification in a variety of ways, such as email or by pager.
An example of a third-party which provides this type of service is WatchDog [23].
 Figure 3: The WatchDog Interface |
It should be noted that the professional version of web statistcal software such as WebTrends [12] will also provide monitoring functions.
As described in a recent article in Ariadne [24] and elsewhere in this issue of Exploit Interactive [25] it is possible to obtain details on links to your web site using services such as LinkPopularity.com [26]. As the article mentions, the quality of the information provided may be questionable. However, as with web usage statistics, as long as the reservations are born in mind, information and trends in the numbers of links to a web site may provide a valuable insight into how useful the community finds the web site.
 Figure 4: LinkPopularity |
If your web site acts as a portal, providing links to external web resources, it may be desirable to provide information on the numbers of users who follow links from your web site. This is not normally possible, if you make use of the <A HREF="http://www.acme.com/"> element. However it is possible to make use of a simple redirection to a CGI script. For example if you use a link such as:
<A HREF="/cgi-bin/redirect-link?http://www.acme.com/">
hen a user follows the link, the /cgi-bin/redirect-link can be used to record user details, before going to the destination.
Depending of the role of the web service, it may be desirable for its content to be available through use of search engines such as AltaVista. As described in a recent article in Exploit Interactive [27] there are a number of approaches to using the web to promote your web site. The approaches discussed in the section on recording the number of links to your web sites can also be used to record the number of resources held in a search engine.
The following checklist is provided as a summary of the potential performance indicators covered in this article.
| Area | Summary | Caveats | Recommendation |
| Web Statistics | Trends in accesses, users, user profiles, errors, referrals, etc. | Results distorted by caches, robots, web site architecture, etc. | Should be provided, accompanied by warnings about limitations. |
| Error Logs | Detecting errors on the web site, such as a variety of broken links | Will only detect broke links which users have attempted to access. | Will be provided as part of an analysis of web statistics. |
| Server Availability | Time the server is available. | May be costly / time-consuming to automate. | Should be provided for large-scale services. |
| Links To Web Site | Information on links to a web site. | Will only report on links which are included in search engine database. | Value not yet clear. |
| Links From Web Site | Information on links followed by users from your web site. | Requires use of intermediary script. | May be useful for portal sites. |
| Search Engine Coverage | Information on number of resources on your web site indexed by a global search engine. | Numbers provided may vary due to "document fluctuation". | May be useful for some sites. |
Brian Kelly
UK Web Focus
UKOLN
University of Bath
Bath
England
BA2 7AY
URL: <http://www.ukoln.ac.uk>
Email: <b.kelly@ukoln.ac.uk>
Brian Kelly is UK Web Focus. He works for UKOLN, which is based at the University of Bath.
For citation purposes:
Brian Kelly, "Performance Indicators for Web Sites",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/indicators/>
The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose "invisible hand" is what maintains its quality. The residual cost of online-only peer review is low enough to be recovered from author-institution-end page charges, covered from institutional subscription savings, thereby vouchsafing a toll-free literature for everyone, everywhere, forever.
Human nature being what it is, it cannot be altogether relied upon to police itself. Individual exceptions there may be, but to treat them as the rule would be to underestimate the degree to which our potential unruliness is vetted by collective constraints, implemented formally.
So it is in civic matters, and it is no different in the world of Learned Inquiry. The "quis custodiet" problem among scholars has traditionally been solved by means of a quality-control and certification (QC/C) system called "peer review" [1]: The work of specialists is submitted to a qualified adjudicator, an editor, who in turn sends it to fellow-specialists, referees, to seek their advice about whether the paper is potentially publishable, and if so, what further work is required to make it acceptable. The paper is not published until and unless the requisite revision can be and is done to the satisfaction of the editor and referees.
Neither the editor nor the referees is infallible. Editors can err in the choice of specialists (indeed, it is well-known among editors that a deliberate bad choice of referees can always ensure that a paper is either accepted or rejected, as preferred); or editors can misinterpret or misapply referees' advice. The referees themselves can fail to be sufficiently expert, informed, conscientious or fair.
Nor are authors always conscientious in accepting the dictates of peer review. (It is likewise well-known among editors that virtually every paper is eventually published, somewhere [2] [3]: There is a quality hierarchy among journals, based on the rigour of their peer review, all the way down to an unrefereed vanity press at the bottom. Persistent authors can work their way down until their paper finds its own level, not without considerable wasting of time and resources along the way, including the editorial office budgets of the journals and the freely given time of the referees, who might find themselves called upon more than once to review the same paper, sometimes unchanged, for several different journals.)
The system is not perfect, but it is what has vouchsafed us our refereed journal literature to date, such as it is, and so far no one has demonstrated any viable alternative to having experts judge the work of their peers, let alone one that is at least as effective in maintaining the quality of the literature as the present imperfect one is [4].
Alternatives have of course been proposed, but to propose is not to demonstrate viability. Most proposals have envisioned weakening the constraints of classical peer review in one way or other. The most radical way being to do away with it altogether: Let authors police themselves; let every submission be published, and let the reader decide what is to be taken seriously. This would amount to discarding the current hierarchical filter -- both its active influence, in directing revision, and its ranking of quality and reliability to guide the reader trying to navigate the ever-swelling literature [5].
There is a way to test our intuitions about the merits of this sort of proposal a priori, using a specialist domain that is somewhat more urgent and immediate than abstract "learned inquiry"; if we are not prepared to generalise this intuitive test's verdict to scholarly/scientific research in general, we really need to ask ourselves how seriously we take the acquisition of knowledge: If someone near and dear to you were ill with a serious but potentially treatable disease, would you prefer to have them treated on the basis of the refereed medical literature or on the basis of an unfiltered free-for-all where the distinction between reliable expertise and ignorance, incompetence or charlatanism is left entirely to the reader, on a paper by paper basis?
A variant on this scenario is currently being tested by the British Medical Journal [6], but instead of entrusting entirely to the reader the quality control function performed by the referee in classical peer review, this variant, taking a cue from some of the developments and goings-on on both the Internet and Network TV chat-shows, plans to publicly post submitted papers unrefereed on the Web and to invite any reader to submit a commentary; these commentaries will then be used in lieu of referee reports as a basis for deciding on formal publication.
Is this peer review? Well, it is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). The expert population in any given speciality is a scarce resource, already overharvested by classical peer review, so one wonders who would have the time or inclination to add journeyman commentary services to this load on their own initiative, particularly once it is no longer a rare novelty, and the entire raw, unpoliced literature is routinely appearing in this form first. Are those who have nothing more pressing to do with their time than this really the ones we want to trust to perform such a critical QC/C function for us all?
And is the remedy for the possibility of bias or incompetence in referee-selection on the part of editors really to throw selectivity to the winds, and let referees pick themselves? Considering all that hangs on being published in refereed journals, it does not take much imagination to think of ways authors could manipulate such a public-polling system to their own advantage, human nature being what it is.
And is peer commentary (even if we can settle the vexed "peer" question) really peer review? Will I say publicly about someone who might be refereeing my next grant application or tenure review what I really think are the flaws of his latest raw manuscript? (Should we then be publishing our names alongside our votes in civic elections too, without fear or favour?) Will I put into a public commentary -- alongside who knows how many other such commentaries, to be put to who knows what use by who knows whom -- the time and effort that I would put into a referee report for an editor I know to be turning specifically to me and a few other specialists for our expertise on a specific paper?
If there is anyone on this planet who is in a position to attest to the functional difference between peer review and peer commentary [4] [7], it is surely the author of the present article, who has been umpiring a peer-reviewed paper journal of Open Peer Commentary, Behavioral and Brain Sciences [BBS] [8] published by Cambridge University Press [9] for over two decades [10], as well as a brave new online-only journal of Open Peer Commentary, likewise peer-reviewed (Psycoloquy [11], sponsored by the American Psychological Association [12]), which entered its second decade with the millennium.
Both journals are rigorously refereed; only those papers that have successfully passed through the peer review filter go on to run the gauntlet of open peer commentary, an extremely powerful and important supplement to peer review, but certainly no substitute for it. Indeed, no one but the editor sees [or should have to see] the population of raw, unrefereed submissions, consisting of some manuscripts that are eventually destined to be revised and accepted after peer review, but also (with a journal like BBS, having a 75% rejection rate) many manuscripts not destined to appear in that particular journal at all. Referee reports, some written for my eyes only, all written for at most the author and fellow referees, are nothing like public commentaries for the eyes of the entire learned community, and vice versa. Nor do 75% of the submissions justify soliciting public commentary, or at least not commentary at the BBS level of the hierarchy.
It has been suggested that in fields such as Physics, where the rejection rate is lower (perhaps in part because the authors are more disciplined and realistic in their initial choice of target journal, rather than trying their luck from the top down), the difference between the unrefereed preprint literature and the refereed reprint literature may not be that great; hence one is fairly safe using the unrefereed drafts, and perhaps the refereeing could be jettisoned altogether.
Support for this possibility has been adduced from the remarkable success of the NSF/DOE-supported Los Alamos Physics Archive [13], a free, public repository for a growing proportion of the current physics literature, with over 25,000 new papers annually and 35,000 users daily. Most papers are initially deposited as unrefereed preprints, and for some (no one knows how many), their authors never bother replacing them with the final revised draft that is accepted for publication [5] and [14]. Yet Los Alamos is actively used and cited by the physics community [15] [16] [17].
Is this really evidence that peer review is not indispensable after all? Hardly, for the "Invisible Hand" of peer review is still there, exerting its civilising influence: Just about every paper deposited in Los Alamos is also destined for a peer reviewed journal; the author knows it will be answerable to the editors and referees. That certainly constrains how it is written in the first place. Remove that invisible constraint -- let the authors be answerable to no one but the general users of the Archive (or even its self-appointed "commentators") -- and watch human nature take its natural course, standards eroding as the Archive devolves toward the canonical state of unconstrained postings: the free-for-all chat-groups of Usenet [18], that Global Graffiti Board for Trivial Pursuit -- until someone re-invents peer review and quality control.
Now it is no secret that I am a strong advocate of a free literature along the lines of Los Alamos [19]. How are we to reconcile the conservative things I've said about QC/C here with the radical things I've advocated elsewhere about public author archiving [20] [21] [22]?
The answer is very simple. The current cost of the refereed paper journal literature is paid for by Subscription, Site License and Pay-Per-View (S/L/P). Both the medium (paper) and the method of cost-recovery (S/L/P) share the feature that they block access to the refereed literature, whereas the authors, who contribute their papers for free, would infinitely prefer free, universal access to their work.
The optimal (and inevitable) solution is an online-only refereed journal literature, which will be much less costly (less than 1/3 of the current price per page) once it is paper-free [23] and resides in open archives [24] but still not entirely cost-free, because the peer review (and editing) still needs to be paid for [25]. If those residual QC/C costs are paid at the author-institution-end (not out of the author's pocket, of course, but out of institutional publication funds redirected from 1/3 of the 3/3 annual institutional savings from serial S/L/P cancellations), the dividend will be that the papers are all accessible for free for all (via interoperable open archives such as CogPrints [26] -- integrated seamlessly into a single global "virtual" archive, mirrored worldwide, which will then have an unrefereed preprint sector and a refereed, published, reprint sector, tagged by journal name). Journal publishers will continue to provide and be paid for their QC/C while the public archive will serve as the "front end" for both journal submissions (tagged "unrefereed prepints") and published articles (tagged "refereed reprints [plus journal name, etc.]" [27].
Peer review is medium-independent, but the online-only medium will make it possible for journals to implement it not only more cheaply and efficiently, but also more equitably and effectively than was possible in paper, through subtle variants of the very means I have criticised above [28] and [29]: Papers will be submitted in electronic form, and archived on the Web (in hidden referee-only sites, or publicly, in open-archive preprint sectors, depending on the author's preferences). Referees need no longer be mailed hard copies; they will access the submissions from the Web [30].
To distribute the load among referees more equitably (and perhaps also to protect editors from themselves), the journal editor can formally approach a much larger population of selected, qualified experts about relevant papers they are invited to referee if they have the time and inclination. Referee reports can be emailed or deposited directly through a password-controlled Web interface. Accepted final drafts can be edited and marked up online, and the final draft can then be deposited in the public Archive for all, superseding the preprint.
Referee reports can be revised, published and linked to the published article as commentaries if the referee wishes; so can author rebuttals. And further commentaries, both refereed and unrefereed, can be archived and linked to the published article, along with author responses. Nor is there any reason to rule out postpublication author updates and revisions of the original article -- 2nd and 3rd editions, both unrefereed and refereed. Learned Inquiry, as I have had occasion to write before [31] is a continuum; reports of its findings -- informal and formal, unrefereed and refereed -- are milestones, not gravestones; as such, they need only be reliably sign-posted. The discerning hitch-hiker in the PostGutenberg Galaxy can take care of the rest [32].
Overall, the dissemination of learned research, once we have attained the optimal and inevitable state described here, will be substantially accelerated, universally accessible, and incomparably more interactive in the age of Scholarly Skywriting than it was in our own pedestrian, papyrocentric one; Learned Inquiry itself -- and hence all of society -- will be the chief beneficiary.
Stevan Harnad
Intelligence, Agents, Multimedia
Electronics and Computer Science Department
Southampton University
Highfield
Southampton
SO17 1BJ
United Kingdom
Email:
<harnad@soton.ac.uk> or
<harnad@princeton.edu>
URL:
<http://www.princeton.edu/~harnad/intpub.html>
<http://cogsci.soton.ac.uk/~harnad/intpub.html>
For citation purposes:
Stevan Harnad, "The Invisible Hand of Peer Review",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/peer-review/>
Sigrid Tollefsen introduces NORDGUIDE, a directory of databases in the Nordic Countries.
In 1984, NORDINFO (The Nordic Council for Scientific Information) initiated the first publication of a directory of Nordic databases, NORDGUIDE. The main purpose of NORDINFO is to promote Nordic cooperation within the field of scientific information and documentation, principally in connection with the research library system. NORDINFO shall also promote Nordic interests in a wider international context.
NORDGUIDE has been available via Internet since 1998. Since 1988 the following national bodies have financed the inventory in each Nordic country: in Denmark, INFOSCAN, in Norway RBT/Riksbibliotektjenesten (National Office for Research Documentation, Academic and Special Libraries), in Sweden Kungliga tekniska högskolan. Biblioteket (Royal Institute of Technology Library), in Iceland Landsbókasafn Íslands - Háskólabókasafn (National and University Library of Iceland) and in Finland VTT Tietopalvelu (VTT Information Service). Each country compiles the relevant databases and VTT in Finland co-ordinates the database. The directory is updated annually as new information becomes available.
The directory covers entries of around 850 databases. The following criteria are applied in selecting a database for the directory:
Portable databases either on CD-ROM or on diskettes are also included, as well as web databases, mainly those for professional purposes.
NORDGUIDE is divided into three sections: database descriptions, producers and hosts. In the Simple Search interface [1] it is possible to search NORDGUIDE either by individual countries or all Nordic countries at the same time. In addition, a more advanced searching facility is available [2] as illustrated in Figure 1.
 Figure 1: Advanced Search Interface in NORDGUIDE |
The subject terms and the database descriptions are in English, the keywords both in English and in a Nordic language. Producer is the organization which produces the database. Host is the organization which supplies passwords and manuals.
NORDGUIDE is free of charge and easy to use. It is available from [3].
For citation purposes:
Sigrid Tollefsen, "NORDGUIDE - A Directory of Databases in the Nordic Countries",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/nordguide/>
Phil Cross, Nicky Ferguson, Tracey Hooper and Emma Place of the Institute for Learning and Research Technology, University of Bristol provide an overview of the DESIRE II project.
How can we make the World Wide Web a better tool for supporting the work of the research community in Europe? This is the question that the DESIRE project [1] has been tackling since 1996, by developing new Internet technologies and using them to create high quality online information services.
DESIRE is a large project funded by the European Union under the Telematics Applications Programme [2]. Now in its second phase, it involves a team of researchers [3] from four countries - The Netherlands, Norway, Sweden and the UK, working at ten institutions including national libraries, university research centres and providers of national research networks.
The project recognises that researchers were one of the first communities to make use of the Web in their work. The Web is used as an information medium - for publishing and disseminating research data and for locating information to support further research. It is also used as a communication medium – for the discussion of research issues, both within projects and in the wider arena.
We know that users of the Web, particularly researchers, still have concerns about the Web as a vehicle for finding and delivering high quality research information:
Users would like to see technology solve these problems for them. They would like some sort of guarantee of quality - and some friendly places to start from - "subject-based community centres". These are the some of issues that DESIRE is dealing with.
The DESIRE project has demonstrated that Internet gateways built by qualified subject experts using standard Web technologies can offer Internet users unparalleled levels of quality when searching the Internet. Distributed teams of academics, university librarians and professionals are now involved in the development of large-scale "subject gateways" across Europe. These gateways offer catalogues of Internet sites and resources, which can be searched or browsed by subject. They guide people quickly and effectively to the kinds of sites that can support academic and professional work and the kinds of sites that people know they can trust. These are the highest quality portals on the Internet.
These gateways use standard metadata formats, formal classification schemes and have strict selection criteria. The editors are all subject or information experts who can make an informed judgement about the quality of an Internet site, based on its semantic content. The gateways use standardised formats and technologies, which means they are all compatible and can interoperate. Examples of such Gateways are: SOSIG [4], EELS [5], and DutchESS [6]
DESIRE is looking at issues such as scalability and interoperability. Interoperability allows us to combine distributed elements to form an integrated infrastructure. That is, to allow cross-searching across different databases or catalogues of information, which may cover different subjects or use different protocols. This includes the use of forward knowledge, a mechanism discovering what might be found in these remote collections before searching them. This speeds up search results by avoiding network overload through unnecessary querying of the remote catalogue. It also allows cross-browsing through different catalogues, creating virtual catalogues from constantly changing collections of information.
DESIRE has also looked at mechanisms for quality assurance. In addition to the quality assurance that comes from the selection of resources by subject experts for the subject gateways, we have taken part in work on the use of metadata for quality labelling systems, in particular using RDF, the Resource Description Framework [7]. Quality ratings on a particular resource being viewed by the user could be made available from a third party ratings bureau, or quality metadata could be used in a ranking algorithm within an subject gateway search engine.
DESIRE is also developing a Web site recommendation system. Gateways such as SOSIG hope to attract communities of users within particular disciplines and we hope to be able to use the expertise of our users to recommend quality resources as an addition to the catalogues we produce. Authorised users can make recommendations including comments about particular sites.
DESIRE also looks at mechanisms for automatically trawling and indexing the Internet - helping to ensure greater relevance and higher quality than the Internet search engines usually provide.
The Combine [8] harvesting and indexing software was developed under DESIRE I and gathers, parses and collects resources specifying rules for the URLs or servers that should be collected. For instance, the Harvester was used in the Nordic Web Index to create a distributed regional index containing all Web pages in the Nordic countries. It can also be used with the URLs that have been catalogued within a quality subject gateway, to index each site listed to a specified depth. This mechanism is used by the Social Science Information Gateway to create an additional database [9] of relevant, high quality pages.
During DESIRE II, the software is being extended. In particular its capabilities to index metadata are being improved, including such standards as Dublin Core [10], taking advantage of new formats such as RDF. Users should, for example, be able to search within a group of harvested data containing particular metadata fields and values, or the results of their searches could be divided into categories defined by the metadata they contain.
The aim of the design and architecture of Combine has been to provide a harvesting system which can be used for building fairly large indexes but no attempt has been made to compete with the world wide commercial search engines. Rather, it is building an index covering a small country or all universities in a region.
The harvesting policies can be formulated flexibly using allow and exclude rules, allowing distributed data collection implying that a number of servers will each have the responsibility for one or more regions or domains in a broad sense. These areas of responsibility can be assigned based upon actual network domains, organizations or geographical domains just as easily as they could be domains of human knowledge.
An important part of the architecture is an easy way to filter the sets of URLs to be indexed according to some subject or domain. Before a random set of URLs is loaded into the scheduler for processing, they are filtered through an external policy-filter. This filter, which is localised for each installation, determines what URLs are to be harvested given the policy adopted by the installation. It thus defines the region or domain a particular installation will cover.
The project has aimed at building a system which:
The range of document types that can be indexed is being extended to include perhaps postscript or Word documents, as is the range of protocols that can be harvested, to include NNTP and FTP, in addition to HTTP.
Another strand related to harvested databases is automatic classification. Although our emphasis is on collections of Web resources catalogued and classified by information professionals, there are occasions, such as with a harvested database, where automatic classification is very useful. It could also be used with the collections of recommended pages.
We have produced a report [11]on the current state of projects, methods, and problems associated with automatic classification. This work took place at Lund University [12] in Sweden, where they have created a collection of engineering resources for experimentation based on a harvested database. They have tested and evaluated different methods for creating the collection, and using different classification methods on the data to test their effectiveness. A pilot service demonstrating some of the above methods is now available [13].
The final strand of resource discovery research within the DESIRE project is the use of LDAP white pages directories. The use of directories as distributed information services is becoming widespread. Although white pages services are the major field of application, other uses are developing in this field. However, one issue which still remains open is the lack of an infrastructure that makes such distributed information more accessible to the end user.
One method the project has looked at is to use crawlers to gather all the data from LDAP directories in the Netherlands onto a single server. A second model is to use forward knowledge of the material contained on distributed directories. This system uses a central server that collects data based on the content of the individual directories - using the Common Indexing Protocol - this essentially produces simplified indexes of the records held by each directory. A search would initially be made on such a central server which would then return a list of directories that held the requested data. Remote searches can then be made on just those directories.
Such an approach would be viable for providing Europe-wide access to directory information - the goal is to have one distributed index for all directory protocols.
The DESIRE project is holding an Indexing workshop [14] in Delft, The Netherlands on 13-14 May 2000. The workshop is aimed at implementers and managers of National Research and Education Networks (NRENs), subject gateways, directory, indexing and searching innovators, information retrieval and automatic classification specialists, networked information developers and digital libraries specialists. Delegate registration costs are being met by the DESIRE project.
The DESIRE team has produced the definitive guide to setting up and maintaining a large-scale information gateway. The DESIRE Information Gateways Handbook [15] was launched in October 1999 and is freely available over the web. It promotes the development of national gateway initiatives among the academic and library communities. It also promotes the adoption of standard procedures in setting up gateways, to ensure compatibility and the potential for integrating services.
The Information Gateways Handbook has three main sections:
The Handbook is an excellent example of collaborative working, which draws together the expertise and experience of the leading gateway practitioners in Europe (in fact, the world). The Handbook received very favourable reviews from all peer reviewers and since its launch has received a great deal of positive feedback.
Internet Detective [16] is an interactive, Web-based tutorial that can be accessed for free from any Web browser. It is designed to teach people to question the quality of information that they find on the Internet, warning them that this information is not always of the quality you’d expect from an academic library and so if used carelessly can degrade academic research, teaching or learning.
The tutorial encourages people to think like a detective when looking at an Internet site:
The tutorial has proven very popular, with over 44,000 registrations to date. We have had extensive feedback revealing that it has been incorporated into many university and school curricula and Internet training programmes. As a result of this feedback we created a second edition which included support materials for lecturers and trainers (a PowerPoint presentation, handouts and ideas for classroom exercises and assessments) as well as a downloadable version for offline use.
The tutorial has been recommended by national media, notably the BBC WebWise campaign, The Independent Newspaper and USA today. Following the success of the original, a Dutch version of the tutorial written by DESIRE staff at Koninklijke Bibliotheek [17] (National Library of the Netherlands) will be made available in May 2000. A series of 10 subject-based tutorials (The Virtual Training Suite) are also being created through the UK's Resource Discovery Network (RDN) [18].
Once a user has located some quality information of interest, the next problem comes from the time it can take to download the resource due to the enormous growth of traffic demand on network backbones.
One remedy is to install a Web caching service. Local caching services are already in widespread use. DESIRE is taking the idea one step further and building a network of interconnected caches which serve local, regional, national and international users, with the ultimate aim of being able to provide a coordinated service across the research networks of Europe. Already, both UNINETT (the national research network of Norway) [19] and SURFnet (the national research network of the Netherlands) [20] have set up national caching hierarchies which are interconnected with each other and with those of other countries. The DESIRE project is attempting to spread such cache meshes across Europe.
Statistics show that Web traffic is reduced by 30-50% by sending it through a web cache system (comprising one or more servers). The analysis shows that on every level of the mesh (institutional cache, top level cache, whole mesh) the benefits of caching exceed the costs involved.
The DESIRE project has now run two workshops [21] aimed at Web cache managers looking at Cost Benefit Analyses, cache architectures and intercache communication protocols.
The DESIRE project covers a wide range of services with the common goal of making the Internet a more useful resource for academic researchers across Europe. Other countries in Europe are now considering a national strategy for gateway and Web cache development, which involves the academic and library communities.
A new project called Renardus [22] will take the DESIRE work on subject gateways forward by developing a broker service offering improved subject-based routes to Internet-accessible collections of cultural and scientific information across Europe. Renardus will be working with - and building on - existing subject gateway initiatives.
Phil Cross, Nicky Ferguson, Tracey Hooper and Emma Place
Institute for Learning and Research Technology
University of Bristol
Bristol
BS8 1HH
United Kingdom
Tel: +44 (0)117 928 7197
Email: t.a.hooper@bristol.ac.uk
URL: < http://www.ilrt.bristol.ac.uk/>
Phil Cross, Nicky Ferguson,
Tracey Hooper and Emma Place are all part
of the DESIRE Team based at
ILRT
(The Institute for Learning and Research Technology), University of Bristol.
Phil Cross is a Senior Technical Researcher who also works on the SOSIG and Renardus
projects. Nicky Ferguson is the Institute's Research Director and directs
both the DESIRE and SOSIG projects. Tracey Hooper is DESIRE Project Manager and
coordinates the project. Emma Place is a Senior Researcher working on SOSIG and
the RDN Virtual Training Suite - a successor to the Internet Detective.
For citation purposes:
Phil Cross, Nicky Ferguson, Tracey Hooper and Emma Place, "DESIRE: Making the Most of the Web",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/desire/>
|
Issue Home | Editorial | Features | Regular Columns | News and Events | Et cetera | ||
|
| ||
| Go to Top |
A UKOLN Service. Contact Us. Copyright © 1999  |
Last Updated: 7 April 2000 |
