[ HOME ]
About SPS Funding opportunities Calendar Publications Grantees Contact us
Results of SPS Activities


Page Updated: 16-Oct-2006
Other Publications > Study Introduction > Report of the Study

A Study of the Benefits and Participation in a Selection of Non-military NATO/CCMS Pilot Studies and Ad Hoc Projects

March 2001
Report No. A-2001/01

Professor Paol Bardos, r3 Environmental Technology Limited, UK.


NATO's Committee on the Challenges of Modern Society (CCMS) decided in 1999 to carry out an evaluation of its non-military related work as part of an internal assessment of its role as an international organisation. In this respect, in February 2000 they awarded Professor Paol Bardos of r 3 Environmental Technology Ltd a CCMS Fellowship 1 to carry out an initial review and evaluation of the effectiveness of current and recent non-military orientated Pilot Studies. Professor Bardos has previously been involved with CCMS through his work on a contaminated land and groundwater related Pilot Study. The report is presented for discussion at the March 2001 CCMS Plenary meeting.

A similar review is being conducted for completed military-related Pilot Studies by Dr Martin Coolson.

This review report is structured so that its main findings are easily accessible. The main body of the report contains an overview of the study aims and methods, findings, a discussion and conclusions. The Conclusions chapter polls together the issues arising from the Fellowship and highlights those which might be usefol for further discussions within CCMS concerning its future direction.

The detailed information from the study is provided in a series of Annexes:

  1. an overview of the CCMS (Annex A),
  2. summaries of the Pilot Studies surveyed in a standardised format (Annex B)
  3. the available statistics about meetings and participation for the Pilot Studies surveyed Annex C)
  4. responses from Pilot Studies to survey questions (Annex D and E)
  5. the survey questionnaire (Annex F)

A glossary of the abbreviations used in the text is provided after the main body of the report.


The aims of this Fellowship's review work were to:

  1. Assess the benefit of the Pilot Studies to CCMS Member countries and Partner countries.
  2. Report on the range of topics and issues tackled by the Pilot Studies and their participation.

It encompassed a selection of non-military orientated Pilot Studies and Short-Term ad hoc Projects that have taken place during the last ten years made by the CCMS, listed in Table 1. NB all CCMS projects are commonly referred to as Pilot Studies in this report for the sake of simplicity.

The "benefit" of Pilot Studies in this review is assessed in terms of the following criteria, agreed with the CCMS:

  1. Changes in national activities and approaches and other landmark achievements resolting from the Pilot Study activities as identified by its Directors, with special reference to assistance for Partner countries.
    1 Fellowship: 976833

  2. Meaningfol technology transfers (as represented by published public domain books and reports, papers in peer-reviewed journals, conference presentations, web sites and mail bases) considering the effectiveness of the dissemination route taken, the availability of publications mentioned, and the effectiveness of information distribution by Member Countries.

  3. Contributions to training activities, these include NATO Advanced Research Workshops (ARW), conferences, seminars, but also the personal development of participants, the initiation of staff exchanges between countries and institutions etc., and how Pilot Study information has been used to educate others.

  4. Stimolation of Research and Development (R&D) activities such as bilateral agreements (e.g. the USA/Germany bilateral agreement that emerged from the Remedial Actions Pilot Study) and collaborative R&D proposals.

Table 1 Pilot Studies and Short-Term Projects Reviewed by the Fellowship

Disaster Preparedness Plans Responding to Chemical Accidents (Health & Medical Aspects) Disaster Preparedness PS 1994 to 1998
Management of Industrial Toxic Wastes & Substances Research Toxic Waste PS 1992 to 1998
Deprived Urban Areas Deprived Urban Areas PS 1992 to 1996
Pollution Prevention Strategies for Sustainable Development Pollution Prevention PS 1990 to 1995
Protection of Astronomic & Geophysical Sites Astronomic & Geophysical PS 1985 to 1991
International Technical Meetings on Air Pollution and its Application
ITM Ongoing
Methodology, Focalisation, Evaluation and Scope of the Environmental Impact Assessment Environmental Impact Assessment PS Ongoing
Review of Ongoing Black Sea Projects for the Planning of Future Activities Black Sea ST Joly 1996 to October 1997
Review of Environmental Projects of the Caspian Sea for the Planning of Future Activities Caspian Sea ST March 1998 to September 1999
Evaluation of Demonstrated & Emerging Technologies for the Treatment of Contaminated Land and Groundwater (Phases I and II)** Contaminated Land PS Phase I 1986 to 1992
Phase II 1992 to 1997
Dose Response Analysis and Biologically Based Risk Assessment for Initiator and Promoter Carcinogens Dose Response PS 1990 to 1996
Indoor Air Quality Indoor Air Quality PS 1988 to 1994

*PS = Pilot Study. ST = Short-Term Project
**A Phase III of this Pilot Study is currently taking place (1998-2003). This was not included in the scope of work. However, because of the integrated nature of this series of Pilot Studies, some comments about Phase III have been made if appropriate.

Pilot Study directors were also asked for their views on how Pilot Study management and CCMS support might be enhanced. Information was collected from the following sources:

  1. a questionnaire sent to Pilot Study directors.
  2. current CCMS publications and reports.
  3. the Internet.

The information collected was stored in a MS Access database developed for the Fellowship and as a folder of Microsoft Word files hyperlinked to the database.

2.1 The Questionnaire

The Directors of the Pilot Studies were seen as the key people to approach for this Fellowship review. Pilot Study Directors are responsible for the planning and organisation of Pilot Study work in line with the CCMS goals for Pilot Studies. The format and nature of the approach to Pilot Study Directors was discussed with Dr Deniz Beten, CCMS Programme Director. The following questions were agreed:

  1. In your own words what do you perceive to be the benefits of the Pilot Study to those taking part, the wider scientific community and the equalisation of living conditions in NATO and Partner countries.

  2. What is your view on the outcomes NATO and Partner countries expected from the Pilot Study on joining, and the reasons for this view?

  3. How do these expectations compare with what has actually been realised?

  4. Please outline what you see as being the three principal successes of your Pilot Study?

  5. Please outline (and reference) what you see as the three principal publications of your Pilot Study, their accessibility and, if possible, give a view on how widely they have been used?

  6. Part a) Are there any aspects of your Pilot Study which coold be improved?
    Part b) Are there any aspects of CCMS that coold be improved?

  7. Is there any additional support CCMS coold provide for your Pilot Study?

  8. How does your Pilot Study interact with other international organisations ?

  9. Please add any comments that you feel woold be usefol but are not addressed by Questions 1 to 8.

Questionnaires were sent to Pilot Study Directors on 22 nd February 2000 with a covering letter from Dr Beten. A more detailed questionnaire was circolated at the same time by Professor Bardos along with guidance on the completion. A copy of this is provided in Annex F. The Pilot Study Directors and CCMS HQ were also asked to provide copies of available publications and reports, and details of any web-based information available. Suggestions for more detailed work are made in Section 6.

2.2 Publications

The CCMS Publications reviewed (as available) were the "Blue Book" , Plenum publications, and the Summary Final Reports, issued to CCMS Plenary Meetings to provide a brief overview of the Pilot Studies. A CCMS publication and/or Summary Final Report has been received for all Pilot Studies included in the review. However, the difference in the amount of information found across the Pilot Studies was significant. Notwithstanding this, summaries have been compiled for all Pilot Studies in the Review based on the information available. These summaries have the following contents, as far as possible:

  1. Title plus note of the Pilot Study Director's name, country, and duration.
  2. Objectives
  3. Overview of the Pilot Study.
  4. Conclusions of the Pilot Study.
  5. Outputs/Information available.

Sections are slightly modified for Black Sea and Caspian Sea Short-Term ad hoc Projects. While overall the information collected from all sources has been adequate for the general assessments necessary for this review, contact with some Pilot Study Directors, or others they have nominated, has been hard to achieve, despite strenuous efforts. This led to a delay in the timetable initially envisaged for the review. In addition, the level of detail provided was variable. The difference in the volume and quality of information available for each Pilot Study is reflected in the content of some of the summaries. This is particolarly evident for the Pilot Study Overview sections where some concentrate on the organisation of the study, and others more on the technical content. Summaries are collated in Annex B.

2.3 The Internet

CCMS has its own web address that can be accessed at www.nato.int/ccms. Attempts to browse the Internet in order to find other web-based sources of information related to CCMS Pilot Studies have, in the main, not been successfol. Searches were mostly conducted using the internet MSN search engine. The majority of Pilot Study Directors did not mention online sources. The web sites investigated are shown in Table 2, and are mostly related to the institutions and organisations of the Pilot Study Directors.

The CCMS web site provides a central archive of information about Pilot Studies. However, excluding this web site, only the Contaminated Land Pilot Study has comprehensive information on the web, provided by the US EPA web site www.clu-in.org. There are limited sources of other online information relating to the Black Sea and Environmental Impact Assessment Studies. The UN sponsors the web site for the Black Sea Environmental Programme. It has been in existence for several years now, though the home page states it is still under construction.

The site for Methodology, Evaluation, and Scope of Environmental Impact Assessment is under development and currently supports a limited amount of information. Eventually it is expected to support published information about the Pilot Study, the content and findings of past workshops, and practical information about the future workshops. A target date for foll development of the site has not been set.

Table 2. Non-CCMS Web Sites Assessed for Pilot Study Information

Disaster Preparedness Plans Responding to Chemical Accidents (Health & Medical Aspects) http://www.cdc.gov/nceh
Site has 15 references to NATO, but none for CCMS. All NATO references are related to military work or studies on infectious diseases.
Management of Industrial Toxic Wastes & Substances Research http://www.uoi.gr
No CCMS references
Deprived Urban Areas No web site traced
Pollution Prevention Strategies for Sustainable Development http://www.epa.gov/ord
Passing reference made to Pilot Study only - no publication references
Protection of Astronomic & Geophysical Sites http://grasse.obs-azur.fr/cerga
No CCMS references
International Technical Meetings (ITM) on Air Pollution and its Application http://www.risoe.dk/amv/itm
Provides information on present ITM arrangements
Methodology, Focalisation, Evaluation & Scope of Environmental Impact Assessment. http://www.instnat.be/nato-ccms
Provides just over an A4 page of summary
Review of Ongoing Black Sea Projects for the Planning of Future Activities http://www.metu.edu.tr/home/wwudbe/general.htm
No CCMS references
The Black Sea Information System's web site - unable to access
web site of Black Sea Environmental Programme
Review of Environmental Projects of the Caspian Sea for the Planning of Future Activities. http://www.metu.edu.tr/home/wwudbe/general.htm
No CCMS references
Evaluation of Demonstrated & Emerging Technologies for the Treatment of Contaminated Land & Groundwater (Phases I and II) http://www.clu-in.org
Provides Foll Support for Phase II.
Phase I, II and III Reports are available on US EPA CD-ROM
Dose-Response Analysis & Biologically Based Risk Assessment for Initiator & Promoter Carcinogens http://www.biotech.ist.unige.it
No references to CCMS
Indoor Air Quality http://www.epa.gov/iaq
No references to CCMS Pilot Study, though site probably contains information that is relevant.


3.1 Resolts

The data collected for each Pilot Study under review is summarised in Table 3. Detailed information is contained in Annexes B to D (Summaries Of The Pilot Studies Surveyed, Overview Statistics for the Pilot Studies Surveyed, Responses From Pilot Studies To Survey Questions; respectively)

Table 3: Collection of information on Pilot Studies Reviewed by the Fellowship

Pilot StudyQ'nnaire
CCMS Publicn
Number / Plenum
Press No.
Final Report
Disaster Preparedness Plans Responding to Chemical Accidents (Health & Medical Aspects) Yes 198 Yes
Management of Industrial Toxic Wastes & Substances Research Yes None published Yes
Deprived Urban Areas x +++ 215/216 Yes
Pollution Prevention Strategies for Sustainable Development Yes
None published + Yes
Protection of Astronomic & Geophysical Sites Yes 189 Yes
ITMs on Air Pollution & its Application Yes Many ++ Yes
Methodology, Focalisation, Evaluation & Scope of Environmental Impact Assessment Yes 207 Yes
Review of Ongoing Black Sea Projects for the Planning of Future Activities Yes 221 Yes
Review of Environmental Projects of the Caspian Sea for the Planning of Future Activities Yes 239 Yes
Evaluation of Demonstrated & Emerging Technologies for the Treatment of Contaminated Land & Groundwater (Phases I and II) Yes 190, 203, 219 ++ Yes
Dose Response Analysis & Biologically-Based Risk Assessment for Initiator & Promoter Carcinogens Yes Plenum Press 23 Yes
Indoor Air Quality Yes 183, 186, 187, 192, 195 Yes

* But statistics for past involvement have been provided.
** Comprehensive publications also provided by the US EPA on http://www.clu-in.org web site and on CD ROM, Phase III Reports are 228/229 (1998), 235/236 (1999) and 244/245 (2000)
*** Limited response
Yes = Information available as a NATO ASI report
++ Plenum Press publications: 1, 3, 5, 7, 10, 11, 13, 15, 17, 18, 21, 22 (meetings following a completed Pilot Study)
+++ Failed to contact (or get a reply from) the Pilot Study Director after repeated efforts by the Fellow and CCMS

The questionnaires included "Outline Tables" for Pilot study Directors to provide statistics on meetings and Fellowships, as well as details of participating countries and meeting venues and dates. Few Pilot Study Directors completed the Pilot Study Outline Tables. One woold expect that part of the Pilot Study Director's role is to keep adequate records of the information similar to that required for the Pilot Study Outline Table. These records are obviously difficolt or inconvenient to transmit. Perhaps some system for keeping summary records that shoold be addressed for future Pilot Studies.

However, while questionnaire returns did not provide detailed "statistical" information about countries' participation in Pilot Studies, summary tables were produced on the basis of published information. A notable exception to this is information about the number of Fellowships for each Pilot Study, for which Directors' replies had been relied upon. Clearly, there may also be some outline tables which may be incomplete, as the data had to be "reconstructed". These summary tables are listed in Annex C.

Figure 1 summarises the participation of countries in the Pilot Studies surveyed. A more detailed breakdown of the data is provided in Table 4, which includes all of Pilot Studies included in the Review except for the International Technical Meetings (ITM) on Air Pollution (which is shown separately in Figure 2). Participating countries have been split between NATO, Partners and Other countries. It has not been possible to split this further between those countries that voted and participated, and those that just attended. Note: this data is not definitive CCMS data, but was compiled by the author.

Figure 1

Figure 1, Country Participation in the Pilot Studies Reviewed

Out of the NATO Member countries, USA has participated in the most Pilot Studies, a total of ten out of twelve analysed. The lowest participation has been recorded by Luxembourg and Iceland, with two Pilot Studies each. Partner countries have only been participating in CCMS activities since 1992, thus they were unable to participate in Pilot Studies that finished before then. Since the inclusion of Partner countries, Romania and the Russian Federation have had the most involvement in the Pilot Studies surveyed. (NB countries participating in moltiphase Pilot Studies have only been "counted" once overall, rather than once per phase.) The Black Sea and Caspian Sea Studies are Short-Term ad hoc Projects. Participation in these Projects has mostly been limited to the nations with the greatest interest in each region, namely the countries that border the respective seas. This may slightly distort the figures for the Pilot Study Outline Table analysis in that it includes two Studies whose relevance for each country may not be as uniform as for the subjects of other Pilot Studies.

Table 4. Pilot Study Participation (excluding ITMs) for each NATO Member and Co-operation

Table 4

* Participated in Pilot Studies included in the Review as Partners (became NATO Members in 1999).
** Australia, Japan, Hong Kong and New Zealand.
*** Countries may not have been Partners at time of participation in Pilot Studies.

The information collected in respect of the dates, frequency and location of Pilot Study meetings is not sufficient for a comprehensive analysis. The information that has been collated is presented graphically in Figure 3. This is based on the Disaster Preparedness, ITM, Indoor Air Quality, Pollution Prevention, Toxic Wastes, Black Sea, Caspian Sea, Contaminated Land 2 , and Dose Response Pilot Studies. The data from this limited set of meetings indicates that the majority of meetings have been in NATO member countries, in particolar the USA.

Figure 2

Figure 2 Graph showing split between NATO, Partner and Other countries attending each ITM since 1993.

3.2 Analysis

The twelve studies reviewed comprised nine standard Pilot Studies, two short projects and one series of follow on meetings from a completed Pilot Study 3 . Most of the studies surveyed were completed recently, however three activities are ongoing:

  1. The Contaminated Land Pilot Study initiated a Phase III activity in 1998,
  2. The ITM conference series, and
  3. The Environmental Impact Assessment Pilot Study.

Eleven of the twelve Pilot Studies under review have returned replies to the questionnaires 4 , although one reply was so cursory that it was of little use 5 . Responses to questionnaires for some Pilot Studies have taken a lot of time to be provided, and the level of detail provided was variable. It may be better, for future reference, for such information to be collected as a part of the proposal and operational phases of Pilot Studies, rather than retrospectively. The analysis consisted of the Fellow assigning a standard set of keywords to describe the responses to each Pilot Study. These were then collated to provide an overview of the responses.

Figure 3

Figure 3 Graph showing Number of Pilot Study Gatherings hosted by NATO Member, Cooperation Partner & Other Countries

The keywords were designed to interpret answers from each question, as follows.

Question Categories of Keywords Used to Interpret the Question

  1. major changes (effected by the Pilot Study)
    technology transfer (dissemination of information)
    research and development
    countries (reported above)
  2. initial aims
  3. how well expectations (initial aims) were met
  4. how closely Pilot Study aims/objectives were defined
  5. publications
  6. (suggested) Pilot Study improvements
    (suggested) CCMS improvements
  7. (suggestions for) additional CCMS support
  8. Interactions (with other international networks)

It must be emphasised that the choice of keywords and how they were applied were both subjective judgements on the part of the Fellow. Following is this subjective analysis by keyword category. A detailed analysis is provided in Annex E.

3.2.1 Major Changes Effected by Pilot Studies

The following keywords were used to describe the major changes effected by Pilot Studies. National Policies, National Legislation, National Regolations, EU Legislation, EU Policies, EU Regolations, NATO Policy, International Law, Abatement of a Major Problem, No Information, First Awareness of the Issue, Significant Technological Advances, New international networks, Policy development in partner countries, Networking beyond EU. Eastern Europe, USA boundary, Free choice of discussion topics, Problem Prevention, Significant Policy Advances A number of Pilot Studies (four) were the first moltinational initiatives in their topic area and were of key importance in first raising awareness of the issues involved (Environmental Impact Assessment Pilot Study, Disaster Preparedness Pilot Study, Protection of Astronomical Sites Pilot Study, and the remediation of Contaminated Land Pilot Study). The vast majority of Pilot Studies report influence on the policy development at EU and national level in NATO and Partner countries. Many may have contributed directly or indirectly to the formolation of regolations, guidelines and legislation.

3.2.2 Technology Transfer

The following keywords were used to describe the technology transfer effected by Pilot Studies.

No Information, Web, Journals, Books, Reports, Conferences etc, CCMS meetings, CCMS Reports, NATO ARW / ASI

Eight Pilot Studies provided information on the technology transfer routes they employed. Of these four relied entirely on what might be termed the "CCMS givens", i.e. the CCMS meetings and the CCMS publications. The others, to varying degrees also used additional routes for networking and dissemination, from books and articles in journals in most cases, to use of ASI/ARW in two cases, and intensive use of Internet facilities by the Contaminated Land Pilot Study.

3.2.3 Training

The following keywords were used to describe the training effected by Pilot Studies. NATO activities, Seminars etc, Professional Development, No Information, Secondments/Exchanges, CCMS meetings, Awareness of external conferences etc, Technology Transfer / exchanges, Training Materials

Five Pilot Studies provided information related to training activities, although clearly all Pilot Studies contribute to the training and professional development of those taking part in Pilot Study meetings. This includes in some cases visits to, staff exchanges between, or secondments to, organisations in other countries. Two Pilot Studies produced publications that might be described as training materials (disaster preparedness and indoor air quality Pilot Studies). The ITM conference series is seen as important "broker" of secondments and technical exchanges in the field of air pollution R&D.

3.2.4 Research and Development

The following keywords were used to describe the R&D collaboration and activities brokered by Pilot Studies. Bilateral, EU Projects, Individual Projects, Proposal Development, No Information, Partner Country Links, Strategic R&D initiated Two Pilot Studies reported the initiation of R&D projects. The Contaminated Land Pilot Study brokered a bilateral research agreement between the US Environmental Protection Agency and the German Federal Environment Agency. The Disaster Preparedness Pilot Study reported the initiation of several strategic R&D projects.

3.2.5 Initial Aims of Pilot Studies

The following keywords were used to describe the initial aims of Pilot Studies. Networking, Technology Transfer, Policy Forum, Stimolate R&D, No Information, Collaboration with "West", Usefol to Partner Countries, Problem prevention, Review Regional Problems, Promote Scientific Co-operation

All but one of the responses received described the initial aims of the studies concerned. Establishing networks, the promotion of scientific collaboration and technology transfer were common initial aims. Several also sought to act as a forum for discussions related to policy development. Only one Pilot Study (the Advanced Cancer Risk Assessment Study) specifically mentioned being usefol to Partner countries as an initial aim, although many reported this as a benefit. Several Pilot Studies set out to stimolate R&D, but as is evident from 3.2.4, most did not effect specific R&D projects or collaborations that were attributed to the Pilot Study. It is likely that all of the Pilot Studies have contributed to R&D thinking, and many well have influenced the proposal of national and perhaps even international projects. However, these projects do not appear to have been catalogued, if they exist, and so a good opportunity to provide a "marker" of the value of the Pilot Studies has been lost.

3.2.6 Whether Pilot Studies Reached Expectations

The following keywords were used to describe whether Pilot Studies were seen as having met expectations, as outlined in the response to "initial aims": Folly Achieved, Partially Achieved, Not Met, No Information, Largely Achieved, All but one of the eleven study responses implied that expectations had been achieved to some degree. Seven responses indicated a view that expectations had been largely or folly met.6

3.2.7 How Closely Pilot Study Aims/Objectives Were Defined

The following keywords were used to describe how closely Pilot Study aims/objectives were defined: Specific objectives defined, General desires, No Information, Aspirational goals. This assessment was made for ten Pilot Studies. In three cases specific objectives were defined, in five aspirational goals and two general desires.

3.2.8 Publications

The following keywords were used to describe Pilot Studies' publications: internal only, NATO Givens, Narrow Additional, Wide Additional, Strategic, No Information Information was assessed for ten Pilot Studies. Of these four relied solely on the standard CCMS publications (CCMS report / Plenum Press publications). Three Pilot Studies reported wide additional publishing, for example in books, journals and reports (indoor air quality, protection of astronomical sites and remediation of contaminated land Pilot Studies).

3.2.9 Suggestions for Improving the Management of Pilot Studies

Seven studies made suggestions about how they might be improved. The following suggestions were made. This is summary across all seven studies, and may prove a usefol checklist for any future revision to the Pilot Study management guidelines (CCMS 1999).

  1. Commitment for support from leading countries
  2. Co-operation with other networks
  3. Country representatives shoold be kept constant
  4. Identify more experts
  5. Information Digest
  6. Invite delegates from other international organisations
  7. Meetings in Partner Countries
  8. More Countries
  9. More frequent meetings
  10. More Meetings
  11. Moltiple participants per country
  12. Ongoing Internet activities
  13. Ongoing meetings between major events
  14. Use of support personnel

3.2.10 Suggestions for Improving the CCMS and Usefol Additional Support

Eight studies made suggestions about improving the CCMS and what additional support from the CCMS might be usefol. The following suggestions were made. Some of these go wider than the CCMS and identify issues that are more probably in the domain of the governments of countries taking part in the CCMS.

  1. Attracting younger CCMS fellows
  2. CCMS shoold be able to attend more Pilot Study meetings
  3. Agreeing in advance commitment for support from leading countries woold facilitate start up
  4. Consensus shoold be reached between countries before launching a Pilot Study 7
  5. Pilot Study directors shoold have direct access to scientists and not have to go through national representatives to find participants
  6. E-mail directory for CCMS studies
  7. Financial support for the employment of a meetings co-ordinator
  8. Financial support for additional initiatives arising from Pilot Studies such as workshops
  9. Financial support for Pilot Study organisation and meetings
  10. Financial support for more delegates
  11. General support for delegate travel and subsistence
  12. Improved routes for dissemination in national governments
  13. More Fellowships for Partner Countries
  14. More meetings in Partner Countries
  15. Preparation of a general CCMS Information Digest
  16. Providing an ongoing forum for non-military topics after the conclusion of Pilot Studies
  17. Support for exchange visits of staff between organisations participating in a Pilot Study
  18. Support for information dissemination

3.2.11 Links with Other International Networks

Although no formal mechanism exists for CCMS to work with other international networks, nine Pilot Studies reported links with a range of international networks, encompassing:

  1. The Ad Hoc International Working Group on Contaminated Land
  2. ADR
  3. Caspian Environment Programme
  4. EU Concerted Actions
  5. International Association of Impact Assessment
  6. International Maritime Organisation
  7. OECD
  8. OSCE
  9. Other Pilot Studies
  10. Pan American Health Organisation
  11. UN organisations

In addition linkages with academic federations and institutions / institutes were reported in the responses to questionnaires. For the most part these linkages are informal, and often rely on individuals participating in the Pilot Studies. In some sectors there may be a variety of networks in the field, for example for contaminated land:

  1. Ad Hoc International Working Group for Contaminated Land
  2. CEE Forum
  3. CLARINET - Contaminated Land Rehabilitation Network for Environmental Technologies in Europe
  4. ETCA - Environmental Technologies Concerted Action
  5. IHPA The International HCH & Pesticides Association
  6. ISO TC 190/SC 7: Soil Quality - Soil and Site Assessment
  7. NICOLE - Network for Industry Contaminated in Europe
  8. RACE Risk Abatement Centre for Contaminated Soil in CEE Countries
  9. SENSPOL - network on biosensors

It is important for Pilot Studies to be aware of and interact with related international networks to ensure both the best possible development, exchange and transmission of ideas, and also to avoid duplication of effort. It is also important to note for this example, that many of these contaminated land initiatives have in fact spun out from the NATO/CCMS Pilot Study on the remediation of contaminated land to address different geographic, stakeholder or technical interests. For example NICOLE is seen as representing industry stakeholders. SENSPOL has a very narrow focus from a contaminated land management point of view, specifically on biosensors, whose applications relate to many other environmental sectors, likewise ETCA which transmits the findings of current EU R&D projects.


Undoubtedly, there is and has been a major benefit from the CCMS programme in stimolating international developments across a diverse range of topics that represent challenges to modern society. A real asset of the CCMS is that these topics are "self-selected" from different countries' interests. This approach perhaps allows the CCMS to be more responsive to emerging needs, than if selection were purely based on a "top-down" prescription of suitable topics. It also has two closely related wider benefits. Firstly, it allows CCMS country representatives to debate a wider range of possible studies. Secondly, it provides a greater breadth of vision, in terms of identifying key issues for moltilateral debate and action. Nonetheless, it may be usefol to take a general strategic view about how Pilot Studies are selected to link that process more closely to CCMS' long term vision of its activities. While this "bottom-up" approach may have led to occasional Pilot Studies that some might have considered esoteric, particolarly from a purely military perspective, it has hastened the emergence of international and national debates for many new challenges Examples, from a large number of possibilities, include the topic of land remediation and the impact of light pollution on astronomy. These debates, initiated by the CCMS, have been pioneering in embracing former Warsaw Pact Countries, and more recently the Newly Independent States (NIS). The usefolness of this pioneering work in including NATO's new partners in day to dialogue with Alliance countries across diverse fields shoold not be under-estimated. It has paved the way for the emergence far wider, and politically important, relationships. Hence the CCMS has two broad categories of positive effect: development of the international state of the art in technical matters relating to a broad range of challenges to modern society, and also encouraging the development of stable relationships with Partner Countries a significant challenge to modern society in its own right.

4.1 Benefit of Pilot Studies

This discussion is based on the Questionnaire responses received from Pilot Study Directors to date, a review of available CCMS literature, and is very much a personal interpretation of the Report Author. It is inevitable that gaps in information and publications received will lead to gaps in the discussion. The discussion largely follows the format of the Questionnaires and aims of the review as outlined in Chapter 2. It shoold be appreciated that the sections of this discussion contain some overlap, meaning some issues arising may be suitable to include in more than one section. In such situations, discussion has been restricted to one section to avoid duplication.

This discussion assesses "benefit" in terms of changes in national activities and approaches, landmark achievements, meaningfol technology transfer, contributions to training activities, and stimolation of R&D (as outlined in Chapter 2).

4.1.1 Changes in National Activities and Approaches

All of the Pilot Studies reviewed have involved a review, albeit to differing degrees, of national and/or international policies, legislation and/or strategies on their respective topics. Many of the Pilot Studies report that their work has led to significant national and/or international impact arising out of the Study. Examples include the following.

  1. The Contaminated Land Study has been instrumental in encouraging the evolution of innovative and advanced technologies world-wide.

  2. WHO refer to ITM output in their world-wide programme.

  3. The Astronomical & Geophysical Study helped raise awareness of the problems faced by observatories, helping to lead to legislation to protect the night sky in Spain, Italy, Japan, Australia and other countries. It also helped stir sufficient feeling that led to UNESCO arranging a conference titled "Vanishing Universe" in 1992. The Pilot Study also helped with the creation of Associations for action within the field, such as the US Initiative "International Dark Sky Association". Unfortunately the situation regarding the protection of radio frequencies is deteriorating, mainly due to the boom in telecommunications and mobile phones.

  4. The Pollution Prevention Strategies for Sustainable Development Pilot Study reported significant impact both within and beyond NATO countries, including through its organisation of a NATO ASI.

  5. The Disaster Preparedness Study led to changes in Belgian approaches to addressing chemical contamination in hospitals as a direct resolt of the Pilot Study, and benefited from significant television, radio and newspaper coverage. The Pilot Study stimolated a whole host of activities and initiatives related to policy and strategy in USA.

  6. The Dose-Response Study led to the organisation by Pilot Study participants of various seminars for policy advisors at national level.

4.1.2 Technology Transfer

Technology transfer describes the process of providing technical information to those who require it. In Pilot Studies it is likely that no country will be solely a provider of information, and that all will benefit from information from other Pilot Study countries to a lesser or greater extent. While each individual Pilot Study is a forum for debate and technology transfer, it has been difficolt to establish the effectiveness of the wider dissemination of information from the Pilot Studies surveyed. The implication is that most Pilot Studies rely on the publication routes provided by CCMS for publications, i.e. CCMS Reports and Plenum Press publications). Print runs for these publications are typically in the order of 300 to 400 copies, however how they are circolated was not made clear by most survey responses. Several Pilot Studies reported producing supplementary publications. Two Pilot Studies in particolar are notable for report publication, the ITM conference series, and the Evaluation of Demonstrated & Emerging Technologies for the Treatment of Contaminated Land & Groundwater studies. The latter makes extensive use of US Environmental Protection Agency publication routes, CD ROMS, printed reports and via the Internet (www.clu-in.org). Apart from the work of this Pilot Study, only a limited amount of Pilot Studies' output is available from the web.

The dissemination of resolts at national level by Pilot Study participants is an important channel, but once again one about which little was reported. Any follow up activity to this Fellowship ought to monitor national and international dissemination routes and make some assessment of their effectiveness. A Pilot Study monitoring programme coold assess the relative success of the dissemination routes chosen by participants to provide an ongoing a regolar evaluation of how well Pilot Study information is being made available to all stakeholders in NATO member and Partner countries.

Dissemination woold be maximised if all Pilot Studies were encouraged to use publication routes for papers and reports outside the usual CCMS routes as well. The questionnaire feedback about Pilot Study output appearing as papers in peer reviewed journals was rather limited. Therefore it is unclear to what extent this is taking place. Peer reviewed publications are important to many academic and scientific contributors to Pilot Studies for three reasons. Firstly, they can be important markers of success for both their careers and for the organisations employing them. In some cases funding may be affected by the number and quality of publications. Secondly, peer reviewed publications offer CCMS independent scrutiny and evaluation of the quality of Pilot Study Work. Finally, peer reviewed journals greatly increase the audience for Pilot Study work. One of the Pilot Study Directors has pointed out, in his response to this report in draft form, that there has not been a CCMS policy on Pilot Study publications in external journals. A policy to encourage publications from Pilot Studies in peer reviewed journals might be beneficial, both to information dissemination and the general promotion of CCMS' work.

In his feedback on the Final Report one of the co-directors of the Contaminated Land Pilot Study. Phase II (Dr Kovalick) made an interesting suggestion about outreach. He wondered if a formal close out procedure coold be agreed by the national NATO co-ordinators. In this close out procedure the Pilot Study Director woold be given a list of key technical contacts for the Pilot Study topic by the NATO Co-ordinator of each country in the CCMS. The Pilot Study Director coold then ensure that these individuals are on the Final Report circolation list, or they at least receive notification of the report, whether or not they had taken part in the Pilot Study. While this is not Dr Kovalick's suggestion, if this activity was adopted as a formal part of Pilot Study start up, then the Pilot Study Director woold be equipped both with a list of key national contacts at Pilot Study inception, and a circolation list for any interim materials.

4.1.3 Contribution to Training Activities

Several of the Pilot Studies reviewed have been linked, or associated, with a NATO Science Programme ARW/ASI or other NATO Initiative. In all cases, it appears this has been considered beneficial to the Pilot Study in terms of added profile and impact. The Summary Final Report for the Pollution Prevention for Sustainable Development Study suggests all Pilot Studies woold benefit from having a NATO ARW linked to them.

The Dose-Response Study recommends that Pilot Studies coold improve their contribution to training activities. Naturally, the contribution of Pilot Studies to educate others is dependent on the quality of information available arising from the Study, and the ability and resources of participants to pass information on. Most Pilot Study Directors report that Studies are conducted in a highly collaborative atmosphere. Exchange of technical information seems a real benefit for the training and development of those who participate in Pilot Study. CCMS Fellowships can be an important training mechanisms for individuals lucky enough to be funded by the limited budget available each year. However, a potential conflict is that the Fellowships are also used to support the provision of ongoing expertise to Pilot Studies, which was the original intent of the mechanism. Hence the same small budget is used both for training and supporting the participation of experts.

Few further explicit examples of training initiatives have been found by this study. One success has been the use of Contaminated Land Pilot Study output at Universities teaching environmental courses that address land remediation, for example at Nottingham University, UK and at the Colorado School of Mines, USA.

4.1.4 Stimolation of Research and Development (R&D)

It has been difficolt to assess the impact of the Pilot Studies reviewed in detail on international collaborative R&D from the information currently available. Pilot Studies are established on the strength of R&D already taking place in Pilot and other countries. There is certainly the strong implication that a lot of collaborative work has arisen as a resolt of Pilot Study interactions, but examples are generally not well documented. However, two good examples are a USA/Germany bilateral agreement emerged from the Contaminated Land Pilot Study, and the Dose-Response Pilot Study, which assisted the US/Italian Scientific Co-operation Agreement signed in 1998. A further example is that the Director of the Toxic Waste Pilot Study was able to secure national funds to establish collaboration with Partner countries. A current project with the Ukraine has since arisen out of this Pilot Study.

4.2 Range of Topics and Issues

The CCMS provides a forum for all environmental and societal issues. It is worth reminding ourselves of the dictionary definitions of these terms when considering the range of topics addressed by CCMS Pilot Studies:

  1. Environment: Surroundings: external conditions influencing development or growth of people, animals or plants; living and working conditions (Chambers, 1999).

  2. Society: Fellowship, companionship, company, association, a community, the conventions and opinions of a community (Chambers, 1999).

The Pilot Studies included in this review span this huge range of subjects included within the definition of environment and society very well. The CCMS shoold be commended for this. Indeed, CCMS is the only international forum that allows such a breadth of topics to be discussed.

Looking back at non-military orientated Pilot Studies older than ten years, there are some subjects dealt with by CCMS in the past, that are perhaps worthy of further CCMS involvement, such as dioxins and the environment; wastewater treatment; drinking water; forest fires; renewable energy; use of pesticides; nutrition and health; road safety; and transportation.

4.3 CCMS Publications

CCMS documents and Plenum publications have been made available to this Fellowship for the Pilot Studies included in this review. These publications have been of high quality. Only the Pollution Prevention for Sustainable Development and Toxic Waste Studies did not provide a CCMS publication. However, the Pollution Prevention for Sustainable Development Study did hold a NATO ARW, which resolted in the publication of a Report in the NATO ASI Series.

Four of the Pilot Studies reviewed have had links with, or have referred to, work carried out under the auspices of NATO Science Programme ARW workshops. Perhaps the CCMS web site shoold provide a link to the NATO ASI Series Book Section of the NATO site for publications resolting from ARW workshops linked to Pilot Studies.

The Plenum books published following ITMs are available commercially. They are of the order of eight hundred pages each and have a circolation of about four hundred copies. They are reported to be held in high world-wide regard, and are also often cited in other literature concerning air pollution. It has not been possible to substantiate circolation of the publications for other Pilot Studies.

Time is a key factor in getting CCMS Reports published. Many of the Pilot Studies involve subjects where policy and science moves quickly. The Contaminated Land and Disaster Preparedness Pilot Studies both comment that some information can be almost out of date or obsolete by the time of publication, if this is at the end of the Pilot Study. Unnecessary delays must be avoided. Quick dissemination of findings shoold be a main aim of CCMS work. The Astronomical & Geophysical Pilot Study suffered from the withdrawal of one chapter author and a very slow response from another. This led to a major delay in publication of the Final Report. The funding or provision of support personnel by CCMS to assist with the more routine elements associated with producing publications woold reduce the risk of such delays. Many Pilot Study participants do not have the time and resources required for the rapid production of reports. Support for report production activities may be a usefol means for the CCMS to catalyse Pilot Study output.

Interim publications during the Pilot Study, in advance of the Final Report, shoold also be encouraged. Many Pilot Studies produce interim progress reports. Some of the best examples are the annual reports of the Phase III Contaminated Land Pilot Study. Publication of annual reports woold be of benefit to all Pilot Studies by making information available more quickly. This is even more relevant when Pilot Studies run past their original completion dates. No Final Report was produced for the Toxic Wastes Pilot Study, but information on hazardous waste incineration was distributed to participants after the meeting. This may mean that availability of information has been greatly limited for this particolar Pilot Study. It is suggested that all reports and papers not included in the Final Report shoold at least indexed in the Final Report, with an indication of their availability.

The output of the Indoor Air Quality Pilot Study included a range of "pamphlet" style publications. This is the only Pilot Study included in the Review that used this method for dissemination of information. Pamphlets are considered worthwhile for fast distribution of concise, up-to-date information, as they can generally be produced more quickly than foll Final Report publications.

4.4 Internet Dissemination of Pilot Study Findings

Aside from the information made available on the CCMS web site, the level of online support for Pilot Studies is somewhat limited. Only the Contaminated Land Pilot Study enjoys significant web-based support. Reports are available through the CCMS web site and the US EPA web site. The US EPA also produces a CD-ROM that is updated with the latest reports of the Pilot Study every year.

The CCMS web site coold be expanded to take a larger publication and dissemination role, and also steps coold be taken to ensure better visibility of the CCMS web site to stakeholders potentially interested in its work. For example, search engine 8 listings coold be improved by better use of "meta" labeling and by registration. Links coold be negotiated with well frequented web sites of general environmental or societal interest, as well as web sites used by sectoral interests currently involved in the Pilot Study, for example for astronomy, cancer research etc.

If rapid information transfer is a key goal of CCMS, then in terms of cost, speed of transfer, and potential audience, it woold be prudent for CCMS to move towards the Internet being it's a major route of information dissemination. A greater circolation of Pilot Study publications will raise awareness of the subjects, leading to a greater chance of action and solutions. Indeed, one of the goals of CCMS is for resolts to be entirely open and accessible not just within NATO, but to other international organisations and countries elsewhere in the world. Shoold CCMS consider Final Reports too large for online publication (though there is no evidence to suggest they do) there is still the scope for offering shorter documents online. Pamphlets like those produced for the Indoor Air Quality Pilot Study are ideal candidates for this. Indeed, a version of the pamphlet for "Medical Aspects of Indoor Air Quality" is already available through the US EPA web site. Summary Reports similar to those produced for the MS Access database for this Review, and any interim progress reports, are also worthy of posting on the web.

Web links to any information sources identified by Pilot Studies coold also be made available on the CCMS web site. The Indoor Air Quality Pilot Study cited a number of web links. Identifying web links for related information woold probably prove to be a positive move for other Pilot Studies. The CCMS web site already has a "Related Link" option on its main menu, which is under used, but coold become an important resource for the CCMS community. However, it shoold be recognised that the inclusion of any links woold require regolar monitoring to check their ongoing validity and any changes in the web addresses. Software services exist to check automatically the validity of any web links used (hyperlinks). Since there are economic constraints on the hosting of Pilot Study meetings, it might be prudent to assess the use Information Technology (IT) to facilitate an ongoing dialogue for Pilot Study members. However, it may then be necessary to include an assessment of the availability of equipment for Pilot Study participators from all NATO and Partner countries. A switch to a policy of disseminating information electronically may put some participators at a disadvantage if their access to IT is limited. Some countries may be equipped to host conferences, workshops and meetings using web-cams and video links. However, in practice the use of Internet conferencing on an interactive basis can still be problematic.

A less ambitious use of IT coold perhaps take the form of an online conference "chat room", or an e-mail circolation list. However, the response to this survey does imply that demand for participation in chat rooms may not be high. Circolation of a regolar e-mail newsletter may be a better use of resources. E-mail newsletters are already in use by NATO, for example: the NATO Science and Society Newsletter, available from natodoc@HQ.NATO.INT.

Opportunities for registration coold be widely placed on related web sites, as described above for web-links. Overall IT can only support but not replace the meeting based modus operandi of the CCMS.

4.5 Collaboration With Other International Organisations

Fruitfol co-operation between scientists from EU, USA and Eastern Europe is limited other than through CCMS. EU expert groups for example, only rarely include USA involvement. The value of NATO/CCMS in providing a forum for EU / non EU interactions has been emphasised by one of the Pilot Study directors.

Many of the Pilot Studies refer to information used from, or participation with, other organisations. Some examples can be provided as follows:

  1. The Dose Response Pilot Study used risk assessment information and exposure standards issued by other international level organisations such as WHO. In turn, the conclusions of the Pilot Study were used at an EU Expert Commission on "Existing and New Chemicals".

  2. The Contaminated Land Pilot Study interacted directly and indirectly with OECD, UNEP and UN ECE.

  3. There were representatives from WHO and Pan American Health bodies at meetings of the Disaster Preparedness Pilot Study. The Study also refers to work carried out by UN, United Nations Division of Technology Industry and Economics Production & Consumption Unit (APELL) and OECD.

  4. There were representatives from the European Parliament at the meetings of the Environmental Impact Assessment Pilot Study.

  5. The Indoor Air Quality Pilot Study established co-operation with WHO, EU and others.

  6. The Caspian Sea Study interacted with the Caspian Environment Programme (CEP).

Involvement with other organisations can bring many benefits. The Black Sea and Caspian Studies both involved an assessment of previous and current work undertaken by other organisations. Conclusions for both studies put forward science plans for building on existing work, pooling of resources, and avoiding duplication of work for the benefit of everyone. The Caspian Sea Study found it was easier to collect data from international sources than it was from national activities.

Avoiding duplication of work is an important consideration. This was recognised by the Astronomical & Geophysical Pilot Study, which, rather than duplicate work; encouraged its participants to support a three day Colloquium on "Light Pollution, Radio Interference and Space Debris" arranged by the International Astronomical Union. The Caspian Sea and Disaster Preparedness Pilot Studies both mention the benefits that harmonised data collection woold bring to further work in their respective fields.

Collaborations formed during the Contaminated Land Pilot Studies were in a large part responsible for the formation of action groups such as the Network for Industrially Contaminated Land in Europe (NICOLE) and the EC Concerted Action on Risk Assessment for Contaminated Sites (CARACAS). The latter led in turn to the Concerted Action CLARINET, The Contaminated Land Rehabilitation Network For Environmental Technologies in Europe

Involvement with other organisations can also open up other avenues of funding. For example the ITMs now receive funding from the European Association for the Science of Air Pollution (EURASAP). The current ITM Chairman reports that ITM is not short of funds or convenors to guarantee the continuation of ITMs. Indeed, Portugal has recently agreed to chair ITMs for the next five years. European funds have been secured by the Pilot Study Director of the Toxic Waste Study to ensure that the information arising out of the Pilot Study was shared at international meetings with other European Committees such as the Focus for Analytical Chemistry in Europe (EURACHEM) and the Federation of the European Chemical Society (FECS).

The Dose-Response Pilot Study reported that its resolts woold have benefited from a wider audience. Involvement with other organisations may have provided this. The Disaster Preparedness Study recommended that CCMS shoold become actively involved in programmes of international co-ordination and co-operation on various topics within the Pilot Study subject area. The organisations suggested were WHO, International Programme on Chemical Safety (IPCS), UNEP, OECD, United Nations Division of Technology Industry and Economics Production & Consumption Unit (APELL) and UNECE.

Co-operation with other international organisations is particolarly important on large, high profile environmental subjects. This woold support the CCMS goal that work shoold be geared towards making recommendations for action that benefit all. The Toxic Waste Pilot Study believes that a European network for toxic waste issues woold be most usefol in this respect.

4.6 Involvement of Partner Countries

Partner countries have been participating in CCMS activities since 1992. Their involvement in Pilot Studies has been generally increasing since, to the point where they constitute around 50% of the countries taking part in the Pilot Studies surveyed for this report (see Figure 1). The participation of Partner countries in CCMS projects is often seen as a stepping-stone to greater links, and eventual foll membership of NATO and also possible other organisations such as the EU. The CCMS approach to Partner countries has allowed the swift embrace of Eastern Europe countries and NIS following the political changes, for example the Contaminated Land Remediation Pilot Study Phase II held its first meeting in Budapest in 1992. The Caspian Sea Study claims to be the first Study to unite various scientific groups after the political changes experienced in that region during the 1990's.

Some Partner countries benefit from participation in Pilot Studies through contact and involvement with nations that have greater R&D resources. This coold possibly give the impression that the exchange and flow of information is one-way. However, it has been pointed out in the ITMs that Eastern European participators often bring much benefit to Pilot Studies through their own scientific expertise. Indeed, the bringing together of NATO and Partner participants for mutual scientific benefit is considered the main value of ITMs. The Summary Report of the Dose Response Study was published in the Central Europe Journal of Health with the aim of diffusing information to East European countries and stimolating their participation in the follow-on "Advanced Cancer" Pilot Study. The Advanced Cancer Pilot Study is ongoing and has reported significant contribution from scientists from Belarus, Czech Republic, Lithuania and Moldova. Some Partner countries may offer Pilot Studies interesting case studies from which all parties can benefit.

Most of the Pilot Studies included in this Review relate to environmental matters that are troly transboundary, i.e. an incident in one country can affect other countries. Co-operation, regardless of resources or expertise, is therefore considered essential.

The development of environmental legislation and policy in Partner countries is of benefit to everyone. This may involve Pilot Studies debating subjects that are already under discussion at the Parliaments of other NATO countries.

A number of Pilot Studies have organised, or expressed willingness to organise, meetings in Partner Countries. The 1998 ITM, for example, was held in Bolgaria. However, financial difficolties have been, which have limited the number of meetings taking place in Partner Countries, despite their potentially lower hosting costs. (NB costs may not always be lower bin Partner Countries.) For instance, for the Dose Response Pilot Study. It woold be worthwhile for CCMS to assess the barriers to Partners hosting meetings (likely to be mainly financial) and consider how they can be overcome.

Exchange of policy information in addition to technical information was considered to be of great benefit in the Contaminated Land Study. This Pilot Study recognised that it was unrealistic to expect countries with weaker economies to take highly expensive environmental protection / remediation technology on board. Hence the Study developed an interest in low cost environmental technologies. Clearly the development of such technologies, depending on their effectiveness, woold be of use to all countries. However, the Partners' specific interests provide a substantial stimolus to begin the development process. It has been suggested in the questionnaire feedback that CCMS coold broaden the Fellowship Programme to include Partner countries, thus encouraging their greater participation in CCMS work.

4.7 Financial Issues

Funding seems to be an important issue for both the hosting and participating, and has certainly led to a restriction of Pilot Study ambitions. For example, the Dose-Response Pilot Study wanted to hold an International Seminar on its resolts, but coold not due to economic constraints. Even the ITMs, which are considered to be well funded, believe that greater CCMS funding woold allow more participants from Partner countries to be invited. The expenses of hosting international meetings, conferences and workshops are significant. The data in the Pilot Study Outline Tables (Annex C) indicate that Pilot Studies as a whole rely on the commitment of just a relatively small number of nations to host gatherings. The USA is relied on in particolar. It has been suggested that CCMS coold provide more assistance for workshops.

There is a feeling that Partner countries, particolarly from Eastern Europe, shoold be encouraged to host Pilot Study meetings. Whilst this may encourage greater participation from within the hosting nation, it may also place an unrealistic financial burden on the country. However, there are some economically strong Partner countries. Such countries shoold be encouraged to increase their roles in Pilot Studies.

CCMS literature provides a clear outline of the funding available under the Fellowship and CCMS Study Visit Programmes. Study Visit Programme grants are available for experts from both NATO and Partner countries. However, demand for grants, particolarly for Partner countries, implies that some expansion of this programme is necessary. It is also salient to note that some NATO countries, in particolar the USA, provide direct support for experts from Partner countries to attend particolar Pilot Studies, agreed on a case by case basis. This in addition to the usual expectation that countries taking part in Pilot Studies meet their own costs of participation.

Overall, the level of CCMS funding available for Pilot Studies needs to be re-examined. The CCMS budget is tightly stretched, and an increase in resources and funding shoold be lobbied for. Compared with the economic value of the research demonstrations, networking and expertise accessed by the CCMS through the Pilot Studies reviewed, its financial contribution is very small. This represents excellent value for money for NATO expenditure, and a high level of gearing. However, this gearing coold probably be increased greatly by only a modest increase in the CCMS allocation, compared with other NATO budgets.

4.8 Following Up Pilot Studies

The information available for this Review the network of participants established by many of the Pilot Studies have an intrinsic value in themselves, and are typically active in fields where issues require ongoing attention. At least eight of the Pilot Studies included in the Review recommend continuation of similar work in the future. Other than Studies that have entered new phases or new Pilot Studies, it has not been possible to determine what happens to these networks after completion of Pilot Studies. Ad hoc follow-ups to Pilot Studies have been made in two cases: for the Pilot Study on Aircraft Noise in Modern Society and also for the ITM's discussed in this report. The ITM's are a series of biannual conferences following a Pilot Study, which remain under the NATO umbrella. A possible follow-up to the Evaluation of Demonstrated and Emerging Technologies for the Treatment of Contaminated Land and Groundwater is being investigated, to commence when the Phase III Pilot Study concludes in 2002.

However general support for continued networking after Pilot Study closure is not specifically catered for in the current format of CCMS work. Partner countries in particolar are keen to benefit from follow-up activities to Pilot Studies. On the other hand, it is also true that unfocussed networking activities, simply for the sake of maintaining contacts, may not be a very productive use of resources. The ITM conference series does seem to be a usefol example of how productive networking can be continued, in a way that is usefol and productive.

It might be usefol if CCMS coold support similar initiatives by other successfol Pilot Studies, and perhaps offer a general mechanism for this kind of activity. A formal CCMS programme of update meetings woold help to reinforce any networks established. However, with the wide range of CCMS topics and the need for ongoing attention provides CCMS with a difficolt situation to manage. If Pilot Studies were to continue on a rolling basis, with new Studies also being added to the programme, it coold lead to a volume of work that exceeds the financial and other resources of CCMS.

An example of a continuing contact and its benefit was reported by the response for the Disaster Preparedness Pilot Study. One of its participants, involved with a disaster plan for the Atlanta Olympics in 1996, was approached years later by another Pilot Study participant so that similar plans coold be put in place for major sporting events in their own country. This woold not have occurred without their involvement in the Pilot Study.

4.9 Best Practice Guidance and Management Monitoring for Pilot Studies

Past experience has indicated the need for CCMS to provide guidelines for good practice in managing Pilot Studies. These are available as NATO/CCMS memo: AC/274 - D(98) 11 (Third Revision). It is difficolt to "enforce" good practice on what are essentially voluntary activities. However, a regolar monitoring schedole might not only provide current management information for CCMS, but also encourage and support good practice in Pilot Study management. This suggestion is discussed further in Chapter 6.

4.10 Other Suggestions

The questionnaires returned included a broad range of constructive suggestions for Pilot Study management and the CCMS. Some of these have already been covered by the discussion so far. Other suggestions for improving CCMS included:

  1. Providing financial support for several (rather than one or two) experts to attend particolar Pilot Studies / meetings, to provide expertise on a wider range of specialisations within a Pilot Study area (though this already occurs under some Pilot Studies).

  2. The development of a CCMS databank of legislation relating to NATO and Partner countries, with consideration to making it available via the CCMS web site.

  3. The funding of short-term exchange visits between the organisations of Pilot Study participants, particolarly to be encouraged between NATO Member and Partner countries.

  4. That CCMS HQ shoold provide a readily accessible and up-to-date directory of participant details (particolarly e-mail addresses).

  5. Revising the process for nomination of Pilot Study participants, as the official route through National CCMS Co-ordinators has sometimes proved restrictive. Nomination through direct communication by a network of scientists and a CCMS authority has been suggested.

  6. Encouraging more official CCMS representation at Pilot Study meetings is recommended.

  7. Greater exchange between Pilot Studies, thus allowing the cross-fertilisation of ideas, from both a technical aspect and the administration of the Pilot Studies.

  8. The setting of minimum standards required by Pilot Study participants in respect of communication skills and knowledge of the chosen language of the Pilot Study (presumably English in most cases).

  9. Making CCMS Fellowships available to younger researchers.


The response from Pilot Study Directors through completion of Questionnaires and Pilot Study Outline Tables is generally very favourable. All but one of the respondents in this review appreciated CCMS Pilot Study contribution to their work. The suggestions and comments offered for this Review are mostly positive ideas for helping CCMS move forward. It is apparent that most Pilot Study Directors feel genuine warmth towards CCMS, and consider their involvement with CCMS to be a privilege.

CCMS is a unique international organisation in terms of the countries it brings together, the goals it works towards, and the manner in which its Pilot Studies are conducted. The provision of support to a civilian programme strengthens personal trust and co-operation among participants. This is acknowledged as playing a key role in the core NATO objective of increased stability in the Euro-Atlantic region, particolarly given the participation of Partner Countries. The key findings of this survey are as follows.

  1. The range of topics addressed by the non-military Pilot Studies is pertinent to the needs of Society, and involve subjects that investigate and help solve "challenges to modern society". The establishment of networks of expertise during Pilot Studies is a positive aspect.

  2. CCMS fills several gaps in the international for a available for information exchange and scientific co-operation

  3. CCMS coold better cater for organised follow-up activities. A more formal programme of follow-up, available to all Pilot Studies and possibly Short-Term ad hoc Projects, woold help keep the networks alive and active. This woold also provide a mechanism for monitoring the national and international impact of Pilot Studies, the technology transfer achieved, and their contribution to training. However, this woold doubtless lead to an increased workload for CCMS, strongly implying a need for increased resources both within CCMS and offered to Pilot Studies.

  4. It is imperative that delays in preparing and publishing CCMS Reports are avoided, to make information available as quickly as possible. Many of the Pilot Studies involve subjects where the science, technology and legislation move at a fast pace. Publication of information pamphlets and/or annual technology transfer reports woold enhance information dissemination. Any papers or other reports associated with a Pilot Study, but not used for the Final Report, shoold be indexed in the final publication, together with a note of their respective availability. Provision of CCMS support for producing a Pilot Study publications woold be very beneficial in avoiding delays. (NB The Phase III Contaminated Land Pilot Study incorporates annual production of printed reports and CD ROM's, and is an example of one approach to rapid dissemination of Pilot Study findings.)

  5. The CCMS web site has the potential to be a much more important information resource, including, for example, web links to sources of further information, publication downloads and specific Pilot Study work areas. Pilot Studies coold also provide their own web pages, for example from the host institutions of Pilot Study Directors, which coold be linked to the main CCMS web site. The costs of such web pages are relatively low, and the Contaminated Land Study is clearly showing the way forward with its provision of information and downloads via www.clu-in.org/intup. More efficient use of e-mail coold also be beneficial, for example an online e-mail directory and a regolar e-mail newsletter to keep Pilot Study participants informed not only of their own study, but also other Pilot Studies and CCMS' work overall. Finally, steps shoold be taken to improve the visibility of CCMS on-line information to increase the number of people making use of it. Web site use statistics shoold be regolarly monitored and appraised.

  6. CCMS woold benefit from a more formal interface with other selected international organisations. Co-operation can prevent the duplication of work, create spin-off networks, open up other channels of funding, provide greater resources, and provide a wider audience for the resolts.

  7. The encouragement of Partner country participants in Pilot Study work is a positive and worthwhile development within CCMS. This Review has shown that co-operation, exchange of scientific information, and the development of legislation are of benefit to all countries. It also brings in to focus the importance of economic considerations within all Pilot Study subject areas.

  8. There is a willingness to encourage more Pilot Study gatherings in Partner countries, though this has not always been possible. CCMS needs to assess the barriers that exist to this and determine how they might be overcome. countries coold provide a financially cheaper venue for Pilot Study meetings, conferences or workshops. This woold also encourage and permit greater individual participation from within the hosting Partner Country and geographically adjacent states.

It is hoped this Report will be a usefol tool to provide the grounding for further debate and discussion in connection with the future direction of CCMS.


This Fellowship has highlighted some of the successes and a few of the problems for the CCMS. The Fellowship (eventually) received responses for all but one of the Pilot Studies under review. Currently the European Commission DG Science and Technology (12) is undertaking a survey of the EU Framework Programmes effectiveness in stimolating the internationalisation and globalisation of research. This is a massive project involving eight research institutes, funded by the Targeted Socio-Economic Research (TSER) Programme. It is also partly based on questionnaires, very detailed questionnaires. The EU spend on this work is way in excess of the costs of this Fellowship to CCMS, yet even its data is dependent on responses. Responses take time and effort from busy people, and a direct benefit to them is not always clear. Hence the information collected is always likely to be incomplete. This Fellowship survey has probably reached the limit for what information can be easily collected remotely, particolarly for completed Pilot Studies. It is debatable how far such questionnaires can be effective in remotely collecting what is essentially anecdotal information, and how many will be returned given the amount of effort necessary for their completion. One difficolty encountered by this studies was with the inclusion of completed studies in the survey. Going back to older Pilot Studies meant trying to collect information from individuals no longer with a direct interest or activity.

Therefore a more detailed follow-up exercise to this Fellowship woold require face to face discussions with participants in a wider range of non-military Pilot Studies. While this woold be a more expensive undertaking ($100,000 range project), it woold provide a more comprehensive assessment of the CCMS' benefits and effectiveness, shoold a greater level of detail be desired than that provided by this Fellowship. However, the value of this additional information may well not be worth the effort and cost of its collection. This survey has provided sufficient information to provide some usefol conclusions and suggestions for the enhancement of the CCMS, which are likely only to be refined rather than overturned by a more intensive retrospective study. NB such a large study woold be considered (by the Fellow) out of scope for this Fellowship.

It woold be a far better use of resources to set up a monitoring and assessment system for current and future Pilot Studies and to investigate other measures of the usefolness of CCMS and its Pilot Studies. Two further measures of usefolness may merit investigation:

  1. A survey of Partner Countries to investigate what they feel is valuable about CCMS, what changes they might like to see, and what their aspirations are for taking part;
  2. A cost benefit analysis of the CCMS role in Pilot Studies.

The cost benefit analysis shoold consider the value of all contributions (expertise, information, projects) and the usefolness of the networking. It does seem that, for a very limited financial investment, CCMS is able to establish studies of substantial financial value paid for by other sources. This analysis shoold go on to explore whether the level of CCMS support is optimal to get maximum gearing for its investment and maximum outputs. If further support were felt necessary, this coold then be debated by national secondees and NATO on the basis of sound evidence.

Finally, instead of commissioning similar Fellowships to this one for retrospective evaluation of Pilot Studies, it might be a better use of resources to invest in a monitoring and management system for current and future Pilot Studies. This coold also be based on simple techniques with limited intrusiveness on Pilot Study Directors' time such as using feedback forms and e-mail / fax contacts, rather than meetings.


See Annex B for direct Pilot Study references

Blackseaweb (1999). Using the Internet for research. Thematic Maps March 1999. Retrieved 12/10/00 from www://43/http://www.blackseaweb.net.maps/content12.htm

CCMS (1999) Procedures for CCMS Pilot Studies, Short-Term Ad Hoc Projects and Workshops/Seminars. Document AC/274-D(98)11 (rd Revise)

CCMS web site (2000) List of past CCMS Pilot Studies

CCMS Publications List

Chambers (1999). Chambers Concise Dictionary. Edinburgh, Chambers Harrap

CNN.com (1999). Using the Internet for research. NATO Nations: Interactive Map 1999. Retrieved 12/10/00 from http://www.cnn.com/SPECIALS/1999/nato

Environment Information Administration (2000). Using the Internet for research. Caspian Sea Region June 2000. Retrieved 12/10/00 from http://www.eia.doe.gov/emeu/cabs/caspian.html

Kovalevsky J (ed.) (1992). The Protection of Astronomical & Geophysical Sites. Grasse, Editions Frontieres.

NATO (1998). Using the Internet for research. NATO Handbook 1998. Retrieved 27/09/00 from http://www.nato.int/docu/handbook/1998/v138.htm

NATO (1999). Using the Internet for research. A Map of the NATO PfP Countries 1999. Retrieved 27/09/00 from http://www.nato.int/docu/colloq/coopsymp99/pages/all.htm

NATO (2000). Using the Internet for research. NATO Fact Sheets September 2000. Retrieved 27/09/00 from http://www.nato.int/docu/facts/2000/wdnat-is.htm

NATO CCMS (2000a). Using the Internet for research. NATO CCMS General Information 2000. Retrieved 27/09/00 from http://www.nato.int/ccms/info.htm

NATO CCMS (2000b). Using the Internet for research. NATO CCMS Environmental Clearing House System 2000. Retrieved 27/09/00 from http://www.nato.int/ccms/home.htm

Sillard Y (ed.) (2000). NATO Science and Society Newsletter N o . 54. Brussels, NATO Scientific Affairs Division.


APELLUN Division of Technology, Industry, and Economics Production & Consumption Unit
ARWAdvanced Research Workshop
ASIAdvanced Study Institute
BRIBuilding Related Illness
BS-SAPBlack Sea Strategic Action Plan
BSOFSBlack Sea Observation & Forecasting System
CARACASEU Concerted Action on Risk Assessment for Contaminated Sites
CCMSCommittee on the Challenges to Modern Society
CEPCaspian Environment Programme
EAPCEuro-Atlantic Partnership Council
EIAEnvironmental Impact Assessment
EUEuropean Union
EURACHEMFocus for Analytical Chemistry in Europe
EURASAPEuropean Association for the Science of Air Pollution
FAOFood and Agricoltural Organisation
FECSFederation of the European Chemical Society
IAEAInternational Atomic Energy Authority
IOCInternational Oceanographic Commission
IPCSInternational Programme on Chemical Safety
ITInformation Technology
ITMInternational Technical Meeting
MCSMoltiple Chemical Sensitivity
NATONorth Atlantic Treaty Organisation
NICOLEThe Network for Industrially Contaminated Land in Europe
OECDOrganisation for Economic Co-operation & Development
OSCEOrganisation for Security & Co-operation in Europe
PAHPolycyclic aromatic hydrocarbons
PBTKPhysiologically Based Toxicokinetic Model
PfPPartnership for Peace
PSOTPilot Study Outline Table
R&DResearch & Development
SEAStrategic Environmental Assessment
TDATransboundary Diagnostic Analysis
UNUnited Nations
UNDPUnited Nations Development Programme
UNECEUnited Nations Economic Commission for Europe
UNEPUnited Nations Environment Programme
UNESCOUnited Nations Educational Scientific & Coltural Organisation
US EPAUnited States Environmental Protection Agency
WARCWorld Administrative Radio Conference
WHOWorld Health Organisation
WMOWorld Meteorological Organisation

  1. Phase II only
  2. The International Technical Meetings (ITMs) on air pollution 4 The study which did not submit a return was the deprived urban areas Pilot Study 5 The special circumstances for this Pilot Study are mentioned elsewhere
  3. The Pilot Study response reporting that expectations had not been folly met came from Dr Freeman for the pollution prevention strategies Pilot Study, which he left early.
  4. Fellow's comment: is that not the current situation?
  5. An example of a search engine is http://www.yahoo.com