Utilization of Technology-Enhanced
Delphi Techniques

Charles G. Andrews
Jeff M. Allen
University of North Texas

Abstract

This paper discusses the Delphi consensus-building technique. Also discussed are the Delphi Technique’s history, the process, and some advantages and disadvantages found in the literature. Finally, this paper examines a technology-enhanced version of the process. The study provides researchers interested in using the Delphi Technique in conjunction with technology with a process that is easily duplicated. Nine university professors/instructors agreed to address one question: “What information should the student be aware of when working on a team project?” The question is unimportant. However, the results of the Delphi consensus regarding this single question are provided, along with a framework for conducting future Delphi studies, using computer technology. Cost savings and time reduction are important advantages to be considered when conducting a Technology-Enhanced Delphi study using computer technology.

Introduction

Throughout history, humans have wanted to know what will happen tomorrow, next week, or even next year. Also, they want to know what to do to ensure that what they want to happen will happen. Greek mythology provides an example of this “need to know.” The oracle of Delphi could answer all questions if the right questions were asked and a tribute was paid (Buckley, 1995; “Predicting the Future,” 1995). This desire to know the future remains, as is demonstrated, for example, by the increasing number of psychic networks available to seekers.

The Delphi Technique is a tool used by organizations and researchers worldwide. Like all processes and techniques, it is natural for the Delphi Technique to be analyzed for ways to enhance the technique and make it even more appealing to both organizations and researchers. The last decade of the 20th century and the beginning of the 21st century have been a time of change. Every organization and every process used by organizations is undergoing change. Few tools of the new technology are so ideal that changes could not improve their function or use. First, it is important to identify the Delphi Technique, its foundation, advantages, and disadvantages and when it can be utilized. Next, the question arises as to how the technique can be enhanced so as to reduce what are seen as its disadvantages. This study specifically addresses the following two questions: How can computer technology be combined with the traditional technique to enhance its performance? Can a merger with this technology help to reduce the effect of one or more of the inherent disadvantages of the Delphi Technique?

Today’s competitive world requires that organizations utilize new and innovative tools. Some tools used are the Focus Group Technique (developed in the 1940s by R. K. Merton and colleagues); the Nominal Group Technique (brainstorming technique developed by A. Delbecq and A. de Ven in 1968); and the Delphi Technique (developed by O. Helmer and N. Dalkey in 1953).

Overview of Delphi Technique

In the mid 1950’s the Rand Corporation hired Dalkey and Helmer, then pioneers in Delphi research. Dalkey and Helmer described the “Delphi Technique” as providing “the most reliable consensus of opinion of a group of experts” (Dalkey & Helmer, 1993, p. 458). The U.S. National Cancer Institute later modified the technique to prioritize corporate projects, and the government of Taiwan used the Delphi Technique to prioritize various aspects of their information technology industry. The technique reduces, if not eliminating, some of the limitations of both the Focus Group Technique and the Nominal Group Technique.

Buckley (1995, p. 17) quoted Cary and Salmon’s description of the Delphi Technique from their July 1976 Agricultural Extension Research report as “`a tool for discovering agreement, and identifying differences rather than forcing consensus.’” The “Predicting the Future” (1999, para. 1) article provided this description by McNamee: The Delphi Technique is “`an interactive and personality-free team approach to decision making.’” According to Kaynak, Bloom, and Leibold, (1994) “The Delphi Technique attempts to make constructive and systematic use of informed intuitive judgment” (p. 19).

Viewed as a qualitative technique (Hinks & McNay, 1999), the Delphi Technique is used (a) when there is an emotionally-charged situation, (b) when a decision is opinion-based, (c) when there is a need for expert input and the experts are not in a central location, and (d) when better results might be achieved if the experts did not meet face-to-face. One way to work around an emotionally-charged atmosphere is not to bring the emotionally-involved individuals to the same location. This action also applies when the decision is opinion-based. When multiple experts are involved, there is the potential for personality conflicts. The Delphi Technique fits these situations by insuring the anonymity of participants.

The main premise of the technique is anonymity; participants are not aware of the identity of other participants. The second premise is based on central tendencies. Simply stated, through repeated cycles of assessment, there is a tendency for opinions to move toward a central point of consensus. What are the basic steps in a Delphi?

Dunham (1998) identified a seven-step process:

  1. Identify the issue and solicit ideas
  2. Response to first questionnaire
  3. Create and send a second questionnaire
  4. Response to second questionnaire
  5. Create and send a third questionnaire
  6. This process is continued until it becomes obvious that no new ideas are being generated and no new strengths and weaknesses are noted
  7. Resolution. (pp. 1-2)

A questionnaire is sent to a group of selected participants (Step 1). Prospective participants return the questionnaire anonymously (Step 2). A new questionnaire is prepared in Step 3 that includes all of the input from the first questionnaire, with space to allow positive and negative comments on each item, along with space to add new ideas. This questionnaire is also returned anonymously in Step 4. A third questionnaire further summarizes the prior input in Step 5. As Step 6 indicates, this process continues until it becomes obvious that no new ideas are being generated and no new strengths and weaknesses are noted. Dunham (1998, p. 2) described the final step (Step 7) as resolution. The results of the last questionnaire determine how the resolution process proceeds. If only one or a few highly valued ideas emerge, then the process ends with the generation of the ideas listed, along with their strengths and weaknesses. If this does not occur, a more formal assessment is conducted. Three methods could meet this need.

First, a questionnaire is prepared listing each idea with a rating scale of -7 to +7. Participants are asked to rate each idea, and the ideas are then ranked based on the responses. The ranked list becomes the final product. A second method is to prepare a questionnaire with instructions to select the top five ideas and identify them, with 5 for the most important, 4 being the next most important, and so forth. The ideas are then ranked, based on the average scores provided, and this ranked list becomes the final product. A third method is the modified resolution technique used by the National Cancer Institute (Cline, 1997). This method has each participant rank each idea on a 3-point scale: 1 (very important), 2 (somewhat important), and 3 (not important). All items with a mean value equal to or greater than 2.0 are removed from the list. This ranking and dropping process continues until the results stabilize. The process may need to be repeated as many as four times before a stabilized list is obtained.

Carter and Beaulieu (1992) provided another example of the process from the University of Florida Institute of Food and Agricultural Sciences.

  1. A questionnaire should be developed focusing on identified issues: problems, causes, solutions, and actions. The intent is for each respondent to list ideas regarding the specified issue.
  2. The questionnaire is distributed to an appropriate group of respondents.
  3. Each respondent independently generates ideas in answering the questions and returns the questionnaires.
  4. The questionnaire is summarized into a feedback report, and a second questionnaire is developed for the same respondent group. The second questionnaire should ask respondents to prioritize or rank input from the first round.
  5. Feedback summary and second questionnaire are distributed.
  6. Respondents review feedback reports, independently rate priority ideas in second questionnaire, and return response.
  7. This process is repeated until general agreement is reached on problems, causes, solutions, and actions.
  8. A final summary and feedback report is prepared and distributed to respondents. The feedback reports throughout this process allow the exchange of opinions and priorities and often result in individual changes in opinions and priorities after respondents evaluate the general group’s perspectives (para. 9).

The Carter and Beaulieu (1992) University of Florida study suggested some amount of researcher manipulation of the response data during the middle of the consensus-building process. This raises the question of the validity of the final results. There is a strong need for the researcher/coordinator to ensure that ethical standards are maintained. One important action is to ensure that participants retain their anonymity. Another is to ensure that ideas are not removed from the list due to the personal preference of the researcher. Any idea dropped from the list must be based on input provided by the participants.

The basic Delphi Technique consists of repeatedly asking questions of a group of individuals (Kaynak et al., 1994) and then analyzing the data received. This two-step process is repeated until a consensus is achieved. The breadth of the study and the resources available determine the number of participants.

There is a practical advantage to using the Delphi Technique (Mitchell & McGoldrick, 1994), in addition to using it when the number of participants is too great or too distributed to have a face-to-face meeting: The cost of assembling the participants in one location can be eliminated. Thus, the size of the panel is determined by time and cost allocation. Additionally, Carter and Beaulieu (1992) provided the following lists of advantages and disadvantages:

Advantages

  1. Allows participants to remain anonymous.
  2. Inexpensive.
  3. Free of social pressure, personality influence, and individual dominance.
  4. Allows sharing of information and reasoning among participants.
  5. Conducive to independent thinking and gradual formulation.
  6. A well-selected respondent panel–a mix of local officials, knowledgeable individuals, citizens of the community, regional official, academic social scientists–can provide a broad analytical perspective on local problems and concerns.
  7. Can be used to reach consensus among groups hostile to each other.

Disadvantages

  1. Judgments are those of a selected group of people and may not be representative.
  2. Tendency to eliminate extreme positions and force a middle-of-the-road consensus.
  3. More time consuming than the nominal group technique.
  4. Should not be viewed as a total solution.
  5. Requires skill in written communication.
  6. Requires adequate time and participant commitment (about 30 to 45 days to complete the entire process, largely due to the slowness of the normal mail system).

The question to address now is whether some of the disadvantages can be reduced or even eliminated by integrating the process with computers. According to Buckley (1995, p. 17), J. W. Sutherland recommended in 1975 that computers could be used to input data for a Delphi so as to lessen human influence. Winch (1995, p. 23) wrote of organizations using computer-based decision programs to assist management. One important feature mentioned is the increased ability for participants to remain anonymous. Kaynak et al. (1994, p. 20) referred to an earlier article by Kaynak and Macaulay (1984) that suggested the use of a computer in what they called a “Delphi Conference,” in which participants communicate in a chat environment in a real-time situation. This process has one major drawback. If existing software is used without changes, each transmission is accompanied with the individual’s identification. A simple solution, available with many software programs, is to log onto the chat system with a fictitious name, thus enabling the individual to maintain his or her anonymity.

Technology-Enhanced Delphi Technique

Technology-Enhanced Delphi Techniques reduce human intervention, with data collected while reducing many of the normal costs involved in collecting data. Table 1 lists the steps that may occur during a Delphi research process and how a computer using existing software could change the process.

Several steps in Table 1 related to computer integration need additional clarification. Use of e-mail in Step 3 eliminates reproduction and mailing costs. This e-mail should be formal in nature, explaining the purpose of the research, the use of a response time frame, and assurance of anonymity. The participant’s reply message serves as confirmation of participation and acceptance of the response requirements. Participants completing the questionnaire select a “submit” button to transmit the data electronically returned to be stored in a database. This technologically-enhanced process reduces data entry errors.

Table 1
Two Ways to View the Delphi Technique: Standard View and Computerized View
Steps for the Standard Method
Impact of Integrating a Computer
1. Select a topic for analysis. 1. No impact.
2. Select a panel of experts or participants and requiest proposed participants to take part in the research. 2. Select and e-mail proposed participants.

3. Develop the first questionnaire.

  • Produce sufficient copies and mail to participants.
  • Wait for responses.
  • Input responses into a database.

3. Develop the questionnaire in HTML format.

  • E-mail participants providing them with the Internet address of the questionnaire. Specify response time frame.
  • Once the time frame passes, deactivate the questionnaire.
  • Participants submit responses electronically.

4. Develop a second questionnaire containing all the responses from the first questionnaire.

  • Proceed with sub-steps as in Step #3.

4. Extract database information and configure into a HTML form.

  • Proceed with sub-steps as in Step #3.

5. Develop a third questionnaire and request participants to rate each item.

  • Proceed with sub-steps as in Step #3.

5. Extract database information and configure into a HTML form with rating fields.

  • Proceed with sub-steps as in Step #3.

6. Develop a second rating questionnaire.

  • Proceed with sub-steps as in Step #3.

6. Extract database information and configure into a HTML form with rating fields.

  • Proceed with sub-steps as in Step #3.

A second questionnaire is generated in Step 4 and made available to the participants. Responses from this questionnaire feed a third questionnaire in Step 5 that includes rating fields for each response. Participants analyze the data and rate each item. Based on predetermined criteria, the researcher identifies the items to build a fourth questionnaire for Step 6.

If the researcher determines that additional refinement of the consen- sus process is needed, Step 6 is repeated as needed. Final analysis is performed following Step 6. Summary reports are prepared and distributed electronically, bypassing mailing and reproduction costs.

There appear to be at least six advantages to the Technology- Enhanced process.

  1. Use of e-mail eliminates reproduction and mailing costs.
  2. Controlling the availability of the questionnaire by specifying an input time frame and removing access to the questionnaire allows the researcher to control the response period.
  3. The ease of responding and awareness of a time frame should result in a time factor reduction, especially if the use of a time frame is clearly explained in the initial correspondence that solicited participation.
  4. Human data entry error is eliminated.
  5. Human influence is greatly reduced during the analysis phases.
  6. Overall duration of the project may be reduced considerably.

There are also a few disadvantages:

  1. Individuals not adept at using a computer may experience some data entry problems.
  2. Individuals not adept at using electronic mail applications may experience some problems.
  3. Individuals who do not have access to a computer connected to the Internet will not be able to participate.

The methods section addresses the process used to test the Technology-Enhanced Delphi Technique.

Method

This study confirmed that technologically-enhanced Delphi Techniques can be worth pursuing. The researcher opted to use the Internet for all communication instead of the postal services. Four software tools were used: Microsoft FrontPage, Microsoft Access, Microsoft Excel, and GroupWise. FrontPage provided the means to develop Internet-based questionnaires, automatically interfaced with the Access database to electronically store responses. GroupWise, the faculty/staff e-mail application used by the University of North Texas, provided the only communication link with faculty members participating in the study. The following steps took place:

Step 1: A topic for analysis was identified.

Step 2: Twelve faculty members responded to an e-mail request for participants. Each respondent received a short paper describing the Delphi Technique and was requested to respond by email whether he or she wished to participate in the study. Nine of the 12 responded and were established in an e-mail group as “blind copy recipients” to maintain their anonymity from the other participants.

Step 3: A Web page questionnaire using FrontPage interfaced automatically with a Microsoft Access database to record the responses. e-mails were sent to the participant group thanking them for agreeing to participate and providing them with the Internet address for the first questionnaire. The questionnaire contained instructions stating the date and time when the questionnaire would be deactivated. Five days were allowed for data entry. One participant left town and returned the afternoon the input phase was to be terminated. This participant requested a few extra hours to input a response, and the time was provided. Subsequent steps included a reminder e-mail halfway through the data entry time period.

Participants were requested to provide responses to the question “What information should the student be aware of when working on a team project?” Participants submitted their responses via the Internet, and their responses were automatically recorded in the Microsoft Access database.

Although no data analysis occurred at this point in the process, the researcher discovered that most of the respondents provided considerable narrative, listing several elements of information that they considered important. Narratives were then broken down into phrases by the researcher, using the exact wording provided. All items were then used to create the second questionnaire.

Step 4: A second questionnaire listing each phrase and a new database were created for the participants. Data entry fields were provided to allow positive and negative comments to be recorded for each data element. Space was also provided allowing for elements to be added. The new database ensured that responses would not be mixed between questionnaires.

Recipients received an e-mail with the Internet address of the new questionnaire. As before, a time period for input was established. The data entry time period was expanded from 5 working days to 10 working days because some participants were scheduled to be out of town. Halfway through the time period, participants were reminded by e-mail to record their responses.

No data analysis took place at this stage. However, because some participants suggested that some data elements possessed a relationship, these elements were grouped in the database without any notation of the grouping. Thus, they appeared in the third questionnaire one after another.

Step 5: A third questionnaire contained each data element and all positive and negative comments. Questionnaire instructions requested that participants review the positive and negative comments for each data element and then to rate the data element using a 3-point scale: 1 (very important), 2 (somewhat important), and 3 (not important). The scale was displayed vertically, with the not important option automatically checked to prevent missing data. Participants submitted their responses electronically within the indicated time frame. Mean values were computed for each data element, and based on the procedure established by the National Cancer Institute, all data elements with a mean score of less than 2.0 were identified. These items provided data for the fourth questionnaire.

Step 6: The fourth questionnaire did not contain the positive and negative comments. A 3-point rating scale was displayed horizontally with the not important option automatically checked to prevent missing data. The change from vertical display to horizontal display was intended to reduce the memory of rating made on the prior questionnaire.

After analyzing the responses, the researcher determined that a consensus was achieved. Participants were advised by e-mail that the process was complete, and copies of the summary report were provided to each participant.

Results

Nine individuals agreed to participate in the study. Table 2 shows various statistics for each questionnaire.

Table 2
Questionnaire Statistics
 
Questionnaire #
1
2
3
4
Number or participants (N=9)
7
6
6
6
Percent of participants active
77%
66%
66%
66%
Number of data elements
NA
40
41
30
Number of data elements with
mean value < 2.0
NA
NA
30
29

Two participants dropped out after receiving the first questionnaire, and 1 after responding to the first questionnaire. Sixty-six percent of the participants responded to Questionnaires 2, 3, and 4. However, because of the rule of anonymity, it cannot be determined who may have dropped out. Forty data elements were identified from the responses of Questionnaire 1. No data elements, regardless of the extent of similarity, were omitted from the second questionnaire. After reading the data elements provided by other participants, the participants recorded any positive and negative comments for any data element they wished. One new data element was added, which brought the total to 41 going into the development of Questionnaire 3. Only the 30 elements receiving a mean rating of less than 2.0 were used to construct Questionnaire 4. Only one item received a mean rating of 2.0 or greater from Questionnaire 4. The researcher determined that consensus had been achieved.

Table 3
Number of Data Elements Receiving a “Very Important” Rating by Percent Of Raters
Number of Data
Elements		 100%
Percentage of Participants Rating Very Important
100%
84%
66%
50%
33%
17%
4
6
4
9
5
1

Conclusions

According to Kaynak et al. (1994, p. 20) in their discussion of the weakness of the Delphi Technique, Laczniak and Lusch (1979) suggested that Delphi Techniques may experience a high participant dropout rate. The number of participants responding in this study remained constant after the first questionnaire. Although this does not predict participation for future studies, it does present an interesting possibility.
Carter and Beaulieu (1992) suggested that Delphi Techniques require 30 to 45 days for the full process, depending on the number of stages and postal delays. This study covered a span of 71 days. This fact is not of concern because, although there was considerable area for compression, the primary goal was to document the electronic process and to identify areas in which the process could be adjusted to improve efficiency. This objective was achieved. The expansion from five workdays for data entry to ten workdays had no major impact on data entry. Rather, the reminder message appeared to be the motivator for completing data entry. This was determined by daily reviewing the database to determine the number of respondent records.

Use of subject matter experts in a field resulted in an initial list of data elements that were considered to be of value by the participants as a whole, apparently reducing the chaff. Thus, it was mainly a task of identifying elements that everyone, or the majority, could agree on as being very important. Only 12 of the 41 total data elements received a rating of 2.0 or greater, causing them to be dropped. Consensus was achieved. Of the participants, 100% agreed that four data elements were very important. Eighty-four percent agreed on an additional six elements, and two thirds agreed that an additional four should be considered very important. Thus, one half of the total 28 elements receiving a rating less than 2.0 were seen as very important by at least two thirds of the participants.

In terms of efficiency, the entire process required little detail work by the researcher due to lack of data entry and mailing tasks. Money was saved by not paying for packaging and postage to send and return the questionnaires. Because neither the researcher nor the participants had to mail printed materials back and forth, no mailing costs were experienced throughout the project. Participants could respond to the questionnaires from almost any personal computer attached to the Internet. They also knew that within moments of submitting their responses, their data would be recorded in a database exactly as they provided the data. Data entry was accomplished automatically, eliminating data entry errors. This also reduced the amount of researcher interaction with the data. Interaction consisted of exporting the Access database to an Excel spreadsheet for analysis. The researcher, by computing mean values, could identify the data elements to be carried on to the following stage.

The Technology-Enhanced Delphi process has at least two delimiting factors: Respondents must have some familiarity with computers, and they must have access to the Internet. These delimiters may prevent some subject matter experts from participating, but time will continue to decrease this circumstance.

Discussion

The Technology-Enhanced Delphi Technique for collecting information and achieving consensus is a strong tool for researchers and organizations. The step-by-step process provided will help to illustrate the simplicity and strength of allowing a computer to perform much of the data-recording task, thus reducing the chances of data transfer error caused by human error.

Customer service is a major factor in the success of most organizations, but the type of customer service depends on the customer. The Delphi can be a method for organizations to identify the core issues to best serve their clients. The emergence of computer technology in the business community provides both parties with a communication channel that eliminates additional paper handling. This may increase the likelihood that clients will respond to a study request. Each situation is different, and the information needs vary. The researcher can determine the degree of consensus needed. This will determine the number of rounds and amount of researcher time involved. Just as with printed questionnaires, the researcher may impact the number of responses by the size of each questionnaire. Future use of Technology-Enhanced Delphi may reveal additional refinements. Such refinements can only improve an already excellent research tool.

Large corporations have a ready-made source of experts who can be called upon to identify solutions. The larger the organization, the greater the likelihood that these experts will be located in several, sometimes distant, locations. The expense of assembling the experts in one location to conduct either a Focus Group Technique or Nominal Group Technique may cause management not to authorize group meetings. The Technology-Enhanced Delphi Technique solves location and expense problems. Anonymity may reduce the influence of individuals whose primary drive is to rise higher in an organization. They may be influenced not to take part in a Technology-Enhanced Delphi because of the anonymity involved.

Many researchers and authors are associated with academic institutions. Assembling representative academicians for a topic in question may be just as difficult as getting together those in business organizations. Also, if they do meet, the problem that arises is how to motivate opinionated individuals to be open to discussions of ideas other than their own. Assurance of anonymity, and thus the loss of recognition, may cause some individuals not to participate in a Technology-Enhanced Delphi. This may open the participants to new ideas and new achievements.

Regardless of the environment, the researcher must answer an important question: Is the participation of any particular individual(s) really that important? The researcher should consider the possibility that those individuals not willing to participate are possibly roadblocks to a successful project, and also that those individuals may not have analyzed their own beliefs to the extent that they can support them to the other participants. In either case, are they desirable participants?

The Technology-Enhanced Delphi Technique provides researchers, regardless of their environment, with a powerful tool. Unfortunately, many researchers appear not to be aware of its existence. Those who have used it to achieve a consensus or to solve a problem may also be keeping their method a secret.

References

Armstrong, M. A. (1999). The Delphi Technique. Retrieved September 10, 1999 from the World Wide Web: Http://www.peicommerce.com/Research/MARKETS/DELPHI.HTM

Buckley, C. (1995). Delphi: A methodology for preferences more than predictions. Library Management, 16(7), 16-19.

Carter, K. A., & Beaulieu, L. J. (1992). Conducting a community needs assessment: Primary data collection techniques. Retrieved September 10, 1999, from the World Wide Web: Http://edis.ifas.ufl.edu/scripts/htmlgen.exe?body&DOCUMENT_HE060

Cline, A. (1997). Prioritization process using Delphi Technique [White paper]. Retrieved September 10, 1999, from the World Wide Web: Http://www.carolla.com/wp-delphi.htm

Dalkey, N., & Helmer, O. (1993). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458.

Dunham, R. B. (1998). The Delphi technique. Retrieved October 9, 1999, from the World Wide Web: Http://www.instruction.bus.wisc.edu/obdemo/readings/delphi.html

Dunham, R. B. (1999). Nominal group technique: A user’s guide. Retrieved September 10, 1999, from the World Wide Web: Http://www.instruction.bus.wisc.edu/obdemo/readings/ngt.html

Hinks, J., & McNay, P. (1999). The creation of a management-by-variance tool for facilities management performance assessment. Facilities, 17(1/2), 31-53.

Kaynak, E., Bloom, J., & Leibold, M. (1994). Using the Delphi Technique to predict future tourism potential. Marketing Intelligence & Planning, 12(7), 18-29.

Laczniak, G. R., & Lusch, R. F. (1979). Future research for managers. Business, 29(1), 41-49.

Mitchell, V.-W., & McGoldrick, P. (1994). The role of geodemographics in segmenting and targeting consumer markets: A Delphi study. European Journal of Marketing, 28(5), 54-72.

Predicting the future: The Delphi tradition (1995). Retrieved September 10, 199, from the World Wide Web: Http://www.business.u-net.com/!acolyte/DelphicSpread.html

Winch, G. W. (1995). Developing consensus: Reflections on a model-supported decision process. Management Decision, 33(6), 23-31.
Charles G. Andrews, a Doctoral Candidate working on his Ph.D., and Jeff M. Allen, Program Coordinator and Associate Professor, are at the Applied Technology, Training and Development program in the Department of Technology and Cognition at the University of North Texas.