Replication in the Democracy & Local Governance Research Program

Henry Teune

University of Pennsylvania

Replication of research is an "in principle" criterion of scientific knowledge. The procedures for approximating it, of course, are soft, matters of judgment of others. That can be achieved by providing "research protocols", such that someone else can do the same research and "in principle" come up with identical findings. Differences between the "doer" and the "replicator" should be easily accounted for. If not, then rule of "triangulation", a third try, kicks in.

Strict replication of the analyses of data should ferret out error but cannot help much with bad design or bad data. The more general criteria of "reliability" and "validity" for making knowledge robust dominate concerns with replication of data made available by researchers. The classic, once much cited paper in political science, by D.T. Campbell and D. W. Fiske, "Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix" (Psychological Bulletin, 56 (1959), 81-105) provides principles that should inform the conduct of empirical research. It has done so in the Democracy and Local Governance research program.

A sketch of the dimensions and processes of the Democracy and Local Governance research program will be presented along with some steps taken to assure its credibility. It is currently funded in part by the National Science Foundation (Grant no. SBR94-23801). It received support from the U.S. Institute of Peace and a variety of other foundations, universities, and institutions for various countries and parts of the research (Contact author for more complete list.)

Democracy and Local Governance: The Problems

This research now has interviewed (about 1-2 hours, face-to-face in most of the countries) about 12,000 local political leaders (mayors, deputy mayors, and major administrators; local council people; and political party leaders, if identifiable) in about 800 localities ranging in population size from 25,000 to 250,000 (adjusted upward or downward, depending on the level and kind of urbanization in a country) between the spring of 1991 and continuing today. Data for 20 countries are in an international file; they are or soon will be for a second point in time (1995) for eight former communist countries; and are being collected or organized now in two countries. Several other countries are in the process of starting the research in 1996. Countries are the starting points; localities within a certain population range are the universe for the sample; political leaders within them are targeted for interviews.

Our objective is to examine the relationship between democratic values of local political leaders, the "globalization" of localities, and the local political orientation of leaders. The data do not contradict that relationship for local leaders, no matter what country they are from (all leaders pooled in analysis): those who are more democratic in their values see their localities as more involved with "foreign" trade, immigrants, workers, television, pollution; and they are more oriented to local problems and solutions. These relationships, however, are complicated by the contexts of countries, regions, localities, time, as well as individual differences among leaders.

We have been especially sensitive about making the data and the statements based on them credible. We moved quickly into former communist countries in 1991, where this kind of research was unfamiliar, the collaborators and their associates variously motivated under conditions of uncertainty, and where telling the truth or saying what one believed were not among the habits of political leaders, or perceived so by western social scientists.

The main initiators, however, had experience dating from the mid-1960s in the International Studies of Values in Politics. The basic sample frame was used, and shown to be exceptionally efficient for getting a national sample; most of the value scale items and many other questions had been used over the years and had proven robust (and have served well in the 1990s) and can be used for cross-time comparisons in some countries; and we were confident that we knew how to do organize collaborative, cross-national research.

The decision was made to undertake this research in June, 1990. By March of 1991 we had a sample for three countries and a questionnaire; by June, we completed the data collection and coding for Poland, Slovenia, and Sweden. That was possible because we had done research in these countries before (Poland beginning in 1966; Slovenia, as one of three Republics of Yugoslavia, in 1966; and Sweden in 1984). We were able to analyze the data and develop some confidence that we could get scales and theoretically interesting cross-national comparisons that would not violate general expectations (e.g. that Swedish leaders were overwhelmingly more democratic than either Polish or Slovenian leaders, the latter being more nationally oriented in face of a crises of independence than either the Swedes of Poles, etc.). We obtained at least "face" validity. Comparative, cross-national or cross-cultural, research generates confounding effects. The most obvious of these is comparing terms for concepts, such as authority, leadership, honesty, across different languages and cultures. Then the observers for obvious reasons are coterminous with the "systems" or countries and communities being observed (ideally they should be randomly distributed among the respondents without regard to country or locality). Then there are honest errors, often stubbornly defended by some collaborating researchers; practical, but problematic adjustments in samples, depending on countries; different, but unknown political events transpiring in the country and locality (a threat of take-over of a political party, being one example). The list goes on.

The Organization of the Research

The research is decentralized in execution and analysis. Each country has a Research Coordinator with few exceptions located in a University of academic institution; there is an International Steering Committee; a Project Director (the author); an International Coordinator (Dr. Krzysztof Ostrowski); and a Data Base Manager (Ms. Tatiana Iskra). The International File is constructed in Warsaw. National files are a bit different, generally containing data either of special interest to the country researchers or of special sensitivity to the respondents.

Major decisions about the research are generally made at meetings of the researchers of by the International Steering Committee. The last one in Krakow adopted principles of confidentiality. During these meetings the researchers have had opportunities to visit a locality in the community and talk with the political leaders interviewed, something that is done at every opportunity.

Checking the Data

A program has been developed to check the "consistency" of the data that is deposited in Warsaw. A few such errors have been discovered. Other errors come from a misunderstanding of a question, in which case it is deleted for that country from the international file.

We have decided that what is in the "International" file are raw data. All transformed, analytical variables that were initially used are being deleted ( See, B. Jacob, K. Ostrowski. and H. Teune (eds., ). Democracy and Local Governance: Ten Empirical Studies. Honolulu: HA: Matsunaga Institute for Peace, 1993). Responses to the open-ended questions are not being coded. They are in the international file as raw responses in the original languages (three at present), accessible through various programs for text analysis. (See, T. Iskra, K. Ostrowski, and H. Teune. "Designing a Data Base for Comparative Research", paper presented to INTERCOCTA, Tampere, Finland, Dec., 1995).

Governance of the International Data File

Research Coordinators who deposit data from their country can become members of a Consortium, the officers and rules of which are now being established. It is understood that the international file will be made accessible to social scientists. Some researchers have requested special provisions involving consultation with them before the data are used in publication. Each contributing member is a signatory to a protocol concerning confidentiality.

Accessibility of the International File

The data from several countries are being put onto a World Wide Web page, along with documentation, such as questionnaires, articles, references, etc. This page will be up-dated, new information added, including perhaps some visuals. A user of the page can ask questions in general and request analysis from the University of Pennsylvania. The raw data are on display for use in a PC environment. Some parts of the page will have restricted access.

Confidentiality

Countries vary on their stringency concerning confidentiality. A few have strict laws; others, loose ones. There are professional norms and different views among the researchers about how confidential their data should be.

The data are coded by country, community, and political position (implicit is the date of that position). There is one mayor in a particular Polish city in 1995. That person can be identified. Individuals, masked by name, are grouped by position in the international file. Mayors and Deputy Mayors and some administrators are coded as administrators. Localities are given approximate area identities in some countries. The latter two steps make it nearly impossible to compare mayors cross-nationally and to define regions precisely. It also makes it very difficult for anyone to identify a particular person from the questionnaire.

Religion, political party affiliation, and ethnicity are among the variables that are sensitive. In some countries asking such questions is illegal. Religion is variously coded and kept in country codes; religiosity is not (attending religious meetings, self-designation as "believer". Political party affiliations are being coded generally, one example being "left", "left-center", "center", etc. Ethnicity is left to the Research Coordinators, but will enable comparisons at levels of generality as "majority" and "minority". In western countries with a few major political parties and where the research is now being extended might require other kinds of codes to insure confidentiality.

All of these data might be available in national files and can be released at the discretion of the Research Coordinator for a particular country. There is general agreement that these data, however, should be released with caution about confidentiality.

Concluding Comments

Care has been taken to check the data that is put into the international file. In almost every case, the Data Manager and others have met to discuss a data file with the Research Coordinator. Whenever possible, the value scale items in the questionnaire have been and can be constructed into many kinds of measures. Most of these items have been used for over 30 years in cross-national research They appear to be robust.

The reliability and validity of this research will be established in the processes of analysis and the gathering of other data. It is good to hear that another research program, initiated without knowledge of the Democracy and Local Governance program, on a population sample along with a different leadership group found that during the past few years the commitment to democratic values in three of the same former communist countries, measured differently, has remained more or less the same, while that to a market economy, also measured differently, has waned. There are theories about why those changes in those countries, some of which can be examined in those two sets of data.

The Democracy and Local Governance Research program is a macro, cross-national, cross level (individuals, localities, regions, countries, and trans-national-regions), and cross-time study of democratization. Cross-time and cross-level, as well as cross-system, research is the core of the logic of comparative analysis. These data will be available not only for replication but also for melding into other kinds of data and research programs to provide a social science foundation for understandings and explanations of the significant historical dynamics of change during the last two decades of this century.