Alhassanain(p) Network for Heritage and Islamic Thought

Part III: Policy Research in Anthropology

10 Anthropology as a Policy Science

The purpose of policy science is to provide information to decision makers in support of the rational formulation, implementation, and evaluation of policy. Policies can be thought of as strategies of action and choice used to achieve desired goals. Mostly we think of policy in the context of various kinds of formal organizations like social agencies, educational institutions, business firms, and governments at all levels. There are many different kinds of policy. We use terms like public policy, social policy, food policy, employment policy, industrial policy, foreign policy, and others to designate the strategies of action and choice used by governments and other organizations in various aspects of life in complex societies. These terms reflect rather different situations in content and scope, yet all relate to the same set of basic issues. That is, all policy is concerned with values.

Policy formulation involves specifying behavior that is to result in achieving a valued condition. In a sense, a policy is a hypothesis about the relationship between behavior and values: if we want to be a certain way, we need to act this way. At a basic level, policies involve allocation decisions--decisions to spend money and time to achieve something. The "something" can be quite diverse, including increases in gross national product, decreases in unemployment, decreases in the relative cost of food staples in urban areas, decreases in the number of teenage pregnancies, or increases in fairness in the allocation of housing. These large-scale national concerns can be matched with smaller-scale, local concerns, such increases in public input in the planning of the construction of a dam, the determination of the usefulness of a particular development project, or the identification of local needs for a certain kind of educational program. Policy research can occur on both sides of a policy issue and can be adversarial. Community groups can carry out policy research as a political counterpoise to research done by the government.

POLICY PROCESS

Policy should be thought of in terms of a process. The policy process is very complex. Stating the process in the simplest possible terms, we can say that the process consists of the following stages:

I.. Awareness of need.

II. Formulation of alternative solutions.

III. Evaluation of alternative solutions.

IV. Formulation of policy.

V. Implementation of policy.

VI. Evaluation of implementations.

This process is carried out in the political arena, in which there is much competition for resources. Thus what would appear in a schematic diagram as neatly rational and orderly in reality may be determined by compromise and blunt applications of political power. The basic problem is that everything can not be done at once. Competition forces more careful allocation decisions. The complexity of the competition creates opportunities for policy science.

Policy science includes a large variety of research activities that in one way or another support the process by which needs are identified and policies are formed, implemented, and evaluated. Each stage in the policy process is associated with research needs and opportunities.

The view of policy science taken here is unusually broad; basically it is synonymous with applied research. Much (probably most) policy is formed without the aid of specific research efforts. Then again, social science tends generally to inform participants in the policy arena so that it is continually brought to bear on policy problems without actually being commissioned for a specific policy formation purpose. In these cases we can speak of policy-relevant research. There are many different points in the policy process where research done by cultural anthropologists can be used. Most research by anthropologists in this arena is done because of an existing policy, rather than to determine what the policy should be. Program evaluation, a type of research commonly done by anthropologists, is a good example of this. Some may want to separate policy research from program research.

In any case, this is not new ground for anthropologists. In fact, one could argue that policy research needs accelerated the development of anthropology as a discipline in the nineteenth century. This view is argued in chapter 2 on the history of the development of applied anthropology. In many countries, anthropology emerged as an organized discipline to fulfill policy research needs associated with colonial administration, both internal and external. At the beginning, this took the form of doing basic ethnography in unknown areas or troubleshooting concerning intercultural relationships. As early as 1895, James Mooney carried out research that had as its goal the determination of what the U.S. Department of War should do in response to the Ghost Dance as practiced among certain Plains Indians ( Wallace 1976). The appointment of the early professors of anthropology at the great English universities was based on the need to train colonial administrators. In spite of this time depth, the use of anthropology as a policy science is quite recent. It was not until the 1970s that anthropologists became involved more extensively in policy research efforts. As stated in chapter 2, this involvement relates to both push and pull factors. The push factor is the collapse of the academic job market. The pull factor is the increase in policy research efforts because of federal legislation. This last factor, of course, is most important in the United States.

As a corollary to the policy research function, anthropologists have to some extent become policy makers. This function is rare and very poorly documented. One interesting example is the work of anthropologist Robert Textorin the Peace Corps. Textor participated in the development of the so-called in-up-and-out personnel policy of the Peace Corps, which restricted the length of employment in the Peace Corps so as to maintain a higher rate of innovation and what might be called "organizational youth" ( Textor 1966). My own experiences in development administration involved small-scale policy formulation in response to a community development effort on an American Indian reservation.

One can not overlook the cases where anthropologists have assumed highlevel administrative positions in federal and state government. Some noteworthy examples are: Philleo Nash, who served both as commissioner of Indian Affairs in the U.S. Department of the Interior, and lieutenant governor of the State of Wisconsin ( Landman and Halpern 1989); Aguirre Beltran, who served as director of the National Indian Institute of Mexico; Jomo Kenyatta, who was the first prime minister of Kenya; and Nirmal Kumar Bose, who was appointed Commissioner for Scheduled Castes and Scheduled Tribes in India ( Sinha 1986). In all these cases these people were intimately involved in policy formation. There are of course a number of knowledgeable applied anthropologists who have argued very eloquently against such involvement. A good example is Homer G. Barnett, who did extensive applied work in the Pacific following World War II. He argued that our effectiveness as applied anthropologists would be reduced if we took over administrative functions ( 1956).

In any case, most involvement of anthropologists in the policy arena is as researchers. In this framework they are said to be most effective at the local level ( Chambers 1977); or, when they work at the level of national policy formation, they function best in large multidisciplinary research teams ( Trend 1976). Both Chambers and Trend seem to be arguing from the same ground, which is that the traditional, holistic, participant-observation-based research methodology works best in smaller-scale contexts. While this is probably true, there are ways of escaping the effects of the constraint. One is to learn other research techniques.

Policy research is not a monolith. There are many different types. For example, each stage in the policy process is associated with different research needs. There are many different types of current policy research practice that see anthropological involvement. Anthropologists conduct evaluation research, needs assessment, social impact assessment, social soundness analysis, and cultural resource assessment, as well as various other kinds of policy research. In addition to the research carried out in support of the development, implementation, and evaluation of specific policies, there is also research that is referenced to general areas of social concern. This can be referred to as policy-relevant research.

In regard to this distinction, it is possible to speak of anthropology in policy, and anthropology of policy. This follows a contrast originally made by the medical sociologist Robert Straus ( 1957), who spoke of researchers serving in support of medical care, as opposed to researchers who study medical care. The first was referred to as sociology in medicine, the second as the sociology of medicine. DeWalt applied this distinction to his analysis of agricultural anthropology ( 1985). Both are very important. It is, however, important to recognize the distinctions between the two kinds of work. All the policy anthropology that we refer to here is of the "anthropology in policy" type.

All the different types of policy anthropology represent important kinds of research activity for anthropologists in many different employment situations. Further, if one considers all the different purposes and funding mechanisms for research by anthropologists, one finds that the contrast between applied and basic research becomes reduced. We have, on the one hand, research that is specified, bought, and paid for by clients to meet some practical need, and, on the other hand, research planned and carried out by researchers referenced only to their curiosity and sense of the direction of the discipline.

What exists between these polar types is the product of a mix of personal inclination and many different incentives. For example, many programs that fund basic research will fund that research in terms of a set of priorities that are derived from general policy questions. These specific economic incentives come to be converted to "hot topics" and short-run tendencies in research topic selection. Under certain circumstances, research produced for specific applied purposes can begin to appear in print as if it were basic research. This then influences research topic selection in yet another way. The point is that the contrast between applied and basic research is rather weak. Further, there is a great deal of flow between the two realms.

CURRENT TYPES OF POLICY RESEARCH PRACTICE

The types of policy research discussed here range from standardized research methods geared to specific policy issues, to large and generalized research orientations applicable in a wide variety of situations. The contribution of anthropologists to the development of the methods and techniques used varies from a great deal to very little. Except for a few cases, the anthropologist involved in the use of these practices needs to know general social science research methods in addition to those more traditionally associated with anthropology. As is mentioned in Chapter 12, on evaluation, one needs an integrated research methodology in which the researcher is capable of drawing on a variety of different techniques, depending on the problem at hand. A glimpse at the various types of policy research is provided below. As suggested above, some of these types have specific technical meanings, while other categories are general and include a wide variety of research functions.

Evaluation.

In evaluation, research is done with the goal of determining the worth of something, such as a project, program, or set of training materials. The process can involve a wide variety of research designs, from highly structured experiments with control groups to descriptive ethnographies. Evaluation can serve many purposes. Many evaluations are done to determine the effects of a specific project or program. Evaluation can also be done to see if some activity is working as expected, with the goal of improving it. Evaluators can use a wide variety of data collection techniques. Evaluation can be used to test the feasibility of wider application of innovations. Research can be used to evaluate alternatives in the design process. Evaluation is one of the most important types of policy research done by the applied anthropologist.

There is currently increased interest in the use of ethnography in evaluation. Using this approach, the task becomes one of finding out what is going on in a specific situation, rather than technical determination of effects. The chapter on evaluation includes case studies of evaluations that involved anthropologists. Anthropologists working in evaluation often use case study methodologies. In some cases they serve as ethnographers studying large-scale projects as part of multidisciplinary teams.

Social Impact Assessment.

In social impact assessment, research is geared toward predicting the social effects of various kinds of projects. Usually the process involves the examination of unplanned effects of major construction projects on families and communities, before the project is built. In this limited sense, social impact assessment is a kind of effect study. Social impact assessment is especially important in the design process. Usually the process involves the consideration of the effects of various design alternatives. Social impact assessment often involves the use of secondary data.

This is an important kind of policy research for cultural anthropologists. An entire chapter is given to social impact assessment in this text. It is worth noting here that often the research methodologies used in social impact assessment are mandated by the contracting agency. In the United States, various kinds of impact assessment research is done in compliance with a number of different federal laws, including those concerned with protecting the environment.

Chapter 11 contains an expanded discussion of social impact assessment, with a case study of a specific assessment project. You will find that social impact assessment, in part, resembles the traditional anthropological/sociological community study, except that it places emphasis on the use of secondary data. The use of secondary data is encouraged because of the need for speed and standardization between different project assessments. While social impact assessment can be done in many different settings, it is specifically geared for use in conjunction with the planning of projects in the United States. It is used in a wide variety of settings involving the projection of the effects of everything from dam construction to fisheries management policies. This kind of research is done to evaluate design alternatives prior to implementation.

Needs Assessment.

In needs assessment, research is done to determine deficiencies that can be treated through policies, projects, and programs. It is done as part of the planning process and is sometimes thought of as a kind of evaluation. Sometimes needs assessment takes the form of large-scale survey research projects that identify and rank preferences for certain developments. Such surveys usually require two waves of standardized data collection, one to identify and one to rank. Needs assessments can also be based on existing census data used as social indicators. Many factors that are targeted by policy can be measured this way, such as education and income levels, number of violent deaths, and disease rates. Working in smaller-scale contexts, the needs assessment process may involve the use of community meetings of various kinds.

Obviously, needs assessment occurs early in the policy process and can set the scene for a variety of policy research procedures. The operation of many intervention strategies may involve needs assessments of various kinds. The identified needs often are used in program monitoring and evaluation at subsequent stages.

Social Soundness Analysis.

Social soundness analysis is used to determine the cultural feasibility of development projects. This generalized approach to project assessment came to be used by researchers working for the U.S. Agency for International Development, starting in the mid-1970s. The approach, in large part, was developed by the anthropologist Glynn Cochrane. Cochrane had done assessment work for various development agencies, including the World Bank and the British Ministry of Overseas Development. The term social soundness analysis comes from U.S. Agency for International Development documents.

The process is described in Cochrane bookThe Cultural Appraisal of Development Projects ( 1979). An important element in social soundness analysis is the identification of the different beneficiary groups associated with the effects of a specific project. This is important because of the policy framework of American international development efforts, which, since the amendments made to the Foreign Assistance Act in 1973, have had a mandate to direct their attention to the needs of the "poorest of the poor." This required a commitment to what Cochrane called social mapping. Social mapping is basically a process of ethnography that involves the collection of data on ethnicity, social organization, belief systems, wealth forms, patterns of mobility, and access to basic human needs.

The project design process as outlined by Cochrane directs the attention of the appraiser to a number of criteria that should be considered during project design.

Criteria Used in Cultural Appraisal of Projects

Contextualism --assuring that the project ideas fit with the cultural landscape.

Incrementalism-- assessing the magnitude of the social change involved.

Minimum participant profiles-- analyzing the social characteristics of project participants.

Spread effects-- estimating the magnitude of project impact.

Motivation-- providing reasons for participation in projects.

Estimating time factors—approximating the length of time required for social change.

Benefit incidence-- observing who gains and who loses during the life of a project.

Communication and learning-- seeking ways of facilitating and encouraging innovation and adaptation.

Design of extension efforts—building the organization of extension work.

Using indigenous organization-- maximizing the use of local management talent. (Cochrane 1979)

The cultural appraisal process outlined by Cochrane is nontechnical, in that it does not present much beyond a checklist with illustrative cases as a means of specifying the research process. In any case, the approach is currently used in the U.S. Agency for International Development project planning process, in conjunction with other research approaches.

Technology Development Research. In an effort to help assure the appropriateness of technology developed for use in less developed countries, a number of agencies have become committed to the use of social science to inform the technology development process. This is well developed in farming systems research ( DeWalt 1989; McCorkle 1989; Norman and Summons 1982; Hildebrand 1976; Ruthenberg et al. 1980). Farming systems research is geared toward linking farmers with those who develop agronomic technology. Part of this linkage is the provision of comprehensive accounts of the farming system. The concept of the farming system is focused on analysis of the production and consumption decisions of farm households. In this research, attention is paid to the identification of development constraints and opportunities. One way that technology development research can operate in the agronomic context is through on-farm research. This involves the actual implementation of agronomic research on the farms rather than in the experiment stations. In this setting the social scientist can serve as a broker for the experimentation program.

Most farming systems research of the type briefly described here is done in conjunction with the international crop research centers, such as CIMMYT (Centro Internacional de Mejoramiento de Maiz y Trigo ), or CIP (Centro Internacional de la Papa ), or the commodity-focused Collaborative Research Support Programs (CRSP), such as INTSORMIL or Small Ruminants. Fanning systems research, although developed outside of anthropology, is congruent with certain anthropological tendencies in methodology.

Cultural Resource Management.

Since the early 1970s a great deal of archaeological research in the United States has been carried out in response to legislative mandates. This has led to the emergence of cultural resources management (CRM). CRM is concerned with identifying the impact of federal and other kinds of development on archaeological sites, historic buildings, and similar things, and then managing the impact in various ways. Management usually involves identification and documentation, but may include mitigation and protection. Mitigation may include thorough research and documentation of the resource. Protection may include physical stabilization and the establishment of zones of protection.

Large numbers of archaeologists, architectural historians, and other researchers have been active in CRM. Recently this assessment process has begun to be directed toward contemporary communities as cultural resources. The emphasis in this research is toward the documentation of the folk knowledge of communities that are displaced by development projects. This kind of research is not common. The research methodology is based on traditional ethnographic practice. An example of this kind of work is the Big South Fork Project, carried out by Benita Howell and the National Park Services Applied Ethnography Program.

Of course, there are other types of policy research besides those mentioned here. Nevertheless, these are important because of the numbers of anthropologists involved in them. Clearly, the most important are social impact assessment and evaluation research. Anthropologists have also been involved in the development of these research methodologies. This is especially apparent in the area of social impact assessment methodology developed for the Army Corps of Engineers. Also, anthropologists have served as evaluators of the products of social impact assessment and evaluation research. The point is that there are many different ways of participating in policy research endeavors.

INCREASING THE USE OF POLICY RESEARCH

Sometimes research just happens, but usually applied researchers have to work hard at it. The crucial question facing the applied or policy researcher after all this hard work is, "How can I get my research used?" In dealing with the question of utilization it is important to be neither naive nor cynical. It is necessary to recognize that our research, however sound, may not affect the situation. Also in many situations decision makers may be poised to act on the basis of the knowledge provided them through policy research.

The literature on the different types of policy research in all cases contain references to the problem of underuse of research results. It is clear that this is a consistent problem in the policy research realm. It is a problem that stimulates its own research. This section of the chapter, written with Barbara Rylko-Bauer and based on an earlier article ( Rylko-Bauer and van Willigen, 1993), is intended to give practical advice on how knowledge utilization can be increased. You could think of it as a theory of research effectiveness. This advice is organized around a series of principles that when followed will help increase the impact of anthropological research. The advice is intended to be general enough to cut across the various research types.

Before discussing the framework some basics need to be established. First, because we can control our own actions we need to think primarily about what we do rather than what others do. I say this because often researchers blame the agency for not making use of the research. While this may be true to some extent, it is more productive to focus on what we can do to improve the potential for getting research used. Second, we need to treat knowledge use as something that needs to be planned into the design of projects. Research designs guide research projects. Applied research should include a knowledge utilization design or plan. The discussion below suggests elements that can be included in such a plan.

Third, we need to think realistically about our goals and look at utilization broadly. Researchers in this area point out that a narrow conception of utilization overlooks the complexity of policy making, and fails to recognize that reducing uncertainty, clarifying issues, and providing new understanding of how programs work are also real effects ( Beyer and Trice 1982; Caplan 1977; Patton 1986; Weiss 1977, 1981). More significantly, research "can gradually cause major shifts in awareness and reorientation of basic perspectives" without seeming to be directly and immediately applied ( Weiss 1981:23).

The following discussion includes factors to be considered in developing a utilization design. The context of a research situation will determine which knowledge utilization factors have more relevance.

Collaboration

The most significant factor in getting research findings used is collaboration between researcher and clients ( Alkin 1985; Burry 1984; Glaser, Abelson, and Garrison 1983; Leviton and Hughes 1981; Patton 1986; Rothman 1980). Collaboration means involving decision makers and other potential stakeholders, such as community members, in the research process. Carefully working with people to identify their information needs and ways they can use the research will increase their commitment to the application of the research. It is important to foster a relationship with an individual that personally cares about the project and the information it generates. Patton refers to this as the "personal factor" ( 1986).

User participation presents some potential ethical dilemmas. A frequently noted concern is cooptation of the researcher, which may occur if decision makers shape the research to provide results that support preferred or already existing policies and actions, and do not challenge their role within the organization ( Ballard and James 1983; Beyer and Trice 1982; Dawson and D'Amico 1985). Selecting stakeholders involves a judgment about whose questions will guide the research ( Mark and Shotland 1985), creating potential for a different sort of cooptation--the preempting of criticism of the project by the inclusion of stakeholders who might have been likely to do so. Finally, if the researcher does not provide stakeholders with the necessary information for effective and knowledgeable collaboration, then user participation can become a form of "pseudoempowerment" ( Mark and Shotland 1985:143-44).

Models of collaborative research are well developed in anthropology (e.g., Stull and Schensul 1987), and evolved from a value orientation that recognizes the validity of self-determination as a major force in sociocultural change. Recently the idea of user participation has been explicitly suggested as a strategy for increasing the use of anthropological knowledge ( Davidson 1987; Schensul 1987; Stern 1985; Whiteford 1987).

Communication

Communication of research findings is often limited to the writing of a final report; yet this is not a very effective way of passing on information, and often results in too much, too late. Perhaps the most important strategy is to discuss preliminary findings throughout the research process and maintain an ongoing dialogue with feedback between researcher and information users ( Glaser, Abelson , and Garrison 1983; Rich 1975). This is much easier to do if decision makers are collaborating in the research process ( Dawson and D'Amico 1985; Patton 1986).

Other communication strategies include using multifaceted and appropriate means of communication, such as workshops, conferences, trade magazines, journals from other disciplines, and widespread distribution of short draft reports ( Ballard and James 1983; Beyer and Trice 1982; O'Reilly and Dalmat 1987; Patton 1986; Schensul 1987). Presenting findings in the language and style of users is supported by our common sense, yet all social scientists have great difficulty avoiding jargon, keeping reports brief, and presenting findings and recommendations in a manner familiar to potential users ( Ballard and James 1983; Rothman 1980). It is important to communicate findings directly to relevant decision makers. Practitioners need to provide concrete, specific recommendations about what is to be done, by whom, and when ( Patton 1986; Rothman 1980). Policy makers do not usually expect primary data and research reports; they want recommendations on what to do ( Cernea 1991).

Client

Collaborative research is more likely to succeed if one understands the client agency, community, or group, and the political context within which the research and knowledge would be used. Do an ethnography of the research situation. Becoming informed about the ways in which communities and groups may be affected by the research, and about the client group and its decision-making process, gives the researcher some understanding of the relationships among relevant groups, who the key decision makers and community leaders are, and the potential areas of conflict and possible forums for resolving them.

In studying the nature of the client group, one can focus on questions such as who are the relevant decision makers and potential users of the information, how are decisions made within the organization, what are the usual channels of communication, and what are the constraints and/or incentives to use of the information within the agency.

Community and Politics

Always be aware of the potential impact of research findings, and try to understand the relationship that exists between the client agency and those individuals, groups, or communities that may be affected. Often, the client may be in a position of relative power vis-à-vis the community, and the agency's values and bureaucratic needs may conflict with those of community members. Recommendations perceived as threatening by those outside the agency may enable a community to mobilize public support to defeat such action. Conversely, the agency may decide not to act on recommendations perceived as going against its best interest, even if they are beneficial to the community that the agency serves. Research based in an established community institution with political clout has greater likelihood of having an impact and bringing about desired social change ( Schensul 1987).

Research Process

Research should be designed, from the onset, with utilization in mind ( Patton 1986). There are three features of research that increase the potential for use.

First, diversity of research methods, in particular the creative combination of quantitative and qualitative methods and analysis, can provide an insightful, valid, and convincing representation of social reality. At the same time, diversity can help meet time constraints, as well as criteria of reliability and generalizability that policy makers often expect ( Beyer and Trice 1982; Fetterman 1989; Schensul 1987; Trotter 1987).

Second, use of research is directly related to the credibility of the research process ( Caplan 1977; O'Reilly and Dalmat 1987; Weiss and Bucuvalas 1980; Whiteford 1987). This includes perceived accuracy, fairness, understandability, and validity of research design and methods ( Patton 1986). Research quality issues become more important in situations of political debate, where the policy maker cannot afford to have the research discounted due to uncertain methodology ( Weiss and Bucuvalas 1980).

Third, the potential for use also increases if the research focuses on variables that can be acted upon, that are accessible to control ( Gouldner 1957). We call this applicability. Several studies suggest that decision makers are more likely to use findings if recommendations are feasible, and the results conform to users' expectations or existing knowledge ( Caplan 1977; Leviton and Hughes 1981; Weiss and Bucuvalas 1980).

Time

Policy research often has a short time frame. Recognition of this has led to many new methods for anthropologists doing policy research ( van Willigen and DeWalt 1985). Perhaps most notable is the development of problem-focused, short-term research techniques such as focus groups and rapid appraisal ( van Willigen and Finan 1991). One example is the informal or reconnaissance survey done in fanning systems research. In these efforts there is a heavy reliance on key informant interviewing, judgmental sampling, use of secondary data, and on-site observation. Another example is "rapid assessment procedures," such as those developed for evaluating and improving primary health services ( Scrimshaw and Hurtado 1987).

Advocacy

Promoting one's research findings and recommendations also can improve the prospects for use ( Barber 1987; Jones 1976; Rothman 1980; Siegel and Tuckel 1985). Advocacy works best from inside the system. One way of personally ensuring that research is used is to become one of the decision makers. It is much harder to influence the policy process from the outside, and increasingly anthropologists encourage direct involvement in program management and policy making ( Cernea 1991). In whatever role you choose, you have to be committed to change.

Clearly there are many variables that influence whether or not research is used. Above all, the policy researcher must include in the design of his or her research a knowledge utilization plan to increase the probability that the research will be used.

SUMMARY

Anthropologists provide a wide variety of research services in response to various needs associated with the process of policy formation, implementation, and evaluation. More detailed charting of the policy process would no doubt produce even more types of applied or policy research. While anthropologists bring certain methodological and conceptual tendencies to these research efforts, the content of these approaches is defined in reference to the policy process itself, as well as other disciplines.

The major types of applied research done in reference to the policy process are: evaluation, social impact assessment, needs assessment, social soundness analysis, technology development research, and cultural resources assessment.

Certainly other specific types will emerge in the future. The differences between these research methods are not based so much on technique and design but on p Further, in some settings the research technique used is really geared toward being appropriate to specific administrative requirements. It is clear that involvement in policy research calls for broad preparation in social science as well as knowledge of the traditions of ethnographic research.

FURTHER READING

Belshaw Cyril S. 1976.The Sorcerer's Apprentice: An Anthropology of Public Policy . New York: Pergamon Press. A view from the perspective of the British Commonwealth.

Chambers Erve. 1989. "The Policy Idea," inApplied Anthropology: A Practical Guide . Prospect Heights, Ill.: Waveland Press. This chapter discusses the nature of policy and the policy process.

Sanday Peggy Reeves, ed. 1976.Anthropology and the Public Interest: Fieldwork and Theory . New York: Academic Press. Useful to get a perspective on policy-relevant anthropology.

van John Barbara Rylko-Bauer Willigen, and Ann McElroy, eds. 1989.Making Our Research Useful: Case Studies in the Utilization of Anthropological Knowledge . Boulder, Colo.: Westview Press. Presents a variety of case examples of anthropologists having an impact through their research.

11 Social Impact Assessment

Social impact assessment (SIA) is a kind of policy research frequently done by cultural anthropologists. While the term is applied to a variety of policy research activities, it usually entails the collection of sociocultural data about a community for use by planners of development projects. The data is usually intended to help project planners decide whether a particular project should be built or how it should be modified. Considered in its most narrow context, SIA is done as part of the environmental impact assessment process carried out under the mandate of the National Environmental Policy Act (NEPA) and other federal statutes.

DEFINITION OF SIA

A useful general definition of SIA is provided by the Social Impact Assessment Committee of the Society for Applied Anthropology. The committee, chaired by Michael Orbach, defined the process in the following way: "Social Impact Assessment (SIA) is the study of the potential effects of natural physical phenomena, activities of government and business, or any succession of events on specific groups of people" ( Orbach 1979). This definition covers a range from the highly structured research mandated by federal statute and regulation, to generalized futures research. In this book our concern is more with the various types of structured research carried out to comply with law than generalized futures research. SIA can be done in reference to a variety of potential events. Some of these are natural disasters; establishment of new federal, state, or local regulations; closure of governmental agencies, such as military bases; changes in industrial production practices; and the construction of large-scale projects like power lines, dams, and highways. SIA can serve many different purposes. The most appropriate to our interests here is the use of SIA in conjunction with the planning of major governmental projects. In this setting, the social impact assessor is called upon to provide projections about future effects to inform all the parties involved in the project, including planners, designers, political leaders, and the public. Often this involves assessing a number of different project options. For example, an interstate highway could be routed in a variety of ways, have interchanges in various locations, and vary in other features of design. Usually one design alternative considered is "no project." This type of SIA will often require a number of determinations of impact as the project is redefined. In addition to the data collection effort, the social impact assessor may also be involved in the process of informing the public about the project and the findings of the research.

Work in SIA can take other forms. Some of these are research and development leading to new SIA methods. These may be expressed in the form of agency regulations and guidelines as well as in general field research manuals. Another type of research that is similar to agency-sponsored SIA work is research done on behalf of a community that is engaged in resistance to a planned project. This type of job, of course, is less frequent, and may be done on a voluntary basis. In the United States, resisted projects end up in court, and this can involve the presentation of a number of different views of a specific project.

NATIONAL ENVIRONMENTAL POLICY ACT

The most practical component of NEPA for anthroplogists and other social scientists is paragraph (c) of section 102. This paragraph enjoins all agencies of the federal government to include in every recommendation or report on proposals for legislation and other major federal actions significantly affecting the quality of the human environment, a detailed statement by the responsible official on (I) the environmental impact of the proposed action, (II) any adverse environmental effects which cannot be avoided should the proposal be implemented, (III), alternatives to the proposed action, (IV) the relationship beween local short-term uses of man's environment and maintenance and enhancement of longterm productivity, and (V) any reversible and irretrievable commitments of resources which would be involved in the proposed action should it be implemented. ( U.S. Congress 1971:853).

This portion of NEPA has led to the emergence of environmental impact assessment (EIA) practice and a new form in the literature of science, the environmental impact statement, or EIS ( Burchell and Listokin 1975:1). SIA is often an aspect of environmental impact assessment.

As is typical with federal laws of this significance, NEPA stimulated a tremendous quantity of interpretive literature. This included general federal administrative guidelines, case law, specific agency guidelines, and various commentaries from the academic community, as well as the EIS's themselves. In addition, the law served as a progenitor for substantial parallel legislation at the state and county level. The "Progeny of NEPA" present interpretive problems of their own ( Burchell and Listokin 1974). Following the development of this practice in the United States, EIA has become important in domestic policy of other countries ( Carley and Bustelo 1984:2), and in international development work ( Derman and Whiteford 1985).Although it is clear that section 102(2) (c) of NEPA requires an EIS on "major federal actions significantly affecting the quality of the human environment" ( U.S. Congress 1971:853), it is not always possible to determine where an EIS is required. In addition, it is not apparent from NEPA what constitutes an appropriate method for determining impact, nor does the law indicate the relative importance of social impact as opposed to environmental impact. Further, there is a great deal of variation in methodologies used in social impact analysis. The extent to which the social dimension is considered is often limited, although social impact assessment would seem to be part of the requirement of an adequate environmental impact assessment.

Environmental Impact Statement

According to federal regulation, agencies of the federal government are subject to the EIS provisions of section 102 (2) (c) of NEPA. Each agency is to view the act as supplementary to its own authorization. A wide range of actions are included in the purview of the act. These include all "new or continuing projects and program activities: directly undertaken by federal agencies; or supported in whole or in part through federal contracts, grants, subsidies, loans, or other forms of funding assistance . or involving a federal lease, permit, license certificate or other entitlement for use" ( Council on Environmental Quality 1973:20551).In order to be reviewed, projects of the types indicated above must be "major" and be capable of "significantly affecting the quality of the human environment." The responsibility for making this determination is in the hands of the specific agency that must consider the "overall, cumulative impact." A number of related small-scale projects may be subjected to this analysis if the projects taken as a whole have significant impact. The significance of a project may relate as much to location as to project design. That is, "the significance of a proposed action may also vary with the setting, with the result that an action that would have little impact in an urban area may be significant in a rural setting or vice versa" ( Council on Environmental Quality 1973:20551). The agencies were instructed to determine guidelines that would in some way define significance.Each EIS is to cover the following points:

I. A. Description of the proposed action. B. Description of the environment affected.

II. Relationship of the proposed action to land use plans, policies, and controls for the affected area.

III. Probable impact of the proposed action on the environment. A. Positive and negative effects.

B. Secondary or indirect, and primary or direct effects.

IV. Alternatives to the proposed action.

V. Any probable adverse environmental effects that cannot be avoided.

VI. Relationship between local short-term uses of the environment and the maintenance and enhancement of long-term productivity.

VII. Any irreversible and irretrievable commitments of resources that would be involved in the proposed action should it be implemented.

NEPA and its related regulations require that individual agencies develop their own guidelines for carrying out their charge. This means that EIS's done for individual federal agencies should follow specific defined sets of practice, many of which are derived from the more general NEPA expectations. In addition, certain agencies, such as the Army Corps of Engineers, are subject to the requirements of their own versions of NEPA.

In the case of the corps, there is section 122 of the River and Harbor and Flood Control Act of 1970 (Public Law 91-611), which provides for economic, environmental, and social assessments. This law states that "the Chief of Engineers . shall . promulgate guidelines designed to insure that possible adverse economic, social and environmental effects relating to any proposed project have been fully considered in developing such projects" ( U.S. Congress 1972:1823). Section 122 of the law goes on to specify a number of adverse effects that must be assessed. These include: (1) air, noise, and water pollution; (2) destruction or disruption of man-made and natural resources, esthetic values, community cohesion, and the availability of public facilities and services; (3) adverse employment effects, and tax and property value losses; (4) injurious displacement of people, business, and farms; and (5) disruption of desirable community and regional growth. Similarly, the Housing and Community Development Act of 1974 also provided for its own type of impact analysis. The River and Harbor and Flood Control Act is especially noteworthy, however, because of its explicit treatment of social variables.

It must be made clear that environmental impact assessment and social impact assessment are not the same thing. The original CEQ guidelines that respond to NEPA do not single out and stress the social component of the total impact of a project. The orientation, although at times vague, is more holistic. Social impacts are seen as part of the total "package" of effects to be reviewed. In this context we might think of SIA as a term that is applicable to a portion of the entire EIA process.

Even a cursory reading of the CEQ guidelines indicates the importance of the social component in total impact determination. This orientation is made even more explicit in many of the developed agency guidelines. For example, the Army Corps of Engineers has funded a number of projects that have led to the publication of various guidance documents for preparing and contracting for what are explicitly labeled as social impact assessments. Thus in certain agencies SIA is an official concept.

Although the EIA procedure originally seemed to be oriented toward biophysical data, it is quite apparent that the role of the social scientist in the EIA process has developed in a significant way.

CASE STUDY: EL LLANO PROJECT

El Llano project is described inSocial Impact Assessment: Experiences in Evaluation Research, Applied Anthropology and Human Ethics , by Sue-Ellen Jacobs ( 1977). The value of this particular volume is enhanced by Jacobs's thoughtful inclusion of a "Field Guide for Conducting Social Impact Assessment," although she stresses that it "presents one person's purview" ( Jacobs 1977:i). In this rapidly changing area of inquiry, various alternative approaches should be reviewed and considered.

The proposed El Llano project would entail the construction of a diversion dam, canal, and lateral system to be built two miles upstream from Velarde, New Mexico, on the Rio Grande. The $13 million project was designed primarily for irrigation rather than flood control or recreation. The official name of the project was El Llano Unit of the San Juan-Chama Project. Its principal social impact zone encompassed an area nineteen miles long and three miles wide, within which there were twenty-two communities. Jacobs's impact assessment activities focused upon the communities in one of the two counties affected.

Community Characteristics

Using 1970 census data, Jacobs determined that there were 11,888 people living in the impact zone studied. The majority of the population was SpanishAmerican, with about 9,930 or 83.5 percent reported. The other large segment of the population was from two Indian pueblos: 1,255 persons from San Juan, and 916 persons from Santa Clara.

The population for the impact zone was increasing, in that the county within which it was located increased its population 4 percent per year between 1960 and 1970. The anticipated growth would be urban in nature. Jacobs notes some of the planners' projections:

Land use is expected to shift from "vacant-agriculture" acreage (which makes up 96 percent of existing acreage) to developed acreage which would provide increased housing, industrial development, school sites and other neighborhood developments in order to accommodate this increased trend to urbanism. Simultaneously, it has been recommended that because of the Llano Unit potential to irrigate new lands, concerted efforts should be made to increase high yield agricultural endeavors. ( Jacobs 1977:4)

The economic activities of the region reflected a mixture of subsistence agriculture and wage labor supplemented by agriculture. The unemployment rate in the county was very high, often over 19 percent ( Jacobs 1977:4). This rate, which is of course extraordinarily high, may not have adequately accounted for gainful traditional agricultural pursuits. The community manifested a pattern of increasing rates of employment and unemployment, which is not an unusual pattern in communities of this type. Data from the 1970 census indicated that over 14 percent of the impacted families received public assistance. The percentage of the population receiving public assistance was substantially lower than the percentage eligible. Jacobs also suggests that there was a great deal of mutual aid, trading, and bartering in rural areas of the impact zone, which helped assure that a person's basic needs were met. Community members traded craftwork, labor, and surplus produce. Some persons sold garden and orchard produce at a local farmers' market in order to obtain money to reinvest in farm improvements. Among the important crops were apples, peaches, apricots, and plums.

The farming system was difficult to characterize because it seemed to manifest features of both traditional and modern systems. That is, the farms produced both for the market and for household consumption. Most farms were small because of the land inheritance patterns of the predominantly Hispanic and Indian population. It was unclear whether these patterns were traditional or a more recent development.

One of the most interesting features of the region was the system of communitycontrolled irrigation ditches and the traditional organization that maintained them. The system, which was established under provisions of the Treaty of Guadalupe Hidalgo, was maintained by an elected supervisor, ormayordomo . Themayordomo role was best performed by a fair-minded person who could allocate water equitably in periods of shortage. He needed to know the requirements of the land and crops, the water laws, and practical ditch management skills. The role, defined largely by tradition, possessed substantial power. Through themayordomo 's power, he could prevent and resolve conflict in the realm of water and land use, as well as aspects of community life beyond the ditches. In addition to the water allocation responsibilities, themayordomo had also to organize annual ditch-cleaning activities.

Parts of the irrigation system extant in the impact zone antedated the system established by the treaties with the Spanish. To the south of the region were the Indian pueblos of San Juan and Santa Clara. This irrigation system was maintained in a somewhat different way.

Because the water development project was threatening the land, in her report Jacobs discusses the land in terms of its role in the religious belief system of the various communities. She stresses the sacred nature of the land both in the Hispanic and Indian portions of the zone of impact. Her discussion of the religious life of the region accentuates the parallels between the Hispanic and the Indian communities. For example, certain parallels also exist in the social organization of the Hispanic and Indian populations. Jacobs emphasizes the importance of family relations during various stages of life among virtually all the people in the area. The Hispanic and Indian groups have been linked through marriage from the late seventeenth century. Jacobs suggests that this pattern has continued until the present day.

One of the most interesting components of the area's traditional social organization was the system of ritual godparenthood, called compadrazgo. This institution provided a means for extending parenthood beyond the consanguineal parents for the purpose of "sponsorship" and support in conjunction with life crises and events such as baptism and confirmation. The roles of comadres (godmothers) and compadres (godfathers) were quite important in both the Indian and Spanish communities.

METHODS AND TECHNIQUES

The following discussion of SIA methods and techniques is based on Jacobs's "Field Guide for Conducting Social Impact Assessment" ( 1977). It is very important to note that SIA methodology is highly variable. The social component review of any project will vary in importance because of the lack of uniform criteria for evaluating SIAs. That is, in Jacobs's work we have examples of assessment directly focused on the social component of impact. In many other projects, the social component is an afterthought, given limited focus. The factors that may influence the nature of the social component are indicated in Table 11.1. It is very important to emphasize that the research standards for SIA are improving through time.

Let us have Jacobs define the framework for her discussion of methodology.

In social impact assessment, there are some specific goals to be reached. We want to know how and upon whom a project will have impact in a region or community. It is certain that any project will have some social effects, but we cannot know with absolute certainty just what the full impact will be, nor can we know whether the impact will be positive or negative because it is not possible accurately to predict the full range of human responses that are likely to occur as a result of intrusions into existing sociocultural systems. However, we can make some good speculations about what might happen. So, we also want to know what impacts are likely to occur, how to judge potentially adverse impacts, how to mitigate against adverse effects, and how people in a community are likely to change as a result of a project. ( Jacobs 1977:1)

Unfortunately much of the methodology that we learn as anthropologists that focuses on culture change is retrospective in nature. Yet in SIA we must predict the future. Even standard trend analysis is difficult to apply because the treatment that we are trying to assess has not yet happened. The most basic studies of culture change attempt to identify baseline data with which to compare postchange data so as to determine the extent of change.

The primary task of the preliminary phases of SIA is the identification of the relevant populations and the assimilation of as much information as possible

Table 11.1 Factors Influencing the Nature of the Social Component of SIA

Project Factors

Project types (dams, channelization, canals, reservoirs, irrigation systems, etc.)

Project scale

Project use (flood control, domestic water, irrigation, water quality improvement,

recreation, etc.)

Community Factors

Population density

Culture and cultural diversity

Residential stability and community cohesion

Uniformity of impact

Community knowledge of project

Administrative Factors

Lead agency

Existence and nature of guidelines

Case law

Researcher Factors

Knowledge of laws and SIA practices

Availability of time and personnel

Quality of secondary data and its availability

Political Factors

Level of community value consensus on the project, for or against

Openness of planning process

Level of organization of resistance or advocacy

Methodological Factors

State of the art concerning the group. Details as to the scope of the area of impact can often be determined from an analysis of the technical specifications of the project. Once the social scientist has been told that a proposed project is expected to be located at a given site, the first attempts are made to locate the people who will be affected. These attempts will include the reading of maps, aerial photographs, [satellite] images, and other locational sources in order to identify human clusters (such as towns, villages, farmsteads, and in the case of cities, neighborhoods). At this point, we also begin collecting preliminary secondary data (books, reports, newspaper articles, letters, vital statistics, and other written materials) that will tell us about people in the area. ( Jacobs 1977:3) Following an initial assessment of the project and project area, the researcher must develop some notion of who he or she will be attempting to draw into the research activities. In certain kinds of projects it is necessary to supplement the research team with special expertise. Special experts may include other social scientists, community members, and "lead" agency personnel who might have special knowledge of the design of the project. Some agencies require multidisciplinary teams. As understanding of the project develops, it is necessary to comprehend how the particular project is justified in legislation. For the social scientist the important factors are the specific laws and regulations that govern social impact assessment. This knowledge will strongly affect the methodology selected.

It is beyond the scope of this text to delineate the complexities of the methodology selection process, but it is necessary to assert that there is not one "right" methodology for this type of analysis. As indicated above, the process is influenced by many factors.

The selected research design may be continually refined. This recognizes the impact of the research on the researcher, who will redefine approaches as understanding emerges. This indeterminate discovery process is important because it will increase the chances of understanding the diversity of the project and its potentially changing nature.

In the case of El Llano Project, Sue-Ellen Jacobs used a generalized and flexible assessment technique. Initially, she spent time developing a sense of the place. This involved "windshield" surveys and consultations with maps. This was followed by intensive analysis of secondary data sources. Jacobs emphasizes that in SIA, even the most trivial personal maintenance activities, such as renting an apartment, buying groceries, and setting up later interviews, are all important sources of data. As the investigator becomes more involved in the assessment process, more and more time is invested in direct purposive interviewing.

Because of the diffuse nature of the research process in its early stages, data recording must be done very carefully. Certainly the data storage system should reflect various areas of potential impact, conditioned by the relevant regulations. Jacobs notes:

Some social scientists function on the premise that to have categories established prior to data collection introduces a bias which causes data to be forced into artificial categories. Those social scientists argue that data will fall into "natural" categories if the researcher enters the field without a preconceived notion of how data should aggregate. ( 1975:5)

The anthropologist must contact individuals who reflect a variety of life experiences. Besides the usual categories of concern such as sex, socioeconomic strata, age, and occupation, there must be an attempt to identify and interview those who are supporters or opponents of a specific project, as well as those who are neutral. Additional stratification can be made based on levels of knowledgeability of the project. One way to monitor this issue is to create a representation grid in which persons in the various significant categories of representation are tallied as compared to the appropriately proportioned ideal. It is assumed that given time constraints and the difficulties in establishing an adequate sampling frame, the number of individuals in various categories will be somewhat disproportionate, and selected judgmentally or opportunistically.The collection of diverse data allows the researcher to "profile" the community. Profiling is the process of organizing and displaying data for later interpretation. Profiling is common to virtually all SIA situations, for example, it is explicitly referred to in the laws that affect SIA for the Army Corps of Engineers. Stated simply, a profile consists of characterization of the "ambient conditions of the human environment" ( Vlachos 1975:20). Or alternately, "profiling is the development of complete baseline data in order to provide a basis for projection and planning" ( Van Tassell and Michaelson 1977:14). The data included in the baseline inventory or profile can vary significantly. Vlachos and colleagues state: "No study can include all the variables since a selectivity always operates as to which ones are considered as important in any given project or any given time" ( Vlachos et al. 1975:21).

A Community Profile Outline

(After Vlachos 1975) Section One: Structural Variables

Human Ecology

A. Demographic Characteristics

Population Size

Composition/ethnic mix

Rate of growth/urbanization

2. Population Density

3. Population Mobility

B. B. Spatial Distribution 1. Land use

2. Housing

3. Land ownership

4. Rural-urban

5. Core-suburban

6. Neighborhoods

7. Transportation patterns

II. Characteristic Institutions A. Family

B. Education

Religion

Political 1. Interest groups

2. Community Services

3. Citizen participation

4. Governmental administration

Economic 1. Occupation

2. Wage structure

3. Income levels

Health

Leisure and Recreation

Social Collectivities

A. Formal associations

B. Informal associations 1. Power groups

2. Ethnic groups

3. Class

Section Two: Cultural Tracts

Life-styles

a. Subsistence

b. Communications

1. Language

2. Other expressive media

3. Transportation

4. Proxemics

Religion

1. Beliefs

2. Practices

3. Sacred places, events, and objects

4. Places of worship

Housing

1. Styles of shelter

2. Clusterings

3. Relationships to kin-networks and work

e. Geographic location of people, businesses, farms, and other physical features of the community

f. Institutional characteristics and relationships

Health 1. Culture-specific definitions of health

2. Local health practices

3. Local health facilities

4. Health practitioners

Education

1. Formal

2. Informal

Leisure, cultural, and recreational activities

Politics (at various levels)

1. Formal

2. Informal

Historical Features

Artifacts

1. Contemporary

2. Archaeological

B. Physical Representations

1. Contemporary physical structures

2. Structures of antiquity

Worldviews, Beliefs, Perceptions, and Definitions of Reality

A. Cognitive and Religious Systems

Value Systems

1. Historic values

2. Aesthetic values

C. Belief systems

D. Perceptions of own group and others

IV. Intercultural Perceptions

The Vlachos profile checklist also includes reference to interrelationships, although it does not include any specific content. In addition, each element should be conceived as being in process. This condition is implicit in a conception of impact assessment.

A review of this list and other similar ones reminds one of other checklists such as the Outline of Cultural Materials, which was published by the Human Relations Area Files to serve as a means of coding world cultures, or the checklists associated with characterizations of the community study method ( Arensberg and Kimball 1965). It is quite clear that cultural anthropology is a scientific discipline that is highly appropriate to the goals of profiling.

The next step in "standard" SIA practice is projection ( Vlachos 1975; VanTassell, Tassell and Michaelson 1976; Jacobs 1977). One source notes that "projection involves the selection of a broad range of alternative courses of action, and elaborating their consequences for the present profiled situation over a specified period of time" ( Van Tassell and Michaelson 1976:24). The first step in projection is the specification of alternative "treatments" or projects. Obvious alternatives would include locational choices, alternative designs, scheduling, and, of course, the "no project" alternative ( Van Tassell and Michaelson 1976:24). The alternatives should reflect those considered in technical studies and public discussions.

The second component of the projection process is "forecasting," that is, actually to predict the future. It is essential in the SIA process to make predictions concerning the future. However, because of the state of the art, this tends to be a "black box" affair. That is, specific data is collected in explicit ways and somehow forecasts are made. There are many ways of doing this, but most of the techniques are largely intuitive.

SOME FORECASTING TECHNIQUES

Flow Chart and Diagram Construction. Some techniques cited are simply means for displaying the relevant data. Flow charts and other graphic displays of this type can be used to depict the directions of change and, more important, the interaction between various components of the project and the "human community." These approaches can vary significantly in terms of sophistication.

Metaphors, Analogies, and Comparison. Project effects can be determined by comparing the unimplemented project with those that are similar. The applicability of this approach is limited by the shortage of comparative data and the lack of bibliographic control and access to relevant case studies. The diversity of each completed project makes it difficult to carry out controlled comparisons.

Delphi Technique. Delphi technique was developed at the Rand Corporation in the late 1940s.

Delphi is an attempt to elicit expert opinion in a systematic manner for useful results. It usually involves iterative questionnaires administered to individual experts in a manner protecting the anonymity of their responses. Feedback of results accompanies each iteration of the questionnaire, which continues until convergence of opinion, or a point of diminishing returns, is reached. The end product is the consensus of experts, including their commentary, on each of the questionnaire items, usually organized as a written report by the Delphi investigator. ( Sackman 1975:xi)

Trend Extrapolation. This technique assumes some constancy in existing processes of directional change. That is, trends determined between the present and the past will relate to conditions at some future point. There are various types of trend extrapolation.

Scenarios. Scenarios are "future histories" or "narrative descriptions of potential courses of developments" ( Vlachos et al. 1975:63). Scenario writing draws, in an intuitive way, on the profile narrative. Scenarios are very useful for presenting the nature of the project and its effects to a lay audience. The scenario should reflect various alternative courses of the project.Cross-impact Matrices. Emerging out of environmental impact assessment practice, matrix analysis explicitly depicts the interaction of various project features. In the absence of a scheme of analysis standards, matrix analysis is merely a way to logically present intuitive judgments. It is defined as an experimental approach by which the probability of each item in a forecasted set can be adjusted in view of judgments relating to potential interactions of the forecasted items. Such an approach analyzes cross effects through an elaboration of potential interrelationships in the direction of the interaction, strength of the interaction and time delay of the effect of one effect or another. ( Vlachos et al. 1975:65)

ASSESSMENT

Formally, the next step in the SIA process as defined here is the assessment of impact. Clearly, the distinction between impact projection and assessment is difficult to demonstrate. Assessment involves summarization and comparison of the various projections. The type of assessment used relates very closely to the kinds of projections. Assessments can deal with the following characteristics of the project/context situation: Affected groups:

How they are affected

The likelihood of effects

The timing of effects

The magnitude of effects

The duration of the impact

The breadth and depth of impact, or the diffusion of effects

The source of impact, or the organization

The controllability of the impact generated by given technologies

(Vlachos et al. 1975:62)

Without discussing the details of the assessment process, let us simply emphasize that it is possible to overestimate its power and utility. Social costs are in particular very difficult to determine prospectively. Needless to say, the realm of social indicator analysis is potentially the most productive. One area where productive assessment is more possible is the determination of who is affected. This was an important consideration in Jacobs's assessment of El Llano project, to which we now return.

POTENTIAL IMPACTS OF EL LLANO PROJECT

Jacobs's research revealed that the residents of the impact zone were highly concerned about the project. These concerns were for the most part negative. Earlier in the predevelopment period, a community group was formed to resist the dam project and the related irrigation system. The group was named Association of Communities United to Protect the Rio Grande (ACUPRG). It carried out an active public education and resistance program.

The perceptions of the project and its impact were colored by the nature of the planning procedure used by the Bureau of Reclamation, the lead agency, and the cooperating agency, the Bureau of Indian Affairs. A thorough reading of Jacobs's account, which includes a fair number of basic documents from the project, suggests that the agencies' planning style was a significant causal factor in generating community resistance to the project. The intent of the various agencies involved was not fully disclosed.

One of the major objections to the construction of the project was that it would result in the obliteration of the three-hundred-year-old irrigation system structures. Project planners did not seem to recognize the antiquity and cultural significance of the traditional irrigation system. The ditches were referred to as "temporary diversion structures." The fact that the old dams associated with the ditches were attached to local descent groups was simply not recognized by the official documents.

The local community was also concerned that the new dam would result in reduced water flow, an issue that was not made clear in the earlier documentation of the project. The people feared and objected to this because of the potential problem it would cause the orchards, and the aesthetic cost of a dry river and the associated dam.

The loss of the traditional irrigation system and the initiation of control of the irrigation water by the irrigation district would result in the loss of control of the water by themayordomos and the governor of San Juan Pueblo. In her report Jacobs suggests the "grave concern" of themayordomos for the "future equitable distribution of water" and the future expression of their responsibilities. Jacobs notes: "If this authority and power is, in fact, taken from the mayordomos then one of the strongest measures of local government in the Spanish American communities will be lost" ( 1977:30).

Jacobs's report to the Bureau of Reclamation in Amarillo, Texas contained references to other kinds of "potential social impacts." These include the negative aesthetic effects of construction noise, the problems associated with the population growth stimulated by the project, as well as the associated relocation problems. Jacobs suggested that the area would lose some of its ethnic diversity because of the increased influx of Anglos. The community members were very concerned about the project's effect on the appearance of the river. They were worried about the river going dry and the disappearance of thebosques , or groves, along the banks of the stream.

The project would have an adverse effect on historic and archaeological resources. A number of houses perhaps as old as two hundred years would potentially be harmed by the project. Various archaeological sites were similarly threatened. Community members were concerned that the construction process would disrupt classroom instruction in local schools as well as cause threats to the wellbeing of the students. Related to this were fears that the physical transformation of the communities' life space would result in changes in the informal processes of socialization within the community.

The communities had made use of the existing recreation potential of the river by swimming and fishing. Residents felt that the marginal improvements in recreation potential would not adequately compensate for the negative impacts.

The report also makes the case that the project, during construction and after completion, would result in various health hazards. These would include increased threat of drowning, breeding of insects, and airborne dust. There was also some concern about the possibility of an earthquake that would cause the dam to break.

EVALUATION--THE LAST STEP

Social impact assessment in the final analysis is not a scientific practice as much as a political one. Which is to say, SIA produces documents that assist the process of decision making. This decision making is based on the evaluations presented in the report and the politicians' interpretation of them. The decision takes the form of a selection from among the alternatives. The analyst does not evaluate in the final accounting. That is the task of the politicans and/or the public. This step of the process is often made more public than earlier stages.

Because evaluation is a political process, it is difficult and unpredictable. Its difficulty is caused by the confrontation of local community and development agency values. The focal point may in fact be the impact statement itself. The stress generated by the confrontation can be mitigated by openness in executing the impact assessment. The amount of stress relates to the amount of controversy concerning the project. More controversial projects should be more public, and the openness should occur right from the very beginning.

El Llano Project became controversial. The planning process became more public. Finally, the project was killed in Congress because it was apparent that the positive benefits of the project would not outweigh its negative effects ( Jacobs 1978). In this way the anthropologist's policy research efforts served as a means of protecting community values.

SUMMARY

Social impact assessment is a type of policy research frequently done by anthropologists. In many ways it is very similar to the community study method used by both anthropologists and sociologists for many years. The process of social impact assessment can be highly structured by agency requirements. As this field has developed, the number of regulations and guidelines that structure the work has increased dramatically. A person engaged in SIA must carefully keep up with agency procedures. Just as agency guidelines change and develop, so does the market for such services. An important factor in shaping the market is the amount of federal spending in construction. Decreases in spending on such construction, recently coupled with changes in federal policy toward the whole environmental impact assessment process, has decreased the amount of this kind of work.

An interesting quality of social impact assessment is that learning its techniques can be very useful for most cultural anthropologists. There is great utility in learning how to acquire secondary data and how to treat it in the context of change. An emerging adjunct to the social impact assessment process is the field of public participation coordination.

Further Reading

Derman William, and Scott Whiteford, Eds. 1985.Social Impact Analysis and Development Planning in the Third World. Boulder, Colo.: Westview Press.

Contains good case material and overview discussions. Jacobs Sue-Ellen. 1978 "Top-down Planning': Analysis of Obstacles to Community Development in an Economically Poor Region of the Southwestem United States."Human Organization 37( 3) :246-256.

Contains an account of the author's involvement in the social impact assessment process inEl Llano project.

Millsap William, ed. 1984.Applied Social Science for Environmental Planning. Boulder, Colo: Westview Press.

Good collection of materials on a number of issues that relate to social impact assessment.

Preister Kevin. 1987. "Issue-Centered Social Impact Assessment." InAnthropological Praxis: Translating Knowledge into Action , Robert M. Wulff and Shirley J. Fiske, eds. Boulder, Colo: Westview Press.

Documents anticipated impacts of the development of a skiing area in Colorado.

12 Evaluation

Evaluation is a kind of policy research. It shares some fundamental features with social impact assessment. First, both are concerned with the impact or effects of different actions on people. Second, both can make use of the same kinds of research methods and techniques. But the two kinds of research are different in certain important ways. SIA is primarily concerned with discoveringbefore the fact any costly unintended effects of an activity. For example, governments build dams to impound water so that floods are reduced, agricultural production is increased, or recreational opportunities can be developed. In this situation an SIA might be done to predict whether this would have adverse effects on nearby communities. The purpose of dams is not to displace communities, so it is important to planners to identify all the effects.

Evaluation is most often concerned with determiningafter the fact whether the intended benefits of an activity occurred, or alternatively discovering whether a project with intended benefits is working. For example, an agency might establish a program to increase employment of high school dropouts, and then do research to determine whether the dropouts became employed. In addition, evaluation can be used to examine program operations as well as program effects.

INTEGRATED RESEARCH METHODOLOGY

Our treatment of evaluation will focus primarily on research design. The discussion will start with consideration of classical experimental design. From this base we will consider a number of research design alternatives. Our perspective will be that of general social science as much as anthropology. The specific uses of ethnography in evaluation will be considered. It is important to note that contemporary evaluation research makes use of many different research strategies and that these methods and techniques are used by evaluators regardless of the discipline they were trained in. One is likely to pick up an ethnographically oriented evaluation research report, based on participant-observation, and find that no anthropologists were involved in the study. And, of course, the reverse is true: it is possible to find anthropologists involved in executing pre-test, posttest, control group experimental designs. Ethnographic practice is one tool, a very useful tool, but only one tool. The implications are clear--evaluation researchers need to know a number of different designs whether they are anthropologists, sociologists, psychologists, political scientists, economists, or other kinds of researchers.

This statement on evaluation takes an integrated research methodology approach, which may combine qualitative and quantitative research ( Cook and Reichardt 1979). The integrated research methodology approach requires that we control a variety of research designs and data collection techniques. This implies the possession of the necessary technical skill to process and analyze the data derived through a variety of techniques. The integrated research methodology approach means carefully identifying which research data collection technique is required to solve a research problem. What works? What carries us the furthest in understanding? What is efficient? What research technique is the most credible? These questions represent some of the dimensions that can be considered when we make a judgment about the research approach that will be used.

The criteria that we use to judge which design and technique we will use are quite broad. Of course, basic notions of validity and reliability are among the most important. Another important consideration is cost, both in money and time. The best design in the world is worthless if one can not afford to implement it. Ethnographic practice sometimes involves a great deal of time spent in research. Yet, a good ethnographer can learn a lot about what is going on in a situation by interviewing one person. In any case, there is a large range of legitimate concerns in terms of research design and technique selection. A final, important question is, Does the researcher have the skill to do the task?

Anthropologists can be involved in evaluation in two different ways. First, and most important, is as a broadly trained social scientist who is prepared to do a variety of evaluation research tasks as needed. The second is as a specialized evaluation ethnographer, who contributes to evaluation through nonexperimental, unobtrusive, qualitative, and participatory research techniques as these skills are needed. In this second role, the anthropologist may also be valued because he or she has knowledge of the group within which the evaluation is taking place, in addition to knowledge of technique. While ethnographic skills are very useful in the evaluation process, as we will discuss later, the most promising approach is the first. There are opportunities for professional, ethnographically oriented evaluators, it is just that pathways for career development are more open to people with somewhat broader technical training in research design and data analysis. M. G. Trend speaks of ethnographic evaluators being stuck at the lower levels of large evaluation programs, being relegated to a "go-fer" role with lower pay and less job security ( 1976). For more discussion of these issues see Chapter 14, "Making a Living." One implication of the view expressed above is that there is not an anthropological way of evaluating. It is more useful to think of a multifaceted and loosely structured social science of evaluation in which individual problems in evaluation are addressed using a variety of techniques. Anthropologists can do their job better if they have control of a variety of techniques. The task is not to mimic sociology or psychology, but to participate in a larger contemporary tradition in social or behavioral science. The effect of this on anthropology will be positive.

This chapter will have three parts:

a discussion of the evaluation process, with description of alternatives for evaluation design;

a discussion of differing perspectives on the role of evaluation; and,

(3) case studies in evaluation done by anthropologists. The cases used are evaluations of the Administrative Agency Experiment of the Department of Housing and Urban Development, the Tumaco ( Colombia) Health Project of Foster Parents Plan International, the Job Shop/ Job Club Program of the Community Education Center of West Philadelphia, and a needs assessment done for the United Way of Saskatoon.

EVALUATION PROCESS

Because this single-chapter treatment of evaluation is necessarily brief, our presentation of evaluation may make the field seem much more orderly than it really is. Evaluation encompasses all the disarray that you would expect in a relatively young field in which persons of many disciplines participate. This situation has been exacerbated by the fact that important segments of the field, most notably educational evaluation, have undergone rapid growth forced by huge federal subsidies. For example, the Elementary and Secondary Education Act of 1965 carried a provision that educators receiving grants in support of education programs had to submit evaluation reports that identified the effects of the program. Basically, a lot happened quickly, and there has been little synthesis. There are a number of competing viewpoints and substantial semantic difficulties. Now then, what is evaluation?

At its very core evaluation is what the dictionary says it is, that is, "the determination of the worth of something. While all of us are constantly evaluating things, activities, and ideas, evaluation in a technical sense requires much more than the casually subjective, and largely private, assessments of worth that we produce everyday. Let us consider evaluation as a process.

First, a general technical definition: "Evaluation is the determination of the worth of a thing. It includes obtaining information for use in judging the worth of a program, product, procedure, or objective, or the potential utility of alternative approaches designed to attain specified objectives" ( Worthen and Sanders 1973:19). When evaluation is done, it is almost always done in reference to activity that is intended to affect people in one way or another. Evaluation can be used to determine worth in both negative and positive aspects. While many research designs used in evaluation stress the determination of whether planned objectives were accomplished, evaluation can also be used to discover unintended consequences of programs and projects. The activities evaluated are always motivated by some desired end state.The evaluation process is a process by which values are rationalized. The idea of treatment borrowed from the literature on experimental design is useful as a label for the actions, projects, and so forth that are carried out to achieve goals. Other dimensions to the evaluation process are the nature and characteristics of both the agency providing the service and the individuals and groups that are the focus of the agency.At a general level there are three types of evaluations:

I.

Effect studies--The basic task here is the determination of whether a program (or other entity) is achieving its goals. This is the classic evaluation task. It has also been referred to as product evaluation (Stufflebeam 1973) or outcome evaluation. Effects studies done during the life of a program that are intended to inform program managers or sponsors about program operations can be thought of as process evaluations ( Stufflebeam 1973). Effects studies can be directed at the dissemination of practices to other settings to help guide decisions about continuance, enhancement, curtailment, and modification.

II.

Process studies--The basic task here is to determine how a program is operating. This is a managerial task. This type of evaluation is also called operations analysis ( Riecken 1972). Both process and effect studies may be designed in the same way. Process evaluations may consist of long-term program monitoring.

III.

Needs assessment--The basic task here is to determine the needs of a potentially served population (McKillip 1987; Neuber 1980; Scriven and Roth 1978). One could include needs assessment in a discussion of planning. Needs assessment can also occur during the life of a so as to allow redefining. That is, it can be part of program planning and management. Needs assessment can be ongoing.

This general typology implies a number of dimensions. These include: the purpose or role of the evaluation, the timing of the research, and to some indirect extent, the research design. There are a number of very useful discussions in the evaluation literature that address these dimensions. Let us focus on design. Research design is what is unique about evaluation research when compared to other types of research. Measurement and data analysis techniques are quite comparable in evaluation and basic research. Let us start our discussion of design by considering the classical design pattern and then expanding from that base. Carol H. Weiss depicts the "traditional formulation" of evaluation research in the following way ( 1972:6):

1.Finding out the goals of the program;

2.Translating the goals into measurable indicators of goal achievement;

3.Collecting data on the indicators for those who have been exposed to the program;

4.Collecting similar data on an equivalent group that has not been exposed to the program (control group);

5.Comparing the data on program participants and controls in terms of goal criteria.

This is, of course, a generalized version of experimental design used in behavioral science.

An understanding of this basic pattern can be supplemented by readingExperimental and Quasi-experimental Designs for Research by Donald T. Campbell and J. C. Stanley ( 1965);Research Design in Anthropology: Paradigms and Paradigmatics in the Testing of Hypotheses by John A. Brim and David H. Spain ( 1974); orQuasi-experimentation: Design and Analysis Issues for Field Settings , by Thomas D. Cook and Donald T. Campbell ( 1979). These volumes very clearly lay out research design alternatives.

Campbell and Stanley ( 1965) define five different research designs. The list starts with the least rigorous, the one-shot case study. In a one-shot case study of a program one would measure the effects of the program only once, after the research subjects had participated in the program. What is absent from this method is a baseline measurement, called a pre-test, to determine the "before condition." One has to assume a great deal about the program participants' prior state. Some researchers attempt to strengthen the one-shot case study design by using documentary evidence or reconstructions based on self-reports. At worst, one-shot case studies take the form of program-serving testimonials. Fortunately, there are many circumstances where the one-shot case study can be valuable, because the design is very common. Much ethnographic evaluation takes the form of a one-shot case study. All the examples discussed later in this chapter are of this type.

An enhancement of the one-shot case study is the one-group, pre-test, posttest design. The addition of the pre-test allows one to measure change more objectively. The pre-test, however, does not allow one conclusively to attribute the change to the program. Change can occur because of other events, such as normal change through time, the pre-test's effects, ineffective measurement, and participant fatigue, as well as other factors.

A third type of research design is the static group comparison, which adds a control group to the one-shot case study. In this method, a group that has experienced the program is compared with a group that has not experienced it. The weakness is that the design does not allow certainty about the differences between the groups before the treatment. It is possible to strengthen this design through matching of participants and the use of retrospective measures.

The fourth type is the pre-test, post-test, control group design, which involves setting up two groups before the program. In terms of research design quality, the best way of doing this is by random assignment. There is a before and after measurement to determine effects of the program. This design, although very good, does not control for the effects of the research procedure. This can be controlled for through the use of the final research design, the Solomon four 231-group design, in which the effects of the research are examined along with the effects of the program. These design alternatives do not address issues relating to needs assessment, but are applicable to many contexts in both effect and process studies. The problems with needs assessment mostly revolve around having the research sample be consistent with the group that actually receives the service after program implementation.

While these are standard designs, and each represents an incremental increase in the capacity to specify cause, there are costs associated with increases in experimental control. This is one reason why much evaluation is carried out using the one-shot case study design. The more complex, error-reducing designs are used quite extensively in education to evaluate curricula in anticipation of wider use. It is clear why; the treatments are usually more readily definable and control groups are easier to find and match. Treatments will consist of a set of test materials administered in an adequately standardized way, and, if you need a control group, other classes of student "subjects" are available. Similar patterns occur in the evaluation of drug treatment programs, in which the participants have diminished control because they are in the by order of the courts.

Outside of certain specialized areas, it may be very difficult to apply the more complex research designs. While there are many statements in the literature on evaluation that present the more complex designs as ideals, it is very important to view these designs as alternatives to be selected for application as appropriate. Selection should always be based on the most appropriate design, not the most elegant one.

A very large array of factors needs to be considered in evaluation planning, in addition to basic research design. Perhaps most important is the intended purpose of the evaluation. Some researchers may place too high a value on the elegance of their design, and too low a value on assisting the program to serve its clients better. Anthropologists in evaluation seem to be more committed to clients' needs than others. Other factors to consider are cost, available time, and the nature of the service population.

Soft Designs

As we know, in many circumstances soft and fuzzy is good. Much of the research methodology literature is geared toward an idealization of hard and definite. This is changing somewhat as researchers become disenchanted with operationalism. Yet one still finds defensiveness on the part of soft methodologists. Hard versus soft is not the same as good versus bad. Both approaches are subject to their own problems of quality. By soft designs we mean, among other things, research that stresses qualitative methods, naturalistic observation, discovery, induction, and holism. By hard designs we mean, among other things, quantitative methods, structured observation, verification, deduction, and particularism ( Cook and Reichardt 1979: 10). For example, ethnography, with its emphasis on key informants and participant-observation, tends toward softness; survey research, with its emphasis on randomly selected subjects and instrumented observation (i.e., questionnaires) tends toward hardness. Again, we are not arguing for anything other than the selection of appropriate methods. When are soft methods appropriate?

Soft methods are useful because they are often less of a burden for the program staff. The more structured the research design is, the greater the chance of the evaluation interfering with program functioning. It is very difficult to burden useful programs with certain kinds of highly structured research. The selection of softer designs is called for where obtrusiveness is an issue. Soft methods are useful where program goals are less well-defined, or are especially complex and diverse. Soft approaches are really useful for discovery. Ethnographers seem to do research more to raise questions than to answer them. Soft methods are often the only way to realistically handle complex situations. The more structured the research design, the fewer variables it can consider. Program goals are often not very well defined. Soft techniques can be fit into ongoing program development better than hard approaches. The before and after measures specified in the experimental designs can be replaced with during-during-during measures, which are more workable with softer techniques.

Softer methods often prepare the way for implementation of results better than hard methods because the researchers often end up with an excellent understanding of the persons managing programs and the constraints under which they must operate. In fact, it may be best to have the evaluation include continual feedback to the program with correction. This kind of arrangement is unworkable with hard designs because it interferes with the outcome of the research. Further, hard research designs assume too much about the stability of programs while they are being evaluated. Mid-study change in program administration disrupts the hard studies, but for the soft designs this kind of activity simply represents more data relevant to the program. We might say that soft techniques are useful in rapidly changing circumstances.

Hard Designs

Hard research designs are especially appropriate if the program has clear-cut, measurable objectives that are identified within the program. Hard designs are appropriate where program staff are familiar with, and value, research along with their commitment to service. This orientation is appropriate to situations where control groups are readily available. The idea of the control group is sometimes antithetical to the service orientation of program administrators. In. some circumstances, establishing control groups requires one to deny access. Hard approaches work well where there is relative program stability and a lower expectation or need for mid-course feedback. A useful application of hard approaches is in the production of the final evaluation of demonstration projects, with the goal being to inform potential adopters of the program. Both hard or soft techniques must be executed in a way that allows their results to be applied to real-world decision making. The inflexibility of the hard techniques can relate to less timeliness--that is, the research has to run its prescribed course. Soft techniques, on the other hand, can be so unfocused that the researcher meanders through a program without attending to the needed research issues. It is important to remember that the primary use of evaluation is to provide information for decision making. Late research is bad research.

PERSPECTIVES ON THE ROLE OF EVALUATION

Evaluation has a number of different roles, both legitimate and illegitimate. A useful understanding of the roles of evaluation can be derived from the ideas of Michael Scriven and Daniel L. Stufflebeam. Both have developed concepts of evaluation that can serve to direct it toward greater utility.

In a very important and frequently reprinted article entitled "The Methodology of Evaluation" ( 1973), Michael Scriven provides a number of concepts useful for thinking about the role of evaluation. Although his discussion is focused on evaluation in education, with emphasis on the evaluation of new teaching methods and curricula, his ideas are very widely applicable. As he notes, the roles of evaluation can be quite variable. All these roles relate to the primary goal of evaluation: to determine worth. The role of evaluation in some ways structures the evaluation itself.

Scriven conceives of two types of evaluation research: formative evaluation and summative evaluation. Formative evaluation is carried out in the course of a project, with the goal of improving project functions or products. The evaluation may be done by an outside consultant, but the information produced by the evaluation is for the use of the agency. As Scriven notes, "the evaluation feedback loop stayswithin the developmental agency (its consultants), and serves to improve the product" ( 1973:62). Formative evaluation is conceptualized as a mid-term outcome study of the product or effects of the program, rather than a more general kind of process study, which might answer the question, What is going on here?

Summative evaluation serves to determine worth at the end of the process and is intended to go outside the agency whose work is being evaluated. The evaluation serves to increase utilization and recognition of the project. According to Scriven, program monitoring is a hybrid type of summative evaluation in that it is intended to go outside the agency being evaluated, but at an intermediate time.

Both formative and summative evaluation can make use of the same research designs. However, because of their different roles they require different communication strategies. The essence of the formative-summative contrast rests in the direction and purpose of the communication of evaluation results. Scriven also contrasts what he calls intrinsic and pay-off evaluation. Intrinsic evaluation evaluates the content of the project's product or treatment, whereas pay-off evaluation is focused on effects. These four concepts--formative versus summative, intrinsic versus pay-off--are useful because they focus the evaluation on a specific purpose. Scriven's ideas make us sensitive to the various roles of evaluation; Stufflebeam's work models an entire process of evaluation. His work, also developed in the context of educational evaluation, rests on the assumption that evaluation is done to aid decision making. The information that it provides should be useful to decision makers. Evaluation is a continuing process and is best organized in coordination with implementation. Data collection needs to be consciously targeted on decision-making needs. The total evaluation process ultimately involves collaboration between evaluator and decision maker.

This view of evaluation is integrated by Stufflebeam into a comprehensive process referred to as the CIPP evaluation model. The model specifies different kinds of evaluation, which serve various purposes and inform various types of decisions. These decision types are:

"planning decisions to determine objectives"

"structuring decisions to design procedures"

"recycling decisions to judge and react to attainments" and,

"Implementing decisions to utilize, control and refine procedures" ( Stufflebeam 1973:133). The four types of decisions are served by four types of evaluation.

These arecontext evaluation ,input evaluation ,product evaluation , andprocess evaluation .

Context evaluation supports planning decisions. This category would include what is called needs assessment by others, but would also identify resources that are not being used, and constraints that affect needs. Products of context evaluation include identification of the client population, and of general goals and objectives. Input evaluation supports structuring decisions. The important task here is the identification of resources that relate to project objectives. Part of the process involves the determination of current agency capability. Also included is the identification of alternate strategies for accomplishing objectives. Input evaluation also involves costing out alternatives. Process evaluation supports implementing decisions. This type of evaluation is used to find defects in procedures and implementation, to inform ongoing decisions, and to document activities of the program. Product evaluation informs recycling decisions. The task here is to evaluate project accomplishments at various points in the life of the project. Product evaluation requires operational definition of objectives, development of a measurement strategy, and standards against which measurements are compared.

CASE STUDIES: FOUR EVALUATIONS

These four projects were selected to illustrate evaluation in different settings, all executed by anthropologists. The fact that all four evaluations are case studies is not an accident. Evaluation anthropologists rarely use control groups. While the evaluations are all case studies, they are quite different. They differ in terms of how they structure analytical comparisons. This is especially apparent in the first case, on housing and the third case, on employment. The second case, on health care, is primarily concerned with documenting project completion. The fourth case is a needs assessment for a community service agency.

Evaluating Housing Policy Innovations

The Administrative Agency Experiment (AAE) was one of three social experiments initiated by the U.S. Department of Housing and Urban Development (HUD) in the early 1970s to test the use of direct-cash housing allowance payments to assist low-income families improve the quality of their housing through the open market ( Trend 1978a). Housing vouchers had been considered as an alternative to subsidized public housing by politicians and policy makers for a considerable amount of time. This program attempted to test the workability of this approach and to develop effective management practices. Most other programs for improving housing for the poor presented beneficiaries with limited choice. The AAE payments were made directly to the families, with the stipulation that they had to spend the money on housing.

The three experiments tested different issues. In addition to the AAE, experiments were designed that tested the effect of direct payments on housing supply and demand. The AAE tested various approaches for management of such a social policy. The AAE was implemented by eight public agencies in different localities throughout the country. The agencies were given substantial control over administrative procedures and implementation at their site. Each local project was allowed to enroll up to nine hundred families for the two-year period of the experiment. The sites were in Springfield, Massachusetts; Jacksonville, Florida; Durham, North Carolina; Tulsa, Oklahoma; San Bernardino, California; Portland, Oregon; Peoria, Illinois; and Bismarck, North Dakota.

The evaluation was carried out by Abt Associates, Incorporated, a large research consulting firm located in Cambridge, Massachusetts. The project made extensive use of anthropologists, especially for on-site observation, but also in subsequent analysis. Most of those employed were fresh out of graduate school.

The AAE was intended to provide information on different methods for administering a housing allowance program. The evaluation task was complex and included documenting both effects and administrative procedures. The AAE was implemented by four different types of organizations: local housing authorities, metropolitan government agencies, state community development agencies, and welfare agencies. HUD provided fewer administrative requirements for implementation than normal. They included certain eligibility rules, payment formulas, a locally defined housing standard requirement, restriction to rental property, a lease requirement, a standard payment plan, and certain reporting procedures. Beyond this, variation was encouraged. Agencies had flexibility in various administrative functions. These included outreach methods, enrollment procedures, certification procedures, payment practices, inspection standards, community relations activities, and overall program management. Variation in the form and effectiveness of these different functions was an important focus of the evaluation task. The local agencies generated the program features that were to be evaluated.

The AAE was a naturalistic experiment and therefore did not test hypotheses. The evaluation involved after-the-fact determination of success, with emphasis on documenting and evaluating the various solutions to the problems of implementation. Thus the evaluation questions were determined by the design decisions that were made as the programs developed in the eight sites.

The evaluation was based on data from a variety of sources; most important were the on-site observers. These observers spent a year documenting administrative agency activities. While these observations focused on the agency as a whole, the observations and field notes they produced were referenced to fourteen functions, such as outreach, certification, and inspection, as mentioned above. This allowed cross-project comparison. When variations between agencies were apparent, these variations were examined to determine relative effectiveness and impact. These assessments were addressed in reports that compared projects across functions. The reports provided information that facilitated knowledge transfer to other similar projects. This is the summative evaluation function at work.

The evaluation design required that work be done toward two kinds of research objectives. These included data collection that focused on operational measures, such as numbers of enrollees and payment levels, as well as qualitative assessments of hard-to-measure features, such as fairness and dignity of treatment of participants.

Some measures of function were defined early in the experiment and collected through the use of program documents. Background data on participants and the local community was collected throughout the project. In addition to the information on cross-agency differences, information on common features was also collected. In this frame, outcome data was compared. In addition, the evaluation process also produced separate case studies in reference to specific functions and agency programs that were problematic. Many of these case studies were based on observational data.

Six types of data were collected and used: direct observation of agency activities; surveys of participating households; data from administrative agency forms on participants; agency management reports; site environmental data; and samplings of administrative records. The full-time, on-site observers produced field notes that were formatted to allow comparison between programs categorized by the previously mentioned functions. These materials were used for various purposes, but were especially useful for interpretation. The survey component collected information on income, household composition, and attitudes toward the program on the part of participants. In addition, the time of project employees was charged to the various function areas. All in all, the amount of data collected was immense. For example, over twenty-five thousand pages of field notes were accumulated.

The analysis attempted to assess the overall utility of the direct payment approach to meeting housing needs, while making recommendations as to how direct payment might best be done. The variation-focused evaluation helped identify what were thought to be more effective administrative practices. For example, the Bismarck agency found that they obtained reliable certification results with simpler income determination procedures than the other agencies. The Jacksonville agency evaluation addressed the problems of making the approach work in a tight housing market.

Perhaps most important, the experiment demonstrated that poor people could effectively operate in the open housing market in terms of their own choice. The project also showed that the allowance program did not raise housing costs in these markets. On the negative side, the AAE did not reduce housing segregation significantly. The evaluators concluded that AAE was a useful tool in dealing with the housing problems of poor people from a policy perspective. Ultimately, many of the approaches demonstrated in the AAE came to be implemented in federal housing law.

Evaluating a Health Project Sponsored by a Private Voluntary Organization

Foster Parents Plan International (PLAN) is a private organization that sponsors community development projects in twenty-two developing countries. As a private voluntary organization, or PVO, it is not funded as a government agency, nor is it a profit-making firm ( Buzzard 1982). Most of the funding for the Foster Parents Plan program comes from individual sponsors (called foster parents) in Australia, Canada, the United Kingdom, the Netherlands, and the United States. These individuals are matched with individual needy foster children. This arrangement is the best-known component of the program. PLAN provides these foster children with various services depending on local resources. A portion of the foster parents' contribution may go directly to the family, often to pay school expenses. In addition, these funds can be aggregated and used for community projects. Community projects may start with a request from the local community. Local PLAN offices are budgeted to fund these projects. In addition, some funds are available from the international headquarters in Rhode Island for special projects. Also PLAN may solicit money from government agencies to fund projects.

The project evaluation considered here was part of the PLAN program funded by the U.S. Agency for International Development (AID) to train community health workers in four field locations--two in Colombia, plus Ecuador and Indonesia. The project discussed here was located in Tumaco, a densely populated seaport town in the Pacific lowlands of Colombia. Tumaco was faced with a very difficult public health situation. There was limited clean water and no means of sanitary human waste and garbage disposal. Houses were typically built on piles driven into the tidal flats of river estuaries and beaches. Wastes were disposed of in the water, garbage accumulated near the simple houses, and people defecated on the beaches or in the fields. Although there was a water system, it was old and dilapidated, providing ineffective treatment. In any case, the transport and storage of the water from the tap to its use resulted in more contamination. The effect of this was endemic parasite infestation and high incidence of gastroenteritis, with its debilitating diarrheal disease. Hand washing was infrequent and probably ineffective given the nature of the water. The boiling of water was rare and difficult to get done.

The health problems related to these conditions were exacerbated by poor nutritional status and bad economic conditions. The local diet was based largely on bananas, rice, and cassava. Although the economy was based on littoral fishing and shrimping, many families could not afford these protein-rich foods. As a result, severe protein deficiency could be observed in some children. The generally poor nutritional status of the population tended to increase the effects of the gastrointestinal problems. In this population, diarrheal disease killed children. Other significant health problems were tetanus (especially among newborns), malaria, measles, as well as accidents among the fishermen. Health care facilities were limited.

It was in this context that PLAN initiated the Tumaco Health Project. The project was funded through an agreement with AID. The primary objectives were the improvement of the health and nutritional status of PLAN families, with emphasis on the needs of mothers and children under six years of age. The specific objectives of the five-site project included:

Decrease the mortality and morbidity from diseases that could be controlled through vaccination.

Improve sanitation in the family setting to an adequate level,

(3) Achieve an adequate nutritional level (monitored by age for weight), with special Attention to children under six years of age,

(4) Improve the level of health care and nutrition of mothers during pregnancy and lactation

Provide to family’s health education concerning prevention and treatment of disease

(6) Improve knowledge and access to effective family planning practices.

The Tumaco Health Project sought to achieve these objectives in a number of ways. The core activity was the recruitment, training, and placement of fourteen health promoters from the local community. This required the establishment of an administrative infrastructure and training. The functioning of the health promoters and related programs was to be monitored through a record-keeping and evaluation system that would improve quality of management. Program objectives showed a concern for integration of the Tumaco Health Project with the regional system through referral to existing clinics and collaboration with the Ministry of Health. The project was also to result in the construction and staffing of four health posts, installation of public water systems, and the construction of public latrines. All these activities were to be carried out with maximal community participation.

PLAN sought to evaluate the project mid-term, following the training and placement of the health promoters and the establishment of the health posts.

Unlike many other PVOs, PLAN has an ongoing of evaluation. The evaluation of the Tumaco Health Project reflected the experiences of the PLAN evaluation organization and the reporting requirements associated with the use of AID funds. The evaluation had two primary purposes: the monitoring of the project by PLAN headquarters and the documentation of the project's functioning for the instruction of others doing similar projects.

The project evaluation was conceived as a case study. Presumably because the evaluation was intended for both internal and external use, there was much reporting of background information on health conditions and the local community. The design of the evaluation was not structured as an experiment to determine program effects. The primary emphasis was the identification of project plan attainments rather than project effects.

The data upon which the evaluation was based was collected using a number of techniques. Participant-observation was used to gain a general familiarity with project functions. The participant-observation data included information gathered by accompanying health promoters on their rounds and the observation of daily activities in the health posts. More data was gathered by interviewing various participants in the program and its planning. This included Ministry of Health administrators as well as health service providers and users. Given the basic task of documenting what went on during the implementation portion of the project, the review of project internal correspondence was quite important. This aspect of the data collection process allowed the evaluator to chronicle the changes in the program. This data was supplemented with reviews of hospital records, health post records, and the data collected through the baseline survey carried out by the health promoters themselves.

The evaluation was more concerned with whether program treatments were implemented than with the actual effects of the treatments, as is often called for in an experimental design. The researcher found that the was sound. This focus was consistent with the holistic case-study approach used. It shows the importance of the ability to use relatively unstructured inquiry and existing documents in the evaluation process. The evaluation report shows the utility of an ethnographic approach to evaluation. This particular evaluation was carried out by anthropologist Shirley Buzzard.

Evaluating a Jobs Program in Philadelphia

One of the most significant problems facing inner-city America is unemployment. Programs designed to solve this problem are widespread. The Job Shop Program of the Community Education Center of West Philadelphia is one such program. Awarded a contract from the Pennsylvania Department of Community Affairs, the center was to provide services to unemployed persons in three Philadelphia neighborhoods ( Simon and Curtis 1983).

The program was to offer job search assistance, employability training, and job development services to help unemployed clients. These services were offered as an integrated package of services. The program had four objectives. First, the training of 100 to 120 workers in job readiness, job-seeking skills, and job retention skills, and second, placement of at least 60 percent of these workers in unsubsidized employment or training programs. Third, the program was to assess the employment needs of local employers and to identify potential sources of employees. Fourth, the center was to help program participants retain their jobs by means of counseling and support services.

The center recruited participants through local media, community groups, advertisements, social service organizations, and local churches. The program was intended to serve people in the local neighborhoods. Program participants took classes in getting and keeping jobs. Instructional activities included writing resumes, dealing with applications, reading want ads, the interview experience, and good work habits. Following completion of class work, the participants were to actually search for work using their new skills with the help of project staff. Throughout the project, the staff was to work with local employers to identify possible jobs.

The contract that brought the funds to the center for the project required that the program be evaluated. The evaluation was carried out by Elaine Simon and Karen Curtis, who used a variety of data sources. Program activities were observed and compared with those specified in the proposal. In this way, the "treatment" that the participants received was described.

The proposal presented a rather generalized view of planned program activities. This pattern is very common in evaluation research. It is often the case that programs do not have clearly defined goals. In that goals are frequently the reference point for evaluation, it is not surprising that evaluators often have to spend time identifying goals or using goal-free evaluation techniques.

The evaluation made use of data on individual participants that included referral source, ethnicity, age, education, and previous work history. This was coupled with information on outcomes in employment, referral to other services, and program completion. This data was derived primarily from program records. An interesting component of the evaluation was extensive description of the content of training. The evaluation was further contextualized through the use of a number of participant histories in which their experiences in the program were documented. The individuals selected for profiling in this way were all successful cases. The profiles showed the interrelationship between aspects of the training and preparation and the program's overall success.

The design did not make use of control groups. Moreover, there was no explicit measure of effects of the training on the participants other than the important one, whether they were employed. There was no assessment of levels of new knowledge or changes in attitude. The core of the evaluation consisted of assessing whether the program achieved the numbers of planned participants and whether the predicted placement rates were achieved. These dimensions were compared in a generalized way with the experiences of other such programs. The quality of the employment that these persons obtained was also considered by examining employment in terms of classification by job type.

The evaluation revealed a program that had largely accomplished its originally stated goals. Coupled with this conclusion were a series of recommendations about the effects of certain participant referral sources. The study showed that participants referred to the program from welfare agencies were less likely to finish the program or to be employed if they did finish.

Assessing Community Needs in Saskatoon

The Saskatoon Needs Assessment Project was carried out by a team from the Department of Anthropology and Archaeology of the University of Saskatchewan, led by Alexander M. Ervin (Ervin et al. 1991). The idea for the project came from the board of the United Way. The executive director of the agency approached Ervin about doing the research. The project was funded in 1990 by a community foundation, the university, and the United Way. Saskatoon's population was about 200,000 at this time. The economy of this prairie city includes agriculture, mining, and forestry, as well a growing manufacturing segment. Unemployment was over 10 percent. There were increases in the use of food banks and soup kitchens.

The project was to provide baseline information for the agency to support their decision making in a number of areas. These included "identifying needs and public perceptions to assist agencies in meeting those needs; allocating funds by working with agencies to target programs in identified needs; evaluating new agencies which have applied to join United Way [and] fundraising by focusing marketing efforts on identified needs which the United Way serve" ( Ervin et al. 1991:1).

The research design was developed by the project leader in consultation with an advisory committee, United Way's staff and board, and the research staff. Designs of other Canadian United Way needs assessments were consulted. The assessment process called for six data collection activities. The team reviewed available reports relevant to Saskatoon's needs. These reports, including those from the city government, nongovernment organizations, and academic programs, were abstracted for the final report. The team attempted to review social and economic indicators with the assistance of Statistics Canada. These indicators included census data, household structure, birth rates, labor force, employment, income, disability, and other data. The research team organized three public forums that were highly publicized. They conducted 135 interviews with key informants from community agencies. Five focus groups were held with client groups and one was held with representatives of self-help groups ( Greenbaum 1988; Merton, Fiske, and Kendall 1990). Data was also collected using a threestage Delphi procedure with an expert panel consisting of 28 United Way agency executive directors. Overall the needs assessment had remarkable breadth of contact with community groups. Over 140 agencies or organizations participated in the interviews and forums, or submitted written briefs.

Delphi procedure was developed as a means of collating the opinions of a panel of experts in a way that allows them to be aware of each others' opinions during the process without them being able to influence each other through their personalities. This technique is also discussed in the chapter on social impact assessment. In this case the process started with a single exercise: "Please list what you consider to be the most important social or human needs that should be addressed in Saskatoon, regardless of what agencies or levels of government are responsible for them." The experts were to type in their answers in ten boxes of equal size. The expert panel never spoke with each other directly, yet they communicated to each other through the research team.

The research team analyzed the responses and put them into standardized phrasing and related clusters. This produced a list of 108 needs. In the second round the experts were asked to choose the top twelve needs in rank order, and then to make comments. This produced another list of 86 needs, ranked in terms of their raw scores. The experts were then asked for any adjustments. The need that was ranked highest was the need "to eliminate hunger and therefore the necessity of food banks." Other highly ranked needs were: "need for more emphasis on preventive services," "need for accessible, affordable, quality accommodation; perhaps based on income (i.e. not low-income ghettos)," and the "need to increase core funding for non-government agencies to enable long term planning and development."

The Delphi panel data was used, along with data from all other sources, to produce an abstract of community needs on "the widespread social problems that are confronting Saskatoon" ( Ervin et al. 1991:20). Also reported were the needs of organizations, referred to here as metaneeds. The more than two hundred needs identified were organized into seventeen sectors, including general health, mental health, seniors, native issues, racism and discrimination, immigrant and refugee resettlement, and rehabilitation, among others. The sectors were derived from a directory of community information published by the public library. The research team produced a series of recommendations for the United Way.

SUMMARY

Evaluation research is a rapidly growing area in applied anthropology. Preparation for careers in evaluation should include training in both experimental and case study design, as well as the appropriate data collection techniques. Research methods traditionally associated with anthropology are useful for a number of important evaluation tasks, but these need to be supplemented to meet the entire array of evaluation problems. The utility of ethnographic evaluation methods is directly related to the purpose of the evaluation. Ethnographic evaluation techniques are especially useful when one of the purposes of the evaluation is the documentation of program operations, or the discovery of what went wrong with a program. Ethnographic techniques serve as a good foundation for providing recommendations for program improvement.

Anthropologists in evaluation do not make extensive use of experimental designs. Usually they rely on various kinds of case study approaches. These approaches are quite variable and represent a significant array of research tools in their own right. The utility of the case study approach can be seen in the interest in these approaches shown by nonanthropologists. The literature on evaluation methodology places an emphasis on the use of experimental designs other than the case study. It is important to recognize that in spite of this, much evaluation is done using the case study approach. The reasons for this are largely practical. In many settings, it is expensive and politically awkward to use the more complex experimental designs. In addition, there are many problems in evaluation where the best and perhaps only approach is the case study method. In spite of the continued importance of the case study method there is relatively little discussion in the literature of refinements to the case study methodology.

FURTHER READING

Cook Thomas D., and Charles S. Reichardt, eds. 1979.Qualitative and Quantitative Methods in Evaluation Research . Sage Research Progress Series in Evaluation, vol. 1. Beverly Hills, Calif.: Sage Publications.

Contains a number of good articles dealing with the qualitative--quantitative contrast. See especially the article by M. G. Trend on the Administrative Agency Experiment.

Fetterman David M., and M. Pitman, eds. 1986.Educational Evaluation: Ethnography in Theory, Practice, and Politics . Beverly Hills, Calif.: Sage Publications.

Fetterman David M. 1988.Qualitative Approaches to Evaluation in Education: The Silent Revolution . New York: Praeger.

David Fetterman has provided leadership in the use of ethnography in evaluation.

13 Technology Development Research

Technology development research is a type of policy or applied research that places the research anthropologist as a communication link between producers and users of new technology. The communication may range from relatively informal interaction to highly organized research and extension projects. The research itself may include elements of needs assessment, evaluation, baseline description, social soundness analysis, as well as extension. This type of policy research requires that the researcher have good understanding of the technology being developed and the occupational culture of the technology developers.

What is called technology development research for the purposes of this chapter includes all those research enterprises that serve the goal of the creation of culturally appropriate technology. These functions are carried out in many contexts, including architecture ( Clement 1976), landscape design ( Low and Simon 1984), medical treatment ( Kendall 1989; Coreil 1989), energy source development ( Practical Concepts, Inc. 1980; Roberts 1981), mariculture ( Stoffle 1986), reforestation ( Murray 1987; Smucker 1981), waste disposal ( Elmendorf and Buckles 1978), and housing ( Esber 1987; Mason 1979; Wulff 1972). This general type of research is most common in agricultural development ( McCorkle 1989; Rhoades 1984).

The achievement of cultural appropriateness is an important goal of most of the intervention and research techniques that are discussed in this book. This value is emphasized in technology development research and cultural brokerage. Cultural appropriateness is, of course, conceptually related to the idea of appropriate technology or intermediate technology, developed by E. F. Schumacher ( Schumacher 1973; Dunn 1979; McRobie 1981; Stewart 1977). Developers working within the framework of appropriate technology emphasize the design and manufacture of "small, simple, capital-saving, nonviolent technologies and their supporting institutions" ( McRobie 1981: 13).

FARMING SYSTEMS RESEARCH

Technology development research is best represented by what is called farming systems research (FSR). This policy research tradition has emerged since the late 1970s as an important approach to rural development. It is the product of the work of agricultural economists, sociologists, and anthropologists. The work of Michael Collinson and David W. Norman is often cited in discussions of the development of the approach. According to Deborah Sands, for example, their work in Tanzania and Nigeria, respectively, demonstrated the complexity of small farmer resource allocation decisions ( 1985:3). This implied "that farmers' management strategies and decisions could only be understood within the context of the whole farm system," and "that ideal management in any specific enterprise is not feasible in the small farm situation" ( Sands 1985:3). Yet this agricultural research was usually based on the assumption of ideal procedures, and was usually achieved at an experiment station, under conditions very different from a real farm.

A useful definition of FSR is provided by Shaner, Philipp, and Schmehl:

[FSR is] an approach to agricultural research and development that views the whole farm as a system and focuses on (1) the interdependencies between the components under the control of members of the farm household and (2) how these components interact with the physical, biological and socioeconomic factors not under the household's control. Farming systems are defined by their physical, biological, and socioeconomic setting and by the farm families's goals and other attributes, access to resources, choices of productive activities (enterprises), and management practices. ( 1982:13)

The definition shows FSR's emphasis on sociocultural factors as salient features of farming systems. In this framework sociocultural factors are not viewed as extraneous to the development process, but as essential components that must be understood if the development process is to work.

The perspective represented in this definition is consistent with some fundamental attributes of anthropological theory and method. Two important consistencies are the commitment to holism and the understanding of ecological relationships. Perhaps it is because of these consistencies that anthropologists seem comfortable working in this framework.

Historic Antecedents

Farming systems research has a number of historical antecedents. Perhaps most important is that agricultural research virtually requires, in the opinion of some, the assumption of a systems perspective ( Dillon and Anderson 1984). Others argue that the approach was influenced by American farm management economics as it developed earlier in the twentieth century. The point is well taken because this type of economic analysis was focused on the efficient use of technology to increase farm income. As part of the farm management perspective, analysts studied the total farm in order to understand the relationship between economic activities. The holism characteristic of the farm management economics came to be largely lost in agricultural research, which became strongly reductionist after World War II ( Gilbert, Norman, and Finch 1980). Research became more bounded by disciplines and focused on specific commodities. The research tradition became more geared to an agribusiness perspective while it grew in experimental rigor. In an effort to control variables experimentally, plant breeding and agronomic research became less appropriate to the needs of poor farmers.

Narrow commodity-focused research did have tremendous impacts on international agricultural development processes. It was the commodity research tradition that produced the hybrid rice and corn seeds that served as the foundation for the Green Revolution. In spite of the ongoing critique of these developments, all would agree that this represents technical success ( Ruttan 1977; Wellhausen 1976; Dalrymple 1974). However, the technical success produced substantial social costs, ranging from displacement of rural populations to loss of genetic diversity of native plant stocks.

The commodity focus in international agricultural development was expressed organizationally through the network of International Agricultural Research Centers (IARCs), such as the International Rice Research Institute (IRRI), the Centro Intenacional de la Papa (CIP), the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), theCentro Intemacional Mejoramiento de Maiz y Trigo (CIMMYT), and others. These efforts were sponsored by the Rockefeller Foundation as well as other donors.

While the IARC-produced revolutions are remarkable, the development of agriculture in the Third World will in all likelihood be more gradual than in the past. Unfortunately, the need for increased production is more urgent now than when the first of the revolutionary seeds were introduced to tropical farmers. Simmonds notes:

The IARCs have done much good work and, either directly or indirectly, have had diverse local impacts on agricultural production. But there have been no more Green Revolutions; in retrospect none was to have been expected. The Green Revolution succeeded where wheat and rice were already grown under irrigation and were susceptible to the application of new technology. ( 1984:5-6)

Limits of the Top-down Approaches

For a number of reasons, the socially uncontextualized, high-technology agricultural innovations developed at the international centers came to be questioned, as did many "top-down" approaches to development. New agricultural technology that was developed without reference to local conditions did not work well. Often the new seed required the use of complex, poorly understood and expensive "packages" of "inputs" that were beyond the reach of small farmers. Local researchers lacked the techniques and facilities to adapt the technology to local needs and conditions. The effect was that the new technologies were not attractive to local small farmers. These innovations were resisted by so-called traditional agriculturalists because they were too risky, unprofitable, and unsuitable to local conditions. As has been demonstrated, these farmers were not simply conservative or traditional in their orientation, but were effective, rational scrutinizers of a largely unsuitable technology, and hence, they rejected the offered innovations ( DeWalt 1979; Schultz 1964). The innovations did not make sense within the framework of the constraints faced by the farmers. The innovation-producing researchers did not adequately understand or account for the conditions under which these local, small farmers operated. For this reason much of the investment in agricultural research was fruitless.

FSR grew out of the need to deal with this failure by creating a flow of reliable information to researchers about farmers' needs and frustrations to increase the probability that the new technology can be used and will meet a need. This perspective can be applied to a wide variety of problems outside the realm of agriculture.

FSR PROCESS

As with many innovative approaches to problem solving, there is some disagreement about exactly what is entailed in doing FSR. There are even different names for the activity, and inappropriate use of the terms associated with FSR to dress up traditional approaches. Nevertheless, there are a number of important features that are part of the perspective.

FSR takes a generalized systems perspective with the goal of understanding the complex relations between the physical, biological, and socioeconomic conditions under which farmers operate. The focus on so many linkages and interactions forces a multidisciplinary approach to the development of technology that links plant breeders, agronomists, animal scientists, agricultural engineers, social scientists, and agriculturalists. Of most significance to us is that FSR "places relatively more importance, than in the past, on integrating the social sciences into the research and development process" ( Shaner, Philipp, and Schmehl 1982:15). The FSR anthropologist is an integral part of the team, not an isolated critic.

FSR is farmer-oriented. The farming systems researcher looks "at the interactions taking place within the whole farm setting and measures the results in terms of farmers' and society's goals" ( Shaner, Philipp, and Schmehl 1982:14). The farmer orientation is expressed in many ways, including the fact that basic research on the farming system is carried out before innovations are developed, and farmers participate directly in aspects of the research process ( Matlon et al. 1984). FSR practice calls for substantial on-farm research that includes breeding trials, engineering prototype testing, and Farming practice evaluation. This results in continual iteration between researcher and farmer, and reduction in the distinction between extension and research. In large part, FSR is a bottom-up strategy. In this way FSR contrasts with typical disciplinary commodity research. In spite of this contrast it is important not to think of FSR as a substitute for or alternative to orthodox commodity-focused disciplinary research, but as a complement to it. FSR is a problem-solving process. Farmer constraints are a continual focus of the research and technology development process. Growth of knowledge is a secondary consideration. The problems that are solved tend to be farmer problems.

FSR programs can be thought of as having four stages ( Gilbert, Norman, and Finch 1980). These are the diagnostic or descriptive stage, design stage, testing stage, and extension stage. The primary product of the diagnostic or descriptive stage is baseline documentation of the farming system, with special reference to constraints and potentials. The design stage produces strategies for dealing with the constraints enumerated during the diagnostic stage. These strategies are evaluated under farm conditions during the testing stage. Developed strategies that are shown to be effective through testing become recommendations for dissemination during the extension stage.

Anthropologists seem to be most involved in FSR at the diagnostic stage, while some have been involved in the testing stage ( Rhoades 1984). Certainly, the diagnostic phase draws heavily on anthropological research skills. FSR diagnostics vary in scale from elaborate in-depth research to short-term informal surveys.

The short-term informal surveys used in the diagnostic stage, often calledsondeo ( Hildebrand 1981), informal agricultural surveys ( Rhoades 1982), or rapid rural reconnaissance ( Honadle 1982; van Willigen et al. 1985), are especially interesting methodologically. They are useful when it is necessary to scope out an unfamiliar situation quickly. While these practices do vary considerably, it is possible to identify some attributes that appear consistently. This type of survey is always done by the researchers themselves rather than enumerators. There is always direct observation involved. While the time allotted to these surveys is limited (one week to a month or two), observer-days are increased through use of a team. The team is always multidisciplinary.

Data is collected through key-informant interviews, sometimes done by the team as a group, and direct observation. Interviews are often carried out in the fields to facilitate researcher understanding of the farmer's situation. The process is iterative, that is, the research team continually meets and discusses their current understanding and data needs. Research questions are continually reformulated, often on a daily basis. Purposive rather than random sampling is used. Researchers are focused on learning from the research subjects rather than testing a series of hypotheses.

Rapid Research Techniques

The diagnostic phase may be based on a rapid survey technique, an in-depth sample survey, or a staged combination of rapid reconnaissance and an in-depth survey. Investment of resources in diagnostic research varies with the length of time of the development commitment, level of regional documentation, and amount of regional variation. There seems to be an awareness of the importance of timely and parsimonious research efforts in FSR. These qualities need to be kept in mind in all policy research settings.

Some anthropologists are too quick to call attempts at parsimony or efficiency "quick and dirty." The criticism is often inappropriate. There is a very strong case against the use of elaborate survey methods. Robert Chambers, for example, develops an argument in reference to research done in support of rural development:

The costs and inefficiencies of rural surveys are often high: human costs for the researchers; opportunity costs for research capacity that might have been better used; and inefficiencies in misleading "findings.". . Preparing, conducting, analyzing and writing up a rural survey are heavily committing activities, the demands of which are habitually ignored or underestimated, and the duration of which almost always exceeds that planned. ( Chambers 1983:51)

DIAGNOSIS AND DESCRIPTION

The diagnostic or descriptive stage includes a number of elements. The first step is the determination of the target area ( Sands 1985). This is usually based on either policy or geographic criteria. Subsequent to defining the area for the FSR activity, it is necessary to identify the types of farming systems to be focused upon. As in much policy research, there is a heavy emphasis on the use of secondary data. It is especially important in the establishment of the baseline. The diagnostic baseline, derived from secondary sources, may include "physical factors such as climate, land, soil, and water resources; as well as socio-economic factors such as input-output markets, infrastructure, population density, land tenure systems, and organization of farm production/consumption units" ( Sands 1985:14).

During this phase the FSR team establishes recommendation domains based on the accumulated data (including both secondary and survey data). A recommendation domain is "a group of roughly homogeneous farmers with similar circumstances for whom we can make more or less the same recommendation. [These domains] may be defined in terms of both natural factors--e.g., rainfall-and economic factors--e.g., farm size" ( Byerlee et al. 1980). This concept is very important in FSR. Recommendation domains can be defined on the basis of a number of different variables. This quality of recommendation domains is communicated well by the following statement from a CIMMYT field manual:

Recommendation domains can be defined on the basis of various farmer circumstances. They may be determined by variations in the natural circumstances of the farmer such as rainfall, soils or diseases. A given region may contain manyagro-climatic environments . These are areas where a crop exhibits roughly the same biological expression so that we would obtain, for example, similar varietal or fertilizer responses,everything else being equal . These agro-climatic environments are, however, often modified by socioeconomic circumstances that produce different recommendation domains. For example, close to a large town maize may be grown largely for sale as fresh ears while further away it is a subsistence grain. Such differences may impose modifications on varietal selection and planting date. More commonly, even if all locations are in the same agroclimatic environment, the resource endowments of farmers may lead to different technological needs. For example, small farmers with scarce capital relative to labor and who place more emphasis on food security may follow quite different cropping patterns and practices from large farmers in the same agro-climatic environment. ( Byerlee et al. 1980:10)

Once the general nature of the recommendation domains is described researchers may return for more focused verification.

The idea of recommendation domains can be transferred to a variety of other contexts outside of FSR. The concept is useful in situations where information or advice is being provided that is specific to a particular type of client. This approach requires that the research accurately deal with internal variation within the community being studied, and a subsequent adjustment of the data collection and analysis procedures. The theoretical significance of documenting variation is great, and goes far beyond the need to make focused recommendations in policy research. For a useful discussion of these issues read "Intra-cultural Diversity: Some Theoretical Issues," by Pertti J. Pelto and Gretel H. Pelto ( 1975).

The coordinated use of qualitative and quantitative research techniques makes FSR diagnostics a very comfortable setting for the contemporary research anthropologist. The contribution of anthropologists to the FSR diagnosis process is substantial. Some examples are the work of Robert Rhoades, Robert Werge, and colleagues at CIP ( Rhoades et al. 1982), Billie R. DeWalt and Kathleen M. DeWalt for the International Sorghum/Millet Project (INTSORMIL) in Honduras ( 1982), Edward B. Reeves and Timothy Frankenberger for INTSORMEL in Sudan ( 1981), Jeffrey Jones forCentro Agronomico Tropical de Investigacion y Ensenanza (CATIE) in Costa Rica ( 1982), Constance M. McCorkle for the Small Ruminants Coordinated Research Support Program ( 1989), and Robert Tripp for CIMMYT ( 1982).

Following the initial diagnostic phase and the subsequent design stage, the FSR process usually involves on-farm experiments ( Rhoades 1984). These activities are based upon the diagnostic research. In this phase technical solutions to farmer production constraints are tested under actual farm conditions. Tests include new varieties of plants and animals, and new equipment and facilities. The degree of researcher and farmer involvement can be quite variable. Early in the process the tests are researcher-managed, with the cooperation of the farmers. Later, the experimental program is managed by farmers.

The on-farm experiment stage improves the researchers' understanding of the farming system* and increases the integration of the farmer into the research process. This also has the effect of blurring the distinction between research and extension. The testing phase emphasizes the evaluation of technical improvements in an area under farmer control. The testing program is referenced to a range of potential users and is geared toward fine-tuning the technology in response to farmer needs, constraints, and management practices. Much is learned from the observation of the farming system's response to the experimental program. The continual interaction and mutual support of the on-farm and experiment station research programs is an important characteristic of FSR.

Anthropologists can be involved in the on-farm experiment activities in a substantial way ( Tripp 1985). They are often the staff members with the most extensive relations with farmers, and thus are in a good position to manage this aspect of the program. Tripp notes: "Anthropological experience on how to elicit opinions from farmers, make and record observations informally, utilize different types of data, and explore a wide range of topics of interest to particular research goals is all directly applicable to field management of on-farm experimentation" ( 1985:117).

The last stage of FSR is extension and evaluation. Tested technology is made available to farmers of a specific recommendation domain through extension programs. As noted, the different stages tend to merge into each other in FSR. This is viewed as a positive outcome. Often, descriptions of early reconnaissance data collection methods will advise that extension workers be included in research teams.

Evaluation is developing in importance. As FSR takes a broader perspective, so too should evaluation considerations. The commodity-focused research tradition tended to focus on yields, productivity, and profit. Anthropologists are making progress in the use of nutrition and health as factors for evaluation ( DeWalt and Fordham 1983; Tripp 1982; Fleuret and Fleuret 1983). While the development of nutritional and health criteria for evaluation appears promising, it is important to use evaluation criteria that are as direct as possible. More research is necessary in this area to develop suitable evaluation criteria.

CASE STUDY: NORTH KORDOFAN (SUDAN) SORGHUM AND MILLET STUDY

The FSR project that is presented as a case example here was done from 1981 to 1982 as part of the socioeconomic component of the International Sorghum and Millet Collaborative Research Support Program, that is, the INTSORMIL CRSP. There are a number of CRSPs, all funded as part of the research and development program of the U.S. Agency for International Development. The CRSPs represent an innovative way of organizing research at United States universities in response to the need for developing agriculture in the Third World. Social scientists participate in a number of these programs, including those on small ruminants, beans and cowpeas, peanuts, and nutrition, as well as sorghum and millet. The research investment is primarily in agronomy, but all programs have social science components.

Research Methods

It was the goal of the North Kordofan diagnostic to "identify the socioeconomic constraints that impede agricultural production in the el-Obeid area [of western Sudan]," with emphasis on the relationship between sorghum and millet and the cash crops ( Reeves and Frankenberger 1982: 1). In addition, the project attempted to contribute to the improvement of understanding of sorghum and millet in the agricultural research community through INTSORMIL. The elObeid FSR team worked in collaboration with the Western Sudan Agricultural Research Project (WSARP) in order to provide extensive baseline data on the farming systems to the staff of this newly developing research center. WSARP was investing in a research station at el-Obeid as part of their total program.

In addition to the WSARP collaboration, the team also established relations with Khartoum University; the regional Ministry of Agriculture, Kordofan Province; the Sudan Agricultural Bank, el-Obeid; the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), Sorghum and Millet Breeding Program; the CARE Renewable Energy Project, Desertification and Control Coordination and Monitoring Unit; and the U.S. Agency for International Development Mission--Sudan.

The research focused on both agricultural production and marketing. These major components were supplemented with research into food preparation and consumption. Each of the research components varied somewhat in their approach, although each topically focused research activity was based on a reconnaissance survey of eighteen villages, and an initial ethnographic study of three villages in the area. The information generated from these activities was used to "design a survey instrument of Farming operations" to be used in an areawide sample of fifteen villages ( Reeves and Frankenberger 1982:5).

Initial data collection was broad and included coverage of "farmers access to land, labor, and capital, how they manage these resources, as well as how these resources are channeled into cropping patterns, animal husbandry and off-farm economic activities" ( Reeves and Frankenberger 1982:5). The initial data collection relied heavily on key-informant interviewing. There was an emphasis on collecting local terminology for farming operations, crops, tools, and stages in the growth cycle of millet.

Following this initial exploratory stage a report was issued to potential users before the research went on. This report focused on the farming system per se. The key informant data was expanded upon with an intensive study of three villages. This included a nearly 100 percent sample household survey dealing with household size, total landholding size and area cultivated in sorghum, millet, sesame, and peanuts. Following this, a sample of forty farmers from the three villages was drawn and stratified in terms of landholding and household size. The preliminary survey allowed the refinement of the data collection approach when it was applied to the survey of fifteen villages.

Throughout the study the researchers consulted with various agricultural scientists. These included agronomists, plant breeders, and entomologists. These specialists provided technical information that served to guide portions of the study. As part of these collaborations, the researchers collected local seed for inclusion in germ plasm banks.

The marketing study made use of documentary evidence in the form of government tax receipts and key-informant interviews. This data was supplemented with direct observations, time series grain price data, and price comparison of a market basket of goods. Data was also collected from farmers on marketing in reference to all fifteen villages. Four market centers were selected for their hetereogeneity and examined more closely. Because of the nature of commerce in the el-Obeid area, it took substantial time to develop rapport with merchants-many of them were also smugglers. Four data collection forms or interview schedules were used in the marketing centers. These were directed at inventorying marketing services, interviewing village merchants (career history, capital assets, and farming activities, as well as other attributes); interviewing periodic vendors (stock-in-trade, residence, and market visiting patterns, as well as other attributes); and interviewing market visitors (occupation, sales and purchase, market visitation patterns).

Farming System

The North Kordofan diagnostic revealed the following Farming system. The farms of the region are poor, combining both subsistence and cash crops. Farmers raise sorghum and millet primarily as subsistence crops. These are supplemented with cash crops, including sesame, peanuts, watermelon, and roselle (a flavoring for drinks). Production is diversified through livestock, including cattle, sheep, goats, camels, donkeys, and horses, and poultry, including chickens and pigeons. Gum arabic is collected from acacia trees and sold. Most families engage in some nonfarm economic enterprises. These include charcoal making, water hauling, and retailing. Poorer families sell labor to better-off families during the rainy season. Others migrate to urban areas or large mechanized farms. Some families' incomes are supplemented by remittances from family members that are working in cities and abroad in places like Saudi Arabia.

The rural population is settled in villages that range from five or six to over a thousand households. Village population expands during the rainy season, when most of the crop-related farming activities take place. The households average between seven and eight members. Nuclear family residence is most typical, although extended families, matrifocal families, and other arrangements are common alternatives. While the household is an important unit of cooperation in both production and consumption, household members may have different decision-making interests. Husbands and wives may manage separate farms; unmarried sons and daughters of sufficient age may also manage farms. Most households have members who work off-farm, usually seasonal.

Just over half the land available for cultivation in a given year is being farmed. This indicates that there is insufficient land available for the fallowing cycle to be effective. Ideally under this regime one would need about 80 percent of the land in fallow ( Reeves and Frankenberger 1982:9). One-third of the total cultivated land is being rented. Most of this is poor farmers renting from the betteroff. Land rents are low; the economic motive to rent land to another person is closely related to the shortage of labor. Poor farmers can not afford to hire labor, which would allow them to expand their production.

The most common crops are millet, sorghum, sesame, and peanuts. Millet and sorghum are mostly consumed at home, although some surpluses are sold. Millet is an important staple of the rural diet. The millet stalks are used universally as building material. Fully 95 percent of the households grow millet; it represents 38 percent of the sown land. While sorghum is less common, it too is widely planted.

Sesame is an important cash crop; almost half the land is sown to it. Often intercropped, over 90 percent of the farmers plant it. Sesame is sold at regional markets for the urban cooking oil trade. Sesame market prices are stable. Peanuts are also planted, but suffer from wide price fluctuations due to the fact that this crop goes to the world market. About 10 percent of the crop land was sown to peanuts during the year of the study.

In addition to the crops discussed here, some less important crops are grown. These include roselle, cowpeas, okra, and watermelon. Watermelon is sometimes used as fodder for livestock because of its water content.

Land clearing starts the crop cycle in January to April. From April to August the major crops are planted. The long-maturing local variety of millet is planted first. If adequate soil moisture is present and germination occurs early, the farmer avoids the depredations of insects and birds that attack later-maturing plants. If the early plantings do not germinate, the farmer may replant or choose a crop with a shorter maturity. Plantings of sorgum, sesame, and peanut occur in June and early July.

Farmers prefer that all crops be weeded twice. Poor farmers are often forced to sell their labor during peak work seasons, and thus are not able to work their own fields. Wealthy farmers may have three or four weedings. Harvest takes place from late August to January, with the bulk of the activity occurring in October and November.

While weather represents a major constraint, labor supply also presents problems in the production cycle. For many farmers the largest production input cost is labor. It was determined that the greatest return on labor was with millet. However millet is subject to pests. Sesame, a popular crop, offered a lower return on labor inputs. Despite this, sesame cultivation was attractive because it represented risk-aversion. Peanuts were more risky and also required substantial labor inputs.

Animals are important in this farming system. Over 90 percent of the households own animals, although there are wide differences in number from household to household. The ownership of cattle and sheep is especially concentrated. Goat ownership is diffuse, on the other hand, because of the relative low cost of goats and their utility as dairy and meat animals. Animals represent a good means of storing resources from a good crop year through bad years. Unfortunately, the animals have a deleterious effect on the environment and seem to contribute to desertification. While some crop residues and crops are used to feed working animals (donkeys and camels), most graze on lands beyond the cultivated zone of the village.

The other important component of the farming system is the activities that occur off-farm. Most every household supplements its income with off-farm activities of various kinds. Some farmers migrate during the dry season for wage labor or other activities such as water hauling, tailoring, carpentry, charcoal manufacturing, or itinerant marketing. Others derive income from village shops, bakeries, flour mills, oil presses, cisterns, and trucks. About a fourth of the families receive remittances from members who have gone elsewhere to work.

Based on their analysis, Reeves and Frankenberger concluded that "farming in this region is not subsistence-oriented" ( 1982: iii). There is continual trading and selling of crops to local merchants to obtain needed commodities. The researchers were able to demonstrate that the farming systems of the el-Obeid area were well adapted to the uncertainties of the environment through diversification of the basic farm enterprise.

Constraints

Perhaps the most important research goal was the identification of constraints in the farming system. The identified constraints serve as foci for the subsequent research and extension that closes the FSR circle. Reeves and Frankenberger divide constraints into three broad categories: crop production constraints, input constraints, and other constraints. These are presented with the farmer's compensating strategies for dealing with the constraint and a set of recommendations for both research and extension. The researchers identified constraints related to wind erosion, pests and diseases, loss of soil fertility, availability of rainfall, access to labor, access to seeds, chemical inputs, availability of drinking water, credit, crop auction procedures, gum arabic price policy, limited farming knowledge, and transport and storage.

To learn more about constraint identification, let us examine one set of constraints, compensating strategies, and recommendations--that of wind erosion. Reeves and Frankenberger describe the wind erosion constraint in the following way:

High winds in this area often blow away freshly planted seeds or newly germinated crop seedlings in farmer's fields. Millet and sesame are particularly susceptible to such wind erosion. This often forces farmers to bear the time and labor cost of replanting. In addition, such wind erosion removes top soil from farmer's fields which adversely effects crop output. ( 1982:90)

The researchers excerpted information from their ethnographic data base that was related to this particular problem. The compensating strategies identified in the research are:

1. Farmers plant sorghum in the same hole with sesame. The firm root structure and sturdy stalks of sorghum make the sesame plant less susceptible to wind erosion.

2. In addition to sorghum, farmers plant a number of other crops with sesame in the same hole. The other crops prevent winds from uprooting the sesame.

3. Farmers often leave trees and bushes in their fields until after the first weeding to protect the soil and newly planted seeds from wind erosion.

4. Farmers plant excessively large amounts of seeds per hole so that the crop germinates as a bush. By increasing the density of plants per hole wind is less likely to blow them away.

5. When farmers clear their fields prior to planting, they sometimes leave cut-up bushes, grasses and crop residue lying on the field to protect the soil from wind erosion. When planting begins, some farmers remove this debris while others plant around it. ( Reeves and Frankenberger 1982:90)

In the framework of this particular constraint, the researchers recommended both research and extension activities. Researchers were encouraged to examine the relationships between intercropped commodities so that the most appropriate crop combinations could be identified. This was seen as a way of encouraging intercropping. Soil scientists were requested to research the effects of early land cleiring and planting in this area. These farming activities may be implicated in the desert encroachment that occurs throughout the Sahel. Field clearing is done to synchronize with labor availability rather than the optimum time for the curtailing of wind erosion. Because of this, fields may sit cleared for a long period of time. The researchers pointed out that suggestions to delay land clearing would be followed with reluctance because of the labor "bottleneck" that it would cause.

The extension workers were encouraged to recommend to farmers that the cut weeds and crop residue be left on the fields altogether so as to reduce wind erosion and increase soil moisture retention. Crops could be planted through this debris using a minimal tillage technique. The researchers also suggested that tree shelter belts be planted around farmers' fields. They also requested that certain potential shelter belt trees be evaluated by researchers.

The recommendations themselves were derived from a number of sources. An important source was the farmers themselves. They, like farmers everywhere, were able to discuss solutions to their problems. In addition, recommendations were developed in conjunction with the collaborating researchers.

SUMMARY

The North Kordofan FSR study illustrates one approach to doing farming systems research. FSR is the most elaborately developed type of what we are calling technology development research. While it may be difficult to estimate the future growth and demand for these kinds of services, the field is developing very rapidly and is associated with the development of academic interest in agricultural research. Practice within FSR may prove instructive for all policy research anthropologists. Some elements that may be useful in other subfields are the idea of recommendation domains, the rapid reconnaissance survey, and the very close level of cooperation between the anthropological producers of research and the agricultural scientist consumers of it.

FURTHER READING

DeWalt Billie R. 1985. "Anthropology, Sociology, and Farming Systems Research."Human Organization 44 (2): 106-114. This review essay serves as an introduction to a series of articles on aspects of farming systems research to be published in Human Organization. The introduction of this chapter is based on this treatment of the topic.

Byerlee Derek, Michael Collinson, et al. 1980.Planning Technologies Appropriate to Farmers: Concepts and Procedures . Mexico: CIMMYT. A useful how-to manual produced by the economics of CIMMYT. Contains a good treament of the idea of recommendation domains.

McCorkle, Constance M. 1989.The Social Sciences in International Agricultural Research: Lessons from the CRSPs . Boulder, Colo.: Lynne Reiner Publishers. Documents the participation of anthropologists in the development of agricultural technology.

Rhoades Robert E. 1984.Breaking New Ground: Agricultural Anthropology . Lima, Peru: International Potato Center. Highly instructive case study of the use of anthropology in the development of new potato technology.

Schultz T. W. 1964.Transforming Traditional Agriculture . New Haven: Yale University Press. While not an example of FSR, this book, by the Nobel laureate Theodore Schultz, provides a clear statement about the essential economic rationality of the traditional agriculturalists.