Evaluation Review recent issues

2012-01-25T01:00:29-08:00

This study evaluates the 2008–2009 "Lose your Excuse" public service advertising (PSA) campaign on energy efficiency targeting 8- to 12-year-olds, intended to increase knowledge, foster proactive attitudes, and change energy usage behaviors. Baseline and two follow-up surveys were conducted with online samples representative of the national population of households with kids with online access. Almost half (47%) of the tweens recognized at least one ad from the campaign. Ad recognition was positively associated with knowledge, proactive attitudes, and energy-saving behavior. Propensity score analysis confirmed a small but measurable and statistically significant effect on energy-saving behavior. The discussion section compares these results to public health campaigns in terms of ghost awareness, reach, and effect size.

2012-01-25T01:00:29-08:00

Many inquiries regarding the causal effects of policies or programs are based on research designs where the treatment assignment process is unknown, and thus valid inferences depend on tenuous assumptions about the assignment mechanism. This article draws attention to the importance of understanding the assignment mechanism in policy and program evaluation studies, and illustrates how information collected through interviews can develop a richer understanding of the assignment mechanism. Focusing on the issue of student assignment to algebra in 8th grade, I show how a preliminary data collection effort aimed at understanding the assignment mechanism is particularly beneficial in multisite observational studies in education. The findings, based on ten interviews and administrative data from a large school district, draw attention to the often ignored heterogeneity in the assignment mechanism across schools. These findings likely extend beyond the current research project in question to related educational policy issues such as ability grouping, tracking, differential course taking, and curricular intensity, as well as other social programs in which the assignment mechanism can differ across sites.

2012-01-25T01:00:29-08:00

Two case studies are presented to compare and contrast the challenges encountered when attempting to conduct participatory evaluations (P-Es) with tribal programs that represented two extremes of collaboration between the programs and evaluators. In one case, the P-E was successful because the principals were invested in the program, whereas in the second case, the absence of a shared program vision hampered the ability to adequately evaluate the program. The authors contend that governmental funding agencies have responsibilities to ensure that their expectations are appropriate, the proposed program is feasible, and that technical assistance should be provided before the grant is written, not after it has been funded.

2012-01-25T01:00:29-08:00

A mathematical model of HIV/sexually transmitted infections (STI) transmission was used to examine how linearity or nonlinearity in the relationship between the number of unprotected sex acts (or the number of sex partners) and the risk of acquiring HIV or a highly infectious STI (such as gonorrhea or chlamydia) affects the utility of sexual behavior change measures as indicators of the effectiveness of HIV/STI risk-reduction interventions. Findings indicate that the risk of acquiring HIV through vaginal intercourse is essentially a linear function of the number of unprotected sex acts and is nearly independent of the number of sex partners. Consequently, the number of unprotected sex acts is an excellent marker for the risk of acquiring HIV through vaginal intercourse, whereas the number of sex partners is largely uninformative. In general, the number of unprotected sex acts is not an adequate marker for the risk of acquiring a highly infectious STI due to the highly nonlinear per act transmission dynamics of these STIs. The number of sex partners is a reasonable indicator of STI risk only under highly circumscribed conditions. A theoretical explanation for this pattern of results is provided. The contrasting extent to which HIV and highly infectious STIs deviate from the linearity assumption that underlies sexual behavior outcome measures has important implications for the use of these measures to assess the effectiveness of HIV/STI risk-reduction interventions.

2011-11-14T16:23:55-08:00

This article describes the methodology used in the 2007 U.S. National Roadside Survey to estimate the prevalence of alcohol- and drug-impaired driving and alcohol- and drug-involved driving. This study involved randomly stopping drivers at 300 locations across the 48 continental U.S. states at sites selected through a stratified random sampling procedure. Data were collected during a 2-hr Friday daytime session at 60 locations and during 2-hr nighttime weekend periods at 240 locations. Both self-report and biological measures were taken. Biological measures included breath alcohol measurements from 9,413 respondents, oral fluid samples from 7,719 respondents, and blood samples from 3,276 respondents.

2011-11-14T16:23:55-08:00

This article examines alcohol-impaired collision metrics around nine sobriety checkpoint locations in Indianapolis, Indiana, before and after implementation of 22 checkpoints, using a pre/post examination, a pre/post nonequivalent comparison group analysis, and an interrupted time series approach. Traffic safety officials used geographical information system (GIS) analysis to help select checkpoint locations amid high alcohol-related collision clusters, then analyzed possible checkpoint impacts. A post hoc analysis examined counts and rates of impaired collisions before and after checkpoint dates within the 2-mile radius zones around each checkpoint site and compared pre-to-post differences to two similar time-matched control areas without checkpoints. As a group, checkpoint zones showed significant though minor declines in pre-to-post collision counts, and no impairment rate change. Considered together, non-downtown checkpoint zones had more favorable impairment rates than comparison areas. After controlling for collision volume, monthly trend, and locational effects, the interrupted time series analysis found that for all checkpoints the count of impaired collisions in post-checkpoint periods was about 19% less than pre-checkpoint counts.

2011-11-14T16:23:55-08:00

The article describes a psychosocial model of intervention with psychiatric patients in long-term hospitalization in a psychiatric ward in Israel and reports the findings of the evaluation conducted of its effectiveness. The model was aimed at maintaining or improving the patients' functioning in four main areas: personal hygiene, environmental maintenance, occupational competency, and social and recreational activities. For each patient, an individualized intervention program was designed and implemented by a personal case manager appointed from the ward’s clinical staff. The programs involved family and ward staff as well as the patients themselves. The evaluation, based on comparison of monthly data for 4 months before and 12 months during the intervention, showed significant improvement in all but two areas of functioning: social functioning and psychiatric status. The findings, although in need of confirmation in larger, representative samples of patients in Israel and elsewhere, hold promise for effective intervention with patients whose treatment today consists largely of medication.

2011-11-14T16:23:55-08:00

The labor market integration of immigrants is a top political priority throughout the Organization for Economic Cooperation and Development (OECD) countries. Social and fiscal gains, as well as sustained future labor supply make governments search for effective policies to increase employment among the mostly disadvantaged. The author studies SIN, a Swedish pilot workplace introduction program targeting these groups, using very detailed individual data and allowing for effects through several channels. The results show increased transitions from unemployment to work experience schemes and improved future employment probabilities for those who entered these schemes. A rough calculation suggests that each job year created cost about Euro 30,000.

2011-11-14T16:23:55-08:00

PartyIntents examines whether portal survey methods could be used to anonymously survey gay and bisexual men about HIV-risk behaviors before and after a weekend party–oriented vacation. The study recruited 97% of eligible men and of these 489 participants 47% completed the follow-up assessment. Approximately one half of the men intended to use illegal drugs over the weekend, and almost 20% thought that they might have anal intercourse and not use a condom. The methodology can be applied and provides useful information about HIV risk at these events, though refinements may be needed to increase the follow-up rates.

2011-09-13T14:44:00-07:00

Along with the late Howard Freeman, Richard Berk was a founding editor of Evaluation Review (then Evaluation Quarterly) in 1977. He resigned as editor of this journal at the end of 2010. In this article, he reflects on his experiences.

2011-09-13T14:44:00-07:00

This article reviews the mechanics of conventional and piecewise growth models to demonstrate the unique affordances of each technique for examining the nature and predictors of children’s early literacy learning during the transition from preschool through first grade. Using the nationally representative Family and Child Experiences Survey (FACES) data set, 1997 cohort, the authors show how piecewise models revealed discrete contributions of child, family, and classroom experiences to children’s literacy skills within particular years, whereas conventional models, which considered the whole 3-year trajectory of change as a single outcome, revealed fewer of these nuanced contributions.

2011-09-13T14:44:00-07:00

Mediation analysis in child and adolescent development research is possible using large secondary data sets. This article provides an overview of two statistical methods commonly used to test mediated effects in secondary analysis: multiple regression and structural equation modeling (SEM). Two empirical studies are presented to illustrate the respective circumstances in which the two methods are most useful. One study examines the mediated effect of parents' social capital on parent involvement in Head Start programs through parent-child bond. The other study assesses the mediating effects of structured routine activities, delinquent association, and prosocial belief on the relationship between religiosity and juvenile delinquency.

2011-09-13T14:44:00-07:00

Complex survey data are collected by means other than simple random samples. This creates two analytical issues: nonindependence and unequal selection probability. Failing to address these issues results in underestimated standard errors and biased parameter estimates. Using data from the nationally representative Head Start Family and Child Experiences Survey (FACES; 1997 and 2000 cohorts), three diverse multilevel models are presented that illustrate differences in results depending on addressing or ignoring the complex sampling issues. Limitations of using complex survey data are reported, along with recommendations for reporting complex sample results.

2011-09-13T14:44:00-07:00

Complex survey data, as highlighted in this issue of Evaluation Review, provide a wealth of opportunities for answering methodological and/or applied research questions. However, the analytic issues of nonindependence and unequal selection probability must be addressed when analyzing this type of data. Thus, to ensure that research questions are accurately answered when using complex survey data, researchers must take extra precautions to ensure complex survey data are correctly analyzed. The purpose of this article is to provide software recommendations for analysis of and tips on troubleshooting when analyzing complex sample data.

2011-05-09T21:43:46-07:00

Evaluation designs for social programs are developed assuming minimal or no disruption from external shocks, such as natural disasters. This is because extremely rare shocks may not make it worthwhile to account for them in the design. Among extreme shocks is the 2010 Chile earthquake. Un Buen Comienzo (UBC), an ongoing early childhood program in Chile, was directly affected by the earthquake. This article discusses (a) the factors the UBC team considered for deciding whether to put on hold or continue implementation and data collection for this experimental study; and (b) how the team reached consensus on those decisions. A lesson learned is that the use of an experimental design for UBC insured that the evaluation’s internal validity was not compromised by the earthquake’s consequences, although cohort comparisons were compromised. Other lessons can be transferred to other contexts where external shocks affect an ongoing experimental or quasi-experimental impact evaluation.

2011-05-09T21:43:46-07:00

Individuals in contact with the criminal justice system are a key population of concern to public health. Record linkage studies can be useful for studying health outcomes for this group, but the use of aliases complicates the process of linking records across databases. This study was undertaken to determine the impact of aliases on sensitivity and specificity of record linkage and how this affects ascertainment of mortality. Records for a cohort of prisoners were linked to methadone maintenance treatment (MMT) and mortality records. The record linkage was conducted in two stages. First, the linkage was undertaken using the participant’s name and date of birth as recorded in a prior study. Then, a second linkage was undertaken using these identifiers, plus all known aliases. Sensitivity was 64%, and specificity 100%, for the first linkage. When aliases were added to the linkage, sensitivity increased to 86% and specificity remained 100%. The standardized mortality ratio was 4.3 for the first linkage, increasing to 6.1 when aliases were used. These results suggest that the potential effects of participant aliases on linkage outcomes, and methods for mitigating these effects should be carefully considered when planning and undertaking record linkage studies with criminal justice populations.

2011-05-09T21:43:46-07:00

Warrantless arrest laws for domestic violence (DV) are generally classified as discretionary, preferred, or mandatory, based on the level of power accorded to police in deciding whether to arrest. However, there is a lack of consensus in the literature regarding how each state’s law should be categorized. Using three classification schemes, this study examined whether variations among these schemes impact research outcomes by analyzing the effects of discretionary, preferred, and mandatory warrantless arrest laws on intimate partner homicide (IPH). Variations in classification schemes and in the dates of law passage presented in the literature resulted in differing estimated effects of the laws.

2011-05-09T21:43:46-07:00

Free to Grow: Head Start Partnerships to Promote Substance-free Communities (FTG) was a national initiative in which local Head Start (HS) agencies, in partnership with other community organizations, implemented a mix of evidence-based family-strengthening and community-strengthening strategies. The evaluation of FTG used a quasi-experimental design to compare 14 communities that participated in the FTG intervention with 14 matched comparison communities. Telephone surveys were conducted with two cohorts of the primary caregivers of children in HS at baseline and then annually for 2 years. The survey was also administered to repeated cross-sectional samples of primary caregivers of young children who were not enrolled in HS. No consistent evidence was found in changes in family functioning or neighborhood conditions when the 14 FTG sites were compared to 14 matched sites. However, caregivers of young children who were not in HS in three high-implementing FTG sites showed evidence of improvements in neighborhood organization, neighborhood norms against substance abuse, and child disciplinary practices. Results provide highly limited support for the concept that family and neighborhood conditions that are likely to affect child development and well-being can be changed through organized efforts implemented by local HS programs.