Opportunity Knocks #40 - The Disconnect Between Policy Makers and Policy Implementers + Lots of Studies and articles
Every Monday, I share reflections, ideas, questions, and content suggestions focused on championing, building and accelerating opportunity for children.
Last week, the GAO issued a report examining schools that have been identified as those needing comprehensive support and improvement (CSI) under the Every Student Succeeds Act (ESSA) of 2015, essentially, the lowest performing schools that need the most support. There was one paragraph that caught my attention. From the report:
“Evidence-based interventions. Several district officials described this as the most challenging of the three required elements we reviewed. First, two district officials noted that the ESEA definition of evidence-based can be hard to understand, making it difficult to determine whether an intervention meets those standards. All seven of the school officials we spoke with were unaware of Education’s Clearinghouse as a resource to help them do so. Two school officials said they often turned to alternative informal means of selecting interventions, including asking for word-of-mouth recommendations from colleagues or posting in informal social media groups. These groups may not include experts on evidence based interventions. Additionally, two district officials and one school official stated that they turned to educational product vendors or paid consultants for assistance in selecting interventions. Finally, one district official stated that teachers—and even a school principal in one case— sometimes resist considering new interventions. This official shared the opinion that many school officials need help understanding why a new approach is needed as their default is to continue with the status quo.
Seven superintendents isn’t representative of the over 7,000 in the United States (it does seem that state-level policy makers are more familiar with the tool). However, for U.S. Ed’s “central and trusted source of scientific evidence on education programs, products, practices, and policies” to be totally unknown to district superintendents is a glaring, if extreme, example of the disconnect between policy makers and policy implementers that pervades youth-facing work. What causes this disconnect? Some ideas:
Message melt and morph: Spillane, Reiser, and Reimer (2002) examine how policy messages get reinterpreted as they move through the system, often leading to significant differences between the policy as intended by its creators and as understood and implemented by folks closest to the policy consumer.
Absent provider voice: Coburn (2001) emphasizes that for policies to be effectively implemented, the perspectives and experiences of those at the ground level need to be integrated into policy formulation.
Political incentives and short-termism: political actors—which all policy makers are—gravitate to the most innovative policy ideas and/or the most straightforward policies to implement in order to optimize for positive electoral outcomes. And for the same reason, they often over-index for perceived benefits of policies. Thus, what is actually needed on the ground is not well understood (or altogether disregarded).
Broken feedback loops: Sabatier (1986) highlights the importance of feedback in the policy process, noting that effective policy implementation requires continuous monitoring and the use of feedback to adjust strategies in response to observed effects and changing conditions.
Distinct intellectual paradigms: this idea is from Fink (2000), “Policy makers operate from a very rational linear intellectual paradigm. They look on education as a technical exercise that can be improved by a mandate here, a new policy there, or an plethora of accountability procedure. They define 'what' the pupils are to learn, divide the learning into convenient chunks, establish time frames, develop a testing regimen, organize suitable materials, and tell teachers how to achieve predetermined targets. . . Implementers face a non-rational, non-linear, complex and some would even suggest chaotic reality.”
So, what can we do to narrow this disconnect?
Implement terms limits.
Dramatically expand participatory budgeting.
The President should issue an executive order requiring all federal agencies to conduct child impact statements—answering fundamental questions about how a policy may benefit or harm a child—for every new regulation, set of guidance or program. Governors should do the same. Individuals closest to children should be systematically involved in answering those questions.
If you read one article this week:
Miller, Lisa (2024). An American Girlhood in the Ozempic Era: Adults are divided about giving children new drugs for weight loss. At 13, Maggie Ervie decided to take them. New York. “By January 2023, when she turned 15, Maggie had lost more than 80 pounds and grown three inches. That month, the American Academy of Pediatrics released new clinical obesity-treatment guidelines recommending that pediatricians steer their patients along the path she had forged: consider bariatric surgery and medication as early as possible. The guidelines created a public furor. These GLP-1 agonists were new and their long-term effects on children’s minds and bodies entirely unknown. The number of children with eating disorders, including those in larger bodies, had surged during the pandemic, and specialists who treat them feared that the guidelines would place even more clinical scrutiny on adolescents’ weight. Some nutritionists pointed out the risks of malnutrition and side effects, while others worried that doctors’ full-throated endorsement of the drugs signaled the end of any hope that American families might ever learn a joyful, unencumbered relationship with eating. Public-health advocates despaired that prescribing a generation of children GLP-1 agonists was nothing but a Band-Aid solution to the ailments caused by Big Food. Most vocal among the critics were activists from the fat-acceptance movement who had spent the past 20 years pushing clinicians to stop conflating larger bodies with illness. The guidelines’ authors, they pointed out, claimed to focus on ensuring children’s health but had devoted most of the document’s 100 pages to addressing their size — a doublespeak familiar to anyone who grew up under the rule of quick-fix diets and faddish drugs.”
If you think about one chart this week: From Dalton, Rosalia et al (2024). The Department of Education’s budget tug of war: Congress vs. presidents. Brookings Institute
Two observations: 1) It’s hard, regardless of the political balance of congress and the presidency, to dramatically cut Department of Ed funding (worth noting as calls to abolish Ed ramp up during the 2024 election cycle); and 2) the share of federal government spending on children effectively has effectively remained flat over the last decade. And while there was a spike in the share of federal government spending on children’s issues in 2021 because of Covid-19 stimulus checks, projections show that by 2025 the share will revert to 2019 levels. This when children are facing unprecedented mental health and learning loss crises.
If you consider a few studies this week:
Vicente, L., & Matute, H. (2023). Humans inherit artificial intelligence biases. Scientific Reports. “Artificial intelligence recommendations are sometimes erroneous and biased. In our research, we hypothesized that people who perform a (simulated) medical diagnostic task assisted by a biased AI system will reproduce the model's bias in their own decisions, even when they move to a context without AI support. In three experiments, participants completed a medical-themed classification task with or without the help of a biased AI system. The biased recommendations by the AI influenced participants' decisions. Moreover, when those participants, assisted by the AI, moved on to perform the task without assistance, they made the same errors as the AI had made during the previous phase. Thus, participants' responses mimicked AI bias even when the AI was no longer making suggestions. These results provide evidence of human inheritance of AI bias.”
Ortiz, R., et al (2024). Evidence for the association between adverse childhood family environment, child abuse, and caregiver warmth and cardiovascular health across the lifespan: The Coronary Artery Risk Development in Young Adults (CARDIA) study. Cardiovascular Quality and Outcomes. “Although risky family environmental factors in childhood increase the odds of poor longitudinal adult CVH, caregiver warmth may increase the odds of CVH, and socioeconomic attainment in adulthood may contextualize the level of risk. Toward a paradigm of primordial prevention of cardiovascular disease, childhood exposures and economic opportunity may play a crucial role in CVH across the life course.”
Maples, B., Cerit, M., Vishwanath, A. et al (2024). Loneliness and suicide mitigation for students using GPT3-enabled chatbots. NPJ. “Mental Health Research. “Mental health is a crisis for learners globally, and digital support is increasingly seen as a critical resource. Concurrently, Intelligent Social Agents receive exponentially more engagement than other conversational systems, but their use in digital therapy provision is nascent. A survey of 1006 student users of the Intelligent Social Agent, Replika, investigated participants’ loneliness, perceived social support, use patterns, and beliefs about Replika. We found participants were more lonely than typical student populations but still perceived high social support. Many used Replika in multiple, overlapping ways—as a friend, a therapist, and an intellectual mirror. Many also held overlapping and often conflicting beliefs about Replika—calling it a machine, an intelligence, and a human. Critically, 3% reported that Replika halted their suicidal ideation. A comparative analysis of this group with the wider participant population is provided.”
Cortes, K. (2024). A scalable approach to high-impact tutoring for young readers: Results of a randomized controlled trial. NBER. “This paper presents the results from a randomized controlled trial of Chapter One, an early elementary reading tutoring program that embeds part-time tutors into the classroom to provide short bursts of 1:1 instruction. Eligible kindergarten students were randomly assigned to receive supplementary tutoring during the 2021-22 school year (N=818). The study occurred in a large Southeastern district serving predominantly Black and Hispanic students. Students assigned to the program were over two times more likely to reach the program’s target reading level by the end of kindergarten (70% vs. 32%). The results were largely homogenous across student populations and extended to district-administered assessments. These findings provide promising evidence of an affordable and sustainable approach for delivering personalized reading tutoring at scale.”
King, C. A., Beetham, et al (2024). Adolescent residential addiction treatment in the US: Uneven access, waitlists, and high costs. Health Affairs. “Drug overdose deaths among adolescents are increasing in the United States. Residential treatment facilities are one treatment option for adolescents with substance use disorders, yet little is known about their accessibility or cost. Using the Substance Abuse and Mental Health Services Administration’s treatment locator and search engine advertising data, we identified 160 residential addiction treatment facilities that treated adolescents with opioid use disorder as of December 2022. We called facilities while role-playing as the aunt or uncle of a sixteen-year-old child with a recent nonfatal overdose, to inquire about policies and costs. Eighty-seven facilities (54.4 percent) had a bed immediately available. Among sites with a waitlist, the mean wait time for a bed was 28.4 days. Of facilities providing cost information, the mean cost of treatment per day was $878. Daily costs among for-profit facilities were triple those of nonprofit facilities. Half of facilities required up-front payment by self-pay patients. The mean up-front cost was $28,731. We were unable to identify any facilities for adolescents in ten states or Washington, D.C. Access to adolescent residential addiction treatment centers in the United States is limited and costly.”
Brough, R., Phillips, D. C., & Turner, P. S. (Forthcoming). High schools tailored to adults can help them complete a traditional diploma and excel in the labor market. American Economic Journal: Economic Policy. “More than 18 million adults in the US have no high school credential. Later on, these adults are less likely to earn full diplomas than GEDs, but diplomas are potentially more valuable. A network of high schools helps adults graduate by providing a tailored curriculum, coaching for non-academic barriers, onsite child care, and transportation. After 5 years, earnings increase by 38% more for graduates than applicants who do not enroll. We address selection by conditioning on 5 years of pre-application earnings and comparing to students who exit after positive shocks. Much of the wage gains can be accounted for by sectoral switching, and evidence on completion of credentials is consistent with a human capital explanation for the results.”
Ciocca Eller, C. (2024). The Power of Evidence to Drive America's Progress. Harvard University & Results for America. “The next 10 years should be focused on 6 key recommendations: 1. Continue to set aside federal funds to build evidence of what works, for whom, and under what circumstances, while also ensuring that federal legislation, regulations, and guidance prioritize data and evidence use: A critical step for directing public resources toward improving outcomes in education, workforce, poverty reduction and other areas of economic mobility. 2. Foster demand for data and evidence through active policy leadership, providing the knowledge and tools for more federal policy leaders to champion the use of evidence and data in the policy-making process. 3. Use federal policy and guidance to build evidence and data capacity in state, local, Tribal, and territorial governments. 4. Advance data and evidence as public goods by expanding public access to both and incorporating underrepresented voices into the evidence ecosystem. 5. Advocate for full implementation of the Evidence Act, ensuring that sufficient federal resources are allocated to evidence and data annually and that evidence and data leaders in federal agencies have a seat at the policy decision-making table. 6. Celebrate progress, tell the story, and mobilize new champions by increasing plain-language storytelling about the impacts of evidence and data on real people, advancing economic mobility, elevating mutual learning, and continuing to expand the evidence ecosystem.”
Until next week, be calm and be kind,
Andrew