The problem isn't just poverty. Poverty's an effect.
"Poverty" where I live, a mostly black neighborhood, means that kids live in single-family dwellings, usually with two cars both less than 10 years old. Some live with one parent. They have cable, Internet with wifi routers (meaning probably at least two computers), HDTV. They're not wealthy, but let's not call it "poverty."
And their scores are 7 points below Latinos, 25 points below whites, and 6-27 points below Asians on standardized tests, in English, science, and math. The baseline's different, but the difference is about the same.
Latinos have slightly higher family income than blacks, but that's because their unemployment numbers are lower. Average wage is a bit lower, but more members in a household work. Many have ESL issues. 7 points higher.
This spread has been true at schools around here for more than a decade. It's shifted maybe a point or two over that time. But during that time, school populations have shifted demographically by 40-70%; administrations have come and gone; schools have replaced 80% or more of the teaching staff and focused on making sure that they "accidentally" hired teachers with the right skin color for their school's kids. Spending's increased. Tech's increased. Monitoring's increased. They've forced teaching styles to change, implemented all kinds of response/intervention strategies, gone from no test review before major standardized tests to having 3 weeks review. They've introduced free breakfasts and Saturday school, more after school tutoring and summer programs. They've built new schools that are the kind of building you drool over.
The schools that had a 67% pass rate for one ethnicity 12 years ago have seen kids enter first grade in a radically different environment graduate with a 67% pass rate.
The administrations, buildings, teachers, teaching styles, responses keep changing. The only thing that doesn't change, really, are the students and their families.
It's gotta be the teachers. So of course the teachers that can escape do escape.