The Illinois Department of Children and Family Services is ending a high-profile program that used computer data mining to identify children at risk for serious injury or death after the agency's top official called the technology unreliable.
"We are not doing the predictive analytics because it didn't seem to be predicting much," DCFS Director Beverly "B.J." Walker told the Tribune.
The $366,000 Rapid Safety Feedback program was central to reforms promised by Walker's predecessor, George Sheldon, who took office in 2015 following a series of child deaths and other problems.
Two Florida firms -- the nonprofit Eckerd Connects and its for-profit partner, Mindshare Technology -- mined electronic DCFS files and assigned a score of 1 to 100 to children who were the subject of an abuse allegation to the agency hotline. The algorithms rated the children's risk of being killed or severely injured during the next two years, according to DCFS public statements.
But caseworkers were alarmed and overwhelmed by alerts as thousands of children were rated as needing urgent protection. More than 4,100 Illinois children were assigned a 90 percent or greater probability of death or injury, according to internal DCFS child-tracking data released to the Tribune under state public records laws.
And 369 youngsters, all under age 9, got a 100 percent chance of death or serious injury in the next two years, the Tribune found.
At the same time, high-profile child deaths kept cropping up with little warning from the predictive analytics software, DCFS officials told the Tribune.
One child who did not get a high-risk score was 17-month-old Semaj Crosby, who was found dead under a couch in her Joliet Township home in April following at least 10 DCFS abuse investigations and an ongoing "intact family" care plan.
Another child the computers failed to flag was 22-month-old Itachi Boyle, who died a month after Semaj in Rock Island following eight DCFS mistreatment investigations into his home and similar "intact family" services from a nonprofit contracted by DCFS, the Tribune found.
"Predictive analytics (wasn't) predicting any of the bad cases," Walker told the Tribune. "I've decided not to proceed with that contract."
A May 2017 Tribune investigation found the arrangement with Eckerd was among a series of no-bid deals Sheldon gave to a circle of associates from his previous work in Florida as a child welfare official, lawyer and lobbyist. Sheldon left Illinois under a cloud a month later, and a July joint report by the Office of Executive Inspector General and the DCFS inspector general concluded that Sheldon and DCFS committed mismanagement by classifying the Eckerd/Mindshare arrangement as a grant, instead of as a no-bid contract.
By doing so, the joint report said, DCFS avoided state bidding transparency requirements, making it impossible to determine if Illinois could have obtained the same services from local companies at a lower cost, a requirement of the state's procurement code.
Predictive analytics has captured the imaginations of human services administrators around the globe and tapped an estimated $270 million state and federal government market for child welfare data collection and analysis. If it is possible to use big data to spotlight a child in trouble and intervene before he or she is hurt, then doing so is government's moral obligation, advocates for the technology say.
Eckerd Connects -- which recently changed its name from Eckerd Kids -- told the Tribune that variants of its Rapid Safety Feedback are used today by child welfare agencies in Ohio, Indiana, Maine, Louisiana, Tennessee, Connecticut and Oklahoma.
But some large child welfare agencies have balked at the expense and ethical questions about predicting children's futures based on the histories of their families. The algorithms could disproportionately select poor children of color for government intervention, critics warn, and automated decision-making may replace the judgment of experienced child welfare professionals.
The effort in Illinois unraveled following missteps acknowledged by both DCFS and Eckerd, Tribune interviews show.
The DCFS automated case-tracking system was riddled with data entry errors in both the Semaj Crosby and Itachi Boyle cases, the Tribune found. In addition, it did not link investigations about many children to cases regarding their siblings, or other adults in the same home.
These and other shortfalls undermined Eckerd's analysis. And state laws forced DCFS to erase "unfounded" child mistreatment investigations, giving the Eckerd analysts less data to work with.
The department is now moving to change the way it indexes and links investigations, and it is also considering legislative changes that would allow it to retain records of past unproven allegations.
For its part, Eckerd told the Tribune it regrets using stark language suggesting the company could predict the probability of harm to a child.
Illinois child care agencies told the Tribune they were alarmed by computer-generated alerts like the one that said: "Please note that the two youngest children, ages 1 year and 4 years have been assigned a 99% probability by the Eckerd Rapid Safety Feedback metrics of serious harm or death in the next two years."
"We all agree that we could have done a better job with that language. I admit it is confusing," said Eckerd spokesman Douglas Tobin.
Eckerd now says the 1-to-100 score is merely meant to represent how closely a child matches historical data on fatality and harm cases.
After the Tribune raised questions about the language Eckerd and Mindshare were using, Eckerd asked DCFS to strike that language from internal communications to child workers. "We are working to change that language," Tobin said.
DCFS used similar language about predicting harm or death in public budget statements as well as in federal court filings for the consent decree where the agency describes its programs and reform efforts.
While Eckerd says details of its risk-assignment algorithms are considered proprietary, the basics of its Rapid Safety Feedback are outlined in state procurement files, federal court reports and marketing presentations from Illinois and other states.
Eckerd retrospectively analyzes thousands of closed abuse cases and from them draws data points that are highly correlated with serious harm. The parents' ages could be a factor -- or their previous criminal records, evidence of substance abuse in the home, or the presence of a new boyfriend or girlfriend.
DCFS gives Eckerd a nightly "data dump" from the state's automated case-tracking system, and the next morning Eckerd generates real-time scores flagging the most imperiled children.
Front-line caseworkers should never get those raw scores, let alone make decisions based on them, Eckerd says; the data instead should be reviewed by DCFS supervisors who are trained and coached by Eckerd to decide which cases need immediate attention and how to tackle them.
Even before arriving in Illinois, Sheldon had professional ties to both Eckerd and Mindshare.
He is quoted on Mindshare's website endorsing that company and its technology. And as head of Florida's child welfare agency, he worked closely with Eckerd, which runs child welfare programs in Florida's Hillsborough County under a $73 million state contract, using for-profit companies as subcontractors.
When Sheldon arrived in Illinois in 2015, he appointed Eckerd's Chief External Relations Officer Jody Grutza to a $125,000 senior DCFS position. While Grutza did not supervise the Eckerd contract, Sheldon put her in charge of overseeing other deals with Sheldon's Florida associates, including a Five Points Technology contract that paid $262,000 to Christopher Pantaleon, a longtime Sheldon aide with whom Sheldon owned Florida property, the Tribune revealed in a July report.
After a year in Illinois, Grutza returned to a top position with Eckerd in Florida.
In a brief interview, Sheldon said it was smart to tap Florida experts he trusted from previous work as he hit the ground running in Illinois, and that the Eckerd/Mindshare partnership had a good national reputation well beyond Florida.
In contract papers submitted to Illinois DCFS, Eckerd described the "remarkable" accomplishments of its predictive analytics method, saying Eckerd had virtually eliminated abuse-related deaths of wards in Hillsborough County since 2012.
But the Tribune's report in May report found at least five Hillsborough County children who died while under Eckerd's supervision in 2015 and 2016, including one whose foster mother faces pending first-degree murder and aggravated child abuse charges.
Eckerd said that four of those five fatalities were accidental deaths related to unsafe sleep or natural causes, and the alleged homicide involved a child in foster care who had not received a Rapid Safety Feedback assessment.
Despite her decision to end the predictive analysis program, Walker told the Tribune that Eckerd did provide useful case-analysis training that is currently being used by 15 agency staffers and three supervisors. This team is reviewing the roughly 2,700 cases of families receiving "intact family services" and prioritizing them to identify the highest-risk cases, Walker told the Tribune.
This review is "going to take several months" but "it's going to teach us a lot," Walker said.
Walker said she also is working to reduce the caseloads of investigators while improving communication between state investigators and the nonprofit organizations that deliver the "intact" services to troubled families.
(c)2017 the Chicago Tribune
