An AI tool used in thousands of criminal cases is facing legal challenges

Law enforcement agencies and prosecutors from Colorado to New York have turned to a little-known artificial intelligence tool in recent years to help investigate, charge and convict suspects accused of murder and other serious crimes.

But as the software, called Cybercheck, has spread, defense lawyers have questioned its accuracy and reliability. Its methodology is opaque, they’ve said, and it hasn’t been independently vetted. 

The company behind the software has said the technology relies on machine learning to scour vast swaths of the web and gather “open source intelligence” — social media profiles, email addresses and other publicly available information — to help identify potential suspects’ physical locations and other details in homicides and human trafficking crimes, cold cases and manhunts.

The tool’s creator, Adam Mosher, has said that Cybercheck’s accuracy tops 90% and that it performs automated research that would take humans hundreds of hours to complete. By last year, the software had been used in nearly 8,000 cases spanning 40 states and nearly 300 agencies, according to a court decision that cited prosecutors in a New York case that relied on the tool.   

In the New York case, a judge barred authorities from introducing Cybercheck evidence last year after having found that prosecutors hadn’t shown that it was reliable or well-accepted, the decision shows. In another ruling last year, an Ohio judge blocked a Cybercheck analysis when Mosher refused to disclose the software’s methodology.

“We’re being asked to trust a company to present evidence that could eventually put people in prison,” said William Budington, a senior staff technologist with the Electronic Frontier Foundation, a civil liberties group. “That goes against the right to due process.”

In a motion filed last month in a fatal robbery in Akron, Ohio, defense lawyers representing two defendants charged with murder demanded that Mosher provide the software’s proprietary code and algorithm. 

In the April 10 filing, the lawyers also made an alarming series of allegations: Mosher lied under oath about his expertise and made false claims about when and where the technology has been used.

Donald Malarcik, one of the defense lawyers who filed the motion, said in an email that it was “shocking” that prosecutors continued to rely on Mosher as an expert.

Mosher didn’t respond to a detailed list of questions about Malarcik’s comments or the filing. An executive for the Canadian company that makes Cybercheck, Global Intelligence Inc., cited ongoing court matters and said it wouldn’t comment. 

Mosher has said he wouldn’t provide Cybercheck’s software to defense experts because it is proprietary, according to an appeals filing in the Ohio homicide case that was ruled on last year. 

A spokesman for the Akron Police Department, which investigated the fatal robbery, didn’t respond to requests for comment. A spokesman for the Summit County Prosecutor’s Office, which is handling the case, wouldn’t comment, citing pending litigation.

At a hearing last year, the prosecutor addressed some of the allegations in the filing and said Mosher was “great at software, great at open source intelligence, a work in progress on the law,” according to a transcript of the September hearing.

The prosecutor, Brian Stano, said his office had no intention of opening an investigation into Mosher, the transcript shows. 

“I believe this is more of a lost-in-translation issue as opposed to some sort of impropriety,” he said.

‘Circumstantial’ evidence 

When law enforcement agencies request assistance from Cybercheck, the software searches parts of the web that aren’t indexed by search engines, as well as the surface web. Those findings are compiled in a report and provided to law enforcement agencies.

Several contracts and proposals reviewed by NBC News show officials from Washington state to Pennsylvania considering or agreeing to pay Global Intelligence $11,000 to $35,000 for Cybercheck. The company secured a $25,000 agreement with Akron beginning in April 2022 that included 50 cases and 40 hours of “real time intelligence,” according to a copy of the contract Malarcik obtained through a public records request and shared with NBC News.

The Akron police spokesman didn’t respond to a request for comment about the agreement.

In the fatal robbery, two men were arrested in July 2021 in connection with the alleged crime nine months earlier, authorities said in a news release. The men, Deshawn Coleman and Eric Farrey Jr., were later indicted on charges of aggravated murder, aggravated robbery and other crimes.

A ballistics analysis on a gun found at Coleman’s home linked him to shell casings at the scene of the homicide, Malarcik said. A car registered to Farrey was captured on camera near the scene, he said. Both men have pleaded not guilty, and Malarcik described that evidence as “entirely circumstantial.”

In December 2022, Malarcik said in an interview, Cybercheck produced a report placing both men at the scene. The report was created after the software searched the web for 21 days in an automated process that sifted through 1.1 petabytes of information — or more than 1 million gigabytes — and created “cyber profiles” for Coleman and Farrey, according to the filing.

Those profiles were assembled from email addresses and social media accounts, according to the filing. Cybercheck connected the profiles to the scene of the killing within minutes of the homicide using a network address — a unique number that identifies devices connected to the internet — from a Wi-Fi-enabled security camera, according to the filing.

At least one device — possibly a phone — with a suspect’s cyber profile had tried to communicate with the camera’s Wi-Fi connection, according to the report, Malarcik said.

The report didn’t cite any video recording of the killing, and it wasn’t clear where Cybercheck found the camera’s network address or how it verified that the device was at the scene. The defense’s forensic experts were unable to locate the social media accounts cited in the report, the filing says, and it wasn’t clear how the software verified an email address that it said belonged to both defendants. 

At a hearing last summer, Mosher testified that the software’s conclusions in the case were 98.2% accurate, according to the filing, which doesn’t provide additional details about how that accuracy rate was calculated.

At the hearing, Mosher said his software has never been peer-reviewed, the filing says, noting that he provided the same testimony in an earlier case in Akron.

But at a hearing in a third homicide case in Akron in October, Mosher said Cybercheck was peer-reviewed by the University of Saskatchewan, according to a ruling this year. The judge in that case ordered Mosher to provide the study to prosecutors and Malarcik, a copy of the ruling shows. 

The 47-page document, which Malarcik said Mosher emailed him in February, is from 2019. It appears to be an instructional document for the software and doesn’t say who performed the review or include their findings.

A University of Saskatchewan spokesman said the school had a research contract with Global Intelligence that was connected to an agreement with Canada’s National Research Council, a science and technology agency focused on research and development.  

The university “was not involved in the creation of the document, nor did we create any content for the document, nor can we say conclusively what information the company used to create it,” the spokesman said in a statement.

The research done under the contract was never submitted anywhere, the spokesman said, “and therefore was not peer-reviewed.”

Global Intelligence and the spokesman for the Summit County Prosecutor’s Office wouldn’t comment on the document.

Data not preserved 

According to the defendant’s April 10 filing in the Akron homicide prosecution, Cybercheck doesn’t preserve the data it uses to create or locate cyber profiles.

Mosher didn’t respond to a request for comment on the apparent practice. In a transcript of July proceedings in the case provided by Malarcik, Stano, the prosecutor, questioned Mosher about what data Cybercheck stores. 

Mosher said the software doesn’t index or collect data because it’s a data processor, not a data collector. He cited the large sizes of the files, as well as “other considerations around governance and compliance,” according to the transcript.

The April 10 filing asks the judge to order Mosher to share Cybercheck’s software and points to other decisions in which judges have ruled that defendants are entitled to review it. Mosher didn’t respond to a question asking whether he planned to share Cybercheck’s methodology, and Malarcik said he wasn’t aware whether Mosher had shared the software in any of the cases in which he has been ordered to. 

In one such case — another Akron homicide, in which the defendant, Javion Rankin, was charged with aggravated murder and other crimes in 2021 — Mosher refused to provide the software’s methodology because it is proprietary, a March 28 appeals filing shows. After that refusal, the judge barred prosecutors from introducing Cybercheck evidence in the case, the filing shows. 

Prosecutors appealed the ruling, saying it “destroyed” the possibility of an effective prosecution, a notice of appeal shows. Rankin was released on his own recognizance last year while the appeal moves forward, said Malarcik, who also represents Rankin.  

Expert testimony under scrutiny

Lawyers for Farrey and Coleman also point to claims Mosher made under oath in the Rankin case in June about his experience as an expert witness. 

Mosher said he has testified 13 times, the filing says, but he failed to provide a list of cases after the judge requested one. At an April 2023 hearing in Colorado for a case alleging child sex abuse image crimes, Mosher told a judge he’d testified just twice, the filing says.

The two cases Mosher cited in the Colorado hearing were linked to child sexual abuse image crimes in Canada, according to the filing. Mosher provided case names and numbers, according to the filing, and during a hearing he said he’d provide trial transcripts.

But when an investigator for the lawyer in Colorado reached out, prosecutors in Canada said Mosher hadn’t testified, the filing says, citing emails from the officials. 

In one case, Mosher provided material to authorities in Calgary, Alberta, that he said he found on the dark web, the filing says.

“But it was not usable disclosure,” the prosecutor in that case said, according to the filing. “Our Internet Child Exploitation Unit technicians were unable to read or analyze it or locate anything in the ‘data dump.’” 

The prosecutor added that the trial ended on the first day, when the accused pleaded guilty, the filing says.

In the second case, in New Brunswick, the person Mosher identified as a defendant hadn’t even been charged, the filing says, citing a corporal with the Royal Canadian Mounted Police’s child exploitation unit.

When prosecutors in Canada learned about Mosher’s “false testimony,” the filing says, they reached out to authorities handling the Colorado case. 

On Aug. 4, prosecutors moved to dismiss the charges, a court filing shows. The filing doesn’t say why the Boulder County prosecutor sought the dismissal. A spokesperson for the DA’s office declined to comment, citing a state law that bars her office from discussing cases that have been dismissed and sealed.

Mosher didn’t respond to requests for comment.