The ruling deals a rare rebuke to U.S. spying programs that have generally withstood legal challenge and review since they were dramatically expanded after the Sept. 11, 2001, attacks. The opinion resulted in the FBI agreeing to better safeguard privacy and apply new procedures, including recording how the database is searched to detect possible future compliance issues.

The intelligence community disclosed Tuesday that the Foreign Intelligence Surveillance Court last year found that the FBI’s efforts to search data about Americans ensnared in a warrantless internet-surveillance program intended to target foreign suspects have violated the law authorizing the program, as well as the Constitution’s Fourth Amendment protections against unreasonable searches. The issue was made public by the government only after it lost an appeal of the judgment earlier this year before another secret court.

The court concluded that in at least a handful of cases, the FBI had been improperly searching a database of raw intelligence for information on Americans—raising concerns about oversight of the program, which as a spy program operates in near total secrecy.

The October 2018 court ruling identifies improper searches of raw intelligence databases by the bureau in 2017 and 2018 that were deemed problematic in part because of their breadth, which sometimes involved queries related to thousands or tens of thousands of pieces of data, such as emails or telephone numbers. In one case, the ruling suggested, the FBI was using the intelligence information to vet its personnel and cooperating sources. Federal law requires that the database only be searched by the FBI as part of seeking evidence of a crime or for foreign-intelligence information.

In other instances, the court ruled that the database had been improperly used by individuals. In one case, an FBI contractor ran a query of an intelligence database—searching information on himself, other FBI personnel and his relatives, the court revealed.

The Trump administration failed to make a persuasive argument that modifying the program to better protect the privacy of Americans would hinder the FBI’s ability to address national-security threats, wrote U.S. District Judge James Boasberg, who serves on the FISA Court, in the partially redacted 138-page opinion released Tuesday.

In one case central to the court’s opinion, the FBI in March 2017 conducted a broad search for information related to more than 70,000 emails, phone numbers and other digital identifiers. The bureau appeared to be looking for data to conduct a security review of people with access to its buildings and computers— meaning the FBI was searching for data linked to its own employees.

Judge Boasberg wrote that the case demonstrated how a “single improper decision or assessment” resulted in a search of data belonging to a large number of individuals. He said the government had reported since April 2017 “a large number of FBI queries that were not reasonably likely to return foreign-intelligence information or evidence of a crime,” the standard required for such searches.

“The court accordingly finds that the FBI’s querying procedures and minimization procedures are not consistent with the requirements of the Fourth Amendment,” Judge Boasberg concluded.

The legal fight over the FBI’s use of the surveillance tool has played out in secret since the courts that adjudicate these issues under the Foreign Intelligence Surveillance Act of 1978 rarely publicize their work. It was resolved last month after the government created new procedures in the wake of losing an appeal to the U.S. Foreign Intelligence Surveillance Court of Review—a secret appeals court that is rarely consulted and seldom releases opinions publicly. That resolution cleared the way for the disclosure Tuesday.

Additionally, FBI Director Chris Wray ordered the creation of a compliance review team following the October decision, a bureau official said.

The program in question, known as Section 702 surveillance, has roots in the national-security tools set up by the George W. Bush administration following the Sept. 11, 2001, terrorist attacks. It was later enshrined in law by Congress to target the electronic communications of non-Americans located overseas. The program is principally used by the National Security Agency to collect certain categories of foreign intelligence from international phone calls and emails about terrorism suspects, cyber threats and other security risks.

Information from that surveillance is often shared with relevant federal government agencies with the names of any U.S. persons redacted to protect their privacy, unless an agency requests that identities be unmasked.

Privacy advocates have long criticized the Section 702 law for allowing broad surveillance that can implicate Americans and doesn’t require individualized warrants. U.S. intelligence officials have defended it as among the most valuable national-security tools at their disposal, even as intelligence agencies have acknowledged that some communications from Americans are swept up in the process.

The court documents released Tuesday reveal unprecedented detail about how communications from Americans were ensnared and searched by intelligence collection programs that U.S. officials have publicly said are aimed mainly at foreigners. They cast doubt on whether law-enforcement and intelligence agencies are carefully complying with privacy procedures Congress has mandated.

Sen. Ron Wyden (D., Ore.), a critic of U.S. surveillance programs, said the disclosure “reveals serious failings in the FBI’s backdoor searches, underscoring the need for the government to seek a warrant before searching through mountains of private data on Americans.”

President Trump signed into law a six-year renewal of the Section 702 program in early 2018. Changes to the law allowed the court to review the FBI’s data handling ultimately led to the October ruling.

The surveillance court opinions are the latest setback for U.S. surveillance practices during the Trump administration. The NSA last year turned off a program that collects domestic phone metadata—the time and duration of a call but not its content—amid at least two compliance issues involving the overcollection of data the spy agency wasn’t authorized to obtain.

The FBI has also been under intense political pressure from Mr. Trump and his allies, who allege that the bureau’s surveillance of a Trump campaign associate was improper. That surveillance of the aide, Carter Page, fell under a different provision of the foreign intelligence law but has nevertheless sparked a major debate about the scope of the bureau’s authorities.