U.S. and Russian negotiators held their first security talks since Russia’s deployment of tens of thousands of troops to the Ukrainian border sparked fears of an invasion, but they left the Monday talks saying they failed to narrow their differences.

Although both sides described the discussions as businesslike, they remain at odds over Moscow’s demands that the North Atlantic Treaty Organization, the postwar military alliance of Western powers, halt its expansion in Europe, and the U.S.’s insistence that Russia remove more than 100,000 Russian troops arrayed near Ukraine’s border.

The troop buildup has prompted concerns that Russian President Vladimir Putin intends to invade a country he considers part of Russia’s sphere of influence or is generating a crisis to extract security concessions from the West.

Deputy Russian Foreign Minister Sergei Ryabkov said that Russia had “no intentions to attack Ukraine” and that the West needn’t fear “any kind of escalation.”

But many U.S. officials believe otherwise.

“We believe the threat of invasion is real,” White House national-security adviser Jake Sullivan said on “NBC Nightly News” after the talks concluded.

In a briefing with reporters after the Monday’s round concluded, Deputy Secretary of State Wendy Sherman, who led Washington’s side, said, “We’re just at the beginning, and we don’t know where all of this is headed quite yet.”

Mr. Ryabkov described the talks as “difficult, long, very professional, specific, without any attempts to embellish anything, to get around any sharp corners.”

But, he added, “the main questions are on hold, and we do not see the American side’s readiness to resolve them in a way that suits us.”Few expected much progress from Monday’s talks in Geneva, because the two parties came to the meeting with starkly different goals. The U.S. hoped to ease tensions over Ukraine by involving the Russians in talks over intermediate-range missiles in Europe and the scope of military exercises. Russia, meanwhile, said its aim was to remake the post-Cold War security arrangements in Europe by halting NATO’s expansion, curtailing the alliance’s ties with Ukraine and parts of the former Soviet Union, and severely restricting military deployments on the territory of the alliance’s Eastern European members.

The differences were displayed immediately in the separate news conferences the two sides gave after the nearly eight-hour meeting.

“We will not allow anyone to slam closed NATO’s open-door policy, which has always been central to the NATO alliance,” Ms. Sherman said.

Mr. Ryabkov countered, “It is very important that Ukraine can never join NATO in the future. We need iron, legal obligations, not promises, but guarantees…This is a matter of Russia’s national security.”

An important objective for U.S. officials was to gauge whether Russia was ready to pursue a reduction of tensions on terms that NATO might accept or was keeping open its military options to attack Ukraine.

“It doesn’t sound like they have come out with an answer to that question,” said Angela Stent, a Brookings Institution fellow and former U.S. national intelligence officer for Russia and Eurasia.

“Nobody walked out of the talks. The question will be how much Russia insists on demands NATO says are nonnegotiable.”

Dimitri Simes, president of the Center for the National Interest, a Washington think tank, said Mr. Ryabkov will convey his sense to the Kremlin on whether Russia can achieve its objectives with further talks.