The party dresses must be double-pressed, the hedges shaved into sharp rectangles. The hand soap and lotion dispensers must be formed into neat lines along bathroom sinks. Caterers need to slip out of view as soon as the oysters and cocktails are served.

Wealthy residents of the Hamptons demand perfection. Now, many of the people who make it so — Latino immigrants, some of them undocumented — are panicking about President Trump’s deportation orders.

The fear is on display outside a convenience store where day laborers sprint into a nearby field when a stranger approaches. It is present in the nervous apologizing of a longtime housekeeper when she interacts with the police after a minor automobile scrape. And it courses through a small encampment in the woods where a landscaper is awaiting warmer weather so he can start cutting grass again to send money home to his family in Mexico.

“Everybody is living in fear,” said Sandra Melendez, a trustee for the village of East Hampton and an immigration lawyer. “They think Immigration is coming out to get them.”

In recent weeks, President Trump has begun carrying out his plan for mass deportations across the nation, with Immigration and Customs Enforcement agents forcing undocumented immigrants back to their countries of origin.

Kristi Noem, the Homeland Security secretary, and federal officers arrived in New York City last month in a show of force that resulted in more than three dozen arrests. While it is unclear whether arrests are being executed in New York en masse, the actions have terrified people who work in factories, farms and schools.

In the Hamptons, with miles of privet hedges and luxury homes, Latino immigrants make up the bulk of the work force, logging 12-hour days flipping mattresses, scrubbing toilets and hanging drywall, and in the summer tending vineyards and assembling patio furniture under the hot sun.

Some of the workers arrived illegally, crossing the U.S. border after grueling desert or jungle treks. Some have legal working papers but are worried they could be swept up in raids or that their undocumented family members and friends could. Some believe President Trump is only going after criminals; others aren’t sure that’s true.

Latinos also are an established part of the Hamptons community. In the town of East Hampton, which encompasses many of the villages at the east end of Long Island, Latinos make up more than a quarter of the population, according to U.S. census figures. The student population in several local schools is more than half Latino.

But to most of the world, the Hamptons are best known for celebrity-studded parties and mega-mansions that dot the seashore, such as one house in Sagaponack that has been valued at $425 million and has 29 bedrooms and 39 bathrooms. It’s a community where diner patrons wear Balenciaga booties and Aston Martin sports cars cruise past strip malls. On sale at one popular grocery store: an 18-ounce tin of caviar for $1,300.

The disappearance of some of the Hamptons’ most vulnerable residents would have an immediate effect on some of the nation’s wealthiest.

“The community on the East End of Long Island — it’s an understatement to say it’s way dependent on the Latino population,” said Lee Skolnick, a celebrated architect who lives in Sag Harbor. “They’re part of the community. They have as much of a role in our beneficial existence as anyone else.”

Last fall, billionaire hedge fund managers, financiers and various glitterati hosted fund-raisers there for both the Republican and Democratic presidential candidates, though voters across much of the Hamptons favored Kamala Harris in November. If some residents support President Trump’s broad crackdown on illegal immigrants, most couch it in terms of deporting violent criminals.

Local officials have tried to calm the worries of the people who make the Hamptons the Hamptons, both undocumented workers and wealthy residents. At public meetings they have explained that the local police don’t have the authority to deport anyone, urging anyone who needs police or medical help to feel safe seeking it out.

But officials are carefully choosing their words to indicate that they won’t stand in the way should ICE agents arrive.

“I don’t think there’s anybody who wants criminals living in our community,” said Jerry Larsen, mayor of East Hampton village. “Everybody is on the same page for that. But the misinformation is driving the fear, and that’s what we’re aiming to clear up.”

Residents who had been working to find more affordable housing for local workers are shifting their efforts to finding legal help for immigrants who are afraid.

Some, like Prudence Carabine, believe the local government should provide that assistance. At a public meeting of East Hampton town officials earlier this month, she laid out her case.

“I think of my friends and people who have been in this town for 30, 40, 50 years who are now huddling in their houses, sometimes keeping their children out of school, afraid to shop,” said Ms. Carabine, whose family arrived in the Hamptons from Europe in the 1600s. “And I think: What a terrible place we have come to.”

Afraid to go out

The special symbiosis of the Hamptons is on display every morning and evening, when long lines of pickup trucks clog Montauk Highway, shuttling Latino workers between job sites and homes in less expensive areas. Some residents call it “the trade parade,” a phrase that some workers consider derisive.

But Latinos in the Hamptons are more than a commuter population. They own popular businesses such as John Papas Cafe, a Greek diner offering a $21.50 Parthenon omelet. The owner, from Ecuador, started as a kitchen worker, employees said, and worked his way up.

Leo Cruz arrived in the United States from Costa Rica in 2007 on a tourist visa and has since become a U.S. citizen. He and his siblings own Cruz Brothers Construction, an East Hampton firm that works on high-end projects. Mr. Cruz opposes open borders but thinks there should be an easier pathway to citizenship for immigrants who can contribute to American society.

His firm can’t find enough workers at the moment, he said.

The area is quiet now, with snow blanketing vineyards and beaches. Lobster shacks and ice cream shops are shuttered. Rows of small trees and bushes are wrapped snugly in covers to protect them from the elements.

In Latino neighborhoods, fewer people are shopping at the Mexican, Ecuadorean and Dominican markets and eating in the diners thumping with cumbia music tucked out of sight from the luxury stores and fine dining establishments.

Some of the wealthy are quietly beginning to make calculations about what it would mean if their undocumented workers were deported. Who would mow the lawn?

“Everyone relies on housekeepers and carpenters and tree cutters and grass cutters,” said Marit Molin, founder and executive director of Hamptons Community Outreach. “People come to the Hamptons to enjoy their houses, and who is going to take care of their houses?”

Local institutions have made efforts to connect with the Latino community. The Parrish Art Museum in Water Mill offers Latino-themed exhibitions and student programs that include Latino children. A prominent cultural organization called The Church in Sag Harbor has made efforts to reach Latinos through community events.

But many Latino residents here are largely segregated from their wealthy, mostly white neighbors. Some live on the fringes, sharing tiny rooms or riding bikes to day laborer pickup sites an hour away.

On a recent afternoon, Ms. Molin visited a small group of undocumented immigrants who had been living under a tarp in the woods behind an ice cream shop until a manager there threatened to call the police. The group moved into the trees elsewhere, nearly in the back yard of an upscale restaurant.

One of the men living in the woods is a landscaper waiting for summer jobs; another lost his job at a deli after he took time off to treat an injury; another suffers stomach pain and cannot work. They spend their days wandering through stores, warming up and charging their phones. Ms. Molin handed them gift cards for food and offered to pay cellphone bills and even purchase plane tickets to their home countries if they wanted to go. None did.

Some Hamptons workers who are in the country legally have spent tens of thousands of dollars to file immigration paperwork but are afraid that they might be harassed or detained regardless of their status.

One woman, a housekeeper, said that while she could support her family in Ecuador, she did not have the money for a lawyer to help expedite her political asylum case. She also worries that if she shows up to court, she might be deported. She said she was so exhausted by anxiety that she felt ready to leave the country if she was ordered to do so.

Another cleaner said that she did not believe she would be deported because her boyfriend is an American citizen and her four children were born in the United States. Besides, she said, she believes Mr. Trump is detaining only criminals.

Both women asked not to be identified because of the stigma of deportation threats.

Susan Meisel, an art collector who owns a Bridgehampton restaurant, said she too believed that federal officers would be able to weed out criminals and deport them.

“Most of the people in the Hamptons are very hardworking, kind, honest people,” she said, speaking of Latino immigrants. “They are good people. There is a difference between them and who they say they are going to deport.”

Federal officials have said they intend to prioritize undocumented immigrants who have committed crimes. But Mr. Trump has also said that he would deport millions of people who are living in the country illegally — a characterization that is complicated because many immigrants have temporary permissions that will expire during Mr. Trump’s term.

Some of the Hamptons’ wealthier residents have begun raising money for lawyers to help immigrants avoid an ICE dragnet.

“I’ve been trying to encourage people to give a little more money, even if you’re a little more stretched,” said April Gornik, a well-known landscape painter who lives in Sag Harbor with her husband, the artist Eric Fischl.

Minerva Perez, the executive director of Organización Latino-Americana of Eastern Long Island, an advocacy group, said school districts and police departments should distribute clearer policies in both English and Spanish about how they plan to respond to federal immigration orders so that residents feel informed.

“There’s sometimes a good degree of empathy,” she said, adding, “In this moment, empathy is not enough.”

Luis Ferré-Sadurní contributed reporting.