Simon Cowell, lower left, and the Silvermans, upper right, with the Hamptons as the backdrop.

Simon Cowell, lower left, and the Silvermans, upper right, with the Hamptons as the backdrop.

Simon Cowell’s torrid sex scandal with Lauren Silverman has turned the normally publicity shy Hamptons on its ear. Not since the 2008 divorce of Christie Brinkley and Peter Cook has the exclusive playground of the rich been so divided.

True Hamptonites hate publicity and even the whiff of scandal. But the Cowell-Silverman sexcapade has blown the lid off the tony beach community.

“It’s really beyond the pale,” said one Hamptons resident who asked not to be named. “Can you believe they really carried on like that?”

“For a minute, I thought we’d get though the summer without a major incident. But this has been like an earthquake,” said another. “It really makes us uncomfortable, because we’re all under the spotlight.”

Andrew Silverman’s family is well known in upper-bracket New York City, where the Hamptons draws the lion’s share of its residents. His father, Allen Silverman is a wealthy real estate developer, who would could match Donald Trump dollar-for-dollar.

Allen founded property company The Andalex Group in the 1980s and named it after sons, Andrew and Alexander, who both work in the family business. It owns millions of dollars in property, mostly Manhattan apartment and office buildings.

Every year since their marriage in 2003, Andrew and Lauren have spent summers at the $4 million Bridgehampton estate owned by his family.

The Silvermans were accepted among the community’s wealthy elite, if not the scattered celebrities who live there. Alec Baldwin, Paul McCartney and Billy Joel are all neighbors, yet they rarely, if ever, socialized with them.

Enter Simon.

He met the Silverman’s at an exclusive resort in Barbados in 2006. It’s hard to see what the “X-Factor” mogul had in common with them, but it’s possible Lauren caught his eye from the get-go. In any event, they became friends.

In the Hamptons, celebrity worship is infectious. The only thing worse than being rich, it’s often said in the tony enclave, is being a rich nobody.

As such, the wealthy, but faceless people who live there often go out of their way to court celebrity friendships. Such as it was with Andrew and Lauren. Having a celebrity pal like Simon added a certain cache that set them apart from the merely wealthy.

So much so, that Andrew was blind to the growing attraction between his wife and the music mogul. When Lauren revealed she was pregnant by Simon, it blew the lid off the resort.

“Simon opened up a world for them, no question, but at a terrible price,” said a Bridgehampton neighbor who views celebrities as crass interlopers.

“I just wish this whole thing would go away,” she lamented. “It really gives all of us a bad name.”