Photo: Rodolfo Martinez/Bravo

Southern Charm - Season 5

Thomas Ravenel‘sSouthern Charmcostars have all but cut ties with him since production on season 5 wrapped, a source tells PEOPLE.

An insider close to the cast says the 55-year-old reality star has been ostracized from his circle of friends over the past few weeks, with former friends likeShep Rose,Cameran Eubanks,Craig Conover,Austen Kroll, Chelsea Meissner andNaomie Olindoputting their focuses elsewhere.

“He doesn’t have friendships with the [main] cast anymore — everyone is done,” the insider explains. “They’re disgusted with him and the way he and Ashley treated Kathryn this season. His eye-opening behavior and the drama surrounding those allegations caused a huge wedge in his friendships.”

“There’s too much dishonesty there, too much manipulation,” the source continues. “They’ve all just been keeping their distance.”

WATCH: Southern Charm’s Thomas Ravenel Accused of Sexual Assault, Allegedly Paid Accuser $200,000

Representatives for Bravo had no comment, but the source adds that only Whitney Sudler-Smith and J.D. Madison — who recur on the Bravo reality show —have remained in contact with Ravenel, as has Dennis. The two share 50/50 custody of their daughterKensie, 4, and sonSaint, 2½.

“Best birthday that I have ever had… and also the only surprise party anyone has ever held for me!” Dennis tweeted. “Thank y’all for making my day so special!”

Ravenel was not invited to the party, the source says — though it appears unlikely he would have even attended.

“Thomas been like a hermit, staying close to home at his Brookland Plantation,” reveals the source. “He’s completely distraught over the allegations and what’s happened with the show.”

“He’sbounced back from scandal before, but this is much harder now that he has kids,” the source continues.

Rodolfo Martinez/Bravo/NBCU Photo Bank via Getty

Southern Charm - Season 5

Ravenel has had a history of trouble with the law, dating back to 2007 when — as South Carolina’s treasurer —he was indicted on federal cocaine distribution charges.He was sentenced and served 10 months in jail, and went to rehab as well.

Since then, Bravo and Haymaker (the production company behindSouthern Charm) have launched an investigation into the claims, saying they “take allegations like these very seriously.”

“Once all the information is gathered and carefully reviewed, appropriate action will be taken,” they said.

No update into the investigationhas been provided yet, but Ravenel did skip the season 5Southern Charmreunion,which taped in June.

RELATED VIDEO:Southern CharmKathryn Dennis on ‘Natural Chemistry’ with Thomas Ravenel: ‘There’s Common Ground from the Past’

When the reunion aired in July,Jacobs said that the allegations have changed Ravenel.

“I have seen a new Thomas,” she explained. “I mean, it’s humbling, of course, and now he’s [like] ‘Let’s stay in, let’s watch movies, let’s cook dinner, let’s just stay under the radar right now.’ And I’ve just seen a much softer, more sensitive, kinder, more supportive boyfriend. You know, everyone wants to be with you when you’re on top. Who’s with you when you’re down there? And I’ve told him, I’ve just said, ‘I’m not leaving.’ ”

“You’re exactly where Kathryn was,” said Conover, with Eubanks adding, “You’re lying to yourself, Ashley … What does it say about a man who talks poorly about the mother of his children? … He’s been a not nice person.”

“I feel bad for you,” Dennis said. “I’m watching you be me years ago. I’m watching it unfold right before me.”

source: people.com