During his very first podcast interview, with his former Step by Step daughters Staci Keanan and Christine Lakin on their rewatch pod Keanan and Lakin Give You Déjà Vu, Patrick Duffy gave a wide-ranging interview looking back on his successful television career.

While they touched on his ’80s favorite Dallas and ’70s breakout Man from Atlantis, they of course wanted to talk all things Step by Step, too.

The ’90s sitcom was like a “new era of The Brady Bunch,” Duffy, 75, shared of the show that followed him as Frank Lambert and Suzanne Somers as his wife, Carol Lambert, plus their blended family of six kids.

He recalled meeting the late Somers for the first time, briefly when they were both offered their parts on the spot without even auditioning. And from there, they just clicked … almost too well.

Keanan noted that some viewers have called the show “horny” since it ended in 1998.

“I could not believe that anyone would say that,” she said. “I thought, ‘What are they talking about? Our show is a very wholesome and sweet family show.’ ” But, she noted, “Now I start [re]watching the show … “

Duffy replied with a laugh, “It’s not like we demanded it. But we always pushed for the fact that the reason these two families got together is we, on a whim, on a vacation, got married after knowing each other for two days,” he noted of the plot. “And it wasn’t because we were both good at playing poker. So every chance that we could, we wanted to go upstairs and we couldn’t because we had a house full of kids, you know. And they wrote to that … and we played to that.”

However, it wasn’t necessarily a bad thing, he added. “We thought it was important. We could be appropriately attracted to one another and also good parenting models.”

Duffy noted that the show is currently quite popular in Europe, and that both there and in the United States, “people have come up to me and have said that watching that show was pivotal for them as a child, wanting, if they didn’t have one, a dad like that,” he shared of his character. “One young man came up and said, ‘I watched the show and ever since I started watching the show as a young person, I said, I want to be a dad like that. I wanna have a dad like that and then I wanna be a dad like that if I ever have children.’ So we fulfilled a certain function, you know, and I I’m sure they felt that way about Suzanne,” he continued.

While he did love the crux of the series, Duffy said there was one plot point he pushed back on.

“I had a disagreement [with the writers] when Suzanne wanted to get pregnant,” he recalled. “I thought it was irresponsible. I thought, we were a six-child family of a woman who was theoretically a beautician — but we never saw her work after the first two seasons — and a struggling contractor.”

“And,” he continued to Keanan and Lakin, “you guys had more fashion clothes than a runway model! And I thought, it sends the wrong message. And I told them that and they said, ‘Stay in your own lane.’ And I went back to my room.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“We had brilliant writers,” he added, noting that he’d never made a suggestion on Dallas and never made a Step by Step suggestion again. Plus, he said, Somers wanted the storyline. (Lilly, their daughter, joined the show in season 4.)

“It wasn’t so much that she wanted another child on the show,” Duffy explained. “She just thought she could act that out, being pregnant, and she could. It’s a funny thing to have her … doing all the Lucille Ball stuff. She was in heaven, it made her happy. So I was fine.”