"We have changed tack once and we can change again", he said.
Conservative backbencher Philip Davies, who on Thursday became the third Conservative to declare publicly that he has submitted a letter of no confidence in Theresa May's leadership, told BBC Newsnight he had had his "differences" with the chief whip.
The Times's Sam Coates led the way on this yesterday and followed it up last night with a highly explosive allegation that Smith openly admitted to "a rival chief whip" that he meant to break Parliamentary convention and order pairs to be broken because the Government was on a knife-edge.
Tory whips had agreed to "pair" her with party chairman Brandon Lewis - meaning he would also skip the vote so Ms Swinson would not be punished for having a baby.
She referred to research published in May by the Equality and Human Rights Commission, showing that more than 10% of mothers interviewed said they had been dismissed or treated so poorly they felt they had to leave their jobs.
Labour has called for the chief whip and party chairman to resign if they can not fully explain what happened.
Conservative Peter Bone (Wellingborough) said he was "very concerned" to hear that a pairing was broken. The Sun reported that up to five MPs were approached and that Smith demanded to know why one of them ignored his instructions to break the whip. Integrity and honesty are fundamental to our democracy.
Jo Swinson is on maternity leaveYUI MOK PA
Reacting to the report, Ms Swinson said: "Well". Then breaks promise to scrape through vote?
The Lib Dems have called for Mr Smith to make a statement in Parliament and "be accountable for this mess".
SNP Westminster leader Pete Wishart asked in the Commons for a full inquiry into the breakdown of pairing.
The twist is they broke the long-term pair with Jo Swinson "by accident", and while they acknowledge Julian Smith did "explore and possibly instruct" shorter-term pairs to be broken, none actually were.
A Conservative spokesman added: "We have apologised for the fact that a pregnancy pairing arrangement was broken in error this week".
Over the course of the Parliament, 66 pairing arrangements have been broken, with 52 of those at the hands of the Opposition, according to Tory sources.
A debate on proxy voting, which would end the need for pairing, was shelved a couple of weeks ago due to other business and will be rescheduled for September, says Dr Thompson.
Conservative whips did consider breaking short-term pairing arrangements - those for MPs who need to be away for another engagement, sources acknowledged.
There are now no formal maternity or paternity arrangements in Parliament, with informal leave arranged at party level.