Liberal senator Cory Bernardi has suggested Prime Minister Malcolm Turnbull should take "all events into consideration" as he considers the future of his "captain's call" for Special Minister of State, Mal Brough.

Senator Bernardi's comments came as Mr Turnbull stood by Mr Brough amid increasing pressure from Labor for the Prime Minister to dump the minister from his frontbench over his role in the James Ashby affair.

Senator Bernardi is the first government MP to publicly concede that Mr Turnbull may need to consider Mr Brough's position in cabinet.

Mr Turnbull told question time on Wednesday Mr Brough had "answered the questions put to him" and "guilt or innocence is not determined by public denunciation".

And Mr Brough backflipped on remarks he made in an interview with 60 Minutes, telling the Parliament he did not ask former political staffer James Ashby to make copies of the diary of his then boss Peter Slipper.

Mr Slipper was the Speaker at the time and the Member for Fisher, a seat Mr Brough went on to win from Mr Slipper in the 2013 election.

The opposition devoted all of its questions in question time in the House of Representatives on Wednesday to Mr Brough's role in the Ashby affair and Mr Turnbull's refusal to sack him.

Mr Brough has been accused of misleading the Parliament after he said on Tuesday a question put to him by 60 Minutes had been selectively edited.

60 Minutes later released footage showing the only cutting that had occurred was to the "ums and ahs" of the interviewer Liz Hayes.

In that interview, Mr Brough admitted he had asked Mr Ashby to make copies of then Mr Slipper's diary.

But on Wednesday, when asked by the same question by Labor, Mr Brough told the Parliament "no", he had not made such a request.

In an interview with Fairfax Media, Senator Bernardi stopped short of calling for Mr Brough to resign or, at least, stand aside in the short term.

But in comments that underscore the pressure on Mr Turnbull over his choice for Special Minister of State - and which echo what many Liberal MPs are saying privately - Senator Bernardi said that "in the end of the appointment of ministers is always a captain's call".

"I'm sure the current captain will take all events into consideration in determining Mr Brough's future," he said.

"It is entirely a matter for the two Malcolms."

Asked if the affair was damaging the government, Senator Bernardi said: "That's for others to make a judgment on. The fact you are calling me suggests it is not helpful".

On Mr Turnbull's first day back in Parliament since returning from climate talks in Paris, Labor said the Prime Minister had to confront the issue, which has dogged the government since Australian Federal Police officers raided Mr Brough and Mr Ashby's home two weeks ago.

"Given the Special Minister of State misled the Parliament yesterday, and has misled the Parliament again today, why hasn't the Prime Minister sacked the Special Minister of State?" Deputy Opposition Leader Tanya Plibersek said.

"I thank the honourable member for her question. And she would be very well aware, as indeed would the shadow attorney-general be very well aware, that guilt or innocence is not determined by public denunciation," Mr Turnbull replied.

"Here or anywhere else. The Minister has answered the questions put to him. I thank the opposition for their interest in this matter."

Shadow-attorney general Mark Dreyfus later moved a motion to censure the Prime Minister for his atrocious judgment for giving the Minister the responsibility for government integrity and "for his complete and utter failure to show any leadership and sack the Minister of State".

The government used its numbers to vote down the censure motion.

Follow us on Twitter