He said that meetings of country heads should only be for "urgent and immediate concerns", when major events in the summit happened "every 30 minutes".
The President has said he is against the recreational marijuana, but is open to legalizing medical marijuana.
But in an interview with reporters, he said he was not serious when he said the statement.
"Even presidents are entitled to a joke", Panelo told reporters.
He said that if Duterte's remark defies logic, it must be a joke.
Duterte later told reporters he was "of course" joking, trying to keep a rambling speech about how much he disliked the experience of attending this year's Association of Southeast Asian Nations (ASEAN) summit from getting too boring.
"Like his so many sexist, misogynistic and violent statements that set into motion killings, human rights abuses and the rabid disrespect of women, President Duterte makes these pronouncements without any sense of responsibility and accountability".
Marijuana, or cannabis, is illegal in the Philippines and is not a stimulant.
Opposition Sen. Risa Hontiveros described the President's joke as "callous" and "cruel". "Sometimes you say I'm a misogynist when I joke but that's my style". But Senator Aquilino Pimentel III said he did not believe that Duterte was using marijuana.
The journalist who leads a news site that has battled Philippine President Rodrigo Duterte turned herself in on Monday to face what she termed a "manufactured" tax fraud case that carries the threat of a decade behind bars.
Duterte, however, said he would push for stricter regulation to ensure that medicinal marijuana would be offered only to those who really need it.
Rappler added that Duterte has previously, without claiming to be joking, to the use of, an extremely addictive and risky opioid believed to have played a role in almost half the opioid overdose deaths in the United States in 2017.
Panelo insisted that Duterte had "never" taken marijuana in his life while admitting that he never discussed the matter with the President.
But he repeated the joke later on Monday when asked about extending martial law on the island of Mindanao, he said: "It might be during the cabinet meeting, after the pot session, so we can think clearly", The Guardian reported.
"There is a disconnect between what the president admitted to do and what the president said he will do to those who use drugs".