He said building by residents between wooded areas was a “crime” that had resulted in blocking escape routes.
Mr Kammenos was confronted by angry locals as he visited areas devastated by fires east of Athens this week.
At least 83 people died and search teams fear they will find further victims as dozens more are missing.
Dozens of rescue workers searching a cliffside in the seaside town of Mati on Thursday found more human remains.
Many of the victims were caught close to the cliff as the flames spread rapidly, whipped up by winds as high as 120km/h (75mph). Relatives have been providing their DNA to help identify the bodies of the victims.
Some 60 people are still being treated in hospital, 11 of them in intensive care.
‘You left us alone’
Arriving in the village of Mati, where many people died, Mr Kammenos was surrounded by residents who accused him of abandoning them.
“You let people burn for no reason. You left us to the mercy of God,” a woman told the minister, who was accompanied by the local mayor and the head of the military,
One man told the BBC he had reached the fire station to ask for help, and found firefighters unaware of the blaze.
Another resident complained that people had been advised to dive into the sea to escape, when most of the elderly were unable to reach the sea in the first place.
Many of the survivors were saved by the sea, but dozens of victims were unable to reach the water.
Mr Kammenos, a combative figure who leads the nationalist Independent Greeks, rejected accusations that the government had failed to protect its citizens.
The actions of some residents had closed the roads to the beach, he argued.
“This is a crime from the past,” he said. “This coast of Athens, all these properties, the majority are without a licence, and they have occupied the coast without rules.”
Rafina Mayor Evangelos Bournous said one in four houses had been completely destroyed by the fire, while 50% more were partially damaged.
A satellite map of the area revealed the scale of the fire damage, showing that more than half the 12,760 acres affected were residential areas.
The mayor defended the role of the military, arguing that if the army had not been able to stop the fire spreading, it would have razed the entire area.
With many people still unaccounted for, a harrowing account emerged from the head of the Greek forestry federation, Georgos Papadias, who said firefighters had focused their resources on a separate fire to the west of Athens, leaving the East Attica coast exposed.
A scene of chaos with no evacuation
Mr Papadias had been in the village of Kokkino Limanaki, on the southern edge of Mati when the fire struck. It was like a horror movie, he said, with injured and elderly pleading for help, and he was not being in a position to do anything,
No alarm was sounded to warn residents or tourists, and no evacuation was organised from the coastal resorts of Maki and Kokkino Limanaki and the village of Neos Voutzas just inland. But once the fire crossed the main road through the area, Marathon Avenue, residents had no means of escape.
Greek newspapers complained that residents could have no confidence in official organisations as civil protection was non-existent, and no lessons had been learned since the last devastating fires to hit Greece, in 2007.
Mr Papadias spoke of cars blocking the roads as he tried to escape. He said he had lost neighbours to the disaster and a friend was in a grave condition in hospital.
Among the missing are twin nine-year-old girls Vassiliki and Sofia Philippopoulou, who were with their grandparents near Mati at the time of the fire. The girls’ father said he thought he had seen them being rescued in footage broadcast by Greek TV.
However, the children in the video turned out to be different girls.