Security forces have been battling with Taliban fighters for more than a week in the northwest Malakand division, but troops have largely avoided entering the militant stronghold in Swat Valley.
But residents of the main city Mingora began leaving Tuesday when officials lifted a curfew in some places to allow civilians to relocate.
One resident who left with his family told VOA that militants completely control the town, have taken up fighting positions in large buildings and placed landmines along key routes. He said the Taliban did not prevent people from leaving during the suspended curfew.
"We are leaving our homes from Mingora. Some people going to Mardan, some are going to Peshawar. The government gave us only three or four hours, so that is why the people of Mingora are leaving Mingora," he said.
The information minister for the North-West Frontier province predicted as many as 500,000 people may flee the Swat Valley. He said the government is preparing temporary camps to house those who have nowhere else to go.
Pakistan's army has not said if it is planning an assault on Swat, which would be the final blow for a much criticized peace deal with the Swat Taliban. Provincial officials had reached a deal with the militants that allowed the establishment of Islamic courts in parts of the northwest in exchange for the disarming of the militants, something they did not do.
But even staunch defenders of the agreement have in recent days harshly criticized the Taliban for failing to hold up their end of the bargain.
Bashir Bilour is a senior member of the Awami National Party, which played a key role in setting up the deal. He defended the recent military action by citing a long list of grievances.
"After we discussed implementing Sharia, the other side smashed our schools, slaughtered our men and hanged our people. They stopped girls from attending school and women from going to the market," he said.
Since last week, Pakistan's army said it killed more than 100 militants and secured three districts that earlier had been controlled by the Taliban. Fighting continues in some of those places, but overall, the military said its operations have gone smoothly.
In Swat, there is concern among residents who have already been through two previous military operations that failed to dislodge the militants. The resident of Mingora who spoke to VOA said people are opposed to the Taliban, but are equally wary of military operations that have killed civilians in the past.
"People want operation against the militancy, but not against the innocent people. This is the big problem for us, the Swati people, also for Buner," he said.
He said residents worry the military's attempts to dislodge the Taliban could fail once again.