Japanese Foreign Minister Koichiro Gemba said Friday that Mr. Lee's visit was "utterly unacceptable," and would have a "strong impact" on South Korean-Japan relations.
"In protest, we are temporarily pulling out Ambassador Mutoh from the Japanese Embassy in South Korea. Additionally, we passed the message to the South Korean ambassador that we were given no choice but to take suitable measures against it."
Lee on Friday became the first ever South Korean president to visit the islands, which have been a source of tension between the two countries for decades. Seoul officials downplayed the visit, saying it was not meant to stir up trouble.
The islets, known in Japan as Takeshima and in Korea as Dokdo, have been under effective South Korean control since the end of Japanese colonial rule after World War II.
Last week, Japan renewed its claim to the islands in its annual defense report, prompting South Korea to summon a Japanese diplomat in protest.
The two countries share strong trade and tourism ties and work together on numerous foreign policy issues of mutual concern. But a series of territorial and historical issues have also plagued the relationship.
Some information for this report was provided by AP, AFP and Reuters.