Gordon Brown is to urge all parties to show a "united front" against those opposing immigration out of prejudice.
In a speech in London, the PM will say it is right for politicians to talk about the issue and address people's concerns about immigration levels.
But he will say debate must be measured and talk that immigration is "out of control" plays into extremists' hands.
The speech comes as a number of leading politicians challenged the main parties to toughen pledges on immigration.
The parliamentarians, including Labour MP Frank Field, have written an open letter challenging all the main parties to toughen their manifesto pledges.
The British National Party (BNP), which wants a stop to all immigration, except in exceptional cases, and to deport all illegal immigrants, won its first seats in the European Parliament last year.
Some Labour and Tory politicians blamed the BNP's breakthrough on the failure of their parties to address concerns about the impact of immigration on jobs, housing and social services.
'Right to talk'
Labour says the points-based system governing the amount of people that can come to work in the UK based on different criteria - introduced in 2008 - is fair, flexible and has contributed to a fall in immigration.
But the Tories say the current system is not working and have urged an annual cap to be set on immigration to reflect the UK's economic needs.
Conservative leader David Cameron has said net immigration levels - the difference between those coming into the UK and those leaving - have been too high in the past 10 years and need to be reduced.
In his second major speech on immigration in the past six months, Mr Brown will say people have a right to talk about the issue.
"As politicians in the mainstream of British politics, we have a duty to listen and engage with them - because if we don't people will listen to whoever does," he is expected to say.
"When we talk of fairness, it is right to talk of immigration and address people's worries and concerns.
"The question of who comes to Britain, and what they have to do to earn that privilege - it is something that should be the subject of open and responsible debate.
"But how we conduct this debate is an important as the debate itself."
'Standing together'
Mr Brown will say there is a consensus among mainstream parties in favour of immigration as a positive force in British society and a necessary contributor to economic growth.
"So I call on all those in the mainstream of our politics to stand together in the coming weeks and present a united front against those who don't value the diverse and outward-looking Britain that we stand for."
In November, Mr Brown announced plans for a points-based test for permanent residence and citizenship and more recently pledged to tighten the rules on student visas. The Lib Dems have said immigrants should be encouraged to go to parts of the UK with specific skills shortages and which have the "will and resources" to accommodate them.
They have also called for improved border controls and for exit checks at all ports to be reinstated.
BBC News
