Labour candidate Michael Jones decided he could not share a platform with the British National Party (BNP).
It led to the Conservative and Liberal Democrat candidates also pulling out.
The BNP said it was a "disgrace". Bognor Regis Civic Society said the audience on Saturday night voted to send a letter of rebuke to all three.
Mr Jones said: "I could not in all good conscience share a platform with the BNP and provide them with respectability and credibility for their racist views which they crave.
"Since my very beginnings in politics, I have always believed in no platform for fascists and I could not betray that."
A Conservative Party spokesman said its candidate Nick Gibb had wanted to take part in the debate, to explain how the BNP was exploiting immigration.
He said: "He was disappointed the Labour candidate withdrew at the last moment.
"Our policy nationally is not to debate when the major parties are not present."
He added that Mr Gibb was now trying to arrange other public meetings which might not involve the BNP.
'Gross bad manners'
Liberal Democrat candidate Simon McDougall said that once the other two main parties had withdrawn, he felt the event had lost any value.
"I believed the Conservative MP should have sat on the platform and defended his record, but he followed his party protocol and withdrew.
"I was outside the event on Saturday night. I spoke to several members of the public that came out, and I explained the situation
"I did turn up. I took some of the flak. I did take flak, but I had many people who respected the decision."
BNP candidate Andrew Moffat said all three candidates had snubbed the electorate, displayed "gross bad manners" and treated voters with contempt.
"It's about the right of the electorate to meet their candidates and to cross-examine them and put them under scrutiny, but these people didn't have the courage.
"They didn't want to discuss multiculturalism, the Afghan war or the EU.
"It's not racist to say we are full up. We were never consulted about immigration by the Labour Party.
"And if that's true, why didn't they debate it? We are very cross. We are very annoyed."
'Anger and upset'
Hugh Coster, deputy chairman of Bognor Regis Civic Society, said: "We are are fiercely non-political, so when we invited the candidates, we invited all six."
After the three main candidates withdrew, the event went ahead with the three other parties fielding candidates in the constituency.
Mr Coster added: "There were 270 people in the audience who were very angry and upset.
"We pride ourselves on being the voice of the people. We arrange public meetings to enable them to question people in authority.
"The actions of these candidates have destroyed the opportunity for free speech."
The candidates for Bognor Regis & Littlehampton are: Independent: Melissa Briggs; UK Independence Party: Douglas Denny; Conservative: Nick Gibb; Labour: Michael Jones; Liberal Democrats: Simon McDougall; British National Party: Andrew Moffat
BBC news
