"It was the main subject - we might have to leave the country at any time," says Baldev Singh Bassey, who was 25 at the time of the speech and living in Wolverhampton.
"We didn't have suitcases [packed], but that was the only talk at the time.
"There was fear in our community, a lot of fear," says Mr Bassey.
"We didn't know what was going to happen tomorrow, if we were going to be kicked out [of Britain]."
Such fears were understandable.
In his speech, Powell had suggested the "encouragement of re-emigration" as a solution to what he saw as Britain's problems.
The MP also referred to a long-running campaign by Sikh busmen in Wolverhampton over the right to wear turbans on duty, which he said was a dangerous example of "communalism".
"When he made the speech the Indian community were really scared," Mr Bassey says.
"After that, we didn't know what was going to happen.
"People were talking about it in the pubs and suddenly the attitude of the [white] people living here was hostile."
One of the many industrious immigrants who came to post-war Britain, Mr Bassey arrived in the country from the Punjab in 1962, initially working at the Qualcast foundry and later on the buses for Wolverhampton Corporation.
"In the 1950s and early 60s, neither Powell nor any other prominent politician in the town showed opposition to the presence of Commonwealth immigrants," says University of Wolverhampton academic Dr Shirin Hirsch.
"They needed these workers for capitalism to function."
Despite this acceptance from the political class that immigration was necessary, Mr Bassey recalls that the prejudice he faced had been bad enough even before the speech.
"We were shouted [at] on the buses, we were spat at," the father of five says.
"I was shouted at, 'black bastard, go back to your country'."
Following the speech, Mr Bassey says, the hostility became even worse - a poll conducted by Gallup in the weeks afterwards found 74% of people agreed with what Powell had said.
Mr Bassey remembers being abused in pubs and in the street.
Walking home from work one night, he was accosted by two youths chanting "Enoch! Enoch!"
"We were scared," he says, "there was no doubt about it after that speech."
