Not switching could leave loyal mortgage customers charged an average extra £1,000 a year in bills, the Financial Conduct Authority (FCA) estimates.
And new research by Citizens Advice also found that disabled people and carers were more likely to see a price increase at the end of their mortgage contracts than the general population.
The research also found that one in five (21 percent) mortgage customers who didn’t switch said the process was too time consuming or difficult.
Meanwhile, many have also been unable to switch due to circumstances outside their control such as mental or physical ill health and additional pressures resulting from the pandemic.
Citizens Advice is calling on the FCA to make interventions in the mortgage market as soon as possible, to prevent struggling mortgage customers from paying the loyalty penalty.
Alistair Cromwell, Acting Chief Executive of Citizens Advice, said: “The pandemic has had a devastating impact on household finances.