Ms Harman went on to use the occasion of a speech to an Age UK conference to announce that the government is to bring forward a planned review of the default retirement age to 2010.
The default limit of 65 was established four years ago and prevented employers from forcing workers to retire at an earlier age except where it could be justified. Under the rules, employees who reach 65 can ask to stay on in their jobs, but employers have the right to refuse the request.
Were the default age to go, Ms Harman said that employers could still retire employees who were no longer physically up to their jobs.
Ms Harman argued that workplace ageist attitudes are costing the economy billions a year.
She said: We still have more to do to tackle the attitude that once you reach 60 you are just treading water until you become frail and dependent. This is important not just for those individuals concerned but for the economy as a whole. We have to banish the ageism in the workplace that costs an estimated up to £31 billion per year due to lost GDP.
Any plans to scrap the default retirement age may have the backing of smaller employers.
The Federation of Small Businesses (FSB) said it supported the government's decision to bring forward the review.
Research from the FSB has revealed that 60 per cent of small businesses do not think the government should set a default retirement age.
But the FSB called for a regulation that would protect those employers that need to retire staff because of ill health.
John Wright, the FSB's national chairman, said: Many small business owners have no intention of putting in place a blanket policy to retire their staff at 65 - they understand the valuable contribution and skills that older workers bring to the business.
In a recent survey by the FSB, 60 per cent of respondents employ staff over 50 years old and a quarter employ staff who are over 65, showing that small firms are flexible employers.
According to the FSB, almost eight out of ten firms reported that they do not use the default retirement age for their staff, while 76 per cent believe that retirement should be based on a mutual decision between the employee and employer.
Some 90 per cent of small businesses would consider an employee going into part time or flexible working, rather than retiring.
Mr Wright added: However, businesses need to be able to make decisions about their workforce without the threat of expensive tribunals from employees who are unable to work because of age-related issues. The ability to let someone go because of ill-health should be made sacrosanct for those employers.