Workers lost their jobs after the industrial revolution, true. Instead, that work force now works in services and as a result, leads a healthier life. Don't get me started on depression and other mental health issues that are associated with the corporate world but as a whole, fewer and fewer people have to do the awful jobs that were common in the pre-industrial age.
You're missing one point - wealthy people get their wealth by providing services and products to the masses (well, mostly). At the same time, they employ those masses which get paid and afford the services and products manufactured. If you lay everyone off and replace them with robots, the unemployed population will no longer afford to buy any products manufactured by those robots, leading to a collapse in demand and subsequently supply.
Mechanical robots aren't that much of a threat outside of war zones. Well, not for the next 50 years probably until their dexterity surpasses humans at a guess. The problem now is AI which could take over the last human feature which is it's mind and intelligence. Why pay 300 workers to do invoicing, stock trading and other desk jobs when you could have 1 AI platform doing it all for you, paying only for maintenance and electricity? This is the biggest problem we're facing now.