I just completed wonder that was India by A L. Bhasham and I ve read many books on Indus and Aryan culture. The thing is we ( as a scholars) concretely don't know how Indus civilisation died and how and exactly when Aryans/vedic people started vedic culture? And there is no proof that inhabitants of Indus even got in direct contact of vedic people. There's a time gap of around 1 century millenna b/w those two things.

And there is no mention of castes in Vedas; scholars have clear notion that purusha sukta which talks about caste is later addition to Rig Veda coz putusha suktas Sanskrit is different from the Sanskrit of Rig Veda. And secondly scholars didn't get any profound mention of caste in other vedic texts mainly in principle Upanishads. And even after some 2000 years of vedic period Buddhists and jaina text (initial ones) do not cite caste practice in vedic tradition, they criticise it for rituals and sacrifices. Cite me any any buddist or jaina text contemporary of 1st Buddhist council, even in the text of maghda dynasty, which was contemporary of Budha and jaina traditions particularly don't talk about caste.

So your or your grandfather's claim that vedic seers were in minority is a sheer foul and proves nothing but utter stupidity.