The bulls are semen tested each year to determine what percentage of "swimmers" are good anymore. Too many duds, or too low motility, and the bull is "retired". Usually they produce decent semen until they're 4 or 5, but sometimes 6 years old. It's not usually related to calf size, but more so that certain kinds of dud semen can result in a pregnancy that can't survive, so the cow miscarries and the farmer is out of luck for that cow/calf for the year (unless it's early and they can rebreed). If the defects are more so ones that prevent motility (can't swim) or prevent pregnancy, then sometimes a farmer will keep it on a bit longer since bulls produce obscene volumes of sperm and if all the hits result in viable pregnancy, then they can manage with a lower percentage of "good" sperm. So, depending on what sorts of defects are seen in the sperm, bulls have varying lifespans.

Sometimes cows will produce successively larger calves, which can cause issues. The calf size is largely dependent on genetics, so depending on what types of bulls/cows you're working with it can be more/less of an issue, but I don't believe the calf size is increased by older bull sperm. AI (artificial insemination) sperm is often selected based on the desired calf size, larger calves increase the risk of dystocia. When AI is used, the genetics of the cow and the bull are both taken into consideration.

*Source: currently in vet school and primarily worked with cattle.