When Rosaries are Red
First, a caution: most if not all of the red beads we talk about are actually just accessories in paintings. So we are talking about something that is really -- literally -- just dots of red paint. There will always be some uncertainty about exactly what they are supposed to represent. Here's a fairly typical example by Joos van Cleve: Portrait of an Unknown Woman, 1527.
Second, we have to weigh the question of how realistic the painter was trying to be. Paintings are not photographs, and we know that many painters don't paint exactly what's before their eyes. For instance, we sometimes see rosaries shown with peculiar numbers of beads -- 28, 39, 16. It's very likely that at least some of these are the painter's version of "how many beads will fill up this space in the painting and still be big enough that you can see what they are."
Mostly the reason I think these rosaries are supposed to be coral is that red coral shows up in many documents as a material popular for rosary and paternoster beads. Coral was popular enough for paternosters that the Dominican Order had to make a rule in 1260 that friars could not carry paternosters of coral or amber (both luxury materials). Coral is also used for other decorative purposes, including adult jewelry and the peculiar table-top sculptures called "coral gardens" -- miniature landscapes made of valuable mineral specimens, precious materials and jewels, including coral "trees."
Mentions of carnelian and other reddish stones, by contrast, seem to be fewer. For carnelian in particular, apparently a lot of carnelian is more orange to yellow as it comes out of the ground: in modern times, it's routinely heat-treated to turn it redder and darker. I don't know whether this was done -- or whether it was possible -- in the Middle Ages.
Certainly you'd expect that some who couldn't afford the real thing would purchase cheap imitations of red coral instead. Again I'm speculating, but I wouldn't be at all surprised to find red glass used this way. (And perhaps more practical than the real thing -- genuine coral is rather soft for a semi-precious stone; glass is harder, though heavier). Nor would I be surprised to find wood or bone beads dyed red, or perhaps an "inferior" type of coral given the same treatment. That's certainly happening today: I think it's fair to say you can assume any "red" coral you buy these days has been dyed -- unless a very reputable supplier is certain that it wasn't. The genuine "gem quality" red coral is terribly scarce now and horribly expensive: a rosary made from it would probably cost $600 or more.
To the point, however: The other reason I think most of the red rosary beads in paintings are supposed to represent coral is that they are painted in the same style as other beads we _know_ are supposed to be coral. The closest and best examples are the many pictures of the Infant Jesus where he is _not_ playing with a rosary, but has a string of red beads around his neck.
A string of coral beads was, and perhaps still is, thought to be an appropriate gift for a baby over much of Europe, because coral has been regarded as having protective properties against the "Evil Eye." For that matter, a good-luck charm representing a coral branch, called a corno or "horn" in Italian, is still sold for adults in Mediterranean countries (although now it may be made out of plastic). The Virgin Mary here has simply given her baby a protective necklace as would any good mother. Here's a version by Joos van Cleve (Flanders, 1507-1540), the same artist who painted the secular portrait above.
The clincher for infant necklaces is when we notice how many of them have a pendant that is clearly a branch of native coral -- perhaps somewhat shaped and polished. In that distinctive color, and in that kind of context, it really can't be anything else.