Really. That was about the biggest challenge, given the lack of modern medicine, plagues, the violence of medieval life, etc
If you did make it to adulthood, your biggest problems would have been keeping your estate profitable, in the context of primitive agricultural techniques, and having to deal with constantly providing men and materiel to your monarch, in whatever wars he was pursuing. Being a medieval noble was very expensive, and many were in constant debt, aside from having to lead their men off to war, which usually involved long periods of time away from home and family. War meant that life was dangerous, too.