Infancy Gospel of Pseudo-Matthew

The Infancy Gospel of James was well known by the early Church Fathers, especially Origen, and possibly Clement of Alexandria. Although this infancy narrative was condemned by Pope Innocent I in 405 and was considered apocryphal by the Gelasian Decree around the year 500, it was one source of the early church tradition of the perpetual virginity of the Virgin Mary, Theotokos, Mother of God. It was especially treasured by the faithful in both the ancient and medieval worlds. St Jerome even speculated that Joseph was also perpetually virgin, and that the Hebrew word translated as his sons were actually his cousins.
Should we reject the Infancy Gospel of James as apocryphal? Would it be fair to label them as heretical? Although this infancy gospel contains many fanciful stories, there are no obviously heretical teachings, nor are the actions by anyone in the Holy Family morally or doctrinally objectionable. Rather, they are similar to the many Jewish midrash Talmudic stories, many of which are similarly fantastic, but are instead cherished for their moral teachings. Indeed, they wouldn’t be midrash if they weren’t improbable. […]