Mary and Joseph did have children, therefore they were Jesus' half-brothers and-sisters. Matthew 13:56 reads: "Isn't his mother's name Mary, and aren't his brothers James, Joseph, Simon and Judas? Aren't all his sisters with us?"
 
So we know he had 4 brothers and at least 3 sisters (if there had been only 2, the word 'both' would have been used instead of 'all') who are accepted as half brothers and sisters. Because of this, it follows that Joseph and Mary were husband and wife, and that all those mentioned were alive when Jesus was.
 
Matthew 12:46 also refers to Jesus' mother and brothers: "While Jesus was still talking to the crowd, his mother and brothers stood outside, wanting to speak to him."

We must also presume that Joseph was already dead when Jesus was crucified. Joseph is never mentioned during Jesus' ministry; and when he was being crucified, Jesus asked John to look after Mary as if she were his own mother (John 19:26,27) At that time, none of Jesus' family believed in him and his message. Yet afterwards some did believe, and the letters written by two of them, James and Jude (or Judas) are part of the New Testament.