Genesis
4:1 And Adam knew Eve his wife; and she conceived, and bare Cain, and
said, I have gotten a man from the LORD.
4:2 And she again bare his brother Abel. And Abel was a
keeper of sheep, but Cain was a tiller of the ground.
4:3 And in process of time it came to pass, that Cain brought of the fruit of
the ground an offering unto the LORD.
4:4 And Abel, he also brought of the firstlings of his
flock and of the fat thereof. And the LORD had
respect unto Abel and to his offering:
4:5 But unto Cain and to his offering he had not respect. And Cain was very
wroth, and his countenance fell.
It is seen in these texts that bringing an offering was something that was expected and that the Lord had given instructions to what was acceptable as an offering. The Lord did not accept Cain's offering because Cain did not follow instructions.
(Note: Jesus said in Matthew 15:9 "in vain they do worship me...". Both Cain and Abel were attempting to worship God when they brought their offerings before Him, but only one gift was acceptable to God because it was what He had asked for. Cain, on the other hand, did things his own way and his worship was "in vain". This raises the question for us, "Is our worship "in vain" because we are doing things our own way, or are we worshiping God the way He wants?")
For further study, read: Cain and Able Tested