First of all, Jesus says, it's obviously okay in certain instances to violate what is holy if it serves a pressing need, as David and his men ate the consecrated bread while they were fleeing from King Saul in 1 Samuel 21:1-8. Nothing in this account hints that David was guilty of any sin, so it must be that his and his men's need for food in that desperate situation took precedence over whether the food they ate was "properly" lawful.
Secondly, the Law itself teaches that the priests break the Sabbath all the time. The priests have to serve in the temple, right? That's work, right? You're not supposed to work on the Sabbath, right? Therefore the Law teaches that the priests can break the Law. Hmm, it sort of hints that keeping the Sabbath isn't the be all and end all of our entire existence.
So human need trumps persnickety law-keeping, and even the Law teaches that work must be done on the Sabbath. Now Jesus goes on to argue his third point, which is that his own presence on this earth changes our entire understanding of the Law. Jesus' arrival signals that all the Old Testament shadows are giving way to something greater, that he is the true temple, priest and Sabbath rest.
And that means, jumping forward to Jesus' fifth argument, that he is the Lord of the Sabbath. The Old Testament Sabbath law is merely a foreshadowing of the true Sabbath rest that Jesus would bring: "Come to me, all who are weary and heavy-laden and I will give you rest" (remember this from our previous passage?). Jesus came to give us rest from our labors under the Law. If you have kept the Old Testament Sabbath in view of that messianic fulfillment, then you understood the point; but if you have made the Sabbath a task-master to further oppress the weary, you have perverted it.
Which is why Jesus quotes from Hosea 6:6 in his fourth point, "I desire compassion and not sacrifice." The Sabbath rest is about having compassion on needy and desperate, upon sinners who seek to be free from the burden of their guilt. By extension the Sabbath should be an occasion for having mercy upon any kind of need, whether it is a need like David's, or the need of priests to do their holy work, or even the need of the disciples to breakfast on their journey with Jesus. That the Pharisees go around using the Sabbath as a club to bludgeon people and build up their own self-righteousness demonstrates just how far astray their blindness has led them from the truth.