When the Pharisees had accused Jesus of casting out demons by Beelzebul, they may have only been whispering among themselves, since v. 25 says that Jesus knew what they were saying because he perceived their thoughts. They weren't proclaiming this accusation before the crowds, and yet they will be judged for it because their words are evidence of the evil in their hearts. A tree is known by its fruit. A good tree produces good fruit and a bad tree produces bad fruit. Likewise the words you speak tell of the state of your heart. Your words are like the fruit of a tree telling everyone what kind of tree it is.
"And I say to you, that every careless word that men shall speak, they shall render account for it in the day of judgment. For by your words you shall be justified, and by your words you shall be condemned." This pronouncement has always scared me. So many idle words come out of my mouth every day. On the Day of Judgment shall I be held to a standard of perfection? But now looking at this passage I don't think Jesus is talking about perfection, but about whether you have true faith and a true heart. As Jesus said in v. 30, "He who is not with me is against me, and he who does not gather with me scatters." What do your words say about where you stand in relation to Jesus? Are you for him or against him? There is no middle ground. The question has to do with where your heart is, which has to do with the kinds of words that proceed from the heart. Idle words can be very telling. The Pharisees can't brush off their accusation against Jesus as just careless words. If they think he casts out demons by the power of the Devil, then it's obvious where their hearts are at. They are most decidedly against Jesus if they would make a mockery of common sense by calling the work of the Holy Spirit an act of devilry.
So the passage is still scary, but at least it makes more sense. Even idle words can indicate unbelief--or belief. But God will judge words only because they tell something about the core of the person who speaks them. God knows the heart even if people are able to conceal their true selves from others, yet on Judgment Day he won't just make a pronouncement upon us based solely on his omniscient reading of our souls. He will try us before angelic witnesses as in a courtroom, and he will bring forth the evidence of our lives. Words, deeds, the testimony of human witnesses. All of it focused on whether you are for Jesus or against him.