"So also the tongue is a small member," James 3:5 says, "yet it boasts of great things. How great a forest is set ablaze by such a small fire!"
Yes, it starts with little sparks. Unchecked, it can turn into an inferno. It starts with little nitpicks and henpecks, little nips at each other that turn into ravenous consumption (Gal. 5:15).
The pastor must not tolerate gossip. When he hears it, he ought to stop it and check it. When he hears of it, he ought to intervene and check it. Gossip often is disguised as concern. Some concern isn't gossip, but a lot is. Gossip is when we say things about someone we won't say to them. If you have a concern about someone that you aren't wiling to take to them, it should be forgiven or forgotten. Any serious concern—about sin, about error, about immodest associations—ought to be brought to the person for discussion.
A pastor will sometimes find himself the recipient of hearsay. What I mean is, he will occasionally receive reports of concerns about his character from anonymous parties delivered by parties willing to deliver them. There are few circumstances in which this might be acceptable. But in general, a pastor facing anonymous criticism will be asked to answer to ghosts. Very few things discourage a pastor more than anonymous criticism. More often than not, a wise pastor will need to say, "If someone is concerned about that, they need to bring it to me personally. As it is, I won't entertain it." The wise pastor will then personally consider whether the concerns are valid, anonymously generated or not, and "cling to what is good." But he is under no obligation to entertain the charges of nobody in particular.
"People are saying… " can be emotionally crippling to even the most secure pastor. Which people? How many people? One person, two people? Members? Regular attenders? Someone who likes me? Someone who doesn't? Suddenly every interaction with someone in the congregation is covered by a dark cloud. Is this one of the people who has concerns? Can I trust this person? Nobody can be trusted now because some anonymous somebody has not trusted the pastor with directly delivered concerns.
The pastor should give no harbor to about-talk that avoids to-talk. The gospel is a great creator of order in this regard. When people are reminded of the gospel constantly, and the church's teaching and programs are centered on the gospel, gossip finds itself in a hostile environment. "Gossiping" the gospel, then, creates an environment where sinners are more inclined to talk to and for each other rather than about and at each other.