with a grain of salt; discounting half the story
Used when you don't fully believe what someone says, treating it as only half true. Often used in the phrase 話半分に聞く (take with a grain of salt).
I always take his boasting with a grain of salt.
You'd better take that rumor with a grain of salt.