It depends on the series, with Dragonball Z I absolutely skip filler episodes since there's a ton of them. Though in some cases it's worth watching all the episodes, for example Steven Universe (I know it's not an anime but a western cartoon) has some episodes that people might consider filler but they ended up making sense later on as the series went on.