Question: However, I am getting too many invitation these days and I need to start saying "No" in order to get work done. However, I am lacking clear rules except for "A dominates B in time-slot C". Any ideas from your side?

Answer 1: I pretty much say 'no' if

a) conference is too unimportant

b) people don't pay for my travel (invitation on my own account are cheap :)

c) I don't see how I would profit from attending this conference at all

d) I simply want to get work done

A good thing is also to promote good students to give talks for you, e.g., at workshops/seminar/etc.