Why should you never turn down a speech or speaking engagement? I say this and it may be a bit controversial. Some say that it is crazy! What if it is not an ideal audience, or if it is too far away and a lot different variables. Btu for most people running business, writers, politicians, and experts, any time you speak it is an opportunity to sell your ideas- whether it is your business, service, or platform. And it is essentially free advertising!
It does take time but you need to look at it from that perspective. Of course there are exceptions. If you are a conservative Christian minister and the National Manboy Love Association wants you to be a keynote speaker, you may want to turn that opportunity down. Btu in other situations, if there is a chance for you to reach people who could be potential customers and prospects, speak to them.
This is what is different in the modern world. You could say in the past that it is not worth your time flying to far off places to speak even if you are getting paid well. That is not an excuse anymore as you can offer to speak for them through free Skype video from the comfort of your home or your office. That is why you cannot turn down speaking opportunities. Offer to speak at reduced costs through online video as it will reduce traveling costs and save your time.