Attention has increasing marginal value; as you remove distractions and drains-on-attention, each successive removal gives you more and more ability to focus and execute.
This is unfortunate, because it means that when one is first considering whether to begin removing drains-on-attention, there's not a great amount to be gained. A good metaphor is "if you are surrounded by ten screaming people, you wouldn't put that much effort into getting rid of just one, because there will still be nine, so what's the point?"
But in fact if there was only one screaming person, you would likely be motivated to try quite hard to get them to shut up.
The key insight is that you will never reach the situation where there's one last high-value drain-on-attention to remove unless you enact a policy of chipping away at the pile of drains-on-attention, even when the marginal value seems low.
Therefore:
Prioritize removing drains on your attention, even when it seems locally like too much effort for too little gain.