Alexey Staroselets<p>Almost 22% of traffic to my website is generated by various web crawlers. More than a half of that belongs to <a href="https://mas.to/tags/MJ12bot" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MJ12bot</span></a>. I'm going to figure out if it truly obeys robots.txt, as it's said on their website.</p><p><a href="https://mas.to/tags/Majestic" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Majestic</span></a> <a href="https://mas.to/tags/crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>crawlers</span></a></p>