Social Media Platforms

Twitterbot checks shared URL's posted in Twitter and records preview content, link integrity and meta data. Twitterbot respects robots.txt, but will only check robots.txt a few times a day, so any updates won't be immediate.

Gnip is the full Twitter firehouse which powers thousands of social listening, influencer and media monitoring platforms. If you get traffic from Twitter and want to get more, then this is the one you need to ensure isn't get blocked.

Pinterest is a pinboard-style social photo sharing website that allows users to create and manage theme-based image collections"pins" such as events, interests, hobbies, and more. The Pinterest crawler aggregates your content and displays it as pins on Pinterest, The crawler is configured to automatically rate limit concurrent requests and does respect robots.txt. If you don't want your content aggregated don't whitelist it.

LINE is a Japanese 'Free' messaging App aimed at consumers. It's crawler collects information based on what their users share. The Bot respects Robots.txt, and can be seen getting data such as favicons and link contents, likely for embedding in messages. Artists and brands can also push and share content with users too.

The LinkedIn crawler visits your site and gets content such as homepage grabs, or image grabs for use when links are shared in the LinkedIn platform.

This Facebook crawler makes a series of HEAD requests when it visits. If you have a Facebook presence for your business you may wish to allow this bot.

The Facebook crawler scrapes the HTML at URL's that are shared with other FB users, to gather, cache and display info about the content on Facebook such as the title, description, and thumbnail image. The crawler can also be triggered by any of Facebook's social plugins on the URL.The Facebook Crawler scrapes the HTML of a website that was shared on Facebook via copying and pasting the link or by a Facebook social plugins on the website. The crawler gathers, caches, and displays information about the website such as its title, description, and thumbnail image.

Facebook Catalogue allows e-commerce website owners to import their products into the Facebook Catalogue Manager. There are several ways to add products to your catalogue. You may be able to import them to Facebook Catalogue Manager directly from the platform so you can continue managing your inventory on your e-commerce platform. Your products will sync automatically with Facebook. This crawler pics up inventory details from your website, such as product images, for use in the Facebook ecosystem.

The Bytedance spider is seen eminating from both TikTok and Toutaio services from Bytedance. If you have no use for either platform in your business then we would recommend you do not allow this bot.