For website owners

MignutiChatbotCrawlerBot

Our crawler reads websites so that our customers' chatbots can answer questions about their content. Customers add websites as a knowledge source, usually their own.

How the crawler behaves

• It respects robots.txt. It does not fetch disallowed paths.
• It only follows links within the same domain and does not execute JavaScript.
• It fetches pages one after another, not in parallel, and reads text only. It does not download images, videos or files.
• How often it returns is set by our customer: manually, daily, weekly or monthly.
• The text it reads is used only as knowledge for that customer's chatbot. We do not train AI models on it.

Block the crawler

Add this to your robots.txt:

User-agent: MignutiChatbotCrawlerBot
Disallow: /

Allow the crawler (Cloudflare)

You use Mignuti Chatbot and your bot cannot read your own website? Usually a bot protection is blocking our crawler, for example Cloudflare “Block AI crawlers”, “Bot Fight Mode” or “Pay per Crawl”. The Mignuti admin then says “Your website blocks our crawler” or “asks crawlers to pay”.

1. IP access rule (all Cloudflare plans): Security → WAF → Tools → IP Access Rules → 46.38.242.234 with action “Allow” for your website. According to Cloudflare, Bot Fight Mode does not trigger when an IP access rule matches first. It cannot be skipped by other rules.
2. Pro plan and above, additionally: a WAF rule with action “Skip” for requests whose user agent contains MignutiChatbotCrawlerBot, skipping Super Bot Fight Mode and the bot rules.
3. Pay per Crawl / AI Crawl Control: if our crawler appears in the list there, set it to “Allow”. Otherwise the IP access rule from step 1 helps.

Then, in the Mignuti admin under Knowledge → URL sources, click “Re-crawl now”. Still not working? Write to us and we will take a look.

Questions about the crawler or reporting abuse: Contact. The source IP may change; this page will then be up to date.