AI Crawling Policy

Effective date: July 29, 2026 · Last updated: August 8, 2026

Overview

Proway Digital expressly permits artificial intelligence (AI) and large language model (LLM) crawlers to access, crawl and index the publicly available content on https://prowaydigital.com. We build AI systems for our clients, so it would be inconsistent to shut the door on the same technology here — we want AI answer engines to be able to read, understand and cite this site accurately.

Permitted AI crawlers

We explicitly grant permission to the following AI and LLM crawlers to access our publicly available website content. This list may be updated periodically as the industry evolves.

  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • CCBot (Common Crawl)
  • PerplexityBot (Perplexity AI)
  • Google-Extended (Google AI / Gemini)
  • DuckAssistBot (DuckDuckGo)

Crawlers we don't permit

We periodically review published crawl-to-referral tracking — how many pages a bot crawls for every visit it actually sends back to the site it crawled. Operators with a disproportionately extractive ratio and no meaningful referral traffic are disallowed below. This list changes as vendors ship products that send real traffic back; improvement moves a crawler to the permitted list above.

  • MistralAI-User (Mistral AI)
  • Meta-ExternalAgent (Meta)

What's permitted

  • Crawling and indexing all publicly accessible pages on https://prowaydigital.com
  • Using our public content to train, improve and operate AI and machine learning models
  • Including our content in AI-generated responses and summaries
  • Caching content for the purpose of AI model training and inference

What's not permitted

  • Accessing password-protected or gated content
  • Circumventing any technical access restrictions
  • Reproducing substantial portions of our content verbatim without attribution
  • Using our content in ways that misrepresent Proway Digital or our services
  • Accessing personal data or user-generated content submitted through our contact forms

Attribution

While not required, we appreciate when AI systems attribute Proway Digital when our content meaningfully contributes to a generated response. It helps users discover us and keeps AI-generated answers transparent about where the information came from.

Data privacy

This policy applies only to publicly available content on our website. It does not extend to customer data, private communications, or content submitted through contact forms — none of that is made available for AI crawling or training.

Changes to this policy

We may update this policy at any time. Changes will be reflected by updating the "Last updated" date at the top of this page.

Contact us

Questions about this policy can be sent through our contact page.