Crawl Limitations

When CloudTrawl crawls a site we apply several reasonable limitations to prevent crawls taking up excessive resources: