Introducing the self-service Quant web crawler

 

We are excited to announce a new feature on the dashboard: the self-service web crawler! If you want to crawl any of your websites to create a static version, you can now do it easily through the dashboard. After crawling, the web pages and all digital assets (images, JavaScript, and CSS files) will be served from the Quant static edge.

Adding a crawler using the self-service option in the dashboard.

How to use the crawler feature?

If you are on the Enterprise Extra or Enterprise Advanced plans, you should see the Crawl settings in your dashboard. For other plans, contact us about getting access to the crawl functionality. Check out the Crawler documentation to see how to use the crawler.

You can also use the command line interface (CLI) to run the crawler. Check out the Crawl and migrate CLI docs and the crawler CLI tutorial for more information.

For the best performance, you should verify your domain (click on the domain in the dashboard for instructions). And, if you want to schedule your website crawls, contact the support team to talk through your options.

Why use the Quant crawler?

There are many reasons why you might want to use the crawler:

• Web Archival: If you have a website that you don’t need to update anymore and don’t want to maintain, you can crawl it and just host the static version for archival purposes. Think of it as your own Wayback Machine.
• Disaster Recovery: You can take regular snapshots of a site in case it goes down and you want a static copy to use as a failover.
• Faster Static Sites: If you want the public to access a completely static version of your site for better performance and don’t want to integrate your backend environment with Quant, you can just crawl it on a daily basis to get the content pulled in.
• Minimal Updates: If you have been using a content management or web framework but find you don’t update the site much anymore, you can create a static version of the site with the crawler and still edit the content using a simple WYSIWYG editor and code editor.
• Revision History: Each crawl will have its own revision so you can go back and see changes that were made easily and revert to an old snapshot as well.
• Creative Uses: Maybe you have two sites you want to merge together (assuming they don’t have the same URLs :). Or you want split up all your blog posts and host them on a different domain. Let your imagination wander!

Crawler process

The crawler process is simple. Have fun and let us know how we can help.

1. Add a crawler
2. Run it
3. Wait a bit
4. Check your content
5. Profit!

Crawler configuration list page in dashboard.

Crawler runs in the dashboard.

List of content in the dashboard after running the crawler.

Web page that was created after running the crawler.

About QuantCDN

Quant is a global static edge; a CDN & WAF combined with static web hosting. We provide solutions to help make WordPress and Drupal sites static, as well as support for all popular static site generators.