Delayed Image Import

Delayed Image Import takes the images out of your import. Instead of downloading every product photo while the import runs, your importer writes the supplier’s image URLs into two post-meta fields and moves straight on to the next row. The import finishes. Afterwards, a background queue works through those URLs at a sensible pace and turns each one into a proper WordPress attachment in your media library.

That is what makes the import fast. On a real catalogue the images are the expensive part, not the data. A product row is a line of text your database swallows in microseconds; each of its photos costs a separate HTTP round trip out to the supplier’s server, and then thumbnail generation for every image size registered on your site. A 50,000-product import means at least 50,000 of those, done one after another, while everything else waits on them. Take them out of the loop and the import goes back to doing what it should be doing – writing rows to the database.

And that is what stops you getting banned. Image hosts and CDNs do not distinguish between a legitimate bulk import and a scraper; they see thousands of requests from one IP address in a few minutes and they rate-limit you, start returning 403s, or block your server outright. Suddenly half your catalogue has no images and your supplier will not talk to you any more. The background queue is deliberately polite instead: it rotates between supplier domains so no single host is ever hit repeatedly, and when a host starts failing it backs off automatically for two minutes, doubling each time up to an hour, rather than hammering it.

It speeds up every importer, not just ours. The whole contract is two post-meta fields, so anything that can write post meta can use it – WP All Import, a WooCommerce CSV import, a custom feed script, whatever you already run. Point your featured-image column at delayed_image_url and your gallery column at delayed_gallery_image_urls, switch off that importer’s own image handling, and it inherits the same deferred downloads, the same politeness and the same background queue. With Super Speedy Imports there is nothing to map at all: deferring switches itself on the moment both plugins are active.

Before the local copy exists, the page uses the supplier’s URL directly. If that source remains reachable and permits hotlinking, shoppers and search engines see the real photograph while the queue works. A supplier outage or hotlink block can affect the image until it has been downloaded locally.

# Check the current backlog and throughput
wp dii status

# Drain everything now instead of waiting for cron
wp dii load-all

# Watch it work in real time
wp dii status --watch

Why image downloads become the bottleneck

Slow importers never exposed this problem, because they trickled image requests out over hours by accident. Super Speedy Imports is built to import or update a million products in about 22 minutes, and at that speed the images stop being a cost and start being a firehose: tens of thousands of requests aimed at one supplier’s server inside a few minutes, from a single IP address. That is the point at which the import is no longer the bottleneck – the supplier is – and the point at which a CDN decides you are scraping it.

So the faster your importer gets, the more you need this. The import itself gets faster again, because it stops downloading images at all. And the image requests get slower and politer, which is the half nobody expects: small batches, rotated across supplier domains, backing off automatically from any host that starts failing.

There is one more piece of ordering that matters on a live shop. Every time a not-yet-downloaded image is rendered, that post’s view counter ticks up, and the queue drains most-viewed first. The products your customers are actually browsing get their local copies tonight; the long tail nobody has opened can take until the morning without costing you anything.

Features

Documentation and development

Frequently asked questions