How Delayed Image Import Works Under the Hood
This guide explains the machinery: how images are hot-linked before they’re downloaded, how the round-robin download queue decides what to fetch next, and how the escalating-timeout retry ladder separates slow suppliers from dead URLs. Read the Quick Start Guide first if you just want it working — this is for understanding why it behaves the way it does.
The plugin’s life cycle for every image is: hot-linked → queued → downloaded (or retried up an escalating-timeout ladder → failed). Three systems, one per stage.
Stage 1: Hot-linking — real images before any download happens
The moment your import finishes, every product already shows its real image. No placeholder phase, no JavaScript. Here’s how.
When WordPress asks for a post’s featured image and there’s no local attachment yet, the plugin steps in via the standard filters (post_thumbnail_html, WooCommerce’s product-image filter, and the WooCommerce product-image block) and emits the supplier’s URL directly:
<img src="https://supplier-cdn.tld/products/rx200.jpg" class="dii-remote" loading="lazy"
onerror="...swap to local SVG..." />The visitor’s browser fetches the image straight from the supplier’s CDN — which is built for exactly that job — so your server does zero image work on the page view. Search engines see real images too.
Galleries get the same treatment with one extra trick: WooCommerce’s gallery template only renders slots for attachment IDs it can iterate, and an un-hydrated product has none. So the plugin injects synthetic gallery IDs (negative numbers it recognises internally, one per delayed URL) and intercepts their rendering to emit the remote URLs — the gallery looks and slides exactly as it will after download.
Three refinements make the hot-linking phase fast and resilient:
- Preconnect hints. For every supplier host referenced by posts in the current page’s query, a
<link rel="preconnect">is emitted in<head>, so the browser has the TLS connection warm before it hits the first<img>. - The
onerrorfallback. If the supplier refuses the browser’s request (dead URL, hot-link blocking), theonerrorhandler swaps in a neutral local SVG — visitors never see the broken-image icon. (Settings → Diagnostics can switch this to “show broken” while you’re hunting down which URLs fail.) - The hot-link blocklist. Some suppliers always 403 browser requests while allowing server downloads. Listing their hostnames in Settings skips the doomed request everywhere — featured images, gallery placeholders and the preconnect hints — and shows the SVG immediately; the background downloads still fetch those images fine. (The “show broken” diagnostic mode overrides the blocklist so you can inspect the failing URLs.)
One more thing happens on every hot-linked render: the post’s hit counter (dii_image_hits) ticks up, once per post per page view. This is the queue’s prioritisation signal — the images your visitors are actually looking at get downloaded first.
Because prioritisation is driven by real page views, the images that matter to your Core Web Vitals hydrate first. The long tail of products nobody has opened can take all night without costing you anything.
Stage 2: The round-robin download queue
Every tick (default: one per minute, WP-Cron), the background worker:
- Picks the next batch — up to 100 posts (configurable) that still have delayed meta, ordered by hit count descending. Most-viewed products hydrate first; the long tail of products nobody has opened waits its turn.
- Expands the batch into work items — one per featured image, one per gallery — and notes each item’s supplier hostname.
- Buckets the items by hostname and round-robins between the buckets. With the default “max consecutive same-domain” of 1, the worker downloads one image from supplier A, then one from supplier B, then C, then back to A — never a burst at any single host. If your catalogue spans three suppliers, each one sees roughly a third of the traffic, evenly spaced.
- Trips a circuit breaker on failing hosts — and remembers. If a host racks up 5 failures within a run (filterable via
dii_circuit_breaker_threshold), the worker stops trying that host and records a persistent backoff: 2 minutes on the first trip, doubling with each consecutive trip, capped at 1 hour (filterdii_host_backoff_max). Subsequent cron ticks skip the host entirely until the backoff expires, so a dead supplier doesn’t burn timeout slots that queued images from healthy hosts could use. A single successful download from the host clears the backoff and its trip count.
Work items are per image, not per post — a 12-image gallery becomes 12 separate items, so even one product’s gallery gets interleaved across suppliers rather than fired at one host back-to-back. Galleries hydrate one image at a time: interrupted runs resume exactly where they stopped, and a partially-downloaded gallery renders complete on the front end (local copies first, hot-linked remainder after).
This is the piece that protects you from the modern fast-import problem: importers like Super Speedy Imports finish so quickly that inline image downloading means thousands of requests hitting one supplier in minutes — which is how servers get rate-limited and banned by image providers. The queue turns that firehose into a polite, spread-out trickle, while stage 1 means nobody is waiting on it.
The bulk WP-CLI commands (wp dii load-all and friends) run through the same round-robin scheduler, and wp dii load-all --dry-run prints the image count per supplier host so you can see the spread before committing. The one difference: CLI runs ignore any active host backoff — you explicitly asked for a drain — though they still record trips on failure.
Stage 3: The escalating-timeout retry ladder
A download that hangs blocks everything behind it in the batch, so timeouts are deliberately aggressive at first and generous later. Every image — featured and each gallery URL alike — climbs the same per-URL ladder:
The timeout doubles on every failure. Attempt 1 uses your download timeout (default 3 seconds — plenty for a healthy CDN, and it means a batch of 100 can’t be wrecked by a few slow URLs). Each failure doubles it: 3, 6, 12, 24… capped at the maximum download timeout (default 30 seconds — effectively “no timeout” for an image; if it can’t arrive in 30 seconds it isn’t coming, and an uncapped call would block the whole batch behind it).
The retry spacing escalates too. After each failure the post waits attempt-count × the cooldown setting (default 60 s) before its next try — 1 minute after the first failure, 2 after the second, and so on. The cron respects that spacing, so a dying supplier isn’t polled every single tick forever, while a blip recovers within a minute.
Three strikes at the ceiling → failed. An image gets 3 attempts at the maximum timeout (with the defaults that’s 7 attempts total: 3, 6, 12, 24, 30, 30, 30). Exhaust them all and it’s marked failed — auto-retry stops, and it appears in the Failed Images tab with its reason (http_404, a cURL timeout, an SSL error…), attempt count, when it failed, a live preview of the remote URL, and one-click Retry now / Dismiss actions. wp dii retry-failed (optionally --host= for one supplier) re-queues failures with a completely fresh ladder once you’ve fixed whatever was wrong at the source.
The effect: dead URLs fail fast and cheap early on, slow-but-working suppliers succeed on the patient rungs, and only genuinely unreachable images end up needing a human — with the evidence attached. Attempt counts are tracked per URL (in one compact meta entry per post), so a 10-image gallery with one bad URL hydrates its 9 good images normally while the bad one climbs its own ladder.
A 3-second first attempt is not impatience — it is what stops one hanging supplier from consuming the entire batch window. The ladder then gives genuinely slow hosts up to 30 seconds on later rungs.
After a successful download
- The image is checked against every previously downloaded image by URL — if an attachment for that exact URL already exists, it’s reused instead of re-downloaded (100 products sharing one image = one file on disk, one attachment).
- A normal WordPress attachment is created with all thumbnail sizes generated immediately,
_thumbnail_id/_product_image_galleryare set, and any deferred alt / title / caption / description meta is applied. - The
delayed_*meta is removed, and once a post has nothing left to hydrate its hit counter is cleaned up too. From this point the post is indistinguishable from one imported the slow way.
A stampede guard sits underneath all of this: a per-post lock ensures that when two processes want the same image at the same moment (say, an admin opens the product while the cron is fetching it), one downloads and the other waits briefly and then uses the result — the same URL is never downloaded twice concurrently.
The knobs, in one table
| Setting | Default | Governs |
|---|---|---|
| Tick interval | 1 minute | How often a batch runs |
| Batch size | 100 posts | Work per tick |
| Max consecutive same-domain | 1 | Round-robin strictness |
| Image download timeout | 3 s | The retry ladder’s starting timeout |
| Maximum download timeout | 30 s | The ladder’s ceiling; 3 failures there = failed |
| Retry spacing base | 60 s | Wait before retrying, × the attempt count |
| Admin-bar badge | on | Pending-count bubble while a backlog exists |
| Hot-link blocklist | empty | Hosts that skip the browser request |
| Force HTTPS | on | Rewrite http:// supplier URLs before download |
Developers can also filter dii_http_timeout, dii_max_timeout (the ladder ceiling), dii_circuit_breaker_threshold, dii_host_backoff_max (cap on the escalating per-host backoff, default 1 hour), dii_failure_spike_threshold, dii_image_url (rewrite URLs before download), dii_after_hydrate (react to each completed image) and the dii_image_failed action (react when an image exhausts its ladder).
Comments