Architecture Overview

Developer/LLM-oriented map of how SSI is put together: the major components, what each does, the notable design decisions, and where the extension points are. For how to build add-ons and templates, see creating-addons and creating-custom-templates; for the stage system in depth see stage-system-reference and how-imports-work-stage-pipeline.

Summary — components at a glance

Unusual / interesting aspects (expanded at the end):

• Set-based, not per-row. Stages are bulk SQL over a staging batch table, not a per-product loop, which is the core reason SSI is fast at scale.
• Per-import disposable batch tables, intentionally left on disk after a run for post-mortem.
• Parallel stages via proc_open worker subprocesses, each owning a segment range; coordination is purely through the shared batch table (no IPC).
• wpdb-or-direct-mysqli switch to dodge per-connection memory overhead and WP filter chains when needed.
• Image dedup table (wp_ssi_image_lookup) with a BINARY(16) URL hash: constant-time lookups regardless of URL length, and INSERT IGNORE makes concurrent workers race-safe.
• Recursive-CTE hierarchy resolution done once in a stage finalizer, not per row.
• One stage plan for every runner: core stage names and priorities come from a locally built stage plan, so CLI, browser and scheduled runs and CLI workers register the same pipeline.

Request lifecycle

wp ssi <id> ──► superspeedyimports_run_import($args,$assoc)  [run-import.php]
                  ├─ stages  → superspeedyimports_run_parallel()   (fan-out orchestrator)
                  └─ <tool>  → $template->runUtility()/tools

Admin Run button ──► REST routes under superspeedyimports/v1  [super-speedy-imports.php]
                  import/setup → import/run per sequential stage;
                  per parallel stage: import/stage-setup, import/run per segment, import/finalize
                  → import/complete
                  import/stage-status: recovery probe after a timed-out request

Schedule ──► Action Scheduler, WP-Cron fallback  [functions/scheduled-imports.php]
                  re-entrant ticks, each running a bounded slice of the stage list

superspeedyimports_run_import builds the template instance (resolving base_template via superspeedyimports_resolve_template_class()), parses flags (--workers, --file, --show-sql, --delete-all, --verbose) and dispatches. All three runners take their stage list, with the import-mode preset applied, from getFilteredStagesWithParallelInfo().

Notable components

Orchestrator (run-import.php)

• superspeedyimports_run_parallel($template,$id,$workers) iterates the stage list from getFilteredStagesWithParallelInfo(). For each stage:
• setupStageAuxTables() (DROP+CREATE any stage-specific aux tables) runs first — in all execution modes.
• If the stage is not parallel → run in-process.
• If parallel and effective_workers === 1 → single-worker bypass: run in-process (no proc_open) but still call finalizeStage().
• If parallel and > 1 → build one worker command per segment (superspeedyimports_build_worker_command) and spawn via superspeedyimports_spawn_workers (proc_open); then finalizeStage().
• Segment sources: csv (byte-offset row ranges, from countCsvRows), batch / terms / hierarchies (ID ranges from captured max-IDs).
• finalizeStage($name) is the reduce step — runs once on the parent after workers finish (e.g. load-csv: dedupe UIDs, build post-load indexes, capture max-IDs; import-taxonomies: recursive-CTE term-taxonomy propagation). Called in every mode, so single- and multi-worker runs produce identical results.
• Parallel stages ($parallel_stages in import-template-abstract.php): load-csv(csv), attach-existing-images(batch), upload-remote-images(batch), attach-gallery-images(batch), fix-attributes(batch). Everything else is sequential.

Template hierarchy (import-template-abstract.php, templates/)

• Abstract SuperSpeedyImports_ImportTemplate; concrete SuperSpeedyImports_PostTemplate, SuperSpeedyImports_WooCommerceProductTemplate (extends PostTemplate), SuperSpeedyImports_CustomPostTypeTemplate. (The classes were renamed twice, from unprefixed names to SSI_* and then to SuperSpeedyImports_*; functions/template-resolver.php::superspeedyimports_resolve_template_class() maps both legacy spellings of a saved base_template to the current class, and only instantiates classes from templates/ or registered through superspeedyimports_register_templates.)
• Templates register stages (registerStage($name,$priority,$callback,$desc)), define the batch-table schema (setupBatchTables), the field-mapping UI shape, and getStagePresets() (import modes: full, only_insert, only_update, prices_only, taxonomies_only, images_only).
• Asset auto-discovery: a template’s JS and CSS are auto-enqueued when they share the basename of the file that defines the template class (MyTemplate.php → MyTemplate.js / MyTemplate.css), found next to that file or in its js/ or css/ subfolder.

Stage pipeline

Core (stages/core/): load-csv → import-taxonomies → match-existing → handle-slug-duplicates → update-posts → insert-posts → update-postmeta → insert-postmeta. Post (stages/post/): attach-existing-images, upload-remote-images, then upsert-relationships. WooCommerce (stages/woocommerce/, product imports only): process-variable-products, assign-product-type, attach-gallery-images, fix-attributes, and sync-variable-prices after process-deletes; variable-product parents are synthesised inside load-csv‘s finalize through superspeedyimports_after_load_csv_synthesise, not as a stage of their own. process-deletes runs last of the import stages and post-import-jobs (After Import) after everything. An import with a fetched remote source gets source-fetch, and an XML or JSON source gets xml-to-csv, just before load-csv. Utils (stages/utils/): save-posts, sanitize-terms, list-files, run only when named. Each stage is set-based SQL over the batch table; full per-stage detail in stage-system-reference.

Database layer

• Per-import tables wp_super_speedy_imports_<type>_<id> and wp_ssi_<type>_<id>: batch (the staging table; UNIQUE on _ssi_unique_item_id), terms, hierarchies (UNIQUE path_key preserves the full source-cell and level identity; parent_path_key resolves nullable parent IDs in one load-csv finalisation UPDATE), gallery_images, deletes, ambiguous_terms/_hierarchies, duplicate_unique_ids, validation_errors, acf_expanded. Left on disk after a run for diagnostics; recreated next run.
• functions/db.php wraps writes/reads with a superspeedyimports_should_use_wpdb() switch: wpdb by default, direct mysqli only inside load-csv’s row loop, where wpdb’s per-query overhead builds up memory. Stage SQL goes through superspeedyimports_query / superspeedyimports_get_results / superspeedyimports_get_var, which stamp each statement with the stage fingerprint /*ssi:<import_id>:<stage>*/ and refresh the run heartbeat; that is how the stage-status probe tells a running stage from a dead one. superspeedyimports_insert is not fingerprinted. The WordPress.org edition ships no direct-mysqli code, so it runs load-csv through wpdb as well; define('SUPERSPEEDYIMPORTS_FORCE_WPDB', true); in wp-config.php puts any edition on that same wpdb-only path: the direct-mysqli helpers are not loaded, so load-csv runs through $wpdb, and WooCommerce attributes go through the prepared $wpdb attribute engine. Use it to run the WordPress.org edition’s database path from a paid build, or where a direct connection is not wanted.
• History: wp_ssi_import_history + _stages via ImportLogger; metrics are numeric and summed across workers (outputAggregatedStageSummary).

Image pipeline

• SuperSpeedyImports_PostTemplate::uploadImage() downloads, validates MIME, and media_handle_sideload()s inside withImageSizesFilter() (which can suppress intermediate-size generation), then registers the result via superspeedyimports_image_lookup_insert().
• wp_ssi_image_lookup (global) maps url_hash (BINARY(16) MD5) → attachment_id. Lookups are O(1)-ish regardless of URL length; INSERT IGNORE on the unique url_hash makes parallel workers race-safe. superspeedyimports_image_lookup_cleanup_stale() (anti-join) prunes rows for attachments deleted in wp-admin, run at the top of image stages. This replaced the pre-2.55 original_image_url postmeta contract — never read/write that postmeta; use functions/image-lookup.php.

Taxonomy handling

load-csv parses each taxonomy column into hierarchy levels (the configured separator, e.g. >) and multiple terms (the | multi-term separator). Flat terms resolve to term_taxonomy_id during import-taxonomies; hierarchical paths are staged and resolved by a recursive-CTE bulk UPDATE in finalizeStage('import-taxonomies'). Slug collisions (two source values sanitising to the same slug) follow term_slug_collision_policy (alert vs merge); genuinely ambiguous paths surface in the ambiguous_* tables.

Hook points (brief)

Add-ons/templates extend SSI through actions/filters — full how-to in creating-addons. The notable ones:

Add-ons detect SSI by checking class_exists('SuperSpeedyImports_ImportTemplate').

The ssi_ prefix is gone. Version 2.79.0 renamed every hook, class and function from ssi_* to superspeedyimports_* for the WordPress.org directory’s prefix rules, and there are no back-compatibility aliases: an add-on still hooking ssi_apply_additional_options or checking class_exists('SSI_ImportTemplate') silently does nothing. Option keys, capabilities (ssi_manage_imports, ssi_run_imports), meta keys (_ssi_unique_item_id, ssi_updated), table names (wp_ssi_*) and wp-config.php constants were deliberately NOT renamed and still use the short prefix.

Why it’s built this way (interesting aspects)

• Bulk SQL over a batch table is the central idea: load the CSV into a staging table once, then every stage is a small number of set-based statements (INSERT … SELECT, UPDATE … JOIN) instead of N per-row WordPress calls. This is what makes it scale (no-op updates short-circuit; unchanged rows cost ~nothing).
• Parallelism is opt-in per stage and segment-based: the orchestrator splits the work into ID/byte ranges and runs wp ssi worker subprocesses; they never talk to each other, only to the shared batch table. The single-worker bypass keeps the same code path correct without spawning.
• Disposable per-import tables trade a little disk for debuggability — after a run you can inspect exactly what each stage saw.
• The image-lookup table is a deliberate move away from wp_postmeta lookups: a dedicated, well-indexed, collation-controlled table the plugin owns, so URL→attachment resolution stays fast and side-effect-free at scale.
• Hierarchy resolution as one recursive CTE in a finalizer avoids per-row term queries and is computed once per import regardless of worker count.
• One stage plan, built locally, gives the orchestrator, its workers and the browser and scheduled runners the same set and ordering of core stages.

Maintenance note: avoid citing line numbers in this doc — they drift. Names of classes, functions, hooks, tables and stages are the stable references.

Written for Super Speedy Imports 3.63.0.