PHP Generators and yield: Memory-Efficient Data Processing in WordPress

PHP generators and yield: memory-efficient data processing in WordPress

A generator is a function with yield: instead of building a full array and returning it, it hands out one value at a time and pauses until the caller asks for the next one. For WordPress jobs over large data sets (going through all products, exporting thousands of posts to CSV, processing a big import) this keeps memory flat: only the current page of data is in memory, not all of it.

Problem: A CLI script or an export loads all products with 'posts_per_page' => -1 and runs out of memory on a store with 50,000 products.

Solution: Wrap a paginated WP_Query in a generator, so the caller writes a simple foreach while the generator fetches page after page, and clear WordPress’ in-memory cache between pages.

A generator over all posts

<?php
function my_iterate_posts( $post_type = 'post', $per_page = 200 ) {
    $page = 1;

    do {
        $query = new WP_Query( array(
            'post_type'              => $post_type,
            'post_status'            => 'publish',
            'posts_per_page'         => $per_page,
            'paged'                  => $page,
            'orderby'                => 'ID',
            'order'                  => 'ASC',
            'update_post_term_cache' => false,
        ) );

        foreach ( $query->posts as $post ) {
            yield $post;
        }

        $page++;
        $more = $page <= $query->max_num_pages;

        // WordPress keeps every loaded post and its meta in the runtime object cache;
        // without this the memory still grows page by page
        wp_cache_flush_runtime();
    } while ( $more );
}

foreach ( my_iterate_posts( 'product' ) as $post ) {
    // only one page of products is in memory at any time
}

Two details matter. The order is fixed by ID, so pages do not overlap or skip posts. And wp_cache_flush_runtime() (WordPress 6.0+) clears the non-persistent object cache: generators keep the PHP side lean, but WordPress caches every post it loads, and in a long loop that cache, not your array, is what eats the memory.

If the loop changes the posts it iterates (for example, changes their status so they no longer match the query), paging by number skips posts. Then collect the IDs first, or query “the next 200 with an ID greater than the last one”.

Stream a CSV export

<?php
add_action( 'admin_post_my_export_products', function () {
    if ( ! current_user_can( 'manage_woocommerce' ) ) {
        wp_die( 'Forbidden', 403 );
    }
    check_admin_referer( 'my_export_products' );

    header( 'Content-Type: text/csv; charset=UTF-8' );
    header( 'Content-Disposition: attachment; filename="products.csv"' );

    $out = fopen( 'php://output', 'w' );
    fputcsv( $out, array( 'ID', 'Title', 'SKU', 'Price' ) );

    foreach ( my_iterate_posts( 'product' ) as $post ) {
        $product = wc_get_product( $post );
        fputcsv( $out, array( $post->ID, $post->post_title, $product ? $product->get_sku() : '', $product ? $product->get_price() : '' ) );
    }

    fclose( $out );
    exit;
} );

Rows go straight to the output, so the file can be any size. For very large exports, run the same loop in WP-CLI or a background job and write to a file instead: a web request has a time limit.

Pipelines

Generators compose: one can filter or transform another without building intermediate arrays.

<?php
function my_filter( iterable $items, callable $keep ) {
    foreach ( $items as $item ) {
        if ( $keep( $item ) ) {
            yield $item;
        }
    }
}

function my_map( iterable $items, callable $fn ) {
    foreach ( $items as $item ) {
        yield $fn( $item );
    }
}

$expensive = my_filter( my_iterate_posts( 'product' ), fn( $p ) => (float) get_post_meta( $p->ID, '_price', true ) > 100 );
$rows      = my_map( $expensive, fn( $p ) => array( $p->ID, $p->post_title ) );

foreach ( $rows as $row ) {
    // ...
}

yield from chains generators: yield from my_iterate_posts( 'post' ); yield from my_iterate_posts( 'page' ); in one function gives one sequence of both.

Things to know

• A generator runs only while it is iterated: calling my_iterate_posts() does nothing until the foreach starts.
• It can be iterated once. For a second pass, call the function again; iterator_to_array() works but loads everything into memory, which is what you wanted to avoid.
• Errors inside it appear in the foreach, at the moment the failing page is fetched.

NOTE: For WP-CLI commands over many posts, also print progress (WP_CLI\Utils\make_progress_bar()) and make the job resumable, so a failure halfway does not mean starting over. Avoiding extra queries inside such loops is covered in avoiding N+1 queries.