Get Started
Ultimate guide to scaling feeds for 10K+ products - AdTribes
  1. Home
  2. Knowledge Base
  3. General Feed Tips
  4. Ultimate guide to scaling feeds for 10K+ products

Ultimate guide to scaling feeds for 10K+ products

Stores with large product catalogs (10,000 or more products) often face slow feed generation, timeouts, or memory exhaustion when creating product feeds. Product Feed Pro and Product Feed Elite are built to handle unlimited products, but you need to configure them correctly for your server environment and catalog size.

This guide explains how to improve feed generation performance for large catalogs. You will learn about batch processing, server limits, scheduled generation, filtering unnecessary products, and other performance considerations.

Why large feeds are different

When you have 10,000+ products, generating a feed in a single request can exceed:

  • PHP Memory Limits – Each product loads data (images, attributes, categories) into memory
  • Max Execution Time – PHP scripts often timeout after 30-60 seconds
  • Server Resources – Shared hosting may throttle long-running processes

The plugin uses batch processing to split feed generation into smaller chunks. Each batch runs separately, so you avoid timeouts and memory exhaustion. Furthermore, you can reduce the number of products in your feed by filtering out items you do not need for advertising.

How batch processing works

The plugin processes feeds in batches. Each batch handles a set number of products (the batch size). When one batch finishes, the next is scheduled automatically. This approach reduces memory use and avoids PHP timeouts.

From Product Feed Pro 13.5.7, the plugin chooses the batch size for you. See Adaptive batch sizing below.

On earlier versions, the plugin used fixed default batch sizes based on your total product count:

Product CountBatch Size
Up to 1,000200 products per batch
1,001 to 5,000300 products per batch
5,001 to 10,000500 products per batch
10,001 to 50,0001,000 products per batch
Over 50,0001,500 products per batch

Smaller batches use less memory and are less likely to hit execution limits.

Adaptive batch sizing

Product Feed Pro 13.5.7 and later time each batch and measure its memory use, then size the next batch from what your host just handled. This is a Product Feed Pro core feature, so it applies whether or not Product Feed Elite is active.

  • First run: The batch size starts from the product-count table above, capped at 500 products, then adjusts within a few batches.
  • Later runs: The plugin saves the learned size for each feed and starts the next run near it.
  • After a crash: The plugin sets a lower size limit for that feed, and the limit recovers gradually as later refreshes complete.
  • Constrained hosts: On a host with a low memory limit (for example, 128MB), the batch size shrinks until batches fit, so the feed completes slowly instead of failing every run.

In testing on an 18,201-product Google Shopping XML feed, adaptive sizing ran about 19% faster on a first run and 24% faster on a repeat run, with identical product counts. The saving comes from running far fewer batches (19 down to 2), which matters most on low-traffic stores where WP-Cron only runs on visitor hits and the wait between batches dominates total time.

From Product Feed Pro 13.5.9, the Manage feeds page also shows the batch size each feed will start from, and any limit set by a previous crash.

Setting a batch size manually turns adaptive sizing off for every feed on the site (Product Feed Pro 13.5.7+). For 10,000+ product catalogs, leave the batch size unset.

Configure custom batch size

Set a batch size manually only if your host has a hard limit you already know, for example a hosting plan that cuts requests off after a fixed time. If your feed gets stuck or times out with a manual size, reduce it.

How to change the batch size

  1. Go to Product Feed → Settings → General
  2. Enable Change products per batch number
  3. Enter a value in Insert batch size: (e.g., 500 or 250)
  4. Click Save next to the Insert batch size: field
Product Feed settings showing the Change products per batch number option and the Insert batch size field

Recommendations

  • Timeouts: Use a smaller batch size (e.g., 250-500) so each batch finishes before your server’s max execution time
  • Memory errors: Reduce batch size further (e.g., 100-200) to lower memory usage per batch

Suggested batch sizes by server type:

Server TypeSuggested Batch Size
Shared Hosting200-300
VPS or Cloud500-1000
Dedicated Server1000-1500

Always test after changing the batch size. If the feed still gets stuck, reduce it further. For more details, see Batch size configuration product feed.

Check and adjust server limits

Your hosting environment imposes limits on PHP execution time and memory. These directly affect feed generation.

max_execution_time

Each batch runs within PHP’s max_execution_time. If this is too low (e.g., 30 or 60 seconds), batches may be cut off before finishing.

  • Recommended: At least 300 seconds (5 minutes) for large catalogs
  • How to check: Go to WP Admin → WooCommerce → Status, and find the PHP Time Limit value from the Server Environment.
  • How to fix: Increase the limit in your server configuration by adding set_time_limit(300) to the wp-config.php file, or contact your hosting provider
WooCommerce Status page, Server Environment section showing the PHP Time Limit value

memory_limit

Feed generation loads product data into memory. When processing thousands of products per batch, a low memory limit can cause fatal errors or incomplete feeds.

  • Recommended: 256MB or higher for 10,000+ products
  • How to check: Go to WP Admin → WooCommerce → Status, and find the WordPress Memory Limit value from the WordPress Environment.
  • How to fix: Increase the memory limit in wp-config.php by adding define('WP_MEMORY_LIMIT', '256M'); or ask your hosting provider to increase the limit.
WooCommerce Status page, WordPress Environment section showing the WordPress Memory Limit value

The plugin logs a warning when available memory is below 128MB before a batch runs (if logging is enabled). This entry appears under log source woo-product-feed-pro in WooCommerce → Status → Logs. Use this to spot potential issues.

Crash-recovery and parked-feed warnings from adaptive batch sizing (Product Feed Pro 13.5.7+) log unconditionally to the same woo-product-feed-pro log source, regardless of the “Enable logging” setting. If your feed keeps getting parked after repeated crashes, check WooCommerce → Status → Logs even if you haven’t enabled logging.

Use scheduled generation instead of manual refresh

For large catalogs, avoid relying on manual refresh from the browser. Browser requests can time out or disconnect before all batches complete. Instead, use scheduled generation.

Refresh interval

  1. Edit your feed
  2. Go to the General tab
  3. Set Refresh interval to Hourly, Daily, or Twice Daily (Product Feed Elite also supports custom intervals)
  4. Save the feed

The plugin uses Action Scheduler to run feed generation in the background at the chosen interval. This is more reliable than manual refresh for large feeds.

Server cron (advanced)

For precise control or high-traffic stores, you can use a server cron job instead of the built-in scheduler. This is especially useful when:

  • You want feeds to run at specific times (e.g., off-peak hours)
  • Your site has low traffic, and WP cron may not run often enough
  • You need to stagger multiple large feeds

For step-by-step instructions, see How to refresh product feed using cron jobs.

Disable HTTP feed generation requests

When you select feeds on the Manage feeds page and choose Refresh from Bulk Actions, the plugin can trigger feed generation via HTTP requests. On large stores or constrained hosting, this can cause timeouts or incomplete feeds.

Disable HTTP feed generation requests forces feed generation to run only via Action Scheduler (or server cron). No HTTP requests are used to start or continue batches.

How to enable

  1. Go to Product Feed → Settings → General
  2. Enable Disable HTTP feed generation requests. The setting saves as soon as you turn it on.
Product Feed settings showing the Disable HTTP feed generation requests option

When to use:

  • Your feed keeps getting stuck or timing out when you refresh it from Manage feeds
  • You use a custom batch size (batch processing disables HTTP processing by default in that case)
  • You run feeds via scheduled refresh or server cron

For more details, see What Does “Disable HTTP Feed Generation Requests” Do?.

Understand the product counts the plugin reports

On a large catalog, the number you see in WooCommerce is usually higher than the number of products a feed can contain. From Product Feed Pro 13.5.9, the plugin reports the feed-eligible count instead of the raw catalog total, so the figures match what feed generation actually works with.

The feed-eligible total is the number of published products that are not variable parents, plus the published variations whose parent product is also published. It leaves out:

  • Variable parent products: Most channels give a variable product no row of its own. Only its variations get rows.
  • Variations whose parent is not published: A variation stays published when its parent is set to draft, pending or private, but no feed can include it.

To see the breakdown, go to Product Feed → Settings → System check. The Products table lists simple and other products, variable parents, published variations (split into parent published and parent not published), and a Total eligible for feeds row. A warning icon appears next to the variations whose parent is not published. This is the figure to compare against your feeds, and the one to share with support.

Each feed’s report uses the same base. Go to Product Feed → Manage feeds and hover over the help icon next to the feed name. The tooltip shows how many products the feed started from and how many ended up in the file.

If your feed holds fewer products than WooCommerce shows for your catalog, this is expected. Compare the feed against Total eligible for feeds, then check your filters and rules for the remaining difference.

On Product Feed Pro versions before 13.5.9, the System check and feed reports showed a raw total that included products no feed could contain. Do not use that figure to judge whether a feed is complete.

Filter out unnecessary products

Including only the products you need in your feed reduces processing time and file size. Use filters to exclude products that are not relevant for your channel.

Common filtering strategies

GoalFilter TypeExample Condition
Include only products that are availableIncludeStock status = instock
Include products from a specific categoryIncludeCategory contains [target category]
Include products above a certain priceIncludePrice greater than X
Exclude out-of-stock productsExcludeStock status = outofstock
Exclude products without imagesExcludeMain image is empty
Exclude very low-priced productsExcludePrice less than X
Exclude specific categoriesExcludeCategory contains [category name]
Exclude specific brands or tagsExcludeBrand/Tag contains [value]

Fewer products mean fewer batches and faster feed generation. For step-by-step guidance, see How to create filters for your product feed.

Variable products and variations

If you use variable products, consider:

These options are in the feed’s General tab and can significantly cut the number of products processed.

Performance considerations summary

FactorRecommendation for 10,000+ Products
Batch sizeLeave unset (see Adaptive batch sizing); for a manual value, see Suggested batch sizes by server type
max_execution_time300 seconds or higher
memory_limit256MB or higher
Refresh methodScheduled (hourly/daily) or server cron, not manual only
HTTP generationDisable if feeds get stuck or timeout
FiltersExclude out-of-stock, irrelevant categories, or low-value
Variable productsUse “lowest priced” or “default variation” when appropriate

Troubleshooting

Feed stuck processing
Turn on Disable HTTP feed generation requests (see above) and check Help, my feed won’t update or is stuck processing!. If you set a manual batch size, reduce it or remove it so adaptive sizing can take over (Product Feed Pro 13.5.7+).
Timeout errors
Increase max_execution_time to 300 or more.
Fatal error or memory exhausted
Increase memory_limit to 256MB or more. On Product Feed Pro 13.5.7+, adaptive sizing shrinks batches on constrained hosts instead of failing outright.
Fewer products than expected
Action Scheduler failures
Go to WooCommerce → Status → Scheduled Actions, open the Failed tab and review the errors, then increase your server limits.

Need help?

If you need help optimizing feed generation for your large catalog:

Product Feed Elite users: Create a support ticket to reach the AdTribes support team.

Product Feed Pro users: Open a topic in the WordPress.org forum.

Was this article helpful?

Related Articles

Complete Your Purchase
AdTribes WooCommerce Product Feed

The best WooCommerce product feed plugin

  • AdTribes Pty Ltd
    ABN: 40 675 636 816
Product
Resources & Info
Partner Sites
Rymera