I am trying to import / update 10000+ products into Magento 1.9 from a given XML file (no Magento import format). This is the current workflow:
In a cron job I am downloading a XML file every night at 1 am. This file includes the 10000+ products. So I am going through the XML file using the XMLReader class from PHP. To figure out if a product already exists, I am first loading the product by sku, like this and after that I am checking if it was loaded, like this:
$product = Mage::getModel('catalog/product')->loadByAttribute('sku', $sku);
if(!$product) {
// create the product
} else {
// update the product
}
Of course this is not the only code, but maybe its my bottleneck? The code is running forever the way I have it written right now. Any ideas on how to quickly (within 3-4 hours maybe) import 10000+ products?
I can't use magmi or the import workflow because the import has to run at night via cron job.
Thanks!
First, regarding the product entity storage in the database: everything is stored using the EAV pattern, which means you can add arbitrary attributes to your product entity and they're treated as being roughly equivalent to another column on the product table. However, they're actually stored across a set of tables which then have to be joined up whenever you request a product model (so you're joining across several large tables, 10000+ times).
Luckily there's a way around this in the form of flat tables - these are tables which take the custom attributes and create a single table which incorporates all of the product attributes (which are configured with the "Use in Product Listing" option). Your bottleneck is probably that you're not using this product flat table (enable it in System configuration under the Catalog section). It should re-build the flat table automatically after being turned on, if not you'll need to reindex manually from Index Management.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments