Welcome, Guest
Username: Password: Remember me

TOPIC: Performance with Large Datasets

Performance with Large Datasets 2 years 9 months ago #1889

  • Biking Zone
  • Biking Zone's Avatar
  • Offline
  • Junior Boarder
  • Posts: 22
  • Thank you received: 2
  • Karma: 0
It appears that there are some inefficiencies when working with large datasets. My implementation started slowing down dramatically once there were a few thousand records. Now I have over 40,000 records and I am getting "out of memory errors". In most of the data world this is NOT a large dataset!

After looking at the code I noticed why this was happening. I found the problem in the following example of code in /administrator/components/com_jomclassifieds/models/adverts.php line :100
$db->setQuery($query);
$items = $db->loadObjectList(); *** Loads the ENTIRE dataset even though only 10 records might be needed
$this->_total = count($items);
if(!$this->limit) {
  return $items;
} else {
  return array_slice($items, $this->limitstart, $this->limit);
}	
So I replaced the above code with the following code:
//Get the paged number of records using LIMIT
$query .= " LIMIT $this->limitstart, $this->limit";
$db->setQuery( $query );
$items = $db->loadObjectList();		
//Get the record count
$db->setQuery( "SELECT count(*) FROM #__jomcl_adverts AS a $where" );
$this->_total = $db->loadResult();		
return $items;

At first glance it may appear inefficient because it's making two calls rather than one, but it works 1,000s of times faster because it only gets what it needs (i.e 10 records at a time). Both calls take ~.1 seconds in MySQL on 40,000+ records.
I would highly recommend that this be implemented for getItems() in the following files:
/administrator/components/com_jomclassifieds/models/adverts.php
/components/com_jomclassifieds/models/adverts.php
/components/com_jomclassifieds/models/search.php
The administrator has disabled public write access.

Performance with Large Datasets 2 years 9 months ago #1899

  • Super User
  • Super User's Avatar
  • Online
  • Administrator
  • Posts: 2756
  • Thank you received: 291
  • Karma: 40
Dear Customer,

My sincere apology on this and sure we will fix this in our immediate next version.

Thanks.
The administrator has disabled public write access.

Performance with Large Datasets 2 years 9 months ago #1905

  • Biking Zone
  • Biking Zone's Avatar
  • Offline
  • Junior Boarder
  • Posts: 22
  • Thank you received: 2
  • Karma: 0
Two more files that need to have getItems() be updated:
/components/com_jomclassifieds/models/user.php
/components/com_jomclassifieds/models/userads.php (A $where variable will need to be created for this one.)
The administrator has disabled public write access.
Time to create page: 0.195 seconds