Ask your Symfony questions! Pay money and get answers fast! (more info)

How to get more than 1,000 rows back from Sphinx? Symfony

Using Symfony 1.4, and the [[LINK href="http://www.symfony-project.org/plugins/sfSphinxPlugin/0_0_9"]]sfSphinx plugin[[/LINK]] for search.

I am up against the Sphinx 1,000 row limit ([[LINK href="http://sphinxsearch.com/info/faq/#row-storage"]]explained here[[/LINK]]).

I need to be able to get 50,000 rows back.

I opened:

/etc/sphinx/sphinx.conf

and I changed this line:

max_matches = 50000

to 50,000, as you can see.

My PHP code is, sadly, still only returning 1000 rows. What else do I need to change?

I see there is a setLimits() method as part of the sfSphinx plugin:

[[LINK href="http://trac.symfony-project.org/browser/plugins/sfSphinxPlugin/trunk/lib/sfSphinxClient.class.php?rev=28182"]]http://trac.symfony-project.org/browser/plugins/sfSphinxPlugin/trunk/lib/sfSphinxClient.class.php?rev=28182[[/LINK]]

But I tried calling it and setting the limit to 50,000, but I still only get 1,000 records back, on a query where I should be getting back 8,000 records. You can see my use of setLimits near the bottom of this method:

private function sphinxSearchForExport($query, $valuesToFilter) {
if($this->sphinxIndexName=='') return array();
//$this->query = ($this->query!='')?implode('* ', explode(' ', $this->query.'*')):'';
$options = array(
'limit' => 50000,
'offset' => ($this->page - 1) * $this->pageSize,
'weights' => array(100, 1),
'sort' => sfSphinxClient::SPH_SORT_EXTENDED,
'sortby' => '@weight DESC',
);
$this->sphinx = new sfSphinxClient($options);

if(!empty($this->filters))
{
foreach ($this->filters as $filter=>$properties)
{
$comparator='=';

if(is_array($properties)) //Check for a multi array.
{
$comparator=(isset($properties['comparator'])?$properties['comparator']:'='); //Comparator, default =
}
else
{
$filter=$properties; //$this->filters is a list (a,b,c)
}

if(!empty($valuesToFilter[$filter]))
{
if ($comparator == 'bet') //Range Comparison
{
$min=(empty($valuesToFilter[$filter]['from_s'])?0:$valuesToFilter[$filter]['from_s']);
//Use max integer 2^32
$max=(empty($valuesToFilter[$filter]['to_s'])?4294967296:$valuesToFilter[$filter]['to_s']);

$this->sphinx->SetFilterRange($filter, $min, $max);
}
else{
$this->sphinx->SetFilter($filter, array($valuesToFilter[$filter]));
}
}
}
}

$this->sphinx->setLimits(0, 50000);
$res = $this->sphinx->Query($this->query, $this->sphinxIndexName);
$this->arrayOfMatches = array();
$this->arrayOfMatches = $res['matches'];

$this->pager = new sfSphinxDoctrinePager($this->class, $options['limit'], $this->sphinx);
$this->pager->setMaxRecordLimit($this->maxRecordLimit);
$this->pager->setPage($this->page);
$this->pager->init();
return (!empty($this->pager)?$this->pager:false);
}



It's in $this->arrayOfMatches where I expect to see more than 1,000 rows, but I'm not getting it.

Answers (1)

2011-03-03

Loban Rahman answers:

From the link you provided:

"Second, there's also a 3rd argument to SetLimits() API call that also defaults to 1000 and must be raised. (With SphinxSE, use "maxmatches" option.)"

Then I checked:

http://sphinxsearch.com/docs/manual-0.9.9.html#api-func-setlimits

Which shows that there is indeed a 3rd argument (and 4th!). So do this:

$this->sphinx->setLimits(0, 50000, 50000);

But before you do that, find out if this method is a symfony wrapper to the actual sphinx api. If it is, then make sure that this wrapper actually accepts and forwards this 3rd parameter to the main sphinx api. If not, you'll have to modify the plugin to include this.