Support for LDAP paged queries about sympa

[ikedas]

Closed as fixed.

[ikedas]

• For example on OpenLDAP, by default, page size limit seems equal to "hard size limit" (size limit of total count of results enforced by server), and not to be enforced by server (cf. Admin manual).
• On Active Directory, page size limit is enforced by MaxPageSize LDAP policy of server, and its value is lower than MaxResultSetSize (size limit of total count of results enforced by server) (cf.
  Doc on technet).

I guess MaxPageSize is the challenge on Active Directory, because its default is slightly low (1000) and it is enforced by server. If several other directory servers also have the same problem, I think introducing paged search seems useful.

[bertaviader]

I was trying to find this link: https://sourcesup.renater.fr/tracker/?aid=7369 but it seems is down. Do you have the code? I need it to find a solution about pagination.
Thanks,
Berta

[lchanouha]

Hello,
Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

    $mesg = $db->do_operation(
        'search',
        base   => "$ldap_suffix",
        filter => "$ldap_filter",
        attrs  => [@attrs],
        scope  => "$source->{'scope'}",
+       sizelimit => 10000
    );

An option (global or per list) would be better.

[rpet]

Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

That is useless when the LDAP server enforces a limit. It also does not scale well.

So supporting paged queries is the way to go IMHO.

[rpet]

Replacing a standard LDAP query with a paged one is pretty straightforward. This is an excerpt of a Perl script I made to query my Active Directory infrastructure:

...

my $ldap = Net::LDAPS->new( $server, onerror => 'die' ) or die "new: $@";

my @controls = ();

# Creates a control for pages of 100 items
my $ctrl_page = Net::LDAP::Control::Paged->new( size => 100 );
push @controls, $ctrl_page;

my @ldap_args = (
	base => $base,
	filter => $filter,
	attrs => @attributes,
	callback => \&processObject,
	control => @controls
);

my $cookie;

while (1) {
	my $mesg = $ldap->search( @ldap_args );

	$mesg->code && die $mesg->error;

	if ($mesg->entries < 1) {
		print STDERR "NO RESULTS\n";
	}

	# Retrieves the page control cookie
	my($resp) = $mesg->control( LDAP_CONTROL_PAGED ) || last;
	$cookie   = $resp->cookie || last;

	# Sets the cookie for the next search
	$ctrl_page->cookie($cookie);
}

[lchanouha]

Instead of modifiying LDAP server soft limit, or di paged control (my hard limit is higher), I just added sizelimit arg on Sympa/List.pm

That is useless when the LDAP server enforces a limit. It also does not scale well.

So supporting paged queries is the way to go IMHO.

Yes you're right this is the best way. sizelimit is a quick workaround at least for openldap for queries between soft and hard limit, if no size.pr is specified, if I understand well openldap documentation.

[ikedas]

If anyone would like to submit a pull request to make this feature request a reality, it would be appreciated.

In my opinion, it is desirable to be able to use the paged operation transparently from the do_operation() method of Sympa::DatabaseDriver::LDAP. To achieve this, directory servers that do not support the paged operation should be considered, such as below:

my $db = Sympa::Database->new('LDAP', ...);

if ($db->__dbh->root_dse->supported_control(
    Net::LDAP::Constant::LDAP_CONTROL_PAGED()
) {
    # Process paged search operations.
} else {
    # Process simple search operation.
}