Max results for search query? about livesdk-for-windows

[peternied]

There is a hard limit of 500 items returned in a given search against OneDrive with the LiveSDK service.

[factormystic]

Thank you for the confirmation. However, this is incredibly irritating in two ways:

First, it's undocumented and I had to spend my own time figuring out another way Onedrive/Live SDK is secretly broken.

Second, this is an absurd cap that make impossible to build real world apps. I understand capping the total number of results per request- paging is no big deal. But to say that Onedrive/Live SDK makes it impossible to get result number 501 means that for me as a developer it's impossible to build an app that requires result number 501... aka, any app that is actually going to use search for real. This means I can't build my app to rely on Onedrive/Live SDK search functionality any more. Very disappointing.

[rgregg]

Thanks for the feedback. The search feature was designed as a user scenario, where the user is looking for content in their OneDrive. Most users never make it past the first page of results, let alone that deep into the query. If they can't find what they're looking for quickly, they refine the search to get a better set of results.

Your scenario sounds different, where you are using the search API to retrieve a set of all files in someone's OneDrive matching the query. For example, perhaps you are searching for all JPG files. That's something the search API wasn't designed for in it's current iteration.

We're actively working on improving our documentation and our API, so I appreciate the feedback. If you can elaborate more on what you're trying to accomplish, it'll help us make sure we've got an API designed for real developer scenarios.

[factormystic]

@rgregg let's say I had an idea for an app that would do something with your music collection... say, analyze your albums and print a histogram of album count by release year. Not too complicated. The Live SDK let's me SSO from a Windows Phone to the same account's OneDrive, and if that account is also connected to a Windows 8+ user, then many users' music collections are already in OneDrive, ready for me to access with my app. So far, so good.

Now, how could I actually get access to those files? This is where things start to break down.

I suppose, given that I know the music is in a root folder called Music Library, I should be able to search for all files in that folder and then pick out the ones with type audio. Except I can't, because searches can't be scoped to folders like that. Well, I could use search api with ?filter=audio to just get music. Except I can't, because searches require a search term and you can't just filter all files by type. Well that's ok, I can use the the (undocumented?) wildcard search (q=*) to do a search across the entire file store including non music, and combine with with &filter=audio to just get music! Except I can't, because search queries can't be combined with filters. Sigh, well, ok fine I'll just do the q=* search and do the filtering myself afterward. Except I can't, because there's an undocumented hard cap on the total number of search results.

You see how many times an idea with the Live SDK/OneDrive caused me to say "I can't". This is how the platform is demonstrating that it's not appropriate for app developers.

So the solution here is to get a folder's items, identify those with type audio, also identify those with type folder, and recursively build up the list of items. Error-prone, slow, and requires many many more http requests than a simple 1 hit (or paged) search.

But oops, even after figuring out how to get all of a user's music files for analysis, I still can't build my app, because even though I see audio metadata like track number, year, copyright, etc on the OneDrive website, it's not returned as part of the Live SDK api. So now it's time to stop building my app with OneDrive.

[mmcdole]

I agree with @factormystic, it makes sense to require paging for search results. It doesn't make sense to not let us access the next pages search results.

None of the other Cloud APIs do this. I can easily pass in a nextPageToken to Google Drive's api and grab the next set.

Can we please have this restriction removed from the OneDrive API?

[rgregg]

I appreciate the details for the scenario you're trying to build out. I've added an item to our backlog to make sure we make search work for this type of scenario. I don't have any details to share about ETA, but I'll provide an update here when I have one.