Count value too low about tweet-display-back

[mbabker]

I'm tempted to go with an advanced parameter to allow the user to define
the multiplier with a fallback/default of 4 (since 3 isn't working right
in this instance).

On 6/28/11 10:33 PM, "Twentronix"
[email protected]
wrote:

I'm using the new dev release at my website and pulling tweets from my
account. I show 2 tweets and have all filters active, so itll fetch 2x3
tweets. This is apparently not enough to get 2 tweets, it shows only 1.

Or we should higher the count value, which decreases loadtime. Or maybe
good idea to add an option @ advanced options to let the user setup the
count value and be able to override. I think this is a good idea to do,
no matter which count we will implement.

Reply to this email directly or view it on GitHub:
#6

If you do that, it would be better understanding to add a parameter for the user like tweetsToScan (or something) in stead of multiplicator they can fill in. So they know exactly what they're filling in.
We can then $count = params get->... $tweetsToScan;

[mbabker]

OK, focusing on this repo for a bit today...

I'm thinking two params here. One to define a maximum number of tweets to scan, and with that an option to let the user use their defined value or the existing $count * 3 method.

The count * 3 method won't do. How more active filters how higher this value should be. We should then make something like:
if ( overrideParameter = set)
count = overrideParameter
else
{
if (filterRetweets)
count=count*3

if (filterReplies)
count=count*3

if (filterMentions)
count = count *3;
}

At this way, how more active filters how higher the count value. That's exactly what we need. We only need to determine the right values. We'll need some statistics for that.

Best to add a notice to the user: how more active filters how more dataprocessing is needed (slower).

[mbabker]

Using this example, theoretically we end up pulling 8 times the number of
tweets the user wants just to grab enough tweets. $count * 3 should work
for one filter, but is obviously inefficient for more than one especially
when working with lower tweet counts. Filtering really isn't something
that was intended to be performed with the REST API, so having available
statistics to work with won't happen. I'd really have to study the SEARCH
API and see exactly how it returns data, and if (and only if) it returns a
JSON in the exact same format as the two endpoints I'm pulling data for, I
might be able to point filtered solutions to that API and get around this
problem.

On 7/2/11 12:16 PM, "Twentronix"
[email protected]
wrote:

The count * 3 method won't do. How more active filters how higher this
value should be. We should then make something like:
if ( overrideParameter = set)
cout = overrideParameter
else
{
if (filterRetweets)
count=count*3

if (filterReplies)
count=count*3

if (filterMentions)
count = count *2;
}

At this way, how more active filters how higher the count value. That's
exactly what we need. We only need to determine the right values. We'll
need some statistics for that.

Reply to this email directly or view it on GitHub:
#6 (comment)
0

The SEARCH API supports JSON, don't know if the format is ok:
http://search.twitter.com/api/

Note that the SEARCH API only finds posts from the last day/hours/..

[mbabker]

Well, the SEARCH API is out of the question. The results set it returns
is too little information to work with.

On 7/2/11 11:18 PM, "Twentronix"
[email protected]
wrote:

The SEARCH API supports JSON, don't know if the format is ok:
http://search.twitter.com/api/

Note that the SEARCH API only finds posts from the last day/hours/..

Reply to this email directly or view it on GitHub:
#6 (comment)
3

[mbabker]

The beta release includes the method I added with commit 00e4298 which does a multiplier based on the number of active filters (mentions, replies, and whether RT's are displayed). Starting at x3, each additional active filter raises the multiplier up by one (max of x5). If necessary, we can extend the multipliers further, but I think this should suffice (I haven't seen any errors in testing with high or low counts).

[mbabker]

Looking at the updated API documentation, I see that a new method has been added (at least I don't remember seeing it). This should help not only with ensuring the correct tweet count, but code specific to filtering replies could possibly be pulled to help with the processing time too. I'm gonna test a little with this parameter.

exclude_replies

This parameter will prevent replies from appearing in the returned timeline. Using exclude_replies with the count parameter will mean you will receive up-to count tweets — this is because the count parameter retrieves that many tweets before filtering out retweets and replies.

Nice!
Sorry I'm quite bussy these days.

[mbabker]

I feel ya on the busy. I think my activity stream says a lot of where my
being busy is coming from... LOL

Just like excluding include_rts from the URL, filtering is done on
Twitter's end after the tweets specified in count are retrieved. Still
gonna have to include this condition in the multiplier, but can definitely
work on processFiltering with this parameter.

On 7/17/11 11:02 AM, "Twentronix"
[email protected]
wrote:

Nice!
Sorry I'm quite bussy these days.

Reply to this email directly or view it on GitHub:
#6 (comment)
6

Ok installed 2.0 beta.
The tweets to scan was not high enough with my profile at the moment. I
hope the new api works :)
The description text of "Define number of tweets to scan" is not complete.

The rest is working ok it seems.
If I'll have some spare time I could translate some to Dutch. Do you
have a translater yet?

[mbabker]

I don't have any "official" translators. I'm listed on Transifex after
some people in the Joomla community started a translators initiative. So
check out https://www.transifex.net/projects/p/tweetdisplayback for that.

I did the "define number of tweets" field to be an override to the count
and multiplier method. From there, the originally defined count is still
used as the check on the number of items that have been processed for
display. I'm always open for ideas to improve it if that just won't work
for whatever reason.

On 7/17/11 11:30 AM, "Twentronix"
[email protected]
wrote:

Ok installed 2.0 beta.
The tweets to scan was not high enough with my profile at the moment. I
hope the new api works :)
The description text of "Define number of tweets to scan" is not complete.

The rest is working ok it seems.
If I'll have some spare time I could translate some to Dutch. Do you
have a translater yet?

Reply to this email directly or view it on GitHub:
#6 (comment)
2

[mbabker]

OK, I can implement exclude_replies and remove all checks for replies in the module, but this method comes with a side effect. If you've retweeted a tweet which was originally a reply, then it gets dropped from your stream with the exclude_replies method. I'm gonna search around a little bit and see if this is intended or not and decide from there what to do.

[mbabker]

https://dev.twitter.com/discussions/427 - Discussion on Twitter Developers site

[mbabker]

exclude_replies pulls out retweeted replies by design. To implement it saves some processing (which probably amounts to milliseconds, the real killer in loading is pulling external data) at the expense of unintentionally filtering out retweets meeting the condition set by that. It still doesn't solve the tweet count issue because they filter tweets after getting the data (which makes no sense to me, but it isn't my API).

I've pretty much got my site set to be migrated up to J! 1.7, and in doing so, I'm adding tutorials and additional documentation to the site. This is going to have to be one of those things I document very clearly.

[mbabker]

Hopefully this block explains things well enough for most users.

Filtering & Tweet Count

Enabling Tweet Display Back's filtering capabilities can at times cause less tweets than expected to be rendered. Depending on your settings and tweet habits, this can be temporary in nature or something that you will need to fix using Tweet Display Back's own parameters.

What's Happening?

Because Twitter doesn't natively filter tweets, the module has to filter them after retrieving the data from Twitter. In this instance, this potentially means that less tweets than you specify would be displayed. To get around this, there is logic in the module to increase the number of tweets that are pulled from Twitter depending on the number of active filters. The logic breaks down as such:

Filter Types
    Show Retweets (defined in Tweet Settings)
    Show Mentions (defined in Filter Settings)
    Show @Replies (defined in Filter Settings)
Module Logic for Active Filtering
    If 1 filter is active, multiply "# of Tweets" parameter by 3
    If 2 filters are active, multiply "# of Tweets" parameter by 4
    If 3 filters are active, multiply "# of Tweets" parameter by 5

I Don't Get Enough Tweets

This method can be somewhat problematic if you are displaying a low number of tweets on your site. In testing, we found that a site that displays only 2 tweets many times doesn't even get 2 tweets with filtering enabled (which would cause 6, 8, or 10 tweets to be pulled, depending on settings). To help get around this, a "bypass" was added to the Filter Settings panel. By enabling the "Define number of tweets to scan" option, the user can specify the number of tweets to pull from Twitter. Using this option is a full bypass to the multiplication logic explained above; the module will pull specifically the number of tweets you specify in the "# of Tweets to Scan" option. For best results, you'll need to set this number to be at least 5 times higher than the "# of Tweets" option in the Twitter Settings panel (since 5 is the highest multiplier used by the code).

How Does The Bypass Affect My Display

Answer - It doesn't. The final output will still be at a maximum the value you specify in the "# of Tweets" option. This method only affects the amount of data being retrieved from Twitter.