Server health tests are a little unfair due to absolute timing values about dir

[MrPetovan]

Here's the latest data from dir.friendica.social with the above query:
20180430-site-health.zip

[MrPetovan]

I'll admit I didn't expect to have so much fun when I agreed to maintain the Friendica Directory.

[AndyHee]

I'm not entirely clear on this point yet. But I think once we understand the probability of how the ping (X) value affects the response time (Y), we should know how to "factor" it.

The log function -- and here I understand this a little better -- is merely to normalise the curve of your factored values. If you were to divided Y/X and your curve consequently looks like this, you cannot really do the zoning properly.

So if you use log(Y/X), you could already do some zoning, even Y/X is just a random factoring. The above graph then should look less skewed.

[MrPetovan]

The MrPetovan/dir:master branch has been updated. You can use it to test the ping feature. Additionally, you can now use a CLI console tool to trigger a probe on a specific domain or site-health-id manually.

[MrPetovan]

I love everything about this.

[MrPetovan]

Just for clarification: When we pull and push as defined in sync-targets, we only transfer information about profiles and not any health related server information? Correct?

Yes, health is computed locally.

[AndyHee]

I'm analysing the data that dir.hubup.pro has generated over the last few days (since we are collecting avg_ping ). I currently have only 176 nodes, about 100 less than the last dir.friendica.social dataset.

Preliminary, it looks promising. I think there are similarities in the quartiles, despite that actually the values for discounted_request_time are very different. I'll wait for a few more days to get more data and will than show a detailed box chart to compare both directories.

I'm currently looking into ways how to calculate the quantiles and extreme values. I found something that looks like a potential direction for how to do quantiles in php (see below).

Once we know Q1 -3 you can also calculate the lower "extreme value" (i.e. Q3 + 1.5 x IQR).

function Median($Array) {
  return Quartile_50($Array);
}
 
function Quartile_25($Array) {
  return Quartile($Array, 0.25);
}
 
function Quartile_50($Array) {
  return Quartile($Array, 0.5);
}
 
function Quartile_75($Array) {
  return Quartile($Array, 0.75);
}

https://blog.poettner.de/2011/06/09/simple-statistics-with-php/

[AndyHee]

Here the code with fixed values.

//Speed scoring.
if (intval($time) > 0) {
	//Pentaly / bonus points.
	if ($time > 515) {
		$current -= 10; //Bad speed.
	} elseif ($time > 251) {
		$current -= 5; //Still not good.
	} elseif ($time > 181) {
		$current += 0; //This is normal.
	} elseif ($time > 71) {
		$current += 5; //Good speed.
	} else {
		$current += 10; //Excellent speed.
	}

and here with dynamic points based on all discounted_request_time values sorted.

//Speed scoring.
if (intval($time) > 0) {
	//Pentaly / bonus points.
	if ($time > $lower_iqr) {
		$current -= 10; //Bad speed.
	} elseif ($time > $Quartile_75) {
		$current -= 5; //Still not good.
	} elseif ($time > $Quartile_50) {
		$current += 0; //This is normal.
	} elseif ($time > $Quartile_25) {
		$current += 5; //Good speed.
	} else {
		$current += 10; //Excellent speed.
	}

lower_iqr = Quartile_75 + 1.5 x (Quartile_75 - Quartile_25)

[AndyHee]

Some preliminary results. Here a comparison of the two directories, one in Western Europe and the other Southeast Asia.

The datasets were taken at different times and have different total number of nodes (181 v. 270).

Results based on this equation:

discounted_request_time = request_time - (avg_ping * coefficient)

The coefficient:
dir.friendica.social-20180430 = 3.49712908930528
dir.hubup.pro-20180512 = 0.146000348540368

[MrPetovan]

Here you are:

2018-05-12-site-probe.csv.zip

I did deduplicate base_url but I didn't added the nurl column. It shouldn't skew the data too much.

[MrPetovan]

Here's my plan to change the policy:

• Ping all the known reachable servers
• Divide the average response time by the average ping latency
• Create break points based on percentiles on the resulting data
• Grade servers on a curve based on this data so that the absolute value doesn't matter as much.

My first impulse would be to do it once, take the resulting break points and hard-code them in the health scoring process, but ideally the break point should be updated regularly. The issue is that it requires to ping every know server which takes significant amount of time especially with 5 tries.

Maybe it could be a rolling thing where the ping is checked at the same time as the server health, and the break points are moved less regularly.

Another idea would be to have a fully proportional score change based on min/max values, this wouldn't require any break points.

[AndyHee]

The following is a proposed direction by a trained econometrician, whom I consulted about how to conceptually handle the issue by determining meaningful values for "server health".

Conditional distribution

request_time (Y) is a random variable and we are interested how it relates to another variable called avg_ping (X).

The most we can know about how X affects Y is contained in the conditional distribution of Y given X. This information is summarised by the conditional probability function, defined by:

for all values of x such that:

The interpretation is most easily be seen when X and Y are discreet. Then,

where the right-hand side is read as "the probability that Y = y given that X = x."

Skewness

Once we understand the probabilities we need to test whether the data set is skewed or normal. If skewed, we use a log function to normalise it.

Zoning

With a normal distribution we can easily define different zones to the left and right of the bell cure crest. Here we can establish the cut off points for the different level of healthiness of the server.

[AndyHee]

Here are the native math functions: http://php.net/manual/en/book.math.php
For statistics, it's a separate module: http://php.net/manual/en/book.stats.php

[MrPetovan]

Thanks for asking and for the work, I've a question: the bell curve studies only one variable (the response time in our case), how do you factor in the ping? Through the potential log function?

[MrPetovan]

Are we waiting on your friend then, or do we have to do anything by ourselves?

[AndyHee]

I'm going to test the log function for the time being. Next Friday, I'll see him again. He will try to get someone who will do the empirical part for us on a spreadsheet. Once we know the factor formula and the benchmark values, it can be coded.

Today's discussion was very brief (after some other meeting) and were merely conceptual. I put up the concept for people to comment. But I understand this is quite advanced and also partially beyond my own expertise.

[MrPetovan]

Beyond my own as well!

[AndyHee]

So the LOG-normal functions works as predicted. Here a before and after on the "request_time" variable:

Skewed (before)

Normalized (after)

This would be ready for zoning, if it had the ping factored in. I'll try to give it go even before the next meeting.

--
But there is a remaining issue, as we have an incomplete data set. I think, we need all known nodes because the inclusion of these 257 nodes out of more than 1000 is done rather arbitrary.

@MrPetovan do you think you could obtain data for all known nodes? All we need to ensure is that there is no duplication (e.g. http vs https).

[MrPetovan]

There’s a difference between “known node” and “active node”. I voluntarily limited the dataset to active nodes only because ping and request times don’t make sense for dead nodes.

[AndyHee]

That's good. Sorry my confusion!

Yes, if that's all active nodes out there that's great.

Do you have a script for generating the data?

[MrPetovan]

No, it was a simple SQL query against the site-health table, including only nodes with a positive health score.

[AndyHee]

Sorry, this is the part I don't understand.

In my site-health table I have 25 entries that are of positive value. The renaming 400 servers have a negative score. This includes for example my own server, your server, libranet.de.

So my question how do you make that leap from a negative health score to being inactive?

[MrPetovan]

Because the score penalty for connection timeout is very high, so anything with the lowest health score (-100) is unlikely to be active.

[AndyHee]

But could this not be part of the latency problem we're trying to solve?

Could you have a look at my table taken from this side of the world? There are some active nodes with -100.

andy-site-health.csv.zip

--
It's not very important to have the complete dataset now, because I hope that we can do the health levels dynamically in the future based on percentage zones with relevant cut off points on each site (see here: #43 (comment)). But it may help us to define the zones properly up front. It would rule out any distortion.

We can do this any time once we have the correct probability formula and agreed on the different zones in a spreadsheet for checking.

In the future, each directory might once in awhile calculate automatically the cut off values for health levels based on our predefined zones. This presumably will be done on the basis of all known nodes in the DB.

[MrPetovan]

The important field for available node is dt_last_seen. It's the last time the directory got a successful response. Compare this value with dt_last_probed which marks the last time the directory tried to probe the server. Any time there's a difference between those two date, a score penalty is applied to the server.

Also remember that high response time will also penalize servers, this is why you have mass-trespass.uk at -20 despite a successful last probe. And since you're far from most servers in this list, your scores will be overall lower than what I would get with a directory hosted in France.

This is why you probably should take into account nodes where dt_last_seen and dt_last_probed have a value and whose values aren't more than X [time unit] apart.

[AndyHee]

Yes, that makes sense. Failed probes, like outdated code, will reduce the overall score of course.

When I said dynamically calculated levels, I purely meant value of the response_time moderate by ping_avg.

My own node running on a piece of junkware is a good example; it has a very high response time but a relatively low ping (from my directory) and high high (from yours). The probability distribution formula will actually reduce my node's response_time value because of the low ping, but it probably will be only a tiny bit. My node will still very likely get a low score in terms of the high RT/ping value. That's the part we are interested in; to see how each percentage zone pans out along the curve at the crest and the base.

Are you able to see why I'm interested in the whole dataset of all known nodes? Do you think I could easily generate this myself for my directory by running a simple query?

[MrPetovan]

Of course I see the interest, but I'm weary of inactive nodes skewing the data set in a specific direction.

The other table to consider is site-probe, you'll find the result of successful probes. You can use average values to get a single set of ping/request time per server, and you can join site-health so that even current inactive nodes historical data will be included, increasing the sample size without skewing the data set.

The query to extract the data would go along those lines (I can't test until 8 PM EST):

SELECT `base_url`, AVG(`request_time`), AVG(`avg_ping`)
FROM `site-probe`
JOIN `site-health` ON `site-health`.`id` = `site-probe`.`site_health_id`
WHERE `request_time` IS NOT NULL
AND `avg_ping` IS NOT NULL
GROUP BY `site-health`.`id`

[AndyHee]

Thanks! No rush at all.

The conditional probability equation will not produce a value for any node that has a zero ping.

The very long request_times that just end up flat-lining at the base will turn into a steep drop through the normal-log function. See the before and after graph (see here: #43 (comment))

We will do a purely qualitative judgment with the zoning as to how narrow each zone will. This will ensure that very slow machines can never have a medium or high rating.

[tobiasd]

How can there be a ping of zero? Or do you mean a missing value?

[MrPetovan]

Less than a millisecond ping. Could happen.

[AndyHee]

Thanks MrPetovan for the data!!

@tobiasd highlights an important point. I think for the equation to give a probability, the ping value must be greater than zero. A superfast 0.001ms ping would still work.

In practice, this means admins who block pings will not have a health value. There are currently nodes that return a fast request_time but have a "0" ping, presumably because it failed.

Rather than giving these nodes a bad health score, they should be given a special status as "unknown health" or something like this. All the nodes that I saw fitting this category, were not open for registration. So this would be the trade off: if you run an open registration node, you need to be pingable. Otherwise you don't get a score.

[tobiasd]

One could use the average ping time for those nodes that block ping. I mean a fix value we find during the evaluation of the health determination round.

[AndyHee]

Yes, that's a possibility, but it's open for manipulation. So if you have a slow request_time and a faster than average ping, you just block your ping and instantly get a better score.

It's a kind of shared risk issue. There are many legitimate reasons why people block ping, but if everyone were to block ping than the directories would not work any more.

Of course we would still provide individual outputs for nodes without ping, similar to screenshot below. But there would not be a final overall score; instead of the heart being green or whatever colour, the heart would just not be shown or something like this, while clearly indicating the node is active. Mostly it wouldn't matter, because only nodes open for registration are listed in ../servers.

[tobiasd]

If someone is gaming the health test that way, we can just introduce a penalty by a horrible factor.

The fixed time could also be at the lower end of the bell-curve; average plus half of the full width half max value or so. Then the fixed time is more likely no desired value for gamers.

[AndyHee]

Yes, that would work. But than it would make the node's health look worse than it really is.

Do you think, it's important to have an overall score in such cases?

[tobiasd]

I don't think there will be gaming of the value, hence my suggestion with the average value to be neutral in that metric. I don't really have an opinion about "no health" value in the listing and if that would be better or worse for the node in terms of "advertising" the node to new users.

[AndyHee]

That's a good point! "Unknown health" can sound worse than "below average health" for some people. We probably need better explanations that would help people to interpret the finer points, either way.

If we give those nodes an overall score, I think a penality would be good mainly for encouraging people to allow pings were this is possible. Mainly a penalty for not sharing the collective risk of contributing to the average and standard deviation ping values.

How exactly that penalty would look like (so that's not an invitation to fiddle the system), we need to see with the data and the calculated probabilities. Something like you said, average or above average ping value, or even a simple 0 = 1.

[AndyHee]

Hypolite I have a question about the datatbase, in particular about the ping value.

Looking at the site-probe table, I cannot see any ping value. What am I missing here?

I am able to query the db for the request_time and able to join it with health-site, by omitting avg_ping`.

[MrPetovan]

Either:

• You need to update your database according to the dfrndir.sql file since there's no automatic update.
  Or:
• I need to update the dfrndir.sql file as the ping could be a dir.friendica.social-only feature at the moment. 😅

(Pretty sure it's the second one)

[AndyHee]

ping could be a dir.friendica.social-only feature

Ahh... it's not in the source yet, you're saying. Oh well. 😀

[AndyHee]

What's the easiest way? I'll update my db and add a table? If you give me a hint, I might be able to do it.

But does the current code actually collect pings or only your unpublished version?

[MrPetovan]

Only my unpublished version as well. The easiest way is for me to commit my code.

[MrPetovan]

Sorry for the untimeliness :/

[AndyHee]

I am outlining here what seems like a workable solution. @Ken-Ko , one of the team members who is the main driver behind this, has just joined us on github and will follows the implementation thorough and advise us further if necessary.

The initially proposed idea of treating this as a conditional probability problem has been rejected, as it will lead to unnecessary levels of complexity. Instead the more plausible way is to tackle this with a linear regression, i. e. ordinary least squares (OLS).

This will give a "discounted_request_time" based on the request time in relation to the ping value. We will only need basic mathematical operations (addition, subtraction, division, multiplication) and exponentiation. This also will avoid the problem with a zero/ failed ping as 0 will get no discount. Here the basic equation:

discounted_request_time = request_time - (avg_ping * Coefficient)

The Coefficient based on the current dataset (dir.friendica.social-20180430-site-health) is 3.49712908930528

This results in the following hard-coded equation:
discounted_request_time = request_time - (avg_ping * 3.49712908930528)

In the next post, I'll outline how to calculate the Coefficient; so we can do this dynamically, as new nodes come online, or disappear, or hardware changes.

For the moment, I would like you to look at the actual data and inspect Column H (Discounted Y_a). There you see all nodes sorted by "speed" discounted_request_time. The faster the node, the lower the value. Some fast/ superfast nodes have negative values. We are not really interested in the actual value itself, but in which (of the six) zones each node will fall accordingly.

There is a very nice chart in the excel file that shows the distribution and will give us an idea about the different zones. Unfortunately, the chart is only visible in a specific propitiatory version (which I think is MS Excel 2016). I hope to show this chart here once we get an exported graphic file. If you happen to have that particular version, you can already see the preview. Otherwise just look at the ods file for the data.

The zoning will be done dynamically too in the future. I'm working on this at the moment, but we can also hard-code the values for the time being. This means with the above equation we can have this up and running for testing in no time.

dir-friendica-social-20180430-site-health-OLS.zip

propitiatory_copy.zip

[MrPetovan]

Wow, thank you and your team for your work!

[AndyHee]

Scoring (Q1-3)

The data gives us the following three quartiles:

First Quartile: 75
Second Quartile: 118
Third Quartile: 251

In a boxplot the whole data looks like this:

Here in more detail with the outliers removed, generated by BoxPlotR:

http://shiny.chemgrid.org/boxplotr/

So the quick hard-coded fix will look like this:

	//Speed scoring.
	if (intval($time) > 0) {
		//Pentaly / bonus points.
		if ($time > 520) {
			$current -= 10; //Bad speed.
		} elseif ($time > 251) {
			$current -= 5; //Still not good.
		} elseif ($time > 181) {
			$current += 0; //This is normal.
		} elseif ($time > 71) {
			$current += 5; //Good speed.
		} else {
			$current += 10; //Excellent speed.
		}

[AndyHee]

AVG(avg_ping)

I'm looking at the average ping values as they come in at dir.hubup.pro.

I think, there seems to be an error in how the average of avg_ping is calculated. E.g. https://libranet.de in Germany gets 0.0894 or https://social.isurf.ca in California 0.9265 from Thailand.

Is it that a certain number of 0 values are ending up in the average?

[MrPetovan]

I don't know, I'm just taking the raw result from the ping command. Have you tried running a manual ping command against the same domains to see if there's a difference?

[AndyHee]

Looking at the avg_ping column in site-probe everything is correct! The ping for https://libranet.de is 236 and for https://social.isurf.ca 227, identical to the values taken manually.

The problem occurs when running a query the database to generate an output. I can actually see some error messages in addition to the distorted output.

So the ping seems to work without any issues. I'll check what's going on with my db or phpmyadmin setup.

Just for clarification: When we pull and push as defined in sync-targets, we only transfer information about profiles and not any health related server information? Correct?

[AndyHee]

I managed to traced the error that occurred when querying the database. It was a result of having modified the table index when I adding the two new columns. It's all fixed now and works as expected. 🙂

[MrPetovan]

This looks good, but I'm not sure what it would be for. Determining the cutoff points?

[AndyHee]

Here I'm just showing again the proposed zoning based the data for dir.friendica.social 20180430

A = Excellent
B = Good
C = Normal
D = Still not good
E = Bad

[AndyHee]

This looks good, but I'm not sure what it would be for. Determining the cutoff points?

Yes. See above.

Quartile_25 = Q1
Quartile_50 = Q2
Quartile_75 = Q3

[AndyHee]

@MrPetovan the explanation for the coefficient, I gave, is incorrect. #43 (comment)

I'll try to give the correct version shortly. Hope you have not already coded this.

[AndyHee]

Duplication of nodes

I have noticed there are some duplications in the base_url table.

Something like:
http://meld.de/
https://meld.de/

But we are quite sure there is only one node running there, despite the difference in protocols.

Even more concerning are duplications of entries with identical protocols. So for instance in Hypolite's dataset there are ten (10!) entries for https://libranet.de and about 15 for https://friendica.ladies.community each with different request_time values.

What's going on there and how to fix this?

[MrPetovan]

The behavior is even stranger than you expect. These are the only 15 redundant base_urls in the dir.friendica.social database:

base_url	COUNT(*)
https://box25.it	7990
http://localhost	302
https://salesnet.tomeetu.de	22
https://friendica.ladies.community	17
https://libranet.de	10
https://privet.su	9
http://192.168.244.183	8
https://www.ladies.community	7
http://192.168.178.208	5
https://friendica.me	3
http://172.16.0.10	2
http://social.pelikancms.pl	2
https://social.gl-como.it	2
https://social.retr.co	2
https://friendica.christsmith.ca	2

The first issue is that there isn't a UNIQUE key on the base URL. The second issue is that there's no reduction to a normalized URL (without https) which would allow to rule out HTTP/HTTPS duplicates.

[AndyHee]

Ohh.. what did we let ourselves in for here... 😲

I think this issue has effected the stats somehow. The coefficients are too different. Could you run this query again with:

WHERE `request_time` IS NOT NULL
AND `avg_ping` IS NOT NULL

We would like to run some further tests. Thanks.

[MrPetovan]

Of course, go ahead!

[AndyHee]

[OK, I deleted some of my redundant posts above]

Ko tested the two new datasets for us and we found some interesting developments. I'm summarising a three page long report here and will give the practical implication.

For "dir.frienica.social" the removal of duplicated nodes seemed to make the relationship between request_time and avg_ping even stronger. This is good.

However, for "dir.hubup.pro" the data showed there was no relationship between request_time and avg_ping . The above (see #43 (comment)) rather different coefficients were already some indication of this. These two graphs might give you further some idea of the problem.

After removing all servers with zero avg_ping value (and some outliers), we now have established that in the Thai dataset, there is also a significant relationship between request_time and avg_ping. Which is good, because it allows us to use the OLS equations as planned.

discounted_request_time = request_time - (avg_ping * coefficient)

Here the coefficients (plus p-values about the likelihood of no relationship)
"dir.frienica.social = 3.316080104 (p = 0.0)
"dir.hubup.pro" = 4.965583808 (p = 0.001)

Practical implication

The calculation of the coefficient and the Q1, Q2, Q3, and IQR values (see here #43 (comment)) must exclude all nodes with zero avg_ping . These nodes (providing they have a request_time that is not zero) will of course still get a health score, but will not contribute to determining the coefficient and speed zones.

[tobiasd]

But then each directory server has to automatically recalculate the coefficients from time to time--right?

[AndyHee]

Correct, and also its speed score zones.

Here an example for zones: #43 (comment)

[AndyHee]

OK, here the coefficient. Please excuse this non-standard notation. I hope this makes sense.

Coefficient=SUM of all x*y / SUM of all x^2

x = avg_ping - (AVERAGE of all avg_ping WHERE avg_ping is NOT zero)
y = request_time - (AVERAGE of all request_time WHERE avg_ping is NOT zero)

[MrPetovan]

Moved to friendica/friendica-directory#4