referrer add-ons discussion about user.js

[publicarray]

Q:

Is it compatible with all the new referer prefs that Mozilla has added to Firefox, or does it require them to be set a certain way?

A:
I just investigated the behaviour of FF prefs by adding console.logs() to the extension. In short to the FF prefs take precedence.

Default prefs:

referer as received by request.requestHeaders: "http://www.onlinehtmleditor.net/?d=asd"
Rejecting referer "http://www.onlinehtmleditor.net/?d=asd" for "https://www.whatismyreferer.com/"  
new referer:

set network.http.referer.trimmingPolicy to 2:

referer as received by request.requestHeaders: "http://www.onlinehtmleditor.net/"
Rejecting referer "http://www.onlinehtmleditor.net/" for "https://www.whatismyreferer.com/"
new referer:

set network.http.sendRefererHeader to 0:

<no output as expected>

This applies all extensions that read the referer header from the API. Note: extensions still have the capability to do whatever they want with this information though.

So if you want the extension to allow referer headers to certain sites than the FF preferences will have to allow this.

I suppose those prefs are applied before the data is passed to the internal network layer?

yes

---

Also note that the Smart Referer does not modify same origin requests (similar to network.http.referer.XOriginPolicy 1 or 2). In the future adding an option to block same origin referers would be a nice addition for advanced users.

[crssi]

uM has limitations since you can control referer only by source and not by destination or preferably both.
See https://www.reddit.com/r/uMatrix/comments/86samo/referal_improvement_suggestion/
while with SmartReferer you can control both, see example here meh/smart-referer#77 (comment).

I never used/tryed RefControl, so I can't comment on this one.

Cheers

Pardon me for being the joker who must have missed something but I don't understand the pertinence of referrer dedicated tools for Firefox, be they add-ons or scripts.

Where am I mistaking when I think and write that network.http.referer.XOriginPolicy when set to 1 allows sites to send their referer if base domains match (and even to 2 if hosts match is required, though more problematic often than 1), and as such is all a user needs? Unless to consider that even between 2 sites of a same domain the user could be trapped? But trapped in what way? Different domains need not to know that they have been accessed by a given user, but within the same domain, where is the problem?

Obviously there are several points I miss in the argumentation which credits the very add-ons/script mentioned above, the associated work still active. My point includes the idea that granularity control is maybe excessive given the per-site problems it creates compared to the actual benefits.

[Atavic]

@zymase See: http://www.ghacks.net/2015/01/22/improve-online-privacy-by-controlling-referrer-information/

@Thorin-Oakenpants #12 (comment)

@Atavic I had read the article you mention but understanding the different built-in Firefox settings to manage http headers is not my problem; what is my concern is to wonder if these settings, considering they seem to handle all possible schemes, are not sufficient by themselves to avoid brainstorming over what would appear as excessive per-site granular control regarding these http headers.

Not that I dislike brainstorming, on the contrary! But is it really necessary when it comes to http headers' management in the Firefox browser?

[Atavic]

Yes. You'll encounter some sites giving you breakages due to the fact that they pretend more referral info for them to work, as Soundcloud or Twitter.
I think uMatrix is a good option: https://github.com/gorhill/uMatrix/wiki/Per-scope-switches#referrer-spoofing

the referrer information will now be foiled using the root URL derived from the URL of the request.

#5 (comment)

So. That's all I was wondering of, @Atavic . Thanks for the info.

I could re-install another FF add-on I've used in the past, not mentioned here above, which handles only the network.http.sendRefererHeader setting with values 0-1-2 and has the advantage of allowing a 10seconds permission for sites requiring value=1or2 before resetting to 0
Change Referer Button at https://addons.mozilla.org/en-US/firefox/addon/change-referer-button/

Or, mentioned above, with a per-site handling, the RefControl FF add-on
Or use uMatrix when up to now uBlockO makes it fine for me.

What triggers my lack of enthusiasm is the pertinence, but reading what you above link to might make me change my mind.

[Atavic]

That one and https://addons.mozilla.org/en-US/firefox/addon/no-referer/ worked fine.

I've never tried this No-Referer FF add-on for the simple reason I hadn't heard about it.
It is mentioned that it is not available for Firefox 52 : if the add-ons mentioned above are seniors, No-Referer is a grand-pa : last updated in 2008!

[Atavic]

Right click and open with no referer... T'was great.

[Gitoffthelawn]

@publicarray Thank you for all your answers. Their brevity was not a problem at all! I look forward to reading your follow-up whenever you have the time.

[Gitoffthelawn]

@Alexander255 Thank you! We look forward to the WebExtensions version too!

[publicarray]

@Thorin-Oakenpants Thanks. Yes I hope @meh finds the time :)

[earthlng]

FYI: addons already on AMO can still be updated after the deadline with FF53. They will be only temporarily disabled until a new version with the flag is released by the addon devs.

[publicarray]

@Thorin-Oakenpants the developers pushed it to AMO about the same time you commented here. See: meh/smart-referer#61 With the automated reviews I think it will only take a few days to land on AMO.

[meh]

Yeah, that wasn't a coincidence, I saw the notification at a time where I had nothing to do, and remembered that pull request was waiting for early October, and early October it was 🐼

[publicarray]

thanks @meh 😍

@Thorin-Oakenpants The (web) extension is now live on AMO.

[Thorin-Oakenpants]

Thanks @meh ... Sweet .. will move it up from Legacy in the wiki

Sidenote: CanvasBlocker became WE on AMO about 12 hrs ago and uMatrix a couple of days ago .... we're getting there (not sure where there is but too late to get off the train)

[vertigo220]

I'm trying to make sense of these referer add-ons and hope someone can provide some insight. As has been mentioned, uMatrix eliminates tracking by spoofing 3rd-party referers, and I haven't had any issues due to this (that I know of) in the ~2-3 years I've been using it. And there's the fact, as was also mentioned, that referers can be handled through about:config. So I'm struggling to understand the point of separate add-ons like RefControl, Smart Referer, etc. What additional benefit, if any, do these provide? It seems there's no need for them and continuing to just use uMatrix for this should be sufficient, but I want to be sure I'm not missing something.

from what I could initially tell...uMatrix does referer blocking only in one direction. It seems to only block referers to sites, but not from sites.

First, what's meant by "from sites?" Are you saying that clicking a link on site A to go to site B will send info from site B back to site A? If so, it doesn't make sense how that would be an issue, since A already knows you're going to B. And did you ever confirm whether or not it is the case?

[vertigo220]

Thanks for the replies, though I'm still not understanding it. Why would you want to control it by source/destination? It seems the key is preventing site B from knowing you came from site A, and AFAICT uM does this. And you say if it were blocking all non-1st party referers it would break stuff, but isn't that what it's doing? My understanding is there's 1st-party (same domain) and 3rd-party (cross-domain), and uM blocks 3rd-party, which not only are the important ones, but is exactly what you say it would break stuff if it did. Clearly I'm missing something with all of this, and I'd love to know what, because I want to be sure I'm not less protected than I think.

[crssi]

@vertigo220 with all the respect, have you even look at the second link in my post?

[vertigo220]

Thanks. I thought I did, but I guess I didn't drill down deep enough. Looking at the ghacks-user.js issue #373 link, if I'm understanding it and the overall subject correctly, blocking 3rd-party referers, which uM does, does in fact provide full tracking protection, but it can break things that require what are essentially reverse referers, such as embedded Vimeo videos (apparently Vimeo won't allow videos to play if the player that's embedded on another site doesn't receive the referer indicating it's coming from Vimeo). I can't recall ever seeing an embedded Vimeo video, and I don't think I've used the main site in probably at least 2-3 years, so it explains why I haven't noticed it, but I can see that such cases do necessitate the ability to whitelist certain referers, and therefore why something more than uM is needed for those fringe cases. Am I understanding that all correctly? So it seems I should install Smart Referer, add a whitelist for referers from Vimeo to other sites, and disable uM's referer protection, correct?

[crssi]

If you don't have a breakages then you should stick to uM, there is no need to overbloat your profile with extensions that you do not need.
If you do, then go for SR. You do not need to white-list vimeo in SR since its already white-listed... the second link, the one you also failed to read the second time, its exactly about that... but in the same time its also a nice example of difference between uM and SR.
There is no reverse referers... it is saying any source site going to player.vimeo.com as destination should preserve referals.

[vertigo220]

I'm sorry that my lack of understanding what was happening in that thread due to the fact I've just started learning about all of this gives the impression that I failed to read it. Now that I have a better understanding of it, rereading that link I see that it was added to the built-in whitelist, though I'm not sure why that was necessary if you can Anyways, despite the fact I haven't (to my knowledge) experienced any breakage from using uM for this, I'd rather ensure there's as little chance of problems as possible while still maintaining adequate protection, both for my own sake and because I want to be able to set others up and their browsing habits and ability to solve said issues are different from my own. So based on what I've learned here, it seems SR is a better option. And hopefully others with the same question will be find the answer here.

One other question: I'm wondering if there would be any benefit to spoofing the referer to always look like I'm coming from duckduckgo (or another, similar site). Their privacy policy states they make revenue when people link from them to sites like Amazon and eBay and make a purchase, so I thought maybe I, and others that are interested, could make it always look like I'm coming from DDG. I'm just not sure if it would work, or how to do it. Looking at their links, I don't see any obvious referer info.

[vertigo220]

@Thorin-Oakenpants Yeah, I'm aware that's what it does (one of the many things I've learned in researching this over the past couple days), but I don't know enough about how referers work to get much more from that info than to know that linking to example.com from a google search will make example.com think I linked from them, not google. But I didn't realize that could/would break anything. It doesn't seem like it would, but apparently there are rare (or not so rare?) cases that it does. My end goal is just to keep site A and site B from sharing info and building a bigger, more accurate picture of me. I don't necessarily mind them knowing how I got there, since it can help them know how to better reach their audience/customers, but unfortunately tracking and profiling has gotten so out of hand that I'd rather put myself first and do what I need to in order to protect myself and others, which is why I'm trying to learn about this stuff and a lot more.

[crssi]

@Thorin-Oakenpants: the SR white-list is not definite, you can simply opt-out.
Even better, you can add your own white-list source which you can share with others. ;)
See meh/smart-referer#78

[vertigo220]

Glad I could confuse you. At least now I'm not the only one 😉

I don't even know anymore which way I was thinking of it with the Vimeo video. But I think (though clearly I'm no expert on this) that the site the video is embedded on is the source and it's requesting the video from vimeo.com, which is the destination, and I'm guessing that if vimeo.com gets a referral from itself to play an embedded video, it cries foul play. Perhaps @crssi can inform if I've got that right or if I'm totally backwards. And I think the issue with uM per crssi is that it can whitelist one way but not the other, so maybe it can whitelist vimeo.com so it can send its referer to other sites, but not so other sites can send their referer to it, which is (I think) what's necessary (and I see you just posted something along these lines--slow down yourself!! 😛 )

As for SR's whitelist, I figured it's scrutinized by users and hopefully at least mostly good. I guess it's a matter of using uM and getting rare breakage or using SR and possibly having leakage every now and then. Maybe uM for me and SR for my parents would be a reasonable strategy. The biggest problem with uM here is not the breakage, but that it's not even readily apparent. If I had gone to that site without knowing about the video, I never would have realized it was there.

[crssi]

The uM biggest problem is that you can "white-list" only source.
So when you do that on particular site, you will white-list referers to any destinations from this site.

[vertigo220]

On a side-note, but I think still related, I've also been looking at add-ons such as clearurls, neat url, and dont track me google, which all appear to do the same thing of cleaning links of their garbage. Interestingly, I'm not even sure if I need that, since apparently NoScript does it as well. In fact, I had to disable all my privacy addons and settings before finally getting to NS (I really didn't think it was responsible, plus I've stopped keeping it on my add-on bar, so I didn't do it until last) and realizing it was the one cleaning them up. But I can't for the life of me figure out what setting in it is responsible, and I'm in the process of searching to see if I can find anything about whether the others are needed if using NS or not.

[vertigo220]

Haha! I think I've got it. As crssi said, you can whitelist vimeo.com, so I went there and told uM not to spoof it. But the video still doesn't work, unsurprisingly, because (again, I'm not 100% sure about this) I've told uM to let vimeo.com send a referer, but I can't whitelist it to allow any site to send one to it. So in this case, vimeo.com appears to be the destination, and the host site (where the video is embedded), is the source, which sends a referer, but that referer is spoofed, causing vimeo to break. Interestingly, allowing the player in uBo but not in uM gives me the ability to click a link to play the video on vimeo, but even then it doesn't work. So vimeo won't even play a video on its own site if the referer isn't right. Crazy.

[vertigo220]

So it seems like the requests to be made for each add-on would be:

SR: Option to notify user when whitelisted item is activated
uM: Option to whitelist domain for referers being sent TO the domain in addition to the current ability to whitelist referers coming FROM it / Edit: AND option to notify when activated

Does that sound right?

[crssi]

@Thorin-Oakenpants SITE A is a source domain.
@vertigo220: going to vimeo and white-list it in uM wont work, since you are telling the uM to preserve referals on from when navigating (or getting any other info) from vimeo to any other site.
But in this case I would need from any site as a source to vimeo as destination (this cannot be accomplish in uM).

[vertigo220]

Right. So my understanding was correct. Sounds like if uM could be updated with the features I stated, it could replace SR and similar add-ons, making one less needed. Otherwise, if SR is updated to provide notifications, it would meet Pants' requirements, and would work in place of uM's spoofing without concern about possible unknown privacy leaks.

[vertigo220]

Well, SR meets your requirements if you disable the built-in whitelist. I was thinking more along the lines of you being able to use it to reduce breakage if they implemented a notification feature to warn you when something in the whitelist was making a request.

As for uM, that's why I said it would be nice if gorhill could add the functionality to whitelist either way. I agree that adding a column would be a good way to do it, since it would make it readily available, could work like the columns in uBo (pretty much as you described), and make the feature more visible (I didn't even realize you could whitelist that until crssi pointed it out, though, again, I only just started looking into referer stuff). I'll play around with SR and probably submit those as feature requests for both add-ons later. Right now I'm trying to unbreak Reddit, which seems to require allowing some trackers in order to view some threads...

[vertigo220]

Still working on the Reddit issue, but it doesn't seem related to trackers. Even though it randomly worked then didn't work then worked again and so on when I enabled or disabled uBo or certain domains within it, I've now narrowed down the problem to a specific setting in Reddit Enhancement Suite (which took me the past couple hours, ugh). Unfortunately, this is in Waterfox, and in Firefox with a mostly fresh profile the issue doesn't exist, so now I have to figure out what's conflicting to cause it. So much fun...