Sync still not working (v. 8.7) about browser-extension

[erosman]

I am going to work on and hopefully improve the sync process for v8.8.

[erosman]

I found the reason.
There are conflicting information on MDN which led to the issue.

storage.onChanged

object. Object describing the change. This object contains properties for all the keys in the storage area included in the storageArea.set call, even if key values are unchanged.

storage.StorageArea.onChanged

object. Object describing the change. This contains one property for each key that changed. The name of the property is the name of the key that changed, and its value is a storage.StorageChange object describing the change to that item.

First one says: "all the keys in the storage area, even if key values are unchanged"
Second one says: "one property for each key that changed."

After further testing both APIs storage.onChanged & storage.sync.onChanged:

• Firefox onChanged returns all the keys in the storage area, even if all values are unchanged
• Chrome onChanged returns only the changed values (event doesn't fire when there is no change)

Now that the problem has been found, I am going to re-write the sync process.

[erosman]

v8.8 was uploaded to AMO.
FoxyProxy Basic is available now and FoxyProxy Standard will be available after the AMO review.

[erosman]

Are you syncing between v8 instances?

[Doc73]

Yes, all instances now are on v. 8, in all my PCs and in all my browsers

[erosman]

I assume both FoxyProxy Sync & Firefox sync are on.
Are all instances on the same machine?

Please try:

• Go to the FoxyProxy Options page
• Open the Dev Tools (F12)
• Go to the Console tab
• Type the following and hit ENTER

browser.storage.sync.get().then(console.log)

What is the result in each of the synced Firefox browsers?

[Doc73]

@erosman
Yes, Firefox sync is enabled!
I already did this test! This command give some result and the next command of your tutorial downloads the json.

[erosman]

Can you post a screenshot?

Is it possible to post the settings?
(you can remove the private information before posting)

[Doc73]

[Doc73]

From another firefox, where I imported the settings manually

[erosman]

Please let me know...

• Is "Enable Sync" on in all of them?
• Can you show a screenshot of browser.storage.sync.get().then(console.log)?
• Do the Privoxy servers and TOR servers have the same hostname:port?

[Doc73]

Please let me know...

* Is "Enable Sync" on in all of them?

Yes

* Can you show a screenshot of `browser.storage.sync.get().then(console.log)`?

* Do the Privoxy servers and TOR servers have the same hostname:port?

Both are on localhost, but the second one is disabled and I left it just as a reminder. Ultimately, these are the same configurations as in v. 7.5, imported manually when you switched to version 8

[Doc73]

Of course, I could do a manual import of the settings, but I think it is more useful now to solve this problem. Until now, I have never seen, since the v. 8, an automatic synchronization!

[erosman]

The screenshot shows the sync data from v7 & v8.

• Use the code on Settings disappeared after the upgrade to v8 to create a backup file of the browser sync data
• If needed repeat for all instances of Firefox
• Run the following in the FoxyProxy Options page -> Dev Tools, which would clear/delete all the sync data

browser.storage.sync.clear()

• In one of the Firefox instances, click SAVE to initiate the creation of new sync data

Let's see how that works out.

[Doc73]

I'm sorry, but really I must clear ALL sync data????

[erosman]

Alternatively, post the sync data so that I can check it.
(you can remove the private information before posting)

[Doc73]

{
  "0": {
    "active": true,
    "title": "Privoxy",
    "type": "http",
    "hostname": "127.0.0.1",
    "port": "8118",
    "username": "",
    "password": "",
    "cc": "",
    "city": "",
    "color": "#66cc66",
    "pac": "",
    "pacString": "",
    "proxyDNS": true,
    "include": [
SOME RULES
  ],
    "exclude": []
  },
  "1": {
    "active": false,
    "title": "Rasp Privoxy",
    "type": "http",
    "hostname": "XXXXX",
    "port": "8118",
    "username": "",
    "password": "",
    "cc": "",
    "city": "",
    "color": "#df46df",
    "pac": "",
    "pacString": "",
    "proxyDNS": true,
    "include": [],
    "exclude": []
  },
  "2": {
    "active": false,
    "title": "Rasp Tor",
    "type": "socks5",
    "hostname": "XXXXX",
    "port": "9050",
    "username": "",
    "password": "",
    "cc": "",
    "city": "",
    "color": "#6db9cc",
    "pac": "",
    "pacString": "",
    "proxyDNS": true,
    "include": [
SOME RULES
    ],
    "exclude": []
  },
  "3": {
    "active": false,
    "title": "Tor",
    "type": "socks5",
    "hostname": "127.0.0.1",
    "port": "9050",
    "username": "",
    "password": "",
    "cc": "",
    "city": "",
    "color": "#cc1d0d",
    "pac": "",
    "pacString": "",
    "proxyDNS": true,
    "include": [
SOME RULES
    ],
    "exclude": []
  },
  "ijl831510855683809": {
    "active": true,
    "address": "127.0.0.1",
    "blackPatterns": [],
    "color": "#66cc66",
    "index": 0,
    "pacURL": "",
    "password": "",
    "port": 8118,
    "title": "Privoxy",
    "type": 1,
    "username": "",
    "whitePatterns": [
SOME RULES
    ],
    "proxyDNS": false
  },
  "oi30qj1504593779192": {
    "active": false,
    "address": "localhost",
    "blackPatterns": [],
    "color": "#cc1d0d",
    "index": 4,
    "port": 9050,
    "title": "Tor",
    "type": 3,
    "whitePatterns": [
SOME RULES
    ],
    "proxyDNS": true,
    "username": "",
    "password": "",
    "pacURL": ""
  },
  "fzse1g1696685685691": {
    "type": 1,
    "color": "#df46df",
    "title": "Rasp Privoxy",
    "active": false,
    "address": "XXXXX",
    "port": 8118,
    "proxyDNS": false,
    "username": "",
    "password": "",
    "whitePatterns": [],
    "blackPatterns": [],
    "pacURL": "",
    "index": 2
  },
  "logging": {
    "size": 100,
    "active": false
  },
  "l7j16r1668524890608": {
    "type": 3,
    "color": "#6db9cc",
    "title": "Rasp Tor",
    "active": false,
    "address": "XXXXX",
    "port": 9050,
    "proxyDNS": true,
    "username": "",
    "password": "",
    "whitePatterns": [
 SOME RULES
    ],
    "blackPatterns": [],
    "pacURL": "",
    "index": 3
  },
  "k20d21508277536715": {
    "active": true,
    "blackPatterns": [],
    "color": "#0055E5",
    "index": 1,
    "notes": "These are the settings that are used when no patterns match a URL.",
    "title": "Default",
    "type": 5,
    "whitePatterns": [
      {
        "title": "all URLs",
        "pattern": "*",
        "type": 1,
        "protocols": 1,
        "active": true
      }
    ]
  },
  "mode": "patterns"
}

[Doc73]

Ah! It seems there are duplicates.... 🙄

[erosman]

Please try the following:

• Add a new dummy proxy to one Firefox instance
• Wait for a while (sync is not immediate)
• Check if it is picked up elsewhere
• Check browser.storage.sync.get().then(console.log) everywhere to see if the new server was added

[Doc73]

The new dummy proxy doesn't appear elsewhere, but, just for curiosity, is this the right syntax of the json settings?

This file is from the foxyproxy of my new instance of Firefox Beta, where nothing appears in settings!!!!

[erosman]

but, just for curiosity, is this the right syntax of the json settings?

No.. that is messed up.

• obj was a mistake in v8.0 code as per #36, it shouldn't be there since v8.1
• it also appears that the container & commands are also messed up

Can you paste the actual above JSON here?

[Doc73]

{
  "mode": "pattern",
  "sync": true,
  "autoBackup": false,
  "showPatternProxy": false,
  "passthrough": "",
  "container": {
    "incognito": "",
    "container-1": "",
    "container-2": "",
    "container-3": "",
    "container-4": ""
  },
  "commands": {
    "setProxy": "",
    "setTabProxy": "",
    "quickAdd": ""
  },
  "data": [
    {
      "active": true,
      "title": "Privoxy",
      "type": "http",
      "hostname": "127.0.0.1",
      "port": "8118",
      "username": "",
      "password": "",
      "cc": "",
      "city": "",
      "color": "#add8e6",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [
        SOME RULES
      ],
      "exclude": []
    },
    {
      "active": false,
      "title": "Tor",
      "type": "socks5",
      "hostname": "127.0.0.1",
      "port": "9050",
      "username": "",
      "password": "",
      "cc": "",
      "city": "",
      "color": "#f66151",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [
        SOME RULES
      ],
      "exclude": []
    },
    {
      "active": false,
      "title": "Rasp Privoxy",
      "type": "http",
      "hostname": "XXX.XXX.XXX.XX",
      "port": "8118",
      "username": "",
      "password": "",
      "cc": "",
      "city": "",
      "color": "#808000",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [],
      "exclude": []
    },
    {
      "active": false,
      "title": "Rasp Tor",
      "type": "socks5",
      "hostname": "XXX.XXX.XXX.XX",
      "port": "9050",
      "username": "",
      "password": "",
      "cc": "",
      "city": "",
      "color": "#66cdaa",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [
        SOME RULES
      ],
      "exclude": []
    }
  ],
  "obj": {
    "data": [
      {
        "active": true,
        "title": "Privoxy",
        "type": "http",
        "hostname": "127.0.0.1",
        "port": "8118",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#66cc66",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [
          SOME RULES
        ],
        "exclude": []
      },
      {
        "active": false,
        "title": "Rasp Privoxy",
        "type": "http",
        "hostname": "XXX.XXX.XXX.XX",
        "port": "8118",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#df46df",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [],
        "exclude": []
      },
      {
        "active": false,
        "title": "Rasp Tor",
        "type": "socks5",
        "hostname": "XXX.XXX.XXX.XX",
        "port": "9050",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#6db9cc",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [
          SOME RULES
        ],
        "exclude": []
      },
      {
        "active": false,
        "title": "Tor",
        "type": "socks5",
        "hostname": "localhost",
        "port": "9050",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#cc1d0d",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [
          SOME RULES
        ],
        "exclude": []
      }
    ]
  }
}

[erosman]

Where is that from?
It appears to be from Export or browser.storage.local.get().then(console.log).

In any case, it is fine. The "obj": {} shouldn't be there, but I dont think it causes a problem.

If it is in browser.storage.local.get().then(console.log), you can delete it with

browser.storage.local.remove('obj')

[Doc73]

From export

[erosman]

You can deleteobj from you settings.

To be sure, since you have a backup, you can reset FoxyProxy and delete everything, then start fresh.

[Doc73]

Yes, I already deleted all the settings and tried to restart it from scratch, but I never get full syncs and some things are still left out.
I'll do some tests in the next few days and let you know.

[Doc73]

Here are some sad facts!

• I recreated a new, clean profile from scratch
• Since I couldn't synchronize the various browsers, I import the configuration into all the others browser
• Now four Firefoxes in two machines are identical, with three proxies inactive and one active
• I add a rule in the proxy
• After a few seconds, the rule is added automatically to another Firefox, but all the three inactive proxies disappear! 😢
• After further investigation and downloading the new json, I found that the surviving profile remained in "data": [, while the three missing, together with a duplicate of the surviving one, were transferred to "obj": { "data": [

Here is the structure of the new json, omitting the proxy rules and some other minor details:

{
  "mode": "pattern",
  "sync": true,
  "autoBackup": false,
  "showPatternProxy": false,
  "passthrough": "",
  "container": {
    "incognito": "",
    "container-1": "",
    "container-2": "",
    "container-3": "",
    "container-4": ""
  },
  "commands": {
    "setProxy": "",
    "setTabProxy": "",
    "quickAdd": ""
  },
  "data": [
    {
      "active": true,
      "title": "Privoxy",
      "type": "http",
      "hostname": "127.0.0.1",
      "port": "8118",
      "username": "",
      "password": "",
      "cc": "",
      "city": "",
      "color": "#99c1f1",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [,
      "exclude": []
    }
  ],
  "obj": {
    "data": [
      {
        "active": true,
        "title": "Privoxy",
        "type": "http",
        "hostname": "127.0.0.1",
        "port": "8118",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#66cc66",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [,
        "exclude": []
      },
      {
        "active": false,
        "title": "Rasp Privoxy",
        "type": "http",
        "hostname": "xxx.xxx.xxx.xx",
        "port": "8118",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#df46df",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [],
        "exclude": []
      },
      {
        "active": false,
        "title": "Rasp Tor",
        "type": "socks5",
        "hostname": "xxx.xxx.xxx.xx",
        "port": "9050",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#6db9cc",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [,
        "exclude": []
      },
      {
        "active": false,
        "title": "Tor",
        "type": "socks5",
        "hostname": "localhost",
        "port": "9050",
        "username": "",
        "password": "",
        "cc": "",
        "city": "",
        "color": "#cc1d0d",
        "pac": "",
        "pacString": "",
        "proxyDNS": true,
        "include": [,
        "exclude": []
      }
    ]
  }
}

[erosman]

I am not sure where the obj comes from. Did you clear the previous sync data?

If you want to text, you need to do the following:

• Clear all the data in all browsers (that means the settings, as well as sync data)
• Enables sync in all browsers
• Set some values in only ONE browser & save
• Wait to see the result

[Doc73]

How can I remove ALL sync data? By creating new FF profiles?
Or is it sufficient to reset the foxyproxy data? In case, how?
I had simply reset all FoxyProxy profiles using "Restore Defaults" among the options, in all browsers, then imported a json without obj.

[erosman]

How can I remove ALL sync data? By creating new FF profiles?

New profile not needed. Run this on all browsers from FoxyProxy Options page ...

browser.storage.sync.clear()

Or is it sufficient to reset the foxyproxy data? In case, how?

Click "Restore Defaults" and then SAVE

I had simply reset all FoxyProxy profiles using "Restore Defaults" among the options, in all browsers, then imported a json without obj.

That is good.

[Doc73]

• browser.storage.sync.clear() in all browsers
• Restore Defaults" and then SAVE in all browsers
• Import right json in ONE browser
• Toggle sync in another browser --> no result! 😢

[Doc73]

Ah!
Although I deleted the profile as you told me, from the second Firefox I downloaded the json. Well, it contains an empty data and always that damn obj!!!!

[erosman]

Restart the browser after clearing everything, to make sure its cache it cleared.

[Doc73]

No, no, no! It is not possible!!!!!
After deleting the data and restarting the browser, without having made any changes to foxyproxy, I exported the json: IT STILL CONTAINS THE OBJ SECTION!

[erosman]

The sync data is sent elsewhere by Firefox and then sent to other browsers from there.

You can try to clear the sync data and then do not enable sync for a while.
You can also try deleting that 'obj' and see ...

browser.storage.local.remove('obj')
browser.storage.sync.remove('obj')

[Doc73]

Done!
Tomorrow we'll see what happens. In the meantime, thanks so much for your help!

EDIT
The obj section remained. I haven't turned on sync

[erosman]

Try to uninstall all Firefox instances, restart the browsers and then install again one by one, and check their storage.

[Doc73]

I can try, but that's definitely not the problem. Now I'm on the laptop, so on another machine, yet the json still contains the obsolete obj, which is evidently what makes syncing problematic.

In fact, I noticed that when I told you that there was only one rule left, this was the one in data, while all the others, including the remaining profile, had migrated to obj.

PS: I'm on a Debian GNU/Linux machine, therefore I can't uninstall Firefox-est without breaking the system. BTW, it will be sufficient to remove the old profile and create a new one.

EDIT
I found that in Chrome the exported json has only these lines related to obj:

  "obj": {
    "proxyDNS": false
  }

[erosman]

Thank you for testing. Let me check again and see what is going wrong.

[Doc73]

Starting with a new profile, here's what I did:

1. sync with mozilla to recover all settings and plugins
2. export json from FoxyProxy --> NO obj
3. create some rules from scratch in FP and restart Firefox --> export json from FoxyProxy --> NO obj
4. enable sync and restart Firefox --> export json from FoxyProxy --> NO obj
5. open another version of Firefox and redo points 1-4 here
6. NO SYNC in the second Firefox 😢

EDIT
After a few minutes the synchronization worked, but there are still some issues:

• Firefox Beta has all 4 proxies and all rules; its Json is regular
• Firefox ESR has only 3/4 proxies; its json has the 3 proxies in data and all 4 again in obj

[erosman]

I also tested but obj did not appear.

I suspect, the ESR has issues.

From experience, sometimes an upgrade is partially completed and the addon ends up with some cached file from the previous version mixed with the new version. In these cases, it is best to uninstall and reinstall the extension.

See also: #73, #76 (comment)

[Doc73]

Reinstalled FoxyProxy.
I imported settings in ESR, then enabled sync in Beta, but nothing has arrived yet here and lots of minutes have passed.

[erosman]

Can you make an edit in the Options page?
Any edit will do, disable a proxy for example.
The edit will cause the sync to be started on SAVE.

Also .....

There is Firefox Sync compatibility between ESR and standard Firefox.
I am asking Mozilla for some clarification.

[Doc73]

OMG! I added a rule in one FF and after some minutes the addition went to other FF; then, I removed this rule in the latest one and... in FF beta all proxy settings are gone, except the active one!
I can't remember which one I started with, but I think from the beta version.

I don't think there are issues in Sync betwenn ESR e Beta: your link is very obsolete and now ESR is at v. 115, Beta at v. 122.

EDIT
However, I'm giving up: it's clear that there are serious synchronization issues and I have much more to do!

[erosman]

I have updated the repo with the reworked sync process.

Here is a zip of the beta for testing
The beta can be tested between 2 Firefox Nightly, or 2 Chrome instances.

foxyproxy.8.8.beta2.chrome.zip
foxyproxy.8.8.beta2.firefox.zip

[Doc73]

@erosman Unfortunately I can't test this zip, because I can't remove official Firefox installed by default in every Debian GNU/Linux system. 😢

[erosman]

@erosman Unfortunately I can't test this zip, because I can't remove official Firefox installed by default in every Debian GNU/Linux system. 😢

No problem.
I don't have multiple computers to try myself.
It would be good to test it before the release.
Maybe someone else can test it.