ooni / ooni.org

The ooni.org homepage and all cross organisational issues

OONI Web

This is the canonical website for OONI, that you can access via:

• https://ooni.org
• https://openobservatory.github.io
• https://ooni.netlify.com

If you are having trouble accessing our website, please contact us at contact [at] openobservatory.org.

Contributing articles

• Fork this repository if you're not a collaborator
• Add your post to content/post/ directory
• Submit a pull request
• Wait for pull request to be reviewed and merged to master or merge it yourself :)

Local build

Setup

Dependencies to build the website manually are hugo, ooni-probe, and sphinx to build ooni-probe docs. Exact versions are codified in "canonical" build procedure in travis.yml.

Running locally

If you want to preview the website while editing styles and posts you can do it running following command:

make server

If you want to publish to Github mirror, you should run:

make publish

End to end local testing

To run the end to end integration tests do:

yarn install
yarn run cypress open

ooni.org's Issues

Document deployment procedure.

It appears that this repository is a source for the blog portion of https://ooni.torproject.org/ .

• What's the deployment process for https://ooni.torproject.org/ ?
• What's the deployment process for only this portion of https://ooni.torproject.org/ ?

Archive sources OONI cites in reports to archive.org

OONI cites other websites heavily. Websites come and go.
So, maybe, we should archive every external reference we link.
Should we?

@agrabeli @hellais @elioqoshi

Terminate [email protected]

We're not in a position to offer legal advice, so I propose we terminate the [email protected] email address.

However, it seems that we're linking to it somewhere, since some people are finding this email address and reaching out to us, seeking legal advice.

We therefore first need to identify the web pages that mention [email protected], edit those pages (to delete the email address), and then proceed with terminating this email address.

Add illustrations for each test under the Tests section

The "Tests" section (https://ooni.torproject.org//nettest/) was recently revamped at OONI's hackathon to improve the accessibility of the test descriptions.

It already looks much better than what it used to, but this section would look even better if an illustration was included for each test. :)

How does this repository relate to https://ooni.torproject.org/ ?

It appears that this repository is the source for a subset of https://ooni.torproject.org/ . Is that true or false?

If so, where does the other content for https://ooni.torproject.org/ come from?

Installation instructions by platform

We should have installation instructions by platform. Installing from pip or from source should really be the last resort.

Publish NDT description under the "Tests" section of OONI's website

OONI mobile is going to be launched soon and it includes the NDT test. As such, the NDT test description should be published under the "Tests" section of OONI's website asap.

The NDT test description can be found here: #80

Remove the github wiki for this repository.

Assuming there is no plan to use the wiki associated with this repository, let's remove it.

Improve the OONI website

The current OONI website can do with some improvements. I made some quick mockups of how I would imagine the future of the OONI homepage.

Related tickets:

• #35
• #36

Create Partnership Page

The new OONI Partnership Page would communicate:

• What is the OONI Partnership Program
• Who are OONI partners (i.e. which organizations)
• Which countries around the world are covered by OONI partnerships
• How OONI partners use OONI Probe and OONI data as part of their work

To this end, I propose that a possible layout for the new OONI Partnership page is the following:

• Write a summary explaining what the OONI Partnership Program is and linking to this page (which provides detailed information): https://ooni.io/get-involved/partnership-program/

• Include a map which highlights the countries we have partnerships in => As a MVP, this map could for starters be static, just highlighting the countries we have partnerships in. In the future (when dev resources permit), we could potentially include an interactive map where not only are countries highlighted, but when you hover over a highlighted country, you get that country's partner (their logo and info about their work).

• List all of the partners, and categorize them based on geographic regions (Latin America, Africa, Asia, Global). In the listing, include their logos & links to their websites.

• Include a "highlights" section (it won't be called that), where we showcase some partner projects that use OONI Probe and OONI data. As an example, we can showcase Sinar Project's Southeast Asia Dashboard (https://sinarproject.org/digital-rights/human-rights-internet-censorship-dashboards), the research reports published by the Digital Security Lab of Ukraine, the projects of our Venezuelan partners, etc.

• In the future, it would be cool to embed a 2-minute video on this page which features our partners, where they tell the world how they lead censorship measurement research and advocacy in their communities.

Include a section to post regular censorship observations

We often come across interesting data in OONI Explorer, when analyzing OONI's data, and/or when running tests. Perhaps it would be good to have a section of the website where we regularly post censorship observations?

I'm not suggesting a section for blogs or reports (we already have that). Instead, it could be something like "OONI's weekly/monthly observations", where we post the following:

• A few sentences explaining what was observed and why it is interesting (e.g. Vanilla Tor measurements in Egypt present anomalies - is Tor actually blocked in that network?).

• A link to the network measurement(s) on OONI Explorer (or elsewhere?)

The important thing here is that we don't make any bold statements about censorship (unless there's a block page), but rather just share our observations. Confirming a censorship event (when there's no block page) requires a fair amount of data analysis (based on the current state of the pipeline), and I'm not suggesting that we invest time in analyzing every single anomalous measurement we come across.

Instead, this would be a place for us to share things that triggered our interest/sparked our attention with the broader OONI community. That way, if someone has the skills/interest to analyze the data further, we will be providing them a starting point (or they could just check the reach-ability of a site if they are based in a country where the anomalous measurement is coming from).

By having this section (and updating it with posts regularly), we can keep our audience engaged with the project and this research in general. Also, it would be a way to outsource some of the analysis to those who are interested.

Thoughts?

update-site.sh should block on travis completion

As noticed by @agrabeli if you don't wait until travis completes building the new website, but run make publish-update then you don't get the latest version, but you get instead the n-1 version.

update-site.sh should wait until travis has done it's job before doing it's business.

This can maybe be done by polling: https://api.travis-ci.org/repos/TheTorProject/ooni-web/builds.

cc @darkk

https://www.ooni.io is giving certificate errors

Visiting https://www.ooni.io gives me SSL_ERROR_BAD_CERT_DOMAIN with a certificate that seems only valid for netlify.com and its subdomains.

Layout width bug on Android

The website is not using full screen width in Android Chrome (portrait orientation, album orientation is OK.
Screenshot:

Add docker file to reproducibly build the site

We should have a Dockerfile that allows us to build the website with a known good version of hugo and has everything setup properly to do it.

The running of the dockerfile should be instrumented from the Makefile.

Licence?

It should probably be clarified under what licence this content is published.

Add OONI Mobile App in Install section of website

We should add OONI's Mobile App in the Install section of the website: https://ooni.torproject.org//install/

Changes to Net Tests section of website

I propose the following changes to the Net Tests section of OONI's website (https://ooni.torproject.org/nettest/):

1. Replace Net Tests (title of webpage) with *Tests (To the normal user, "net tests" might potentially be quite confusing).

2. List the test descriptions under the following categories (and include the following short descriptions next to each one):

Core OONI tests:

• Web connectivity: Examine whether and how websites are blocked

• HTTP header field manipulation: Examine whether systems that could be responsible for censorship and/or surveillance are in your network

• HTTP invalid request line: Examine whether systems that could be responsible for censorship and/or surveillance are in your network

Circumvention tool tests:

• Vanilla Tor: Examine whether the Tor network is blocked in your network

• Tor Bridge Reachability: Examine whether Tor bridges - used to circumvent the blocking of the Tor network - are blocked in your network

• Psiphon: Examine whether Psiphon is reachable/blocked in your network

• Lantern: Examine whether Lantern is reachable/blocked in your network

• OpenVPN: Examine whether OpenVPN is reachable/blocked in your network

Other OONI network measurement tests:

• HTTP Host

• DNS consistency...

etc etc etc

In short, all test descriptions would be grouped under 3 categories:

1. Core OONI tests

2. Circumvention tool tests

3. Other OONI network measurement tests

Of course, we could reconsider the name of these groups, but this is a general idea. Then we can include a 1 line description of each test, and upon clicking on the test name the user will be presented with the full description of that test.

Thoughts?

untitled "Thematic OONI Probe testing" page

https://ooni.org/get-involved/run/
<title>OONI - </title>
The value of {{.Title}} is not received.
https://github.com/TheTorProject/ooni-web/blob/f12b92321478e90e6e22c8039567f298db8fcb95/layouts/partials/head.html#L10

GPG key error

To install OONI in Debian, there is a problem getting GPG key.
https://github.com/TheTorProject/ooni-web/blob/master/content/install/ooniprobe.md

root@dbn:/home/user# gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
gpg: keyserver receive failed: No keyserver available
root@dbn:/home/user# gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
gpg: keyserver receive failed: No keyserver available
root@dbn:/home/user# apt-key adv --keyserver keys.gnupg.net --recv-key A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89     
Executing: /tmp/apt-key-gpghome.6FwcbNgQd8/gpg.1.sh --keyserver keys.gnupg.net --recv-key A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
gpg: keyserver receive failed: No keyserver available
root@dbn:/home/user# apt-key adv --keyserver hkp://keys.gnupg.net --recv-key A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
Executing: /tmp/apt-key-gpghome.KTXyWIAuiM/gpg.1.sh --keyserver hkp://keys.gnupg.net --recv-key A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
gpg: keyserver receive failed: No keyserver available

Previously, the solution to this problem was to use the following commands:

curl https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --import
​
gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add - 

Result:

root@dbn:/home/user# curl https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --import
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19665  100 19665    0     0   4702      0  0:00:04  0:00:04 --:--:--  5011
gpg: key EE8CBC9E886DDD89: 36 signatures not checked due to missing keys
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key EE8CBC9E886DDD89: public key "deb.torproject.org archive signing key" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: no ultimately trusted keys found
root@dbn:/home/user# gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add -
OK

I think the reference to the solution on that page is good.
And the question is, why is it so? :)

Improve documentation for raspberry pi installation

Taken from #68:

I think the installation instructions for lepidopter should be restructured to take into account the fact that there are two types of users of lepidopter (or maybe even 3):

A user of lepidopter that has gotten it via our partnership program. These from what I can tell are actually the majority of lepidopter users and are people that don't care how to burn the image to an SD card. This user is not technically savvy.

A user of lepidopter that wants to download the image and burn it onto their SD card. These are a minority of users that are quite technically savvy.

(if we include this as well) Power users or developers that want to contribute in some way to the project. Very technically savvy.

In light of this I think the guide should be divided into at least 2 sections.

User guide

How to burn an image

Inside the user guide there should be clear and simple explanations as to how to connect the raspberry pi to the power and internet (maybe even including screenshot). @andresazp showed me the document he created for VI and it looks great.
It should also cover information on how to initialise the pi (answering the informed consent) and check that measurements are being run, by visiting the web UI.

An outline of the User Guide could be:

Setting up the device (covering how to connect it to internet and power, putting in the SD card etc.)

First start (covering how to reach the pi over the network, using lepidopter or lepidopter.local) and answering the informed consent

Checking that everything works (checking that measurements are being run)

The How to burn an image can remain as the current section titled: "Downloading and verifying the Lepidopter Raspberry Pi image"

I would also try to make this guide overall much shorted and where possible point to external resources instead of having so much content in here (for example explaining the tests).

I am also thinking that maybe the whole part on burning the image could be tucked inside of another page entirely as it makes the whole guide much less user friendly.

Another comment is that the whole section "Lepidopter services" is far too technical and contains a bunch of information that will not be useful to most cases (why would the normal user care to know that there is a mDNS service for SSH?).

I would tuck this stuff as well inside of some advanced section and try to have a minimal guide that is easy for most users to follow.

The experience we had with real deployment of these is that even just explaining how to plug in a pi into a router and connecting to power can be challenging for some non technically savvy users and since this guide is addressed to them we should take that into account.

Link to the latest lepidopter release in documentation

A lepidopter latest-image link should be used in order to have a canonical URL
for downloading the latest lepidopter release.

Add OONI mailing lists on the contact page

We should add the OONI (ooni-dev, ooni-talk, ooni-operators) mailing lists in ooni website on the contact page.

PGP key appears to have expired on March 23rd, 2017

Page at https://github.com/TheTorProject/ooni-web/blob/master/themes/ooni/layouts/section/about.html (https://ooni.torproject.org/about/) says the PGP key expired on March 23rd, 2017. The same key is quoted in https://github.com/TheTorProject/ooni-web/blob/master/themes/ooni/layouts/index.html (https://ooni.torproject.org).

Also, I cannot seem to find the actual key anywhere but perhaps this is just me.

Create OONI glossary

OONI uses a lot of technical jargon, as well as terms which may create ambiguity (such as "middlebox").

To improve our communication, help resolve any ambiguity, and better support the work of translators, I think it would be great to have a glossary which explains all of the technical terms used by OONI. More specifically, I imagine this glossary containing a list of technical terms (used by OONI), and brief 1-2 sentence descriptions (in plain English).

I plan to get started on this soon. Any suggestions/ideas are welcome!

Mention platform support for each test

As per the discussion here:
#132 (comment)

is that possible to create my own TOR network on the LAN ??

Capitalize "l" in .RSSLink to prevent hugo build errors

The .RSSlink field seems to have been renamed to .RSSLink ( https://gohugo.io/news/0.18-relnotes/ ) and fails to build with a miniscule "l".

Invoking make server also gives a warning that .RSSLink will be deprecated in the future:

Building sites … WARN 2019/09/23 11:41:22 Page's .RSSLink is deprecated and will be removed in a future release. Use the Output Format's link, e.g. something like: {{ with .OutputFormats.Get "RSS" }}{{ .RelPermalink }}{{ end }}.

( gohugoio/hugo#4427 (comment) )

Fix display of reports/blogs on homepage

The homepage of the OONI website normally displays a list of a few of the most recently published reports and blog posts, but it's currently empty...

Create pages for test groups

As we are adding the test methodology information in the apps (ooni/probe#822) we discovered that the only way to give information about a test group was to link it to the anchor of the page. ex.
https://ooni.io/nettest/#im

It would be probably wise to create pages for the test groups on the website so we can link them inside the app.
@agrabeli can you write some content?

License error

In bower.json the license is listed as MIT but in the license file is a Creative Commons 3 License. These licenses are not compatible https://creativecommons.org/share-your-work/licensing-considerations/compatible-licenses/

Create censorship report submission form

Many community members reach out to OONI reporting on new censorship events around the world.

Perhaps it would be useful to have an online form on our website that community members can use to report censorship events. This form could also include the standardized set of questions that we always ask when censorship events are reported.

Furthermore, this form could potentially include information on:

• How to find the data that interests them via OONI Explorer (or via the OONI API)

• How to run tests to measure the resources that are reportedly blocked

Add rtl support for blog posts

This came up as part of: #288

Add page about OONI users

Once upon a time we used to have a page that contained information about who our past and present users were. We should create such page again and add it to the list of pages.

blog posts may use special markup to get better snipptets in social media

It would be nice to see rich snippets for OONI blog posts in facebook/twitter/telegram/whatever.

Facebook has some heuristics to generate rich snippet without special markup, you can see it here, but it's imperfect.

Twitter has no heuristics and just shows the link as-is: https://twitter.com/TheMusta/status/791974185857847296
Twitter also has some process to whitelist domains to be able to generate twitter cards. I'm unsure if this process is trivial or not.

Opengraph works for facebook/google/telegram and allows to control image/title/text shown in the snippet.
Here are some examples of opengraph markup: https://moz.com/blog/meta-data-templates-123

Chores: cleanup old OONI webs

• Update the Readme to mention the new deployment procedure
• Remove all references to ooni.github.io
• Remove all the dead code from Makefile
• Mention the diff so people can review it
• Mention in the Readme (if there is anything required) what people should do to update their workflow to deploy with the new procedure
• Fix local build

Update all references to ooniprobe with OONI Probe

Since we are changing the app name, we should be sure everywhere in our website where we say ooniprobe, we now say OONI Probe.

This can be easily spotted by running:

$ grep -rn ooniprobe content

and

$ grep -rn ooniprobe themes

And being careful to just replace the occurrences of it in text and not those as a command (the ooniprobe command will stay the same).

This is the list of files that should be looked at:

content/about/data-policy.md
content/about/inform-users-long.md
content/about/risks.es.md
content/about/risks.md
content/data/content.md
content/faq/faq.md
content/get-involved/contribute-test-lists.md
content/get-involved/partnership-program.md
content/install/lepidopter.md
content/install/ooniprobe.md
content/nettest/dash.md
themes/ooni/layouts/index.html
themes/ooni/layouts/section/about.html
themes/ooni/layouts/section/get-involved.html
themes/ooni/layouts/section/install.html

I excluded the posts from the search as we should probably keep those as they were at the time of writing.

cc @agrabeli

Highlight.js is making certain words bold in code snippets

Some of the code snippets on our website have certain keywords highlighted. In certain cases this doesn't really make sense, for example in:

sudo apt-get install ooniprobe

get is highlighted.

It appears to be the work of highlight.js:

Add more contact information to the homepage

We should have a shared project email address that is listed as contact information on the homepage of the website.

Un-rendered markdown on install page

The instructions to install and run ooni are formatted with markdown, but the markdown is not interpretted/rendered correctly. https://ooni.org/install/ooniprobe/

Write and publish the IM test announcement post

When 2.1.0 is out we should write an publish an announcement blog post about it.

Add support for comments

For a brief window of time (couple of hours), I enabled comments via disqus, thinking it would fit our use case.

However their claim to support "anonymous" comments is a lie (they actually require an email address to post "guest" comments), the UI is very intrusive (it contains a bunch of spam from disqus) and in general it's probably not nice to send data to some third party whenever they read our articles.

To this end I think we should look into adding support for comments via some hosted alternative. I found this thing: https://github.com/posativ/isso/, that seems to be fairly well maintained and support the features we need (truly anonymous comments, minimal UI and self-hosted).

I would say we set this up and try it out on our post pages.

Add direct link to OONI Probe app apk

To overcome the fact that in some countries there is no Play Store.

We can possibly use a direct link to the f-droid apk or to the apk in the github release.

Add multilingual support

Since hugo 0.17 there is now multilingual support:
https://gohugo.io/content/multilingual/

We should add support for translating our site with it!

footer is missing on nettest pages

within any of the nettest description pages, the footer is missing:
https://ooni.torproject.org/nettest/

The same footer as on any other page on the website should be incorporated her as well

Add test specs to the website

xref: ooni/spec#41.

Change license to CC-BY-SA

We should add a footer to every page with the CC-BY-SA notice.
See: https://github.com/OpenObservatory/legal

Restructuring the OONI website

This is a draft.

Menu items

About

• What is OONI?
• How does OONI measure network anomalies?
• What are network anomalies?
• How and why do network anomalies occur?
• Why do network anomalies matter?
• How is OONI funded? End of main page
• Awards
• Privacy Policy (linkable)
• Team (add later)

Research

• OONI research paper: https://www.usenix.org/system/files/conference/foci12/foci12-final12.pdf
• Other projects relying on OONI data (who uses OONI?)
• Papers using OONI data

Measurements

• Explorer (including highlights)
• Raw measurements

Methodology

• Creation of data collection software based on various tests, each of which tests for different symptoms of network interference
• Creation of test lists for countries around the world
• Running the software/tests against country-specific test lists in various countries around the world
• Experiment-Control Methodology: Comparison of collected data with “control data” (this happens on the client side, before the data is collected)
• Collection and processing of data (measurements are sent to a Tor hidden service or to a HTTPS collector)
• Open Data: Making the data accessible through an API
• Data analysis
• Visualization of data (OONI explorer)

Documentation

• Developer Documentation

Code

• Github access to the code

Contribute

• Run ooniprobe
• Adopt a probe
• Use data
• Join the community (IRC + mailing list)

Reports

• EEEP and Greek Internet Censorship
• Zambia, a country under Deep Packet Inspection
• Tab-Tab, come in! Bypassing Internet blocking to categorize DPI devices
• Hadara Palestine
• T-Mobile USA Web Guard

Blog

• Short, weekly posts on anything around censorship, network anomalies, measurements, etc. --- Anyone can write here. Open to guest posts.

Events

• ADINA (upcoming)
• Berlin 2014 (past)

FAQ

For example:

• Is it safe to run OONI?
• Can I send results not over Tor hidden service?
• Can I run my own collectors? Yes.
• Can I write my own tests? Yes. This is how (link).
• I ran OONI. Where are my measurements?
• The test I am running is not working. What now?
• Does ooniprobe run on smartphones? Soon, yes!
• Which websites are tested?
• How do you decide which websites to include in test lists?
• Do results/measurements change across time?
• Why are websites blocked?
• Where can I find my country's test lists?
• Why are there no results for my country?
• How can I contribute to the test lists of my country?
• How can I contribute measurements for my country?
• Can I run ooni over Tor?
• Can I run ooni over a VPN? Sure, but more accurate measurements without (depends on the type of test). Learn more about tests here.
• Once I have run ooni, should I send the results to you?
• Can I avoid submitting results? Yes. You can do this by -n.

What can I do? How to circumvent censorship

• Tor
• Different ways depending on different types of blocking (explain them!)

Contact

• Email address ([email protected]? or [email protected]?) & PGP key
• Twitter: @OpenObservatory
• IRC: ..

PGP key have expired on March 2018

Expired on: 2018-03-26

Ref to #128

pub   4096R/6B2943F00CB177B7 2016-03-23 [expires: 2018-03-26]
      Key fingerprint = 4C15 DDA9 96C6 C0CF 48BD  3309 6B29 43F0 0CB1 77B7
uid       [ultimate] OONI - Open Observatory of Network Interference
sub   4096R/8EBD2087374399AB 2016-03-23 [expires: 2018-03-26]

Specify canonical URL for mirrors

The website has four mirrors listed in README.md. Google Search serves documents both from ooni.io and ooni.torproject.org

• define the canonical domain for URLs: ooni.torproject.org or ooni.io
• add markup to codify it