GithubHelp home page GithubHelp logo

global19-atlassian-net / tracker-radar-detector Goto Github PK

View Code? Open in Web Editor NEW

This project forked from duckduckgo/tracker-radar-detector

0.0 2.0 0.0 1.51 MB

Code used to build a Tracker Radar data set from raw crawl data.

License: Other

JavaScript 100.00%

tracker-radar-detector's Introduction

DuckDuckGo Tracker Radar Detector

This is the code used to build a Tracker Radar data set using crawl data from the Tracker Radar Collector.

Getting Started

To generate a Tracker Radar data set follow these steps:

  1. Clone the Tracker Radar data repo

  2. Generate 3rd party request data using the Tracker Radar Collector

  3. Update the paths in config.json to point to your newly created crawler data files and the location of your Tracker Radar data repository

trackerDataLoc path to your Tracker Radar data repository
crawlerDataLoc path to your crawler data directory
performanceDataLoc path to your performance crawler data

Generating Tracker Radar data

  • Install dependencies

npm install

  • Build site performance summary (optional)

npm run build-performance

  • Update entity data (optional) note: requires some manual validation of the output data, see here for more info
npm run update-entities
npm run apply-entity-changes
  • Build Tracker Radar data files

npm run build

Note that if you wish to resolve CNAME's, node version 12+ is required. You can disable CNAME resolution by setting the option treatCnameAsFirstParty=true and keepFirstParty=false in the config file.

Postgresql data source

Crawler data can also be read from a PostgreSQL database. To enable this, set the crawlerDataLoc to postgres, and set the crawlId and region options in config.json. Database details should be provided via environment variables, for example with envdir:

envdir /etc/ddg/dbenv/tracker_radar_readonly/ npm run build

See the node-postgres documentation for more details on connection options.

Contributing

Reporting bugs

  1. Check to see if the bug has not already been reported
  2. Create a bug report issue

New features

Right now all new feature development is handled internally.

Bug fixes

Most bug fixes are handled internally, but we will accept pull requests for bug fixes if you first:

  1. Create an issue describing the bug.
  2. Get approval from DDG staff before working on it. Since most bug fixes and feature development are handled internally, we want to make sure that your work doesn't conflict with any current projects

Questions or help with anything else DuckDuckGo related?

See DuckDuckGo Help Pages.

This software is licensed under the terms of the Apache License, Version 2.0 (see LICENSE).

tracker-radar-detector's People

Contributors

jdorweiler avatar kdzwinel avatar charlie-belmer avatar sammacbeth avatar slayterdev avatar dharb avatar ladamski avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.