ubclaunchpad / sleuth Goto Github PK

Keep improving our general page parser, particularly its versatility and improving the quality of the data we keep

Allow users to submit sites to add to our crawl list

Create an input field on the front end that, when submitted, makes a search request to our backend and receives the results.

This ticket is just to implement something that uses Solr (or something better?) to return a list of suggestions given some query. In certain cases we should return suggestions as part of the results.

What data do we want to store?

Define some sort of model for Solr and have Scrapy pipeline organize item data into appropriate format

e.g. SolrConnection

Update for new changes from #62 and #66 , documentation here

courseItem: ["2017-10-28T01:46:41Z"]
genericPage: ["2017-10-28 01:28:28"]

Possibly relevant code:

• pipeline
• courseItem
• genericPage

Nothing seems wrong at first glance

Be able to extract a set of results from the search API response and display them on the front end

Ideally we should display the results as a tree, but if we don't have time to implement this by the end of this sprint, we can do a simple list view like Google.

dependent on #63

This should use the existing suggestion functionality to stream search suggestions to the client as they type their query into the search input field. It might be a good idea to do this over websockets, since the query will be changing quickly as the user types, but there's also a good chance it won't be necessary.

Also start researching how React works

• Results show up as bubbles, connected to all linked children
• Results can be deleted, and children disappear with it
• Children can be "focused" to pull up all its children
• Hovering over result shows blurb, other details etc
  

• Initialize Django project (this will probably be the first commit to our repo)
• Come up with a few simple use cases for the backend
• Create high level design for backend service

sklearn

Set up the simple Django source template in our repo under a folder called "backend"

Also update search endpoint for these new fields

• Set up DB
• Connect backend to the DB
• Research Solar integration with backend service

On our React frontend, define a input field that makes a call to /search on our Sleuth API and displays the results, or displays errors if there were any.

• Come up with a few simple user stories for what the front end should do
• Initialize front end framework (React)
• Research React and create a high level design for how we'll modularize/develop the front end

This mostly exists as an exercise in React

We want the crawler to be able to:

• Connect to a generic DB (like Postgres) and insert web page info/content info the DB
• Access web pages and download their content
• Jump from one page to another
• Connect to Solar and insert content
• Maybe process data and connect directly to the DB, or just give raw data straight to the backend

ReactThree