themarshallproject / klaxon Goto Github PK

If you're not signed into a Klaxon session, the bookmarklet will take over a page, but the right sidebar won't appear, so you can't save or close.
The preferred behavior would be for that sidebar to pop up with a signin box that would then reload the page (and bookmarklet) for you.

I had been watching an agency's site that had an HTTP URL. That site implemented HTTPS by default in late October. (Good on the government! :) But a distracting time for me.) The site would issue a 301 for HTTP requests to point to the same URL under HTTPS.

This confused Klaxon. After the redirect was set up, the response body was empty, so my selector didn't match, so the email I got from Klaxon showed only a deletion (no green text). This led to me not getting notification of changes, since the page Klaxon was watching really didn't change (since its response body was empty). It's an easy issue to fix for my particular case -- switch the URL to HTTPS -- but it may take a user a while to realize, especially if their site doesn't update approximately daily as mine did.

Now, you all have certainly thought more about the "theory" of Klaxon, so I don't know if this is expected behavior or not. I can totally see it both ways and want to defer to y'all's thinking.

But if y'all think this is a fixable problem, the RestClient gem will follow redirects automatically and I think RestClient.get would be a drop-in replacement for Net:HTTP.get here -- though behavior with respect to encoding might differ...

For users comfortable with git, it can be much easier for them to upgrade to the newest release.

One of our reporters set up an alert for the main content section of gawker.com so they could get an alert if and when it was ever updated or altered. It's been two days since then and Klaxon has sent three alerts for changes, yet the substance of all the alerts don't seem to indicate anything has changed.

I attached screenshots of all the alerts in question (each one appears to be more or less the same), and of the alert itself in the Klaxon control panel.

@arossback can tweak this. We probably don't need to move people to the email announcement list. Rather, they can go directly to Github to "Set it up in your newsroom."

The copy at the bottom of the page will need to be updated accordingly.

I believe @rdmurphy is working on this here. Not sure how much more there is to add, or if it should be referenced explicitly in the main README.

Tried to test the Heroku deployment and got an error message during the build that seems to indicate a missing table in the PostGres setup.

Here's the log:

Configure environment
Build app  Hide build log
-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.3.0
-----> Installing dependencies using bundler 1.11.2
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Fetching gem metadata from https://rubygems.org/............
       Fetching version metadata from https://rubygems.org/...
       Fetching dependency metadata from https://rubygems.org/..
       Using json 1.8.3
       Installing rake 11.1.2
       Installing i18n 0.7.0
       Installing minitest 5.8.4
       Installing builder 3.2.2
       Installing erubis 2.7.0
       Installing thread_safe 0.3.5
       Installing mini_portile2 2.0.0
       Installing mime-types-data 3.2016.0221
       Installing rack 1.6.4
       Installing arel 6.0.3
       Installing addressable 2.4.0
       Installing bcrypt 3.1.11 with native extensions
       Installing coffee-script-source 1.10.0
       Installing execjs 2.6.0
       Installing concurrent-ruby 1.0.1
       Installing thor 0.19.1
       Installing dotenv 2.1.1
       Installing diffy 3.1.0
       Installing htmlentities 4.3.4
       Installing multi_xml 0.5.5
       Installing jwt 1.5.4
       Installing kramdown 1.10.0
       Using bundler 1.11.2
       Installing pg 0.18.4 with native extensions
       Installing puma 3.2.0 with native extensions
       Installing rails_serve_static_assets 0.0.5
       Installing rails_stdout_logging 0.0.5
       Installing sass 3.4.21
       Installing tilt 2.0.2
       Installing rdoc 4.2.2
       Installing tzinfo 1.2.2
       Installing nokogiri 1.6.7.2 with native extensions
       Installing mime-types 3.0
       Installing rack-test 0.6.3
       Installing rack-cache 1.6.1
       Installing css_parser 1.3.7
       Installing coffee-script 2.4.1
       Installing uglifier 3.0.0
       Installing sprockets 3.5.2
       Installing httparty 0.13.7
       Installing rails_12factor 0.0.3
       Installing sdoc 0.4.1
       Installing activesupport 4.2.5.1
       Installing mail 2.6.4
       Installing rails-deprecated_sanitizer 1.0.3
       Installing premailer 1.8.6
       Installing globalid 0.3.6
       Installing activemodel 4.2.5.1
       Installing activejob 4.2.5.1
       Installing activerecord 4.2.5.1
       Installing rails-dom-testing 1.0.7
       Installing loofah 2.0.3
       Installing rails-html-sanitizer 1.0.3
       Installing actionview 4.2.5.1
       Installing actionpack 4.2.5.1
       Installing sprockets-rails 3.0.4
       Installing railties 4.2.5.1
       Installing actionmailer 4.2.5.1
       Installing premailer-rails 1.9.1
       Installing simple_form 3.2.1
       Installing coffee-rails 4.1.1
       Installing jquery-rails 4.1.1
       Installing sass-rails 5.0.4
       Installing rails 4.2.5.1
       Installing turbolinks 2.5.3
       Bundle complete! 27 Gemfile dependencies, 66 gems now installed.
       Gems in the groups development and test were not installed.
       Bundled gems are installed into ./vendor/bundle.
       Post-install message from rdoc:
       Depending on your version of ruby, you may need to install ruby rdoc/ri data:
       <= 1.8.6 : unsupported
       = 1.8.7 : gem install rdoc-data; rdoc-data --install
       = 1.9.1 : gem install rdoc-data; rdoc-data --install
       >= 1.9.2 : nothing to do! Yay!
       Post-install message from httparty:
       When you HTTParty, you must party hard!
       Bundle completed (30.67s)
       Cleaning up the bundler cache.
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       failed to set default_host, reason: PG::UndefinedTable: ERROR:  relation "app_settings" does not exist
       LINE 5:                WHERE a.attrelid = '"app_settings"'::regclass
       ^
       :               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
       pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
       FROM pg_attribute a LEFT JOIN pg_attrdef d
       ON a.attrelid = d.adrelid AND a.attnum = d.adnum
       WHERE a.attrelid = '"app_settings"'::regclass
       AND a.attnum > 0 AND NOT a.attisdropped
       ORDER BY a.attnum
       I, [2016-05-04T15:36:03.132671 #1086]  INFO -- : Writing /tmp/build_4a7006eeb7404807023d62e92fb2b7a9/themarshallproject-klaxon-fe46e0c/public/assets/klaxon-logo-100px-c4c978b074949e6706cc0bc17d59e0260b7ee3c22c7249ae0cb743e7c2cd9501.png
       I, [2016-05-04T15:36:05.879619 #1086]  INFO -- : Writing /tmp/build_4a7006eeb7404807023d62e92fb2b7a9/themarshallproject-klaxon-fe46e0c/public/assets/application-ce4b02c94ce9d4bc18d4b7e67a7a9de3e1f5a04605e8ffd4d03320b35ced23a0.js
       I, [2016-05-04T15:36:05.879912 #1086]  INFO -- : Writing /tmp/build_4a7006eeb7404807023d62e92fb2b7a9/themarshallproject-klaxon-fe46e0c/public/assets/application-ce4b02c94ce9d4bc18d4b7e67a7a9de3e1f5a04605e8ffd4d03320b35ced23a0.js.gz
       I, [2016-05-04T15:36:10.176761 #1086]  INFO -- : Writing /tmp/build_4a7006eeb7404807023d62e92fb2b7a9/themarshallproject-klaxon-fe46e0c/public/assets/application-aedc3aa66c9871bda1986138e49412820f788b214b419e9cf495af594c3d6401.css
       I, [2016-05-04T15:36:10.176966 #1086]  INFO -- : Writing /tmp/build_4a7006eeb7404807023d62e92fb2b7a9/themarshallproject-klaxon-fe46e0c/public/assets/application-aedc3aa66c9871bda1986138e49412820f788b214b419e9cf495af594c3d6401.css.gz
       Asset precompilation completed (8.90s)
       Cleaning assets
       Running: rake assets:clean
       failed to set default_host, reason: PG::UndefinedTable: ERROR:  relation "app_settings" does not exist
       LINE 5:                WHERE a.attrelid = '"app_settings"'::regclass
       ^
       :               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
       pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
       FROM pg_attribute a LEFT JOIN pg_attrdef d
       ON a.attrelid = d.adrelid AND a.attnum = d.adnum
       WHERE a.attrelid = '"app_settings"'::regclass
       AND a.attnum > 0 AND NOT a.attisdropped
       ORDER BY a.attnum
-----> Discovering process types
       Procfile declares types     -> web
       Default types for buildpack -> console, rake, worker
-----> Compressing...
       Done: 33.9M
-----> Launching...
       Released v7
       https://klaxon-test.herokuapp.com/ deployed to Heroku

Let me know if we're approaching config overload, but should config.time_zone also be an ENV variable?

Just noticed our version is an hour ahead. 😁

Sometimes a page changes URL or design and then you never hear from it again.
It would be good to get a notice if something hasn't updated in a month, or maybe 45 days, to alert a user to check if the page they're watching has moved rather than failing silently.

Maybe I am just not finding the button, but is there no way of deleting a site from the watch list?
I had to delete using SQL query.
Looks like there is a destroy endpoint https://github.com/themarshallproject/klaxon/blob/master/app/controllers/pages_controller.rb#L68-L72, but maybe just button missing?
My Ruby knowledge is extremely limited, so no pull request for now :)

(BTW great project!)

@ivarvong had incorporated this into previous versions of Klaxon and likely has some thoughts on how it might be accomplished here.

Does Klaxon seem like a thing that should always turn away spiders? I know the login will stop them, but I could see the argument for never announcing this exists to search engines.

Should this section on how to merge new changes into your Klaxon instance be absorbed into the main user help within the app?
This was a suggestion we got from our former intern Emily Hopkins.

ReferenceError: event is not defined      inject:98:9

I assume that the solution is just to add event as an argument here:
https://github.com/themarshallproject/klaxon/blob/master/app/views/embed/inject.js.erb#L97

but I'm not sure.

Worked fine in Chrome on my install.

Hey y'all,

I've noticed wonky behavior when trying to set up an alert for a table. Often, the only visible space in the table is a bunch of elements. When I set up an alert using tr as the selector (possibly with additional context, like .whatever #whatever tr), the output looks pretty bad. If I fix it to be table, everything works perfectly. I wonder if it'd be worthwhile to special-case the thing that generates CSS selectors to automatically select the parent table if the user has clicked on a tr.

(I guess the problem here is that the user's click is ambiguous -- do they want to watch the whole table, that one row, or that one cell? But that's a theoretical problem.)

I should clarify that I haven't tested this and there may be problems I'm not foreseeing. Obviously I defer to y'all if you've thought about this case. And, at the risk of repeating myself, I'm a big fan of Klaxon and thank you all for your work! Klaxon works super well and we're using it a ton. :)

Per a report from @jeremybmerrill:

if you’re trying to watch a second thing on the same page, it appears that the Klaxon bookmarklet edits the first, rather than (as this user expected) creating a second one.

The current workaround is to manually add a second watcher, but we should think about how the bookmarklet should handle multiple alerts for a single page.

What can we do to make the documentation for users and for developers/maintainers more clear and easier to use? We're hoping our new community can bring fresh eyes to these and offer some suggestions (and pull requests).

This is an undocumented feature on the Watching page. Should we explain to people how to use it?

You should be able to destroy a klaxon from its edit page rather than just the watch page.

SendGrid is very slow to deliver emails. This is terrible UX for a login email. This is apparently a matter of policy for them: https://sendgrid.com/docs/Classroom/Send/When_Emails_Are_Sent/seeing_a_delay_with_your_first_few_sends.html

Should we be using Postmark or something similar that doesn't have this drawback?

After I'm already watching items, it should no longer say "watch your first item".

That box also appears at the top of the help page for some reason.

What would it take for Klaxon to integrate with an arduino that gave audio or strobe light alerts when an update is detected?

We've chosen Heroku to make this as easy as pie, and some contributors have added Docker support. Where else would you like to deploy this? Can you offer the setup scripts to share with similar users?

Twitter feed or RSS were two ideas we had batted around. SMS? What other alerts would you like to get?

I think this would be doable by adding additional ENV VARS here to set the SMTP hostname and SMTP port; this would, I think, address the question of the Sendgrid dependency out of #8.

Shouldn't overlap with content when scrolling down.

Message currently says: "You're recieving this email because you just tried to log into your Klaxon account." (emphasis mine)

Not 100% sure if the bug is on my end or not — I was trying to set up Klaxon to use the Tribune's SES account, and it threw an unverified email address error.

[ActiveJob] [ActionMailer::DeliveryJob] [f9740ecf-5e14-477d-932d-418a9e4bfaa2] Performed ActionMailer::DeliveryJob from Inline(mailers) in 1621.73ms
Completed 500 Internal Server Error in 1694ms (ActiveRecord: 6.3ms)

Net::SMTPFatalError (554 Message rejected: Email address is not verified.
):
  app/controllers/sessions_controller.rb:11:in `create'

The error makes sense — our SES account hasn't verified [email protected], so it turns it away. Should it be possible to set the email address to use? Is that something else that can be passed into ActionMailer's config? I was able to get it to work by switching out that line with the email I originally tried.

I'm trying to evaluate Klaxon for possible use in my newsroom, and I've been seeing some weird behavior from the check:all Rake task: It sometimes detects too many changes (both the addition of content and its removal, at the same time) or ends up with a rollback in the middle of looking for changes. (Happy to split this up into two issues if you'd prefer.)

More details below. Would love to figure out what's going on here, because Klaxon does seem really useful!

Change reversal

I tried tracking new posts on the home page of Source. (I think the selector was body div#snap-content-wrapper.snap-content.overthrow div.content div.container article.homepage.)

When I did this, I sometimes would get two emails at roughly the same time (i.e., not from separate rake check:all runs): One email with a new top post (and a corresponding post getting bumped to the next page), and another email with roughly the inverse of that change (i.e., the new top post being removed and the old bumped post coming back), and not necessarily in that order.

Two examples:

These are weird, of course, because they aren't perfect inverses, but they seem to be happening at the same time regardless.

I don't have logs for these incidents, and since I've kept the scheduler task running, they're back beyond the 1,500-line maximum I can get from heroku logs.

Unexplainable rollbacks

I also tried watching some subreddits and saw some odd notes in my Heroku logs (again, don't have those), so I decided to try to spin up the app on my laptop instead so I could more easily observe/control what's going on.

It turns out my selectors didn't work, so no changes were ever going to be detected (more details below), but oddly some pages never get that far and log something about rolling back some sort of database transaction:

$ bundle exec rake check:all
  AppSetting Exists (0.4ms)  SELECT  1 AS one FROM "app_settings" WHERE "app_settings"."key" = $1 LIMIT 1  [["key", "default_host"]]
  AppSetting Load (0.3ms)  SELECT  "app_settings".* FROM "app_settings" WHERE "app_settings"."key" = $1 LIMIT 1  [["key", "default_host"]]
setting default_host to localhost:5000
  Page Load (0.4ms)  SELECT "pages".* FROM "pages"
downloading url=https://www.reddit.com/r/dataisbeautiful/new/ for page.id=1
  PageSnapshot Load (0.8ms)  SELECT  "page_snapshots".* FROM "page_snapshots" WHERE "page_snapshots"."sha2_hash" = $1 AND "page_snapshots"."page_id" = 1 LIMIT 1  [["sha2_hash", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]]
   (0.2ms)  BEGIN
  SQL (0.3ms)  UPDATE "page_snapshots" SET "updated_at" = '2016-05-03 14:50:16.574080' WHERE "page_snapshots"."id" = $1  [["id", 1]]
   (0.4ms)  COMMIT
page 1 has not changed
downloading url=https://www.reddit.com/r/aww/new/ for page.id=2
  PageSnapshot Load (0.4ms)  SELECT  "page_snapshots".* FROM "page_snapshots" WHERE "page_snapshots"."sha2_hash" = $1 AND "page_snapshots"."page_id" = 2 LIMIT 1  [["sha2_hash", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]]
   (0.1ms)  BEGIN
  PageSnapshot Exists (0.3ms)  SELECT  1 AS one FROM "page_snapshots" WHERE "page_snapshots"."sha2_hash" = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' LIMIT 1
   (0.5ms)  ROLLBACK
  Page Load (0.3ms)  SELECT "pages".* FROM "pages"
  PageSnapshot Load (0.3ms)  SELECT "page_snapshots".* FROM "page_snapshots" WHERE "page_snapshots"."page_id" = $1  [["page_id", 1]]
  PageSnapshot Load (0.3ms)  SELECT "page_snapshots".* FROM "page_snapshots" WHERE "page_snapshots"."page_id" = $1  [["page_id", 2]]

The page hashes shown in the logs here match the SHA-256 for an empty string, suggesting my selectors don't work for the pages as Klaxon downloaded them, which makes sense because my selectors (picked via the bookmarklet) suggest a logged-in user with Reddit Gold. (For /r/dataisbeautiful, I have body.subscriber.listing-page.loggedin.gold.hot-page div.content div.spacer, and for /r/aww, I have body.subscriber.listing-page.loggedin.new-page.gold div.content div.spacer div#siteTable.sitetable.linklisting.)

2.3.0 :001 > require "digest"
 => true
2.3.0 :002 > Digest::SHA256.hexdigest('')
 => "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

Using a config variable on the Heroku page, allow users to customize the name of their Klaxon instance so it's clear to users it is owned by their newsroom.

This could appear in the <title> tag or the header could say "Klaxon at the Marshall Project", for instance, instead of just "Klaxon".

If we do do that, we should allow users to customize this through the settings dropdown as well.

What version should this be, @ivarvong?

This is on the feed page. It seems to be showing today's date instead...

We got feedback from our user testing that some people found the use of the bookmarklet confusing. Some people weren't familiar with CSS selectors or html and weren't quite sure what exactly they were selecting. Other users didn't really know what a bookmarklet is or how it works.

How can we improve the user documentation to make this powerful feature easier to grasp and use for new users?

if a url starts with a space, the watching page breaks because there is a parse error on the url. example:

016-06-24T15:41:54.101794+00:00 app[web.1]:   Rendered pages/index.html.erb within layouts/application (38.7ms)
2016-06-24T15:41:54.102002+00:00 app[web.1]: Completed 500 Internal Server Error in 45ms (ActiveRecord: 16.7ms)
2016-06-24T15:41:54.103252+00:00 app[web.1]:
2016-06-24T15:41:54.103267+00:00 app[web.1]: ActionView::Template::Error (Invalid scheme format:  https):
2016-06-24T15:41:54.103269+00:00 app[web.1]:     40:         <tr>
2016-06-24T15:41:54.103269+00:00 app[web.1]:     41:           <td><%= page.created_at.strftime("%m/%d/%Y at %I:%M%p") %></td>
2016-06-24T15:41:54.103272+00:00 app[web.1]:     42:           <td><%= page.name %></td>
2016-06-24T15:41:54.103273+00:00 app[web.1]:     43:           <td><%= page.domain.truncate(40) %></td>
2016-06-24T15:41:54.103273+00:00 app[web.1]:     44:           <td><code><%= page.css_selector %></code></td>
2016-06-24T15:41:54.103274+00:00 app[web.1]:     45:           <td><%= page&.user&.display_name %></td>
2016-06-24T15:41:54.103275+00:00 app[web.1]:     46:           <td><% if page.page_snapshots.count > 1 %><%= link_to "Latest Change", latest_change_page_path(page), class: "btn btn-default btn-block" %><% end %></td>
2016-06-24T15:41:54.103276+00:00 app[web.1]:   app/models/page.rb:16:in `domain'
2016-06-24T15:41:54.103277+00:00 app[web.1]:   app/views/pages/index.html.erb:43:in `block in _app_views_pages_index_html_erb__2887604200305143239_70290127052820'
2016-06-24T15:41:54.103277+00:00 app[web.1]:   app/views/pages/index.html.erb:39:in `_app_views_pages_index_html_erb__2887604200305143239_70290127052820'
2016-06-24T15:41:54.103278+00:00 app[web.1]:
2016-06-24T15:41:54.103279+00:00 app[web.1]:

after removing the leading whitespace, everything worked again. can we trim urls and/or add some graceful fallback for assuming all URLs can parse correctly?

The diff is much easier to read on the email alert than in the change page in Klaxon itself. I wonder if we limit the width of the diff and force it to wrap if that wouldn't make it more accessible.

on the site

on email

The change page for a Klaxon alert states how many times a page has changed, like this:

But when I look at the number of snapshots, I see this:

That would say to me that it has changed one time since I started watching it, with the first snapshot being the baseline. Am I misunderstanding that, or is the first snapshot actually the first change and we're missing the baseline?

This is related to #74, I suspect. In the emails, it's giving us the wrong number of changes by counting the original, baseline snapshot.

Once you've activated the Klaxon bookmarklet on a page you want to watch, you can't get rid of it and stay on the page without reloading the page. Should we add a cancel/close button?

We could handle this a number of ways. It'd be nice to have a visual cue in the app (maybe a badge next to the help button?) to alert users to a new tagged release and to nudge them to upgrade.

Insert this at end of 2nd graf:

[If you need help using Klaxon, [you can find it here.](https://github.com/themarshallproject/klaxon/blob/master/data/help.md)]

Klaxon is about making it dead simple for someone to watch a basic HTML page, the kind many government websites still use. The feed, the alerts and the snapshot comparisons are all very useful for this. But I could imagine some people might like to add more complex scrapers of their own devising but still make use of the alerts and the feeds.
What would this entail?

We should recognize everyone who has contributed to the project in code, documentation or other support.

As best I can tell, there's no way to remove a page from the list of monitored sites within the GUI.

I've made do by clearing the URL, and I suppose I could delete the entry by jumping into the database, but am I missing something?

It's pretty confusing how the selector continues to update after you have clicked to select an element. There should be some visual indication of what the selected element is, or something similar.