GithubHelp home page GithubHelp logo

jvasile / planeteria Goto Github PK

View Code? Open in Web Editor NEW
7.0 3.0 7.0 2.05 MB

Multiple Blog Planet Site

Home Page: http://planeteria.org

License: Other

Python 27.56% D 0.01% CSS 46.98% JavaScript 25.32% Shell 0.13%

planeteria's Introduction

Planeteria Readme

About Planeteria

A planet is a collection of posts from many different blogs, all somewhat related to one topic. It's a great way to keep tabs on a subject, a community, a technology, a team, a project or anything else that attracts a diverse range of bloggers.

Planeteria.org is a hosted planet reader. Planeteria allows anyone to make a planet, host it and administer it on Planeteria.org. After you make your planet, tell people about it. That's the whole point of community-building!

Planeteria was created by James Vasile (james at hackervisions dot net) in 2010 and has been maintained in his copious spare time since its creation.

Command Line Options to planeteria.py

Planeteria is run as a command line tool (planeteria.py) and two CGI scripts (admin.py and new_planet.py). The command line tool is used as follows:

Usage: planeteria.py [options]

Options:

  • -h, --help show this help message and exit
  • --force force downloading of cached upstream feeds
  • --no-update prevent downloading of upstream feeds
  • --delete-missing delete planets from db if they are not in file system
  • --clean remove missing planets, unused feeds

The --clean flag doesn't remove old posts from the cache (just unused feeds), and it doesn't delete unused planets, but in a future version, it will.

About Planeteria's code

This is a free open source project licensed under the AGPLv3 license.

This software was originally built on Venus. Venus is great if you're only running one planet, I highly recommend it. However trying to scale up the code for a single planet to make one system do many planets proved to be tricky, so those dependencies were discarded for more flexibility. The code is less robust as a result and could use some love.

In 2013, Aleta Dunne (aleta dot dunne at gmail dot com) took on fixing up the site as an internship project for the GNOME Outreach Program for Women (OPW). We always welcome more help, though!

Please let us know about bugs on the Github issue tracker, or submit a patch!

We have a roadmap in ROADMAP.md, please ping us if you want to tackle any of those!

Unit Tests

There are some tests in the test directory. Run them from the main directory with test/nose.sh . and you will see the start of some unit testing. Please don't use the tests on a live, deployed install. They create and destroy test planets.

The tests include tests of the web functionality. To run those tests, you'll want to do be logged in as the same user that your apache instance runs under (try sudo su www-data; bash). Otherwise, there is a permissions mismatch when the user running the tests tries to delete a file created by the web server and the 644 permissions prevent it. If you get permission denied errors when running nosetests, this might be the problem.

If you want to see the debug output while running nosetest, do tail -f log/planeteria.log in another screen or console.

Once you have complete coverage of everything except util.py, any uncovered functions in util.py can be moved to util_unused.

Installation

Planeteria is compatible with Python versions 2.3 - 2.7. For an overview of its components, see doc/overview.dia.

Grab the code from the master repo:

git clone git://github.com/jvasile/Planeteria.git

Dependencies

Planeteria depends on several libraries. Follow the instructions for your OS below. We've only tested the installation on Debian and Mac OS X so far; we welcome any contributors who would like to help develop installation instructions for other operating systems.

We recommend installing these in a virtual environment (virtualenv) to avoid conflicts with dependencies for other projects run on the same machine. To learn more about installing and using virtualenv, go to http://www.virtualenv.org/en/latest/

Package installation on Debian To install them on Debian, you can

get them all by typing (or pasting) the following into a terminal window:

aptitude install python-feedparser python-utidylib python-simplejson \
python-beautifulsoup python-lxml python-htmltmpl python-dateutil

Package installation on Mac OS X

Most packages can be installed on a Mac with pip, however one of them needs to be downloaded before running pip. Go to http://sourceforge.net/projects/htmltmpl/files/htmltmpl/ and download the 1.22 version (dated 2001-12-17).

WARNING: Do not click the link at the top of the page that says "Looking for the latest version?" Although the version number seems to be the correct version number, the link will actually download htmltmpl 1.22 for PHP, not the Python version which is what this project uses. The download should have the file name htmltmpl-1.22.tar.gz; you should not see php in the filename.

Once you have downloaded the htmltmpl tarball, note the file path for the pip install. Copy and paste the following command into a terminal window, and before you hit Enter, replace path/to/htmltmpl-1.22.tar.gz with the file path on your system.

pip install feedparser pytidylib simplejson beautifulsoup lxml python-dateutil path/to/htmltmpl-1.22.tar.gz

The last step for Mac users is to install tidy, which is required for pytidylib to work. We used Homebrew to install tidy but it requires an extra step because it's not in the default Homebrew repository. First, enter

brew tap homebrew/dupes  

which tells homebrew to look in the dupes library, then:

brew install homebrew/dupes/tidy  

which installs tidy.

Loading the site on your machine

Note that without running it on an Apache server, form submission won't work, so you won't be able to create or administer a planet, however you can view the static pages just fine.

Run planeteria.py which should generate the html files for the site and place them in the /www folder. Then open /www/index.html in a browser window.

Running the site on an Apache server virtual host

The site requires sqlite3.

Permissions

Make sure that the Planeteria directory is accessible by the web server (read/execute permission). You also need to give the server write permission for the following directories:

/data
/www
/log

Server Configuration

For those setting up a virtual host on a Mac, this site walks you through the process in more detail than described below.

In your /etc/hosts file, add a new line underneath 127.0.0.1 localhost that says

127.0.0.1 planeteria.local

In your httpd.conf file, make sure it points to the /extra/httpd-vhosts.conf file like below. Verify the file path!

# Virtual hosts
Include /private/etc/apache2/extra/httpd-vhosts.conf

(The second line is commented out by default.)

In your /extra/httpd-vhosts.conf file, add the following settings:

<VirtualHost *:80>
    ServerName  planeteria.local (it should match the server name in your /etc/hosts file)
    ServerAdmin [email protected]
    DocumentRoot "/path/to/Planeteria/www"
    ErrorLog ${APACHE_LOG_DIR}/planeteria-error.log
    TransferLog ${APACHE_LOG_DIR}/planeteria-access.log
    LogLevel debug
</VirtualHost>

<Directory "/path/to/Planeteria/www/">    
    Options +ExecCGI +FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
    AddHandler cgi-script cgi py pl
</Directory>

Make sure to replace /path/to/Planeteria with the full file path to the location of the cloned Planeteria repo. Be especially attentive to the trailing slashes and quotation marks around the file paths, they may vary from machine to machine and can make the difference of whether the server can find the root directory or not.

Once your settings are saved, reboot the server and follow the directions above to load the site.

Set base href

You need to tell Planeteria what domain and directory it lives in by creating a data/base_href file with the domain. It must start with http://. On the server which runs the site, that file contains http://planeteria.org. On Aleta's virtual host, the base_href file contains http://planeteria.local which just adds http:// to the server name used in her /etc/hosts file.

Automatic updates

In a sandbox environment, you can choose to generate the html files as needed by running planeteria.py in the Terminal. However if you want to deploy the site on a server to run on its own, you will need to set up a cron job to run planeteria.py automatically every so often. Planeteria.org runs it every 15 minutes.

To set this up, add a line like this to your crontab:

15 * * * * cd /path/to/Planeteria; ./planeteria.py

Coders

 git log --pretty='format:%an'|sort|uniq -c|sort --reverse

 457 James Vasile
   3 jvasile
  39 dtiburon
   1 Wraithan (Chris McDonald)
   1 Wacky Hacky
   1 Sumana Harihareswara

planeteria's People

Contributors

brainwane avatar dtiburon avatar jvasile avatar wackyhackypdx avatar wraithan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

planeteria's Issues

Feature Request: Automatic Image Resizing

It would be nice if Planeteria displayed all the images for the feeds with the same width. Some are way too large and it seems to display them with the actual image size right now. 150 px seems to work nicely. If they're much smaller, we should leave them be since making them bigger could make them look really fuzzy/grainy, depending on the image quality.

Error on saving a newly added feed

I'm using Firefox and MacOSX 10.8.2. I just added a new blog to a feed, and got this error:

<type 'exceptions.AttributeError'> Python 2.7.3rc2: /usr/bin/python
Wed Mar 27 21:11:35 2013

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.
/var/www/planeteria2/www/feministhackers/admin.py in ()
215 a = Admin(template_vars(planet, Form)).render()
216 sys.stdout.write(a)
217
218 if name == "main":
=> 219 main()
main =
/var/www/planeteria2/www/feministhackers/admin.py in main()
196 if Form.has_key('PlanetName'):
197 orig_pass = planet.password
=> 198 planet = update_config(planet)
199
200 if Form.getvalue('Timestamp') != str(planet.last_config_change):
planet = <planet.Planet instance>, global update_config =
/var/www/planeteria2/www/feministhackers/admin.py in update_config(planet=<planet.Planet instance>)
135 urls_seen = []
136 while (Form.has_key('section%d' % feed_count)):
=> 137 url = Form.getvalue('feedurl%d' % feed_count,'').strip()
138 urls_seen.append(url)
139 if not url:
url undefined, global Form = FieldStorage(None, None, [MiniFieldStorage('Time..._'), MiniFieldStorage('Pass', '_')]), Form.getvalue = <bound method FieldStorage.getvalue of FieldStor...****'), MiniFieldStorage('Pass', '****')])>, feed_count = 0, ).strip undefined

<type 'exceptions.AttributeError'>: 'list' object has no attribute 'strip'
args = ("'list' object has no attribute 'strip'",)
message = "'list' object has no attribute 'strip'"

</strong> tag doesn't work, entire feed is bold

This may be related to [this issue https://github.com//issues/10] or may not be.

I noticed today that a post from wordpress.com had used <strong> tags but the </strong> tag is not honored so the text continues to be bolded, not only through the end of that entry, but into all the feeds displayed below it!
Screen Shot 2013-01-28 at 2 22 06 PM

I checked the HTML code in that feed and the following feed to confirm that it didn't coincidentally have an un-closed <strong> tag at the beginning of the following entry.

The entry where this starts appears like this on Planeteria:
Screen Shot 2013-01-28 at 2 21 41 PM

But it looks like this in her own blog:
Screen Shot 2013-01-28 at 2 21 24 PM

The HTML code from her blog is:

<p><strong>2nd reading, wherein Paul makes the famous &#8220;there are many parts, but one body&#8221; analogy</strong></p>
<p><strong></strong>Perhaps this story is new to you;

Notice that there is an empty pair of strong tags, but they are definitely closed (a WYSIWYG artifact). However, all the text that appears afterward is still bold on Planeteria. The HTML of the Planeteria.org page looks like this:

<p><strong>2nd reading, wherein Paul makes the famous &#8220;there are many parts, but one body&#8221; analogy</strong></p>
<p><strong/>Perhaps this story is new to you;

Here, the empty pair of strong tags have been changed to <strong/> for some reason. The slash is on the wrong side, so understandably the browser thinks it's an open strong tag.

Mono site not updating

I have added three new feeds to the mono planeteria and they have not updated after over 12 hours.

Publish latest 50 and then also the new posts

We currently publish the newest 50 posts, but that leaves some dormant feeds with nothing represented. We should do 50 + the latest for every feed that isn't represented in the newest 50.

Feature request: Improve speed of planet updates

Currently there's a cron job that updates the planets that updates every 15 mins or so. It would be nice if it could selectively update when a feed is added/changed so there's no wait after a change is made.

Unescaped characters in Admin Name field cause issues in Atom feed reader

From a user:
I can't open the Women In Free Software with my feed reader (NewsFox)
because it claims that the feed isn't well formed. The WC3 Feed
Validation Service
complains about an unescaped "&":

This feed does not validate.
line 7, column 24: XML parsing error: :7:24: not
well-formed (invalid token) [help]

     <name>James Vasile & Aleta Dunne</name>

This was something I (Aleta) recently edited in the Admin Name field under Planet Config. As a short-term solution, I replaced the ampersand with the word "and".

To do: Add escaping to characters in that field.

<type 'exceptions.AttributeError'> while adding a feed

Feed URL: http://fredjoiner.wordpress.com/feed/atom/

Error

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.

 /var/www/planeteria2/www/allarounddc/admin.py in ()
    215    a = Admin(template_vars(planet, Form)).render()
    216    sys.stdout.write(a)
    217 
    218 if __name__ == "__main__":
=>  219    main()
main = <function main>
 /var/www/planeteria2/www/allarounddc/admin.py in main()
    196    if Form.has_key('PlanetName'):
    197       orig_pass = planet.password
=>  198       planet = update_config(planet)
    199 
    200       if Form.getvalue('Timestamp') != str(planet.last_config_change):
planet = <planet.Planet instance>, global update_config = <function update_config>
 /var/www/planeteria2/www/allarounddc/admin.py in update_config(planet=<planet.Planet instance>)
    135    urls_seen = []
    136    while (Form.has_key('section%d' % feed_count)):
=>  137       url = Form.getvalue('feedurl%d' % feed_count,'').strip()
    138       urls_seen.append(url)
    139       if not url:
url undefined, global Form = FieldStorage(None, None, [MiniFieldStorage('Time...d3.jpeg'), MiniFieldStorage('Pass', '[censored]')]), Form.getvalue = <bound method FieldStorage.getvalue of FieldStor...3.jpeg'), MiniFieldStorage('Pass', '[censored]')])>, feed_count = 0, ).strip undefined
<type 'exceptions.AttributeError'>: 'list' object has no attribute 'strip' 
      args = ("'list' object has no attribute 'strip'",) 
      message = "'list' object has no attribute 'strip'"

Admin front end should populate via OPML import

Whenever possible, we should be emitting and accepting standard blocks. Currently, we populate the feeds in the admin interface by prepopulating a javascript function that formats the feed rows in a table. We should instead, grab the opml file, parse it and pass that to the display function.

This issue depends on issue #28.

New planet creation gives an error - can't find the log file

Creating a new planet from the main page goes to http://planeteria.org/new_planet.py and results in this error:

<type 'exceptions.IOError'> Python 2.7.3rc2: /usr/bin/python
Wed Mar 27 22:10:04 2013

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.
/var/www/planeteria2/www/new_planet.py in ()
16 cgitb.enable()
17
=> 18 from config import *
19 log = logging.getLogger('planeteria')
20 from util import Msg
config undefined
/var/www/planeteria2/config.py in ()
57 logger = logging.getLogger('planeteria')
58 logger.setLevel(logging.DEBUG)
=> 59 fh = logging.FileHandler(os.path.join(opt['log_dir'], 'planeteria.log'), encoding = "UTF-8")
60 fh.setLevel(logging.DEBUG)
61 fh_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh undefined, logging = None, logging.FileHandler undefined, os = None, os.path undefined, opt = None, encoding undefined
/usr/lib/python2.7/logging/init.py in init(self=<logging.FileHandler object>, filename='/var/www/planeteria2/log/planeteria.log', mode='a', encoding='UTF-8', delay=0)
895 self.stream = None
896 else:
=> 897 StreamHandler.init(self, self._open())
898
899 def close(self):
global StreamHandler = <class 'logging.StreamHandler'>, StreamHandler.init = , self = <logging.FileHandler object>, self._open = <bound method FileHandler._open of <logging.FileHandler object>>
/usr/lib/python2.7/logging/init.py in _open(self=<logging.FileHandler object>)
916 stream = open(self.baseFilename, self.mode)
917 else:
=> 918 stream = codecs.open(self.baseFilename, self.mode, self.encoding)
919 return stream
920
stream undefined, global codecs = <module 'codecs' from '/usr/lib/python2.7/codecs.pyc'>, codecs.open = , self = <logging.FileHandler object>, self.baseFilename = '/var/www/planeteria2/log/planeteria.log', self.mode = 'a', self.encoding = 'UTF-8'
/usr/lib/python2.7/codecs.py in open(filename='/var/www/planeteria2/log/planeteria.log', mode='ab', encoding='UTF-8', errors='strict', buffering=1)
879 # Force opening of the file in binary mode
880 mode = mode + 'b'
=> 881 file = builtin.open(filename, mode, buffering)
882 if encoding is None:
883 return file
builtin file = <type 'file'>, global builtin = <module 'builtin' (built-in)>, builtin.open = , filename = '/var/www/planeteria2/log/planeteria.log', mode = 'ab', buffering = 1

<type 'exceptions.IOError'>: [Errno 2] No such file or directory: '/var/www/planeteria2/log/planeteria.log'
args = (2, 'No such file or directory')
errno = 2
filename = '/var/www/planeteria2/log/planeteria.log'
message = ''
strerror = 'No such file or directory'

Feature request: Instructions for new feeds

Add instructions for those who want to add their blog to a planet: What info is needed, how to add only one tag / category of your blog, image requirements. This will save some work for the curator/admin and streamline the process of adding new feeds to a planet.

UI: Clarify language under Planet Config on the Admin page

The Planet Config section, which allows you to change the Planet's admin contact information, currently looks like this:
Screen Shot 2013-01-14 at 3 11 20 PM
I initially misunderstood and thought it was asking for the name & email of the person who making that specific change (me), when really it is asking for the administrator's contact info.

Quick fix: instead of
Your name:
Your email:

change to
Adminstrator name:
Administrator email:

Longer term: consider putting the settings for Administrator contact info on a separate page from the Feeds so that someone doesn't accidentally change that when they only meant to make a change to the feeds.

RSS to Email gateway

Allow users to sign up to have posts emailed to them, possibly in gpg-encrypted form.

User login for planet administration

Having a single sign-on makes sense for the admin page.

Planets with multiple administrators could give each administrator a unique login/password

Obfuscate passwords

Passwords are currently stored in plain text. Users are warned about this, but we really should obfuscate them.

OPML export should be extended to include feed tags and face url

The OPML spec at http://dev.opml.org/spec2.html says:

An OPML file may contain elements and attributes not described on this page, only if 
those elements are defined in a namespace, as specified by the W3C.

OPML can also be extended by the addition of new values for the type attribute. 
When specifying such an extension, following the example of this specification, say
which attributes are required and which are optional, and explain the roles each of the 
attributes plays, how they relate to each other, and what rules they must conform to.
There is a mechanism in the OPML Editor that is based on this form of extension.

Developers should, whenever possible, use capabilities that are already in use by 
others, or included in this spec, or recommendations or guidelines.

Our feeds have tags and face (hackergotchi) urls. We should export those in the OPML file planeteria should import from such an OPML file, as per issue #4.

Quotes in Name field on admin page disappear

I noticed when I first put quotes around Skud's name on WFS that the first quote and all the text that appears after it disappears when the Admin page next loads. And as a consequence of that, if you don't add it back, then when you next save changes, it'll save the name without the text after the quotes.

I dug into it a bit today when I was fixing an issue with escaped apostrophes in the name field. It is not fixed by escaping the quotes. Looking at the DOM in Chrome Developer Tools, it seems to be related to the form field code. The value= wraps the string in double quotes, so when it sees double quotes it thinks the string has ended.

In the example below, I typed in the phrase Skud "missile" test and then submitted the change. This is what appeared on the admin page when it next loaded.

screen shot 2013-07-29 at 12 15 50 am

Feed display does not register </center> tag

A post from Blogger had text that used <center> tags; on the Planteria feed page, it did not honor the </center> tag, so all the text after the </center> tag continues to be centered.

This is the code from the Blogger post:

<center>(<span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><span style="font-size: xx-small;">Music<span style="font-size: xx-small;">: </span>I can't get started</span></span>) </center>

Which is displayed like this on her blog:
Screen Shot 2013-01-17 at 11 12 16 AM

And the feed on Planeteria.org looked like this:

Screen Shot 2013-01-17 at 10 19 53 AM

Feature/UI: Sort a planet's feed into subsets with tags

The curator would decide what tags to assign. Users could select a tag or multiple tags to display only those blog posts with that tag.

This has been requested by multiple internship participants.

Another idea that's been suggested as an alternative to tagging is to be able to copy a planet to create a new one and change the subscriptions as desired.

Licenses

This page vendor/withsqlite/withsqlite.py
says it's released under the GNU General Public License, version 3 or later; but the bottom of the main site says the code is under the terms of the AGPLv3.

What is the license that should be used? Let's make it consistent throughout the codebase.

Testing platform needed

We need to automatically test the major functions: planet creation, add/remove feed, edit planet metadata, update planets, generate atom/html/opml files. Maybe twill can help.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.