humandynamics / openpds

openPDS - Personal Data Store / Server

The personal data store component of openPDS handles storage of raw personal data, provides endpoints for writing such data via connectors, a computation engine to perform analysis on the raw data, as well as storage and REST endpoints for results of such analysis (answers to questions, in openPDS lingo).

Please note: openPDS requires a separate registry server for account management and to act as an OAuth 2.0 provider. This can be found at https://github.com/HumanDynamics/openPDS-RegistryServer. If you run your own registry server, the domain for your registry server must be provided to the openPDS setup script.

Contributors: The dev branch is for ongoing development. Please submit pull requests to this brach.

Getting started with openPDS

(You must have python pip, virtualenv, and mongodb server installed on your machine)

# install python and database dependencies
apt-get install python pip
apt-get install python-virtualenv
apt-get install postgresql postgresql-contrib
apt-get install mongodb mongodb-server
apt-get install python-dev libpq-dev

service mongodb start

Creating a python virtual environment for openPDS

You must clone the repo into the virtualenv directory for setup.py to work:

# create the virtual environment
virtualenv pdsEnv
cd pdsEnv
source bin/activate

# get the latest openPDS code
git clone https://github.com/HumanDynamics/openPDS.git
cd openPDS

# install openPDS and its requirements
pip install -r requirements.txt

# set up and run your local openPDS
python start.py
python manage.py syncdb
python manage.py runserver 0.0.0.0:8002 (for access to local VM)

The above steps will start openPDS with default configuration settings on port 8002 of the loopback interface (local access only) on your machine. The openPDS setup script generates a wsgi file that can be used to run openPDS with other web server software, such as Apache.

openpds's Issues

Create Chef cookbook for Open PDS

I'm eager to get a cookbook working to making standing up a server as simple as possible in a secure fashion with server hardening, etc.

Is anyone close to running openpds in a production-ish environment?

Make a landing page

The site needs a landing page. When users go to the root directory, it should just show them something simple, like "It Works!" or redirect them to /admin

Sample DB files

Can anybody please upload the sample database files, to check DB schema for educational purpose.

Thanks 😃

ImportError: No module named internal.mongo

@brian717 👍

Hi,
In oms_pds/ internal folder is not available, so i have corrected the line as below in tasks.py file. Please correct me.

oms_pds/tasks.py

from oms_pds.internal.mongo import InternalDataStore

+from oms_pds.pds.internal import InternalDataStore

As the class " InternalDataStore" is defined only in oms_pds/pds/internal/.py

Investigate integration with Tapiriik, a sync tool for fitness tracker services

Tapiriik is a open source service for syncing data between fitness services. Seems like an interesting first application for OpenPDS, as user can just sync their combined fitness data into it.

https://github.com/cpfair/tapiriik

Thoughts?

cc: @cpfair

Investigate integration with EggTimer, a menstrual cycle tracker

@jessamynsmith is working on an open source menstrual tracker. It strikes me as a great opportunity to make use of Open PDS's model, as this is such personal information.

• https://github.com/jessamynsmith/eggtimer-server
• https://github.com/jessamynsmith/eggtimer-ui

Anyone have any thoughts on whether this is a good application? I'm relatively new to OpenPDS :)

(As I understand it, it's being built with cordova, so the intention is to package it as a cross-platform app. Correct me if I'm wrong though, Jessamyn!)

"No module named django": something wrong with kombu

When you get the error "No module named django", maybe it's noting about your django but a package called kombu.....

When installing the packages in requirment.txt, it installed
django-celery-with-mongodb-3.0
django-celery-3.1.17
celery-4.0.0
kombu-4.0.0

Here the reason for kombu-4.0.0:
"Collecting kombu<5.0,>=4.0 (from celery>=3.1.15->django-celery<4.0,>=3.0->django-celery-with-mongodb->-r requirements.txt (line 5))"

But ! ! !
In openPDS\openpds\settings.py Line 142
'kombu.transport.django' is in the list of INSTALLED_APPS, while there no kombu.transport.django module in kombu-4.00, it's only in the version below 4.00......

And you will just get an error: 'no module named django'. In fact you can hardly find that it's noting wrong with your path of django, your enviroment and so on.

Rolling back the version of kombu to 3.X is complex because a list of package about celery rely on it. So I want to know about which version of django-celery-with-mongodb should be installed.

Clean up unnecessary pymongo imports

Some files just import it and never use it - these are easy, just remove the import.

Some answer modules actually use it after importing it. These uses should be replaced with an InternalDataStore instance, and then the import should be removed.

`python manage.py syncdb` fails in ubuntu 14.04 VM

(pdsEnv)kitchen@60aa1b8653be:~/pdsEnv/openPDS$ python start.py 


##########################################################################
NOTE: This must be run from the root directory of the openPDS project
and for the virtualenv to be located in the directory above it.
ie: you should be in /pdsEnv/openPDS
if necessary, press ctrl+c to exit and then move to the correct directory.
##########################################################################
Enter the path to the openPDS virtual environment (or nothing for default: /home/kitchen/pdsEnv): 

Enter the Registry Server domain name (or nothing for MIT default: linkedpersonaldata.org): 

Which backend would you like to use for personal data storage?
1. MongoDB (openpds.backends.mongo)
2. SQLite (openpds.backends.sqlite)
3. Postsgres (openpds.backends.postgres)
Enter 1, 2, or 3 (default is 1): 
wsgi.py and settings.py generated
Setup done. Please continue with django setup by running: ./manage.py syncdb
Make sure that the user running your server process (www-data for apache, for example) has write access to all directories from your PDS virtual env to the openpds package directory
Finally, after running syncdb, provide the user running the server process with write access to the test.db file generated in the openpds directory
(pdsEnv)kitchen@60aa1b8653be:~/pdsEnv/openPDS$ ./manage.py syncdb
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 69, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 8, in <module>
    from django.core.management.sql import custom_sql_for_model, emit_post_sync_signal
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/core/management/sql.py", line 6, in <module>
    from django.db import models
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/db/__init__.py", line 11, in <module>
    if DEFAULT_DB_ALIAS not in settings.DATABASES:
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/kitchen/pdsEnv/local/lib/python2.7/site-packages/django/conf/__init__.py", line 95, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'openpds.settings' (Is it on sys.path?): cannot import name Connection
(pdsEnv)kitchen@60aa1b8653be:~/pdsEnv/openPDS$

Any immediate thoughts?

Decouple the project from Virtualenv

User should be able to completely remove a virtualenv (or switch to another one) completely separate from their django project.

Blank page when running local instance

I'm running ubuntu 14.04

I cloned the repo yesterday, and installed via the instructions in the readme. I didn't see any issue during install, as accepting all the default options. I set up my superuser as prompted during setup, and ran the server, but only a blank page showed up at 127.0.0.1:8002

This showed up in the logs:

(pdsEnv)~/longer-tmp/pdsEnv/openPDS (master ✔) ᐅ python manage.py runserver 0.0.0.0:8002
Validating models...

0 errors found
Django version 1.4.13, using settings 'openpds.settings'
Development server is running at http://0.0.0.0:8002/
Quit the server with CONTROL-C.
[07/Nov/2014 20:22:02] "GET / HTTP/1.1" 404 15
[07/Nov/2014 20:22:02] "GET / HTTP/1.1" 404 15
[07/Nov/2014 20:22:02] "GET / HTTP/1.1" 404 15
[ etc...]

Any thoughts? I'm pretty technically inclined, so I don't think I'm missing anything obvious if the install instructions are still up-to-date.

Thanks!

Project Feedback

This is essentially the same question and feedback that I gave at HumanDynamics/openPDS-RegistryServer#18, with a couple of minor differences.

I've recently pulled down this project and attempted to get it (with openPDS-RegistryServer) running. A couple of things struck me as surprising and/or unexpected:

• It's unusual for a django project to be distributed (i.e. with a whole settings.py file, ). Typically I'd expect a django app or a collection of apps.
• Your installation instructions tightly couple the project with a virtualenv. These two things should be separated. I should be able to completely remove a virtualenv (or switch to another one) completely separate from my django project.
• There appear to be a few tests, but no information on how to run them.
• The included setup.py, which a neat idea is not what I expect from a setup.py (e.g. a setuptools installation script). Perhaps this should be renamed to something like project_bootstrap.py?

A few things I'd typically expect from a reusable django app are:

• Documentation for any app-specific settings (not a settings file)
• A requirements.txt in the root of the project. This project has this in conf/, but the # XXX - comments don't instill a lot of confidence. (e.g. does this project actually need/use everything in the requirements file?)
• A setuptools-compatible setup.py that allowed me to install with: pip install openPDS
• Some level of tests, runnable with: python manage.py test openpds or instructions for running the test suite in the readme or the docs.

If the goal of this project is to be something that fits within the django ecosystem, the barrier to usage and contributions would be lower if it followed some of the typical community conventions. Some of this information is now available in the django docs, as well.

I hope this is helpful feedback!

How to run openPDS application

I have followed the steps given on link "https://github.com/HumanDynamics/openPDS" to setup on linux environment.

after that how do I run the openPDS ?

import a db_requirements file instead of each one separately.

This will allow us to augenerate db_requirements and requirements.txt, so that we don't require people to have installed both mongo and postgres. See #22 (comment)

how to launch openPDS project in a andriod emulator and run it

Please help me !!!

I am working on linux environment(ubuntu).

1. I have made setup of registry server and openPDS as per the steps provided in links:
   https://github.com/HumanDynamics/openPDS
   https://github.com/HumanDynamics/openPDS-RegistryServer
2. I have setup Andriod ADT bundle to make use of Emulator.

Open Points:

In openPDS project path, after passing the below two commands the terminal is displaying " Developement Server is running at http://0.0.0.0:8002 "

python manage.py syncdb
python manage.py runserver 0.0.0.0:8002 (for access to local VM)

In openPDS-RegistryServer project path, after passing the below two commands the terminal is displaying " Developement Server is running at http://0.0.0.0:8000 "

python manage.py syncdb
python manage.py runserver 0.0.0.0:8000 (for access to local VM)

I have questions now:
How to run openPDS project in andriod emulator ?
How to import openPDS project to emulator ?

"cluster" dependency cannot be installed

Hi,

I just tried to install openPDS, following the instructions as stated in the Wiki. The "cluster" package fails to install for me, complaining about a missing README.rst file. I tried to manually install that dependency, running:

pip install -v cluster

but to no avail:

(pdsEnv)balko@ubuntu:~/dev/pdsEnv$ pip install -v cluster
Downloading/unpacking cluster
  Ignoring link https://pypi.python.org/packages/source/c/cluster/cluster-1.1.1b3.tar.gz#md5=a549edf712328540b08d394c79af2506 (from https://pypi.python.org/simple/cluster/), version 1.1.1b3 is a pre-release (use --pre to allow).
  Using version 1.2.1 (newest of versions: 1.2.1, 1.2.0, 1.1.2)
  Downloading cluster-1.2.1.tar.gz (42kB): 
  Downloading from URL https://pypi.python.org/packages/source/c/cluster/cluster-1.2.1.tar.gz#md5=0134f50d2623dc342c630d53a18e22fe (from https://pypi.python.org/simple/cluster/)
...Downloading cluster-1.2.1.tar.gz (42kB): 42kB downloaded
  Running setup.py (path:/home/balko/dev/pdsEnv/build/cluster/setup.py) egg_info for package cluster
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/balko/dev/pdsEnv/build/cluster/setup.py", line 3, in <module>
        readme_contents = open("README.rst").read()
    IOError: [Errno 2] No such file or directory: 'README.rst'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/balko/dev/pdsEnv/build/cluster/setup.py", line 3, in <module>

    readme_contents = open("README.rst").read()

IOError: [Errno 2] No such file or directory: 'README.rst'

----------------------------------------
Cleaning up...
  Removing temporary dir /home/balko/dev/pdsEnv/build...
Command python setup.py egg_info failed with error code 1 in /home/balko/dev/pdsEnv/build/cluster
Exception information:
Traceback (most recent call last):
  File "/home/balko/dev/pdsEnv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/home/balko/dev/pdsEnv/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/balko/dev/pdsEnv/local/lib/python2.7/site-packages/pip/req.py", line 1229, in prepare_files
    req_to_install.run_egg_info()
  File "/home/balko/dev/pdsEnv/local/lib/python2.7/site-packages/pip/req.py", line 325, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/balko/dev/pdsEnv/local/lib/python2.7/site-packages/pip/util.py", line 697, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command python setup.py egg_info failed with error code 1 in /home/balko/dev/pdsEnv/build/cluster

Storing debug log for failure in /home/balko/.pip/pip.log

Any ideas?

Thanks heaps,
Soeren

unit testing the connectors

Now that the FUNF and OpenSense connectors are both working, we need to start working out testing. The amount of time that I've sunk into manually doing integration testing is pretty extreme, so I'm inclined to think that testing will considerably speed things up.

Here's how I think that the problem should be approached. I'm not a django programmer, so would really welcome feedback and insight into this (@bradmontgomery, @patcon, @chieni, @mitchellgu ):

• Stub out the authorization server, so that the project can be testing without it:

authorization = PDSAuthorization("ios_write", audit_enabled=True)

• Unit test each of the opensense/views.py and funf/views.py methods
• use django.test.client to make test requests to the connectors, and actually simulate testing data into a special Mongo collection. (If we can't use a special test collection, delete the data from mongo after testing.)

Thoughts?

Project State/Goals

Hi there,

I'm taking a breather from working on openPDS. Here are a few notes on what we've been working on and a roadmap for going forward:

What we're doing (as of December 2, 2014):

• Separating database requirements, so that files don't have to import sqlite3, pymongo, and psycopg2 all at the same time. #25 #27
• Related to the above: The requirements.txt file should be auto-generated by start.py #25 #27
• Decoupling the project from virtualenv. Seriously - people should be able to store their virtualen environments anywhere they want. #29
• Testing. This should exist. #39

These are mostly derived from @bradmontgomery's #22

Things that I would do next:

(This is obviously subject to interpretation and debate)

• make the code confirm to pep8 standards.
• I'd remove access control, and replace with a user login. Contentious, I know, but access control creates some complications, and I don't think that the project is ready for it.
  -- at the very least, access control should exist as a separate django app.
• Remove visualizations and meetup
• Create a connector to datahub

datahub notes

This is for the creation of a datahub backend, through which users would be able to directly access and edit their data.

• Create an openPDS user on DataHub.
• Automatically create accounts and openPDS repositories for new users, including passwords.
• Create openPDS repositories for each new account
• grant select, create, insert, update, delete, to openPDS
• send the user the username/password, remove it from openPDS

Add default admin theme

This is what I see on /admin:
https://imgur.com/ydGkhD6

Assuming that a lack of admin theme is not my install being buggy, might it make sense to get a drop-in theme in place?

Rename setup.py

To avoid confusion with setuptools. Actual name will be left as an exercise to the reader.

coupling with funf

Hey, new to the project, been cleaning up the code and migrating the codebase to django 1.7 and other stuff like that on my own fork.

I'm wondering why some modules - AccessControlledInternalDataStore in particular - are coupled so tightly with Funf. For example, that class has a mapping field that directly relates to funf data:

        self.mapping = {'ActivityProbe': 'activity_probe',
                        'SmsProbe': 'sms_probe',
                        'CallLogProbe': 'call_log_probe',
                        'BluetoothProbe': 'bluetooth_probe',
                        'WifiProbe': 'wifi_probe',
                        'LocationProbe': 'simple_location_probe',
                        'ScreenProbe': 'screen_probe',
                        'RunningApplicationsProbe': 'running_applications_probe',
                        'HardwareInfoProbe': 'hardware_info_probe',
                        'AppUsageProbe': 'app_usage_probe'}

If it's just because the first projects to use this have funf-based, that's fine, but it seems to me that it's in the project's best interest to decouple it entirely and abstract the datastore objects from the kind of data being stored.

If it's part of some design decision I'm not privy to, could you explain it to me?

thanks!

manage.py syncdb troubles on osx

I'm getting this error when running syncdb (on Mac)

No idea what's going on with it. All of the requirements, databases, etc are installed. Any ideas, @brian717 ?

File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/init.py", line 443, in execute_from_command_line
utility.execute()
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/init.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(_args, *_options.dict)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(_args, *_options)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
cursor = connection.cursor()
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/db/backends/init.py", line 319, in cursor
cursor = util.CursorWrapper(self._cursor(), self)
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 288, in _cursor
self._sqlite_create_connection()
File "/Users/jbt/Documents/code/pdsEnv/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 278, in _sqlite_create_connection
self.connection = Database.connect(**kwargs)
sqlite3.OperationalError: unable to open database file

Is openPDS usable standalone?

I've been playing around with openPDS from this repo over the last few days, followed the instructions, hacked around with some things, read through parts of the source, etc and am still unable to get an instance running (or if I get it running, it's full of 500 server errors)

So I was wondering, is openPDS (as it exists publicly in this repo currently) actually runnable? I noticed a lot of internal references to Open Mustard Seed; will it only run within that environment?

I'd like to check it out/have a bit of a play (while I wait for access to OMS), but I don't want to waste more time on it if there's no chance of actually getting a local version running currently.

Would really appreciate some input/guidance on how to get it going if it can be :)

Thanks so much!!

Project didn't gain too much traction. What's missing?

Hello.
A lot has changed in the 6 or so years since the project's inception. Everybody involved has probably followed the developments in the privacy world closely.

Did any of you write any summary or "what I learned" articles based on this project yet? If not, I think reflecting on this project's goals and how they could be accomplished better is something I'd like to read about, especially coming from someone who has worked for building such a world.
Perhaps those goals shouldn't be accomplished? Maybe it's better to allow data aggregation so our slow-learning AIs can recognize more patterns.
Perhaps there are better cryptographic primitives now that would make more targeted (as opposed to en-masse) data acquisition viable and private (zero knowledge proofs for ensuring only the people benefiting from an analysis would contribute data?).

I'm thinking about building similar tools as yours. I want to know if that's actually a useful thing with the current status quo.

pin down requirements.txt version numbers

For beginners, this should probably be set. That way, updates won't break their build process. See #22

Step by Step instructions to setup stable version OpenPDS (and it's components) on local machine for academic demo.

Hi,

I want to present OpenPDS for one of my courses. And its not working. I am using Linux Mint 17.2 (Ubuntu 14.04). And while setting it up i am facing few problems and error, which i am not clear are bugs in the code or related to setup process. So i request clear step by step process (from scratch considering system has no apache etc) including directory names (which i found in setting files) to setup the whole system.

Try to provide stable version (links) of software for presentations. 😄
thanks :) 😄