aral / gaebar

Google App Engine Backup and Restore

Please note: I am no longer supporting this project.

The Google App Engine account that the demo is hosted on is also in the process of being shut down.

I do not wish to further support a company whose business model is corporate surveillance.

For more information, please see my talk on digital feudalism and my startup to create a consumer alternative to the closed silos: Indie Phone.

Gaebar (Google App Engine Backup and Restore) Beta 3

A Naklab™ production sponsored by the web conference - http://headconference.com

Copyright (c) 2009 Aral Balkan. http://aralbalkan.com

Released under the GNU GPL v3 License. See license.txt for the full license or read it here: http://www.gnu.org/licenses/gpl-3.0-standalone.html

Downloading and Installing Gaebar

A. From an archive.

You can get a .zip or .tar of Gaebar from: http://github.com/aral/gaebar/tree/master

(Click on the Download link and choose your poison.)

Unzip the Gaebar archive to a folder called gaebar/ off the root of your Django project. You must place Gaebar at this location for the app to work properly.

B. From GitHub

You can install the latest Gaebar trunk into your projects from GitHub using Git.

(a) If you're using Git for your main project

Add Gaebar to your project as a submodule:

git submodule add git://github.com/aral/gaebar-gaed.git

(b) If you're not using Git for your main project

Clone Gaebar into a folder called gaebar off the root folder of your project:

git clone git://github.com/aral/gaebar-gaed.git

2. To check for updates, go into gaebar/ and git pull

(Don't forget to git commit your main project after you've updated Gaebar to a new version via git pull.)

Configuring your project to use Gaebar

IMPORTANT Patch your dev_appserver.py as per the instructions here: http://aralbalkan.com/1440 (and please star issue 616 if you'd like Google to fix this so we can remove this step: http://code.google.com/p/googleappengine/issues/detail?id=616).

This is required in order to override some of the local dev server restrictions to allow automatic downloads of backups. Gaebar will not work unless you implement this patch.

1. Add to installed apps

---

Add Gaebar to your list of INSTALLED_APPS in your application's settings.py file. e.g.

INSTALLED_APPS = (
	# Other apps...
	'gaebar',
)

2. Add to urls.py

---

In your main urls.py, map the Gaebar app to the URL shown below. You must map Gaebar to the exact URL shown below or the app will not work.

urlpatterns = patterns('',

# ...other URLs

url(r'^gaebar/', include('gaebar.urls')),

)

3. Add the static folder

---

In your app.yaml file, add the following entry before any other static entries to map Gaebar's static files (images, js, etc.) correctly:

Static: Gaebar

• url: /gaebar/static static_dir: gaebar/static

4. Add indices

---

If you are declaring your indices manually, add the following to your index.yaml file (or run Gaebar locally in the dev server so that the index is created for you automatically):

• kind: GaebarCodeShard properties:
  • name: backup
  • name: created_at

(Note: indices may take some time to create on the deployment environment. Until they are ready, backups will fail.)

5. Modify settings.py

---

Modify settings.py to add the GAEBAR_LOCAL_URL, GAEBAR_SECRET_KEY, GAEBAR_SERVERS, and GAEBAR_SERVERS settings to your application.

GAEBAR_LOCAL_URL: Absolute URL of your local development server. Is used when downloading your remote backup to your local machine.

GAEBAR_SECRET_KEY: A secret key that is used (a) to authenticate communication between your local deployment environment and the remote backup environment to facilitate the download of backups via urlfetch and (b) used during the restore process to authenticate the client.

GAEBAR_SERVERS: Dictionary of named servers. Not essential but makes it easy to identify the servers by name when backing up and restoring. Also makes it easier to identify which server you're running Gaebar on.

GAEBAR_MODELS: Tuple of models from your app that you want backed up.

Here is a sample batch of settings that you can use as a guide:

Gaebar

GAEBAR_LOCAL_URL = 'http://localhost:8000'

GAEBAR_SECRET_KEY = 'change_this_to_something_random'

GAEBAR_SERVERS = { u'Deployment': u'http://www.myapp.com', u'Staging': u'http://myappstaging.appspot.com', u'Local Test': u'http://localhost:8080', }

GAEBAR_MODELS = ( ( 'app1.models', (u'Profile', u'GoogleAccount', u'AllOtherTypes', u'PlasticMan'), ), ( 'app2.models', (u'Simple',), ), )

A note on templates:

If you get any template-related errors, try adding gaebar/templates to your TEMPLATE_DIRS settings variable.

How it works

Gaebar backs up the data in your datastore to Python code. It restores your data by running the generated Python code.

Since a backup is a long running process, and since Google App Engine doesn't support long-running processes, Gaebar fakes a long running process by breaking up the backup and restore processes into bite-sized chunks and repeatedly hitting the server via Ajax calls.

By default, Gaebar backs up 5 rows at a time to avoid the short term CPU and 10-second call duration quotas and splits the generated code into

gaebar's Issues

Django alpha -> final update?

The packaged (ziped) Django version is the alpha version of the 1.1 release. As the final one is out, it might be a good idea to update the zip file.

Models not in db._kind_map

Hi,

I try to run gaebar on my application, but I get an error on line 451 (pulled from git) in gaebar/views.py

Its seems that my models are not in the db._kind_map

What could be the problem ?

Thanks,

Running gaebar... app.yaml issues InvalidAppConfigError

I have dropped the gaebar folder into my gaebar-aep folder and run ./manage.py runserver. It runs, but when I try to request a page I get...

'google.appengine.api.appinfo' from '/Ap...ogle_appengine/google/appengine/api/appinfo.pyc'>, appinfo.LoadSingleAppInfo = <function LoadSingleAppInfo at 0x73c570>
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py in ReadAppConfig(appinfo_path='/Users/everythingability/Desktop/gae/gaebartest/app.yaml', parse_app_config=<function LoadSingleAppInfo at 0x73c570>)
2958 except IOError, e:
2959 raise InvalidAppConfigError(
2960 'Application configuration could not be read from "%s"' % appinfo_path)
2961 try:
2962 return parse_app_config(appinfo_file)
appinfo_path = '/Users/everythingability/Desktop/gae/gaebartest/app.yaml'

<class 'google.appengine.tools.dev_appserver.InvalidAppConfigError'>: Application configuration could not be read from "/Users/everythingability/Desktop/gae/gaebartest/app.yaml"
args = ('Application configuration could not be read from...verythingability/Desktop/gae/gaebartest/app.yaml"',)
message = 'Application configuration could not be read from...verythingability/Desktop/gae/gaebartest/app.yaml"'