[PATCH] Mako bytecode caching only works when use_dotted_templatenames = False about tg2

[pedersen]

Original Author: pedersen, Original Timestamp: 2011-03-29 03:32:06.251000

Original Body: If you are using Mako templates in your project, Mako bytecode caching doesn't seem to work unless you set use_dotted_templatenames to False in your project's app_cfg.

#2192 is probably related to this, and it should be noted that templating.mako.compiled_templates_dir only works with use_dotted_templatenames as False as well.

---

11/30/09 13:04:17 changed by seedifferently

version changed from 2.0.3 to trunk.
milestone changed from 2.* bugfix to __unclassified__.

Delete 12/01/09 22:02:20 changed by jorge.vargas

milestone changed from __unclassified__ to 2.1b1.

Delete 02/22/10 13:02:45 changed by lambacck

I assume you are actually referring to Mako's creation of .py (and .pyc) files from .mak files. If so, this is caused by not letting the Mako template loader actually load the file and reading the file and passing a string instead.

If you are referring to something else, let me know.

Also there is a bug in Mako that is related to this: http://www.makotemplates.org/trac/ticket/128 which I have also provided a patch for.
02/22/10 14:05:17 changed by lambacck

attachment mako_caching.diff added.

Fix mako caching problem
Delete 02/22/10 14:39:48 changed by lmacken

owner set to lmacken.
status changed from new to assigned.

I just committed pushed this patch, which is similar to yours:

http://bitbucket.org/turbogears/tg-dev/changeset/020db0b5043e/

All of the test suite passes, and this doesn't require any Mako changes. Can you please test and verify that it still works for you in windows?

Thanks!
Delete 02/22/10 14:57:52 changed by lmacken

Ok, so my patch breaks custom templating.mako.compiled_templates_dir when used in conjunction with 'use_dotted_templatenames', since passing an absolute path to the mako Template(module_directory=) will still put them in myproject/templates/
Delete 02/22/10 15:28:28 changed by lmacken

status changed from assigned to closed.
resolution set to fixed.

Ok, I pushed out a couple of small tweaks, and everything seems to be working fine (aside from lambacck's windows-specific mako issue mentioned above). Closing...
Delete (follow-up: 8 ) 03/01/10 15:38:04 changed by lmacken

cc set to percious.
status changed from closed to reopened.
resolution deleted.

Reopening, since I'm not 100% satisfied with how this works by default.

Right now out of the box (with this patch) use_dotted_templatenames is enabled and Mako stores your compiled bytecode in myproject/templates/the/full/path/to/myproject/templates/index.pyc. This behavior is documented by Mako (http://www.makotemplates.org/docs/usage.html#usage_file).

templating.mako.compiled_templates_dir defaults to your templates directory, and when use_dotted_templatenames is disabled Mako compiles and stores templates alongside the existing templates.

I don't know if we can easily have the same behavior with and without dotted templatenames, but I think we could do a better job of docmenting this and making it easy to change, without polluting the project's templates directory.

So, how about we add something like this in the default ini?

{{if mako}}

The location to store your compiled templates

templating.mako.compiled_templates_dir = %(here)/data/templates
{{endif}}

This way, we're sharing %(here)/data with Beaker sessions and we're not touching the projects templates directory.
Delete (in reply to: 7 ) 03/02/10 13:23:21 changed by lambacck

lmacken's comment applies to all Mako template caching and not just dotted names since we copied the behaviour of attempting to pick a suitable location from the non-dotted method.

I think that defaulting to explicit location is a good idea in general, because most users are going to want to have the compiled templates cached if only to assist with debugging. This means a better experience for new TG + Mako developers.

With the current code I don't think there is anyway to avoid writing the templates somewhere to disk (what happens when the process does not have write permissions?), so we might want to make no caching of templates an option at some point in the future. We might want to let "templating.mako.compiled_templates_dir = None" mean "don't cache the templates" or let the option be more explicit and don't cache the templates to disk unless a valid path is provided (maybe log a warning if there is a problem with the path).

In TG1, the equivalent of templating.mako.compiled_templates_dir defaulted to makotemplates (or something similar) in the development.ini but the line was commented out. If it was commented out then no template modules were written to disk.
Delete (follow-up: 10 ) 03/08/10 10:25:15 changed by lambacck

Would you like me to submit patches for what is discussed above?
Delete (in reply to: 9 ) 03/09/10 16:50:56 changed by lmacken

Replying to lambacck:

Would you like me to submit patches for what is discussed above?

Sure. Sorry, I meant to get this done before the weekend, but but caught up with other things. If you're willing and able to throw this patch together, please do so. If not, I can hack it together later this week. Cheers.
Delete 03/16/10 12:43:09 changed by lmacken

So I think this patch may be sufficient.

--- a/devtools/templates/turbogears/development.ini_tmpl
+++ b/devtools/templates/turbogears/development.ini_tmpl
@@ -68,18 +68,17 @@ templating.mako.reloadfromdisk = true

the compiled template dir is a directory that must be readable by your

webserver. It will be used to store the resulting templates once compiled

by the TemplateLookup system.

During development you generally don't need this option since paste's HTTP

server will have access to you development directories, but in production

you'll most certainly want to have apache or nginx to write in a directory

that does not contain any source code in any form for obvious security reasons.

-#
-#templating.mako.compiled_templates_dir = /some/dir/where/webserver/has/access
+templating.mako.compiled_templates_dir = %(here)s/data/templates

WARNING: THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT

Debug mode will enable the interactive debugging tool, allowing ANYONE to

execute malicious code after an exception is raised.

#set debug = false

Logging configuration

Add additional loggers, handlers, formatters here

Delete 03/16/10 12:44:52 changed by lmacken

The issue here is setting this value to None will create a 'None' directory to store your templates in. If we want None to default to in-memory template caching, we'll have to refactor some of the code in configuration.py which looks in a few places to see if it has the permissions to store templates there.
Delete 03/17/10 16:48:06 changed by lmacken

I pushed changeset 307 to set a sane default template cache directory.

If we want to handle None to only use an in-memory cache, this will still require a little bit more work.
Delete 03/17/10 17:03:16 changed by lmacken

actual tgdevtools-dev changeset 307
Delete 05/16/10 15:04:24 changed by lmacken

I threw together a quick patch that simplifies the heuristics of determining where to cache the compiled mako templates. If no cache directory is specified, it currently tries every pylons template path, and will then fall back to %(data)/templates. This patch removes these heuristics and makes the configuration more explicit. If templating.mako.compiled_templates_dir is not defined, None, False, or not writable, it will setup an in-memory cache for compiled template bytecode. Otherwise, it'll throw the cached templates wherever you specify.

I tested this with dotted template lookups, and filesystem based template lookups, both of which seem to work as expected. The only difference is that with dotted template lookups, the cache will have a different directory heirarchy than with path-based lookups. This shouldn't cause any problems as far as I can tell.

If someone else familiar with corner of TG could +1 this patch, that would be re-assuring.

--- a/tg/configuration.py Sun Apr 18 16:26:13 2010 +0200
+++ b/tg/configuration.py Sun May 16 15:51:41 2010 -0400
@@ -317,35 +317,26 @@
# a file system based template lookup mechanism.
compiled_dir = tg.config.get('templating.mako.compiled_templates_dir', None)

•    if not compiled_dir:
  

•        # Try each given templates path (when are they > 1 ?) for writability..
  

•        for template_path in self.paths['templates']:
  

•            if os.access(template_path, os.W_OK):
  

•                compiled_dir = template_path
  

- break # first match is as good as any

•        # Last recourse: project-dir/data/templates (pylons' default directory)
  

•        if not compiled_dir:
  

•    if not compiled_dir or compiled_dir.lower() in ('none', 'false'):
  

•        # Cache compiled templates in-memory
  

•        compiled_dir = None
  

•    else:
  

•        bad_path = None
  

•        if os.path.exists(compiled_dir):
  

•            if not os.access(compiled_dir, os.W_OK):
  

•                bad_path = compiled_dir
  

•                compiled_dir = None
  

•        else:
           try:
  

•                root = os.path.dirname(os.path.abspath(self.package.**file**))
  

•            except AttributeError:
  

•                # Thrown during unit tests when self.package.**file** doesn't exist
  

- root = None

•            if root:
  

•                pylons_default_path = os.path.join(root, '../data/templates')
  

•                if os.access(pylons_default_path, os.W_OK):
  

- compiled_dir = pylons_default_path

•            if not compiled_dir:
  

•                if use_dotted_templatenames:
  

•                    # Gracefully digress to in-memory template caching
  

•                    pass
  

•                else:
  

•                    raise IOError("None of your templates directory, %s, are "
  

•                        "writable for compiled templates. Please set the "
  

•                        "templating.mako.compiled_templates_dir variable in your "
  

•                        ".ini file" % str(self.paths['templates']))
  

•                os.makedirs(compiled_dir)
  

•            except:
  

•                bad_path = compiled_dir
  

•                compiled_dir = None
  

•        if bad_path:
  

•            log.warn("Unable to write cached templates to %r; falling back "
  

•                     "to an in-memory cache. Please set the `templating.mak"
  

•                     "o.compiled_templates_dir` configuration option to a "
  

•                     "writable directory." % bad_path)
  
   if use_dotted_templatenames:
       # Support dotted names by injecting a slightly different template
  

--- a/devtools/templates/turbogears/development.ini_tmpl Sun May 16 12:51:38 2010 -0400
+++ b/devtools/templates/turbogears/development.ini_tmpl Sun May 16 16:03:58 2010 -0400
@@ -66,13 +66,15 @@

deployed in an unzipped manner, then you can leave this option to true

templating.mako.reloadfromdisk = true

-# the compiled template dir is a directory that must be readable by your
-# webserver. It will be used to store the resulting templates once compiled
-# by the TemplateLookup system.
+# the compiled template dir is a directory that must be readable and writable
+# by your webserver. It will be used to store the resulting templates once
+# compiled by the TemplateLookup system.

During development you generally don't need this option since paste's HTTP

server will have access to you development directories, but in production

you'll most certainly want to have apache or nginx to write in a directory

-# that does not contain any source code in any form for obvious security reasons.
+# that does not contain any source code in any form for obvious security
+# reasons. If disabled, None, False, or not writable, it will fall back
+# to an in-memory cache.
templating.mako.compiled_templates_dir = %(here)s/data/templates

WARNING: THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT

Delete (follow-up: 17 ) 06/08/10 12:39:02 changed by lmacken

keywords set to patch, needsreview.
summary changed from Mako bytecode caching only works when use_dotted_templatenames = False to [PATCH] Mako bytecode caching only works when use_dotted_templatenames = False.

Delete (in reply to: 16 ) 09/25/10 09:56:20 changed by mramm

Replying to lmacken:

I'm +1

[pedersen]

Original Author: pedersen, Original Timestamp: 2011-08-19 03:52:26.226000

Original Body: - milestone: 2.2.0 --> 2.1.3

[pedersen]

Original Author: pedersen, Original Timestamp: 2011-09-22 03:04:01.199000

Original Body: - milestone: 2.1.3 --> 2.1.4

[pedersen]

Original Author: pedersen, Original Timestamp: 2011-11-25 22:10:35.553000

Original Body: - version: 2.1.0 --> 2.1.3

• milestone: 2.1.4 --> 2.2.0

[pedersen]

Original Author: pedersen, Original Timestamp: 2012-08-24 01:44:10.731000

Original Body: - version: 2.1.3 --> 2.1.5

• milestone: 2.2.0 --> 2.3.0