The TurboGears Documentation¶
TurboGears is a Python web framework based on the ObjectDispatch paradigm, it is meant to make possible to write both small and concise applications in Minimal mode or complex application in Full Stack mode.
Get Started with TurboGears2¶
The fastest way to start using TurboGears is through the minimal mode, when using TurboGears with minimal mode a default setup that minimizes dependencies and complexity is provided.
Note
While minimal mode is well suited for small simple web applications or web services, for more complex projects moving to a package based configuration is suggested.
Installing TurboGears2¶
This tutorial takes for granted that you have a working Python environment with Python2.6+ or Python3.3+, with pip installed and you have a working browser to look at the web application you are developing.
This tutorial doesn’t cover Python at all. Check the Python Documentation page for more coverage of Python.
Creating Virtual Environment¶
First we are going to create a virtual environment where to install the framework, if you want to
proceed without using a virtual environment simply skip to Install TurboGears.
Keep in mind that using a virtual environment is the suggested way to install TurboGears without
messing with your system packages and python modules. To do so we need to install the virtualenv package:
$ pip install virtualenv
Now the virtualenv command should be available and we can create and activate a virtual environment for our TurboGears2 project:
$ virtualenv tgenv
$ . tgenv/bin/activate
If our environment got successfully created and activated we should end up with a prompt that looks like:
(tgenv)$
Installing TurboGears¶
Now we are ready to install TurboGears itself:
(tgenv)$ pip install TurboGears2
Hello World¶
A TurboGears application consists of an AppConfig application configuration and an application RootController.
The first is used to setup and create the application itself, while the latter is used to dispatch requests
and take actions.
For our first application we are going to define a controller with an index method that just tells Hello World.
Just create a new tgapp.py file and declare your RootController:
from tg import expose, TGController, AppConfig
class RootController(TGController):
@expose()
def index(self):
return 'Hello World'
Now to make TurboGears serve our controller we must create the actual application from an AppConfig:
config = AppConfig(minimal=True, root_controller=RootController())
application = config.make_wsgi_app()
then we must actually serve the application:
from wsgiref.simple_server import make_server
print("Serving on port 8080...")
httpd = make_server('', 8080, application)
httpd.serve_forever()
Running python tgapp (the python module we just created) will start a server on port 8080
with the our hello world application, opening your browser and pointing it
to http://localhost:8080 should present you with an Hello World text.
Serving Multiple Pages¶
Now that we have a working application it’s time to say hello to our user instead of greeting the world,
to do so we can extend our controller with an hello method which gets as a parameter the person to greet:
class RootController(TGController):
@expose()
def index(self):
return 'Hello World'
@expose()
def hello(self, person):
return 'Hello %s' % person
Restarting the application and pointing the browser to http://localhost:8080/hello?person=MyName should
greet you with an Hello MyName text.
Note
How and why requests are routed to the index and hello methods is explained in
Object Dispatch documentation
Passing parameters to your controllers is as simple as adding them to the url with the same name of the parameters in your method, TurboGears will automatically map them to function arguments when calling an exposed method.
Serving Templates¶
Being able to serve text isn’t usually enough for a web application, for more advanced output using a template is usually preferred. Before being able to serve a template we need to install a template engine and enable it.
The template engine used by TurboGears is Kajiki Template Language which is a fast and validated template engine with python3 support. To install Kajiki simply run:
(tgenv)$ pip install kajiki
Now that the template engine is available we need to enable it in TurboGears, doing so is as
simple as adding it to the list of the available engines inside our AppConfig:
config = AppConfig(minimal=True, root_controller=RootController())
config.renderers = ['kajiki']
application = config.make_wsgi_app()
Now our application is able to expose templates based on the Kajiki template engine,
to test them we are going to create an hello.xhtml file inside the same directory
where our application is available:
<html>
<title>Hello</title>
<py:if test="person">
<h1>Hello ${person}</h1>
</py:if><py:else>
<h1>Hello World!</h1>
</py:else>
</html>
then the hello method will be changed to display the newly created template
instead of using a string directly:
class RootController(TGController):
@expose()
def index(self):
return 'Hello World'
@expose('hello.xhtml')
def hello(self, person=None):
return dict(person=person)
Restarting the application and pointing the browser to http://localhost:8080/hello or
http://localhost:8080/hello?person=MyName will display an hello page greeting the person
whose name is passed as parameter or the world itself if the parameter is missing.
Enabling Helpers¶
Helpers are python functions which render small HTML snippets that can be useful in your
templates. This might include your user avatar, a proper date formatter or whatever might
come in hand in your templates. Those are usually provided by turbogears with the h name
inside all your templates.
TurboGears2 usually provides the WebHelpers2 package in applications quickstarted in
full stack mode, but this can be easily made available in minimal mode too.
First we are going to install the WebHelpers2 package:
$ pip install webhelpers2
Then we are going to import webhelpers2 and register it in our configuration as the application helpers (any python module or object can be registered as the helpers):
import webhelpers2
import webhelpers2.text
config['helpers'] = webhelpers2
Now the helpers are available in all our templates as h.helpername and in this case
we are going to use the text.truncate helper to truncate strings longer than 5 characters
in our hello.xhtml template:
<html>
<title>Hello</title>
<py:if test="person">
<h1>Hello ${h.text.truncate(person, 5)}</h1>
</py:if><py:else>
<h1>Hello World!</h1>
</py:else>
</html>
By restarting the application you will notice that pointing the browser to
http://localhost:8080/hello?person=World prints Hello World while pointing it to
http://localhost:8080/hello?person=TurboGears will print Hello Tu... as TurboGears is
now properly truncated.
Serving Static Files¶
Even for small web applications being able to apply style through CSS or serving javascript scripts is often required, to do so we must tell TurboGears to serve our static files and from where to serve them:
config = AppConfig(minimal=True, root_controller=RootController())
config.renderers = ['kajiki']
config.serve_static = True
config.paths['static_files'] = 'public'
application = config.make_wsgi_app()
After restating the application, any file placed inside the public directory will be
served directly by TurboGears. Supposing you have a style.css file you can access
it as http://localhost:8080/style.css.
Working With Database¶
TurboGears2 supports both SQL dbms through SQLAlchemy and MongoDB through Ming, both can be enabled with some options and by providing a Model for the application.
The following will cover how to work with SQLAlchemy and extend the sample application to log and retrieve a list of greeted people. First we will need to enable SQLAlchemy support for our application:
config['use_sqlalchemy'] = True
config['sqlalchemy.url'] = 'sqlite:///devdata.db'
Now TurboGears will configure a SQLAlchemy engine for us, but it will require that we provide a data model, otherwise it will just crash when starting up. This can be done by providing a database Session and a model initialization function:
from tg.util import Bunch
from sqlalchemy.orm import scoped_session, sessionmaker
DBSession = scoped_session(sessionmaker(autoflush=True, autocommit=False))
def init_model(engine):
DBSession.configure(bind=engine)
config['model'] = Bunch(
DBSession=DBSession,
init_model=init_model
)
This will properly make our application work and able to interact with the database, but it won’t do much as we are not actually declaring any table or model to work with.
Accessing Data¶
To start working with tables and the data they contain we need to declare the table itself, this can be done through the SQLAlchemy declarative layer by using a Declarative Base class:
from sqlalchemy.ext.declarative import declarative_base
DeclarativeBase = declarative_base()
From this class we can then inherit all our models:
from sqlalchemy import Column, Integer, DateTime, String
from datetime import datetime
class Log(DeclarativeBase):
__tablename__ = 'logs'
uid = Column(Integer, primary_key=True)
timestamp = Column(DateTime, nullable=False, default=datetime.utcnow)
person = Column(String(50), nullable=False)
This will allow us to read and write rows from the logs table, but before we are able
to do so we must ensure that the table actually exists, which can be done by extending our
model initialization function to create the tables:
def init_model(engine):
DBSession.configure(bind=engine)
DeclarativeBase.metadata.create_all(engine) # Create tables if they do not exist
Now we can finally extend our controller to log the people we greet and provide us the list of past greetings:
class RootController(TGController):
@expose(content_type='text/plain')
def index(self):
logs = DBSession.query(Log).order_by(Log.timestamp.desc()).all()
return 'Past Greetings\n' + '\n'.join(['%s - %s' % (l.timestamp, l.person) for l in logs])
@expose('hello.xhtml')
def hello(self, person=None):
DBSession.add(Log(person=person or ''))
DBSession.commit()
return dict(person=person)
Going Full Stack¶
While it is possible to manually enable the TurboGears features like the SQLAlchemy and Ming
storage backends, the application helpers, app_globals, i18n features through the
AppConfig object, if you need them you probably want to switch to full stack mode and
to create a full stack application through the gearbox quickstart command.
The Full Stack Tutorial provides an introduction to more complex applications with all the TurboGears features enabled, follow it if you want to unleash all the features that TurboGears provides!
Full Stack TurboGears: Wiki in 20 Minutes¶
How does TurboGears2 help you get development done quickly? We’ll show you by developing a simple wiki application that should take you no more than 20 minutes to complete. We’re going to do this without explaining the steps in detail (that is what this book is for, after all). As a result, you’ll see how easily you can make your own web applications once you are up to speed on what TurboGears2 offers.
If you’re not familiar with the concept of a wiki you might want to check out the Wikipedia entry. Basically, a wiki is an easily-editable collaborative web content system that makes it trivial to link to pages and create new pages. Like other wiki systems, we are going to use CamelCase words to designate links to pages.
If you have trouble with this tutorial ask for help on the TurboGears discussion list, or on the IRC channel #turbogears. We’re a friendly bunch and, depending what time of day you post, you’ll get your answer in a few minutes to a few hours. If you search the mailing list or the web in general you’ll probably get your answer even faster. Please don’t post your problem reports as comments on this or any of the following pages of the tutorial. Comments are for suggestions for improvement of the docs, not for seeking support.
If you want to see the final version you can download a copy of the wiki code.
Installing the Development Tools¶
The TurboGears2 Development Tools are a bunch of commands and extensions useful when
developing TurboGears2 applications. They provide the gearbox suite of commands
to create new full stack projects, quickly create controllers, templates, models and
the TurboGears debugbar.
(tgenv)$ pip install tg.devtools
Quickstart¶
TurboGears2 provides a suite of tools for working with projects by
adding several commands to the Python command line tool gearbox. A
few will be touched upon in this tutorial. (Check the
GearBox section for a full listing.) The first tool
you’ll need is quickstart, which initializes a TurboGears project.
Go to a command line window and run the following command:
(tgenv)$ gearbox quickstart wiki20
This will create a project called wiki20 with the default template engine and with authentication. TurboGears2 projects usually share a common structure, which should look like:
wiki20
├── __init__.py
├── config <-- Where project setup and configuration relies
├── controllers <-- All the project controllers, the logic of our web application
├── i18n <-- Translation files for the languages supported
├── lib <-- Utility python functions and classes
├── model <-- Database models
├── public <-- Static files like CSS, javascript and images
├── templates <-- Templates exposed by our controllers.
├── tests <-- Tests
└── websetup <-- Functions to execute at application setup. Like creating tables, a standard user and so on.
Note
We recommend you use the names given here: this documentation looks for files in directories based on these names.
You need to update the dependencies in the file Wiki-20/setup.py.
Look for a list named install_requires and append the docutils
entry at the end. TurboGears2 does not require docutils,
but the wiki we are building does.
Your install_requires should end up looking like:
install_requires=[
"TurboGears2 >= 2.3.9",
"Babel",
"Beaker",
"Kajiki",
"zope.sqlalchemy >= 0.4",
"sqlalchemy",
"alembic",
"repoze.who",
"tw2.forms",
"tgext.admin >= 0.6.1",
"WebHelpers2",
"docutils"
]
Now to be able to run the project you will need to install it and
its dependencies. This can be quickly achieved by running from
inside the wiki20 directory:
$ pip install -e .
Note
If you skip the pip install -e . command you might end up with an error that looks
like: pkg_resources.DistributionNotFound: tw2.forms: Not Found for: wiki20 (did you run python setup.py develop?)
This is because some of the dependencies your project depend on the options you choose while
quickstarting it.
You should now be able to start the newly create project with the gearbox serve command:
(tgenv)$ gearbox serve --reload --debug
Starting subprocess with file monitor
Starting server in PID 32797.
serving on http://127.0.0.1:8080
Note
The --reload option makes the server restart whenever a file is changed, this greatly speeds
up the development process by avoiding to manually restart the server whenever we need to try
our changes.
Note
The --debug option provides full stacktrace in case the server was unable to start, this
is useful in case your server didn’t start due to a configuration error.
Pointing your browser to http://127.0.0.1:8080/ should open up the TurboGears2 welcome page. By default newly quickstarted projects provide a bunch of pages to guide the user through some of the foundations of TurboGears2 web applications.
Controller And View¶
TurboGears follows the Model-View-Controller paradigm (a.k.a. “MVC”), as do most modern web frameworks like Rails, Django, Struts, etc.
Taking a look at the http://127.0.0.1:8080/about page is greatly suggested to get an overview of your newly quickstarted project and how TurboGears2 works.
If you take a look at the code that quickstart created, you’ll see
everything necessary to get up and running. Here, we’ll look at the
two files directly involved in displaying this welcome page.
Controller Code¶
Wiki-20/wiki20/controllers/root.py (see below) is the code that
causes the welcome page to be produced. After the imports the first
line of code creates our main controller class by inheriting from
TurboGears’ BaseController:
class RootController(BaseController):
The TurboGears 2 controller is a simple object publishing system; you
write controller methods and @expose() them to the web. In our
case, there’s a single controller method called index. As you
might guess, this name is not accidental; this becomes the default
page you’ll get if you go to this URL without specifying a particular
destination, just like you’ll end up at index.html on an ordinary
web server if you don’t give a specific file name. You’ll also go to
this page if you explicitly name it, with
http://localhost:8080/index. We’ll see other controller methods
later in the tutorial so this naming system will become clear.
The @expose() decorator tells TurboGears which template to use to
render the page. Our @expose() specifies:
@expose('wiki20.templates.index')
This gives TurboGears the template to use, including the path
information (the .xhtml extension is implied). We’ll look at this
file shortly.
Each controller method returns a dictionary, as you can see at the end
of the index method. TG takes the key:value pairs in this
dictionary and turns them into local variables that can be used in the
template.
from tg import expose, flash, require, url, request, redirect
#Skipping some imports here...
class RootController(BaseController):
secc = SecureController()
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
error = ErrorController()
def _before(self, *args, **kw):
tmpl_context.project_name = "Wiki 20"
@expose('wiki20.templates.index')
def index(self):
"""Handle the front-page."""
return dict(page='index')
#more controller methods from here on...
Displaying The Page¶
Wiki-20/wiki20/templates/index.xhtml is the template
specified by the @expose() decorator, so it formats what you view
on the welcome screen. Look at the file; you’ll see that it’s standard
XHTML with some simple namespaced attributes. This makes it very
designer-friendly, and well-behaved design tools will respect all the
Kajiki Template Language attributes and tags. You can even open it directly in your
browser.
Kajiki directives are elements and/or attributes in the template that
are prefixed with py:. They can affect how the template is
rendered in a number of ways: Kajiki provides directives for
conditionals and looping, among others. We’ll see some simple Kajiki
directives in the sections on Editing pages and
Adding views.
The following is the content of a newly quickstarted TurboGears2 project at 2.3 release time:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">Welcome to TurboGears 2.3, standing on the shoulders of giants, since 2007</title>
</head>
<body py:block="body" py:strip="True">
<div class="row">
<div class="col-md-8">
<div class="jumbotron">
<h1>Welcome to TurboGears 2.3</h1>
<p>If you see this page it means your installation was successful!</p>
<p>TurboGears 2 is rapid web application development toolkit designed to make your life easier.</p>
<p>
<a class="btn btn-primary btn-lg" href="http://www.turbogears.org" target="_blank">
${h.icon('book')} Learn more
</a>
</p>
</div>
</div>
<div class="col-md-4 hidden-xs hidden-sm">
<a class="btn btn-info btn-sm active" href="http://turbogears.readthedocs.io/en/latest">${h.icon('book')} TG2 Documentation</a> <span class="label label-success">new</span><em> Get Started</em><br/>
<br/>
<a class="btn btn-info btn-sm active" href="http://turbogears.readthedocs.io/en/latest/cookbook/cookbook.html">${h.icon('book')} TG2 CookBook</a><em> Read the Cookbook</em> <br/>
<br/>
<a class="btn btn-info btn-sm active" href="http://groups.google.com/group/turbogears">${h.icon('comment')} Join the Mail List</a> <em>for help/discussion</em><br/>
<br/>
<a class="btn btn-info btn-sm active" href="http://runnable.com/TurboGears">${h.icon('play')} Play on Runnable</a> <em>for basic examples</em><br/>
<br/>
<a class="btn btn-info btn-sm active" href="http://stackoverflow.com/questions/tagged/turbogears2">${h.icon('search')} Search Stackoverflow</a> <em>for questions</em>
</div>
</div>
<div class="row">
<div class="col-md-4">
<h3>Code your data model</h3>
<p> Design your data <code>model</code>, Create the database, and Add some bootstrap data.</p>
</div>
<div class="col-md-4">
<h3>Design your URL architecture</h3>
<p> Decide your URLs, Program your <code>controller</code> methods, Design your
<code>templates</code>, and place some static files (CSS and/or Javascript). </p>
</div>
<div class="col-md-4">
<h3>Distribute your app</h3>
<p> Test your source, Generate project documents, Build a distribution.</p>
</div>
</div>
<em class="pull-right small"> Thank you for choosing TurboGears.</em>
</body>
</html>
Wiki Model¶
quickstart produced a directory for our model in
Wiki-20/wiki20/model/. This directory contains an __init__.py
file, which makes that directory name into a python module (so you can
use import model).
Since a wiki is basically a linked collection of pages, we’ll define a
Page class as the name of our model.
Create a new file called Wiki-20/wiki20/model/page.py:
from sqlalchemy import *
from sqlalchemy.orm import mapper, relation
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Text
from wiki20.model import DeclarativeBase, metadata, DBSession
class Page(DeclarativeBase):
__tablename__ = 'page'
id = Column(Integer, primary_key=True)
pagename = Column(Text, unique=True)
data = Column(Text)
Now to let TurboGears know that our model exists we must make it available inside the Wiki-20/wiki20/model/__init__.py
file just by importing it at the end:
# Import your model modules here.
from wiki20.model.auth import User, Group, Permission
from wiki20.model.page import Page
Warning
It’s very important that this line is at the end because
Page requires the rest of the model to be initialized
before it can be imported:
Initializing The Tables¶
Now that our model is recognized by TurboGears we must create the table that it is going to use to store its data. By default TurboGears will automatically create tables for each model it is aware of, this is performed during the application setup phase.
The setup phase is managed by the Wiki-20/wiki20/websetup python module, we are just
going to add to``websetup/boostrap.py`` the lines required to create a FrontPage page for
our wiki, so it doesn’t start empty.
We need to update the file to create our FrontPage data just before
the DBSession.flush() command by adding:
page = model.Page(pagename="FrontPage", data="initial data")
model.DBSession.add(page)
You should end up having a try:except: block that should
look like:
def bootstrap(command, conf, vars):
#Some comments and setup here...
try:
#Users and groups get created here...
model.DBSession.add(u1)
page = model.Page(pagename="FrontPage", data="initial data")
model.DBSession.add(page)
model.DBSession.flush()
transaction.commit()
except IntegrityError:
#Some Error handling here...
The transaction.commit() call involves the transaction manager used
by TurboGears2 which helps us to support cross database transactions, as well as
transactions in non relational databases.
Now to actually create our table and our FrontPage we simply need to run
the gearbox setup-app command where your application configuration file is available
(usually the root of the project):
(tgenv)$ gearbox setup-app
Running setup_app() from wiki20.websetup
Creating tables
A file named Wiki-20/devdata.db should be created which contains
your sqlite database.
For other database systems refer to the sqlalchemy.url
line inside your configuration file.
Adding Controllers¶
Controllers are the code that figures out which page to display, what data to grab from the model, how to process it, and finally hands off that processed data to a template.
quickstart has already created some basic controller code for us
at Wiki-20/wiki20/controllers/root.py.
First, we must import the Page class from our model. At the end of
the import block, add this line:
from wiki20.model.page import Page
Now we will change the template used to present the data, by changing
the @expose('wiki20.templates.index') line to:
@expose('wiki20.templates.page')
This requires us to create a new template named page.xhtml in the wiki20/templates directory; we’ll do this in the next section.
Now we must specify which page we want to see. To do this, add a
parameter to the index() method. Change the line after the
@expose decorator to:
def index(self, pagename="FrontPage"):
This tells the index() method to accept a parameter called
pagename, with a default value of "FrontPage".
Now let’s get that page from our data model. Put this line in the
body of index:
page = DBSession.query(Page).filter_by(pagename=pagename).one()
This line asks the SQLAlchemy database session object to run a query
for records with a pagename column equal to the value of the
pagename parameter passed to our controller method. The
.one() method assures that there is only one returned result;
normally a .query call returns a list of matching objects. We only
want one page, so we use .one().
Finally, we need to return a dictionary containing the page we
just looked up. When we say:
return dict(wikipage=page)
The returned dict will create a template variable called
wikipage that will evaluate to the page object that we looked
it up.
Your index controller method should end up looking like:
from tg import expose, flash, require, url, request, redirect
#More imports here...
from wiki20.model.page import Page
class RootController(BaseController):
secc = SecureController()
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
error = ErrorController()
def _before(self, *args, **kw):
tmpl_context.project_name = "Wiki 20"
@expose('wiki20.templates.page')
def index(self, pagename="FrontPage"):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
#more controller methods from here on...
Now our index() method fetches a record from the database
(creating an instance of our mapped Page class along the way), and
returns it to the template within a dictionary.
Adding Views (Templates)¶
quickstart also created some templates for us in the
Wiki-20/wiki20/templates directory: master.xhtml and index.xhtml.
Back in our simple controller, we used @expose() to hand off a
dictionary of data to a template called 'wiki20.templates.index',
which corresponds to Wiki-20/wiki20/templates/index.xhtml.
Take a look at the following line in index.xhtml:
<html py:extends="master.xhtml" py:strip="True">
This tells the index template to extend the master
template. Using inheritance lets you easily maintain a cohesive look and
feel throughout your site by having each page include a common master
template.
Copy the contents of index.xhtml into a new file called page.xhtml. Now modify it for our purposes:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">${wikipage.pagename} - The TurboGears 2 Wiki</title>
</head>
<body py:block="body" py:strip="True">
<div class="main_content">
<div style="float:right; width: 10em;"> Viewing
<span py:replace="wikipage.pagename">Page Name Goes Here</span>
<br/>
You can return to the <a href="/">FrontPage</a>.
</div>
<div py:replace="wikipage.data">Page text goes here.</div>
<div>
<a href="/edit/${wikipage.pagename}">Edit this page</a>
</div>
</div>
</body>
</html>
This is a basic XHTML page with three substitutions:
- In the
<title>tag, we substitute the name of the page, using thepagenamevalue ofpage. (Remember,wikipageis an instance of our mappedPageclass, which was passed in a dictionary by our controller.):
<title>${wikipage.pagename} - The TurboGears 2 Wiki</title>
- In the second
<div>element, we substitute the page name again withpy:replace:
<span py:replace="wikipage.pagename">Page Name Goes Here</span>
- In the third
<div>, we put in the contents of our``wikipage``:
<div py:replace="wikipage.data">Page text goes here.</div>
When you refresh the output web page you should see “initial data” displayed on the page.
Note
py:replace replaces the entire tag (including start and end tags) with the value of the variable provided.
Editing pages¶
One of the fundamental features of a wiki is the ability to edit the page just by clicking “Edit This Page,” so we’ll create a template for editing. First, make a copy of page.xhtml:
cd wiki20/templates
cp page.xhtml edit.xhtml
We need to replace the content with an editing form and ensure people
know this is an editing page. Here are the changes for edit.xhtml.
Change the title in the header to reflect that we are editing the page:
<head py:block="head" py:strip="True"> <title>Editing: ${wikipage.pagename}</title> </head>
Change the div that displays the page:
<div py:replace="wikipage.data">Page text goes here.</div>
with a div that contains a standard HTML form:
<div> <form action="/save" method="post"> <input type="hidden" name="pagename" value="${wikipage.pagename}"/> <textarea name="data" py:content="wikipage.data" rows="10" cols="60"/> <input type="submit" name="submit" value="Save"/> </form> </div>
Now that we have our view, we need to update our controller in order
to display the form and handle the form submission. For displaying the
form, we’ll add an edit method to our controller in
Wiki-20/wiki20/controllers/root.py:
from tg import expose, flash, require, url, request, redirect
#More imports here...
from wiki20.model.page import Page
class RootController(BaseController):
secc = SecureController()
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
error = ErrorController()
def _before(self, *args, **kw):
tmpl_context.project_name = "Wiki 20"
@expose('wiki20.templates.page')
def index(self, pagename="FrontPage"):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
@expose(template="wiki20.templates.edit")
def edit(self, pagename):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
#more controller methods from here on...
For now, the new method is identical to the index method; the only
difference is that the resulting dictionary is handed to the edit
template. To see it work, go to
http://localhost:8080/edit/FrontPage . However, this only works because
FrontPage already exists in our database; if you try to edit a new
page with a different name it will fail, which we’ll fix in a later
section.
Don’t click that save button yet! We still need to write that method.
Saving Our Edits¶
When we displayed our wiki’s edit form in the last section, the form’s
action was /save. So, we need to make a method called
save in the Root class of our controller.
However, we’re also going to make another important change. Our
index method is only called when you either go to / or
/index. If you change the index method to the special method
_default, then _default will be automatically called whenever
nothing else matches. _default will take the rest of the URL and
turn it into positional parameters. This will cause the wiki to become
the default when possible.
Here’s our new version of root.py which includes both _default
and save:
from tg import expose, flash, require, url, request, redirect
#More imports here...
from wiki20.model.page import Page
class RootController(BaseController):
secc = SecureController()
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
error = ErrorController()
def _before(self, *args, **kw):
tmpl_context.project_name = "Wiki 20"
@expose('wiki20.templates.page')
def _default(self, pagename="FrontPage"):
"""Handle the front-page."""
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
@expose(template="wiki20.templates.edit")
def edit(self, pagename):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
@expose()
def save(self, pagename, data, submit):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
page.data = data
redirect("/" + pagename)
#more controller methods from here on...
Unlike the previous methods we’ve made, save just uses a plain
@expose() without any template specified. That’s because we’re
only redirecting the user back to the viewing page.
Although the page.data = data statement tells SQLAlchemy that you
intend to store the page data in the database, you would usually
need to flush the SQLAlchemy Unit of Work and commit the currently
running transaction, those are operations that TurboGears2
transaction management will automatically do for us.
You don’t have to do anything to use this transaction management system, it should just work. So, you can now make changes and save the page we were editing, just like a real wiki.
What About WikiWords?¶
Our wiki doesn’t yet have a way to link pages. A typical wiki will automatically create links for WikiWords when it finds them (WikiWords have also been described as WordsSmashedTogether). This sounds like a job for a regular expression.
Here’s the new version of our RootController._default method,
which will be explained afterwards:
from tg import expose, flash, require, url, request, redirect
#More imports here...
from wiki20.model.page import Page
import re
from docutils.core import publish_parts
wikiwords = re.compile(r"\b([A-Z]\w+[A-Z]+\w+)")
class RootController(BaseController):
secc = SecureController()
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
error = ErrorController()
def _before(self, *args, **kw):
tmpl_context.project_name = "Wiki 20"
@expose('wiki20.templates.page')
def _default(self, pagename="FrontPage"):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
content = publish_parts(page.data, writer_name="html")["html_body"]
root = url('/')
content = wikiwords.sub(r'<a href="%s\1">\1</a>' % root, content)
return dict(content=content, wikipage=page)
@expose(template="wiki20.templates.edit")
def edit(self, pagename):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
return dict(wikipage=page)
@expose()
def save(self, pagename, data, submit):
page = DBSession.query(Page).filter_by(pagename=pagename).one()
page.data = data
redirect("/" + pagename)
#more controller methods from here on...
We need some additional imports, including re for regular
expressions and a method called publish_parts from docutils.
A WikiWord is a word that starts with an uppercase letter, has a
collection of lowercase letters and numbers followed by another
uppercase letter and more letters and numbers. The wikiwords
regular expression describes a WikiWord.
In _default, the new lines begin with the use of publish_parts,
which is a utility that takes string input and returns a dictionary of
document parts after performing conversions; in our case, the
conversion is from Restructured Text to HTML. The input
(page.data) is in Restructured Text format, and the output format
(specified by writer_name="html") is in HTML. Selecting the
fragment part produces the document without the document title,
subtitle, docinfo, header, and footer.
You can configure TurboGears so that it doesn’t live at the root of a
site, so you can combine multiple TurboGears apps on a single
server. Using tg.url() creates relative links, so that your links
will continue to work regardless of how many apps you’re running.
The next line rewrites the content by finding any WikiWords and
substituting hyperlinks for those WikiWords. That way when you click
on a WikiWord, it will take you to that page. The r'string' means
‘raw string’, one that turns off escaping, which is mostly used in
regular expression strings to prevent you from having to double escape
slashes. The substitution may look a bit weird, but is more
understandable if you recognize that the %s gets substituted with
root, then the substitution is done which replaces the \1 with
the string matching the regex.
Note that _default() is now returning a dict containing an
additional key-value pair: content=content. This will not break
wiki20.templates.page because that page is only looking for
page in the dictionary, however if we want to do something
interesting with the new key-value pair we’ll need to edit
wiki20.templates.page:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">${wikipage.pagename} - The TurboGears 2 Wiki</title>
</head>
<body py:block="body" py:strip="True">
<div class="main_content">
<div style="float:right; width: 10em;"> Viewing
<span py:replace="wikipage.pagename">Page Name Goes Here</span>
<br/>
You can return to the <a href="/">FrontPage</a>.
</div>
<div py:replace="Markup(content)">Formatted content goes here.</div>
<div>
<a href="/edit/${wikipage.pagename}">Edit this page</a>
</div>
</div>
</body>
</html>
Since content comes through as XML, we can strip it off using the
Markup() function to produce plain text (try removing the function
call to see what happens).
To test the new version of the system, edit the data in your front page to include a WikiWord. When the page is displayed, you’ll see that it’s now a link. You probably won’t be surprised to find that clicking that link produces an error.
Hey, Where’s The Page?¶
What if a Wiki page doesn’t exist? We’ll take a simple approach: if the page doesn’t exist, you get an edit page to use to create it.
In the _default method, we’ll check to see if the page exists.
If it doesn’t, we’ll redirect to a new notfound method. We’ll add
this method after the _default method and before the edit
method.
Here are the new notfound and the updated _default
methods for our RootController class:
@expose('wiki20.templates.page')
def _default(self, pagename="FrontPage"):
from sqlalchemy.exc import InvalidRequestError
try:
page = DBSession.query(Page).filter_by(pagename=pagename).one()
except InvalidRequestError:
raise redirect("notfound", pagename=pagename)
content = publish_parts(page.data, writer_name="html")["html_body"]
root = url('/')
content = wikiwords.sub(r'<a href="%s\1">\1</a>' % root, content)
return dict(content=content, wikipage=page)
@expose("wiki20.templates.edit")
def notfound(self, pagename):
page = Page(pagename=pagename, data="")
DBSession.add(page)
return dict(wikipage=page)
In the _default code we now first try to get the page and
then deal with the exception by redirecting to a method that
will make a new page.
As for the notfound method, the first two lines of the method add
a row to the page table. From there, the path is exactly the same it
would be for our edit method.
With these changes in place, we have a fully functional wiki. Give it a try! You should be able to create new pages now.
Adding A Page List¶
Most wikis have a feature that lets you view an index of the pages. To add one, we’ll start with a new template, pagelist.xhtml. We’ll copy page.xhtml so that we don’t have to write the boilerplate.
cd wiki20/templates
cp page.xhtml pagelist.xhtml
After editing, our pagelist.xhtml looks like:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">Page Listing - The TurboGears 2 Wiki</title>
</head>
<body py:block="body" py:strip="True">
<div class="main_content">
<h1>All Pages</h1>
<ul>
<li py:for="pagename in pages">
<a href="${tg.url('/' + pagename)}"
py:content="pagename">
Page Name Here.
</a>
</li>
</ul>
Return to the <a href="/">FrontPage</a>.
</div>
</body>
</html>
The highlighted section represents the template code of interest. You can
guess that the py:for is a python for loop, modified to fit
into Kajiki’s XML. It iterates through each of the pages (which
we’ll send in via the controller, using a modification you’ll see
next). For each one, Page Name Here is replaced by pagename,
as is the URL. You can learn more about the Kajiki Template Language.
We must also modify the RootController class to implement pagelist and to
create and pass pages to our template:
@expose("wiki20.templates.pagelist")
def pagelist(self):
pages = [page.pagename for page in DBSession.query(Page).order_by(Page.pagename)]
return dict(pages=pages)
Here, we select all of the Page objects from the database, and
order them by pagename.
We can also modify page.xhtml so that the link to the page list is available on every page:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">${wikipage.pagename} - The TurboGears 2 Wiki</title>
</head>
<body py:block="body" py:strip="True">
<div class="main_content">
<div style="float:right; width: 10em;"> Viewing
<span py:replace="wikipage.pagename">Page Name Goes Here</span>
<br/>
You can return to the <a href="/">FrontPage</a>.
</div>
<div py:replace="Markup(content)">Formatted content goes here.</div>
<div>
<a href="/edit/${wikipage.pagename}">Edit this page</a>
<a href="/pagelist">View the page list</a>
</div>
</div>
</body>
</html>
You can see your pagelist by clicking the link on a page or by going directly to http://localhost:8080/pagelist .
Further Exploration¶
Now that you have a working Wiki, there are a number of further places to explore:
- Continue to the Rapid Prototyping Tutorial: Using the TurboGears Admin.
- You can learn more about the Kajiki Template Language.
- You can learn more about the SQLAlchemy ORM.
If you had any problems with this tutorial, or have ideas on how to make it better, please let us know on the mailing list! Suggestions are almost always incorporated.
Rapid Prototyping Tutorial: Using the TurboGears Admin¶
TurboGears2 Wikier application is inspired by the TurboGears2 20 Minutes Wiki Tutorial.
While the 20 Minutes Wiki is a great way to learn writing custom TurboGears2 applications the Wikier tutorial tries to provide more focus on rapid prototyping with the ambitious target of making it possible to create a full featured Wiki in minimum possible time.
It will showcase some basic concepts of the TurboGears2 Web Framework and how to best use rapid prototyping tools available in the standard extension modules.
Contents:
Creating Project Structure¶
Hint
This tutorial has been written for TurboGears 2.3 on Python2.7. While it might work with previous or later versions, it has been tested only for version 2.3.
Setting up our Environment¶
If this is your first TurboGears2 project you need to create an environment and install the TurboGears2 web framework to make the development commands available.
Completed version of this tutorial is available on http://runnable.com/U8P0CQTKHwNzQoYs/turbogears-wikier-tutorial-for-python.
If you want to play around with this tutorial without installing TurboGears on your computer you can freely edit the Runnable version.
Creating the Environment¶
First we are going to create a Virtual Environment where we will install the framework.
This helps keeping our system clean by not installing the packages system-wide.
To do so we need to install the virtualenv package:
$ pip install virtualenv
Now the virtualenv command should be available and we can create and activate a virtual environment for our TurboGears2 project:
$ virtualenv tgenv
$ . tgenv/bin/activate
If our environment got successfully created and activated we should end up with a prompt that looks like:
(tgenv)$
Installing TurboGears2¶
TurboGears2 can be quickly installed by installing the TurboGears2 development tools. This will install TurboGears2 itself and a bunch of commands useful when developing TurboGears applications:
(tgenv)$ pip install tg.devtools
Note
The -i http://tg.gy/VERSION option is used to make sure that we install TurboGears2 version and its dependencies at the right version. TurboGears2 doesn’t usually enforce version dependencies to make it possible for developers to upgrade them if they need a bugfix or new features. It is suggested to always use the -i option to avoid installing incompatible packages.
Creating the Project¶
If the install correctly completed the gearbox quickstart command should be available
in your virtual environment:
(tgenv)$ gearbox quickstart wikir
This will create a project called wikir with the default template engine and with authentication. TurboGears2 projects usually share a common structure, which should look like:
wikir
├── __init__.py
├── config <-- Where project setup and configuration is located
├── controllers <-- All the project controllers, the logic of our web application
├── i18n <-- Translation files for the languages supported
├── lib <-- Utility python functions and classes
├── model <-- Database models
├── public <-- Static files like CSS, javascript and images
├── templates <-- Templates exposed by our controllers
├── tests <-- Tests
└── websetup <-- Functions to execute at application setup like creating tables, a standard user and so on.
Installing Project and its Dependencies¶
Before we can start our project and open it into a browser we must install any dependency that is not strictly related to TurboGears itself. This can easily be achieved running the develop command which will install into our environment the project itself and all its dependencies:
(tgenv)$ cd wikir
(tgenv)$ pip install -e .
Project depndencies are specified inside the setup.py file in the install_requires list.
Default project dependencies should look like:
install_requires=[
"TurboGears2 >= 2.3.0",
"Kajiki",
"zope.sqlalchemy >= 0.4",
"sqlalchemy",
"sqlalchemy-migrate",
"repoze.who",
"tgext.admin >= 0.5.1",
"repoze.who.plugins.sa",
"tw2.forms",
]
Kajiki dependency is the template engine our application is going to use, the zope.sqlalchemy, sqlalchemy and sqlalchemy-migrate dependencies are there to provide support for SQLALchemy based database layer. repoze.who and repoze.who.plugins.sa are used by the authentication and authorization layer. tgext.admin and tw2.forms are used to generate administrative interfaces and forms.
Serving our Project¶
Note
If you skipped the pip install -e . command you might end up with an error that looks
like: pkg_resources.DistributionNotFound: tw2.forms: Not Found for: wikir (did you run python setup.py develop?)
This is because some of the dependencies your project has depend on the options you choose while
quickstarting it.
You should now be able to start the newly create project with the gearbox serve command:
(tgenv)$ gearbox serve --reload
Starting subprocess with file monitor
Starting server in PID 32797.
serving on http://127.0.0.1:8080
Note
The –reload option makes the server restart whenever a file is changed, this greatly speeds up the development process by avoiding having to manually restart the server whenever we need to try our changes.
Pointing your browser to http://127.0.0.1:8080/ should open up the TurboGears2 welcome page. By default newly quickstarted projects provide a bunch of pages to guide the user through some of the foundations of TurboGears2 web applications. Taking a look at the http://127.0.0.1:8080/about page can provide a great overview of your newly quickstarted project.
Creating and Managing Wiki Pages¶
If you correctly quickstarted your project with sqlalchemy database support and authentication
you should end up having a model directory which contains database layer initialization and
User, Group and Permission models.
Those are the standard turbogears2 authentication models. You can freely customize them, but for now we will stick to the standard ones.
To manage our pages we are going to add model that represent a Wiki Page with attributes to store the title of the page, page data and last time the page got modified.
WikiPage Model¶
To define the model we are going to add a wiki.py file inside the wikir/model directory
which contains the model definition itself:
# -*- coding: utf-8 -*-
from sqlalchemy import *
from sqlalchemy.orm import mapper, relation, relation, backref
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, DateTime
from wikir.model import DeclarativeBase, metadata, DBSession
from datetime import datetime
class WikiPage(DeclarativeBase):
__tablename__ = 'wiki_page'
uid = Column(Integer, primary_key=True)
updated_at = Column(DateTime, default=datetime.utcnow, nullable=False)
title = Column(Unicode(255), nullable=False, unique=True)
data = Column(Unicode(4096), nullable=False, default='')
Now to let TurboGears know that our model exists we must make it available inside the wikir/model/__init__.py
file just by importing it at the end:
# Import your model modules here.
from wikir.model.auth import User, Group, Permission
from wikir.model.wiki import WikiPage
Creating Tables and setting up Application¶
Now that our model is recognized by TurboGears we must create the table that it is going to use to store its data. By default TurboGears will automatically create tables for each model it is aware of. This is performed during the application setup phase.
To setup your application you simply need to run the gearbox setup-app command where your application
configuration file is available (usually the root of the project):
(tg22env)$ gearbox setup-app
Running setup_app() from wikir.websetup
Creating tables
The Application setup process, apart from creating tables for the known models, will also execute the
wikir/websetup/boostrap.py module, which by default is going to create an administrator
user for our application.
Managing Pages through the Admin¶
Through the manager user that has been created during the setup phase it is possible to get access to the TurboGears Admin at http://localhost:8080/admin. The first time the page is accessed it will ask for authentication. Simply provide the username and password of the user that the setup-app command created for us:
Username: manager
Password: managepass
You should end up being redirected to the administration page. One of the links on the page should
point to WikiPages administration page http://localhost:8080/admin/wikipages/
On this page a list of the existing pages is provided with a link to create a New WikiPage.
Note
If you don’t find the WikiPages link on the administrator page, make sure you correctly
imported the WikiPage model at the end of wikir/model/__init__.py and run the
setup-app command again.
Customizing Management¶
Now that we have a working adiministration page for our WikiPages, we are going to tune a bunch of things to improve it.
First of all we are going to hide the updated_at fields. This will get automatically updated to the current time, so we don’t really want to let users modify it.
Then if you tried to click the New WikiPage link you probably saw that for the title of our web page a TextArea is used, probably a TextField would be a better match to make more clear that the user is supposed to provide a short single line title.
Last but not least we are going to provide a bit more space for the page data, to make it easier to edit the page.
All these changes can be made from our model by specifying a special attribute called __sprox__
which will be used by the administrative interface to tune the look and feel of the tables and
forms it is going to generate:
# -*- coding: utf-8 -*-
"""Wiki Page module."""
from sqlalchemy import *
from sqlalchemy.orm import mapper, relation, relation, backref
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode
from wikir.model import DeclarativeBase, metadata, DBSession
from datetime import datetime
from tw2.forms import TextField
from tw2.core import IntValidator
class WikiPage(DeclarativeBase):
__tablename__ = 'page'
uid = Column(Integer, primary_key=True)
updated_at = Column(DateTime, default=datetime.utcnow, nullable=False)
title = Column(Unicode(255), nullable=False, unique=True)
data = Column(Unicode(4096), nullable=False, default='')
class __sprox__(object):
hide_fields = ['updated_at']
field_widget_args = {'data': {'rows':15}}
Going back to our administration page at http://localhost:8080/admin/wikipages/ and clicking on the New WikiPage link you will see a form with just a single line entry field for the title and a wide textarea for the page data.
Feel free to add as many pages as you like; we are going to see later how to display them.
Serving Wiki Pages¶
We are now able to create, edit and delete Wiki Pages, but we are still unable to serve them.
Without serving pages our wiki is actually useless, so we are going to add a controller and template to make them available.
WebSite Index¶
To make our wiki navigable we are going to create a new index page with a sidebar containing all the available wiki pages, so the user can easily move around.
Pages Slug and Content¶
To create links to the pages and display their content we are going to add url and html_content properties to the page model. The first property will create the slug for the model and provide the url where the page is available, while the second will give back the page content parsed accordingly to the Markdown language.
To generate the slugs we are going to use tgext.datahelpers, and to
process markdown we are going to use the markdown library. So
the first thing we are going to do is add them to our project setup.py
file inside the install_requires list:
install_requires=[
"TurboGears2 >= 2.3.4",
"Kajiki",
"zope.sqlalchemy >= 0.4",
"sqlalchemy",
"alembic",
"repoze.who",
"tw2.forms",
"tgext.admin >= 0.6.1",
"markdown",
"tgext.datahelpers"
]
Then we need to run again pip install -e . to install our new
project dependency:
(tg22env)$ pip install -e .
Successfully installed tgext.datahelpers markdown wikir
Cleaning up...
Now that we installed the datahelpers we can add the url and html_content properties to our WikiPage model. Our model should end up looking like:
#all the other sqlalchemy imports here...
import tg
from tgext.datahelpers.utils import slugify
from markdown import markdown
class WikiPage(DeclarativeBase):
__tablename__ = 'page'
uid = Column(Integer, primary_key=True)
updated_at = Column(DateTime, default=datetime.utcnow, nullable=False)
title = Column(Unicode(255), nullable=False, unique=True)
data = Column(Unicode(4096), nullable=False, default='')
@property
def url(self):
return tg.url('/'+slugify(self, self.title))
@property
def html_content(self):
return markdown(self.data)
class __sprox__(object):
hide_fields = ['updated_at']
field_widget_args = {'data': {'rows':15}}
Index Controller¶
Now that we are able to retrieve the url for each wiki page, we need to retrieve the list of the wiki pages with their urls so that our index page can display the sidebar.
Our index page is a wiki page itself, so we are also going to load up it’s content from the page titled “index”.
To do so we must edit the RootController class inside the wikir/controllers/root.py
file and look for the index method. When you found it change it to look like:
@expose('wikir.templates.index')
def index(self):
wikipages = [(w.url, w.title) for w in DBSession.query(model.WikiPage).filter(model.WikiPage.title!='index')]
indexpage = DBSession.query(model.WikiPage).filter_by(title='index').first()
if not indexpage:
content = 'Index page not available, please create a page titled index'
else:
content = indexpage.html_content
return dict(page='index', wikipages=wikipages, content=content)
TurboGears2 controllers are just plain python methods with an @expose decorator.
The @expose decorator tells to TurboGears2 which template the controller is going to display
and make so that all the data that our controller returns will be available inside
the template itself.
If you are still asking yourself why connecting to http://localhost:8080/ you ended up being served by the RootController.index method you probably want to take a look at TurboGears2 documentation about Request Dispatching & Controllers and try to understand how Object Dispatch routing works.
Index Template¶
Now, if you reloaded to your index page you probably already noticed that nothing changed. This is because our controller retrieved the wiki pages, but we didn’t expose them in the index template in any place.
The index template is available as wikir/templates/index.html which is exactly
the same path written inside the @expose decorator but with / replaced by dots and
without the template extension.
We are going to provide a really simple template, so what is currently available inside the file is going to just be removed and replaced with:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">Wikier Index</title>
</head>
<body py:block="body" py:strip="True">
<div class="row">
<div class="col-md-3">
<ul>
<li py:for="url, title in wikipages">
<a href="${url}">${title}</a>
</li>
</ul>
</div>
<div class="col-md-9">
<div>
${Markup(content)}
</div>
</div>
</div>
</body>
</html>
Serving all Wiki pages¶
If you tried clicking on any link in our sidebar your probably noticed that they all lead to a 404 page. This is because we still haven’t implemented any controller method that is able to serve them.
Page Template¶
First we are going to create a template for our wiki pages and save it as
wikir/templates/page.html. The content of our template will look like:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">${title}</title>
</head>
<body py:block="body" py:strip="True">
<div class="row">
<div class="col-md-12">
<h2>${title}</h2>
${Markup(content)}
<a py:if="request.identity and 'managers' in request.identity['groups']"
href="${tg.url('/admin/wikipages/%s/edit' % page_id)}">
edit
</a>
</div>
</div>
</body>
</html>
Page Controller¶
Now that we have our template we just need to bind it a controller
which is going to render the page. To do this we are going to use
the special _default controller method. This is a method that
turbogears will call if it’s unable to find the exact method request
by the url.
As our wiki pages have a all different names they will all end up
in _default and we will be able to serve them from there. Just
edit wikir/controller/root.py and add the _default method
to the RootController:
from tg import validate
from tgext.datahelpers.validators import SQLAEntityConverter
from tgext.datahelpers.utils import fail_with
@expose('wikir.templates.page')
@validate({'page':SQLAEntityConverter(model.WikiPage, slugified=True)},
error_handler=fail_with(404))
def _default(self, page, *args, **kw):
return dict(page_id=page.uid, title=page.title, content=page.html_content)
The @validate decorator makes possible to apply validators
to the incoming parameters and if validation fails the specified
error_handler is called. In this case we are checking if there
is a web page with the given slug. If it fails to find one
it will just return a 404 page.
If the page is available the page instance is returned, so our controller ends just returning the data of the page to the template.
If you now point your browser to the index and click any of the links in the sidebar you will see that they now lead to the linked page instead of failing with a 404 like before.
Note
If you don’t have any links in the left bar, just go to the admin page and create as many pages as you like.
Our wiki is actually finished, but in the upcoming sections we are going to see how we can improve it by introducing caching.
Advanced Admin Customizations¶
TurboGears admin configurations work through the TGAdminConfig class, which
makes it possible to change the behavior for each model. We are going to use the
EasyCrudRestController to perform quick tuning of our administrative interface.
Displaying the Slug¶
Right now our admin shows us the page id and title, but doesn’t provide a link to the page itself, so it’s hard to see how a page looks after we edit it.
To solve this issue we are going to replace the page id with a link to the page itself inside the administration table.
Custom Admin Config¶
The first step is provide a custom admin config which removes the page id field.
We are going to add this in wikir/controllers/root.py:
from tgext.crud import EasyCrudRestController
from tgext.admin.config import CrudRestControllerConfig
class WikiPageAdminController(EasyCrudRestController):
__table_options__ = {'__omit_fields__':['uid']}
class CustomAdminConfig(TGAdminConfig):
class wikipage(CrudRestControllerConfig):
defaultCrudRestController = WikiPageAdminController
Once you declared your custom admin config, inside your RootController
there should be a line which looks like:
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
Replace that one with:
admin = AdminController(model, DBSession, config_type=CustomAdminConfig)
When you reload the wiki pages administration table you should see that the page uid is not there anymore.
The Slug Column¶
We are now going to replace the previous uid field with a url column which contains a link to the page.
To do so we have to tell our table that there an html type column (so its
content doesn’t get escaped) and how to generate the content for that column.
This can be done inside the WikiPageAdminController that we just declared:
class WikiPageAdminController(EasyCrudRestController):
__table_options__ = {
'__omit_fields__':['uid'],
'__field_order__':['url'],
'__xml_fields__':['url'],
'url': lambda filler, row: '<a href="%(url)s">%(url)s</a>' % dict(url=row.url)
}
Note
The __field_order__ option is necessary to let the admin know that we
have a url field that we want to show. Otherwise it will just know
how to show it thanks to the __xml_fields__ and slug properties
but won’t know where it has to be displayed.
Extending the Admin Further¶
If you want to further customize the admin behaviour have a look at the Working with the TurboGears Admin documentation.
Wiki Page Generation Caching¶
This section of the tutorial will show how to use the updated_at field of our models for different kinds of caching that can greatly speed up our website.
Updating updated_at¶
Right now the updated_at field of our models will only contain the time they get created, because we are never updating it. We could add a SQLAlchemy event that updates it each time the WikiPage object is updated, and that would probably be the suggested way to handle this.
As that solution would be out of scope for this tutorial, we are going to perform the same by customizing the TurboGears Admin.
To do so we have to customize the WikiPageAdminController.put method:
from tgext.admin.config import CrudRestControllerConfig
from datetime import datetime
class WikiPageAdminController(EasyCrudRestController):
__table_options__ = {'__omit_fields__':['uid'],
'__field_order__':['url'],
'__xml_fields__':['url'],
'url': lambda filler, row: '<a href="%(url)s">%(url)s</a>' % dict(url=row.url)
}
@expose(inherit=True)
def put(self, *args, **kw):
kw['updated_at'] = datetime.utcnow()
return super(WikiPageAdminController, self).put(*args, **kw)
This way each time a wiki page is modified its updated_at field will be updated accordingly.
Caching Page¶
Now that we know when the page got updated we can use it speed up our wiki by caching wiki page generation.
The first thing we need to do is move the html content generation inside our template
instead of using it directly from the controller. This can easily be done by
updating our RootController._default method accordingly:
@expose('wikir.templates.page')
@validate({'page':SQLAEntityConverter(model.WikiPage, slugified=True)},
error_handler=fail_with(404))
def _default(self, page, *args, **kw):
return dict(wikipage=page)
Our controller now just retrieves the page and passes it to our template,
so we have to do some minor tuning to the wikir/templates/page.html template
too:
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">${wikipage.title}</title>
</head>
<body py:block="body" py:strip="True">
<div class="row">
<div class="col-md-12">
<h2>${wikipage.title}</h2>
${Markup(wikipage.html_content)}
<a py:if="request.identity and 'managers' in request.identity['groups']"
href="${tg.url('/admin/wikipages/%s/edit' % wikipage.uid)}">
edit
</a>
</div>
</div>
</body>
</html>
Now that the work of converting the page markdown content to HTML is done by our template we can simply cache the template rendering process.
This way we will both skip the template generation phase and the page content
conversion phase at once. TurboGears2 provides a great tool for template caching.
You just need to generate a cache key and provide it inside the tg_cache dictionary
returned by your controller:
@expose('wikir.templates.page')
@validate({'page':SQLAEntityConverter(model.WikiPage, slugified=True)},
error_handler=fail_with(404))
def _default(self, page, *args, **kw):
cache_key = '%s-%s' % (page.uid, page.updated_at.strftime('%Y%m%d%H%M%S'))
return dict(wikipage=page, tg_cache={'key':cache_key, 'expire':24*3600, 'type':'memory'})
This will keep our template cached in memory up to a day and will still regenerate
the page whenever our wikipage changes as we are using the updated_at field
to generate our cache key.
Page Caching Performances Gain¶
By just the minor change of caching the template the throughput of the applications on my computer greatly increased. A quick benchmark can give the idea of the impact of such a change:
$ /usr/sbin/ab -c 1 -n 500 http://127.0.0.1:8080/this-is-my-first-page-1
Requests per second: 97.55 [#/sec] (mean)
$ /usr/sbin/ab -c 1 -n 500 http://127.0.0.1:8080/this-is-my-first-page-1
Requests per second: 267.18 [#/sec] (mean)
TurboGears2 Documentation¶
This section covers documentation about getting started with TurboGears2.
The following chapters cover the most common features of TurboGears, introductions to them and some intermediate features.
Request Dispatching & Controllers¶
The nerve center of your TurboGears application is the controller. It ultimately handles all user actions, because every HTTP request arrives here first. The controller acts on the request and can call upon other TurboGears components (the template engines, database layers, etc.) as its logic directs.
Basic Dispatch¶
When the TurboGears server receives an HTTP request, the requested URL
is mapped as a call to your controller code located in the
controllers package. Page names map to other controllers or
methods within the controller class.
For example:
| URL | Maps to |
|---|---|
http://localhost:8080/index |
RootController.index() |
http://localhost:8080/mypage |
RootController.mypage() |
Index and Catch-All pages¶
Suppose using gearbox quickstart you generate a TurboGears project
named “HelloWorld”. Your default controller code would be created in
the file HelloWorld/helloworld/controllers/root.py.
Modify the default root.py to read as follows:
"""Main Controller"""
from helloworld.lib.base import BaseController
from tg import expose, flash
#from tg import redirect, validate
#from helloworld.model import DBSession
class RootController(BaseController):
@expose()
def index(self):
return "<h1>Hello World</h1>"
@expose()
def _default(self, *args, **kw):
return "This page is not ready"
When you load the root URL http://localhost:8080/index in your web
browser, you’ll see a page with the message “Hello World” on it. In
addition, any of these URLs will return the same result.
Implementing A Catch-All Url Via The _default() Method¶
URLs not explicitly mapped to other methods of the controller will
generally be directed to the method named _default(). With the
above example, requesting any URL besides /index, for example
http://localhost:8080/hello, will return the message “This page is
not ready”.
Adding More Pages¶
When you are ready to add another page to your site, for example at the URL
http://localhost:8080/anotherpage
add another method to class RootController as follows:
@expose()
def anotherpage(self):
return "<h1>There are more pages in my website</h1>"
Now, the URL /anotherpage will return:
There are more pages in my website
Line By Line Explanation¶
"""Main Controller"""
from helloworld.lib.base import BaseController
from tg import expose, flash
from tg.i18n import ugettext as _
#from tg import redirect, validate
#from helloworld.model import DBSession
First you need to import the required modules.
There’s a lot going on here, including some stuff for internationalization. But we’re going to gloss over some of that for now. The key thing to notice is that you are importing a BaseController, which your RootController must inherit from. If you’re particularly astute, you’ll have noticed that you import this BaseController from the lib module of your own project, and not from TurboGears.
TurboGears provides ObjectDispatch system through the TGController class which is imported in the lib folder of the current project (HelloWorld/helloworld/lib) so that you can modify it to suit the needs of your application. For example, you can define actions which will happen on every request, add parameters to every template call, and otherwise do what you need to the request on the way in, and on the way out.
The next thing to notice is that we are importing expose from tg.
BaseController classes and the expose decorator are the basis of TurboGears
controllers. The @expose decorator declares that your method should be
exposed to the web, and provides you with the ability to say how the results
of the controller should be rendered.
The other imports are there in case you do internationalization, use the HTTP redirect function, validate inputs/outputs, or use the models.
class RootController(BaseController):
RootController is the required standard name for the
RootController class of a TurboGears application and it should inherit
from the BaseController class. It is thereby specified as the
request handler class for the website’s root.
In TurboGears 2 the web site is represented by a tree of controller
objects and their methods, and a TurboGears website always grows out
from the RootController class.
def index(self):
return "<h1>Hello World</h1>"
We’ll look at the methods of the RootController class next.
The index method is the start point of any TurboGears controller
class. Each of the URLs
is mapped to the RootController.index() method.
If a URL is requested and does not map to a specific method, the
_default() method of the controller class is called:
def _default(self):
return "This page is not ready"
In this example, all pages except the these urls listed above will map to the _default method.
As you can see from the examples, the response to a given URL is determined by the method it maps to.
@expose()
The @expose() seen before each controller method directs
TurboGears controllers to make the method accessible through the web
server. Methods in the controller class that are not “exposed” can
not be called directly by requesting a URL from the server.
There is much more to @expose(). It will be our access to TurboGears sophisticated rendering features that we will explore shortly.
Exposing Templates¶
As shown above, controller methods return the data of your website. So far, we have returned this data as literal strings. You could produce a whole site by returning only strings containing raw HTML from your controller methods, but it would be difficult to maintain, since Python code and HTML code would not be cleanly separated.
Expose + Template == Good¶
To enable a cleaner solution, data from your TurboGears controller can be returned as strings, or as a dictionary.
With @expose(), a dictionary can be passed from the controller to a template
which fills in its placeholder keys with the dictionary values and then returns
the filled template output to the browser.
Template Example¶
A simple template file called sample could be made like
this:
<html>
<head>
<title>TurboGears Templating Example</title>
</head>
<body>
<h2>I just want to say that ${person} should be the next
${office} of the United States.</h2>
</body>
</html>
The ${param} syntax in the template indicates some undetermined
values to be filled.
We provide them by adding a method to the controller like this …
@expose("helloworld.templates.sample")
def example(self):
mydata = {'person':'Tony Blair','office':'President'}
return mydata
… then the following is made possible:
The web user goes to
http://localhost:8080/example.The
examplemethod is called.The method
examplereturns a Pythondict.@expose processes the dict through the template file named
sample.html.The dict values are substituted into the final web response.
The web user sees a marked up page saying:
I just want to say that Tony Blair should be the next President of the United States.
Template files can thus house all markup information, maintaining clean separation from controller code.
For more on templating have a look at Templating
SubControllers And The URL Hierarchy¶
Sometimes your web-app needs a URL structure that’s more than one level deep.
TurboGears provides for this by traversing the object hierarchy, to find a method that can handle your request.
To make a sub-controller, all you need to do is make your
sub-controller inherit from the object class. However there’s a
SubController class Controller in your project’s lib.base
(HelloWorld/helloworld/lib/base.py) for you to use if you want a
central place to add helper methods or other functionality to your
SubControllers:
from lib.base import BaseController
from tg import redirect
class MovieController(BaseController):
@expose()
def index(self):
redirect('list/')
@expose()
def list(self):
return 'hello'
class RootController(BaseController):
movie = MovieController()
With these in place, you can follow the link:
and you will be redirected to:
Unlike turbogears 1, going to http://localhost:8080/movie will not redirect you to http://localhost:8080/movie/list. This is due to some interesting bit about the way WSGI works. But it’s also the right thing to do from the perspective of URL joins. Because you didn’t have a trailing slash, there’s no way to know you meant to be in the movie directory, so redirection to relative URLs will be based on the last / in the URL. In this case the root of the site.
It’s easy enough to get around this, all you have to do is write your redirect like this:
redirect('/movie/list/')
Which provides the redirect method with an absolute path, and takes you exactly where you wanted to go, no matter where you came from.
Passing Parameters To The Controller¶
Now that you have the basic routing dispatch understood, you may be wondering how parameters are passed into the controller methods. After all, a framework would not be of much use unless it could accept data streams from the user.
TurboGears uses introspection to assign values to the arguments in your controller methods. This happens using the same duck-typing you may be familiar with if you are a frequent python programmer. Here is the basic approach:
- The dispatcher gobbles up as much of the URL as it can to find the
- correct controller method associated with your request.
- The remaining url items are then mapped to the parameters in the method.
- If there are still remaining parameters they are mapped to *args in the method signature.
- If there are named parameters, (as in a form request, or a GET request with parameters), they are mapped to the
- args which match their names, and if there are leftovers, they are placed in **kw.
Here is an example controller and a chart outlining the way urls are mapped to it’s methods:
class WikiController(TGController):
def index(self):
"""returns a list of wiki pages"""
...
def _default(self, *args):
"""returns one wikipage"""
...
def create(self, title, text, author='anonymous', **kw):
wikipage = Page(title=tile, text=text, author=author, tags=str(kw))
DBSession.add(wikipage)
def update(self, title, **kw):
wikipage = DBSession.query(Page).get(title)
for key, value in kw:
setattr(wikipage, key, value)
def delete(self, title):
wikipage = DBSession.query(Page).get(title)
DBSession.delete(wikipage)
| URL | Method | Argument Assignments |
|---|---|---|
| / | index | |
| /NewPage | _default | args : [‘NewPage’] |
| /create/NewPage?text=More Information | create | text: ‘More Information’ |
| title: ‘NewPage’ | ||
| /update/NewPage?author=Lenny | update | kw: {‘author’:’Lenny’} |
| title: ‘NewPage’ | ||
| /delete/NewPage | delete | title :’NewPage’ |
The parameters that are turned into arguments arrive in string format. It is a good idea to use Python’s type casting capabilities to change the arguments into the types the rest of your program expects. For instance, if you pass an integer ‘id’ into your function you might use id = int(id) to cast it into an int before usage. Another way to accomplish this feat is to use the @validate decorator, which is explained in Parameters Validation
Ignore Unused Parameters¶
By default TurboGears2 will complain about parameters that the controller
method was not expecting. If this is causing any issue as you need to share
between all the urls a parameter that it is used by your javascript framework
or for any other reason, you can use ignore_parameters option to have
TurboGears2 ignore them. Just add the list of parameters to ignore in
config/app_cfg.py:
base_config.ignore_parameters = ['timestamp', 'param_name']
You will still be able to access them from the tg.request object if you
need them for any reason.
Templating¶
TurboGears enables template rendering though the tg.decorators.expose decorator to
link controller methods to template and through the tg.render_template() function.
Each template is rendered using a template engine, TurboGears provides some builtin engines
but additional can be configured. The default_renderer for TurboGears applications is
Kajiki which permits to write templates in pure xhtml and validates them to detect issues
at compile time and prevent serving broken pages. For documentation on Kajiki templates see
the Kajiki Template Language.
By default TurboGears references to templates using dotted notation, this is the path of the template file in terms of python packages. This makes possible to refer to template files independently from where the application is installed and started as it refers to the python package where the template file is provided.
Typical dotted notation path looks like: mypackage.templates.template_file and it doesn’t include any extention. If an extension is provided TurboGears will try to read the path as a file system path, not as a dotted notation path.
Explicit Engine in Exposition¶
The @expose decorator template files will always be rendered using the default_renderer
specified into the application configurator unless explicitly set. To explicitly provide
the template engine to use just prepend it to the template path in the form engine:template_path
like kajiki:mypackage.templates.template_file.
Refer to Rendering Engines Configuration documentation for information on setting up available renderers and specifying the default one.
Template Variables¶
To pass template variables the controller is expected to return a dict with all the
variables inside. Those will be available inside the template with the same name.
In addition to variables explicitly specified by the user, TurboGears adds some additional variables and utilities. The most useful one are probably:
- h which is the lib.helpers module, this usually includes every utility function for formatting text and html in templates.
- request, response, tmpl_context, app_globals, config which are the same available inside controllers.
- identity which is the currently logged used when recognized
- tg.url which is the utility function to create urls in TurboGears.
For a complete list of those variables refer to the tg.render_template() documentation.
You can add additional variable to every single template by setting a variable_provider
function inside the Application Configurator (app_cfg.base_config object).
This function is expected to return a dict with any variable that should be added
the default template variables. It can even replace existing variables.
Kajiki Template Language¶
Kajiki provides a XML-based template language that is heavily inspired by Kid, and Genshi which in turn was inspired by a number of existing template languages, namely XSLT, TAL, and PHP.
This document describes the template language and will be most useful as reference to those developing Kajiki XML templates. Templates are XML files of some kind (such as XHTML) that include processing directives (elements or attributes identified by a separate namespace) that affect how the template is rendered, and template expressions that are dynamically substituted by variable data.
Template Directives¶
Directives are elements and/or attributes in the template that are identified
by the namespace py:. They can affect how the
template is rendered in a number of ways: Kajiki provides directives for
conditionals and looping, among others.
All directives can be applied as attributes, and some can also be used as
elements. The if directives for conditionals, for example, can be used in
both ways:
<html>
...
<div py:if="foo">
<p>Bar</p>
</div>
...
</html>
This is basically equivalent to the following:
<html>
...
<py:if test="foo">
<div>
<p>Bar</p>
</div>
</py:if>
...
</html>
The rationale behind the second form is that directives do not always map
naturally to elements in the template. In such cases, the py:strip
directive can be used to strip off the unwanted element, or the directive can
simply be used as an element.
Conditional Sections¶
py:if¶
The element and its content is only rendered if the expression evaluates to a truth value:
<div>
<b py:if="foo">${bar}</b>
</div>
Given the data foo=True and bar='Hello' in the template context, this
would produce:
<div>
<b>Hello</b>
</div>
But setting foo=False would result in the following output:
<div>
</div>
This directive can also be used as an element:
<div>
<py:if test="foo">
<b>${bar}</b>
</py:if>
</div>
py:switch¶
The py:switch directive, in combination with the directives py:case
and py:else provides advanced conditional processing for rendering one
of several alternatives. The first matching py:case branch is rendered, or,
if no py:case branch matches, the py:else branch is rendered.
The nested py:case directives will be tested for equality to the
parent py:switch value:
<div>
<py:switch test="1">
<span py:case="0">0</span>
<span py:case="1">1</span>
<span py:else="">2</span>
</py:switch>
</div>
This would produce the following output:
<div>
<span>1</span>
</div>
Note
The py:switch directive can only be used as a standalone tag and cannot be applied as an attribute of a tag.
Looping¶
py:for¶
The element is repeated for every item in an iterable:
<ul>
<li py:for="item in items">${item}</li>
</ul>
Given items=[1, 2, 3] in the context data, this would produce:
<ul>
<li>1</li><li>2</li><li>3</li>
</ul>
This directive can also be used as an element:
<ul>
<py:for each="item in items">
<li>${item}</li>
</py:for>
</ul>
Snippet Reuse¶
py:def¶
The py:def directive can be used to create macros, i.e. snippets of
template code that have a name and optionally some parameters, and that can be
inserted in other places:
<div>
<p py:def="greeting(name)" class="greeting">
Hello, ${name}!
</p>
${greeting('world')}
${greeting('everyone else')}
</div>
The above would be rendered to:
<div>
<p class="greeting">
Hello, world!
</p>
<p class="greeting">
Hello, everyone else!
</p>
</div>
If a macro doesn’t require parameters, it can be defined without the parenthesis. For example:
<div>
<p py:def="greeting" class="greeting">
Hello, world!
</p>
${greeting()}
</div>
The above would be rendered to:
<div>
<p class="greeting">
Hello, world!
</p>
</div>
This directive can also be used as an element:
<div>
<py:def function="greeting(name)">
<p class="greeting">Hello, ${name}!</p>
</py:def>
</div>
Variable Binding¶
py:with¶
The py:with directive lets you assign expressions to variables, which can
be used to make expressions inside the directive less verbose and more
efficient. For example, if you need use the expression author.posts more
than once, and that actually results in a database query, assigning the results
to a variable using this directive would probably help.
For example:
<div>
<span py:with="y=7; z=x+10">$x $y $z</span>
</div>
Given x=42 in the context data, this would produce:
<div>
<span>42 7 52</span>
</div>
This directive can also be used as an element:
<div>
<py:with vars="y=7; z=x+10">$x $y $z</py:with>
</div>
Structure Manipulation¶
py:attrs¶
This directive adds, modifies or removes attributes from the element:
<ul>
<li py:attrs="foo">Bar</li>
</ul>
Given foo={'class': 'collapse'} in the template context, this would
produce:
<ul>
<li class="collapse">Bar</li>
</ul>
Attributes with the value None are omitted, so given foo={'class': None}
in the context for the same template this would produce:
<ul>
<li>Bar</li>
</ul>
Note
This directive can only be used as an attribute.
py:content¶
This directive replaces any nested content with the result of evaluating the expression:
<ul>
<li py:content="bar">Hello</li>
</ul>
Given bar='Bye' in the context data, this would produce:
<ul>
<li>Bye</li>
</ul>
This directive can only be used as an attribute.
py:replace¶
This directive replaces the element itself with the result of evaluating the expression:
<div>
<span py:replace="bar">Hello</span>
</div>
Given bar='Bye' in the context data, this would produce:
<div>
Bye
</div>
This directive can also be used as an element (since version 0.5):
<div>
<py:replace value="title">Placeholder</py:replace>
</div>
py:strip¶
This directive conditionally strips the top-level element from the output. When
the value of the py:strip attribute evaluates to True, the element is
stripped from the output:
<div>
<div py:strip="True"><b>foo</b></div>
</div>
This would be rendered as:
<div>
<b>foo</b>
</div>
As a shorthand, if the value of the py:strip attribute is empty, that has
the same effect as using a truth value (i.e. the element is stripped).
Includes¶
To reuse common snippets of template code, you can include other files using py:include and py:import.
py:include¶
Includes the text of another template verbatim. The precise semantics of this tag depend on the TemplateLoader being used, as the TemplateLoader is used to parse the name of the template being included and render its contents into the current template. For instance, with the FileLoader, you might use the following:
<py:include href="path/to/base.txt"/>
whereas in the PackageLoader you would use
<py:include href="package1.package2.base"/>
py:import¶
With py:import, you can make the functions defined in another template available without expanding the full template in-place. Suppose that we saved the following template in a file lib.xml:
<py:def function="evenness(n)">
<py:if test="n%2==0">even</py:if><py:else>odd</py:else>
</py:def>
Then (using the FileLoader) we could write a template using the evenness function as follows:
<div>
<py:import hef="lib.xml" alias="lib"/>
<ul>
<li py:for="i in range(sz)">$i is ${lib.evenness(i)}</li>
</ul>
</div>
Converting Genshi Templates to Kajiki¶
Kajiki is a fast template engine which is 90% compatible with Genshi, all of Genshi directives work in Kajiki too apart those involved in templates inheritance as Kajiki uses blocks instead of XInclude and XPath.
Simple templates hierarchies (like the one coming from TurboGears quickstart)
can be moved to Kajiki blocks in a matter of seconds through the gearbox patch
command.
Note
Please note that this guide only works on version 2.3.6 and greater.
Note
It’s suggested to try this steps on a newly quickstarted Genshi application and then test them on your real apps when you are confident with the whole process.
Enabling Kajiki Templates¶
Enabling Kajiki support involves changing the base_config.default_renderer
option in your app_cfg.py and adding kajiki to the renderers:
# Add kajiki support
base_config.renderers.append('kajiki')
# Set the default renderer
base_config.default_renderer = 'kajiki'
Adapting the Master Template¶
The only template we will need to adapt by hand is our master.html
template, everything else will be done automatically. So the effort
of porting an application from Genshi to Kajiki is the same independently
from the size of the application.
First of all we will need to remove the py:strip and xmlns attributes
from the html tag:
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:py="http://genshi.edgewall.org/"
xmlns:xi="http://www.w3.org/2001/XInclude"
py:strip="">
should became:
<html>
Then let’s adapt our head tag to make it so that the content
from templates that extend our master gets included inside it:
<head py:match="head" py:attrs="select('@*')">
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<meta charset="${response.charset}" />
<title py:if="False">Your generic title goes here</title>
<meta py:replace="select('*')"/>
<link rel="stylesheet" type="text/css" media="screen" href="${tg.url('/css/bootstrap.min.css')}" />
<link rel="stylesheet" type="text/css" media="screen" href="${tg.url('/css/style.css')}" />
</head>
should became:
<head>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<meta charset="${response.charset}" />
<title py:if="False">Your generic title goes here</title>
<py:block name="head"/>
<link rel="stylesheet" type="text/css" media="screen" href="${tg.url('/css/bootstrap.min.css')}" />
<link rel="stylesheet" type="text/css" media="screen" href="${tg.url('/css/style.css')}" />
</head>
Then we do the same with the body tag by disabling it as a block and
placing a block with the same name inside of it:
<body py:match="body" py:attrs="select('@*')">
<!-- Navbar -->
[...]
<div class="container">
<!-- Flash messages -->
<py:with vars="flash=tg.flash_obj.render('flash', use_js=False)">
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div py:if="flash" py:replace="Markup(flash)" />
</div>
</div>
</py:with>
<!-- Main included content -->
<div py:replace="select('*|text()')"/>
</div>
</body>
Which should became:
<body>
<!-- Navbar -->
[...]
<div class="container">
<!-- Flash messages -->
<py:with vars="flash=tg.flash_obj.render('flash', use_js=False)">
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div py:if="flash" py:replace="Markup(flash)" />
</div>
</div>
</py:with>
<!-- Main included content -->
<py:block name="body"/>
</div>
</body>
What is did is replacing all the XPath expressions that lead to
insert content from the child templates into head and body
with two head and body blocks. So our child templates will be
able to rely on those blocks to inject their content into the master.
Last importat step is renaming the master template, as Kajiki
in turbogears uses .xhtml extension we will need to rename
master.html to master.xhtml:
$ find ./ -iname 'master.html' -exec sh -c 'mv {} `dirname {}`/master.xhtml' \;
Note
The previous expression will rename the master file if run from within your project directory.
Upgrading Child Templates¶
There are three things we need to do to upgrade all our child templates to Kajiki:
- Replace
xi:includewithpy:extends- Strip
<html>tags to avoid a duplicated root tag- Replace
<head>tag with a kajiki block- Replace
<body>tag with a kajiki block
To perform those changes we can rely on a simple but helpful gearbox command
to patch all our templates by replacing xi:include with py:extends which is used
and recognized by Kajiki.
Just move inside the root of your project and run:
$ gearbox patch -R '*.html' 'xi:include href="master.html"' -r 'py:extends href="master.xhtml"'
You should get an output similar to:
6 files matching
! Patching /private/tmp/prova/prova/templates/about.html
! Patching /private/tmp/prova/prova/templates/data.html
! Patching /private/tmp/prova/prova/templates/environ.html
! Patching /private/tmp/prova/prova/templates/error.html
! Patching /private/tmp/prova/prova/templates/index.html
! Patching /private/tmp/prova/prova/templates/login.html
Which means that all our templates apart from master.html got patched
properly and now correctly use py:extends.
Now we can start adapting our tags to move them to kajiki blocks.
First of all we will need to strip the html from all the templates apart
master.xhtml to avoid ending up with duplicated root tag:
$ gearbox patch -R '*.html' 'xmlns="http://www.w3.org/1999/xhtml"' -r 'py:strip=""'
Then we will need to do the same for the head tag:
$ gearbox patch -R '*.html' '<head>' -r '<head py:block="head" py:strip="True">'
Then repeat for body:
$ gearbox patch -R '*.html' '<body>' -r '<body py:block="body" py:strip="True">'
Now that all our templates got upgraded from Genshi to Kajiki, we must remember to rename them all, like we did for master:
$ find ./ -iname '*.html' -exec sh -c 'mv {} `dirname {}`/`basename {} .html`.xhtml' \;
Restarting your application now should lead to a properly working page equal to the original Genshi one.
Congratulations, you successfully moved your templates from Genshi to Kajiki.
Scaffolding¶
Scaffolding is the process of creating a new component of your web application through a template or preset.
Creating new Controllers, Models and Templates¶
On TurboGears this is provided by the gearbox scaffold
command which will look for .template files inside your
project directory to create new scaffolds for your web app.
Since version 2.3.5 projects quickstarted by TurboGears provide
.template files for model, controller and template so
new models, controllers and templates can be easily created
in your project by running gearbox scaffold.
For example to create a new Photo model simply run:
$ gearbox scaffold model photo
It will create a model/photo.py file with a Photo
class inside which you just need to import inside model/__init__.py
to make it available inside your app.
Creating All Together¶
Multiple scaffolds can be created together, so in case you need to create a new model with a controller to handle it and an index page you can run:
$ gearbox scaffold model controller template photo
Which will create a new controller with the associated page
and model. To start using the controller mount it inside
your application RootController.
Creating Packages¶
There are cases when your controllers, templates and model might become complex and be better managed by a python package instead of a simple module. This is common for templates which will usually be more than one for each controller and so are usually grouped in a package for each controller.
To create scaffold in a package just provide the
-s [PACKAGE] option to the scaffold command:
$ gearbox scaffold -s photo controller template photo
This will create a photo controller and template inside a photo package where multiple templates can be placed.
Parameters Validation¶
When using TurboGears, your controller methods get their arguments
built from the various GET, POST, and URL mechanisms provided by
TurboGears. The only downside is that all the arguments will be
strings and you’d like them converted to their normal Python datatype:
numbers to int, dates to datetime, etc.
This conversion functionality is provided by the FormEncode package
and is applied to your methods using the validate
decorator. FormEncode provides both validation and conversion as a
single step, reasoning that you frequently need to validate something
before you can convert it or that you’ll need to convert something
before you can really validate it.
The @validate() decorator can evaluate both widget-based forms and
the standard form arguments so they are not dependent on widgets at
all.
Furthermore, the @validate() decorator is not really required at
all. It just provides a convenience so that you can assume that you
have the right kind of data inside your controller methods. This helps
separate validation logic from application logic about what to do with
valid data.
If you don’t put a @validate() decorator on your method, you’ll
simply have to do the string conversion in your controller.
Validating Parameters¶
When not using forms, the story gets a bit more complex. Basically,
you need to specify which validator goes with which argument using in
the validate decorator. Here’s a simple example:
from tg import request, validate, expose, TGController
from formencode import validators
class RootController(TGController):
@expose('json')
@validate({"a":validators.Int(not_empty=True), "b":validators.Email})
def two_validators(self, a=None, b=None, *args):
validation_status = tg.request.validation
errors = [{key, value} in validation_status['errors'].iteritems()]
values = validation_status['values']
return dict(a=a, b=b, errors=str(errors), values=str(values))
The dictionary passed to validators maps the incoming field names to
the appropriate FormEncode validators, Int in this example.
In case of a validation error TurboGears will provide the errors
and values inside tg.request.validation.
FormEncode provides a number of useful pre-made validators for you to
use: they are available in the formencode.validators module.
For most validators, you can pass keyword arguments for more specific constraints.
Validation Process Information¶
TurboGears provides some information on the currently running validation process while it is handling the validation error.
Whenever an error handling is in process some properties are available in
the tg.request.validation to provide overview of the validation error:
tg.request.validation['values']The submitted values before validationtg.request.validation['errors']The errors that triggered the error handlingtg.request.validation['exception']The validation exception that triggered the error handlingtg.request.validation['error_handler']The error handler that is being executed
The Error Handler¶
In many cases you don’t need the granularity provided by tg.request.validation
and probably in case of an error you just want to send the user somewhere else
(maybe to reinsert the data he provided).
This can be achieved by using the error_handler argument of validate.
The provided function or controller method will be called to generate
a response for the user in case of an error instead of continuing with the current
action:
from tg import request, validate, expose, TGController
from formencode import validators
class RootController(TGController):
@expose()
def onerror(self, **kwargs):
return 'An error occurred: %s' % request.validation['errors']
@expose()
@validate({"a": validators.Int(not_empty=True), "b": validators.Email},
error_handler=onerror)
def two_validators(self, a=None, b=None, *args):
return 'Values: %s, %s, %s' % (a, b, args)
Heading to /two_validators without providing a value for a will lead
to an "An error occurred" message as the onerror method is executed
instead of continuing with two_validators.
Note
The method in question will be called, with the unvalidated data as
its parameters, so it’s usually best to accept **kwargs.
And error validation messages will be stored in tg.request.validation.
Validating Forms¶
For manually written forms you can use @validate on the action that
processes the submitted data and add the errors in your template from
tg.request.validation.
TurboGears also provides a more convenient way to create forms, validate submitted data and display error messages, those can be managed through Widgets & Forms which work together with validation by Validating Fields
Any widget based form can then be passed to the @validate which
will automatically validate the submitted data against that form.
Validators¶
TurboGears applications will usually rely on three kind of validators:
Convertwhich is builtin into TurboGears an can be used for simple conversions like integers, floats and so on…tw2.core.validationwhich provide ToscaWidgets validators for Formsformencode.validatorsvalidators which can be used Standalone or with a Form
While in many cases Convert will suffice, the FormEncode library provides a pretty
complete set of validators:
- Attribute
- Bool
- CIDR
- ConfirmType
- Constant
- CreditCardExpires
- CreditCardSecurityCode
- CreditCardValidator
- DateConverter
- DateTime
- DateValidator
- DictConverter
- Empty
- False
- FancyValidator
- FieldStorageUploadConverter
- FieldsMatch
- FileUploadKeeper
- FormValidator
- IDeclarative
- IPhoneNumberValidator
- ISchema
- IValidator
- Identity
- IndexListConverter
- Int
- Interface
- Invalid
- MACAddress
- MaxLength
- MinLength
- NoDefault
- NotEmpty
- Number
- OneOf
- PhoneNumber
- PlainText
- PostalCode
- Regex
- RequireIfMissing
- RequireIfPresent
- Set
- SignedString
- StateProvince
- String
- StringBool
- StringBoolean
- StripField
- TimeConverter
- True
- URL
- UnicodeString
- Validator
- Wrapper
For the absolute most up-to date list of available validators, check the FormEncode validators module. You can also create your own validators or build on existing validators by inheriting from one of the defaults.
See the FormEncode documentation for how this is done.
You can also compose compound validators with logical operations,
the FormEncode compound module provides All (all must pass),
Any (any one must pass) and Pipe (all must pass with the results of
each validator passed to the next item in the Pipe). You can use these
like so:
from formencode.compound import All
...
the_validator=All(
validators.NotEmpty(),
validators.UnicodeString(),
)
Writing Custom Validators¶
If you can’t or don’t want to rely on the FormEncode library you can write your own validators.
Validators are simply objects that provide a to_python method
which returns the converted value or raise tg.validation.TGValidationError
For example a validator that converts a paramter to an integer would look like:
from tg.validation import TGValidationError
class IntValidator(object):
def to_python(self, value, state=None):
try:
return int(value)
except:
raise TGValidationError('Integer expected')
Then it is possible to pass an instance of IntValidator to the TurboGears @validate
decorator.
Schema Validators¶
Sometimes you need more power and flexibility than you can get from validating individual form fields. Fortunately FormEncode provides just the thing for us – Schema validators.
If you want to do multiple-field validation, reuse validators or just
clean up your code, validation Schema``s are the way to go. You
create a validation schema by inheriting from
:class:`formencode.schema.Schema` and pass the newly created ``Schema
as the validators argument instead of passing a dictionary.
Create a schema:
class PwdSchema(schema.Schema):
pwd1 = validators.String(not_empty=True)
pwd2 = validators.String(not_empty=True)
chained_validators = [validators.FieldsMatch('pwd1', 'pwd2')]
Then you can use that schema in @validate rather than a dictionary of validators:
@expose()
@validate(validators=PwdSchema())
def password(self, pwd1, pwd2):
if tg.request.validation['errors']:
return "There was an error"
else:
return "Password ok!"
Besides noticing our brilliant security strategy, please notice the
chained_validators part of the schema that guarantees a pair of
matching fields.
Again, for information about Invalid exception objects, creating
your own validators, schema and FormEncode in general, refer to the
FormEncode Validator documentation and don’t be afraid to check the
Formencode.validators source. It’s often clearer than the
documentation.
Note that Schema validation is rigorous by default, in particular, you must declare every field you are going to pass into your controller or you will get validation errors. To avoid this, add:
class MySchema( schema.Schema ):
allow_extra_fields=True
to your schema declaration.
Flash & Alert Messages¶
TurboGears provides a way to display short messages inside the current or next page. This works by using the WebFlash module which stores short text messages inside a cookie so that it can be retrieved when needed.
Default Setup¶
By Default the master.html of a quickstarted project provides a div where flash messages will be displayed, this is achieved with the following lines of code:
<py:with vars="flash=tg.flash_obj.render('flash', use_js=False)">
<div py:if="flash" py:replace="Markup(flash)" />
</py:with>
The tg.flash_obj is the WebFlash object which is available inside
any rendered template. This object permits to retrieve the current
flash message and display it.
Storing Flash Messages¶
Flash messages can be stored using the tg.flash command
this allows to store a message with a status option to configure
the flash style.
tg.flash('Message', 'status')
If the method that called flash performs a redirect the flash will be visible inside the redirected page. If the method directly exposes a template the flash will be visible inside the template itself.
Caching with Flash Messages¶
When using tg_cache variable in rendered templates (Prerendered Templates Caches)
the flash will get into the cached template causing unwanted messages to be displayed.
To solve this issue the tg.flash_obj.render method provides the use_js option.
By default this option is set at False inside the template, changing it to True
will make the flash message to be rendered using javascript. This makes so that the same
template is always rendered with a javascript to fetch the flash message and display it
due to the fact that the template won’t change anymore it will now be possible to
correctly cache it.
Customizing Flash¶
CSS Styling¶
By default warning, error, info, ok statuses
provide a style in public/css/style.css for quickstarted applications.
Any number of statuses can be configured using plain css:
#flash > .warning {
color: #c09853;
background-color: #fcf8e3;
border-color: #fbeed5;
}
#flash > .ok {
color: #468847;
background-color: #dff0d8;
border-color: #d6e9c6;
}
#flash > .error {
color: #b94a48;
background-color: #f2dede;
border-color: #eed3d7;
}
#flash > .info {
color: #3a87ad;
background-color: #d9edf7;
border-color: #bce8f1;
}
Flash Options¶
Flash messages support can be styled using options inside the flash. namespace,
those are documented in TGFlash and can be specified in config/app_cfg.py
or in your .AppConfig instance.
For example to change the default message status (when status is omitted) you can use the
flash.default_status option and set it to any string. To change the default flash template
you can use flash.template and set it to a string with the HTML that should be displayed
to show the flash (note that flash.template only works for static rendered flash, not for
JS version).
Custom Flash HTML¶
For example to render the flash using the toastr library you might want to remove the
py:with code block from your master.html and move it to the bottom of your <body>
right after the usage of bootstrap and jquery libraries:
<body>
<!-- YOUR CURRENT BODY CONTENT -->
<script src="http://code.jquery.com/jquery.js"></script>
<script src="${tg.url('/javascript/bootstrap.min.js')}"></script>
<py:with vars="flash=tg.flash_obj.render('flash')">
<py:if test="flash">${Markup(flash)}</py:if>
</py:with>
</body>
This will ensure that we can provide custom Javascript that depends on JQuery inside our
flash template.
Now we can switch flash template to use the toastr library to display our flash by setting
inside your app_cfg.py:
base_config['flash.default_status'] = 'success'
base_config['flash.template'] = '''\
<script src="//cdnjs.cloudflare.com/ajax/libs/toastr.js/latest/js/toastr.min.js"></script>
<script>toastr.$status("$message");</script>
'''
This will ensure that each time the flash is displayed the toastr library with the given status is used.
Last, to correctly display the flash with the right look and feel, don’t forget to add the
toastr CSS to the head of your master.html:
<link rel="stylesheet" type="text/css" media="screen"
href="//cdnjs.cloudflare.com/ajax/libs/toastr.js/latest/css/toastr.min.css" />
If everything is correct you should see your flash messages as baloon into the top-right corner of your webpage.
Custom Flash JavaScript¶
Javascript based flashes are usually common when Caching is involved, so the cached version
of the webpage will not have the flash inside but you still want to be able to display
the flash messages. In this case instead of providing a custom flash.template you
should provide a custom flash.js_call which is the javascript code used to display the
message.
For example to use the toastr library you might want to ensure toastr CSS and JS are available
and add the following to your app_cfg.py:
base_config['flash.default_status'] = 'success'
base_config['flash.js_call'] = '''\
var payload = webflash.payload();
if(payload) { toastr[payload.status](payload.message); }
'''
The webflash object is provided by TGFlash itself and the webflash.payload()
method will fetch the current message for you.
Widgets & Forms¶
TurboGears relies on ToscaWidgets2 for Widgets and Forms definition. Widgets are small reusable HTML components with attached values, and a form is a collection of one or more widgets (one for each field) that display the fields.
ToscaWidgets2 Widgets provide:
- Templating support to generate the HTML
- Support for widget configuration options
- Resources declaration and injection (for css & JS required by the widget)
Widgets¶
Creating Widgets¶
Widgets can be created by subclassing tw2.core.Widget and defining
a template argument for it. By default the template argument has to be
the dotted format path of a template file (same syntax you provide to @expose),
but an inline_engine_name option can be provided to tell ToscaWidgets2
that the template argument is actually the template itself.
import tw2.core as twc
class UserAvatarWidget(twc.Widget):
inline_engine_name = 'kajiki'
template = '<div class="username">${w.name}</div>'
To display the widget just call the .display(**kwargs) method of the
widget passing any parameter you want to provide to the widget:
>>> UserAvatarWidget.display(name='John Doe')
Markup(u'<div class="username">John Doe</div>')
The passed name is available inside the template as w.name.
All the arguments passed to the tw2.core.Widget.display() function will be available
as properties of the widget instance inside the template itself.
Widgets Parameters¶
While our previous example worked as expected, you probably noticed that when
the name argument to display is omitted it will lead to a crash and we
didn’t provide any detail to developers that they must provide it.
to solve this ToscaWidgets provides parameters support through the tw2.core.Param
class.
To make the name parameter explicit and provide a default value for it we
can add it to the widget as a Param:
import tw2.core as twc
class UserAvatarWidget(twc.Widget):
name = twc.Param(default='Unknown User', description='Name of the logged user')
inline_engine_name = 'kajiki'
template = '<div class="username">${w.name}</div>'
Trying to display the widget without a name argument will now just provide the
default value instead of leading to a crash:
>>> UserAvatarWidget.display()
Markup(u'<div class="username">Unknown User</div>')
>>> UserAvatarWidget.display(name='John Doe')
Markup(u'<div class="username">John Doe</div>')
The passed value will be available inside the template as properties
of the widget instance and the widget instance will be available as w.
So in the previous example the user name was available as w.name.
Note
Whenever you need to know which options a widget provides you want to have a look at its parameters. They will usually provide a short description of the parameter purpose.
Widgets Resources¶
To implement more advanced widgets you will probably need to add styling and javascript to them. This can easily be done through resources support provided by ToscaWidgets.
A resource is an instance of tw2.core.JSLink, tw2.core.JSSource,
tw2.core.CSSLink and tw2.core.CSSSource, which allow to provide
access to a CSS and Javascript file or to inline code for the widget.
Resources are injected for you in the <head> tag by a middleware that
handles resources injection for widgets. Each resource has an id attribute
so the same resource won’t be injected twice as far as all instances of the
resources share the same id.
The following example adds an inline CSS to make user avatars bold and provides jQuery as a dependency to add a click function that shows a dialog with the username inside when clicked:
import tw2.core as twc
class UserAvatarWidget(twc.Widget):
name = twc.Param(default='Unknown User', description='Name of the logged user')
inline_engine_name = 'kajiki'
template = '''
<div class="useravatar" id="${w.id}">
<div class="username">${w.name}</div>
<script>$('#${w.id}').click(function() { alert('${w.name}') })</script>
</div>
'''
resources = [
twc.CSSSource(id='useravatarstyle',
src='.useravatar .username { font-weight: bold; }'),
twc.JSLink(id='jquery',
link='https://ajax.googleapis.com/ajax/libs/jquery/3.0.0/jquery.min.js')
]
Calling UserAvatarWidget.display will generate the short html snippet:
<div class="useravatar" id="id_a9a9fbba90744ff2a894f5ea5ae99f44">
<div class="username">John Doe</div>
<script>$('#id_a9a9fbba90744ff2a894f5ea5ae99f44').click(function() { alert('John Doe') })</script>
</div>
but will also inject the required resources into the head tag:
<head>
<style type="text/css">.useravatar .username { font-weight: bold; }</style>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.0.0/jquery.min.js" type="text/javascript" id="jquery"></script>
<meta content="width=device-width, initial-scale=1.0" name="viewport">
...
</head>
Note that display the widgets twice on the page won’t inject the resources twice. That’s
because ToscaWidgets will recognize that useravatarstyle and jquery resources already
got injected and won’t insert them again.
Widgets also provide an .id attribute automatically generated by ToscaWidgets2
(it can be overwritten at display time), this allows to uniquely identify
each widget instance from any javascript injected into the page.
In the previous example we leveraged this feature to point jQuery to each specific
widget instance through $('#${w.id}'). If you display two UserAvatarWidget
with different names on the same page, you will notice that clicking each one of them
will properly show the right name thanks to this.
Forms¶
Displaying Forms¶
Forms are actually a particular kind of Widget, actually a particular kind of
tw2.core.CompoundWidget as they can contain more widgets inside themselves.
A Forms is actually a widget that provides a template to displays through a
Layout all the other widgets that are provided inside the form.
To create a form you will have to declare it specifying:
- the form action (where to submit the form data)
- the form layout (how the form will be displayed)
- the form fields
The action can be specified as an attribute of the form itself, while the layout
must be a class named child which has to inherit from tw2.forms.BaseLayout.
Any of tw2.forms.TableLayout or tw2.forms.ListLayout will usually do, but you
can easily write your own custom layouts. The form fields can then be specified
inside the child class.
import tw2.core as twc
import tw2.forms as twf
class MovieForm(twf.Form):
class child(twf.TableLayout):
title = twf.TextField()
director = twf.TextField(value='Default Director')
genres = twf.CheckBoxList(options=['Action', 'Comedy', 'Romance', 'Sci-fi'])
action = '/save_movie'
To display the form we can return it from the controller where it must be rendered:
@expose('tw2test.templates.index')
def index(self, *args, **kw):
return dict(page='index', form=MovieForm)
and display it inside the template itself.
Any field of the form can be filled using the value argument passed to the
display function. The values provided inside this argument will override the
field default ones.
<div id="getting_started">
${form.display(value=dict(title='default title'))}
</div>
When submitting the form the save_movie controller declared in the action attribute of the form will receive the submitted values as any other provided GET or POST parameter.
@expose()
def save_movie(self, **kw):
return str(kw)
Validating Fields¶
ToscaWidgets2 is able to use any FormEncode validator for validation of
both fields and forms. More validators are also provided inside the
tw2.core.validators module.
To start using validation we have to declare the validator for each form field.
For example to block submission of our previous form when no title or director
is provided we can use the tw2.core.Required validator:
class MovieForm(twf.Form):
class child(twf.TableLayout):
title = twf.TextField(validator=twc.Required)
director = twf.TextField(value="Default Director", validator=twc.Required)
genres = twf.CheckBoxList(options=['Action', 'Comedy', 'Romance', 'Sci-fi'])
action = '/save_movie'
Now the forms knows how to validate the title and director fields,
but those are not validated in any way.
To enable validation in TurboGears we must use the validate decorator
and place it at our form action:
@expose()
@validate(MovieForm, error_handler=index)
def save_movie(self, *args, **kw):
return str(kw)
Now every submission to /save_movie url will be validated against the MovieForm and if it doesn’t pass validation will be redirected to the index method where the form will display an error for each field not passing validation.
Note
TurboGears keeps track of the form that failed validation
when running the error_handler, so if we display that
form during the error_handler it will automatically
display the validation error messages, nothing particular
is required to show errors apart displaying the form after
validation failed.
More about TurboGears support for validation is available inside the Parameters Validation page.
Validating Compound Fields¶
Suppose that you are afraid that people might enter a wrong director name for your movies. The most simple solution would be to require them to enter the name two times to be sure that it is actually the correct one.
How can we enforce people to enter two times the same name inside our form?
Apart from fields, ToscaWidgets permits to set validators to forms.
Those can be used to validate form fields together instead of one by one.
To check that our two directors equals we will use the
formencode.validators.FieldsMatch validator:
import tw2.core as twc
import tw2.forms as twf
from formencode.validators import FieldsMatch
class MovieForm(twf.Form):
class child(twf.TableLayout):
title = twf.TextField(validator=twc.Required)
director = twf.TextField(value="Default Director", validator=twc.Required)
director_verify = twf.TextField()
genres = twf.CheckBoxList(options=['Action', 'Comedy', 'Romance', 'Sci-fi'])
action = '/save_movie'
validator = FieldsMatch('director', 'director_verify')
Nothing else of our code needs to be changed, our /save_movie controller already has validation for the MovieForm and when the form is submitted after checking that there is a title and director will also check that both director and director_verify fields equals.
Manual Validation¶
Usually you will rely on the @validate decorator to check for form errors
on submission and display the form with proper error messages, but there might
be cases where you want to manually validate the form and then pass the errors
to it.
To validate a Form just call the tw2.forms.Form.validate() method on it
with the dictionary of values to validate:
MovieForm.validate({})
That will raise a tw2.core.ValidationError in case the validation failed,
the validation error itself will contain an instance of the widget already configured
to display the error messages:
try:
MovieForm.validate(dict())
except twc.ValidationError as e:
# Display widget with error messages inside.
e.widget.display()
Relocatable Widget Actions¶
Whenever you run your application on a mount point which is not the root of the domain name your actions will have to poin to the right path inside the mount point.
In TurboGears2 this is usually achieved using the tg.url function which
checks the SCRIPT_NAME inside the request environment to see where
the application is mounted. The issue with widget actions is that widgets
actions are globally declared and tg.url cannot be called outside of
a request.
Calling tg.url while declaring a form and its action will cause a crash
to avoid this TurboGears provides a lazy version of the url method which
is evaluated only when the widget is displayed (tg.lurl):
from tg import lurl
class MovieForm(twf.Form):
class child(twf.TableLayout):
title = twf.TextField(validator=twc.Required)
director = twf.TextField(value="Default Director", validator=twc.Required)
genres = twf.CheckBoxList(options=['Action', 'Comedy', 'Romance', 'Sci-fi'])
action = lurl('/save_movie')
Using tg.lurl the form action will be correctly written depending on
where the application is mounted.
Please pay attention that usually when registering resources on ToscaWidgets (both
tw1 and tw2) it won’t be necessary to call neither tg.url or tg.lurl as
all the Link subclasses like JSLink, CSSLink and so on will already
serve the resource using the application mount point.
Custom Layouts¶
While using tw2.forms.TableLayout and tw2.forms.ListLayout it’s easy to perform
most simple styling and customization of your forms, for more complex widgets
a custom template is usually the way to go.
You can easily provide your custom layout by subclassing tw2.forms.widgets.BaseLayout
and declaring a template for it inside your forms.
For example it is possible to create a name/surname form with a side field for notes using the bootstrap CSS framework:
from tw2.core import Validator
from tw2.forms.widgets import Form, BaseLayout, TextField, TextArea, SubmitButton
class SubscribeForm(Form):
action = '/submit'
class child(BaseLayout):
inline_engine_name = 'kajiki'
template = '''
<div py:strip="">
<py:for each="c in w.children_hidden">
${c.display()}
</py:for>
<div class="form form-horizontal">
<div class="form-group">
<div class="col-md-7">
<div py:with="c=w.children.name"
class="form-group ${c.error_msg and 'has-error' or ''}">
<label for="${c.compound_id}" class="col-md-3 control-label">${c.label}</label>
<div class="col-md-9">
${c.display()}
<span class="help-block" py:content="c.error_msg"/>
</div>
</div>
<div py:with="c=w.children.surname"
class="form-group ${c.error_msg and 'has-error' or ''}">
<label for="${c.compound_id}" class="col-md-3 control-label">${c.label}</label>
<div class="col-md-9">
${c.display()}
<span class="help-block" py:content="c.error_msg"/>
</div>
</div>
</div>
<div class="col-md-4 col-md-offset-1">
${w.children.notes.display()}
</div>
</div>
</div>
</div>
'''
name = TextField(label=l_('Name'), validator=Validator(required=True),
css_class="form-control")
surname = TextField(label=l_('Surname'), validator=Validator(required=True),
css_class="form-control")
notes = TextArea(label=None, placeholder=l_("Notes"),
css_class="form-control", rows=8)
submit = SubmitButton(css_class='btn btn-primary', value=l_('Create'))
Access Control and Authorization¶
This document describes how authentication is integrated into TurboGears and how you may get started with it.
How authentication and authorization is set up by default¶
If you enabled authentication and authorization in your project when it was generated by TurboGears, then it’s been set up to store your users, groups and permissions in SQLAlchemy-managed tables or Ming collections.
Your users’ table is used by repoze.who to authenticate them.
When the authentication has success TurboGears uses the TGAuthMetadata
instance declared in config.base_config.sa_auth.authmetadata to
retrieve the informations about your user which are required for authorization.
You are free to change the authmetadata object as you wish, usually if your authentication model changes, your authmetadata object will change accordingly.
You can even get rid of authorization based on groups and permissions and use
other authorization patterns (e.g., roles, based on network components) or
simply use a mix of patterns. To do this you can set authmetadata to None
and register your own metadata providers for repoze.who.
Restricting access with tg.predicates¶
tg.predicates allows you to define access rules based on so-called
“predicate checkers”. It is a customized version of the repoze.what
module which has been merged into TurboGears itself to make easier
to support different authentication backends.
A predicate is the condition that must be met for the user to be able to
access the requested source. Such a predicate, or condition, may be made
up of more predicates – those are called compound predicates. Action
controllers, or controllers, may have only one predicate, be it single or
compound.
A predicate checker is a class that checks whether a predicate or
condition is met.
If a user is not logged in, or does not have the proper permissions, the
predicate checker throws a 401 (HTTP Unauthorized) which is caught by the
repoze.who middleware to display the login page allowing
the user to login, and redirecting the user back to the proper page when they
are done.
For example, if you have a predicate which is “grant access to any authenticated user”, then you can use the following built-in predicate checker:
from tg.predicates import not_anonymous
p = not_anonymous(msg='Only logged in users can read this post')
Or if you have a predicate which is “allow access to root or anyone with the ‘manage’ permission”, then you may use the following built-in predicate checker:
from tg.predicates import Any, is_user, has_permission
p = Any(is_user('root'), has_permission('manage'),
msg='Only administrators can remove blog posts')
As you may have noticed, predicates receive the msg keyword argument to
use its value as the error message if the predicate is not met. It’s optional
and if you don’t define it, the built-in predicates will use the default
English message; you may take advantage of this functionality to make such
messages translatable.
Note
Good predicate messages don’t explain what went wrong; instead, they describe the predicate in the current context (regardless of whether the condition is met or not!). This is because such messages may be used in places other than in a user-visible message (e.g., in the log file).
- Really bad: “Please login to access this area”.
- Bad: “You cannot delete an user account because you are not an administrator”.
- OK: “You have to be an administrator to delete user accounts”.
- Perfect: “Only administrators can delete user accounts”.
Below are described the convenient utilities TurboGears provides to deal with predicates in your applications.
Action-level authorization¶
You can control access on a per action basis by using the
tg.decorators.require() decorator on the actions in question. All you have
to do is pass the predicate to that decorator. For example:
# ...
from tg import require
from tg.predicates import Any, is_user, has_permission
# ...
class MyCoolController(BaseController):
# ...
@expose('yourproject.templates.start_vacations')
@require(Any(is_user('root'), has_permission('manage'),
msg='Only administrators can remove blog posts'))
def only_for_admins():
flash('Hello admin!')
dict()
# ...
Controller-level authorization¶
If you want that all the actions from a given controller meet a common
authorization criteria, then you may define the allow_only attribute of
your controller class:
from yourproject.lib.base import BaseController
class Admin(BaseController):
allow_only = predicates.has_permission('manage')
@expose('yourproject.templates.index')
def index(self):
flash(_("Secure controller here"))
return dict(page='index')
@expose('yourproject.templates.index')
def some_where(self):
"""This is protected too.
Only those with "manage" permissions may access.
"""
return dict()
If you need to specify additional options for the requirement you can
assign a require instance instead of a plain predicate to the
allow_only attribute:
from yourproject.lib.base import BaseController
from tg import require
class AdminAPI(BaseController):
allow_only = require(predicates.has_permission('manage'),
smart_denial=True)
@expose('json')
def create_user(self, user_name):
pass
Warning
Do not use this feature if the login URL would be mapped to that controller, as that would result in a cyclic redirect.
Built-in predicate checkers¶
These are the predicate checkers that are included with tg.predicates,
although the list below may not always be up-to-date:
Single predicate checkers¶
-
class
tg.predicates.not_anonymous Check that the current user has been authenticated.
-
class
tg.predicates.is_user(user_name) Check that the authenticated user’s user name is the specified one.
Parameters: user_name (str) – The required user name.
-
class
tg.predicates.in_group(group_name) Check that the user belongs to the specified group.
Parameters: group_name (str) – The name of the group to which the user must belong.
-
class
tg.predicates.in_all_groups(group1_name, group2_name[, group3_name ...]) Check that the user belongs to all of the specified groups.
Parameters: - group1_name – The name of the first group the user must belong to.
- group2_name – The name of the second group the user must belong to.
- .. (group3_name) – The name of the other groups the user must belong to.
-
class
tg.predicates.in_any_group(group1_name[, group2_name ...]) Check that the user belongs to at least one of the specified groups.
Parameters: - group1_name – The name of the one of the groups the user may belong to.
- .. (group2_name) – The name of other groups the user may belong to.
-
class
tg.predicates.has_permission(permission_name) Check that the current user has the specified permission.
Parameters: permission_name – The name of the permission that must be granted to the user.
-
class
tg.predicates.has_all_permissions(permission1_name, permission2_name[, permission3_name...]) Check that the current user has been granted all of the specified permissions.
Parameters: - permission1_name – The name of the first permission that must be granted to the user.
- permission2_name – The name of the second permission that must be granted to the user.
- .. (permission3_name) – The name of the other permissions that must be granted to the user.
-
class
tg.predicates.has_any_permission(permission1_name[, permission2_name ...]) Check that the user has at least one of the specified permissions.
Parameters: - permission1_name – The name of one of the permissions that may be granted to the user.
- .. (permission2_name) – The name of the other permissions that may be granted to the user.
-
class
tg.predicates.Not(predicate) Negate the specified predicate.
Parameters: predicate – The predicate to be negated.
Custom single predicate checkers¶
You may create your own predicate checkers if the built-in ones are not enough to achieve a given task.
To do so, you should extend the tg.predicates.Predicate
class. For example, if your predicate is “Check that the current month is the
specified one”, your predicate checker may look like this:
from datetime import date
from tg.predicates import Predicate
class is_month(Predicate):
message = 'The current month must be %(right_month)s'
def __init__(self, right_month, **kwargs):
self.right_month = right_month
super(is_month, self).__init__(**kwargs)
def evaluate(self, environ, credentials):
if date.today().month != self.right_month:
self.unmet()
Warning
When you create a predicate, don’t try to guess/assume the context in which the predicate is evaluated when you write the predicate message because such a predicate may be used in a different context.
- Bad: “The software can be released if it’s %(right_month)s”.
- Good: “The current month must be %(right_month)s”.
If you defined that class in, say, {yourproject}.lib.auth, you may use it
as in this example:
# ...
from spain_travels.lib.auth import is_month
# ...
class SummerVacations(BaseController):
# ...
@expose('spain_travels.templates.start_vacations')
@require(is_month(7))
def start_vacations():
flash('Have fun!')
dict()
# ...
Built-in compound predicate checkers¶
You may create a compound predicate by aggregating single (or even compound) predicate checkers with the functions below:
-
class
tg.predicates.All(predicate1, predicate2[, predicate3 ...]) Check that all of the specified predicates are met.
Parameters: - predicate1 – The first predicate that must be met.
- predicate2 – The second predicate that must be met.
- .. (predicate3) – The other predicates that must be met.
-
class
tg.predicates.Any(predicate1[, predicate2 ...]) Check that at least one of the specified predicates is met.
Parameters: - predicate1 – One of the predicates that may be met.
- .. (predicate2) – Other predicates that may be met.
But you can also nest compound predicates:
# ...
from yourproject.lib.auth import is_month
# ...
@authorize.require(authorize.All(
Any(is_month(4), is_month(10)),
predicates.has_permission('release')
))
def release_ubuntu(self, **kwargs):
return dict()
# ...
Which translates as “Anyone granted the ‘release’ permission may release a version of Ubuntu, if and only if it’s April or October”.
Pagination in TurboGears¶
Paginate Decorator¶
TurboGears provides a convenient paginate() decorator that you can
combine with expose(). To use it, you simply have to pass it the
name of a collection to paginate. In controller/root.py:
from tg.decorators import paginate
@expose("paginatesample.templates.movie_list_deco")
@paginate("movies", items_per_page=5)
def decolist(self):
"""
List and paginate all movies in the database using the
paginate() decorator.
"""
movies = DBSession.query(Movie)
return dict(movies=movies, page='paginatesample Movie list')
In your template, you can now use the collection direction since it
will be trimed to only contain the current page. You will also have
have a basic page navigation with ${tmpl_context.paginators.movies.pager()}:
<ol>
<li py:for="movie in movies" py:content="movie">Movie title and year</li>
</ol>
<p class="pagelist">
${tmpl_context.paginators.movies.pager()}
</p>
Advanced Pagination¶
The pager method of the paginator supports various customization options to tune
the look and feel of the paginator, make sure you take a look at
tg.support.paginate.Page for more details:
${tmpl_context.paginators.movies.pager(format='~3~', page_param='page', show_if_single_page=True)}
Adding Parameters to Links¶
You can pass any number of arguments to the pager function and they will be used to create the links to the other pages.
For example with the following code:
${tmpl_context.paginators.movies.pager(param1='hi', param2='man')}
the resulting links will be:
- /list?page=1¶m1=hi¶m2=man
- /list?page=2¶m1=hi¶m2=man
and so on…
By default the url used to generate links will be the same of the page where the paginated data will be visible, this can be changed by passing the link argument to the pager function:
${tmpl_context.paginators.movies.pager(link='/otherlink', param1='hi', param2='man')}
and the resulting link will be generated by using the provided url:
- /otherlink?page=1¶m1=hi¶m2=man
Adding Previous And Next Links¶
Apart from providing the pager method the paginator Page object we receive
inside our template context provides the previous_page and next_page properties
which can be used to create previous/next links:
<p class="pagelist">
<a class="prevPage" href="/list?page=${tmpl_context.paginators.movies.previous_page}"><<<</a>
${tmpl_context.paginators.movies.pager(format='~3~', page_param='page', show_if_single_page=True)}
<a class="nextPage" href="/list?page=${tmpl_context.paginators.movies.next_page}">>>></a>
</p>
Adding Some Arrow Images¶
Once you added your own previous/next page entities you can style them as you prefer, one common need is to display an image instead of the text:
a.prevPage {
background: url("/images/icons/png/32x32/arrow-left.png") no-repeat;
padding-left: 18px;
padding-right: 18px;
padding-top: 12px;
padding-bottom: 15px;
text-decoration: none;
}
a.nextPage {
background: url("/images/icons/png/32x32/arrow-right.png") no-repeat;
padding-left: 18px;
padding-right: 18px;
padding-top: 12px;
padding-bottom: 15px;
text-decoration: none;
}
Web Session¶
| Status: | Work in progress |
|---|
Why Use Sessions?¶
Sessions are a common way to keep simple browsing data attached to a user’s browser. This is generally used to store simple data that does not need to be persisted in a database.
Sessions in TurboGears can be backed by the filesystem, memcache, the database, or by hashed cookie values. By default, cookies are used for storing the session data, which is only good for storing very little amounts of data in the session since all data will be sent back and forth within the cookie. If you are storing lots of data in the session, Memcache is recommended.
Warning
Using cookies for storing the whole session’s content exposes your application to possible exploits if the attacker gets to know the secret key which is used for the encryption of the cookies. Considering this, it is probably better to use the filesystem storage if you don’t want to set up memcache.
Note
When using the filesystem backed storage, you must be aware of the fact, that beaker does not clean up the session files at all. You have to make sure to clean up the data directory on a regular basis yourself. Refer to the Beaker documentation for more details.
How To Use Sessions?¶
If you just quickstarted a TurboGears 2 application, the session system is pre-configured and ready to be used.
By default we are using the Beaker session system. This system is configured to use hashed cookies for session storage.
Each time a client connects, the session middleware (Beaker) will inspect the cookie using the cookie name we have defined in the configuration file.
If the cookie is not found it will be set in the browser. On all subsequent visits, the middleware will find the cookie and make use of it.
When using the cookie based backend, all data that you put into the session will be pickled, hashed and encrypted by the middleware when sending the response to the browser and vice-versa when reading the request.
In the other backends, the cookie only contains a large random key that was set at the first visit and has been associated behind the scenes to a file in the file system cache. This key is then used to lookup and retrieve the session data from the proper datastore.
OK, enough with theory! Let’s get to some real life (sort of) examples. Open up your root controller and add the following import at the top the file:
from tg import session
What you get is a Session instance that is always request-local, in other words, it’s the session for this particular user. The session can be manipulated in much the same way as a standard python dictionary.
Here is how you search for a key in the session:
if session.get('mysuperkey', None):
# do something intelligent
pass
and here is how to set a key in the session:
session['mysuperkey'] = 'some python data I need to store'
session.save()
You should note that you need to explicitly save the session in order for your keys to be stored in the session.
You can delete all user session with the delete() method of the session object:
session.delete()
Even though it’s not customary to delete all user sessions on a production environment, you will typically do it for cleaning up after usability or functional tests.
Avoid automatic session extension¶
TurboGears by default automatically extends session life time at every request if a session is already available. You can avoid this behavior by changing your application configuration
beaker.session.tg_avoid_touch = true
This will also prevent TurboGears from causing an automatic session save at every request.
Caching¶
Caching is a common techneque to achieve performance goals, when a web application has to perform some operation that could take a long time. There are two major types of caching used in Web Applications:
- Whole-page caching – works at the HTTP protocol level to avoid entire requests to the server by having either the user’s browser, or an intermediate proxy server (such as Squid) intercept the request and return a cached copy of the file.
- Application-level caching – works within the application server to cache computed values, often the results of complex database queries, so that future requests can avoid needing to re-caculate the values.
Most web applications can only make very selective use of HTTP-level caching, such as for caching generated RSS feeds, but that use of HTTP-level caching can dramatically reduce load on your server, particularly when using an external proxy such as Squid and encountering a high-traffic event (such as the Slashdot Effect).
For web applications, application-level caching provides a flexible way to cache the results of complex queries so that the total load of a given controller method can be reduced to a few user-specific or case-specific queries and the rendering overhead of a template. Even within templates, application-level caching can be used to cache rendered HTML for those fragments of the interface which are comparatively static, such as database-configured menus, reducing potentially recursive database queries to simple memory-based cache lookups.
Application-level Caching¶
TurboGears comes with application-level caching middleware enabled by default in QuickStarted projects. The middleware, Beaker is the same package which provides Session storage for QuickStarted projects. Beaker is the cache framework used by TurboGears 2.3.12.
Beaker supports a variety of backends which can be used for cache or session storage:
- memory – per-process storage, extremely fast
- filesystem – per-server storage, very fast, multi-process
- “DBM” database – per-server storage, fairly fast, multi-process
- SQLAlchemy database – per-database-server storage, integrated into your main DB infrastructure, so potentially shared, replicated, etc., but generally slower than memory, filesystem or DBM approaches
- Memcached – (potentially) multi-server memory-based cache, extremely fast, but with some system setup requirements
Each of these backends can be configured from your application’s configuration file, and the resulting caches can be used with the same API within your application.
Controller Caching¶
TurboGears provides an helper for quick controller caching.
By using the tg.decorators.cached() decorator on a
controller you can specify that the whole controller body
has to be cached depending on the parameters of the request.
By default also the Content-Type and Content-Length
headers will be cached, so that we ensure that the content
related headers get restored together with the cached content.
Using @cached is pretty straightforward:
class SimpleController(TGController):
@cached(expire=100, type='memory')
@expose()
def simple(self):
return "Hey, this is a cached controller!"
When using @cached`` decorator, the *namespace* and
*cache key* used are available in ``request.caching
as request.caching.namespace and request.caching.key.
You can specify additional headers to be cached or change the
cache key by specifying the @cached arguments. Please refer
to tg.decorators.cached() reference for details.
Note
Please note that @cached will only cache the controller
body, the template will still be rendered. To cache both
the controller and template join it with template_cache
using the cache key provided be request.caching.key.
Manually Using the Cache¶
The configured Beaker is more properly thought of as a CacheManager, as it provides access to multiple independent cache namespaces.
To access the cache from within a controller module:
from tg import cache
@expose()
def some_action(self, day):
# hypothetical action that uses a 'day' variable as its key
def expensive_function():
# do something that takes a lot of cpu/resources
return expensive_call()
# Get a cache for a specific namespace, you can name it whatever
# you want, in this case its 'my_function'
mycache = cache.get_cache('my_function')
# Get the value, this will create the cache copy the first time
# and any time it expires (in seconds, so 3600 = one hour)
cachedvalue = mycache.get_value(
key=day,
createfunc=expensive_function,
expiretime=3600
)
return dict(myvalue=cachedvalue)
The Beaker cache is a two-level namespace, with the keys at each level being string values. The call to cache.get_cache() retrieves a cache namespace which will map a set of string keys to stored values. Each value that is stored in the cache must be pickle-able.
Pay attention to the keys you are using to store your cached values. You need to be sure that your keys encode all of the information that the results being cached depend upon in a unique manner. In the example above, we use day as the key for our cached value, on the assumption that this is the only value which affects the calculation of expensive_function, if there were multiple parameters involved, we would need to encode each of them into the key.
Note
The Beaker API exposed here requires that your functions for calculating complex values be callables taking 0 arguments. Often you will use a nested function to provide this interface as simply as possible. This function will only be called if there is a cache miss, that is, if the cache does not currently have the given key recorded (or the recorded key has expired).
Other Cache Operations¶
The cache also supports the removal values from the cache, using the key(s) to identify the value(s) to be removed and it also supports clearing the cache completely, should it need to be reset.
# Clear the cache
mycache.clear()
# Remove a specific key
mycache.remove_value('some_key')
Configuring Beaker¶
Beaker is configured in your QuickStarted application’s main configuration file in the app:main section.
To use memory-based caching:
[app:main]
beaker.cache.type = memory
To use file-based caching:
[app:main]
beaker.cache.type = file
beaker.cache.data_dir = /tmp/cache/beaker
beaker.cache.lock_dir = /tmp/lock/beaker
To use DBM-file-based caching:
[app:main]
beaker.cache.type = dbm
beaker.cache.data_dir = /tmp/cache/beaker
beaker.cache.lock_dir = /tmp/lock/beaker
To use SQLAlchemy-based caching you must provide the url parameter for the Beaker configuration. This can be any valid SQLAlchemy URL, the Beaker storage table will be created by Beaker if necessary:
[app:main]
beaker.cache.type = ext:database
beaker.cache.url = sqlite:///tmp/cache/beaker.sqlite
Memcached¶
Memcached allows for creating a pool of colaborating servers which manage a single distributed cache which can be shared by large numbers of front-end servers (i.e. TurboGears instances). Memcached can be extremely fast and scales up very well, but it involves an external daemon process which (normally) must be maintained (and secured) by your sysadmin.
Memcached is a system-level daemon which is intended for use solely on “trusted” networks, there is little or no security provided by the daemon (it trusts anyone who can connect to it), so you should never run the daemon on a network which can be accessed by the public! To repeat, do not run memcached without a firewall or other network partitioning mechanism! Further, be careful about storing any sensitive or authentication/authorization data in memcache, as any attacker who can gain access to the network can access this information.
Ubuntu/Debian servers will generally have memcached configured by default to only run on the localhost interface, and will have a small amount of memory (say 64MB) configured. The /etc/memcached.conf file can be edited to change those parameters. The memcached daemon will also normally be deactivated by default on installation. A basic memcached installation might look like this on an Ubuntu host:
sudo apt-get install memcached
sudo vim /etc/default/memcached
# ENABLE_MEMCACHED=yes
sudo vim /etc/memcached.conf
# Set your desired parameters...
sudo /etc/init.d/memcached restart
# now install the Python-side client library...
# note that there are other implementations as well...
easy_install python-memcached
You then need to configure TurboGears/Pylon’s beaker support to use the memcached daemon in your .ini files:
[app:main]
beaker.cache.type = ext:memcached
beaker.cache.url = 127.0.0.1:11211
# you can also store sessions in memcached, should you wish
# beaker.session.type = ext:memcached
# beaker.session.url = 127.0.0.1:11211
You can have multiple memcached servers specified using ; separators. Usage, as you might imagine is the same as with any other Beaker cache configuration (that is, to some extent, the point of the Beaker Cache abstraction, after all):
References¶
- Beaker Caching – discussion of use of Beaker’s caching services
- Beaker Configuration – the various parameters which can be used to configure Beaker in your config files
- Memcached – the memcached project
- Python Memcached – Python client-side binding for memcached
- Caching for Performance – Stephen Pierzchala’s general introduction to the concept of caching in order to improve web-site performance
Template Caching¶
Template Loader Cache¶
Kajiki and Genshi will retrieve the templates from a cache if they have not changed.
On Kajiki the cache is unlimited, every template loaded from file system is cached in memory, while on Genshi this cache has a default size of 25, when there are more than 25, the least recently used templates will be removed from this cache.
You can change this behavior by setting the genshi.max_cache_size option
into the development.ini:
[app:main]
genshi.max_cache_size=100
Another speed boost can be achieved by disabling template automatic reloading, this will avoid having Genshi and Kajiki compare the timestamp of files whenever they are rendered.
[app:main]
auto_reload_templates = false
Prerendered Templates Caches¶
In templates, the cache namespace will automatically be set to the name of
the template being rendered. To cache a template you just have to return
the tg_cache option from the controller that renders the cached template.
tg_cache is a dictionary that accepts the following keys:
- key: The cache key. Default: None
- expire: how long the cache must stay alive. Default: never expires
- type: memory, dbm, memcached. Default: dbm
if any of the keys is available the others will default, if all three are missing caching will be disabled. For example to enable caching for 1 hour for the profile of an user:
@expose('myproj.templates.profile')
def profile(self, username):
user = DBSession.query(User).filter_by(user_name=user_name).first()
return dict(user=user, tg_cache=dict(key=user_name, expire=3600))
HTTP-Level Caching¶
HTTP supports caching of whole responses (web-pages, images, script-files and the like). This kind of caching can dramatically speed up web-sites where the bulk of the content being served is largely static, or changes predictably, or where some commonly viewed page (such as a home-page) requires complex operations to generate.
HTTP-level caching is handled by external services, such as a Squid proxy or the user’s browser cache. The web application’s role in HTTP-level caching is simply to signal to the external service what level of caching is appropriate for a given piece of content.
Note
If any part of you page has to be dynamically generated, even the simplest fragment, such as a user-name, for each request HTTP caching likely will not work for you. Once the page is HTTP-cached, the application server will not recieve any further requests until the cache expires, so it will not generally be able to do even minor customizations.
Browser-side Caching with ETag¶
HTTP/1.1 supports the ETag caching system that allows the browser to use its own cache instead of requiring regeneration of the entire page. ETag-based caching avoids repeated generation of content but if the browser has never seen the page before, the page will still be generated. Therefore using ETag caching in conjunction with one of the other types of caching listed here will achieve optimal throughput and avoid unnecessary calls on resource-intensive operations.
Caching via ETag involves sending the browser an ETag header so that it knows to save and possibly use a cached copy of the page from its own cache, instead of requesting the application to send a fresh copy.
The etag_cache() function will set the proper HTTP headers if the browser
doesn’t yet have a copy of the page. Otherwise, a 304 HTTP Exception will be
thrown that is then caught by Paste middleware and turned into a proper 304
response to the browser. This will cause the browser to use its own
locally-cached copy.
ETag-based caching requires a single key which is sent in the ETag HTTP header back to the browser. The RFC specification for HTTP headers indicates that an ETag header merely needs to be a string. This value of this string does not need to be unique for every URL as the browser itself determines whether to use its own copy, this decision is based on the URL and the ETag key.
from tg.controllers.util import etag_cache
def my_action(self):
etag_cache('somekey')
return render('/show.myt', cache_expire=3600)
Or to change other aspects of the response:
from tg.controllers.util import etag_cache
from tg import response
def my_action(self):
etag_cache('somekey')
response.headers['content-type'] = 'text/plain'
return render('/show.myt', cache_expire=3600)
Note
In this example that we are using template caching in addition to ETag caching. If a new visitor comes to the site, we avoid re-rendering the template if a cached copy exists and repeat hits to the page by that user will then trigger the ETag cache. This example also will never change the ETag key, so the browsers cache will always be used if it has one.
The frequency with which an ETag cache key is changed will depend on the web application and the developer’s assessment of how often the browser should be prompted to fetch a fresh copy of the page.
- ETag
- From Wikipedia An ETag (entity tag) is an HTTP response header returned by an HTTP/1.1 compliant web server used to determine change in content at a given URL.
Todo
Add links to Beaker region (task-specific caching mechanisms) support.
Todo
Document what the default Beaker cache setup is for TG 2.3.12 quickstarted projects (file-based, likely).
Todo
Provide code-sample for use of cache within templates
Internationalization And Localization¶
| Status: | Work in progress |
|---|
Table of Contents
Turbogears2 relies on Babel for i18n and l10n support. So if this document is not enough you will want to check their respective documentation:
- Babel’s UserGuide
A quickstarted project comes fully i18n enabled so you should get started quickly.
If you’re lucky enough you’ll even see “Your application is now running” message in your language.
Language Auto-Select¶
Turbogears2 contains the logic to setup request’s language based on browser’s preferences(*).
[*] - Every modern browser sends a special header along with every web request which tells the server which language it would prefer to see in a response.
The language in use during the request is available through the
tg.i18n.get_lang() function. This will report the currently
selected languages both in case of an auto-detected language preference
or in case of languages forced by the developer.
Languages returned by tg.i18n.get_lang() are ordered by
user preference. By default all the requested languages are returned,
if you want to get only the ones your application supports call
get_lang(all=False).
The current language in use is usually get_lang(all=False)[0].
Forcing a Language¶
Developer can force the currently used language for each request
using the tg.i18n.set_temporary_lang() function. This
will change the language only for the current request.
If you need to permanently change the language for the user
session duration, tg.i18n.set_lang() can be used.
If TurboGears session support is enabled it will store the
choosen language inside the session and recover it whenever
the user comes back on next request.
Making your code international¶
Whenever you write a message that has to displayed you must let TurboGears know that it has to be translated.
Even though TurboGears is able to automatically detect content inside tags and mark them for translation all the strings inside controllers must be explicitly marked as translated.
This can be achieved with the tg.i18n.ugettext() and
tg.i18n.lazy_ugettext() calls which are usually imported
with _ and l_ names:
from tg.i18n import ugettext as _
class RootController(BaseController):
@expose('myproj.templates.index')
def index(self):
return dict(msg=_('Hello World'))
In the previous example the ‘Hello World’ string will be detect by TurboGears when collecting translatable text and will display in the browser language if a translation for that language is available.
While ugettext() works perfectly to translate
strings inside a request it does not allow translating strings outside a request.
This is due to the fact that TurboGears won’t know the browser language when
there isn’t a running request. To translate global variables, parameters
default values or any other string which is created outside a controller
method the lazy_ugettext method must be used:
from tg.i18n import lazy_ugettext as l_
class RootController(BaseController):
@expose('myproj.templates.index')
def index(self, msg=l_('Hello World')):
return dict(msg=msg)
In this case the msg parameter is translated using lazy_ugettext()
as it is constructed at controller import time when no request is available.
This will create an object that will translate the given string only when
the string itself is displayed or evaluated.
Keep in mind that as the lazy string object built by lazy_ugettext() is
translated whenever the string is evaluated joining strings or editing it
will force the translation. So the resulting object must still be evaluated
only inside a request or it will always be translated to the default project
language only.
An i18n Quick Start¶
After quickstarting your project, you will notice that the setup.py file contains the following section:
message_extractors = {'yourapp': [
('**.py', 'python', None),
('templates/**.xhtml', 'kajiki', None),
('public/**', 'ignore', None)]},
This is an extraction method mapping that indicates to distutils which files should be searched for strings to be translated. TurboGears2 uses Babel to extract messages to a .pot file in your project’s i18n directory. Don’t forget to add it to your revision control system if you use one.
1. Extract all the translatable strings from your project’s files by using the following command:
python setup.py extract_messages
This command will generate a “pot” file in the i18n folder of your application. This pot file is the reference file that serves for all the different translations.
2. Create a translation catalog for your language, let’s take ‘zh_tw’ for example:
python setup.py init_catalog -l zh_tw
3. Edit your language in i18n/[country code]/LC_MESSAGES/[project-name].po
If you’re not an expert in i18n or if you would like to give the files to someone else so that he helps you we recommend that you use the really nice poedit program. This program works nicely on GNU/Linux and Windows and provides a nice user-interface to edit po files.
Compile your lang:
python setup.py compile_catalog
Config development.ini:
[app:main] use = egg: my-project full_stack = true lang = zh_tw
Start server:
gearbox serve --reload
And see the local message show on the screen.
Commands¶
To fresh start a translation, you could use the following command to handle your locales:
init_catalog¶
You can extract all messages from the project with the following command:
python setup.py init_catalog -l [country code]
The country code could be es(Spanish), fr(France), zh_tw(Taiwan), jp(JAPAN), ru(Russian), or any other country code.
Compile Catalog¶
You can extract all messages from the project with the following command:
python setup.py compile_catalog
Update Catalog¶
You can update the catalog with the following command:
python setup.py update_catalog
Writing TurboGears Extensions¶
TurboGears provides a bunch of hook points and ways to extend the framework behavior
to write extensions. Most of the needs can usually be satisfied by relying on
tg.configuration.AppConfig, Configuration Milestones and Hooks.
Creating an Extension¶
TurboGears extensions are identified by the tgext.* package name, since
version 2.3.2 the devtools provide a command to quickly create a new TurboGears
extension which can be both used by itself or plugged through tgext.pluggable.
To quickstart the extension run:
$ gearbox tgext -n myextension -a "My Name" -e "my@email.com"
This will create a tgext.myextension directory which a simple sample extension
inside. To tune the author, description, license of your extension have a look
at the tgext command options using gearbox help tgext.
Tuning on the Extension¶
If you created the tgext.myextension using the tgext command you can
quickly turn it on by editing your application config/app_cfg.py and appending
at the end the following lines:
import tgext.myextension
tgext.myextension.plugme(base_config)
of through the pluggable application interface if tgext.pluggable is available:
from tgext.pluggable import plug
plug(base_config, 'tgext.myextension')
By enabling the autogenerated extension and starting gearbox serve again,
you will notice that it just provides a bunch of logging hooks and not much more:
17:52:30,019 INFO [tgext.myextension] Setting up tgext.myextension extension...
17:52:30,023 INFO [tgext.myextension] >>> Public files path is /home/USER/myapp/myapp/public
17:52:30,023 INFO [tgext.myextension] + Application Running!
17:52:30,040 INFO [gearbox] Starting server in PID 22484.
Starting HTTP server on http://0.0.0.0:8080
17:52:30,740 INFO [tgext.myextension] Serving: /
The Sample Extension¶
The sample extension created by the tgext command provides its actual setup
inside the __init__.py file. This file showcases some of the extension
points provided by TurboGears.
If you actually created tgext.myextension you should get a
tgext.myextension/tgext/myextension/__init__.py file with the following content:
from tg import config
from tg import hooks
from tg.configuration import milestones
import logging
log = logging.getLogger('tgext.myextension')
# This is the entry point of your extension, will be called
# both when the user plugs the extension manually or through tgext.pluggable
# What you write here has the same effect as writing it into app_cfg.py
# So it is possible to plug other extensions you depend on.
def plugme(configurator, options=None):
if options is None:
options = {}
log.info('Setting up tgext.myextension extension...')
milestones.config_ready.register(SetupExtension(configurator))
return dict(appid='tgext.myextension')
# Most of your extension initialization should probably happen here,
# where it's granted that .ini configuration file has already been loaded
# in tg.config but you can still register hooks or other milestones.
class SetupExtension(object):
def __init__(self, configurator):
self.configurator = configurator
def __call__(self):
log.info('>>> Plugin files path is %s' % config['paths']['static_files'])
hooks.register('startup', self.on_startup)
def echo_wrapper_factory(handler, config):
def echo_wrapper(controller, environ, context):
log.info('Serving: %s' % context.request.path)
return handler(controller, environ, context)
return echo_wrapper
# Application Wrappers are much like easier WSGI Middleware
# that get a TurboGears context and return a Response object.
self.configurator.register_wrapper(echo_wrapper_factory)
def on_startup(self):
log.info('+ Application Running!')
The core parts of the previous example are:
- plugme function, this is the function used to turn on your extension.
will be automatically called by
tgext.pluggablewhen the extension is enabled using the pluggable application interface or by the user itself when manually enabling your extension. Inside this method the application configurator (AppConfig) object is available and the options the user specified for the extension, but not application configuration as it has not been loaded yet.
- SetupExtension.__call__, this is a callable that is registered
by the plugme function for the
config_readmilestone so that it is executed when the .ini configuration has been loaded and merged with the options declared through the application configurator in config/app_cfg.py.Here you can register additional milestones, functions or access and modify the application configurator through the
self.configuratorobject.
- SetupExtension.on_startup This is a sample hook registered on
application startup by
SetupExtension.__call__that says hello when the application has started. Have at look at Hooks for a complete list of available hooks.
Extensions with models and controllers¶
If your extension needs to expose models and controllers you probably want to have a look at Pluggable Applications which are meant to create reusable turbogears applications that can be plugged inside other applications to extend their features.
Pluggable Applications with TurboGears¶
TurboGears 2.1.4 introduced support for pluggable applications using tgext.pluggable.
tgext.pluggable is now the official supported way in TurboGears to create pluggable
reusable applications.
Currently only SQLAlchemy based applications are supported as pluggable applications.
Official documentation for tgext.pluggable can be found at: http://pypi.python.org/pypi/tgext.pluggable
Supported Features¶
Pluggable applications can define their own:
- controllers, which will be automatically mounted when the application is purged.
- models, which will be available inside and outside of the plugged application.
- helpers, which can be automatically exposed in
hobject in application template.- bootstrap, which will be executed when setup-app is called.
- statics, which will be available at their own private path.
Mounting a pluggable application¶
In your application config/app_cfg.py import plug from tgext.pluggable and
call it for each pluggable application you want to enable.
The plugged package must be installed in your environment.
from tgext.pluggable import plug
plug(base_config, 'package_name')
Creating Pluggable Applications¶
tgext.pluggable provides a quickstart-pluggable command
to create a new pluggable applications:
$ gearbox quickstart-pluggable plugtest
...
The quickstarted application will provide an example on how to use models, helpers, bootstrap, controllers and statics.
RESTful Requests Dispatching¶
If you are developing an application where you want to expose your
database using a stateless API, tg.RestController
might be for you. If you want to serve resources with multiple
formats, and handle embedded resource lookup, you might find
RestController useful. If you want to provide simple URLs which are
consistent across all of the data shared and manipulated in your
application, RestController is probably worth a look.
Unlike TGController, RestController provides a mechanism to access the request’s method, not just the URL. If you are not familiar with how HTTP requests work, think for a minute about the difference between sending a form with GET and POST. Primarily, developers use POST to send data to modify the database. They use GET to retrieve data. These are HTTP methods.
Standard HTTP verbiage includes: GET, POST, PUT, and DELETE. RestController supports these, and also adds a few shortcuts for URL dispatch that makes displaying the data as forms and lists a little easier for the user. The API docs describe each of the supported controller functions in brief, so use that if you already understand REST and need a quick way to start using it, this document will be your guide. It is intended to provide a step-by-step example of how to implement REST using RestController.
To explain how RESTful works with TurboGears we are going to define a simple webservice that exposes a list of movies. WebServices are usually an ideal candidate for RESTful dispatch and so provide a simple and clean showcase of the feature.
Here is the Model used to develop this chapter:
from sqlalchemy import Column, Integer, String, Date, Text, ForeignKey, Table
from sqlalchemy.orm import relation
from moviedemo.model import DeclarativeBase, metadata
movie_directors_table = Table('movie_directors', metadata,
Column('movie_id', Integer, ForeignKey('movies.movie_id'), primary_key = True),
Column('director_id', Integer, ForeignKey('directors.director_id'), primary_key = True))
class Genre(DeclarativeBase):
__tablename__ = "genres"
genre_id = Column(Integer, primary_key=True)
name = Column(String(100))
class Movie(DeclarativeBase):
__tablename__ = "movies"
movie_id = Column(Integer, primary_key=True)
title = Column(String(100), nullable=False)
description = Column(Text, nullable=True)
genre_id = Column(Integer, ForeignKey('genres.genre_id'))
genre = relation('Genre', backref='movies')
release_date = Column(Date, nullable=True)
class Director(DeclarativeBase):
__tablename__ = "directors"
director_id = Column(Integer, primary_key=True)
title = Column(String(100), nullable=False)
movies = relation(Movie, secondary=movie_directors_table, backref="directors")
I am isolating Movies, Genres, and Directors for the purpose of understanding how objects might relate to one another in a RESTful context. For purposes of this demonstration, Movies can only have one Genre, but may be related to one or more Directors. Directors may be related to one or more Movies.
Listing Resources¶
Lets provide a simple listing of the movies in our database.
Our controller class is going to look like this:
from tg import RestController
from tg.decorators import with_trailing_slash
class MovieController(RestController):
@expose('json')
def get_all(self):
movies = DBSession.query(Movie).all()
return dict(movies=movies)
Supposing our MovieController is mounted with the name movies inside
our RootController going to http://localhost:8080/movies will provide
the list of our movies encoded in json format.
If you ware looking for a way to fill some sample movies, just jump to http://localhost:8080/admin and create any data you need to make sure your controller is working as expected.
Creating New Items¶
We use the post method to define how we go about saving our movie to the database. This method gets called whenever the http://localhost:8080/movies url is accessed using a POST request:
from datetime import datetime
class MovieRestController(RestController):
@expose('json')
def post(self, title, description, directors=None, genre_id=None, release_date=None):
if genre_id is not None:
genre_id = int(genre_id)
if directors is not None:
if not isinstance(directors, list):
directors = [directors]
directors = [DBSession.query(Director).get(director) for director in directors]
else:
directors = []
if release_date is not None:
release_date = datetime.strptime(release_date, "%m/%d/%y")
movie = Movie(title=title, description=description, release_date=release_date,
directors=directors, genre_id=genre_id)
DBSession.add(movie)
DBSession.flush()
return dict(movie=movie)
If the insertion is successful we are going to receive back the newly created
movie with its movie_id. The DBSession.flush() call is explicitly there
to make SQLAlchemy get a movie_id for the newly inserted movie.
This will not be the case if the user enters some weird date format for “release_date” or doesn’t provide a title or description.
One way to counteract this problem is by writing a validator when the parameters don’t respect the expected format.
If you don’t know how to test this controller, check for browser extension to make POST requests. Most browser have one, for Google Chrome you can try PostMan which does a good job.
Validating The User’s Input¶
Before we add our record to the database, it is probably a good idea to validate the data so we can prompt the user if there are mistakes. RestController uses the same machinery that TGControllers use for validation. We use FormEncode’s validators to test that our fields are not empty, and that the release_date has correct formatting:
from tg import request, validate
from formencode.validators import NotEmpty, Int, DateConverter
@validate({'title':NotEmpty,
'description':NotEmpty,
'genre_id':Int(not_empty=True),
'release_date':DateConverter(not_empty=True)})
@expose('json')
def post(self, **kw):
if request.validation['errors']:
return dict(errors=dict([(field, str(e)) for field, e in request.validation['errors'].items()]))
#...proceed like before...
Note that the validation errors are stored in request.validation. This is done by the TG dispatch on a failed validation.
Getting one Item¶
Using the get_one() method, we can display one item from the database to the user.:
@expose('json')
def get_one(self, movie_id):
movie = DBSession.query(Movie).get(movie_id)
return dict(movie=movie)
Updating an Existing Item¶
PUT is the method used for updating an existing record using REST. We can validate in the same manner as before:
@validate({'title':NotEmpty,
'description':NotEmpty,
'genre_id':Int(not_empty=True),
'release_date':DateConverter(not_empty=True)})
@expose('json')
def put(self, movie_id, title, description, directors, genre_id, release_date, **kw):
if request.validation['errors']:
return dict(errors=dict([(field, str(e)) for field, e in request.validation['errors'].items()]))
movie = DBSession.query(Movie).get(movie_id)
if not movie:
return dict(errors={'movie':'Movie not found'})
genre_id = int(genre_id)
if not isinstance(directors, list):
directors = [directors]
directors = [DBSession.query(Director).get(director) for director in directors]
movie.genre_id = genre_id
movie.title=title
movie.description = description
movie.directors = directors
movie.release_date = release_date
return dict(movie=movie)
Deleting An Item From Our Resource¶
The work-horse of delete is attached to the post_delete method. Here we actually remove the record from the database, and then redirect back to the listing page:
@expose('json')
def post_delete(self, movie_id, **kw):
movie = DBSession.query(Movie).get(movie_id)
if not movie:
return dict(errors={'movie':'Movie not found'})
DBSession.delete(movie)
return dict(movie=movie.movie_id)
Nesting Resources With RestControllers¶
RestControllers expect nesting as any TG controller would, but it uses a different method of dispatch than regular TG Controllers. This is necessary when you need resources that are related to other resources. This can be a matter of perspective, or a hard-link which filters the results of the sub controller. For our example, we will use a nested controller to display all of the directors associated with a Movie.
The challenge for design of your RESTful interface is determining how to associate parts of the URL to the resource definition, and defining which parts of the URL are part of the dispatch.
To do this, RestController introspects the get_one method to determine
how many bits of the URL to nip off and makes them available inside the
request.controller_state.routing_args dictionary.
This is because you may have one or more identifiers to determine an object; for instance you might use lat/lon to define a location. Since our MovieController defines a get_one which takes a movie_id as a parameter, we have no work to do there.
All we have to do now is define our MovieDirectorController, and provide linkage into the MovieController to provide this functionality:
from tg import request
class MovieDirectorController(RestController):
@expose('json')
def get_all(self):
movie_id = request.controller_state.routing_args.get('movie_id')
movie = DBSession.query(Movie).get(movie_id)
return dict(movie=movie, directors=movie.directors)
class MovieRestController(RestController):
directors = MovieDirectorController()
@expose('json')
def get_one(self, movie_id):
movie = DBSession.query(Movie).get(movie_id)
return dict(movie=movie)
This example only defines the get_all function, I leave the other RESTful verbiage as an exercise for you to do.
One trick that I will explain, is how to use _before to get
the related Movie object within all of your MovieDirectorController
methods with a single define.
Here is what the Controller looks like with _before added in:
from tg import tmpl_context, request
class MovieDirectorController(RestController):
def _before(self, *args, **kw):
movie_id = request.controller_state.routing_args.get('movie_id')
tmpl_context.movie = DBSession.query(Movie).get(movie_id)
@with_trailing_slash
@expose('json')
def get_all(self):
return dict(movie=tmpl_context.movie, directors=tmpl_context.movie.directors)
Non-RESTful Methods¶
Let’s face it, REST is cool, but sometimes it doesn’t meet our needs or time constraints. A good example of this is a case where you want an autocomplete dropdown in your “edit” form, but the resource that would provide the Json for this dropdown has not been fleshed out yet. as a hack, you might add a field_dropdown() method in your controller which sends back the json required to feed your form. RestController allows methods named outside of the boundaries of the default methods supported. In other words, it’s just fine to include a method in your RestController that does not fit the REST HTML verbiage specification.
Supporting TGController’s Inside RestController¶
Just as RestController supports obscure names for methods, it can handle nested TGController classes as well. When dispatch encounters a URL which maps to a non-RestController, it switches back to the normal TG dispatch. Simply said, you may include regular classes for dispatch within your RestController definition.
Database Schema Migrations¶
Since version 2.1.1 TurboGears has integrated migrations support for each new quickstarted project.
TurboGears 2.3 and newer rely on the alembic project to automate database schema migration.
Getting Started¶
TurboGears provides a gearbox migrate command to manage schema migration.
You can run gearbox migrate db_version to see the current version
of your schema:
$ gearbox migrate -c development.ini db_version
Context impl SQLiteImpl.
Will assume transactional DDL.
Current revision for sqlite:////tmp/migr/devdata.db: None
By default the database version is None until a migration is applied.
The first time a migration is applied the migrate_version table is
created. This table will keep the current version
of your schema to track when applying migrations is required.
If you examine your database, you should be able to see schema version tracking table and check what it is the current version of your schema:
sqlite> .headers on
sqlite> select * from migrate_version;
version_num
4681af2393c8
This is exactly like running the gearbox migrate db_version command, both
should tell you the same database version.
In this case the reported version is 4681af2393c8.
Integrating Migrations in the Development Process¶
With the database under version control and a repository for schema change scripts, you are ready to begin regular development. We will now walk through the process of creating, testing, and applying a change script for your current database schema. Repeat these steps as your data model evolves to keep your databases in sync with your model.
Creating migrations¶
The gearbox migrate script command will create an empty change script for you,
automatically naming it and placing it in your repository:
$ gearbox migrate create 'Initial Schema'
The command will return by just printing the migrations repository where it is going to create the new script:
$ gearbox migrate create 'Initial Schema'
Generating /tmp/migr/migration/versions/2a3f515bad0_initial_schema.py... done
$ ls migration/versions
2a3f515bad0_this_is_an_example.py
Edit the Script¶
Each change script provides an upgrade and downgrade method, and
we implement those methods by creating and dropping the account table
respectively:
revision = '2a3f515bad0'
down_revision = '4681af2393c8'
from alembic import op
import sqlalchemy as sa
def upgrade():
op.create_table(
'account',
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('name', sa.String(50), nullable=False),
sa.Column('description', sa.Unicode(200)),
)
def downgrade():
op.drop_table('account')
Test the Script¶
Anyone who has experienced a failed schema upgrade on a production database knows how uniquely uncomfortable that situation can be. Although testing a new change script is optional, it is clearly a good idea. After you execute the following test command, you will ideally be successful:
$ gearbox migrate test
Context impl SQLiteImpl.
Will assume transactional DDL.
Running upgrade 4681af2393c8 -> 2a3f515bad0
Context impl SQLiteImpl.
Will assume transactional DDL.
Running downgrade 2a3f515bad0 -> 4681af2393c8
If you receive an error while testing your script, one of two issues is probably the cause:
- There is a bug in the script
- You are testing a script that conflicts with the schema as it currently exists.
If there is a bug in your change script, you can fix the bug and rerun the test.
Applying migrations¶
The script is now ready to be deployed:
$ gearbox migrate upgrade
If your database is already at the most recent revision, the command will produce no output. If migrations are applied, you will see output similar to the following:
Context impl SQLiteImpl.
Will assume transactional DDL.
Running upgrade 4681af2393c8 -> 2a3f515bad0
Keeping your websetup on sync¶
Each time you create a new migration you should consider keeping your
websetup in sync with it. For example if you create a new table inside
a migration when you will run gearbox setup-app on a new database
it will already have the new table as you probably declared it in your
model too but the migrations version will be None. So trying to run any
migration will probably crash due to the existing tables.
To prevent this your websetup script should always initialize the
database in the same state where it would be after applying all the
available migrations. To ensure this you will have to add at the end
of the websetup/schema.py script a pool of commands to set the
schema version to the last one:
import alembic.config, alembic.command
alembic_cfg = alembic.config.Config()
alembic_cfg.set_main_option("script_location", "migration")
alembic_cfg.set_main_option("sqlalchemy.url", config['sqlalchemy.url'])
alembic.command.stamp(alembic_cfg, "head")
Downgrading your schema¶
There are some cases in which downgrading your schema might be required.
In those cases you can perform the gearbox migrate downgrade command:
$ gearbox migrate downgrade
Context impl SQLiteImpl.
Will assume transactional DDL.
Running downgrade 2a3f515bad0 -> 4681af2393c8
The GearBox Toolkit¶
The GearBox toolkit is a set of commands available since TurboGears 2.3 that replaced the paster command previously provided by pylons.
GearBox provides commands to create new full stack projects, serve PasteDeploy based applications, initialize them and their database, run migrations and start an interactive shell to work with them.
By default launching gearbox without any subcommand will start the interactive mode.
This provides an interactive prompt where gearbox commands, system shell commands and python statements
can be executed. If you have any doubt about what you can do simply run the help command to get
a list of the commands available (running help somecommand will provide help for the given sub command).
To have a list of all the available commands simply run gearbox --help
QuickStart¶
The gearbox quickstart command creates a new full stack TurboGears application,
just provide the name of your project to the command to create a new one:
$ gearbox quickstart myproject
The quickstart command provides a bunch of options to choose which template engine to use, which database engine to use various other options:
optional arguments:
-a, --auth add authentication and authorization support
-n, --noauth No authorization support
-m, --mako default templates mako
-j, --jinja default templates jinja
-k, --kajiki default templates kajiki
-g, --geo add GIS support
-p PACKAGE, --package PACKAGE
package name for the code
-s, --sqlalchemy use SQLAlchemy as ORM
-i, --ming use Ming as ORM
-x, --nosa No SQLAlchemy
--disable-migrations disable sqlalchemy-migrate model migrations
--enable-tw1 use toscawidgets 1.x in place of 2.x version
--skip-tw Disables ToscaWidgets
--skip-genshi Disables Genshi default template
--minimal-quickstart Throw away example boilerplate from quickstart project
Setup-App¶
The gearbox setup-app command runs the websetup.setup_app function of your project
to initialize the database schema and data.
By default the setup-app command is run on the development.ini file, to change this
provide a different one to the --config option:
$ gearbox setup-app -c production.ini
Serve¶
The gearbox serve command starts a PasteDeploy web application defined by the provided
configuration file. By default the development.ini file is used, to change this provide
a different one to the --config option:
$ gearbox serve -c production.ini --reload
The serve command provides a bunch of options to start the serve in daemon mode,
automatically restart the application whenever the code changes and many more:
optional arguments:
-c CONFIG_FILE, --config CONFIG_FILE
application config file to read (default:
development.ini)
-n NAME, --app-name NAME
Load the named application (default main)
-s SERVER_TYPE, --server SERVER_TYPE
Use the named server.
--server-name SECTION_NAME
Use the named server as defined in the configuration
file (default: main)
--daemon Run in daemon (background) mode
--pid-file FILENAME Save PID to file (default to gearbox.pid if running in
daemon mode)
--reload Use auto-restart file monitor
--reload-interval RELOAD_INTERVAL
Seconds between checking files (low number can cause
significant CPU usage)
--monitor-restart Auto-restart server if it dies
--status Show the status of the (presumably daemonized) server
--user USERNAME Set the user (usually only possible when run as root)
--group GROUP Set the group (usually only possible when run as root)
--stop-daemon Stop a daemonized server (given a PID file, or default
gearbox.pid file)
Changing HTTP Server¶
gearbox serve will look for the [server:main] configuration section
to choose which server to run and one which port and address to listen.
Any PasteDeploy compatible server can be used, by default the egg:gearbox#wsgiref
one is used, which is single threaded and based on python wsgiref implementation.
This server is idea for debugging as being single threaded removes concurrency issues and keeps around request local data, but should never be used on production.
On production system you might want to use egg:gearbox#cherrypy or egg:gearbox#gevent
servers which run the application on CherryPy and Gevent, it is also possible to use
other servers like Waitress (egg:waitress#main) if available.
TGShell¶
The gearbox tgshell command will load a TurboGears application and start
an interactive shell inside the application.
The application to load is defined by the configuration file, by default
development.ini is used, to load a different application or under a
different configuration provide a configuration file using the --config
option:
$ gearbox tgshell -c production.ini
The tgshell command provides an already active fake request which makes
possible to call functions that depend on tg.request, it is also
provided an app object through which is possible to make requests:
$ gearbox tgshell
TurboGears2 Interactive Shell
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3]
All objects from myapp.lib.base are available
Additional Objects:
wsgiapp - This project's WSGI App instance
app - WebTest.TestApp wrapped around wsgiapp
>>> tg.request
<Request at 0x3c963d0 GET http://localhost/_test_vars>
>>> app.get('/data.json').body
'{"params": {}, "page": "data"}'
>>> model.DBSession.query(model.User).first()
<User: name=manager, email=manager@somedomain.com, display=Example manager>
Running Scripts from Command Line¶
You will notice that writing a python script that depends on TurboGears and running from the command line will fail with an error.
Suppose you have a script like:
import tg
print 'Hello From', tg.request.path
Saving it as myscript.py and running it will fail with TurboGears
complaining about a missing context:
$ python myscript.py
Hello From
Traceback (most recent call last):
File "myscript.py", line 2, in <module>
print 'Hello From', tg.request._path
File "/Users/amol/wrk/tg/tg2/tg/support/objectproxy.py", line 19, in __getattr__
return getattr(self._current_obj(), attr)
File "/Users/amol/wrk/tg/tg2/tg/request_local.py", line 214, in _current_obj
return getattr(context, self.name)
File "/Users/amol/wrk/tg/tg2/tg/support/objectproxy.py", line 19, in __getattr__
return getattr(self._current_obj(), attr)
File "/Users/amol/wrk/tg/tg2/tg/support/registry.py", line 72, in _current_obj
'thread' % self.____name__)
TypeError: No object (name: context) has been registered for this thread
That because we are trying to access the current request (tg.request) but we are outside
of a web application, so no request exists.
TGShell command solves this problem as it uses same features described in Testing Outside Controllers to run any script inside a fake request:
$ gearbox tgshell myscript.py
15:30:13,925 INFO [tgext.debugbar] Enabling Debug Toolbar
15:30:13,953 INFO [auth] request classification: browser
15:30:13,953 INFO [auth] -- repoze.who request started (/_test_vars) --
15:30:13,953 INFO [auth] no identities found, not authenticating
15:30:13,953 INFO [auth] no challenge required
15:30:13,953 INFO [auth] -- repoze.who request ended (/_test_vars) --
Hello From /_test_vars
So through TGShell it’s also possible to run turbogears based scripts like you were inside an HTTP request for a controller.
Adding your own command¶
To add commands to available gearbox commands, add them to your project entry points under the gearbox.commands group. Here is en example with setuptools:
setup(
name='SampleApp',
# [...]
entry_points={
'gearbox.commands': [
'my_command = sample_app.my_commands_module.my_command_module:MyCommandClass'
]
)
Where your command class extend gearbox.command.Command:
# -*- coding: utf-8 -*-
from gearbox.command import Command
class MyCommandClass(Command):
def take_action(self, parsed_args):
print('Hello world')
Using MongoDB¶
TurboGears supports MongoDB out of the box by using the Ming ORM. Ming was made to look like SQLAlchemy, so if you are proficient with SQLAlchemy and MongoDB it should be easy for you to get used to the Ming query language. This also makes easy to port a TurboGears SQLAlchemy based application to MongoDB.
QuickStarting with MongoDB¶
To create a project using MongoDB you just need to pass the --ming
option to the gearbox quickstart command.
$ gearbox quickstart --ming myproj
The quickstarted project will provide an authentication and authorization layer like the one that is provided for the SQLAlchemy version. This means that you will have the same users and groups you had on the standard quickstarted project and also that all the predicates to check for authorization should work like before.
The main difference is that you won’t be able to use the application without having a running MongoDB database on the local machine.
By default the application will try to connect to a server on port 27017 on local machine using a database that has the same name of your package.
This can be changed by editing the development.ini file:
ming.url = mongodb://localhost:27017/myproject
Now that everything is in place to start using MongoDB as your database server you just need to proceed the usual way by filling your database.
$ gearbox setup-app
The quickstart command from above will create the authentication collections and setup a default user/password for you:
user: manager
password: managepass
Working With Ming¶
If you don’t know how Ming works at all, please take a few minutes to read over the Ming documentation as this documentation will only cover Ming integration with TurboGears.
Your quickstarted project will have a subpackage called model, made up of the following files:
- __init__.py: This is where the database access is set up. Your collections should be imported into this module, and you’re highly encouraged to define them in a separate module - entities, for example.
- session.py: This file defines the session of your database
connection. By default TurboGears will use a Session object
with multi-threading support. You will usually need to import
this each time you have to declare a
MappedClassto specify the session that has to be used to perform queries. - auth.py: This file will be created if you enabled authentication
and authorization in the quickstart. It defines three collections
repoze.whorelies on:User(for the registered members in your website and the groups they belong to),Group(for groups of users) andPermission(a permission granted to one or more groups).
Defining Your Own Collections¶
By default TurboGears configures Ming in Declarative mode.
This is similar to the SQLAlchemy declarative support and needs
each model to inherit from the MappedClass class.
The tables defined by the quickstart in model/auth.py are based on the declarative method, so you may want to check it out to see how columns are defined for these tables.
To see how to define your models refer to Ming UserGuide
Once you have defined your collections in a separate module in the model
package, they should be imported from model/__init__.py. So the end
of this file would look like this:
# Import your model modules here.
from auth import User, Permission
# Say you defined these three classes in the 'movies'
# module of your 'model' package.
from movies import Movie, Actor, Director
Indexing Support¶
TurboGears supports also automatic indexing of MongoDB fields.
If you want to guarantee that a field is unique or indexed you
just have to specify the unique_indexes or indexes variables
for the __mongometa__ attribute of the mapped class.
class Permission(MappedClass):
class __mongometa__:
session = DBSession
name = 'tg_permission'
unique_indexes = [('permission_name',),]
TurboGears will ensure indexes for your each time the application
is started, this is performed inside the init_model function.
Indexes are covered in detail in Ming Indexing Documentation.
Handling Relationships¶
Ming comes with support to one-to-many, many-to-one and many-to-many
Relations they provide an easy to use access to related objects.
The fact that this relation is read only isn’t a real issue as the
related objects will have a ForeignIdProperty which can be changed
to add or remove objects to the relation.
TurboGears comes with a bunc of Many-to-Many relations already defined
so you can see them in action in the Permission and Group classes:
class Group(MappedClass):
"""
Group definition.
"""
class __mongometa__:
session = DBSession
name = 'tg_group'
unique_indexes = [('group_name',),]
_id = FieldProperty(s.ObjectId)
group_name = FieldProperty(s.String)
display_name = FieldProperty(s.String)
permissions = RelationProperty('Permission')
class Permission(MappedClass):
"""
Permission definition.
"""
class __mongometa__:
session = DBSession
name = 'tg_permission'
unique_indexes = [('permission_name',),]
_id = FieldProperty(s.ObjectId)
permission_name = FieldProperty(s.String)
description = FieldProperty(s.String)
_groups = ForeignIdProperty(Group, uselist=True)
groups = RelationProperty(Group)
You can see the permissions and groups properties that provide
the interface to the relation and the _groups property that stores
ids of groups related to each Permission in a mongodb array.
In this case each user will have one or more groups stored with their group_name inside the Permission._groups array. Accessing Permission.groups will provide a list of the groups the user is part of.
For a complete coverage of Relationships with Ming refer to Ming Relations guide.
Custom Properties¶
There are cases when you will want to adapt a value from the database before loading and storing it. A simple example of this case is the password field, this will probably be encrypted with some kind of algorithm which has to be applied before saving the field itself.
To handle those cases TurboGears Ming allows subclassing field property to declare CustomProperties. This provides a way to hook two functions which have to be called before storing and retrieving the value to adapt it through Python Descriptors Protocol:
class PasswordProperty(FieldProperty):
@classmethod
def _hash_password(cls, password):
salt = sha256()
salt.update(os.urandom(60))
salt = salt.hexdigest()
hash = sha256()
# Make sure password is a str because we cannot hash unicode objects
hash.update((password + salt).encode('utf-8'))
hash = hash.hexdigest()
password = salt + hash
# Make sure the hashed password is a unicode object at the end of the
# process because SQLAlchemy _wants_ unicode objects for Unicode cols
password = password.decode('utf-8')
return password
def __set__(self, instance, value):
value = self._hash_password(value)
return FieldProperty.__set__(self, instance, value)
In the previous example the password property automatically hashed
each time a new value is assigned to the property. That is performed
by PasswordProperty.__set__ which calls _hash_password before
calling FieldProperty.__set__ which actually saves the password.
For additional details on working with custom properties refer to CustomProperties Ming Documentation.
ObjectDispatch and TGController¶
The TGController is the basic controller class that provides an easy method for nesting of controller classes to map URL hierarchies. There are however a few methods which provide ways to implement custom dispatching and some entry points that will make easy for the developer to track the progress of request dispatch.
Dispatching Entry Points¶
Dispatching entry points are methods that get executed while the dispatch is moving forward, this permits to run custom code which is related to the controller we are dispatching and not a specific method itself:
class Controller(BaseController):
def _before(self, *remainder, **params):
# Executed before running any method of Controller
def _after(self, *remainder, **params):
# Executed after running any method of Controller
def _visit(self, *remainder, **params):
# Executed when visiting a controller during dispatch.
_beforegets executed whenever the dispatch process- decides that the request has to be served by a method
of the controller, before calling the method itself.
It is executed before any method requirement specified
through
@requirehas been evaluated, but after the controllerallow_onlyhas been evaluated.
_aftergets executed after the request has been dispatched- to one of the controller methods.
_visitgets executed whenever the controller is visited- during the dispatch process. Actual request target might be
a subcontroller and not the controller itself.
Might be called multiple times and gets executed before
allow_onlyhas been evaluated.
The Default Method¶
The developer may decide to provied a _default method within their
controller which is called when the dispatch mechanism cannot find
an appropriate method in your controllers to call. This
_default method might look something like this:
class WikiController(BaseController):
@expose('mytgapp.wiki.new)
def _default(self, *args):
"""
Return a page to prompt the user to create a new wiki page."""
"""
return dict(new_page_slug=args)s
The Lookup Method¶
_lookup and _default are called in identical situations: when
“normal” object traversal is not able to find an exposed method, it
begins popping the stack of “not found” handlers. If the handler is a
“_default” method, it is called with the rest of the path as positional
parameters passed into the default method.
The not found handler stack can also contain “lookup” methods, which are different, as they are not actual controllers.
A lookup method takes as its argument the remaining path elements and returns an object (representing the next step in the traversal) and a (possibly modified) list of remaining path elements. So a blog might have controllers that look something like this:
class BlogController(BaseController):
@expose()
def _lookup(self, year, month, day, id, *remainder):
dt = date(int(year), int(month), int(day))
blog_entry = BlogEntryController(dt, int(id))
return blog_entry, remainder
class BlogEntryController(object):
def __init__(self, dt, id):
self.entry = model.BlogEntry.get_by(date=dt, id=id)
@expose(...)
def index(self):
...
@expose(...)
def edit(self):
...
@expose()
def update(self):
....
So a URL request to …/2007/6/28/0/edit would map first to the BlogController’s _lookup method, which would lookup the date, instantiate a new BlogEntryController object (blog_entry), and pass that blog_entry object back to the object dispatcher, which uses the remainder do continue dispatch, finding the edit method. And of course the edit method would have access to self.entry, which was looked up and saved in the object along the way.
In other situations, you might have a several-layers-deep “_lookup” chain, e.g. for editing hierarchical data (/client/1/project/2/task/3/edit).
The benefit over “_default” handlers is that you return an object that acts as a sub-controller and continue traversing rather than being a controller and stopping traversal altogether. This allows you to use actual objects with data in your controllers.
Plus, it makes RESTful URLs much easier than they were in TurboGears 1.
Subclassing Controllers¶
When overriding a parent controller method you will usually have to expose it again and place any validation or event hook it previously had.
While this is possible, it is not the best way to add additional behavior to existing controllers. If they are implemented in an external library or application, you will have to look at the code of the library, see any template it exposed, any hook it registered and place them again.
If the library will change in any future release your code will probably stop working.
To avoid this behavior and the issues it raises since TurboGears 2.2 it is possible to subclass controllers inheriting the configuration the parent methods had.
The inherit parameter of the tg.decorators.expose decorator
enables this behavior:
class OriginalController(TGController):
@expose('mylib.templates.index')
def index(self):
return dict()
@expose('mylib.templates.about')
def about(self):
return dict()
@expose('json')
def data(self):
return {'v':5}
class MyCustomizedController(OriginalController):
@expose(inherit=True)
def index(self, *args, **kw):
dosomething()
return super(MyCustomizedController, self).index(*args, **kw)
@expose('myapp.templates.newabout', inherit=True)
def about(self):
return super(MyCustomizedController, self).about(*args, **kw)
def _before_render_data(remainder, params, output):
output['child_value'] = 'CHILDVALUE'
@expose(inherit=True)
@before_render(_before_render_data)
def data(self, *args, **kw):
return super(MyCustomizedController, self).data(*args, **kw)
Mount Points and Dispatch¶
Since TurboGears 2.1.4 it is possible to ask for various informations about the request dispatchment and controllers mount points.
Those informations can be useful when writing controllers that you plan to reuse in multiple applications or mount points, making possible for example to generate all the urls knowing where they are mounted.
For statically mounted controllers the exposed informations are:
- The
mount_pointproperty of a controller. If statically mounted it will return where the controller is mounted. This is the url to call when you want to access that controller. - The
mount_stepsproperty of a controller. If statically mounted it will return the complete list of parents of that controller.
In the case you are dispatching the request yourself, for example
through a _lookup method, the mount_point and mount_steps
informations won’t be available. In this case you can rely
on some other functions exposed by TG:
- The
tg.request.controller_stateobject keeps track of all the steps provided to dispatch the request. - The
tg.dispatched_controller()method when called inside a request will return the last statically mounted controller. This can be useful to detect which controller finished the request dispatch using the_lookupmethod.
The application RootController can usually be retrieved from
tg.config['application_root_module'].RootController
Testing with TurboGears¶
TurboGears quickstart command already provides a fully working test suite for your newly quickstarted app.
Running Tests¶
The only required dependency to run the testsuite is the nose package, which
can be installed with:
$ pip install nose
Other dependencies used by your tests will automatically be installed when the
test suite is run through the install_requires and testpkgs variables in setup.py,
so if your application requires any dependency specific for testing just make sure is
listed there.
To actually run the test suite you can run:
$ python setup.py nosetests
This will install all the required dependencies and will run the tests. You should get an output similar to:
Wrong password keeps user_name in login form ... ok
Anonymous users are forced to login ... ok
Logouts must work correctly ... ok
Voluntary logins must work correctly ... ok
The data display demo works with HTML ... ok
The data display demo works with JSON ... ok
Displaying the wsgi environ works ... ok
The front page is working properly ... ok
Anonymous users must not access the secure controller ... ok
The editor cannot access the secure controller ... ok
The manager can access the secure controller ... ok
plain.tests.functional.test_root.TestWithContextController.test_i18n ... ok
Model objects can be created ... ok
Model objects can be queried ... ok
Model objects can be created ... ok
Model objects can be queried ... ok
Model objects can be created ... ok
Users should be fetcheable by their email addresses ... ok
User objects should have no permission by default. ... ok
The obj constructor must set the email right ... ok
The obj constructor must set the user name right ... ok
Model objects can be queried ... ok
----------------------------------------------------------------------
Ran 22 tests in 11.931s
OK
Those are the default tests that TurboGears generated for you.
Tests Collection¶
When running the nosetests command nose will look for any tests
directory inside your application package.
Nose itself will look in all files whose name starts with test_[something].py for all the classes which name starts with Test[Something] and will consider them as Test Cases, for each method inside the test case whose name starts with test_[something] they will be treated as Test Units.
Writing Tests¶
When quickstarting an application you will notice that there is a tests package inside it.
This package is provided by TurboGears itself and contains the fixture already creates a TestApp
instance for you and loads configuration from test.ini instead of development.ini.
The TestApp which is available inside Test Cases as self.app is an object with methods
that emulate HTTP requests: .get, .post, .put and so on and is able to understand
both html and json responses.
Take note that test.ini actually inherits from development.ini and just overwrites some options.
For example for tests by default a sqlalchemy.url = sqlite:///:memory: is used which forces
SQLAlchemy to use an in memory database instead of a real one, so that it is created
and discarded when the test suite is run withouth requiring you to use a real database.
All your application tests that call a web page should inherit from tests.TestController
which ensure:
- For each test unit the database is created and initialized by calling setup-app.
- For each test unit the
self.appobject is provided which is aTestAppinstance of your TurboGears2 application loaded fromtest.ini.- After each test unit the database is deleted.
- After each test unit the SQLAlchemy session is cleared.
For a sample test see tests/functional/test_root:
from nose.tools import ok_
from testapp.tests import TestController
class TestRootController(TestController):
"""Tests for the method in the root controller."""
def test_index(self):
response = self.app.get('/')
msg = 'TurboGears 2 is rapid web application development toolkit '\
'designed to make your life easier.'
ok_(msg in response)
def test_environ(self):
response = self.app.get('/environ.html')
ok_('The keys in the environment are:' in response)
Simulating Authentication Requests¶
To simulate authentication you can just pass to the .get, .post and so on methods an
extra_environ parameter (which is used to add WSGI environ values available
in tg.request.environ) named REMOTE_USER.
For example if you want to behave like you are logged as the editor user you just pass:
def test_secc_with_editor(self):
environ = {'REMOTE_USER': 'editor'}
self.app.get('/secc', extra_environ=environ, status=403)
The previous test will check that when user is logged as editor a 403 error is returned
from the /secc url instead of the 401 which is returned when user is not logged at all.
Checking HTML Responses¶
Within the tests it is also possible to check complex HTML structures if the pyquery module
is installed.
To install pyquery just add it to your testpkgs in setup.py so that it will be
automatically installed when running the test suite.
PyQuery is a python module that works like jQuery and permits easy traversing of the DOM:
from testapp.tests import TestController
class TestHelloWorldApp(TestController):
"""Tests an app that returns a simple HTML response with:
<html>
<head>
<title>Hello to You</title>
</head>
<body>
<h1>Hello World</h1>
</body>
</html>
"""
def test_hello_world(self):
res = self.app.get('/')
assert res.pyquery('h1').text() == 'Hello World'
def test_title(self):
res = self.app.get('/')
assert res.pyquery('title').text() == 'Hello to You'
For pyquery documentation please rely on https://pythonhosted.org/pyquery/
Submitting Forms¶
The TestApp permits also to easily fill and submit forms,
this can be used to test features that require submission of form values:
from testapp.tests import TestController
class TestFormApp(TestController):
"""Tests an app that contains a simple HTML form with:
<form id="form1" action="/submit" method="POST">
<input type="text" name="value"/>
</form>
That submits to:
@expose('json')
def submit(self, value=None, **kwargs):
return dict(value=value)
"""
def test_form_submission(self):
page = self.app.get('/')
form = page.forms['form1']
form['value'] = 'prova'
res = form.submit()
assert res.json['value'] == 'prova', res
The form itself is identified by its id, so the page.forms['form1'] works as
the form has id="form1".
Testing Outside Controllers¶
There might be cases when you are required to test something outside a controller, this is common with validators or utility methods.
In those cases you can inherit from tests.TestController as usual, and you
will probably won’t use the self.app object. Unless you are required to have a
request in place during your test.
This might be the case if your utility function or class uses TurboGears features that
depend on a request like tg.url, tg.i18n.ugettext and so on…
Since version 2.3.6 the test_context context is available, when used
together with a with statement, the whole body of the with will run with
a fake TurboGears context, much like the one you get when using /_test_vars:
from tg.util.webtest import test_context
with test_context(self.app):
hello = ugettext('hello')
assert hello == 'hello', hello
On 2.3.5 the same behaviour could be achieved using the special /_test_vars
url which initializes a fake TurboGears context which will be used until removed:
from testapp.tests import TestController
class TestWithContextController(TestController):
def test_i18n(self):
self.app.get('/_test_vars') # Initialize a fake request
hello = ugettext('hello')
assert hello == 'hello', hello
Make sure you reset the request context after using /_test_vars otherwise
you might end up with a messy environment as you have left behind the globally
registered objects. It is a good practice to perform another another request to
properly reset the global object status at the end of the test unit:
from testapp.tests import TestController
class TestWithContextController(TestController):
def tearDown(self):
self.app.get('/_del_test_vars', status=404) # Reset fake request
super(TestWithContextController, self).tearDown()
def test_i18n(self):
self.app.get('/_test_vars') # Initialize a fake request
hello = ugettext('hello')
assert hello == 'hello', hello
Coverage¶
Coverage is the process of identifying all the paths of execution that the Test Suite is not checking. Aiming at 100% code coverage means that we are sure that our tests pass through all branches in our code and all the code we wrote has been run at least once.
Note that Coverage is able to guarantee that we checked everything we wrote, it is not able to measure code that we should have written but didn’t. Missing errors handling won’t be detected in coverage but it is a fairly reliable tool to ensure that everything your wrote has been checked at least once.
By default coverage reporting is disabled in turbogears test suite, but it can easily be turned
on by changing with-coverage option in setup.cfg:
[nosetests]
verbosity = 2
detailed-errors = 1
with-coverage = true # CHANGED TO true TO ENABLE COVERAGE
cover-erase = true
cover-package = plain
When coverage is enabled, after the tests results, you will get the coverage report:
..
Name Stmts Miss Cover Missing
---------------------------------------
_opnums 13 8 38% 2-3, 6-9, 12, 16-17
----------------------------------------------------------------------
Ran 2 tests in 0.002s
OK
The Configuration System¶
Configuration Process¶
TurboGears 2 provides a configuration system that attempts to be both extremely flexible for power users and very simple to use for standard projects.
Overview¶
The application configuration is separated from the deployment specific information. In TurboGears 2.3.12 there is a config module, containing several configuration specific python files – these are done in python (not as INI files), because they actually setup the TurboGears 2.3.12 application and its associated WSGI middleware. Python provides an incredibly flexible config system with all kinds of tools to keep you from having to repeat yourself. But it comes with some significant drawbacks, python is more complex than INI, and is less declarative so can be less obvious.
But we believe these drawbacks are more than overcome by the power and flexibility of python based configuration for the app because these files are intended to be edited only by application developers, not by those deploying the application. We’ve also worked hard to create an environment that is generally declarative.
At the same time the deployment level configuration is done in simple .ini files, in order to make it totally declarative, and easy for deployers who may not be python programmers.
Configuration in the INI files¶
A TurboGears quickstarted project will contain a couple of .ini files which are used to define what WSGI app ought to be run, and to store end-user created configuration values, which is just another way of saying that the .ini files should contain deployment specific options.
By default TurboGears provides a development.ini, test.ini,
and production.ini files. These are standard ini file formats.
These files are standard INI files, as used by PasteDeploy. The
individual sections are marked off with []’s.
See also
Configuration file format and options are described in great detail in the Paste Deploy documentation.
If want to add some configuration option (let’s say an administrator’s
email) here is how you would do so. First you would edit your
development.ini file and go to the end of the [app:main]
section.
You can then choose a sensible name for your configuration key and add it to the section:
mail.from.administrator = someemail@somedomain.com
This would make sure this variable is now part of the configuration and can be accessed from anywhere in your code. For example let’s imagine that you wanted to get this config option from a controller’s code:
import tg
admin_emailfrom = tg.config.get('mail.from.administrator', 'notconfigured@nodomain.com')
If the person who deployed your application forgot to add the variable to his config file he would get the default value provided as the second argument of the get() call.
Warning
If you set a value like enable_subsystem = false, it will be loaded into python as the string ‘false’ which if used in a conditional will give you a very wrong result
The correct way of loading boolean values for your use is
from paste.deploy.converters import asbool
if asbool(config['enable_subsystem']):
... sub systems is enabled...
Configuration Milestones¶
Since TurboGears 2.3 the configuration process got divided in various milestones, each of those milestones is bound to an advancement in the framework setup process.
Whenever a milestone is reached all the registered callbacks are fired and the configuration process can continue. If the milestone is already passed when a callback is registered, the callback gets instantly fired.
Note
The tg.config object is available at import time but until the
configuration file is parsed, it only contains the system
defaults. If you need to perform startup time setup based on the
supplied configuration, you should do so in a milestone.
Milestones are available through the tg.configuration.milestones
module, the currently provided milestones are:
milestones.config_ready- Configuration file has been loaded and is- available in
tg.config
milestones.renderers_ready- Renderers have been registered and all- of them are available
milestones.environment_loaded- Full environment have been loaded- but application has not been created yet.
Registering an action to be executed whenever a milestone is reach
can be done using tg.configuration.milestones._ConfigMilestoneTracker.register()
method of each milestone. The registered action takes no parameters.
Milestones are much like Hooks but they are only related to the configuration process. The major difference is that while an hook can fire multiple times a milestone can be reached only once.
Milestones and Hooks order of execution¶
The order of execution of the milestones and hooks provided during the application startup process is:
milestones.config_ready- startup Hook
milestones.renderers_readymilestones.environment_loaded- before_config Hook
- after_config Hook
The config module¶
Tip
A good indicator of whether an option should be set in the
config directory code vs. the configuration file is whether or
not the option is necessary for the functioning of the
application. If the application won’t function without the
setting, it belongs in the appropriate config/ directory
file. If the option should be changed depending on deployment, it
belongs in the ini files.
Our hope is that 90% of applications don’t need to edit any of the
config module files, but for those who do, the most common file to
change is app_config.py:
from tg.configuration import AppConfig
import wiki20
from wiki20 import model
from wiki20.lib import app_globals, helpers
base_config = AppConfig()
base_config.renderers = []
base_config.package = wiki20
#Set the default renderer
base_config.default_renderer = 'kajiki'
base_config.renderers.append('kajiki')
#Configure the base SQLALchemy Setup
base_config.use_sqlalchemy = True
base_config.model = wiki20.model
base_config.DBSession = wiki20.model.DBSession
app_cfg.py exists primarily so that middleware.py and environment.py
can import and use the base_config object.
The base_config object is an AppConfig() instance which allows
you to access its attributes like a normal object, or like a standard
python dictionary.
One of the reasons for this is that AppConfig() provides some
defaults in its __init__. But equally important it provides us
with several methods that work on the config values to produce the two
functions that set up your TurboGears app.
We’ve taken care to make sure that the entire setup of the
TurboGears 2.3.12 framework is done in code which you as the
application developer control. You can easily customize it to your needs.
If the standard config options we provide don’t do what you need, you
can subclass and override AppConfig to get exactly the setup you want.
The base_config object that is created in app_cfg.py should be
used to set whatever configuration values that belong to the
application itself and are required for all instances of this app, as
distinct from the configuration values that you set in the
development.ini or production.ini files that are intended to
be editable by those who deploy the app.
As part of the app loading process the base_config object will be
merged in with the config values from the .ini file you’re using to
launch your app, and placed in tg.config.
As we mentioned previously, in addition to the attributes on the
base_config object there are a number of methods which are used to
setup the environment for your application, and to create the actual
TurboGears WSGI application, and all the middleware you need.
You can override base_config’s methods to further customize your
application’s WSGI stack, for various advanced use cases, like adding
custom middleware at arbitrary points in the WSGI pipeline, or doing
some unanticipated (by us) application environment manipulation.
And we’ll look at the details of how that all works in the advanced configuration section of this document.
Configuring your application¶
Here’s are some of the more general purpose configuration attributes:
Configuration Attributes¶
The configuration object has a number of attributes that automate the majority of what you need to do with the config object. These shortcuts eliminate the need to provide your own setup methods for configuring your TurboGears application.
To see the list of available configuration options refer to AppConfig.
Advanced Configuration¶
Sometimes you need to go beyond the basics of setting configuration options. We’ve created a number of methods that you can use to override the way that particular pieces of the TurboGears 2.3.12 stack are configured. The basic way you override the configuration within app.cfg looks something like this:
from tg.configuration import AppConfig
from tw2.core.middleware import TwMiddleware
class MyAppConfig(AppConfig):
def add_tosca2_middleware(self, app):
app = TwMiddleware(app,
default_engine=self.default_renderer,
translator=ugettext,
auto_reload_templates = False
)
return app
base_config = MyAppConfig()
# modify base_config parameters below
The above example shows how one would go about overridding the ToscaWidgets2
middleware. See the AppConfig for more ideas on how you
could modify your own custom config
Template Rendering Config Settings¶
| Status: | Official |
|---|
Table of Contents
The most common configuration change you’ll likely want to make here is to add a second template engine or change the template engine used by your project.
By default TurboGears sets up the Kajiki engine, but we also provide
out of the box support for Genshi, Mako and Jinja. To tell TG to prepare these
templating engines for you all you need to do is install the package
and append 'mako' or 'jinja' to the renderer’s list here in
app_config.
To change the default renderer to something other than Kajiki, just
set the default_renderer to the name of the rendering engine. So, to
add Mako to the list of renderers to prepare, and set it to be the
default, this is all you’d have to do:
base_config.default_renderer = 'mako'
base_config.renderers.append('mako')
Configuration Attributes¶
base_config.default_renderer – set to the name of the default
render function you want to use.
base_config.renderers – This is a list of rendering engines that
ought to be prepared for use in the app. To make it available in
your application you must specify here the name of the engine you
want to use.
TG provides built-in renderers for:
'kajiki', 'genshi', 'mako', 'jinja', 'json' and 'jsonp'.
In 2.3.2 and newer versions, If you would like to add additional renderers, you can
add it to the renderers list, and then register a rendering engine factory
through the tg.configuration.AppConfig.register_rendering_engine()
method.
base_config.use_dotted_templatenames – Generally you will not
want to change this. But if you want to use the standard
genshi/mako/jinja file system based template search paths, set this to
False. The main advantage of dotted template names is that it’s
very easy to store template files in zipped eggs, but if you’re not
using packaged TurboGears 2.3.12 app components there are some
advantages to the search path syntax.
Making a module available to all Templates¶
Sometimes you want to expose an entire module to all of the templates in your templates directory. Perhaps you have a form library you like to use, or a png-txt renderer that you want to wrap with <pre>. This is possible in TG.
First, we must modify our app_cfg.py so that you can share your link across all templates:
base_config.variable_provider = helpers.add_global_tmpl_vars
Next, you want to modify the lib/helpers.py module of your application
to include the newly added add_global_tmpl_vars method:
import mymodule
def add_global_tmpl_vars():
return dict(mymodule=mymodule)
That’s pretty much it, you should have access to mymodule in every template now.
Overriding AppConfig Rendering Methods¶
Please have a look at tg.configuration.AppConfig.register_rendering_engine()
for informations on how to setup custom rendering engines.
Identification & Authentication Layer¶
This document describes how repoze.who is integrated into TurboGears
and how you make get started with it. For more information, you may want
to check repoze.who’s website.
repoze.who is a powerful and extensible authentication package for
arbitrary WSGI applications. By default TurboGears2 configures it to log using
a form and retrieving the user information through the user_name field of the
User class. This is made possible by the authenticator plugin that TurboGears2
uses by default which asks base_config.sa_auth.authmetadata to authenticate
the user against given login and password.
How it works in TurboGears¶
The authentication layer it’s a WSGI middleware which is able to authenticate the user through the method you want (e.g., LDAP or HTTP authentication), “remember” the user in future requests and log the user out.
You can customize the interaction with the user through four kinds of plugins, sorted by the order in which they are run on each request:
- An
identifier plugin, with no action required on the user’s side, is able to tell whether it’s possible to authenticate the user (e.g., if it finds HTTP Authentication headers in the HTTP request). If so, it will extract the data required for the authentication (e.g., username and password, or a session cookie). There may be many identifiers andrepoze.whowill run each of them until one finds the required data to authenticate the user. - If at least one of the identifiers could find data necessary to authenticate
the current user, then an
authenticator pluginwill try to use the extracted data to authenticate the user. There may be many authenticators andrepoze.whowill run each of them until one authenticates the user. - When the user tries to access a protected area or the login page, a
challenger pluginwill come up to request an action from the user (e.g., enter a user name and password and then submit the form). The user’s response will start another request on the application, which should be caught by an identifier to extract the login data and then such data will be used by the authenticator. - For authenticated users, the
IdentityApplicationWrapperprovides the ability to load related data (e.g., real name, email) so that it can be easily used in the application. Such a functionality is provided by so-calledApplicationAuthMetadatain yourapp_cfg.py.
When the IdentityApplicationWrapper retrieves the user identity and its
metadata it makes them available inside request as request.identity.
For example, to check whether the user has been authenticated you may use:
# ...
from tg import request
# ...
if request.identity:
flash('You are authenticated!')
request.identity will equal to None if the user has not been
authenticated.
Also the whole repoze.who authentication information are available
in WSGI environment with repoze.who.identity key, which can be
accessed using the code below:
from tg import request
# The authenticated user's data kept by repoze.who:
who_identity = request.environ.get('repoze.who.identity')
The username will be available in identity['repoze.who.userid']
(or request.identity['repoze.who.userid'], depending on the method you
select).
The FastFormPlugin¶
By default, TurboGears 2.3.12 configures repoze.who to use
tg.configuration.auth.fastform.FastFormPlugin as the first
identifier and challenger – using /login as the relative URL that will
display the login form, /login_handler as the relative URL where the
form will be sent and /logout_handler as the relative URL where the
user will be logged out. The so-called rememberer of such identifier will
be an instance of repoze.who.plugins.cookie.AuthTktCookiePlugin.
All these settings can be customized through the config.app_cfg.base_config.sa_auth
options in your project. Identifiers, Authenticators and Challengers can be overridden
providing a different list for each of them as:
base_config.sa_auth['identifiers'] = [('myidentifier', myidentifier)]
You don’t have to use repoze.who directly either, unless you decide not
to use it the way TurboGears configures it.
Customizing authentication and authorization¶
It’s very easy for you to customize authentication and identification settings
in repoze.who from {yourproject}.config.app_cfg.base_config.sa_auth.
Customizing how user information, groups and permissions are retrieved¶
TurboGears provides an easy shortcut to customize how your authorization
data is retrieved without having to face the complexity of the underlying
authentication layer. This is performed by the TGAuthMetadata object
which is configured in your project config.app_cfg.base_config.
This object provides three methods which have to return respectively the user, its groups and its permissions. You can freely change them as you wish as they are part of your own application behavior.
Advanced Customizations¶
For more advanced customizations or to use repoze plugins to implement
different forms of authentication you can freely customize the whole
authentication layer using through the {yourproject}.config.app_cfg.base_config.sa_auth
options.
The available directives are all optional:
form_plugin: This is a replacement for the FriendlyForm plugin and will be- always used as a challenger. If
form_identifiesoption is True it will also be appended to the list of identifiers.
ìdentifiers: A custom list ofrepoze.whoidentifiers.- By default it contains the
form_pluginand theAuthTktCookiePlugin.
challengers: A custom list ofrepoze.whochallengers.- The
form_pluginis always appended to this list, so if you have only one challenger you will want to change theform_plugininstead of overridding this list.
authmetadata: This is the object that TG will use to fetch authorization metadata.- Changing the authmetadata object you will be able to change how TurboGears fetches your user data, groups and permissions.
mdproviders: This is a list ofrepoze.whometadata providers.- Those usually to the same work that
authmetadatadoes and in case arepoze.whometadata provider already provided identity metadata it will be available insideidentityinauthmetadataand can be used.
Customizing the model structure assumed by the quickstart¶
Your auth-related model doesn’t have to be like the default one, where the
class for your users, groups and permissions are, respectively, User,
Group and Permission, and your users’ user name is available in
User.user_name. What if you prefer Member and Team instead of
User and Group, respectively?
First of all we need to inform the authentication layer that our user is stored
in a different class. This makes repoze.who know where to look for the user
to check its password:
# what is the class you want to use to search for users in the database
base_config.sa_auth.user_class = model.Member
Then we have to tell out authmetadata how to retrieve the user, its groups
and permissions:
from tg.configuration.auth import TGAuthMetadata
#This tells to TurboGears how to retrieve the data for your user
class ApplicationAuthMetadata(TGAuthMetadata):
def __init__(self, sa_auth):
self.sa_auth = sa_auth
def authenticate(self, environ, identity):
user = self.sa_auth.dbsession.query(self.sa_auth.user_class).filter_by(user_name=identity['login']).first()
if user and user.validate_password(identity['password']):
return identity['login']
def get_user(self, identity, userid):
return self.sa_auth.user_class.query.get(user_name=userid)
def get_groups(self, identity, userid):
return [team.team_name for team in identity['user'].teams]
def get_permissions(self, identity, userid):
return [p.permission_name for p in identity['user'].permissions]
base_config.sa_auth.authmetadata = ApplicationAuthMetadata(base_config.sa_auth)
Now our application is able to fetch the user from the Member table and
its groups from the Team table. Using TGAuthMetadata makes also possible
to introduce a caching layer to avoid performing too many queries to fetch
the authentication data for each request.
SimpleToken Example¶
The following is an example of a customization of the authentication stack to allow identification of the user through a token provided through a custom header.
This example is not secure and the token is simply the username itself, it’s simply intended to showcase how to implement your own identification, never use this in production.
Identifying User¶
We will be identifying the user through the value provided in X-LogMeIn header.
This can be done by registering in TurboGears an object with identify, remember
and forget methods.
The identify method is the one we are looking to catch the token value
and return an identity that TGAuthMetadata can use to authenticate our user.
remember and forget methods are intended when the server can also drive
the fact that the values requred to identify the user must be provided on subsequent
requests or not (IE: set or remove cookies). In this case we are not concerned
as we expect the client to explicitly provide the token for each request:
class SimpleTokenIdentifier(object):
def identify(self, environ):
logmein_header = environ.get('HTTP_X_LOGMEIN')
if logmein_header:
return {'login': logmein_header, 'password': None, 'identifier': 'simpletoken'}
def forget(self, environ, identity):
return None
def remember(self, environ, identity):
return None
Then our SimpleTokenIdentifier must be registered in identifiers list of
simple authentication options to allow its usaged:
base_config.sa_auth.identifiers = [('simpletoken', SimpleTokenIdentifier()), ('default', None)]
We also keep the ('default', None) entry to have TurboGears configure cookie based
identification for us, such that we can continue to login through the usual username and
password form.
Authenticating User¶
Once we have an identity for the user it’s authenticators job to ensure that identity
is valid. This means that the identity will be passed to TGAuthMetadata for
authentication.
Note
It’s required that your identity has a password field even though it doesn’t
have a password. Or it will be discarded and won’t be passed to TGAuthMetadata.
We need to modify TGAuthMetadata.authenticate a little to allow identities
that do not provide a valid password but has been identified by SimpleTokenIdentifier.
We can do this by adding a specific check before the one for password:
def authenticate(self, environ, identity):
login = identity['login']
user = self.sa_auth.dbsession.query(self.sa_auth.user_class).filter_by(
user_name=login
).first()
if not user:
login = None
elif identity.get('identifier') in ('simpletoken', ):
# User exists and was identified by simpletoken, skip password validation
pass
elif not user.validate_password(identity['password']):
login = None
# ... rest of method here ...
Now you can try sending requests with X-LogMeIn: manager header and you should
be able to get recognised as the site manager.
BasicAuth Example¶
The following is an example of an advanced authentication stack customization to use browser basic authentication instead of form based authentication.
Declaring a Custom Authentication Backend¶
First required step is to declare that we are going to use a custom authentication backend:
base_config.auth_backend = 'htpasswd'
When this is valued to ming or sqlalchemy TurboGears will configure
a default authentication stack based on users stored on the according database,
if auth_backend is None the whole stack will be disabled.
Then we must remove all the simple authentication options, deleting all the
basic_config.sa_auth from app_cfg.py is usually enough. Leaving
unexpected options behind (options our authentication stack doesn’t use)
might lead to a crash on application startup.
Using HTPasswd file for users¶
Next step is storing our users inside an htpasswd file,
this can be achieved by using the HTPasswdPlugin authenticator:
from repoze.who.plugins.htpasswd import HTPasswdPlugin, plain_check
base_config.sa_auth.authenticators = [('htpasswd', HTPasswdPlugin('./htpasswd', plain_check))]
This will make TurboGears load users from an htpasswd file inside the directory
we are starting the application from. The plain_check function is the
one used to decode password stored inside the htpasswd file. In this case
passwords are expected to be in plain text in the form:
manager:managepass
Challenging and Identifying users with BasicAuth¶
Now that we are correctly able to authenticate users from an htpasswd file, we need to use BasicAuth for identifying returning users:
from repoze.who.plugins.basicauth import BasicAuthPlugin
base_auth = BasicAuthPlugin('MyTGApp')
base_config.sa_auth.identifiers = [('basicauth', base_auth)]
This will correctly identify users that are already logged using BasicAuth, but we are still sending users to login form to perform the actual login.
As BasicAuth requires the login to be performed through the browser we must disable the login form and set the basic auth plugin as a challenger:
# Disable the login form, it won't work anyway as the credentials
# for basic auth must be provided through the browser itself
base_config.sa_auth.form_identifies = False
# Use BasicAuth plugin to ask user for credentials, this will replace
# the whole login form.
base_config.sa_auth.challengers = [('basicauth', base_auth)]
Providing User Data¶
The previous steps are focused on providing a working authentication layer,
but we will need to also identify the authenticated user so that
also request.identity and the authorization layer can work as
expected.
This is achieved through the authmetadata option, which tells
TurboGears how to retrieve the user and it’s informations. In this
case as we don’t have a database of users we will just provide a
simple user with only display_name and user_name so that
most things can work. For manager user we will also provide the
managers group so that user can access the TurboGears admin:
from tg.configuration.auth import TGAuthMetadata
class ApplicationAuthMetadata(TGAuthMetadata):
def __init__(self, sa_auth):
self.sa_auth = sa_auth
def get_user(self, identity, userid):
# As we use htpasswd for authentication
# we cannot lookup the user in a database,
# so just return a fake user object
from tg.util import Bunch
return Bunch(display_name=userid, user_name=userid)
def get_groups(self, identity, userid):
# If the user is manager we give him the
# managers group, otherwise no groups
if userid == 'manager':
return ['managers']
else:
return []
def get_permissions(self, identity, userid):
return []
base_config.sa_auth.authmetadata = ApplicationAuthMetadata(base_config.sa_auth)
Removing Login Form¶
As the whole authentication is now performed through BasicAuth the login form is now unused, so probably want to remove the login form related urls which are now unused:
- /login
- /post_login
- /post_logout
Disabling authentication and authorization¶
If you need more flexibility than that provided by the quickstart, or you are
not going to use repoze.who, you should prevent TurboGears from dealing
with authentication/authorization by removing (or commenting) the following
line from {yourproject}.config.app_cfg:
base_config.auth_backend = '{whatever you find here}'
Then you may also want to delete those settings like base_config.sa_auth.*
– they’ll be ignored.
Hooks and Wrappers¶
TurboGears defines three ways to plug behaviors inside existing applications and plugins: Hooks, Controller Wrappers and Application Wrappers.
Hooks work in an event registration and notification manner, they permit to emit events and notify registered listeners which are able to perform actions depending on the event itself.
Controller Wrappers sits between TurboGears and the controller body code, they permit to extend controllers code much like a decorator, but can be attached to third party controllers or application wide to any controller.
Application Wrappers are much like WSGI middlewares but behave and work in the TurboGears context, so they receive the TurboGears context and Request instead of the WSGI environ and are expected to return a webob.Response object back.
Hooks¶
TurboGears allows you to attach callables to a wide set of events. Most of those are available as both controller events and system wide events.
To register a system wide even you can use the register method
of the tg.hooks object. As some hooks require being registered
before the application is running, it’s common practice to
register them in your app_cfg.py file:
def on_startup():
print 'hello, startup world'
def on_shutdown():
print 'hello, shutdown world'
def before_render(remainder, params, output):
print 'system wide before render'
# ... (base_config init code)
tg.hooks.register('startup', on_startup)
tg.hooks.register('shutdown', on_shutdown)
tg.hooks.register('before_render', before_render)
To register controller based hooks you can use the event decorators:
from tg.decorators import before_render
def before_render_cb(remainder, params, output):
print 'Going to render', output
class MyController(TGController):
@expose()
@before_render(before_render_cb)
def index(self, *args, **kw):
return dict(page='index')
Or register them explicitly (useful when registering hooks on third party controllers):
tg.hooks.register('before_render', before_render_cb, controller=MyController.index)
See tg.configuration.hooks.HooksNamespace.register() for more details on registering
hooks.
Apart from Hooks TurboGears also provide some Configuration Milestones you might want to have a look at to check whenever it is more proper to register an action for a configuration milestone or for an hook.
Available Hooks¶
initialized_config(configurator, config)- new configuration got loaded by the application configurator, application not yet created.startup()- application wide only, called when the application is going to startshutdown()- application wide only, called when the application exitsconfigure_new_app(app)- new application got created by the application configurator.- This is the only call that can guarantee to receive the TGApp instance before any middleware wrapping.
before_config(app) -> app- application wide only, called right after creating application,- but before setting up most of the options and middleware. Must return the application itself. Can be used to wrap the application into middlewares that have to be executed having the full TG stack available.
after_config(app) -> app- application wide only, called after finishing setting everything up.- Must return the application iself. Can be used to wrap the application into middleware that have to be executed before the TG ones. Can also be used to modify the Application by mounting additional subcontrollers inside the RootController.
before_validate(remainder, params)- Called before performing validationbefore_call(remainder, params)- Called after valdation, before calling the actual controller methodbefore_render(remainder, params, output)- Called before rendering a controller template,outputis the controller return valueafter_render(response)- Called after finishing rendering a controller template
Notifying Custom Hooks¶
Custom hooks can be notified using tg.hooks.notify, listeners can register
for any hook name, so it as simple as notifying your own hook and documenting
them in your library documentation to make possible for other developers to listen
for them:
tg.hooks.notify('custom_global_hook')
See tg.configuration.hooks.HooksNamespace.notify() for more details.
Controller Wrappers¶
Controller wrappers behave much like decorators, they sit between the controller code and TurboGears. Whenever turbogears has to call that controller it will process all the registered controller wrappers which are able to forward the request to the next in chain or just directly return an alternative value from the controller.
Registering a controller wrapper can be done using AppConfig.register_controller_wrapper.
It is possible to register a controller wrapper for a specific controller or
for the whole application, when registered to the whole application they will be
applied to every controller of the application or third party libraries:
def controller_wrapper(next_caller):
def call(*args, **kw):
try:
print 'Before handler!'
return next_caller(*args, **kw)
finally:
print 'After Handler!'
return call
base_config.register_controller_wrapper(controller_wrapper)
Due to the registration performance cost, controller wrappers can only be registered before the application started.
See AppConfig.register_controller_wrapper() for more details.
Application Wrappers¶
Application wrappers are like WSGI middlewares but are executed in the context of TurboGears and work with abstractions like Request and Respone objects.
Application wrappers are callables built by passing the next handler in chain and the current TurboGears configuration.
They are usually subclasses of ApplicationWrapper
which provides the expected interface.
Every wrapper, when called, is expected to accept
the WSGI environment and a TurboGears context as parameters
and are expected to return a tg.request_local.Response
instance:
from tg.appwrappers.base import ApplicationWrapper
class AppWrapper(ApplicationWrapper):
def __init__(self, handler, config):
super(AppWrapper, self).__init__(handler, config)
def __call__(self, controller, environ, context):
print 'Going to run %s' % context.request.path
return self.next_handler(controller, environ, context)
Application wrappers can be registered from you application
configuration object in app_cfg.py:
base_config.register_wrapper(AppWrapper)
When registering a wrapper, it is also possible to specify after which other wrapper it has to run if available:
base_config.register_wrapper(AppWrapper, after=OtherWrapper)
Wrappers registered with after=False will run before any
other available wrapper (in order of registration):
base_config.register_wrapper(AppWrapper, after=False)
See AppConfig.register_wrapper() for more details.
TurboGears2 CookBook¶
The CookBook is a collection of documentation and patterns common to TurboGears2
JSON and JSONP Rendering¶
JSON Renderer¶
TurboGears always provided builtin support for JSON Rendering, this is provided by the
JSONRenderer and the json.encode() function.
The first is what empowers the @expose('json') feature while the second is an
utility function you can call whenever encoding to json is needed. Both rely on
on tg.jsonify.JSONEncoder which is able to handle more types than the standard
one provided by the python json module and can be extended to support more types.
Using it is as simple as:
@expose('json')
def jp(self, **kwargs):
return dict(hello='World')
Which, when calling /jp would result in:
{"hello": "World"}
Customizing JSON Encoder¶
While you can create your own encoder, turbogears has a default instance of JSONEncoder
which is used for all encoding performed by the framework itself. Behavior of this encoder
can be driven by providing a __json__ method inside objects for which you want to
customize encoding and can be configured using AppConfig which supports the
following options:
json.isodates-> Whenever to encode dates in ISO8601 or not, the default isFalsejson.custom_encoders-> Dictionary oftype: functionmappings which can specify custom encoders for specific types. Custom encoders are functions that are called to get a basic object the json encoder knows how to handle.
For example to configure a custom encoder for dates your project app_cfg.py would look
like:
from datetime import date
def dmy_encoded_date(d):
return d.strftime('%d/%m/%Y')
base_config['json.custom_encoders'] = {date: dmy_encoded_date}
That would cause all datetime.date instances to be encoded using dmy_encode_date function.
If the encoded object provides a __json__ method this is considered the custom encoder
for the object itself and it is called to get a basic type the json encoder knows how to handle
(usually a dict).
Note
json.custom_encoders take precedence over __json__, this is made so that
users can override behavior for third party objects that already provide a __json__
method.
Per method customization¶
The same options available inside the json. configuration namespace are available
as render_params for the expose decorator. So if you want to turn
on/off iso formatted dates for a single method you can do that using:
from datetime import datetime
@expose('json', render_params=dict(isodates=True))
def now(self, **kwargs):
return dict(now=datetime.utcnow())
JSONP Renderer¶
Since version 2.3.2 TurboGears provides built-in support for JSONP rendering.
JSONP works much like JSON output, but instead of providing JSON response it provides
an application/javascript response with a call to a javascript function providing
all the values returned by the controller as function arguments.
To enable JSONP rendering you must first append it to the list of required engines
inside your application config/app_cfg.py:
base_config.renderers.append('jsonp')
Then you can declare a JSONP controller by exposing it as:
@expose('jsonp')
def jp(self, **kwargs):
return dict(hello='World')
When accessing /jp?callback=callme you should see:
callme({"hello": "World"});
If you omit the callback parameter an error will be returned as
it is required to know the callback name when using JSONP.
Custom callback parameter¶
By default TurboGears will expect the callback name to be provided
in a callback parameter. This parameter has to be accepted by your
controller (otherwise you can use **kwargs like the previous examples).
If you need to use a different name for the callback parameter just provide
it in the render_params of your exposition:
@expose('jsonp', render_params={'callback_param': 'call'})
def jp(self, **kwargs):
return dict(hello='World')
Then instead of opening /jp?callback=callme to get the JSONP response
you will need to open /jp?call=callme as stated by the callback_param
option provided in the render_params.
Exposing both JSON and JSONP¶
If you want to expose a controller as both JSON and JSONP, just provide both expositions. You can then use TurboGears request extensions support to choose which response you need:
@expose('json')
@expose('jsonp')
def jp(self, **kwargs):
return dict(hello='World')
To get the JSON response simply open /jp.json while to get the
JSONP response go to /jp.js?callback=callme. If no extension is provided
the first exposition will be returned (in this case JSON).
DataGrid Tutorial¶
DataGrid is a quick way to present data in tabular form.
The columns to put inside the table are specified with the fields constructor argument in a list.
Each entry of the list can be an accessor (attribute name or function), a tuple (title, accessor) or a tw2.forms.datagrid.Column instance.
Preparing Application¶
This tutorial will show an addressbook with a set of people each with a name, surname and phone number. Model used will be:
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Unicode, Integer, DateTime
class Person(DeclarativeBase):
__tablename__ = 'person'
uid = Column(Integer, autoincrement=True, primary_key=True)
name = Column(Unicode(255))
surname = Column(Unicode(255))
phone = Column(Unicode(64))
and we will populate with this data:
for i in [['John', 'Doe', '3413122314'],
['Lucas', 'Darkstone', '378321322'],
['Dorian', 'Gray', '31337433'],
['Whatever', 'Person', '3294432321'],
['Aaliyah', 'Byron', '676763432'],
['Caesar', 'Ezra', '9943243243'],
['Fahd', 'Gwyneth', '322313232'],
['Last', 'Guy', '23132321']]:
DBSession.add(Person(name=i[0], surname=i[1], phone=i[2]))
Basic DataGrid¶
With a model and some data set up, we can now start declaring our DataGrid and the fields it has to show:
from tw2.forms import DataGrid
addressbook_grid = DataGrid(fields=[
('Name', 'name'),
('Surname', 'surname'),
('Phone', 'phone')
])
After declaring the grid itself we will need to fetch the data to show inside the grid from our controller. For this example we will do it inside the RootController.index method:
@expose('dgridt.templates.index')
def index(self):
data = DBSession.query(Person)
return dict(page='index', grid=addressbook_grid, data=data)
Now the grid can be displayed in the template like this:
Template code necessary to show the grid in templates/index.xhtml:
<div>${grid.display(value=data)}</div>
Paginating DataGrid¶
Now that the grid can be displayed next probable improvement would be to paginate it. Displaying 10 results is fine, but when results start to grow it might cause performance problems and make results harder to view.
The same things explained in the Pagination in TurboGears tutorial apply here. First of all it is needed to adapt the controller method to support pagination:
from tg.decorators import paginate
@expose('dgridt.templates.index')
@paginate("data", items_per_page=3)
def index(self):
data = DBSession.query(Person)
return dict(page='index', grid=addressbook_grid, data=data)
If you run the application now you will see only 3 results as they get paginated three by three and we are still missing a way to change page. What is needed now is a way to switch pages and this can be easilly done as the paginate decorator adds to the template context a paginators variable where all the paginators currently available are gathered. Rendering the “data” paginator somewhere inside the template is simply enough to have a working pagination for our datagrid.
Template in templates/index.xhtml would become:
<div>${grid.display(value=data)}</div>
<div>${tmpl_context.paginators.data.pager()}</div>
Now the page should render with both the datagrid and the pages under the grid itself, making possible to switch between the pages.
Sorting Columns¶
DataGrid itself does not provide a way to implement columns sorting, but it can be easilly achieved by inheriting
from tw2.forms.datagrid.Column to add a link that can provide sorting.
First of all we need to declare or SortableColumn class that will return the link with the sorting request as the title for our DataGrid:
from sqlalchemy import asc, desc
from tw2.forms.datagrid import Column
from markupsafe import Markup
class SortableColumn(Column):
def __init__(self, title, name):
super(SortableColumn, self).__init__(name)
self._title_ = title
def set_title(self, title):
self._title_ = title
def get_title(self):
current_ordering = request.GET.get('ordercol')
if current_ordering and current_ordering[1:] == self.name:
current_ordering = '-' if current_ordering[0] == '+' else '+'
else:
current_ordering = '+'
current_ordering += self.name
new_params = dict(request.GET)
new_params['ordercol'] = current_ordering
new_url = url(request.path_url, params=new_params)
return Markup('<a href="%(page_url)s">%(title)s</a>' % dict(page_url=new_url, title=self._title_))
title = property(get_title, set_title)
It is also needed to tell to the DataGrid that it has to use the SortableColumn for its fields:
addressbook_grid = DataGrid(fields=[
SortableColumn('Name', 'name'),
SortableColumn('Surname', 'surname'),
SortableColumn('Phone', 'phone')
])
Now if we reload the page we should see the clickable links inside the headers of the table, but if we click one the application will crash because of an unexpected argument. We are now passing the ordercol argument to our constructor to tell it for which column we want the data to be ordered and with which ordering.
To handle the new parameter the controller must be modified to accept it and perform the ordering:
@expose('dgridt.templates.index')
@paginate("data", items_per_page=3)
def index(self, *args, **kw):
data = DBSession.query(Person)
ordering = kw.get('ordercol')
if ordering and ordering[0] == '+':
data = data.order_by(asc(ordering[1:]))
elif ordering and ordering[0] == '-':
data = data.order_by(desc(ordering[1:]))
return dict(page='index', grid=addressbook_grid, data=data)
Now the ordering should work and clicking two times on a column should invert the ordering.
Edit Column Button¶
DataGrid also permits to pass functions in the fields parameter to build the row content. This makes possible for example to add and Actions column where to put an edit button to edit the entry on the row.
To perform this it is just required to add another field with the name and the function that will return the edit link. In this example addressbook_grid would become:
addressbook_grid = DataGrid(fields=[
SortableColumn('Name', 'name'),
SortableColumn('Surname', 'surname'),
SortableColumn('Phone', 'phone'),
('Action', lambda obj: Markup('<a href="%s">Edit</a>' % url('/edit', params=dict(item_id=obj.uid))))
])
TurboGears Automatic CRUD Generation¶
Overview¶
This is a simple extension that provides a basic controller class that can be extended to meet the needs of the developer. The intention is to provide a fast path to data management by allowing the user to define forms and override the data interaction with custom manipulations once the view logic is in place. The name of this extensible class is CrudRestController.
What is CRUD?¶
CRUD is a set of functions to manipulate the data in a database: create, read, update, delete.
Um, REST?¶
REST is a methodology for mapping resource manipulation to meaningful URL. For instance if we wanted to edit a user with the ID 3, the URL might look like: /users/3/edit. For a brief discussion on REST, take a look at the microformats entry.
Before We Get Started¶
Here is the model definition we will be using for this tutorial:
from sqlalchemy import Column, Integer, String, Date, Text, ForeignKey
from sqlalchemy.orm import relation
from moviedemo.model import DeclarativeBase
class Genre(DeclarativeBase):
__tablename__ = "genres"
genre_id = Column(Integer, primary_key=True)
name = Column(String(100))
class Movie(DeclarativeBase):
__tablename__ = "movies"
movie_id = Column(Integer, primary_key=True)
title = Column(String(100), nullable=False)
description = Column(Text, nullable=True)
genre_id = Column(Integer, ForeignKey('genres.genre_id'))
genre = relation('Genre', backref='movies')
release_date = Column(Date, nullable=True)
EasyCrudRestController¶
The first thing we want to do is instantiate a EasyCrudRestController.
We import the controller from the extension, and then provide it with a
model class that it will use for its data manipulation. For this
example we will utilize the Movie class.:
from tgext.crud import EasyCrudRestController
from moviedemo.model import DBSession, Movie
class MovieController(EasyCrudRestController):
model = Movie
class RootController(BaseController):
movies = MovieController(DBSession)
That will provide a simple and working CRUD controller already configured with some simple views to list, create, edit and delete objects of type Movie.
Customizing EasyCrudRestController¶
The EasyCrudRestController provides some quick customization tools.
Having been thought to quickly prototype parts of your web applications
the EasyCrudRestController permits both to tune forms options and to
add utility methods on the fly:
class MovieController(EasyCrudRestController):
model = Movie
title = "My admin title"
__form_options__ = {
'__hide_fields__':['movie_id'],
'__field_order__':['title', 'description'],
'__field_widget_types__':{'description':TextArea}
}
__table_options__ = { # see Sprox TableBase and Sprox TableFiller
'__limit_fields__': ['title', 'desc'],
'__add_fields__': {'computed': None},
'computed': lambda filler, row: row.some_field * 2
}
The title option provides a way to customize the title displayed in the titlebar
of your browser.
The __form_options__ dictionary will permit to tune the forms configuration.
The specified options will be applied to both the form used to create new entities
and to edit the existing ones.
To have a look at the available options refer to
Sprox FormBase
The __table_options__ dictionary will permit to tune the forms configuration.
To have a look at the available options refer to
Sprox TableBase,
Sprox TableFiller,
and their parents as well.
Enabling SubString Searches¶
The CrudRestController provides ready to use search function, when opening
the controller index you will see a list of entries and a search box.
By default the search box looks for perfect matches, this is often not the case
especially if you are looking in long text entries that the user might not remember,
this behavior can be changed by using the substring_filters option.
You can enable substring searches for all the text fields by setting it to True:
class MovieController(EasyCrudRestController):
model = Movie
substring_filters = True
__table_options__ = {
'__omit_fields__':['movie_id'],
}
This will permit to search for text inside our movies title and descriptions. If you want to restrict substring searches to only some fields you can specify them explicitly:
class MovieController(EasyCrudRestController):
model = Movie
substring_filters = ['description']
__table_options__ = {
'__omit_fields__':['movie_id'],
}
Remembering Previous Values¶
The default behavior of the CrudRestController is to set fields to the submitted
value, if the user submits an empty value the object property gets emptied,
there are cases where you might prefer it to keep the previous value when an empty one
is provided. This behavior can be enabled using the remember_values option.
This is specially the case with images, you usually prefer to keep the previous image if a new one is not provided instead of deleting it at all.
Suppose we have a Photo model which has an image field using tgext.datahelpers
AttachedImage to provide an image field (pease refer to
tgext.datahelpers documentation
for more details). By default each time the user submits the edit form without specifying a
new image we would lose our previous image, to avoid this behavior and just keep our previous
image when none is specified we can use the remember_values option:
class PhotoManageController(EasyCrudRestController):
model = Photo
remember_values = ['image']
__table_options__ = {
'__omit_fields__':['uid'],
'__xml_fields__' : ['image'],
'image': lambda filler,row: Markup('<img src="%s"/>' % row.image.thumb_url) if row.image else ''
}
__form_options__ = {
'__field_widget_types__':{'image':FileField},
'__field_validator_types__' : {'image':FieldStorageUploadConverter},
'__field_widget_args__': {'image':{'label':'Photo PNG (640x280)'}},
'__hide_fields__':['uid']
}
Customizing Pagination¶
The CrudRestController provides pagination support, by default this is enabled
and provides 7 entries per page.
To tune pagination you can set the pagination set of options. To change
the number of entries displayed you can set pagination['items_per_page'].
To display 20 items per page you can for example use:
class MovieController(EasyCrudRestController):
model = Movie
pagination = {'items_per_page': 20}
To totally disable pagination just set the pagination option to False:
class MovieController(EasyCrudRestController):
model = Movie
pagination = False
Custom CrudRestController¶
The EasyCrudRestController provides a preconfigured CrudRestController
but often you will need to deeply customize it for your needs. To do that
we can start over with a clean controller and start customizing it:
from tgext.crud import CrudRestController
from moviedemo.model import DBSession, Movie
class MovieController(CrudRestController):
model = Movie
class RootController(BaseController):
movies = MovieController(DBSession)
Well that won’t actually get you anywhere, in fact, it will do nothing at all. We need to provide CrudRestController with a set of widgets and datafillers so that it knows how to handle your REST requests. First, lets get all of the Movies to display in a table.
Sprox¶
Sprox is a library that can help you to generate
forms and filler data. It utilizes metadata extracted from the
database definitions to provide things like form fields, drop downs,
and column header data for view widgets. Sprox is also customizable,
so we can go in and modify the way we want our data displayed once we
get going with it. Here we define a table widget using Sprox’s
sprox.tablebase.TableBase class for our movie table.:
from sprox.tablebase import TableBase
class MovieTable(TableBase):
__model__ = Movie
__omit_fields__ = ['genre_id']
movie_table = MovieTable(DBSession)
Filling Our Table With Data¶
So, now we have our movie_table, but it’s not going to do us much good
without data to fill it. Sprox provides a
sprox.fillerbase.TableFiller class which will retrieve the
relevant data from the database and package it in a dictionary for
consumption. This is useful if you are creating JSON. Basically,
you can provide CrudRestController with any object that has a
get_value function and it will work because of duck typing. Just make
certain that your get_value function returns the right data type for
the widget you are filling. Here is what the filler would look like
instantiated.:
from sprox.fillerbase import TableFiller
class MovieTableFiller(TableFiller):
__model__ = Movie
movie_table_filler = MovieTableFiller(DBSession)
Putting It All Together¶
Let’s modify our CrudRestController to utilize our new table. The new RootController would look like this:
from tgext.crud import CrudRestController
from moviedemo.model import DBSession, Movie
from sprox.tablebase import TableBase
from sprox.fillerbase import TableFiller
class MovieTable(TableBase):
__model__ = Movie
movie_table = MovieTable(DBSession)
class MovieTableFiller(TableFiller):
__model__ = Movie
movie_table_filler = MovieTableFiller(DBSession)
class MovieController(CrudRestController):
model = Movie
table = movie_table
table_filler = movie_table_filler
class RootController(BaseController):
movie = MovieController(DBSession)
You can now visit /movies/ and it will display a list of movies.
Forms¶
One of the nice thing about Sprox table definitions is that they
provide you with a set of RESTful links. CrudRestController provides
methods for these pages, but you must provide the widgets for the
forms. Specifically, we are talking about the edit and new forms.
Here is one way you might create a form to add a new record to the
database using sprox.formbase.AddRecordForm:
class MovieAddForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
movie_add_form = MovieAddForm(DBSession)
Adding this to your movie controller would look make it now look something like this:
class MovieController(CrudRestController):
model = Movie
table = movie_table
table_filler = movie_table_filler
new_form = movie_add_form
You can now visit /movies/new.
Edit Form¶
Now we just need to map a form to the edit function so that we can close the loop on our controller. The reason we need separate forms for Add and Edit is due to validation. Sprox will check the database for uniqueness on a “new” form. On an edit form, this is not required since we are updating, not creating.:
from sprox.formbase import EditableForm
class MovieEditForm(EditableForm):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
movie_edit_form = MovieEditForm(DBSession)
The biggest difference between this form and that of the “new” form is
that we have to get data from the database to fill in the form. Here
is how we use sprox.formbase.EditFormFiller to do that:
from sprox.fillerbase import EditFormFiller
class MovieEditFiller(EditFormFiller):
__model__ = Movie
movie_edit_filler = MovieEditFiller(DBSession)
Now it is a simple as adding our filler and form definitions to the
MovieController and close the loop on our presentation.
Declarative¶
If you are interested in brevity, the crud controller may be created in a more declarative manner like this:
from tgext.crud import CrudRestController
from sprox.tablebase import TableBase
from sprox.formbase import EditableForm, AddRecordForm
from sprox.fillerbase import TableFiller, EditFormFiller
class DeclarativeMovieController(CrudRestController):
model = Movie
class new_form_type(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
class edit_form_type(EditableForm):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
class edit_filler_type(EditFormFiller):
__model__ = Movie
class table_type(TableBase):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
class table_filler_type(TableFiller):
__model__ = Movie
Options reference¶
The tgext.crud.CrudRestController and tgext.crud.EasyCrudRestController
provide a bunch of configuration options that can be changed by subclassing
the controller and providing them in a declarative way:
-
class
tgext.crud.CrudRestController(session, menu_items=None)¶ Initialization options: - session
database session
- menu_items
Dictionary or mapping type of links to other CRUD sections. This is used to generate the links sidebar of the CRUD interface. Can be specified in the form
model_items['lower_model_name'] = ModelClassormodel_items['link'] = 'Name'.
Class attributes: - title
Title to be used for each page. default:
'Turbogears Admin System'- model
Model this class is associated with.
- remember_values
List of model attributes that need to keep previous value when not provided on submission instead of replacing the existing value with an empty one. It’s commonly used with file fields to avoid having to reupload the file again when the model is edited.
- keep_params
List of URL parameters that need to be kept around when redirecting between the various pages of the CRUD controller. Can be used to keep around filters or sorting when editing a subset of the models.
- filters
Dictionary of filters that must be applied to queries. Those are only “equals to” filters that can be used to limit objects to a subset of entries that have that value. Filters are also applied on POST method to create new object which by default have the value specified by the filter. If filter is callable it will be called to retrieve the filter value.
For example to display a crud that only shows, creates and edits entities owned by logged user through a user_id ForeignKey you can use a filter like:
{'user_id': lambda: request.identity['user'].user_id}
- search_fields
Enables searching on some fields, can be
True,Falseor a list of fields for which searching should be enabled.- substring_filters
Enable substring filtering for some fields, by default is disabled. Pass
Trueto enable it on all fields or pass a list of field names to enable it only on some fields.- json_dictify
TrueorFalse, enables advanced dictification of retrieved models when providing JSON responses. This also enables JSON encoding of related entities for the returned model.- conditional_update_field
Name of the field used to perform conditional updates when
PUTmethod is used as a REST API.Nonedisables conditional updates (which is the default).- pagination
Dictionary of options for pagination.
Falsedisables pagination. By default{'items_per_page': 7}is provided. Currently the only supported option isitems_per_page.- response_type
Limit response to a single format, can be: ‘application/json’ or ‘text/html’. By default tgext.crud will detect expected response from Accept header and will provide response content according to the expected one. If you want to avoid HTML access to a plain JSON API you can use this option to limit valid responses to application/json.
- provider_type_selector_type
Use a custom provider type selector class. By default the
sprox.providerselector.ProviderTypeSelectorclass will be used to instantiate a provider selector that can select the right provider for ming and sqlalchemy. In case you are not using ming or sqlalchemy or you just want to change the default behavior of the provider selector, you can override this.- resources
A list of CSSSource / JSSource that have to be injected inside CRUD pages when rendering. By default
tgext.crud.resources.crud_styleandtgext.crud.resources.crud_scriptare injected.- table
The
sprox.tablebase.TableBaseWidget instance used to display the table. By defaulttgext.crud.utils.SortableTableBaseis used which permits to sort table by columns.- table_filler
The
sprox.fillerbase.TableFillerinstance used to retrieve data for the table. If you want to customize how data is retrieved override theTableFiller._do_get_provider_count_and_objsmethod to return different entities and count. By defaulttgext.crud.utils.RequestLocalTableFilleris used which keeps track of the numer of entities retrieved during the current request to enable pagination.- edit_form
Form to be used for editing an existing model. By default
sprox.formbase.EditFormis used.- edit_filler
sprox.fillerbase.RecordFillersubclass used to load the values for an entity that need to be edited. Override theRecordFiller.get_valuemethod to provide custom values.- new_form
Form that defines how to create a new model. By default
sprox.formbase.AddRecordFormis used.
-
class
tgext.crud.EasyCrudRestController(session, menu_items=None)¶ A CrudRestController that provides a quick way to setup Sprox forms and Table.
Form options are available through the
__form_options__dictionary which can contain any option accepted by sproxFormBase. Options specific to NewForm and EditForm can be provided through__form_new_options__and__form_edit_options__.Table options are available through the
__table_options__dictionary which can contain any option accepted by sproxTableBase. Dictionary keys that do not start with __ will be threated asTableFillerattributes apart from__actions__which is always assigned to theTableFiller.Usually
_optionsfields will replace the previous values with theirs in case a parent class provided previous values. You can avoid this behaviour and extend the previous values usingaddopts()instead of dictionaries and lists as the option values.Class attributes: - __form_options__
Sprox options that need to be applied to both EditableForm and AddRecordForm forms
- __form_new_options__
Options that need to be applied only to AddRecordForm form
- __form_edit_options__
Options that need to be applied only to EditableForm form
- __table_options__
Options that need to be applied to TableBase and TableFiller
-
tgext.crud.addopts(*args, **kwargs)¶ Specifies options to be added to __form_options__ and __table_options__.
When options are specified using addopts like:
__form_options__ = { '__omit_fields__': addopts('name', 'surname') }
they get added to the current option value instead of replacing it.
In case of lists they always extend the list, so it’s not possible to change the order of the elements.
In case of dictionaries keys already in the option the value of the option will be merged too if it’s a dictionary or a list.
Customizing Crud Operations¶
We have really been focusing on the View portion of our controller.
This is because CrudRestController performs all of the applicable
creates, updates, and deletes on your target object for you. This
default functionality is provided by
sprox.saormprovider.SAORMProvider. This can of course be
overridden.
Overriding Crud Operations¶
CrudRestController extends RestController, which means that any methods available through RestController are also available to CRC.
| Method | Description | Example Method(s) / URL(s) |
|---|---|---|
| get_all | Display the table widget and its data | GET /movies/ |
| new | Display new_form | GET /movies/new |
| edit | Display edit_form and the containing record’s data | GET /movies/1/edit |
| post | Create a new record | POST /movies/ |
| put | Update an existing record | POST /movies/1?_method=PUT |
| PUT /movies/1 | ||
| post_delete | Delete an existing record | POST /movies/1?_method=DELETE |
| DELETE /movies/1 | ||
| get_delete | Delete Confirmation page | Get /movies/1/delete |
If you are familiar with RestController you may notice that get_one is
missing. There are plans to add this functionality in the near
future. Also, you may note the ?_method on some of the URLs. This is
basically a hack because existing browsers do not support the PUT and
DELETE methods. Just note that if you decide to incorporate a TW in
your edit_form description you must provide a
HiddenField('_method') in the definition.
Adding Functionality¶
REST provides consistency across Controller classes and makes it easy to override the functionality of a given RESTful method. For instance, you may want to get an email any time someone adds a movie. Here is what your new controller code would look like:
class MovieController(CrudRestController):
# (...)
@expose(inherit=True)
def post(self, **kw):
email_info()
return super(MovieController, self).post(**kw)
You might notice that the function has the @expose decorator. This is
required because the expose decoration occurs at the class-level, so
that means that when you override the class method, the expose is
eliminated. We add it back to the method by adding @expose with the
inherit parameter to inherit the behavior from the parent method.
For more details you can refer to the TGController Subclassing documentation.
Overriding Templates¶
To override the template for a given method, you would simple re-define that method, providing an expose to your own template, while simply returning the value of the super class’s method.:
class MovieController(CrudRestController):
# (...)
@expose('movie_demo.templates.my_get_all_template', inherit=True)
def get_all(self, *args, **kw):
return super(MovieController, self).get_all(*args, **kw)
Removing Functionality¶
You can also block-out capabilities of the RestController you do not wish implemented. Simply define the function that you want to block, but do not expose it. Here is how we “delete” the delete functionality.:
class MovieController(CrudRestController):
# (...)
def post_delete(self, *args, **kw):
"""This is not allowed."""
pass
Working with the TurboGears Admin¶
TurboGears provides the tgext.admin extension which is powered by tgext.crud
and sprox. This can be used to automatically create simple administration pages
and is the toolkit powering the /admin page in newly quickstarted applications.
By default the admin will provide autogenerated access to all the models
imported in your project models/__init__.py.
While the default configuration for the admin usually just works it doesn’t provide the best experience for users and might require some customizations.
To do so the TurboGears Admin permits to provide custom configurations throught the
AdminController config_type paramter.
The default turbogears admin is created as:
admin = AdminController(model, DBSession, config_type=TGAdminConfig)
which creates an admin for all the models with the default TurboGears admin configuration.
Restricting Access to some Models¶
Restricting access to some models is possible by specifying them explicitly
instead of passing model as the first argument to the AdminController:
from myproject.model import User, Group
admin = AdminController([User, Group], DBSession, config_type=TGAdminConfig)
Switching to a Custom Admin Configuration¶
First step to perform if you want to switch to a customized administration
configuration is to declare your own config. This can easily be done by
subclassing TGAdminConfig:
class CustomAdminConfig(TGAdminConfig):
pass
if you replace config_type=TGAdminConfig with your new CustomAdminConfig:
admin = AdminController(model, DBSession, config_type=CustomAdminConfig)
You will notice that everything works as before, as we didn’t change the configuration in any way. The custom configuration will contains properties for the admin itself and for each CRUD inside the admin.
Apart from configuration for each crud, there are some general admin configuration options:
-
class
tgext.admin.config.AdminConfig(models, translations=None)¶ Class attributes: - layout
The look and feel of the admin. Three builtin layouts are available:
tgext.admin.layouts.BasicAdminLayout,tgext.admin.layouts.BootstrapAdminLayoutandtgext.admin.layouts.GroupedBootstrapAdminLayout.Note
GroupedBootstrapAdminLayoutonly works with the Genshi and Kajiki template languages, so it cannot be used when quickstarting with Jinja or Mako.- project_name
Name of the project, by default is same as config[‘package_name’].
- index_payoff
Custom payoff to display into the admin index page.
- default_index_template
This is the template of the /admin page, by default the one specified inside the
layoutis used.- allow_only
Predicate to restrict access to the whole admin. By default
in_group('managers')is used.- include_left_menu
boolthat states if the sidebar should be included in admin pages or not. By default the sidebar is visible, please note that hiding the sidebar might break the admin layout.- DefaultControllerConfig
The default CRUD configuration for models that do not provide a custom CRUD configuration. By default
tgext.admin.config.CrudRestControllerConfig
Customizing Models CRUD¶
The admin page can be configured using the TGAdminConfig class,
supposing we have a game with running Match and a list of Settings
we can declared MatchAdminController and SettingAdminController
which inherit from EasyCrudRestController and tell TurboGears
Admin to use them for the administration of matches and settings:
class CustomAdminConfig(TGAdminConfig):
class match(CrudRestControllerConfig):
defaultCrudRestController = MatchAdminController
class setting(CrudRestControllerConfig):
defaultCrudRestController = SettingAdminController
class RootController(BaseController):
admin = AdminController([model.Match, model.Setting], DBSession,
config_type=CustomAdminConfig)
This will create an administration controller which uses our custom CrudRestControllers to manage Match and Settings instances.
Each class attribute of our CustomAdminConfig that has the lower case
name of a Model will be used to configure the admin CRUD for that model.
This can be done by subclassing CrudRestControllerConfig with
a defaultCrudRestController class attribute that points to the
CrudRestController class to use for the related model.
Defining Custom Controllers¶
The explicit way to define models configuration is by declaring
CrudRestControllers and configurations separately, but in case
the custom controllers are required only for the admin it might be
shortest to just define them together:
class MyModelAdminCrudConfig(CrudRestControllerConfig):
class defaultCrudRestController(EasyCrudRestController):
__table_options__ = {
# options here...
}
class CustomAdminConfig(TGAdminConfig):
photo = MyModelAdminCrudConfig
For a complete list of options available inside defaultCrudRestController
refer to TurboGears Automatic CRUD Generation.
Photos Admin Tutorial¶
Now suppose we have a photo model that looks like:
class Photo(DeclarativeBase):
__tablename__ = 'photos'
uid = Column(Integer, primary_key=True)
title = Column(Unicode(255), nullable=False)
image = Column(LargeBinary, nullable=False)
mimetype = Column(Unicode(64), nullable=False, default='image/jpeg')
user_id = Column(Integer, ForeignKey('tg_user.user_id'), index=True)
user = relationship('User', uselist=False,
backref=backref('photos',
cascade='all, delete-orphan'))
we might want to customize our admin to use the GroupedBootstrapAdminLayout
layout, put the photos inside the Media group and improve the table view
for the crud by removing the user_id and mimetype columns
and declare the image column as xml so that we can show the image inside:
import base64
from tgext.admin import CrudRestControllerConfig
from tgext.admin.tgadminconfig import BootstrapTGAdminConfig as TGAdminConfig
from tgext.admin.layouts import GroupedBootstrapAdminLayout
from tgext.admin.controller import AdminController as TGAdminController
from tgext.crud import EasyCrudRestController
class PhotoAdminCrudConfig(CrudRestControllerConfig):
icon_class = 'glyphicon-picture'
admin_group = 'Media'
class defaultCrudRestController(EasyCrudRestController):
__table_options__ = {
'__omit_fields__': ['user_id', 'mimetype'],
'__xml_fields__': ['image'],
'image': lambda filler, row: '<img src="data:%s;base64,%s" width="256"/>' % (
row.mimetype,
base64.b64encode(row.image).decode('ascii')
)
}
class CustomAdminConfig(TGAdminConfig):
layout = GroupedBootstrapAdminLayout
photo = PhotoAdminCrudConfig
Now our admin works as expected and uses the PhotoAdminCrudConfig to
manage photos. There is still an open issue, when uploading photos we
have to manually provide the photo user and mimetype.
To solve this issue we will customize the form hiding the two values from the form and forcing them when data is submitted:
import base64, imghdr
from tg import expose, request
from tgext.admin import CrudRestControllerConfig
from tgext.admin.tgadminconfig import BootstrapTGAdminConfig as TGAdminConfig
from tgext.admin.layouts import GroupedBootstrapAdminLayout
from tgext.admin.controller import AdminController as TGAdminController
from tgext.crud import EasyCrudRestController
class PhotoAdminCrudConfig(CrudRestControllerConfig):
icon_class = 'glyphicon-picture'
admin_group = 'Media'
class defaultCrudRestController(EasyCrudRestController):
__table_options__ = {
'__omit_fields__': ['user_id', 'mimetype'],
'__xml_fields__': ['image'],
'image': lambda filler, row: '<img src="data:%s;base64,%s" width="256"/>' % (
row.mimetype,
base64.b64encode(row.image).decode('ascii')
)
}
__form_options__ = {
'__hide_fields__': ['user', 'mimetype']
}
@expose(inherit=True)
def post(self, *args, **kw):
kw['user'] = request.identity['user'].user_id
kw['mimetype'] = 'image/%s' % imghdr.what(kw['image'].file)
return EasyCrudRestController.post(self, *args, **kw)
@expose(inherit=True)
def put(self, *args, **kw):
image = kw.get('image')
if image is not None and image.filename:
kw['mimetype'] = 'image/%s' % imghdr.what(kw['image'].file)
return EasyCrudRestController.put(self, *args, **kw)
class CustomAdminConfig(TGAdminConfig):
layout = GroupedBootstrapAdminLayout
photo = PhotoAdminCrudConfig
Rapid Prototyping REST API¶
TurboGears provides a great way to rapidly prototype rest APIS using the EasyCrudRestController.
Defining a basic ready to use API is as simple as:
from tgext.crud import EasyCrudRestController
class APIController(EasyCrudRestController):
pagination = False
model = model.Permission
class RootController(BaseController):
permissions = APIController(model.DBSession)
Accessing the /permissions.json URL you should get:
{
value_list: [
{
permission_id: 1,
description: "This permission give an administrative right to the bearer",
permission_name: "manage"
}
]
}
Blocking non JSON requests¶
The first side effect is that accessing the url without the .json
extension you will get the CRUD page. This is usually something you
don’t want when exposing an API and can be easily prevented by blocking
requests which are not in JSON format:
from tgext.crud import EasyCrudRestController
from tg import abort
class APIController(EasyCrudRestController):
pagination = False
model = model.Permission
def _before(self, *args, **kw):
if request.response_type != 'application/json':
abort(406, 'Only JSON requests are supported')
super(APIController, self)._before(*args, **kw)
Accessing the HTML page will now report a Not Acceptable error while JSON output will still be accepted.
Getting Relationships¶
You probably noticed that even though our permissions are related to groups we didn’t get the list of groups the permission is related to.
This is due to the fact that for performance reasons the EasyCrudRestController
doesn’t automatically resolve relationships when providing responses for APIs.
To enable this behavior it is sufficient to turn on the json_dictify option:
class APIController(EasyCrudRestController):
pagination = False
json_dictify = True
model = model.Permission
def _before(self, *args, **kw):
if request.response_type != 'application/json':
abort(406, 'Only JSON requests are supported')
super(APIController, self)._before(*args, **kw)
Reloading the /permissions.json page will now provide the list of groups
each Permission is related to with a response like:
{
value_list: [
{
permission_id: 1,
description: "This permission give an administrative right to the bearer",
groups: [
1
],
permission_name: "manage"
}
]
}
Leveraging your REST API on AngularJS¶
One of the main reasons to rapidly prototype a REST api is to join it with a frontend framework to perform logic and templating on client side.
The following is an example of a working AngularJS application
that permits creation and deletion of Permission objects through
the previously created API:
Note
Please note the custom $resource to adapted tgext.crud responses to the style expected by AngularJS
Note
Pay attention to the double $$ required to escape the dollar sign on Kajiki
<html py:extends="master.xhtml" py:strip="True">
<head py:block="head" py:strip="True">
<title py:block="master_title">AngularTG</title>
<script src="//ajax.googleapis.com/ajax/libs/angularjs/1.2.12/angular.min.js"></script>
<script src="//ajax.googleapis.com/ajax/libs/angularjs/1.2.12/angular-resource.min.js"></script>
<style>
.ng-cloak {
display: none !important;
}
</style>
</head>
<body py:block="body" py:strip="True">
<div class="row">
<div class="col-md-12">
<div ng-app="myApp" class="ng-cloak">
<div ng-controller="PermissionsCtrl">
<h1>Permissions</h1>
<form ng-submit="addPermission(newPerm)">
<input placeholder="Create Permission" ng-model="newPerm.permission_name" autofocus="true"/>
</form>
<div ng-repeat="perm in permissions">
<span ng-click="delPermission($$index)">X</span>
{{perm.permission_name}}
<p>{{perm.description}}</p>
</div>
</div>
</div>
</div>
</div>
<script>
//<![CDATA[
var myApp = angular.module('myApp', ['ngResource']);
myApp.factory('Permissions', ['$$resource', function($$resource) {
return $$resource("${tg.url('/permissions/:id.json')}", {'id': "@permission_id"}, {
query: {
method: 'GET',
isArray: true,
transformResponse: function (data) {
var data = angular.fromJson(data);
return data.value_list;
}
},
save: {
method: 'POST',
transformResponse: function(data) {
var data = angular.fromJson(data);
return data.value;
}
}
});
}]);
myApp.controller('PermissionsCtrl', ['$$scope', 'Permissions',
function ($$scope, Permissions) {
$$scope.permissions = Permissions.query();
$$scope.addPermission = function(permData) {
var perm = new Permissions(permData);
permData.permission_name = "";
perm.$$save(function(data) {
$$scope.permissions.push(new Permissions(data));
});
}
$$scope.delPermission = function(index) {
perm = $$scope.permissions[index];
perm.$$delete(function(data) {
$$scope.permissions.splice(index, 1);
});
}
}]);
//]]>
</script>
</body>
</html>
Limiting API Results¶
The previous API returns all the Permissions available, which is the most simple case but not always what you are looking for. It is often needed to filter the results by a constraint, for example it is common to get only the objects for a specific user.
While this can be easily achieved by passing any filter to the API itself
when it is called: /permissions.json?groups=1. It is common to need to
perform this on server side.
If we want to permanently only get the permissions for the manage groups
we can make it by extending the get_all method:
from tgext.crud import EasyCrudRestController
from tg import abort
class APIController(EasyCrudRestController):
pagination = False
json_dictify = True
model = model.Permission
def _before(self, *args, **kw):
if request.response_type != 'application/json':
abort(406, 'Only JSON requests are supported')
super(APIController, self)._before(*args, **kw)
@expose(inherit=True)
def get_all(self, *args, **kw):
kw['groups'] = 1
return super(APIController, self).get_all(*args, **kw)
If you point your browser to /permissions.json and had multiple
permissions you will se that only those for the managers group
are now reported.
Now if you tried to use the filtered controller with the previously created AngularJS application your probably noticed that the new permissions you create are not listed back when you reload the page. This is because they are actually created without a group, so they don’t match our managers group filter.
To avoid this we can also force the group on creation by extending
also the post method:
from tgext.crud import EasyCrudRestController
from tg import abort
class APIController(EasyCrudRestController):
pagination = False
json_dictify = True
model = model.Permission
def _before(self, *args, **kw):
if request.response_type != 'application/json':
abort(406, 'Only JSON requests are supported')
super(APIController, self)._before(*args, **kw)
@expose(inherit=True)
def get_all(self, *args, **kw):
kw['groups'] = 1
return super(APIController, self).get_all(*args, **kw)
@expose(inherit=True)
def post(self, *args, **kw):
kw['groups'] = [1]
return super(APIController, self).post(*args, **kw)
This will now correctly create all the new permissions for the managers group.
Custom Queries when fetching data¶
While extending the get_all method is quick and easy, you are limited
to the filtering possibilities that sprox exposes you.
For more advanced filtering or even custom queries it is possible to declare your own TableFiller with a totally custom query:
from tgext.crud import EasyCrudRestController
from sprox.fillerbase import TableFiller
from tg import abort
class APIController(EasyCrudRestController):
pagination = False
json_dictify = True
model = model.Permission
class table_filler_type(TableFiller):
__entity__ = model.Permission
def _do_get_provider_count_and_objs(self, **kw):
manager_group = model.DBSession.query(model.Group).filter_by(group_name='managers').first()
results = model.DBSession.query(model.Permission).filter(model.Permission.groups.contains(manager_group)).all()
return len(results), results
def _before(self, *args, **kw):
if request.response_type != 'application/json':
abort(406, 'Only JSON requests are supported')
super(APIController, self)._before(*args, **kw)
SQLAlchemy Master Slave Load Balancing¶
Since version 2.2 TurboGears has basic support for Master/Slave load balancing and provides a set of utilities to use it.
TurboGears permits to declare a master server and any number of slave servers, all the writes will automatically redirected to the master node, while the other calls will be dispatched randomly to the slave nodes.
All the queries executed outside of TurboGears controllers will run only on the master node, those include the queries performed by the authentication stack to initially look up an already logged in user, its groups and permissions.
Enabling Master Slave Balancing¶
To enable Master Slave load Balancing you just need to edit your model/__init__.py
making the sessionmaker use the TurboGears BalancedSession:
from tg.configuration.sqla.balanced_session import BalancedSession
maker = sessionmaker(autoflush=True, autocommit=False,
class_=BalancedSession,
extension=ZopeTransactionExtension())
Doing this by itself will suffice to make load balancing work, but still
as there is only the standard database configuration the BalancedSession
will just be redirecting all the queries to the only available serve.
Configuring Balanced Nodes¶
To let load balancing work we must specify at least a master and slave server inside our application configuration. The master server can be specified using the sqlalchemy.master set of options, while any number of slaves can be configured using the sqlalchemy.slaves options:
sqlalchemy.master.url = mysql://username:password@masterhost:port/databasename
sqlalchemy.master.pool_recycle = 3600
sqlalchemy.slaves.slave1.url = mysql://username:password@slavehost:port/databasename
sqlalchemy.slaves.slave1.pool_recycle = 3600
The master node can be configured also to be a slave, this is usually the case when we want the master to also handle some read queries.
Driving the balancer¶
TurboGears provides a set of utilities to let you change the default behavior of the load balancer. Those include the @with_engine(engine_name) decorator and the DBSession().using_engine(engine_name) context.
The with_engine decorator¶
The with_engine decorator permits to force a controller method to
run on a specific node. It is a great tool for ensuring that some
actions take place on the master node, like controllers that edit
content.
from tg import with_engine
@expose('myproj.templates.about')
@with_engine('master')
def about(self):
DBSession.query(model.User).all()
return dict(page='about')
The previous query will be executed on the master node, if the @with_engine decorator is removed it will get execute on any random slave.
The with_engine decorator can also be used to force turbogears
to use the master node when some parameters are passed by url:
@expose('myproj.templates.index')
@with_engine(master_params=['m'])
def index(self):
DBSession.query(model.User).all()
return dict(page='index')
In this case calling http://localhost:8080/index will result in queries performed on a slave node, while calling http://localhost:8080/index?m=1 will force the queries to be executed on the master node.
Pay attention that the m=1 parameter can actually have any value, it just has to be there. This is especially useful when redirecting after an action that just created a new item to a page that has to show the new item. Using a parameter specified in master_params we can force TurboGears to fetch the items from the master node so to avoid odd results due to data propagation delay.
Keeping master_params around¶
By default parameters specified in with_engine master_params will be
popped from the controller params. This is to avoid messing with validators
or controller code that doesn’t expect the parameter to exist.
If the controller actually needs to access the parameter a dictionary can be passed to @with_engine instead of a list. The dictionary keys will be the parameters, while the value will be if to pop it from the parameters or not.
@expose('myproj.templates.index')
@with_engine(master_params={'m':False})
def index(self, m=None):
DBSession.query(model.User).all()
return dict(page='index', m=m)
Forcing Single Queries on a node¶
Single queries can be forced to execute on a specific node using the
using_engine method of the BalancedSession. This method
returns a context manager, until queries are executed inside this
context they are run on the constrained engine:
with DBSession().using_engine('master'):
DBSession.query(model.User).all()
DBSession.query(model.Permission).all()
DBSession.query(model.Group).all()
In the previous example the Users and the Permissions will be fetched from the master node, while the Groups will be fetched from a random slave node.
Debugging Balancing¶
Setting the root logger of your application to DEBUG will let
you see which node has been choose by the BalancedSession
to perform a specific query.
Deploying TurboGears¶
This section describes standard deployment technics for TurboGears2.
Running TurboGears under Apache with mod_wsgi¶
mod_wsgi is an Apache module developed by Graham Dumpleton.
It allows WSGI programs to be served using the Apache web
server.
This guide will outline broad steps that can be used to get a TurboGears
application running under Apache via mod_wsgi.
The tutorial assumes you have Apache already installed on your system. If you do not, install Apache 2.X for your platform in whatever manner makes sense.
Once you have Apache installed, install
mod_wsgi. Use the (excellent) installation instructions for your platform into your system’s Apache installation.Create a virtualenvironment with the specific TurboGears version your application depends on installed.
$ virtualenv /var/tg2env $ /var/tg2env/bin/pip install tg.devtools
Activate the virtualenvironment
$ source /var/tg2env/bin/activate (tg2env)$ #virtualenv now activated
Install your TurboGears application.
(tg2env)$ cd /var/www/myapp (tg2env)$ python setup.py develop
Within the application director, create a script named
app.wsgi. Give it these contents:APP_CONFIG = "/var/www/myapp/myapp/production.ini" #Setup logging import logging.config logging.config.fileConfig(APP_CONFIG) #Load the application from paste.deploy import loadapp application = loadapp('config:%s' % APP_CONFIG)
Edit your Apache configuration and add some stuff.
<VirtualHost *:80> ServerName www.site1.com WSGIProcessGroup www.site1.com WSGIDaemonProcess www.site1.com user=www-data group=www-data threads=4 python-path=/var/tg2env/lib/python2.7/site-packages WSGIScriptAlias / /var/www/myapp/app.wsgi #Serve static files directly without TurboGears Alias /images /var/www/myapp/myapp/public/images Alias /css /var/www/myapp/myapp/public/css Alias /js /var/www/myapp/myapp/public/js CustomLog logs/www.site1.com-access_log common ErrorLog logs/www.site1.com-error_log </VirtualHost>
Restart Apache
$ sudo apache2ctl restart
Visit
http://www.site1.com/in a browser to access the application.
See the mod_wsgi configuration documentation for more in-depth configuration information.
Running TurboGears under Circus and Chaussette¶
Circus is a process & socket manager.
It can be used to monitor and control processes and sockets, when paired
with the Chaussette WSGI server it can became a powerful tool to
deploy your application and manage any related process your applications needs.
Circus can take care of starting your memcached, redis, database server or batch process with your application itself providing a single point where to configure the full application environment.
This guide will outline broad steps that can be used to get a TurboGears
application running under Chaussette through Circus.
The tutorial assumes you have
Circusalready installed on your system. If you do not, install it in whatever manner makes sense for your environment.A possible way is by performing:
$ pip install circus
Create a virtual environment with the specific TurboGears version your application depends on installed.
$ virtualenv /var/tg2env $ /var/tg2env/bin/pip install tg.devtools
Activate the virtual environment
$ source /var/tg2env/bin/activate (tg2env)$ #virtualenv now activated
Once you have the environment enabled you will need to install the
ChaussetteWSGI Server:(tg2env)$ pip install chaussette
Chaussette supports many backends to serve the requests. The default one is based on
wsgiref, which is not really fast. Have a look at the Chaussette Documentation for the available backends:waitress,gevent,meinheldand many more are supported.For this tutorial we are going to use
Waitress, which is a multithreaded WSGI server, so we need to install it inside virtual environment:(tg2env)$ pip install waitress
Now the environment is ready for deploy, you just need to install the TurboGears application.
(tg2env)$ cd /var/www/myapp (tg2env)$ python setup.py develop
We now create a circus configuration file (named
circus.ini) with the informations required to load and start your application. This can be performed using thegearbox deploy-circuscommand from gearbox-tools package or by manually writing it:[circus] check_delay = 5 endpoint = tcp://127.0.0.1:5555 debug = true [env:myapp] PATH=/var/tg2env/bin:$PATH VIRTUAL_ENV=/var/tg2env [watcher:myapp] working_dir = /var/www/myapp cmd = chaussette --backend waitress --fd $(circus.sockets.myapp) paste:production.ini use_sockets = True warmup_delay = 0 numprocesses = 1 stderr_stream.class = FileStream stderr_stream.filename = /var/log/circus/myapp.log stderr_stream.refresh_time = 0.3 stdout_stream.class = FileStream stdout_stream.filename = /var/log/circus/myapp.log stdout_stream.refresh_time = 0.3 [socket:myapp] host = localhost port = 8080
Now start circus with the configuration file, after being started it will load your application:
$ circusd circus.ini 2013-02-15 18:19:54 [20923] [INFO] Starting master on pid 20923 2013-02-15 18:19:54 [20923] [INFO] sockets started 2013-02-15 18:19:54 [20923] [INFO] myapp started 2013-02-15 18:19:54 [20923] [INFO] Arbiter now waiting for commands
Visit
http://localhost:8080/in a browser to access the application. You can now proxy it behind Apache, Nginx or any other web server or even use the VHostino project for circus to serve multiple applications through virtual hosts
See the circus documentation for more in-depth configuration information.
Running TurboGears under Heroku¶
This recipe assumes that you have a TurboGears app setup using a Paste INI file, inside a package called ‘myapp’. If you are deploying a custom TurboGears application in minimal mode you might have to tune the following instructions.
Step 0: Install heroku¶
Install the heroku gem per their instructions.
Step 1: Add files needed for heroku¶
You will need to add the following files with the contents as shown to the root of your project directory (the directory containing the setup.py).
requirements.txt:
You can autogenerate this file by running:
$ pip freeze > requirements.txt
You will have probably have a line in your requirements file that has your project name in it. It might look like either of the following two lines depending on how you setup your project. If either of these lines exist, delete them.
projectname=0.1dev
or
-e git+git@xxxx:<git username>/xxxxx.git....#egg=projectname
Now that you have properly frozen your application dependencies it is required to add the webserver you want to use to actually serve your application requests.
This tutorial uses the Waitress webserver, so we need to add it to the dependencies
declared in the requirements.txt
$ echo "waitress" >> requirements.txt
Step 2: Editing Configuration File¶
As heroku passes some configuration options in ENVIRON variables, it is necessary
for our application to read them from the Heroku environment. Those are typically
the PORT where your application server has to listen, the URL of your
database and so on…
First of all we need to copy the development.ini to a production.ini file
we are going to use for the heroku deployment:
$ cp development.ini production.ini
The only options you are required to change are the one related to the server.
So your [server:main] section should look like:
[server:main]
use = egg:waitress#main
host = 0.0.0.0
get port = heroku_port
Then probably want to disable the debug inside the [DEFAULT] section.
Note
If you want to use a different server instead of waitress, as gevent,
CherryPy or something else, as far as it is compatible with PasteDeploy
changing the use = egg:waitress#main to whatever you want usually is enough.
In case you want to use gevent, for you can change it to use = egg:gearbox#gevent.
Step 3: Starting the application¶
Procfile:
Generate this by running:
$ echo "web: ./run" > Procfile
run:
Create run with the following:
#!/bin/bash
python setup.py develop
gearbox serve --debug -c production.ini heroku_port=$PORT
Note
Make sure to chmod +x run before continuing.
The ‘develop’ step is necessary because the current package must be
installed before paste can load it from the INI file.
Step 4: Setup git repo and heroku app¶
Navigate to your project directory (directory with setup.py) if not already there. If you project is already under git version control, skip to the ‘Initialize the heroku stack’ section.
Inside your projects directory, if this project is not tracked under git it is recommended that you first create a good .gitignore file (you can skip this step). You can get the recommended python one by running:
$ wget -O .gitignore https://raw.github.com/github/gitignore/master/TurboGears2.gitignore
Once that is done, run:
$ git init
$ git add .
$ git commit -m "initial commit"
Step 5: Initialize the heroku stack¶
$ heroku create
Step 6: Deploy¶
To deploy a new version, push it to heroku:
$ git push heroku master
Make sure to start one worker:
$ heroku scale web=1
Check to see if your app is running
$ heroku ps
Take a look at the logs to debug any errors if necessary:
$ heroku logs -t
Running TurboGears under AppEngine¶
This recipe describes how to create a minimal mode TurboGears application under Google AppEngine.
Step 0: Install AppEngine¶
Install the AppEngine SDK from https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python The following guide uses the AppEngine SDK for OSX.
Step 1: Create Application¶
First required step is to create an AppEngine application, this can easily be done through the AppEngine SDK.
If you are not using the AppEngine SDK for OSX please refer to the AppEngine Documentation to create a new application.
This will create a new application in the specified directory with the following files:
$ tree
.
├── app.yaml
├── favicon.ico
├── index.yaml
└── main.py
By default the created application relies on the Webapp2 framework,
to remove this dependency edit the app.yaml file and delete the
following part:
libraries:
- name: webapp2
version: "2.5.2"
Step 2: Setup Dependencies¶
On AppEngine, all the dependencies of you application should be provided with the application itself. This makes so that we are required to install TurboGears inside the application directory.
Supposing your application is named tgtest and you created it in your HOME,
to do so we first need to create a temporary virtual environment we will throw
away when everything is installed:
$ cd ~/tgtest
$ virtualenv --no-site-packages tmpenv
New python executable in tmpenv/bin/python
Installing setuptools............done.
Installing pip...............done.
Note
Depending on your virtualenv command version, the --no-site-packages option
might not be required.
Now will enable the virtualenv and install the TurboGears2 dependency, only
difference is that instead of installing it inside the virtual environment itself
we will tell pip to install it inside the packages directory of our application:
. tmpenv/bin/activate
(tmpenv)$ pip install -t packages -I TurboGears2
As AppEngine doesn’t provide setuptools the last required step is to also
provide setuptools inside our packages directory:
(tmpenv)$ pip install -t packages -I setuptools
Note
Please note the -I option to force pip ignoring the currently installed packages.
Now all the required dependencies are installed inside the packages directory
and our virtual environment can be deleted:
(tmpenv)$ deactivate
$ rm -r tmpenv
Step 3: The TurboGears Application¶
Now we can proceed editing the main.py file which is started by
AppEngine to run our application and actually write a TurboGears application there.
The first required step is to tell python to load our dependencies from
the packages directory inside our application. So at the begin of
your main.py, right after the leading comment, add the following lines:
import os
import site
site.addsitedir(os.path.join(os.path.dirname(__file__), 'packages'))
Then, inside the main.py file, after the site.addsitedir line,
you can create the actual WSGI application which must be named app
for AppEngine to serve it:
from tg import expose, TGController, AppConfig
class RootController(TGController):
@expose()
def index(self):
return "<h1>Hello World</h1>"
config = AppConfig(minimal=True, root_controller=RootController())
app = config.make_wsgi_app()
Step 4: Start The Application¶
Now that the application is ready, we can start it from the GoogleAppEngineLauncher
and point our browser to http://localhost:8080 to get its output
Note
if something went wrong, you can press the Logs button and check
the application output.
Step 5: Deploy It!¶
Now that your application is correctly running locally, it’s time to send it to Google servers.
First required step, is to let google know that our application exists.
To do so, point your browser to https://appengine.google.com/ and
press the Create Application button to create a new application.
Once you created your application and have a valid Application Identifier
available, edit the app.yaml file inside tgtest to set the correct
identifier:
application: turbogearstest
Note
replace turbogearstest with your own app identifier.
Now get back to the GoogleAppEngineLauncher and press the Deploy button. It will ask for your google account credentials, and will then proceeed with the deploy.
As soon as it finished the deploy, your application will be available online
at the reserved URL. Which is in the form APPID.appspot.com, in our example
we can point the browser to http://turbogearstest.appspot.com and get
our application output.
Setting up logging in your Application¶
TurboGears relies on the standard Python logging module,
so to add logging to your application simply add the following
lines at the begin of your files:
import logging
log = logging.getLogger(__name__)
Then you can report logging messages with the standard logger
methods: log.warning(), log.info(), log.error(),
log.exception() and so on:
class SimpleController(TGController):
@expose()
def simple(self):
log.debug("My first logged controller!")
return "OK"
Refer to the Python Logger documentation for additional details.
By default TurboGears configures the logging module so that all the
log messages from your application are displayed from DEBUG level
on. So even debug messages will be displayed.
This is specified at the end of the development.ini file.
When starting your application with gearbox it will automatically
load the logging configuration from your development.ini or provided
configuration file.
When you are deploying your application on mod_wsgi or any other environment
that doesn’t rely on gearbox to run the application, remember to load
the logging configuration before creating the actual WSGI application:
APP_CONFIG = "/var/www/myapp/myapp/production.ini"
#Setup logging
import logging.config
logging.config.fileConfig(APP_CONFIG)
#Load the application
from paste.deploy import loadapp
application = loadapp('config:%s' % APP_CONFIG)
Otherwise the logging configuration will be different from the one
available when starting the application with gearbox and you might
end up not seeing logging messages.
Logging Output¶
In the default configuration all your logging output goes to sys.stderr.
What exactly that is depends on your deployment environment.
In case of mod_wsgi it will be redirected to the Apache ErrorLog,
but in case your environment doesn’t provide a convenient way to
configure output location your can set it up through the development.ini
in the [handler_console] section:
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
For example to change it to log to a specific file you can replace the
StreamHandler with a FileHandler:
[handler_console]
class = FileHandler
args = ('application.log', 'a')
level = NOTSET
formatter = generic
Note
Please not that the best practice is not to change the console handler
but creating a new handler and switch the various loggers to it.
WSGI Errors Output¶
The WSGI standard defines a wsgi.errors key in the environment
which can be used to report application errors. As this feature is
only available during a request (when the WSGI environment is provided),
applications won’t usually rely on it, preferring instead the logging
module which is always available.
Please note that many WSGI middlewares will log to it, instead of using the logging module,
such an example is the backlash error reporting middleware used
by TurboGears for its errorware stack.
Setting up wsgi.errors is usually a task that your application server
does for you, and will usually point to the same location sys.stderr points
to. So your wsgi.errors and logging outputs should be available at
the same destination.
In case your deploy environment isn’t setting up wsgi.errors correctly or you
changed the logging output you might have to change where wsgi.errors points too.
This has to be done by code, replacing the environ['wsgi.errors'] key,
on every request, with a stream object.
Being it sys.stderr or something else.
It is usually best practice to leave both the logging output on sys.stderr and
wsgi.errors as is, as they will usually end up at the same location on most
application servers. Then you can tune the output from the application server
configuration itself.
In case of gearbox serve, wsgi.errors will point to sys.stderr which is then
redirected to a logfile, if provided with the --log-file option.
In case of mod_wsgi they will both point to the Apache ErrorLog file so you
can tune your whole logging output configuration from Apache itself.
Streaming Response¶
Streaming permits to your controller to send data that yet has to be created to the client, this can be really useful when your app needs to send an huge amount of data to the client, much more data than you are able to keep in memory.
Streaming can also be useful when you have to send content to the client that yet has to be generated when you provide an answer to the client keeping an open connection between the application and the client itself.
In TurboGears2 streaming can be achieved returning a generator from your controllers.
Making your application streaming compliant¶
So the first thing you want to do when using streaming is disabling any middleware that edits the content of your response. Since version 2.3 disabling debug mode is not required anymore.
Most middlewares like debugbar, ToscaWidgets and so on will avoid touching your response if it is not of text/html content type. So streaming files or json is usually safe.
Streaming with Generators¶
Streaming involves returning a generator from your controller, this will let the generator create the content while being read by client.
@expose(content_type='application/json')
def stream_list(self):
def output_pause():
num = 0
yield '['
while num < 9:
num += 1
yield '%s, ' % num
time.sleep(1)
yield '10]'
return output_pause()
This simple example will slowly stream the numbers from 1 to 10 in a json array.
Accessing TurboGears objects¶
While streaming content is quite simple some teardown functions get executed before your streamer, this has the side effect of changing how your application behaves.
Accessing Request¶
Since version 2.3 all the global turbogears objects are accessible while running the generator. So if you need to have access to them, you can freely read and write to them. Just keep in mind that your response has already been started, so it is not possible to change your outgoing response while running the generator.
@expose(content_type='text/css')
def stream(self):
def output_pause(req):
num = 0
while num < 10:
num += 1
yield '%s/%s\n' % (tg.request.path_info, num)
time.sleep(1)
return output_pause()
This example, while not returning any real css, shows how it is possible to access the turbogears request inside the generator.
Reading from Database¶
Since version 2.3 is is possible to read from the database as usual:
@expose(content_type='application/json')
def stream_db(self):
def output_pause():
num = 0
yield '['
while num < 9:
u = DBSession.query(model.User).filter_by(user_id=num).first()
num += 1
yield u and '%d, ' % u.user_id or 'null, '
time.sleep(1)
yield 'null]'
return output_pause()
Writing to Database¶
If you need to write data on the database you will have to manually flush the session and commit the transaction. This is due to the fact that TurboGears2 won’t be able to do it for you as the request flow already ended.
@expose(content_type='application/json')
def stream_list(self):
def output_pause():
import transaction
num = 0
while num < 9:
DBSession.add(model.Permission(permission_name='perm_%s'%num))
yield 'Added Permission\n'
num += 1
time.sleep(1)
DBSession.flush()
transaction.commit()
return output_pause()
Scheduling Tasks¶
On Posix systems, using cron is a perfectly valid way to schedule tasks for your application.
However, it sometimes happen that you need to interact intimately with the runtime environment of your application, that you need to schedule jobs dynamically, or that your hosting service does not provide access to cron. In those cases, you can schedule jobs with the TGScheduler module.
Installation¶
TGScheduler is registered on PyPI and therefore can be installed with pip install:
$ pip install tgscheduler
Setup¶
TGScheduler is not started by default. To allow your tasks to run, simply start the scheduler when your application is loaded configured.
You can do that in config/app_cfg.py:
def start_tgscheduler():
import tgscheduler
tgscheduler.start_scheduler()
from tg.configuration import milestones
milestones.config_ready.register(start_tgscheduler)
Using a Configuration Milestones ensures that our scheduler is not
started twice, it is also suggested to register tasks inside
the start_tgscheduler function so that they are not
scheduled twice.
Scheduling Tasks¶
To you have four ways to schudule tasks:
- add_interval_task();
- add_monthly_task();
- add_single_task();
- add_weekday_task().
Each of those receive a callable and a time specifier that defines when to run a function. As an example, if you want to update the stock prices in your database every 15 minutes, you would do something like the following:
def update_stocks():
url = 'http://example.com/stock_prices.xml'
data = urllib2.urlopen(url).read()
etree = lxml.etree.fromtsring(data)
for el in etree.xpath("//stock"):
price = model.StockPrice(el.get("name"), int(el.get("price")))
model.DBSession.add(price)
def start_tgscheduler():
import tgscheduler
tgscheduler.start_scheduler()
tgscheduler.add_interval_task(60*15, update_stocks)
from tg.configuration import milestones
milestones.config_ready.register(start_tgscheduler)
Note
Inside the scheduled tasks you won’t have TurboGears automatically
flushing and committing transactions for you, so remember to add
a transaction.commit() call a the end of your tasks if your
application is relying on the transaction manager.
Advanced Caching Recipes¶
Caching Template and Controller¶
A simple form of joint controller+template caching can be achieved by using both cached decorator
and tg_cache parameter as described in Caching.
While it is more common having to perform some kind of minimal computation in controller to decide which cache key to use and rely on the template caching only, the caching system can be leveraged to turn the whole request in a direct cache hit based on the request parameters.
This can be achieved by relying on the cached decorator and using render_template() to
actually render the template during controller execution and caching it together with the controller itself:
from tg import cached, render_template
@cached()
@expose(content_type='text/html')
def cached_func(self, what='about'):
return render_template(dict(page=what, time=time.time()),
'kajiki', 'myproj.templates.cached_func')
Note
While @cached caches the controller itself, any hook or validation associated to the
controller will still be executed. This might be what you want (for example when tracking
page views through an hook) or it might not, depending on your needs you might want to move
hooks and validation inside the controller itself to ensure they are cached.
Caching Authentication¶
Authentication in TurboGears is provided by repoze.who through the ApplicationAuthMetadata class
as described in Identification & Authentication Layer.
ApplicationAuthMetadata provides two major steps in authentication:
- One is authenticating the user itself for the first time (done by
ApplicationAuthMetadata.authenticate) which is in charge of returning theuser_namethat uniquely identifies the user (or any other unique identifier) that is then received by the other methods to lookup the actual user data.- The second step, which is the metadata provisioning, is performed by
IdentityApplicationWrapperand receives the previously identified user as returned by the authentication step. This is performed byget_user,get_groupsandget_permissionswhich are in charge of returning the three aforementioned information regarding the user.
It’s easy to see that this second step is usually the one that has most weight over the request throughput as it involves three different queries to the database.
We can easily change the ApplicationAuthMetadata in our code to rely on the cache to fetch user data instead
of loading it back from database:
from tg import cache
class ApplicationAuthMetadata(TGAuthMetadata):
def __init__(self, sa_auth):
self.sa_auth = sa_auth
def authenticate(self, environ, identity):
# This should be your current authenticate implementation
...
def get_user(self, identity, userid):
identity.update(self._get_cached_auth_metadata(userid))
return identity['user']
def get_groups(self, identity, userid):
return identity['groups'] or []
def get_permissions(self, identity, userid):
return identity['permissions'] or []
def _get_cached_auth_metadata(self, userid):
"""Retrieves the user details from the cache when available"""
auth_cache = cache.get_cache('auth')
auth_metadata = auth_cache.get_value(key=userid,
createfunc=lambda: self._retrieve_auth_metadata(userid),
expiretime=3600)
auth_metadata['user'] = self.sa_auth.dbsession.merge(auth_metadata['user'], load=False)
return auth_metadata
def _retrieve_auth_metadata(self, userid):
"""Retrieves user details from the database"""
user = self.sa_auth.dbsession.query(self.sa_auth.user_class).filter_by(user_name=userid).first()
return {
'user': user,
'groups': user and [g.group_name for g in user.groups],
'permissions': user and [p.permission_name for p in user.permissions]
}
This is usually enough to cache authentication requests in an environment where user data, permissions and groups change rarely. A better cache management, invalidating the user cache whenever the user itself or its permission change, is required for more volatile scenarios.
Using both SQLAlchemy and MongoDB¶
TurboGears since version 2.3.8 allows to enable both Ming and SQLAlchemy into
the same project. This can be achieved by specifying both the use_ming=True
and use_sqlalchemy=True options in AppConfig.
By Default the SQLAlchemy session is considered the primary and is installed as
config['DBSession'] unless it’s explicitly set in AppConfig. When a new
project is created, the quickstart will automatically set this according to the
--ming or --sqlalchemy option, so you usually are ensured that the primary
database is the one you quickstarted the project with.
Both databases will call model.init_model with their engine, according to the
engine type you can take proper action and return the right database session.
To configure both Ming and SQLAlchemy sessions your model/__init__.py
will probably look like:
from sqlalchemy.engine import Engine
# SQLAlchemy Configuration
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
maker = sessionmaker(autoflush=True, autocommit=False,
extension=ZopeTransactionExtension())
DBSession = scoped_session(maker)
DeclarativeBase = declarative_base()
metadata = DeclarativeBase.metadata
# Ming Configuration
from ming import Session
from ming.orm import ThreadLocalORMSession
mingsession = Session()
ODMSession = ThreadLocalORMSession(mingsession)
def init_model(engine):
if isinstance(engine, Engine):
# SQLAlchemy
DBSession.configure(bind=engine)
return DBSession
else:
# Ming
mingsession.bind = engine
return ODMSession
The returned session will be available as config['MingSession'] and config['SQLASession']
according to the initialized engine. You just have to ensure your models use the right session.
Using Multiple Databases In TurboGears¶
| Status: | RoughDoc |
|---|
Table of Contents
The goal of this tutorial is to configure TurboGears to use multiple databases. In this tutorial we will simply set up two different databases engines that will use db session handlers of DBSession and DBSession2, db metadata names of metadata and metadata2, and DeclarativeBase objects of DeclarativeBase and DeclarativeBase2.
Note
When using multiple databases you won’t be able to create relations (foreign keys) between tables on two different databases.
Note
Most plugins and extensions will take for granted that you have a single database connection, and might not work properly when multiple databases are used.
Define your database urls in the [app:main] section of your .ini file(s)¶
The first thing you will need to do is edit your .ini file to specify multiple url options for the sqlalchemy configuration.
In myapp/development.ini (or production.ini, or whatever.ini you are
using), comment out the original sqlalchemy.url assignment and add the
multiple config options:
# We need two different connection URLs for the two engines,
# so we comment the default one to avoid unexpected usages.
# sqlalchemy.url = sqlite:///%(here)s/devdata.db
sqlalchemy.first.url = sqlite:///%(here)s/database_1.db
sqlalchemy.second.url = sqlite:///%(here)s/database_2.db
Change The Way Your App Loads The Database Engines¶
Now we need to instruct the app configurator to load the multiple databases correctly. This requires telling base_config (in app_cfg.py) to load our own custom AppConfig with the proper multi-db assignments and a call to the model’s init_model method (more on that in the next step).
In myapp/config/app_cfg.py:
# make sure these imports are added to the top
from tg.configuration import AppConfig
from myapp import model
# add this before base_config =
class MultiDBAppConfig(AppConfig):
def _setup_sqlalchemy(self, conf):
from sqlalchemy import engine_from_config
engine1 = engine_from_config(conf, 'sqlalchemy.first.')
engine2 = engine_from_config(conf, 'sqlalchemy.second.')
# We will consider engine1 the "default" engine
conf['tg.app_globals'].sa_engine = engine1
conf['tg.app_globals'].sa_engine2 = engine2
# Pass the engines to init_model, to be able to introspect tables
model.init_model(engine1, engine2)
conf['SQLASession'] = conf['DBSession'] = model.DBSession
conf['SQLASession2'] = conf['DBSession2'] = model.DBSession2
def _add_sqlalchemy_middleware(self, conf, app):
# We need to ensure that both sessions are closed at the end of a request.
from tg.support.middlewares import DBSessionRemoverMiddleware
dbsession = conf.get('SQLASession')
app = DBSessionRemoverMiddleware(dbsession, app)
dbsession2 = conf.get('SQLASession2')
app = DBSessionRemoverMiddleware(dbsession2, app)
return app
# base_config = AppConfig()
base_config = MultiDBAppConfig()
Update Your Model’s __init__ To Handle Multiple Sessions And Metadata¶
Switching the model’s init from a single-db config to a multi-db simply means we have to duplicate our DBSession and metata assignments, and then update the init_model method to assign/configure each engine correctly.
In myapp/model/__init__.py:
# after the first maker/DBSession assignment, add a 2nd one
maker2 = sessionmaker(autoflush=True, autocommit=False,
extension=ZopeTransactionExtension())
DBSession2 = scoped_session(maker2)
# after the first DeclarativeBase assignment, add a 2nd one
DeclarativeBase2 = declarative_base()
# uncomment the metadata2 line and assign it to DeclarativeBase2.metadata
metadata2 = DeclarativeBase2.metadata
# finally, modify the init_model method to allow both engines to be passed (see previous step)
# and assign the sessions and metadata to each engine
def init_model(engine1, engine2):
"""Call me before using any of the tables or classes in the model."""
# DBSession.configure(bind=engine)
DBSession.configure(bind=engine1)
DBSession2.configure(bind=engine2)
metadata.bind = engine1
metadata2.bind = engine2
Tell Your Models Which Engine To Use¶
Now that the configuration has all been taken care of, you can instruct your models to inherit from either the first or second DeclarativeBase depending on which DB engine you want it to use.
For example, in myapp/model/spam.py (uses engine1):
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase
class Spam(DeclarativeBase):
__tablename__ = 'spam'
def __init__(self, id, variety):
self.id = id
self.variety = variety
id = Column(Integer, autoincrement=True, primary_key=True)
variety = Column(Unicode(50), nullable=False)
And then in myapp/model/eggs.py (uses engine2):
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode, Boolean
from myapp.model import DeclarativeBase2
class Eggs(DeclarativeBase2):
__tablename__ = 'eggs'
def __init__(self, id, pkg_qty):
self.id = id
self.pkg_qty = pkg_qty
id = Column(Integer, autoincrement=True, primary_key=True)
pkg_qty = Column(Integer, default=12)
If you needed to use the DBSession here (or in your controllers), you would use DBSession for the 1st engine and DBSession2 for the 2nd (see the previous and next sections).
Create And Populate Each Database In Websetup¶
If you want your setup_app method to populate each database with data, simply use the appropriate metadata/DBSession objects as you would in a single-db setup.
In myapp/websetup/schema.py:
def setup_schema(command, conf, vars):
from tgmultidb import model
print("Creating tables")
model.metadata.create_all(bind=config['tg.app_globals'].sa_engine)
model.metadata2.create_all(bind=config['tg.app_globals'].sa_engine2)
transaction.commit()
In myapp/websetup/bootstrap.py:
def setup_app(command, conf, vars):
from sqlalchemy.exc import IntegrityError
try:
# populate spam table
spam = [model.Spam(1, u'Classic'), model.Spam(2, u'Golden Honey Grail')]
# DBSession is bound to the spam table
model.DBSession.add_all(spam)
# populate eggs table
eggs = [model.Eggs(1, 12), model.Eggs(2, 6)]
# DBSession2 is bound to the eggs table
model.DBSession2.add_all(eggs)
model.DBSession.flush()
model.DBSession2.flush()
transaction.commit()
print "Successfully setup"
except IntegrityError:
print('Warning, there was a problem adding your auth data, '
'it may have already been added:')
import traceback
print(traceback.format_exc())
transaction.abort()
print('Continuing with bootstrapping...')
Additional Support¶
There are some additional features that TurboGears2 provides out of the box for single databases that might require change when multiple DBs are involved.
Authentication¶
Your User/Group/Permission and support tables usually need to
be all in the same database. In case this database is not the
one managed by primary DeclarativeBase and primary DBSession
you need to provide to base_config.sa_auth.dbsession the
right session.
Admin¶
The default turbogears admin is mounted to handle all the models
through DBSession. If you moved any mode to DBSession2 you
will have to accordingly configure two admins:
class RootController(BaseController):
admin = AdminController([model.Spam], DBSession, config_type=TGAdminConfig)
admin2 = AdminController([model.Eggs], DBSession2, config_type=TGAdminConfig)
Migrations¶
Code in myapp/websetup/schema.py that initializes the migrations will have to be duplicated to allow migrations for both DB1 and DB2:
print('Initializing Primary Migrations')
import alembic.config
alembic_cfg = alembic.config.Config()
alembic_cfg.set_main_option("script_location", "migration1")
alembic_cfg.set_main_option("sqlalchemy.url", config['sqlalchemy.first.url'])
import alembic.command
alembic.command.stamp(alembic_cfg, "head")
print('Initializing Secondary Migrations')
import alembic.config
alembic_cfg = alembic.config.Config()
alembic_cfg.set_main_option("script_location", "migration2")
alembic_cfg.set_main_option("sqlalchemy.url", config['sqlalchemy.second.url'])
import alembic.command
alembic.command.stamp(alembic_cfg, "head")
You will need also to provide two different migration repositories for the two
db. The easiest way is usually to take the migration directory and rename
it to migration1 and migration2, then make sure to update references
to sqlchemy. inside the two directories migration1/env.py and migration2/env.py
so that they point to sqlalchemy.first. and sqlalchemy.second..
You can then choose for which database run the migrations by providing the
--location option to gearbox migrate command:
$ gearbox migrate -l migration1 db_version
198f81ba8170 (head)
$ gearbox migrate -l migration2 db_version
350269a5537c (head)
Using LDAP for user authentication and authorization¶
| Status: | RoughDoc |
|---|
Table of Contents
This recipe shows how to configure TurboGears to use an LDAP directory for user authentication and authorization.
Requirements¶
We will use the who_ldap plugin for repoze.who for this purpose, since it supports both Python 2 and Python 3.
You can find the documentation for who_ldap on PyPi. From there, you can also install who_ldap, just run:
pip install who_ldap
You should also add this requirement to your project’s setup.py file:
install_requires=[
...,
"who_ldap",
]
Note that who_ldap itself requires the ldap3 package (formerly known as python3-ldap), which is a pure Python implementation of an LDAP v3 client. See the documentation for ldap3 for details.
Configuration¶
Here is an example configuration that you can put into the config/app_cfg.py file of your project:
# Configure the base SQLALchemy setup:
base_config.use_sqlalchemy = False
# Configure the authentication backend:
# YOU MUST CHANGE THIS VALUE IN PRODUCTION TO SECURE YOUR APP
base_config.sa_auth.cookie_secret = 'secret'
base_config.auth_backend = 'ldapauth'
from who_ldap import (LDAPSearchAuthenticatorPlugin,
LDAPAttributesPlugin, LDAPGroupsPlugin)
# Tell TurboGears how to connect to the LDAP directory
ldap_url = 'ldaps://ad.snake-oil-company.com'
ldap_base_dn = 'ou=users,dc=ad,dc=snake-oil-company,dc=com'
ldap_bind_dn = 'cn=bind,cn=users,dc=ad,dc=snake-oil-company,dc=com'
ldap_bind_pass = 'silverbullet'
# Authenticate users by searching in LDAP
ldap_auth = LDAPSearchAuthenticatorPlugin(
url=ldap_url, base_dn=ldap_base_dn,
bind_dn=ldap_bind_dn, bind_pass=ldap_bind_pass,
returned_id='login',
# the LDAP attribute that holds the user name:
naming_attribute='sAMAccountName',
start_tls=True)
base_config.sa_auth.authenticators = [('ldapauth', ldap_auth)]
# Retrieve user metadata from LDAP
ldap_user_provider = LDAPAttributesPlugin(
url=ldap_url, bind_dn=ldap_bind_dn, bind_pass=ldap_bind_pass,
name='user',
# map from LDAP attributes to TurboGears user attributes:
attributes='givenName=first_name,sn=last_name,mail=email_address',
flatten=True, start_tls=True)
# Retrieve user groups from LDAP
ldap_groups_provider = LDAPGroupsPlugin(
url=ldap_url, base_dn=ldap_base_dn,
bind_dn=ldap_bind_dn, bind_pass=ldap_bind_pass,
filterstr='(&(objectClass=group)(member=%(dn)s))',
name='groups',
start_tls=True)
base_config.sa_auth.mdproviders = [
('ldapuser', ldap_user_provider),
('ldapgroups', ldap_groups_provider)]
from tg.configuration.auth import TGAuthMetadata
class ApplicationAuthMetadata(TGAuthMetadata):
"""Tell TurboGears how to retrieve the data for your user"""
# map from LDAP group names to TurboGears group names
group_map = {'operators': 'managers'}
# set of permissions for all mapped groups
permissions_for_groups = {'managers': {'manage'}}
def __init__(self, sa_auth):
self.sa_auth = sa_auth
def get_user(self, identity, userid):
user = identity.get('user')
if user:
name ='{first_name} {last_name}'.format(**user).strip()
user.update(user_name=userid, display_name=name)
return user
def get_groups(self, identity, userid):
get_group = self.group_map.get
return [get_group(g, g) for g in identity.get('groups', [])]
def get_permissions_for_group(self, group):
return self.permissions_for_groups.get(group, set())
def get_permissions(self, identity, userid):
permissions = set()
get_permissions = self.get_permissions_for_group
for group in self.get_groups(identity, userid):
permissions |= get_permissions(group)
return permissions
base_config.sa_auth.authmetadata = ApplicationAuthMetadata(
base_config.sa_auth)
# Override this if you would like to provide a different who plugin for
# managing login and logout of your application:
base_config.sa_auth.form_plugin = None
# Page where you want users to be redirected to on login:
base_config.sa_auth.post_login_url = '/post_login'
# Page where you want users to be redirected to on logout:
base_config.sa_auth.post_logout_url = '/post_logout'
You will need to change the connection parameters to point to your user base in your LDAP directory and login with a bind user and password that is authorized to search over the directory.
You may also need to change some of the other parameters according to your requirements. The configuration for retrieving user metadata and user groups is optional if you want to use LDAP solely for authentication and not for authorization.
Autogenerated Forms with Sprox¶
This is a succinct explanation on how to use sprox’s form rendering
capabilities within TurboGears2. We will assume the reader is somewhat
versed in TurboGears2’s tg.controllers.RestController. Note
that this is the same technology the Turbogears2 admin is based on, so
this knowledge is valuable to understand how to configure the admin
for your purposes.
Establishing the Model Definition¶
Let us first assume the following model for this demonstration.:
from sqlalchemy import Column, Integer, String, Date, Text, ForeignKey, Table
from sqlalchemy.orm import relation
from moviedemo.model import DeclarativeBase, metadata
movie_directors_table = Table('movie_directors', metadata,
Column('movie_id', Integer, ForeignKey('movies.movie_id'), primary_key = True),
Column('director_id', Integer, ForeignKey('directors.director_id'), primary_key = True))
class Genre(DeclarativeBase):
__tablename__ = "genres"
genre_id = Column(Integer, primary_key=True)
name = Column(String(100))
description = Column(String(200))
class Movie(DeclarativeBase):
__tablename__ = "movies"
movie_id = Column(Integer, primary_key=True)
title = Column(String(100), nullable=False)
description = Column(Text, nullable=True)
genre_id = Column(Integer, ForeignKey('genres.genre_id'))
genre = relation('Genre', backref='movies')
release_date = Column(Date, nullable=True)
class Director(DeclarativeBase):
__tablename__ = "directors"
movie_id = Column(Integer, primary_key=True)
title = Column(String(100), nullable=False)
movies = relation(Movie, secondary_join=movie_directors_table, backref="directors")
The Basic Sprox Form¶
Here is how we create a basic form for adding a new Movie to the database:
class NewMovieForm(AddRecordForm):
__model__ = Movie
new_movie_form = NewMovieForm(DBSession)
And our controller code would look something like this:
@expose('moviedemo.templates.sproxdemo.movies.new')
def new(self, **kw):
tmpl_context.widget = new_movie_form
return dict(value=kw)
You may have noticed that we are passing keywords into the method. This is so that the values previously typed by the user can be displayed on failed validation.
And finally, our template code:
<html py:extends="master.xhtml" py:strip="True">
<body py:block="body" py:strip="True">
<div style="height:0px;"> </div>
<div>
<div style="float:left width: 80%">
<h2 style="margin-top:1px;>Create New Movie</h2>
${tmpl_context.widget(value=value)}
</div>
</div>
</body>
</html>
Which produces a form like this:
Omitting Fields¶
Now, we can use the __omit_fields__ modifier to remove the “movie_id” and “genre_id” fields, as they will be of little use to our users. Our form code now becomes:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['genre_id', 'movie_id']
The rendered form now looks like this:
Limiting fields¶
If you have more omitted fields than required fields, you might want to use the __limit_fields__ operator to eliminate the fields you don’t want. The same above form will be rendered with the following code:
class NewMovieForm(AddRecordForm):
__model__ = Movie[
__limit_fields__ = ['title', 'description', 'release_date', 'genre', 'directors']
Field Ordering¶
If you want the fields displayed in a ordering different from that of the specified schema, you may use field_ordering to do so. Here is our form with the fields moved around a bit:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_order__ = ['title', 'description', 'genre', 'directors']
Notice how the release_date field that was not specified was still appended to the end of the form.
Overriding Field Attributes¶
Sometimes we will want to modify some of the HTML attributes associated with a field. This is as easy as passing a __field_attrs__ modifier to our form definition. Here is how we could modify the description to have only 2 rows:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_attrs__ = {'description':{'rows':'2'}}
Here is the resultant form:
Overriding a Form Field¶
Sometimes you want to override a field all together. Sprox allows you to do this by providing an attribute to your form class declaratively. Simply instantiate your field within the widget and it will override the widget used for that field. Let’s change the movie title to a password field just for fun.:
from tw.forms.fields import PasswordField
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_attrs__ = {'description':{'rows':'2'}}
title = PasswordField
You can see now that the title is “starred” out. Note that you may also send an “instance” of a widget for a field, but you must pass in the fieldname to the widget. This is a limitation of ToscaWidgets. (You may not change the “id” of a widget after it has been created.):
title = PasswordField('title')
Field Widget Args¶
Sometimes you want to provide sprox with a class for a field, and have sprox set the arguments to a widget, but you either want to provide an additional argument, or override one of the arguments that sprox chooses. For this, pass a dictionary into the __field_widget_args__ parameter with the key being the field you would like to pass the arg into, and the value a dictionary of args to set for that field. Here is an example of how to set the rows and columns for the description field of a form.:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__field_widget_args__ = {'description':{'rows':30, 'cols':30}}
Custom Dropdown Field Names¶
Sometimes you want to display a field to the user for the dropdown that has not been selected by sprox. This is easy to override. Simply pass the field names for the select boxes you want to display into the __dropdown_field_names__ modifier.:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_order__ = ['title', 'description', 'genre', 'directors']
__dropdown_field_names__ = ['description', 'name']
If you want to be more specific about which fields should display which field, you can pass a dictionary into the __dropdown_field_names__ modifier.:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_order__ = ['title', 'description', 'genre', 'directors']
__dropdown_field_names__ = {'genre':'description', 'directors':'name'}
Either will produce a new dropdown like this:
Creating Custom Dropdown Data¶
Sometimes providing a fieldname alone is not enough of a customization to inform your users into what they should be selecting. For this example, we will provide both name and description for the Genre field. This requires us to override the genre widget with one of our choosing. We will exend the existing sprox dropdown widget, modifying the update_params method to inject both name and description into the dropdown. This requires some knowledge of ToscaWidgets in general, but this recipe will work for the majority of developers looking to modify their dropdowns in a custom manner.
First, we extend the Sprox SingleSelect Field as follows:
from sprox.widgets import PropertySingleSelectField
class GenreField(PropertySingleSelectField):
def _my_update_params(self, d, nullable=False):
genres = DBSession.query(Genre).all()
options = [(genre.genre_id, '%s (%s)'%(genre.name, genre.description))
for genre in genres]
d['options']= options
return d
Then we include our new widget in the definition of the our movie form:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_order__ = ['title', 'description', 'genre', 'directors']
__dropdown_field_names__ = {'genre':'description', 'directors':'name'}
genre = GenreField
Here is the resulting dropdown:
Adding a New Field¶
There may come a time when you want to add a field to your view which is not part of your database model. The classic case for this is password validation, where you want to provide a second entry field to ensure the user has provided a correct password, but you do not want/need that data to be stored in the database. Here is how we would go about adding a second description field to our widget.:
from tw.forms.fields import TextArea
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__field_order__ = ['title', 'description', 'description2', 'genre', 'directors']
description2 = TextArea('description2')
For additional widgets, you must provide an instance of the widget since sprox will not have enough information about the schema of the widget in order to populate it correctly. Here’s what our form now looks like:
Validation¶
Turbogears2 has some great tools for validation that work well with sprox. In order to validate our form, we must first give the form a place to POST to, with a new method in our controller that looks like:
@validate(new_movie_form, error_handler=new)
@expose()
def post(self, **kw):
del kw['sprox_id']
kw['genre'] = DBSession.query(Genre).get(kw['genre'])
kw['directors'] = [DBSession.query(Director).get(id) for id in kw['directors']]
kw['release_date'] = datetime.strptime(kw['release_date'],"%Y-%m-%d")
movie = Movie(**kw)
DBSession.add(movie)
flash('your movie was successfully added')
redirect('/movies/')
A couple of things about this. First, we must remove the sprox_id from the keywords because they conflict with the Movie definition. This variable may go away in future versions. genre and directors both need to be converted into their related objects before they are applied to the object, and the release_date needs to be formatted as a datetime object if you are using sqlite.
Here is what the form looks like on a failed validation:
Overriding a Validator¶
Often times you will want to provide your own custom field validator. The best way to do this is to add the validator declaratively to your Form Definition:
from formencode.validators import String
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
title = String(min=4)
The resulting validation message looks like this:
Overriding both Field and Validator¶
Ah, you may have realized that sometimes you must override both widget and validator. Sprox handles this too, by providing a :class:sprox.formbase.Field class that you can use to wrap your widget and validator together.:
from formencode.validators import String
from sprox.formbase import Field
from tw.forms.fields import PasswordField
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
title = Field(PasswordField, String(min=4))
Again, the field class does not care if you pass instances or class of the widget.
Required Fields¶
You can tell sprox to make a field required even if it is nullable in the database by passing the fieldname into a list of the __require_fields__ modifier.:
class NewMovieForm(AddRecordForm):
__model__ = Movie
__omit_fields__ = ['movie_id', 'genre_id']
__require_fields__ = ['description']
And the form now sports a validation error:
Form Validation¶
You can validate at the form level as well. This is particularly interesting if you need to compare two fields. See Parameters Validation.
Conclusion¶
sprox.formbase.FormBase class provides a flexible mechanism for creating customized forms.
It provides sensible widgets and validators based on your schema, but can be overridden for your own
needs. FormBase provides declarative addition of fields, ways to limit and omit fields to a set that
is appropriate for your application. Sprox provides automated drop-down boxes, as well as providing
a way to override those widgets for your purposes.
Contributing to TurboGears¶
Preparing Your Development Environment¶
Installing and Using Git¶
Please refer to the Git site for directions on how to install a current version of Git on your system. Note that we do not recommend using a version less than 1.5 for working with Git. Versions earlier than that seemed overly complex to use.
The best way to use a version control system (and especially a distributed version control system like Git) is a subject that could span several books.
Instead of going through all of the detail of the many ways to use Git, we refer you to the Git documentation site for a long list of tutorials, online documentation, and books (both paper and ebook) for you to read that will teach you the many options you can use with Git.
Create A virtualenv¶
A virtualenv is extremely recommended for development work.
Make a new blank virtualenv and activate it. This way, you will have all of your work isolated, preventing conflicts with anything else on which you might be working.
Do not do any easy_install’s of any sort. We will cover that in the next step.
Installing TurboGears2¶
On the TurboGears2 project pages, there are a bunch of repositories related to the turbogears project. The most important are:
- TG2.x Core
- This is the actual core of TurboGears2. Unless you are working on modifying a template or one of the Gearbox based tools, or even the documentation, this is the repository you want.
- TG2.x Devtools
- This repository is the add-on tools. It gets updated when you wish to make a change to help an application developer (as opposed to an application installer). It contains all the stock TurboGears2 templates, and references the Gearbox toolchain to provide an HTTP server, along with other tools.
- TG2.x Docs
- This repository contains two versions of the documentation. The first version (located in the docs directory) is the older docs, and is gradually being phased out. The newer version (located under the book directory) contains this file (and others) and is gradually being brought on par with the old, and will eventually replace the older version entirely.
The best way to prepare your development environment is to take the following steps:
- Clone the first three repositories (TG2.x Core, TG2.x Devtools, and TG2.x Docs).
- Enter the top level directory for your TG2.x Core clone, and run
python setup.py tgdevelopandpython setup.py tgdeps - Enter the top level directory for your TG2.x Devtools clone, and
run
python setup.py tgdevelopandpython setup.py tgdeps - Enter the
bookdirectory for your TG2.x Docs clone, and runpython setup.py tgdevelopandpython setup.py tgdeps
After you’ve done all this, you have a working copy of the code sitting in your system. You can explore the code and begin working through any of the tickets you wish, or even on your own new features that have not yet been submitted.
Note that, for all repositories, work is to be done off of the
development branch. Either work directly on that branch, or do the
work on a branch made from the development branch. The master
branch is reserved for released code.
When working on your feature or ticket, make certain to add the test cases. Without them, the code will not be accepted.
Testing Your Changes¶
After doing your development work, before sending to the main repositories (or sending in a pull request), you must make sure that your code does not break anything.
This process is actually rather painless. The first time you do it,
you need to run python setup.py nosetests from the top of your
working tree. This will download any missing packages that are
required for testing, and then run the tests.
After that first time, you may use nosetests, and all of the tests
will be run.
In either case, you will be told about any failures. If you have any, either fix the code or (if the test case is wrong) fix the test. Then re-run the tests.
If you are interested, you can also see the current status of the tests, and how much of the core code is actually being tested. Run this command:
$ nosetests --with-coverage --cover-erase --cover-package=tg
You will now see which lines are being tested, which ones are not, and have a thorough report on the status of the testing coverage.
Preparing a Release of TurboGears¶
Prerequisites¶
- You have a working knowledge of how to use a virtualenv.
- You have shell access to the turbogears site.
- You know how to run the
nosetestson your local git clones of TurboGears.- You have to have a working knowledge of git, and how to do merging and branching.
- You need permission to update the TurboGears2 and tg.devtools packages on PyPI
With those prerequisites in mind, this document will not cover all the command lines that you could run. Instead, we will specify steps as “activate your virtualenv” or “run nosetests” or the like.
Summary of Steps¶
Preparing a release of TurboGears is going to take some time and effort. It is unlikely to be completed in a mere day or two, so please plan on taking some time to work through it all.
The steps for the release, in summary, are as follows:
- Repository Branching
- Finalizing Changes On ‘next’ Branch
- Preparing Changelog And Release Announcement
- Preparing Packages And The Documentation
- Uploading The Documentation
- Pushing to PyPI
- Publishing Release Annoucement And Closing Milestones
- Final Cleanup
Below, we discuss each of these steps in detail.
Repository Branching¶
We have three main repositories: TG2, TG2Devtools, and TG2Docs. Each of them functions in a similar fashion for this process:
- Clone the repository
- Checkout the “development” branch and update it
- Checkout the “next” branch and merge development into it
Once the “next” branch is made, this is where you will do all work until the release is done.
Finalizing Changes On ‘next’ Branch¶
After all the changes that you’ve made so far, the final changes are simply to get the new version numbers into the distributed files.
- In TG2:
- Update
tg/release.pyto have the new version number.- Update CHANGES.txt:
- Add new release with changelog generate with:
git log --no-merges --format="* %s" LAS_RELEASE_TAG..HEAD(LAST_RELEASE_TAG is the tag of the previous release, like tg2.3.2). Please take only meaningful changes.- In TG2Devtools:
- Update setup.py:
- Update the version number
- Update the install requirements so that it requires TurboGears2 >= the new version number
- Update
devtools/templates/turbogears/setup.py_tmplanddevtools/gearbox/tgext/template/setup.py_tmpl:
- Update the install requirements so that it requires TurboGears2 >= the new version number
- In TG2Docs:
- Update requirements.txt:
- Change TurboGears dependency line from
@developmentto current release (EX:@tg2.3.2).- Update docs/conf.py:
- Update the version number
Commit all of these changes, but do not push them public, not yet.
Preparing Changelog And Release Announcement¶
For each of the three repositories, you will need to review the commit logs since the last release. Gather up the summaries for each one, and prepare a new file. Use the standard GNU Changelog format. However, instead of recording individual file changes, record only the summaries. We don’t need the file changes since Git records those changes for us.
Review the GitHub tickets for this milestone, and record any tickets that were closed for this repository but were not referenced in the summaries you’ve already recorded.
The changelog files you’ve made will be the commit message for the tags you are about to make.
In addition, prepare a release announcement. Anything I can say here sounds condescending. You should prepare it, though, so that as soon as you reach the “Publish” step, it’s all done in a few minutes.
Preparing Packages And The Documentation¶
First, merge the branch “next” onto the branch “master”. Then, tag the
master branch with the new version number, and use the changelog
you’ve generated as the commit message. The tag should be an annotated
tag (i.e.: git tag -a").
Do this for each of the three repositories.
Pushing master branch of documentation will automatically trigger a new
documentation build in readthedocs.io.
Making The Source Distribution to PyPI¶
At this point, everything is prepared, with one exception: The source
distributions for TurboGears2 and tg.devtools must be placed in the
eggbasket. Enter your local repository directory for both TG2.x
Core and TG2.x DevTools and run python setup.py sdist upload.
Publishing Release Annoucement And Closing Milestones¶
Publish your release announcement to the places of your choice. We recommend your blog(s) and twitter. In addition, update the turbogears “Current Status” page to reflect the new release.
Final Cleanup¶
For each of the three repositories, merge the “master” branch to the “development” branch.
You’re done. Sit back and enjoy having accomplished a release.
TurboGears Reference¶
Upgrading Your TurboGears Project¶
From 2.3.11 to 2.3.12¶
No backward incompatible changes happened in 2.3.12.
From 2.3.10 to 2.3.11¶
No backward incompatible changes happened in 2.3.11.
From 2.3.9 to 2.3.10¶
By default Custom Error Pages for content types != text/html got disabled
(to avoid responding with a html error page to a software client expecting JSON or something else).
To re-enable custom error pages for all content types set:
base_config['errorpage.content_types'] = []
In your app_cfg.py.
From 2.3.8 to 2.3.9¶
Quickstart with Genshi¶
Due to incompatibilities with Python3 and due to slower development Genshi has been replaced by Kajiki as the default template engine in newly quickstarted projects.
To quickstart a project with genshi you will need to use:
$ gearbox quickstart --genshi --skip-default-template PROJECT_NAME
This will quickstart a project with genshi as the template engine same as before.
From 2.3.7 to 2.3.8¶
Configuration Process tweaks¶
With TG2.3.5 the configuration process has started a refactoring process which is proceeding one step at time to minimize backward incompatibilities.
In 2.3.8 all the functions that setup helpers, globals, persistence, renderers and middleares are now guaranteed to read and write options from configuration dictionary instead of the application configurator object.
In case you provided your own setup_ or add_ functions that override the
default AppConfig those have been renamed as internal method (_setup_something
and _add_some_middleware), each one of them will now accept the current configuration
dictionary as the first argument. Make sure you read/write from that configuration instead
of self or tg.config.
Otherwise you might be reading/setting options that other steps ignore.
tmpl_context is now always strict¶
Since TG 2.3.8 the tg.strict_tmpl_context option no longer changes
depending on the debug option. By default it’s always True, to
keep a consistent behaviour between development and production environments.
Dispatcher state renamed as dispatch_state¶
Previously the dispatcher state was available as tg.request.controller_state.
The .controller_state attribute is now deprecated in favour of .dispatch_state
attribute.
Action parameters are now always read from the dispatch state¶
Previously the action arguments were always read from request, even when the dispatcher modified them the changes were ignored. Now they are read from the dispatch state and when the dispatcher modifies them the modified values is now used.
Arguments not accepted by dispatched action are now discarded¶
When a request provides parameters unexpected by the action they are now discarded.
Previously TG would keep them around which lead to a crash if the action didn’t provide
a **kwargs argument. Original parameters are still available from the tg.request.
Builtin routes support removed¶
Routes support was deprecated since version 2.3 in favour of tgext.routes and has now been removed.
This is because it is now possible to implement a totally custom routing by
overriding RootController._dispatch to return a new DispatchState instance.
From 2.3.6 to 2.3.7¶
Kajiki Templates Extension¶
In TG 2.3.7 Kajiki templates have switched to have .xhtml extension, this
suites them better than the previous .xml extenion as HTML is actually generated
and is widely supported by IDEs which will correctly highlight them.
From 2.3.5 to 2.3.6¶
Beaker Dependency¶
TurboGears 2.3.6 now doesn’t enlist beaker as a dependency anymore.
If your application makes use of sessions and caching make sure that it requires
beaker in the dependencies or session and caching will be disabled.
Identity provider¶
TurboGears 2.3.6 introduced the IdentityApplicationWrapper which is now
in charge of retrieving identity metadata (user, group, permissions) in place of the
old repoze.who metadata provider. No changes are required to your configuration to
start using the new application wrapper and it provides some direct benefits like
being able to rely on tg.cache and the whole TurboGears context during identity
metadata retrieval (See Caching Authentication for an example).
In case you face problems you can go back to the previous behaviour by adding the
following code to your app_cfg.py:
from zope.interface import implementer
from repoze.who.interfaces import IMetadataProvider
from repoze.who.api import Identity
@implementer(IMetadataProvider)
class RepozeWhoAuthMetadataProvider(object):
"""
repoze.who metadata provider to load groups and permissions data for
the current user. This uses a :class:`TGAuthMetadata` to fetch
the groups and permissions.
"""
def __init__(self, tgmdprovider):
self.tgmdprovider = tgmdprovider
# IMetadataProvider
def add_metadata(self, environ, identity):
# Get the userid retrieved by repoze.who Authenticator
userid = identity['repoze.who.userid']
# Finding the user, groups and permissions:
identity['user'] = self.tgmdprovider.get_user(identity, userid)
if identity['user']:
identity['groups'] = self.tgmdprovider.get_groups(identity, userid)
identity['permissions'] = self.tgmdprovider.get_permissions(identity, userid)
else:
identity['groups'] = identity['permissions'] = []
# Adding the groups and permissions to the repoze.what
# credentials for repoze.what compatibility:
if 'repoze.what.credentials' not in environ:
environ['repoze.what.credentials'] = Identity()
environ['repoze.what.credentials'].update(identity)
environ['repoze.what.credentials']['repoze.what.userid'] = userid
base_config['identity.enabled'] = False
base_config.sa_auth.mdproviders = [
('authmd', RepozeWhoAuthMetadataProvider(base_config.sa_auth.authmetadata))
]
Keep in mind that using a repoze.who metadata provider you won’t be able to rely on TurboGears context and you might face issues with the transaction manager as you are actually retrieving the user before the transaction has started.
From 2.3.4 to 2.3.5¶
Genshi Work-Around available for Python3.4¶
Genshi 0.7 suffers from a bug that prevents it from working on Python 3.4
and causes an Abstract Syntax Tree error, to work-around this issue
TurboGears provides the templating.genshi.name_constant_patch option that
can be set to True to patch Genshi to work on Python 3.4.
Configuration Flow Refactoring¶
In previous versions the AppConfig object won over the
.ini file options for practically everything, now the configurator
has been modified so that AppConfig options are used as a template
and for most options the .ini file wins over them.
There are still some options that are immutable and can only be
defined in the AppConfig itself, but most of them can now
be changed from the ini files.
Now the tg.config object will always be reconfigured from scratch
when an application is created. Previously each time an application
was created it incrementally modified the same config object leading
to odd behaviours. This means that if you want a value to be available
to all instances of your application you should store it in base_config`
and not in ``tg.config. This should not impact your app unless you
called AppConfig.setup_tg_wsgi_app multiple times (which is true
for test suites).
Another minor change is that AppConfig.after_init_config
is now expected to accept a parameter with the configuration
dictionary. So if you implemented a custom after_init_config
method it is required to accept the config dictionary and
make configuration changes in it.
tg.hooks is not bound to config anymore¶
Hooks are not bound to config anymore, but are now managed by an
HooksNamespace. This means that they are now registered per
process and namespace instead of being registered per-config.
This leads to the same behaviour when only one TGApp is configured
per process but has a much more reliable behaviour when multiple
TGApp are configured.
For most users this shouldn’t cause any difference, but hooks will now be registered independently from the tg.config status.
Application Wrappers now provide a clearly defined interface¶
ApplicationWrapper abstract base class has been defined
to provide a clear interface for application wrappers, all TurboGears
provided application wrappers now adhere this interface.
I18N Translations now provided through an Application Wrapper¶
I18NApplicationWrapper now provides support for translation
detection from browser language and user session. This was previously
builtin into the TurboGears Dispatcher even though it was not related
to dispatching itself.
The behaviour should remain the same apart from the fact that it is now executed before entering the TurboGears application and that some options got renamed:
langoption has been renamed toi18n.lang.i18n_enabledhas been renamed toi18n.enabledbeaker.session.tg_avoid_touchoption has been renamed toi18n.no_session_touchas it is only related to i18n.lang_session_keygot renamed toi18n.lang_session_key.
For a full list of option available please refer to
I18NApplicationWrapper itself.
Session and Cache Middlewares replaced by Application Wrappers¶
The SessionMiddleware and CacheMiddleware were specialized
Beaker middleware for session and caching. To guarantee better
integration with TurboGears and easier configuration they have been
switched to Application Wrappers.
The use_sessions=True option got replaced by session.enabled=True
and an additional cache.enabled=True option has been added.
For a full list of options refer to the CacheApplicationWrapper
and SessionApplicationWrapper references.
To deactivate the application wrappers and switch back to the old middlewares, use:
base_config['session.enabled'] = False
base_config['use_session_middleware'] = True
and:
base_config['cache.enabled'] = False
base_config['use_cache_middleware'] = True
StatusCodeRedirect middleware replaced by ErrorPageApplicationWrapper¶
The StatusCodeRedirect middleware was inherited from Paste project,
and was in charge of intercepting status codes and redirect to an
error page in case of one of those.
So the status_code_redirect=True option got replaced by the
errorpage.enabled=True option. For a full list of options refer
to the ErrorPageApplicationWrapper reference.
As StatusCodeRedirect worked at WSGI level it was pretty slow and
required to read the whole answer just to get the status code.
Also the TurboGears context (request, response, app_globals and so on)
were lost during the execution of the ErrorController.
In 2.3.5 this got replaced by the ErrorPageApplicationWrapper,
which provides the same feature using an Application Wrappers.
If you are still relying on pylons.original_response key in your
ErrorController make sure to uprade to the tg.original_response key,
otherwise it won’t work anymore.
The change should be transparent for most users, in case you want to get back the
old StatusCodeRedirect behaviour you use the following option:
base_config['status_code_redirect'] = True
Keep in mind that the other options from ErrorPageApplicationWrapper
apply and are converted to options for the StatusCodeRedirect
middleware.
Transaction Manager is now an application wrapper¶
Transaction Manager (the component in charge of committing or rolling back your
sqlalchemy transaction) is now replaced by TransactionApplicationWrapper
which is an application wrapper in charge of committing or rolling back the transaction.
So the use_transaction_manager=True option got replaced by
the tm.enabled=True option. For a full list of options refer to the
TransactionApplicationWrapper reference.
There should be no behavioural changes with this change, the only difference is now that the transaction manager applies before the WSGI middlewares as it is managed by TurboGears itself. So if your application was successfull and there was an error in a middleware that happens after (for example ToscaWidgets resource injection) the transaction will be commited anyway as the code that created the objects and for which they should be committed was successful.
If you want to recover back the old TGTransactionManager middleware you can use the following option:
base_config['use_transaction_manager'] = True
TurboGears provides its own ming ODMSession manager as an Application Wrapper¶
The major change is that MingApplicationWrapper now behaves like SQLAlchemy
session when streaming responses.
The session is automatically flushed for you at the end of the request, in case of stramed responses instead you will have to manually manage the session youself if it is used inside the response generator as specified in Streaming Response.
To recover the previous behavior set ming.autoflush=False and replace
the AppConfig.add_ming_middleware method with the following:
def add_ming_middleware(self, app):
import ming.odm.middleware
return ming.odm.middleware.MingMiddleware(app)
From 2.3.3 to 2.3.4¶
JSON Support no longer supports simplegeneric¶
To provide support for customization the json.isodates and json.custom_encoders
options are now available during application configuration. Those are also available
in @expose('json') render_params, see JSON and JSONP Rendering.
lang option is now fallback when i18n is enabled¶
TurboGears provided a lang configuration option which was only meaningful when
i18n was disabled with i18n_enabled = False. The lang option would force the specified
language for the whole web app, independently from user session or browser languages.
Now the lang option when specified is used as the fallback language when i18n is
actually enabled (which is the default).
tg.util is now officially public¶
As tg.util provided utilities that could be useful to app developers the module has been
cleaned up keeping only public features and is now documented at tg.util
From 2.3.2 to 2.3.3¶
abort can now skip error/document and authentication¶
tg.controllers.util.abort() can now provide a
pass-through abort which will answer as is instead of
being intercepted by authentication layer to redirect
to login page or by Error controller to show a custom
error page. This can be helpful when writing API
responses that should just provide output as is.
@require can now be used for allow_only¶
It is now possible to use tg.decorators.require()
as value for controllers allow_only to enable
smart_denial or provide a custom denial_handler
for Controller-level authorization
@require is now a TurboGears decoration¶
@require decorator is now a TurboGears decoration, the order
it is applied won’t matter anymore if other decorators are placed
on the controller.
@beaker_cache is now replaced by @cached¶
@beaker_cache decorator was meant to work on plain function,
the new @cached decorator is meant to work explicitly on TurboGears
controllers. The order the decorator is applied won’t matter anymore
just like the other turbogears decorations.
@beaker_cache is still provided, but it’s use on controllers
is discouraged.
controller_wrappers now get config on call and not on construction¶
Whenever a controller wrapper is registered it won’t get the
app_config parameter anymore on construction, instead it will
receive the configuration as a parameter each time it is called.
The controller wrapper signature has changed as following:
def controller_wrapper(next_caller):
def call(config, controller, remainder, params):
return next_caller(config, controller, remainder, params)
return call
If you still need to access the application configuration into
the controller wrapper constructor, use tg.config.
TurboGears will try to setup the controller wrapper with the new method signature, if it fails it will fallback to the old controller wrappers signature and provide a DeprecationWarning.
get_lang always returns a list¶
Since 2.3.2 get_lang supports the all option, which made possible
to ask TurboGears for all the languages requested by the user to return only
those for which the application supports translation (all=False).
When get_lang(all=True) was called, two different behaviors where
possible: Usually the whole list of languages requested by the user was
returned, unless the application supported no translations. In that case
None was returned.
Now get_lang(all=True) behaves in a more predictable way and always
returns the whole list of languages requested by the user. In case i18n
is not enabled an empty list is returned.
From 2.3.1 to 2.3.2¶
Projects quickstarted on 2.3 should work out of the box.
Kajiki support for TW2 removed¶
If your application is using Kajiki as its primary rendering engine, TW2 widget will now pick the first supported engine instead of Kajiki.
This is due to the fact that recent TW2 version removed support for Kajiki.
AppConfig.setup_mimetypes removed¶
If you were providing custom mimetypes by overriding the setup_mimetypes method
in AppConfig this is not supported anymore. To register custom mimetypes just
declare them in base_config.mimetype_lookup dictionary in your config/app_cfg.py.
Custom rendering engines support refactoring¶
If you were providing a custom rendering engine through AppConfig.setup_NAME_renderer
methods, those are now deprecated. While they should continue to work it is preferred
to update your rendering engine to the new factory based
tg.configuration.AppConfig.register_rendering_engine()
Chameleon Genshi support is now provided by an extension¶
Chameleon Genshi rendering support is now provided by tgext.chameleon_genshi
instead of being bult-in inside TurboGears itself.
Validation error_handlers now call their hooks and wrappers¶
Previous to 2.3.2 controller methods when used as error_handlers didn’t
call their registered hooks and controller wrappers, not if an hook
or controller wrapper is attached to an error handler it will correctly
be called. Only exception is before_validate hook as error_handlers
are not validated.
AppConfig.add_dbsession_remover_middleware renamed¶
If you were providing a custom add_dbsession_remover_middleware method
you should now rename it to add_sqlalchemy_middleware.
Error Reporting options grouped in .ini file¶
Error reporting options have been grouped in trace_errors options.
While previous option names continue to work for backward compatibility, they will be removed in future versions. Email error sending options became:
trace_errors.error_email = you@yourdomain.com
trace_errors.from_address = turbogears@localhost
trace_errors.smtp_server = localhost
trace_errors.smtp_use_tls = true
trace_errors.smtp_username = unknown
trace_errors.smtp_password = unknown
From 2.3 to 2.3.1¶
Projects quickstarted on 2.3 should work out of the box.
AppConfig.register_hook Deprecation¶
register_hook function in application configuration got deprecated
and replaced by tg.hooks.register and tg.hooks.wrap_controller.
register_hook will continue to work like before, but will be removed in
future versions. Check Hooks Guide and upgrade
to tg.hooks based hooks to avoid issues on register_hook removal.
Exposition and Wrappers now resolved lazily¶
Due to Configuration Milestones support controller exposition is now resolved lazily when the configuration process has setup the renderers. This enables a smarter exposition able to correctly behave even when controllers are declared before the application configuration.
Application wrappers dependencies are now solved lazily too, this makes possible to reorder them before applying the actual wrappers so that the order of registration doesn’t mapper when a wrapper ordering is specified.
Some methods in AppConfig got renamed¶
To provide a cleaner distinction between methods users are expected to subclass to customize the configuration process and methods which are part of TurboGears setup itself.
Validation error reporting cleanup¶
TurboGears always provided information on failed validations in a
unorganized manner inside tmpl_context.form_errors and other
locations.
Validation information are now reported in request.validation
dictionary all together. tmpl_context.form_errors and
tmpl_context.form_values are still available but deprecated.
From 2.2 to 2.3¶
Projects quickstarted on 2.2 should mostly work out of the box.
GearBox replaced PasteScript¶
Just by installing gearbox itself your TurboGears project will be able to use gearbox system wide
commands like gearbox serve, gearbox setup-app and gearbox makepackage commands.
These commands provide a replacement for the paster serve, paster setup-app and paster create commands.
The main difference with the paster command is usually only that gearbox commands explicitly set the
configuration file using the --config option instead of accepting it positionally. By default gearbox
will always load a configuration file named development.ini, this mean you can simply run gearbox serve
in place of paster serve development.ini
Gearbox HTTP Servers¶
If you are moving your TurboGears2 project from paster you will probably end serving your
application with Paste HTTP server even if you are using the gearbox serve command.
The reason for this behavior is that gearbox is going to use what is specified inside the server:main section of your .ini file to serve your application. TurboGears2 projects quickstarted before 2.3 used Paste and so the projects is probably configured to use Paste#http as the server. This is not an issue by itself, it will just require you to have Paste installed to be able to serve the application, to totally remove the Paste dependency simply replace Paste#http with gearbox#wsgiref.
Enabling GearBox migrate and tgshell commands¶
To enable gearbox migrate and gearbox tgshell commands make sure that your setup.py entry_points
look like:
entry_points={
'paste.app_factory': [
'main = makonoauth.config.middleware:make_app'
],
'gearbox.plugins': [
'turbogears-devtools = tg.devtools'
]
}
The paste.app_factory section will let gearbox serve know how to create the application that
has to be served. Gearbox relies on PasteDeploy for application setup, so it required a paste.app_factory
section to be able to correctly load the application.
While the gearbox.plugins section will let gearbox itself know that inside that directory the tg.devtools
commands have to be enabled making gearbox tgshell and gearbox migrate available when we run gearbox
from inside our project directory.
Removing Paste dependency¶
When performing python setup.py develop you will notice that Paste will be installed.
To remove such dependency you should remove the setup_requires and paster_plugins
entries from your setup.py:
setup_requires=["PasteScript >= 1.7"],
paster_plugins=['PasteScript', 'Pylons', 'TurboGears2', 'tg.devtools']
WebHelpers Dependency¶
If your project used WebHelpers, the package is not a turbogears dependency anymore,
you should remember to add it to your setup.py dependencies.
Migrations moved from sqlalchemy-migrate to Alembic¶
Due to sqlalchemy-migrate not supporting SQLAlchemy 0.8 and Python 3, the migrations
for newly quickstarted projects will now rely on Alembic. The migrations are now handled
using gearbox migrate command, which supports the same subcommands as the paster migrate one.
The gearbox sqla-migrate command is also provided for backward compatibility for projects that need
to keep using sqlalchemy-migrate.
Pagination module moved from tg.paginate to tg.support.paginate¶
The pagination code, which was previously imported from webhelpers, is now embedded in the
TurboGears distribution, but it changed its exact location.
If you are using tg.paginate.Page manually at the moment, you will have to fix your imports to
be tg.support.paginate.Page.
Anyway, you should preferrably use the decorator approach with tg.decorators.paginate -
then your code will be independent of the TurboGears internals.
From 2.1 to 2.2¶
Projects quickstarted on 2.1 should mostly work out of the box.
Main points of interest when upgrading from 2.1 to 2.2 are related to some features deprecated in 2.1 that now got removed, to the new ToscaWidgets2 support and to the New Authentication layer.
Both ToscaWidgets2 and the new auth layer are disabled by default, so they should not get in your way unless you explicitly want.
Deprecations now removed¶
tg.url changed in release 2.1, in 2.0 parameters for the url could be passed as
paremeters for the tg.url function. This continued to work in 2.1 but provided a
DeprecationWarning. Since 2.1 parameters to the url call must be passed in the params
argument as a dictionary. Support for url parameters passed as arguments have been totally
removed in 2.2
use_legacy_renderer option isn’t supported anymore. Legacy renderers (Buffets) got
deprecated in previous versions and are not available anymore in 2.2.
__before__ and __after__ controller methods got deprecated in 2.1 and are not
called anymore, make sure you switched to the new _before and _after methods.
Avoiding ToscaWidgets2¶
If you want to keep using ToscaWidgets1 simply don’t install ToscaWidgets2 in your enviroment.
If your project has been quickstarted before 2.2 and uses ToscaWidgets1 it can continue to work that way, by default projects that don’t enable tw2 in any way will continue to use ToscaWidgets1.
If you install tw2 packages in your environment the admin interface, sprox, crud and all the
functions related to form generation will switch to ToscaWidgets2.
This will force you to enable tw2 wit the use_toscawidgets2 option, otherwise they will
stop working.
So if need to keep using ToscaWidgets1 only, don’t install any tw2 package.
Mixing ToscaWidgets2 and ToscaWidgets1¶
Mixing the two widgets library is perfectly possible and can be achieved using both the
use_toscawidgets and use_toscawidgets2 options. When ToscaWidgets2 is installed
the admin, sprox and the crud controller will switch to tw2, this will require you to
enable the use_toscawidgets2 option.
If you manually specified any widget inside Sprox forms or CrudRestController you will have to migrate those to tw2. All the other forms in your application can keep being ToscaWidgets1 forms and widgets.
Moving to ToscaWidgets2¶
Switching to tw2 can be achieved by simply placing the prefer_toscawidgets2 option in
your config/app_cfg.py. This will totally disable ToscaWidgets1, being it installed or
not. So all your forms will have to be migrated to ToscaWidgets2.
New Authentication Layer¶
2.2 release introduced a new authentication layer to support repoze.who v2 and prepare for moving forward to Python3. When the new authentication layer is not in use, the old one based on repoze.what, repoze.who v1 and repoze.who-testutil will be used.
As 2.1 applications didn’t explicitly enable the new authentication layer they should continue to work as before.
Switching to the new Authentication Layer¶
Switching to the new authentication layer should be quite straightforward for applications
that didn’t customize authentication. The new layer gets enabled only when a
base_config.sa_auth.authmetadata object is present inside your config/app_cfg.py.
To switch a plain project to the new authentication layer simply add those lines to your
app_cfg.py:
from tg.configuration.auth import TGAuthMetadata
#This tells to TurboGears how to retrieve the data for your user
class ApplicationAuthMetadata(TGAuthMetadata):
def __init__(self, sa_auth):
self.sa_auth = sa_auth
def get_user(self, identity, userid):
return self.sa_auth.dbsession.query(self.sa_auth.user_class).filter_by(user_name=userid).first()
def get_groups(self, identity, userid):
return [g.group_name for g in identity['user'].groups]
def get_permissions(self, identity, userid):
return [p.permission_name for p in identity['user'].permissions]
base_config.sa_auth.authmetadata = ApplicationAuthMetadata(base_config.sa_auth)
If you customized authentication in any way, you will probably have to port forward all your customizations, in this case, if things get too complex you can keep remaining on the old authentication layer, things will continue to work as before.
After enabling the new authentication layer you will have to switch your repoze.what imports to tg imports:
#from repoze.what import predicates becames
from tg import predicates
All the predicates previously available in repoze.what should continue to be available. Your project should now be able to upgrade to repoze.who v2, before doing that remember to remove the following packages which are not in use anymore and might conflict with repoze.who v2:
- repoze.what
- repoze.what.plugins.sql
- repoze.what-pylons
- repoze.what-quickstart
- repoze.who-testutil
The only repoze.who packages you should end up having installed are:
- repoze.who-2.0
- repoze.who.plugins.sa
- repoze.who_friendlyform
Configuration Options¶
This section reports a reference of various configuration options available inside TurboGears app_cfg or .ini file. As they are automatically extracted from the source code this list might be incomplete until the whole configuration process is updated to automatically declare the expected options.
For example if you want to enable isodates in your application JSON encoder
you might want to add to your app_cfg.base_config the following:
base_config['json.isodates'] = True
All the configuration options provided by Application Wrappers can
usually be set both through the app_cfg.py or through your .ini file,
while AppConfig configuration options can only be set through the app_cfg.py
unless explicitly stated otherwise.
AppConfig¶
-
class
tg.configuration.AppConfig(minimal=False, root_controller=None) Class to store application configuration.
This class should have configuration/setup information that is necessary for proper application function. Deployment specific configuration information should go in the config files (e.g. development.ini or deployment.ini).
AppConfig instances have a number of methods that are meant to be overridden by users who wish to have finer grained control over the setup of the WSGI environment in which their application is run.
This is the place to configure your application, database, transaction handling, error handling, etc.
Configuration Options provided:
debug-> Enables / Disables debug mode. Can be set from .ini fileserve_static-> Enable / Disable serving static files. Can be set from .ini fileuse_dotted_templatenames-> Use template names as packages in @expose instead of file paths. This is usually the default unless TG is started in Minimal Mode. Can be set from .ini fileregistry_streaming-> Enable streaming of responses, this is enabled by default. Can be set from .ini filepaths-> Dictionary of directories where templates, static files and controllers are found:{ 'controllers': 'my/path/to/controlllers', 'static_files': 'my/path/to/files', 'templates': ['list/of/paths/to/templates'] )
use_toscawidgets-> Enable ToscaWidgets1, this is deprecated.use_toscawidgets2-> Enable ToscaWidgets2prefer_toscawidgets2-> When both TW2 and TW1 are enabled prefer TW2. Can be set from .ini filecustom_tw2_config-> Dictionary of configuration options for TW2, refer totw2.core.middleware.Configfor available options.auth_backend-> Authentication Backend, can beNone,sqlalchemyorming.sa_auth-> Simple Authentication configuration dictionary. This is a Dictionary that contains the configuration options forrepoze.who, see Identification & Authentication Layer for available options. Basic options include:cookie_secret-> Secret phrase used to verify auth cookies.authmetadata-> Authentication and User Metadata Provider for TurboGearspost_login_url-> Redirect users here after loginpost_logout_url-> Redirect users here when they logout
package-> Application Package, this is used to configure paths as being inside a pythonapp_globals-> Application Globals class, by default build frompackage.lib.app_globals. package. Which enables serving templates, controllers, app globals and so on from the package itself.helpers-> Template Helpers, by defaultpackage.lib.helpersis used.model-> The models module (or object) where all the models, DBSession and init_models method areavailable. By default
package.modelis used.
renderers-> List of enabled renderers names.default_renderer-> When not specified, use this renderer for templates.auto_reload_templates-> Automatically reload templates when modified (disable this on production for a performance gain). Can be set from .ini fileuse_ming-> Enable/Disable Ming as Models storage.ming.url-> Url of the MongoDB databaseming.db-> If Database is not provided inming.urlit can be specified here.ming.connection.*-> Options to configure the ming connection, refer toming.datastore.create_datastore()for available options.use_sqlalchemy-> Enable/Disable SQLalchemy as Models storage.sqlalchemy.url-> Url of the SQLAlchemy database. Refer to SQLAlchemy Master Slave Load Balancing for configuring master-slave urls.
Error Reporting¶
-
tg.error.ErrorReporter(app, global_conf, **errorware)¶ Provides Error reporting through Backlash on TurboGears.
This is enabled/disabled through the
debugconfiguration option. Currently EMail and Sentry backlash reporters can be enabled.All the options available for error reporting are configured as
trace_errors.*options in yourapp_cfgor.inifiles.The available options for EMail reporter are:
trace_errors.enable-> Enable or disable error reporting, by default is enabled if backlash is available anddebug=falsetrace_errors.smtp_server-> SMTP Server to connect to for sending emailstrace_errors.smtp_port-> SMTP port to connect totrace_errors.from_address-> Address sending the error emailstrace_errors.error_email-> Address the error emails should be sent to.trace_errors.smtp_username-> Username to authenticate on SMTP server.trace_errors.smtp_password-> Password to authenticate on SMTP server.trace_errors.smtp_use_tls-> Whenever to enable or not TLS for SMTP.trace_errors.error_subject_prefix-> Prefix to append to error emails, by defaultWebApp Error:is prepended.trace_errors.dump_request-> Whenever to attach a request dump to the email so that all request data is provided.trace_errors.dump_request_size-> Do not dump request if it’s bigger than this value, useful for uploaded files. By default 50K.trace_errors.dump_local_frames-> Enable dumping local variables in case of crashes.trace_errors.dump_local_frames_count-> Dump up to X frames when dumping local variables. The default is 2
Available options for Sentry reporter are:
trace_errors.sentry_dsn-> Sentry instance where to send the errors.
Slow Requests Reporting¶
-
tg.error.SlowReqsReporter(app, global_conf, **errorware)¶ Provides slow requests reporting for TurboGears through BackLash.
This is enabled through the
trace_slowreqs.enableoption and is only enabled whendebug=false.All the options available for error reporting are configured as
trace_slowreqs.*options in yourapp_cfgor.inifiles:trace_slowreqs.enable-> Enable/Disable slow requests reporting, by default it’s disabled.trace_slowreqs.interval-> Report requests slower than this value (default: 25s)trace_slowreqs.exclude-> List of urls that should be excluded
Slow requests are reported using EMail or Sentry, the same options available in
ErrorReporterapply withtrace_slowreqs.instead oftrace_errors..
JSON Encoding¶
-
JSONEncoder.configure(isodates=False, custom_encoders=None, allow_lists=False, **kwargs) JSON encoder can be configured through
AppConfig(app_cfg.base_config) using the following options:json.isodates-> encode dates using ISO8601 formatjson.custom_encoders-> List of tuples(type, encode_func)to register custom encoders for specific types.json.allow_lists-> Allows lists to be encoded, this is usually disabled for security reasons due to JSON hijacking. See http://stackoverflow.com/questions/16289894 for additional details.
Flash Messages¶
-
TGFlash.configure(cookie_name='webflash', default_status='ok', template=<string.Template object>, js_call='webflash.render()', js_template=<string.Template object>, allow_html=False) Flash messages can be configured through
AppConfig(app_cfg.base_config) using the following options:flash.cookie_name-> Name of the cookie used to store flash messagesflash.default_status-> Default message status if not specified (okby default)flash.template->string.Templateinstance used as the flash template when rendered from server side, will receive$container_id,$messageand$statusvariables.flash.allow_html-> Turns on/off escaping in flash messages, by default HTML is not allowed.flash.js_call-> javascript code which will be run when displaying the flash from javascript. Default iswebflash.render(), you can usewebflash.payload()to retrieve the message and show it with your favourite library.flash.js_template->string.Templateinstance used to replace full javascript support for flash messages. When rendering flash message for javascript usage the following code will be used instead of providing the standardwebflashobject. If you replacejs_templateyou must also ensure cookie parsing and delete it for already displayed messages. The template will receive:$container_id,$cookie_name,$js_callvariables.
Sessions¶
-
class
tg.appwrappers.session.SessionApplicationWrapper(handler, config)¶ Provides the Session Support
The Session Application Wrapper will make a lazy session instance available every request under the
environ['beaker.session']key and inside TurboGears context assession.- Supported options which can be provided by config are:
session.enabled: Whenever sessions are enabled or not.- Beaker Options prefixed with
session., see https://beaker.readthedocs.org/en/latest/configuration.html#session-options
Caching¶
-
class
tg.appwrappers.caching.CacheApplicationWrapper(handler, config)¶ Provides Caching Support.
The Cache Application Wrapper will make a CacheManager instance available every request under the
environ['beaker.cache']key and inside the TurboGears request context ascache.- Supported options which can be provided by config are:
cache.enabled: Whenever caching is enabled or not.- Beaker Options prefixed with
cache., see https://beaker.readthedocs.org/en/latest/configuration.html#cache-options
Internationalization¶
-
class
tg.appwrappers.i18n.I18NApplicationWrapper(handler, config)¶ Provides Language detection from request and session.
The session language(s) take priority over the request languages.
- Supported options which can be provided by config are:
i18n.enabled: Whenever language detection is enabled or not.i18n.lang: Fallback language for the application, works both when language detection is enabled or disabled. If this is set and language detection is dislabled, the application will consider that all gettext wrapped strings must be translated to this language.i18n.lang_session_key: Session key from which to read the saved language (tg_langby default).i18n.no_session_touch: Avoid causing a session save when reading it to retrieve the favourite user language. This isFalseby default, setting it toFalsecauses TurboGears to save and update the session for each request.
User Identity¶
-
class
tg.appwrappers.identity.IdentityApplicationWrapper(handler, config)¶ Provides user identity when authentication is enabled.
The repoze.who provided identity takes precedence over the identity provided by IdentityApplicationWrapper if available.
- Supported options which can be provided by config are:
sa_auth.authmetadata: The TGAuthMetadata object that should be used to retrieve identity metadata.identity.enabled: Enable the Identity Application Wrapper. By default enabled if authmetadata available.identity.allow_missing_user: Whenever the identity should be discarded or not when the authmetadata is unable to find an user.
Transaction Manager¶
-
class
tg.appwrappers.transaction_manager.TransactionApplicationWrapper(handler, config)¶ Wraps the whole application in zope.transaction transaction manager and rollbacks transaction in case of crashes.
Supported options which can be provided by config are:
tm.enabled: Wenever the transaction manager is enabled or not.tm.attempts: Number of times the transaction should be retried if it fails (no retry by default)tm.commit_veto: A function that will be called for every transaction to check if it should abort transaction or let it go. Function signature should be:function(environ, status_code, headers) -> bool.
Custom Error Pages¶
-
class
tg.appwrappers.errorpage.ErrorPageApplicationWrapper(handler, config)¶ Given an Application it intercepts the response code and shows a custom page.
Supported options are:
errorpage.enabled: Whenever the custom error page is enabled or not.errorpage.status_codes: List of HTTP errors that should be trapped. By default 403, 404, 500.errorpage.content_types: List of Content-Types for which the custom error page should be displayed. IE:["text/html"]. An empty list means all. An entry with valueNonemeans no content type provided. Default is["text/html", None].errorpage.handle_exceptions: Whenever exceptions should be trapped and treated as a 500 error or not. By default this isTruewhendebug=false.errorapge.path: Path of the controller should be displayed in case of errors. By default/error/document.
Ming Session Manager¶
-
class
tg.appwrappers.mingflush.MingApplicationWrapper(handler, config)¶ Automatically flushes the Ming ODMSession.
In case an exception raised during excution it won’t flush the session and it will instead close it throwing away any change.
Supported options which can be provided by config are:
ming.autoflush: Whenever to flush session at end of request if no exceptions happened.
Rendering Engines¶
Genshi¶
-
class
tg.renderers.genshi.GenshiRenderer(loader, config)¶ Configuration Options available as
templating.genshi.*:templating.genshi.name_constant_patch-> Enable/Disable patch for Python3.4 compatibility.templating.genshi.max_cache_size-> Maximum number of templates to keep cached, by default 30.templating.genshi.method-> Genshi rendering method (html or xhtml).
Supported
render_params:- Caching options supported by
cached_template() doctype-> To override the global doctypemethod-> To override the global rendering method
Kajiki¶
-
class
tg.renderers.kajiki.KajikiRenderer(loader)¶ Configuration Options available as
templating.kajiki.*:templating.kajiki.force_mode-> Kajiki Rendering Mode (html, html5, xml). Defaulthtml5.templating.kajiki.template_extension-> Kajiki Templates extension, default.xhtmltemplating.kajiki.xml_autoblocks-> List of tags that should be automatically converted to blocks.templating.kajiki.cdata_scripts-> Automatically wrap scripts in CDATA.templating.kajiki.html_optional_tags-> Allow unclosed html, head and body tags.templating.kajiki.strip_text-> Strip leading/trailing spaces from text nodes.
Supported
render_params:- Caching options supported by
cached_template() - All arguments supported by
kajiki.xml_template.XMLTemplate()
Jinja2¶
-
class
tg.renderers.jinja.JinjaRenderer(jinja2_env)¶ Currently Jinja2 support uses a bunch of options from the AppConfig and doesn’t provide its own namespace.
Mako¶
-
class
tg.renderers.mako.MakoRenderer(use_dotted_templatenames, template_extension, dotted_loader, normal_loader)¶ Configuration Options available as
templating.mako.*:templating.mako.template_extension-> Mako Templates extension, default.maktemplating.mako.compiled_templates_dir-> Where to store mako precompiled templates. By default templates are only stored in memory and not on disk.
JSON¶
-
class
tg.renderers.json.JSONRenderer¶ JSON rendering can be configured using options supported by
JSONEncoder.configure()Supported
render_params:- All supported by
JSONEncoder.configure() key-> Render a single key of the dictionary returned by controller instead of rendering the dictionary itself.callback_param-> Name of the callback to call in rendered JS for jsonp
- All supported by
Classes and Functions¶
This page provides a quick access reference to the classes and functions provided by TurboGears
Decorators¶
Decorators use by the TurboGears controllers.
Not all of these decorators are traditional wrappers. They are much simplified from the TurboGears 1 decorators, because all they do is register attributes on the functions they wrap, and then the DecoratedController provides the hooks needed to support these decorators.
-
class
tg.decorators.Decoration(controller)¶ Simple class to support ‘simple registration’ type decorators
-
lookup_template_engine(tgl)¶ Return the template engine data.
Provides a convenience method to get the proper engine, content_type, template, and exclude_names for a particular tg_format (which is pulled off of the request headers).
-
register_custom_template_engine(custom_format, content_type, engine, template, exclude_names, render_params)¶ Registers a custom engine on the controller.
Multiple engines can be registered, but only one engine per custom_format.
The engine is registered when @expose is used with the custom_format parameter and controllers render using this engine when the use_custom_format() function is called with the corresponding custom_format.
exclude_names keeps track of a list of keys which will be removed from the controller’s dictionary before it is loaded into the template. This allows you to exclude some information from JSONification, and other ‘automatic’ engines which don’t require a template.
render_params registers extra parameters which will be sent to the rendering method. This allows you to influence things like the rendering method or the injected doctype.
-
register_template_engine(content_type, engine, template, exclude_names, render_params)¶ Registers an engine on the controller.
Multiple engines can be registered, but only one engine per content_type. If no content type is specified the engine is registered at / which is the default, and will be used whenever no content type is specified.
exclude_names keeps track of a list of keys which will be removed from the controller’s dictionary before it is loaded into the template. This allows you to exclude some information from JSONification, and other ‘automatic’ engines which don’t require a template.
render_params registers extra parameters which will be sent to the rendering method. This allows you to influence things like the rendering method or the injected doctype.
-
-
class
tg.decorators.after_render(hook_func)¶ A list of callables to be run after the template is rendered.
Will be run before it is returned returned up the WSGI stack.
-
class
tg.decorators.before_call(hook_func)¶ A list of callables to be run before the controller method is called.
-
class
tg.decorators.before_render(hook_func)¶ A list of callables to be run before the template is rendered.
-
class
tg.decorators.before_validate(hook_func)¶ A list of callables to be run before validation is performed.
-
class
tg.decorators.cached(key=<class 'tg.support.NoDefault'>, expire='never', type=None, query_args=None, cache_headers=('content-type', 'content-length'), invalidate_on_startup=False, cache_response=True, **b_kwargs)¶ Decorator to cache the controller.
The namespace and cache key used to cache the controller are available asrequest.caching.namespaceandrequest.caching.key. This only caches the controller, not the template, validation or the hooks associated to the controller. If you also want to cache template remember to returntg_cacheoption with the same cache key from the controller.The following parameters are accepted:
key- Specifies the controller parameters used to generate the cache key.NoDefault - Uses function name and parameters (excluding *args) as the key (default)
None - No variable key, uses only function name as key
string - Use function name and only “key” parameter
list - Use function name and all parameters listed
expire- Time in seconds before cache expires, or the string “never”. Defaults to “never”
type- Type of cache to use: dbm, memory, file, memcached, or None for Beaker’s default
cache_headers- A tuple of header names indicating response headers that will also be cached.
invalidate_on_startup- If True, the cache will be invalidated each time the application starts or is restarted.
cache_responseDetermines whether the response at the time the cache is used should be cached or not, defaults to True.
Note
When cache_response is set to False, the cache_headers argument is ignored as none of the response is cached.
-
class
tg.decorators.decode_params(format='json')¶ Decorator that enables parsing parameters from request body.
By default the arguments are parsed in JSON format (which is currently the only supported format).
-
class
tg.decorators.expose(template='', content_type=None, exclude_names=None, custom_format=None, render_params=None, inherit=False)¶ Register attributes on the decorated function.
Parameters: - template
Assign a template, you could use the syntax ‘genshi:template’ to use different templates. The default template engine is genshi.
- content_type
Assign content type. The default content type is ‘text/html’.
- exclude_names
Assign exclude names
- custom_format
Registers as a custom format which can later be activated calling use_custom_format
- render_params
Assign parameters that shall be passed to the rendering method.
- inherit
Inherit all the decorations from the same method in the parent class. This will let the exposed method expose the same template as the overridden method template and keep the same hooks and validation that the parent method had.
The expose decorator registers a number of attributes on the decorated function, but does not actually wrap the function the way TurboGears 1.0 style expose decorators did.
This means that we don’t have to play any kind of special tricks to maintain the signature of the exposed function.
The exclude_names parameter is new, and it takes a list of keys that ought to be scrubbed from the dictionary before passing it on to the rendering engine. This is particularly useful for JSON.
The render_parameters is also new. It takes a dictionary of arguments that ought to be sent to the rendering engine, like this:
render_params={'method': 'xml', 'doctype': None}
Expose decorator can be stacked like this:
@expose('json', exclude_names='d') @expose('kid:blogtutorial.templates.test_form', content_type='text/html') @expose('kid:blogtutorial.templates.test_form_xml', content_type='text/xml', custom_format='special_xml') def my_exposed_method(self): return dict(a=1, b=2, d="username")
The expose(‘json’) syntax is a special case. json is a rendering engine, but unlike others it does not require a template, and expose assumes that it matches content_type=’application/json’
If you want to declare a desired content_type in a url, you can use the mime-type style dotted notation:
"/mypage.json" ==> for json "/mypage.html" ==> for text/html "/mypage.xml" ==> for xml.
If you’re doing an http post, you can also declare the desired content type in the accept headers, with standard content type strings.
By default expose assumes that the template is for html. All other content_types must be explicitly matched to a template and engine.
The last expose decorator example uses the custom_format parameter which takes an arbitrary value (in this case ‘special_xml’). You can then use the
use_custom_format()function within the method to decide which of the ‘custom_format’ registered expose decorators to use to render the template.
-
tg.decorators.override_template(view, template)¶ Override the template to be used.
Use override_template in a controller method in order to change the template that will be used to render the response dictionary dynamically.
The
viewargument is the actual controller method for which you want to replace the template.The
templatestring passed in requires that you include the template engine name, even if you’re using the default.So you have to pass in a template id string like:
"genshi:myproject.templates.index2"future versions may make the genshi: optional if you want to use the default engine.
-
class
tg.decorators.paginate(name, use_prefix=False, items_per_page=10, max_items_per_page=0)¶ Paginate a given collection.
This decorator is mainly exposing the functionality of
webhelpers.paginate().Usage: You use this decorator as follows:
class MyController(object): @expose() @paginate("collection") def sample(self, *args): collection = get_a_collection() return dict(collection=collection)
To render the actual pager, use:
${tmpl_context.paginators.<name>.pager()}
It is possible to have several
paginate()-decorators for one controller action to paginate several collections independently from each other. If this is desired, don’t forget to set theuse_prefix-parameter toTrue.Parameters: - name
the collection to be paginated.
- items_per_page
the number of items to be rendered. Defaults to 10
- max_items_per_page
the maximum number of items allowed to be set via parameter. Defaults to 0 (does not allow to change that value).
- use_prefix
if True, the parameters the paginate decorator renders and reacts to are prefixed with “<name>_”. This allows for multi-pagination.
-
class
tg.decorators.require(predicate, denial_handler=None, smart_denial=False)¶ Decorator that checks if the specified predicate it met, if it isn’t it calls the denial_handler to prevent access to the decorated method.
The default authorization denial handler of this protector will flash the message of the unmet predicate with
warningorerroras the flash status if the HTTP status code is 401 or 403, respectively.Parameters: - predicate – An object with a check_authorization(environ) method which must raise a tg.predicates.NotAuthorizedError if not met.
- denial_handler – The callable to be run if authorization is
denied (overrides
default_denial_handlerif defined). - smart_denial – A list of response types for which to trigger
the smart denial, which will act as an API providing a pass-through
tg.controllers.util.abort(). IfTrue,('application/json', 'text/xml')will be used.
If called,
denial_handlerwill be passed a positional argument which represents a message on why authorization was denied.Use
allow_onlyproperty ofTGControllerfor controller-wide authorization.-
default_denial_handler(reason)¶ Authorization denial handler for protectors.
-
tg.decorators.use_custom_format(controller, custom_format)¶ Use use_custom_format in a controller in order to change the active @expose decorator when available.
-
class
tg.decorators.validate(validators=None, error_handler=None, form=None, chain_validation=False)¶ Registers which validators ought to be applied.
If you want to validate the contents of your form, you can use the
@validate()decorator to register the validators that ought to be called.Parameters: - validators – A dictionary of FormEncode/TW2 validators, a
tg.validation.Convertor any callable that might throwtg.validation.TGValidationError. - error_handler – Function or action that should be used to handle the errors.
- form – A TW2 or ToscaWidgets form to validate ( to be provided instead of
validators) - chain_validation – Whenever
error_handlershould perform validation too in case it’s a controller action or not. By default it’s disabled.
The first positional parameter can either be a dictonary of validators, a FormEncode schema validator, or a callable which acts like a FormEncode validator.
- validators – A dictionary of FormEncode/TW2 validators, a
-
class
tg.decorators.with_engine(engine_name=None, master_params=None)¶ Decorator to force usage of a specific database engine in TurboGears SQLAlchemy BalancedSession.
Parameters: - engine_name – ‘master’ or the name of one of the slaves, if is
Noneit will not force any specific engine. - master_params – A dictionary or GET parameters that when present will force usage of the master node. The keys of the dictionary will be the name of the parameters to look for, while the values must be whenever to pop the parameter from the parameters passed to the controller (True/False). If master_params is a list then it is converted to a dictionary where the keys are the entries of the list and the value is always True.
- engine_name – ‘master’ or the name of one of the slaves, if is
-
class
tg.caching.cached_property(func)¶ Works like python @property but the decorated function only gets executed once, successive accesses to the property will just return the value previously stored into the object.
The
@cached_propertydecorator can be executed within a provided context, for example to make the cached property thread safe a Lock can be provided:from threading import Lock from tg.caching import cached_property class MyClass(object): @cached_property def my_property(self): return 'Value!' my_property.context = Lock()
Validation¶
-
class
tg.decorators.validate(validators=None, error_handler=None, form=None, chain_validation=False) Registers which validators ought to be applied.
If you want to validate the contents of your form, you can use the
@validate()decorator to register the validators that ought to be called.Parameters: - validators – A dictionary of FormEncode/TW2 validators, a
tg.validation.Convertor any callable that might throwtg.validation.TGValidationError. - error_handler – Function or action that should be used to handle the errors.
- form – A TW2 or ToscaWidgets form to validate ( to be provided instead of
validators) - chain_validation – Whenever
error_handlershould perform validation too in case it’s a controller action or not. By default it’s disabled.
The first positional parameter can either be a dictonary of validators, a FormEncode schema validator, or a callable which acts like a FormEncode validator.
- validators – A dictionary of FormEncode/TW2 validators, a
-
class
tg.validation.Convert(func, msg=<tg.util.lazystring.LazyString object>, default=None)¶ Applies a conversion function as a validator.
This is meant to implement simple validation mechanism.
Any callable can be used for
funcas far as it accepts an argument and returns the converted object. In case of exceptions the validation is considered failed and themsgparameter is displayed as an error.A
defaultvalue can be provided for values that are missing (evaluate to false) which will be used in place of the missing value.Example:
@expose() @validate({ 'num': Convert(int, 'Must be a number') }, error_handler=insert_number) def post_pow2(self, num): return str(num*num)
-
exception
tg.validation.TGValidationError(msg, value=None, error_dict=None)¶ Invalid data was encountered during validation.
The constructor can be passed a short message with the reason of the failed validation.
Authorization¶
-
class
tg.decorators.require(predicate, denial_handler=None, smart_denial=False) Decorator that checks if the specified predicate it met, if it isn’t it calls the denial_handler to prevent access to the decorated method.
The default authorization denial handler of this protector will flash the message of the unmet predicate with
warningorerroras the flash status if the HTTP status code is 401 or 403, respectively.Parameters: - predicate – An object with a check_authorization(environ) method which must raise a tg.predicates.NotAuthorizedError if not met.
- denial_handler – The callable to be run if authorization is
denied (overrides
default_denial_handlerif defined). - smart_denial – A list of response types for which to trigger
the smart denial, which will act as an API providing a pass-through
tg.controllers.util.abort(). IfTrue,('application/json', 'text/xml')will be used.
If called,
denial_handlerwill be passed a positional argument which represents a message on why authorization was denied.Use
allow_onlyproperty ofTGControllerfor controller-wide authorization.
Built-in predicate checkers.
This is mostly took from repoze.what.precidates
This is module provides the predicate checkers that were present in the original “identity” framework of TurboGears 1, plus others.
-
class
tg.predicates.CompoundPredicate(*predicates, **kwargs)¶ A predicate composed of other predicates.
-
class
tg.predicates.All(*predicates, **kwargs)¶ Check that all of the specified predicates are met.
Parameters: predicates – All of the predicates that must be met. Example:
# Grant access if the current month is July and the user belongs to # the human resources group. p = All(is_month(7), in_group('hr'))
-
evaluate(environ, credentials)¶ Evaluate all the predicates it contains.
Parameters: - environ – The WSGI environment.
- credentials – The
repoze.whatcredentials.
Raises: NotAuthorizedError – If one of the predicates is not met.
-
-
class
tg.predicates.Any(*predicates, **kwargs)¶ Check that at least one of the specified predicates is met.
Parameters: predicates – Any of the predicates that must be met. Example:
# Grant access if the currest user is Richard Stallman or Linus # Torvalds. p = Any(is_user('rms'), is_user('linus'))
-
evaluate(environ, credentials)¶ Evaluate all the predicates it contains.
Parameters: - environ – The WSGI environment.
- credentials – The
repoze.whatcredentials.
Raises: NotAuthorizedError – If none of the predicates is met.
-
-
class
tg.predicates.has_all_permissions(*permissions, **kwargs)¶ Check that the current user has been granted all of the specified permissions.
Parameters: permissions – The names of all the permissions that must be granted to the user. Example:
p = has_all_permissions('view-users', 'edit-users')
-
class
tg.predicates.has_any_permission(*permissions, **kwargs)¶ Check that the user has at least one of the specified permissions.
Parameters: permissions – The names of any of the permissions that have to be granted to the user. Example:
p = has_any_permission('manage-users', 'edit-users')
-
class
tg.predicates.has_permission(permission_name, **kwargs)¶ Check that the current user has the specified permission.
Parameters: permission_name – The name of the permission that must be granted to the user. Example:
p = has_permission('hire')
-
class
tg.predicates.in_all_groups(*groups, **kwargs)¶ Check that the user belongs to all of the specified groups.
Parameters: groups – The name of all the groups the user must belong to. Example:
p = in_all_groups('developers', 'designers')
-
class
tg.predicates.in_any_group(*groups, **kwargs)¶ Check that the user belongs to at least one of the specified groups.
Parameters: groups – The name of any of the groups the user may belong to. Example:
p = in_any_group('directors', 'hr')
-
class
tg.predicates.in_group(group_name, **kwargs)¶ Check that the user belongs to the specified group.
Parameters: group_name (str) – The name of the group to which the user must belong. Example:
p = in_group('customers')
-
class
tg.predicates.is_user(user_name, **kwargs)¶ Check that the authenticated user’s username is the specified one.
Parameters: user_name (str) – The required user name. Example:
p = is_user('linus')
-
class
tg.predicates.is_anonymous(msg=None)¶ Check that the current user is anonymous.
Example:
# The user must be anonymous! p = is_anonymous()
New in version 1.0.7.
-
class
tg.predicates.not_anonymous(msg=None)¶ Check that the current user has been authenticated.
Example:
# The user must have been authenticated! p = not_anonymous()
Pagination¶
-
class
tg.decorators.paginate(name, use_prefix=False, items_per_page=10, max_items_per_page=0) Paginate a given collection.
This decorator is mainly exposing the functionality of
webhelpers.paginate().Usage: You use this decorator as follows:
class MyController(object): @expose() @paginate("collection") def sample(self, *args): collection = get_a_collection() return dict(collection=collection)
To render the actual pager, use:
${tmpl_context.paginators.<name>.pager()}
It is possible to have several
paginate()-decorators for one controller action to paginate several collections independently from each other. If this is desired, don’t forget to set theuse_prefix-parameter toTrue.Parameters: - name
the collection to be paginated.
- items_per_page
the number of items to be rendered. Defaults to 10
- max_items_per_page
the maximum number of items allowed to be set via parameter. Defaults to 0 (does not allow to change that value).
- use_prefix
if True, the parameters the paginate decorator renders and reacts to are prefixed with “<name>_”. This allows for multi-pagination.
-
class
tg.support.paginate.Page(collection, page=1, items_per_page=20)¶ TurboGears Pagination support for @paginate decorator. It is based on a striped down version of the WebHelpers pagination class This represents a page inside a collection of items
-
pager(format='~2~', page_param='page', partial_param='partial', show_if_single_page=False, separator=' ', onclick=None, symbol_first='<<', symbol_last='>>', symbol_previous='<', symbol_next='>', link_attr=None, curpage_attr=None, dotdot_attr=None, page_link_template='<a%s>%s</a>', page_plain_template='<span%s>%s</span>', **kwargs)¶ Return string with links to other pages (e.g. “1 2 [3] 4 5 6 7”).
- format:
Format string that defines how the pager is rendered. The string can contain the following $-tokens that are substituted by the string.Template module:
- $first_page: number of first reachable page
- $last_page: number of last reachable page
- $page: number of currently selected page
- $page_count: number of reachable pages
- $items_per_page: maximal number of items per page
- $first_item: index of first item on the current page
- $last_item: index of last item on the current page
- $item_count: total number of items
- $link_first: link to first page (unless this is first page)
- $link_last: link to last page (unless this is last page)
- $link_previous: link to previous page (unless this is first page)
- $link_next: link to next page (unless this is last page)
To render a range of pages the token ‘~3~’ can be used. The number sets the radius of pages around the current page. Example for a range with radius 3:
‘1 .. 5 6 7 [8] 9 10 11 .. 500’
Default: ‘~2~’
- symbol_first
String to be displayed as the text for the %(link_first)s link above.
Default: ‘<<’
- symbol_last
String to be displayed as the text for the %(link_last)s link above.
Default: ‘>>’
- symbol_previous
String to be displayed as the text for the %(link_previous)s link above.
Default: ‘<’
- symbol_next
String to be displayed as the text for the %(link_next)s link above.
Default: ‘>’
- separator:
String that is used to separate page links/numbers in the above range of pages.
Default: ‘ ‘
- page_param:
- The name of the parameter that will carry the number of the page the user just clicked on.
- partial_param:
When using AJAX/AJAH to do partial updates of the page area the application has to know whether a partial update (only the area to be replaced) or a full update (reloading the whole page) is required. So this parameter is the name of the URL parameter that gets set to 1 if the ‘onclick’ parameter is used. So if the user requests a new page through a Javascript action (onclick) then this parameter gets set and the application is supposed to return a partial content. And without Javascript this parameter is not set. The application thus has to check for the existence of this parameter to determine whether only a partial or a full page needs to be returned. See also the examples in this modules docstring.
Default: ‘partial’
Note: If you set this argument and are using a URL generator callback, the callback must accept this name as an argument instead of ‘partial’.
- show_if_single_page:
if True the navigator will be shown even if there is only one page
Default: False
- link_attr (optional)
A dictionary of attributes that get added to A-HREF links pointing to other pages. Can be used to define a CSS style or class to customize the look of links.
Example: { ‘style’:’border: 1px solid green’ }
Default: { ‘class’:’pager_link’ }
- curpage_attr (optional)
A dictionary of attributes that get added to the current page number in the pager (which is obviously not a link). If this dictionary is not empty then the elements will be wrapped in a SPAN tag with the given attributes.
Example: { ‘style’:’border: 3px solid blue’ }
Default: { ‘class’:’pager_curpage’ }
- dotdot_attr (optional)
A dictionary of attributes that get added to the ‘..’ string in the pager (which is obviously not a link). If this dictionary is not empty then the elements will be wrapped in a SPAN tag with the given attributes.
Example: { ‘style’:’color: #808080’ }
Default: { ‘class’:’pager_dotdot’ }
- page_link_template (optional)
A string with the template used to render page links
Default: ‘<a%s>%s</a>’
- page_plain_template (optional)
A string with the template used to render current page, and dots in pagination.
Default: ‘<span%s>%s</span>’
- onclick (optional)
This paramter is a string containing optional Javascript code that will be used as the ‘onclick’ action of each pager link. It can be used to enhance your pager with AJAX actions loading another page into a DOM object.
In this string the variable ‘$partial_url’ will be replaced by the URL linking to the desired page with an added ‘partial=1’ parameter (or whatever you set ‘partial_param’ to). In addition the ‘$page’ variable gets replaced by the respective page number.
Note that the URL to the destination page contains a ‘partial_param’ parameter so that you can distinguish between AJAX requests (just refreshing the paginated area of your page) and full requests (loading the whole new page).
[Backward compatibility: you can use ‘%s’ instead of ‘$partial_url’]
- jQuery example:
- “$(‘#my-page-area’).load(‘$partial_url’); return false;”
- Yahoo UI example:
- “YAHOO.util.Connect.asyncRequest(‘GET’,’$partial_url’,{
- success:function(o){YAHOO.util.Dom.get(‘#my-page-area’).innerHTML=o.responseText;} },null); return false;”
- scriptaculous example:
- “new Ajax.Updater(‘#my-page-area’, ‘$partial_url’,
- {asynchronous:true, evalScripts:true}); return false;”
- ExtJS example:
- “Ext.get(‘#my-page-area’).load({url:’$partial_url’}); return false;”
- Custom example:
- “my_load_page($page)”
Additional keyword arguments are used as arguments in the links.
-
Configuration¶
-
class
tg.configuration.AppConfig(minimal=False, root_controller=None)¶ Class to store application configuration.
This class should have configuration/setup information that is necessary for proper application function. Deployment specific configuration information should go in the config files (e.g. development.ini or deployment.ini).
AppConfig instances have a number of methods that are meant to be overridden by users who wish to have finer grained control over the setup of the WSGI environment in which their application is run.
This is the place to configure your application, database, transaction handling, error handling, etc.
Configuration Options provided:
debug-> Enables / Disables debug mode. Can be set from .ini fileserve_static-> Enable / Disable serving static files. Can be set from .ini fileuse_dotted_templatenames-> Use template names as packages in @expose instead of file paths. This is usually the default unless TG is started in Minimal Mode. Can be set from .ini fileregistry_streaming-> Enable streaming of responses, this is enabled by default. Can be set from .ini filepaths-> Dictionary of directories where templates, static files and controllers are found:{ 'controllers': 'my/path/to/controlllers', 'static_files': 'my/path/to/files', 'templates': ['list/of/paths/to/templates'] )
use_toscawidgets-> Enable ToscaWidgets1, this is deprecated.use_toscawidgets2-> Enable ToscaWidgets2prefer_toscawidgets2-> When both TW2 and TW1 are enabled prefer TW2. Can be set from .ini filecustom_tw2_config-> Dictionary of configuration options for TW2, refer totw2.core.middleware.Configfor available options.auth_backend-> Authentication Backend, can beNone,sqlalchemyorming.sa_auth-> Simple Authentication configuration dictionary. This is a Dictionary that contains the configuration options forrepoze.who, see Identification & Authentication Layer for available options. Basic options include:cookie_secret-> Secret phrase used to verify auth cookies.authmetadata-> Authentication and User Metadata Provider for TurboGearspost_login_url-> Redirect users here after loginpost_logout_url-> Redirect users here when they logout
package-> Application Package, this is used to configure paths as being inside a pythonapp_globals-> Application Globals class, by default build frompackage.lib.app_globals. package. Which enables serving templates, controllers, app globals and so on from the package itself.helpers-> Template Helpers, by defaultpackage.lib.helpersis used.model-> The models module (or object) where all the models, DBSession and init_models method areavailable. By default
package.modelis used.
renderers-> List of enabled renderers names.default_renderer-> When not specified, use this renderer for templates.auto_reload_templates-> Automatically reload templates when modified (disable this on production for a performance gain). Can be set from .ini fileuse_ming-> Enable/Disable Ming as Models storage.ming.url-> Url of the MongoDB databaseming.db-> If Database is not provided inming.urlit can be specified here.ming.connection.*-> Options to configure the ming connection, refer toming.datastore.create_datastore()for available options.use_sqlalchemy-> Enable/Disable SQLalchemy as Models storage.sqlalchemy.url-> Url of the SQLAlchemy database. Refer to SQLAlchemy Master Slave Load Balancing for configuring master-slave urls.
-
after_init_config(conf)¶ Override this method to set up configuration variables at the application level. This method will be called after your configuration object has been initialized on startup. Here is how you would use it to override the default setting of tg.strict_tmpl_context
from tg.configuration import AppConfig class MyAppConfig(AppConfig): def after_init_config(self, conf): conf['tg.strict_tmpl_context'] = False base_config = MyAppConfig()
-
make_load_environment()¶ Return a load_environment function.
The returned load_environment function can be called to configure the TurboGears runtime environment for this particular application. You can do this dynamically with multiple nested TG applications if necessary.
-
register_controller_wrapper(wrapper, controller=None)¶ Registers a TurboGears controller wrapper.
Controller Wrappers are much like a decorator applied to every controller. They receive
tg.configuration.AppConfiginstance as an argument and the next handler in chain and are expected to return a new handler that performs whatever it requires and then calls the next handler.A simple example for a controller wrapper is a simple logging wrapper:
def controller_wrapper(app_config, caller): def call(*args, **kw): try: print 'Before handler!' return caller(*args, **kw) finally: print 'After Handler!' return call base_config.register_controller_wrapper(controller_wrapper)
It is also possible to register wrappers for a specific controller:
base_config.register_controller_wrapper(controller_wrapper, controller=RootController.index)
-
register_rendering_engine(factory)¶ Registers a rendering engine
factory.Rendering engine factories are
tg.renderers.base.RendererFactorysubclasses in charge of creating a rendering engine.
-
register_wrapper(wrapper, after=None)¶ Registers a TurboGears application wrapper.
Application wrappers are like WSGI middlewares but are executed in the context of TurboGears and work with abstractions like Request and Respone objects.
See
tg.appwrappers.base.ApplicationWrapperfor complete definition of application wrappers.The
afterparameter defines their position into the wrappers chain. The default valueNonemeans they are executed in a middle point, so they run after the TurboGears wrappers likeErrorPageApplicationWrapperwhich can intercept their response and return an error page.Builtin TurboGears wrappers are usually registered with
after=Truewhich means they run furthest away from the application itself and can intercept the response of any other wrapper.Providing
after=Falsemeans the wrapper will be registered near to the application itself (so wrappers registered at default position and with after=True will be able to see its response).afterparameter can also accept an application wrapper class. In such case the registered wrapper will be registered right after the specified wrapper and so will be a little further from the application then the specified one (can see the response of the specified one).
-
setup_tg_wsgi_app(load_environment=None)¶ Create a base TG app, with all the standard middleware.
load_environment- A required callable, which sets up the basic evironment needed for the application.
setup_vars- A dictionary with all special values necessary for setting up the base wsgi app.
-
class
tg.wsgiapp.TGApp(config=None, **kwargs)¶ -
find_controller(controller)¶ Locates a controller for this TGApp.
This is the same af
lookup_controller()but will reuse configuration of the application and will cache results.Parameters: controller (str) – The controller name, this will be the name of the python module containing the controller.
-
classmethod
lookup_controller(config, controller)¶ Locates a controller by attempting to import it then grab the SomeController instance from the imported module.
Override this to change how the controller object is found once the URL has been resolved.
Parameters:
-
WebFlash¶
Flash messaging system for sending info to the user in a non-obtrusive way
-
class
tg.flash.TGFlash(**options)¶ Support for flash messages stored in a plain cookie.
Supports both fetching flash messages on server side and on client side through Javascript.
When used from Python itself, the flash object provides a
TGFlash.render()method that can be used from templates to render the flash message.When used on Javascript, calling the
TGFlash.render()provides awebflashjavascript object which exposes.payload()and.render()methods that can be used to get current message and render it from javascript.For a complete list of options supported by Flash objects see
TGFlash.configure().-
configure(cookie_name='webflash', default_status='ok', template=<string.Template object>, js_call='webflash.render()', js_template=<string.Template object>, allow_html=False)¶ Flash messages can be configured through
AppConfig(app_cfg.base_config) using the following options:flash.cookie_name-> Name of the cookie used to store flash messagesflash.default_status-> Default message status if not specified (okby default)flash.template->string.Templateinstance used as the flash template when rendered from server side, will receive$container_id,$messageand$statusvariables.flash.allow_html-> Turns on/off escaping in flash messages, by default HTML is not allowed.flash.js_call-> javascript code which will be run when displaying the flash from javascript. Default iswebflash.render(), you can usewebflash.payload()to retrieve the message and show it with your favourite library.flash.js_template->string.Templateinstance used to replace full javascript support for flash messages. When rendering flash message for javascript usage the following code will be used instead of providing the standardwebflashobject. If you replacejs_templateyou must also ensure cookie parsing and delete it for already displayed messages. The template will receive:$container_id,$cookie_name,$js_callvariables.
-
message¶ Get only current flash message, getting the flash message will delete the cookie.
-
pop_payload()¶ Fetch current flash message, status and related information.
Fetching flash message deletes the associated cookie.
-
render(container_id, use_js=True)¶ Render the flash message inside template or provide Javascript support for them.
container_idis the DIV where the messages will be displayed, whileuse_jsswitches between rendering the flash as HTML or for Javascript usage.
-
status¶ Get only current flash status, getting the flash status will delete the cookie.
-
Rendering¶
-
tg.render_template(template_vars, template_engine=None, template_name=None, **kwargs)¶ Renders a specific template in current TurboGears context.
Permits to manually render any template like TurboGears would for expositions. It also guarantees that the
before_render_callandafter_render_callhooks are called in the process.Parameters: - template_vars (dict) – This is the dictonary of variables that should
become available to the template. Template
vars can also include the
tg_cachedictionary which enables template caching. - template_engine (str) – This is the template engine name, same as specified inside AppConfig.renderers.
- template_name (str) – This is the template to render, can be specified both as a path or using dotted notation if available.
TurboGears injects some additional variables in the template context, those include:
- tg.config -> like tg.config in controllers
- tg.flash_obj -> the flash object, call
renderon it to display it. - tg.quote_plus -> function to perform percentage escaping (%xx)
- tg.url -> like tg.url in controllers
- tg.identity -> like tg.request.identity in controllers
- tg.session -> like tg.session in controllers
- tg.locale -> Languages of the current request
- tg.errors -> Validation errors
- tg.inputs -> Values submitted for validation
- tg.request -> like tg.request in controllers
- tg.auth_stack_enabled -> if authentication is enabled or not
- tg.predicates -> like tg.predicates in controllers
- tmpl_context -> like tg.tmpl_context in controllers
- response -> like tg.response in controllers
- request -> like tg.request in controllers
- config -> like tg.config in controllers
- app_globals -> like tg.app_globals in controllers
- session -> like tg.session in controllers
- url -> like tg.url in controllers
- h -> Your application helpers
- translator -> The current gettext translator
- _ -> like tg.i18n.ugettext
Additional variables can be added to every template by a
variable_providerfunction inside the application configuration. This function is expected to return adictwith any variable that should be added the default template variables. It can even replace existing variables.- template_vars (dict) – This is the dictonary of variables that should
become available to the template. Template
vars can also include the
-
tg.render.cached_template(template_name, render_func, ns_options=(), cache_key=None, cache_type=None, cache_expire=None, **kwargs)¶ Cache and render a template, took from Pylons
Cache a template to the namespace
template_name, along with a specific key if provided.Basic Options
template_name- Name of the template, which is used as the template namespace.
render_func- Function used to generate the template should it no longer be valid or doesn’t exist in the cache.
ns_options- Tuple of strings, that should correspond to keys likely to be
in the
kwargsthat should be used to construct the namespace used for the cache. For example, if the template language supports the ‘fragment’ option, the namespace should include it so that the cached copy for a template is not the same as the fragment version of it.
Caching options (uses Beaker caching middleware)
cache_key- Key to cache this copy of the template under.
cache_type- Valid options are
dbm,file,memory,database, ormemcached. cache_expire- Time in seconds to cache this template with this
cache_keyfor. Or use ‘never’ to designate that the cache should never expire.
The minimum key required to trigger caching is
cache_expire='never'which will cache the template forever seconds with no key.
-
class
tg.renderers.base.RendererFactory¶ Factory that creates one or multiple rendering engines for TurboGears. Subclasses have to be registered with
tg.configuration.AppConfig.register_rendering_engine()and must implement thecreatemethod accordingly.-
classmethod
create(config, app_globals)¶ Given the TurboGears configuration and application globals it must create a rendering engine for each one specified into the
engineslist.It must return a dictionary in the form:
{'engine_name': rendering_engine_callable, 'other_engine': other_rendering_callable}
Rendering engine callables are callables in the form:
func(template_name, template_vars, cache_key=None, cache_type=None, cache_expire=None, **render_params)
render_paramsparameter will contain all the values provide through@expose(render_params={}).
-
options= {}¶ Here specify the list of engines for which this factory will create a rendering engine and their options. They must be specified like:
engines = {'json': {'content_type': 'application/json'}}
Currently only supported option is
content_type.
-
with_tg_vars= True¶ Here specify if turbogears variables have to be injected in the template context before using any of the declared engines. Usually
Trueunless engines are protocols (ie JSON).
-
classmethod
-
class
tg.jsonify.JSONEncoder(**kwargs)¶ TurboGears custom JSONEncoder.
Provides support for encoding objects commonly used in TurboGears apps, like:
- SQLAlchemy queries
- Ming queries
- Dates
- Decimals
- Generators
Support for additional types is provided through the
__json__method that will be called on the object by the JSONEncoder when provided and through the ability to register custom encoder for specific types usingJSONEncoder.register_custom_encoder().-
configure(isodates=False, custom_encoders=None, allow_lists=False, **kwargs)¶ JSON encoder can be configured through
AppConfig(app_cfg.base_config) using the following options:json.isodates-> encode dates using ISO8601 formatjson.custom_encoders-> List of tuples(type, encode_func)to register custom encoders for specific types.json.allow_lists-> Allows lists to be encoded, this is usually disabled for security reasons due to JSON hijacking. See http://stackoverflow.com/questions/16289894 for additional details.
-
register_custom_encoder(objtype, encoder)¶ Register a custom encoder for the given type.
Instead of using standard behavior for encoding the given type to JSON, the
encoderwill used instead.encodermust be a callable that takes the object as argument and returns an object that can be encoded in JSON (usually a dict).
-
tg.jsonify.encode(obj, encoder=None, iterencode=False)¶ Return a JSON string representation of a Python object.
-
tg.jsonify.encode_iter(obj, encoder=None)¶ Encode object, yielding each string representation as available.
Request & Response¶
-
class
tg.request_local.Request(environ, charset=None, unicode_errors=None, decode_param_names=None, **kw)¶ WebOb Request subclass
The WebOb
webob.Requesthas no charset, or other defaults. This subclass adds defaults, along with several methods for backwards compatibility with paste.wsgiwrappers.WSGIRequest.-
args_params¶ Arguments used for dispatching the request.
This mixes GET and POST arguments.
-
controller_url¶ Url of the current controller.
-
disable_auth_challenger()¶ Disable authentication challenger for current request.
This will forward your response as is in case of 401 bypassing any repoze.who challenger.
-
disable_error_pages()¶ Disable custom error pages for the current request.
This will forward your response as is bypassing the
ErrorPageApplicationWrapper
-
dispatch_state¶ Details and info about dispatcher that handled this request.
-
languages¶ Return the list of browser preferred languages ensuring that
i18n.langis listed.
-
plain_languages¶ Return the list of browser preferred languages
-
quoted_path_info¶ PATH used for dispatching the request.
-
response_ext¶ URL extension when URL Extensions are enabled.
In case URL Request Extension is enabled this will be the extension of the url.
disable_request_extensionsdrives this is enabled or not.
-
response_type¶ Expected response content type when URL Extensions are enabled.
In case URL Request Extension is enabled this will be the content type of the expected response.
disable_request_extensionsdrives this is enabled or not.
Extract a signed cookie of
namefrom the requestThe cookie is expected to have been created with
Response.signed_cookie, and thesecretshould be the same as the one used to sign it.Any failure in the signature of the data will result in None being returned.
-
-
class
tg.request_local.Response(body=None, status=None, headerlist=None, app_iter=None, content_type=None, conditional_response=None, charset=<object object>, **kw)¶ WebOb Response subclass
Save a signed cookie with
secretsignatureSaves a signed cookie of the pickled data. All other keyword arguments that
WebOb.set_cookieaccepts are usable and passed to the WebOb set_cookie method after creating the signed cookie value.
-
class
crank.dispatchstate.DispatchState(request, dispatcher, params=None, path_info=None, ignore_parameters=None, strip_extension=True, path_translator=None)¶ This class keeps around all the pertainent info for the state of the dispatch as it traverses through the tree. This allows us to attach things like routing args and to keep track of the path the controller takes along the system.
- Arguments:
- request
- object, must have a path_info attribute if path_info is not provided
- dispatcher
- dispatcher object to get the ball rolling
- params
- parameters to pass into the dispatch state will use request.params
- path_info
- pre-split list of path elements, will use request.pathinfo if not used
- strip_extension
- Whenever crank should strip the url extension or not resolving the path
- path_translator
- Function used to perform path escaping when looking for controller methods, can be None to perform no escaping or True to use default escaping function.
DispatchState instance for current request is made available in TurboGears2 as
tg.request.dispatch_stateas soon as the dispatch process is completed.-
action¶ Method in charge of processing the request
-
add_controller(location, controller)¶ Add a controller object to the stack
-
add_routing_args(current_path, remainder, fixed_args, var_args)¶ Add the “intermediate” routing args for a given controller mounted at the current_path. This is mostly used during REST dispatch to keep track of intermediate arguments and make them always available.
-
controller¶ Controller currently handling the request
-
controller_path¶ Controllers that got traversed to dispatch the request
-
extension¶ Extension of the URL (only if strip_extension is enabled).
If the path ends with an extension and strip_extension is enabled the extension is stripped from the url and is available here.
-
params¶ Parameters passed to the action
-
path¶ The path (URL) that has to be dispatched
-
remainder¶ Part of the URL path remaining after lookup of the action.
Those is usually passed as positional arguments to the method in charge of the action together with params.
-
request¶ The request that originated the dispatch process
-
resolve()¶ Once a DispatchState is created resolving it performs the dispatch.
Returns the updated DispatchState where
.controller,.action,.paramsand.remainderproperties all point to the controller and method that should process the request and to the method arguments.
-
root_dispatcher¶ Root Dispatcher instance that initiated the dispatch flow
-
routing_args¶ Parameters detected by the routing system.
This includes Request parameters and parameters extracted in other ways (usually added through
add_routing_args()). In case of REST it will include intermediate arguments retrieved during dispatch of parent controllers.
-
set_action(method, remainder)¶ Add the final method that will be called in the _call method
-
set_params(params)¶ Set parameters that will be passed to the called action
-
set_path(path_info)¶ Update path that needs to be dispatched.
In case this is changed during the dispatch process it won’t have any effect on the dispatch currently under execution.
Hooks¶
-
class
tg.configuration.hooks.HooksNamespace¶ Manages hooks registrations and notifications
-
disconnect(hook_name, func, controller=None)¶ Disconnect an hook.
The registered function is removed from the hook notification list.
-
notify(hook_name, args=None, kwargs=None, controller=None, context_config=None, trap_exceptions=False)¶ Notifies a TurboGears hook.
Each function registered for the given hook will be executed,
argsandkwargswill be passed to the registered functions as arguments.It permits to notify both application hooks:
tg.hooks.notify('custom_global_hook')
Or controller hooks:
tg.hooks.notify('before_render', args=(remainder, params, output), controller=RootController.index)
-
notify_with_value(hook_name, value, controller=None, context_config=None)¶ Notifies a TurboGears hook which is expected to return a value.
hooks with values are expected to accept an input value an return a replacement for it. Each registered function will receive as input the value returned by the previous function in chain.
The resulting value will be returned by the
notify_with_valuecall itself:app = tg.hooks.notify_with_value('before_config', app)
-
register(hook_name, func, controller=None)¶ Registers a TurboGears hook.
Given an hook name and a function it registers the provided function for that role. For a complete list of hooks provided by default have a look at Hooks and Wrappers.
It permits to register hooks both application wide or for specific controllers:
tg.hooks.register('before_render', hook_func, controller=RootController.index) tg.hooks.register('startup', startup_function)
-
-
class
tg.appwrappers.base.ApplicationWrapper(next_handler, config)¶ Basic interface of the TurboGears Application Wrappers.
Application wrappers are like WSGI middlewares but are executed in the context of TurboGears and work with abstractions like Request and Respone objects.
Application Wrappers can be registered using
AppConfig.register_wrapper()which will inject them into the next TGApp created.While they can be any callable, inheriting from this base class is strongly suggested as enables additional behaviours and third party code might depend on them.
Application Wrappers require a
next_handlerwhich is the next handler to call in the chain andconfigwhich is the current application configuration.-
__call__(controller, environ, context)¶ This is the actual wrapper implementation.
Wrappers are called for each request with the
controllerin charge of handling the request, theenvironof the request and the TurboGearscontextof the request.They should call the
next_handler(which will accept the same parameters) and return atg.request_local.Responseinstance which is the request response. Usually they will return the same response object provided by the next handler unless they want to replace it.A simple logging wrapper might look like:
class LogAppWrapper(ApplicationWrapper): def __init__(self, handler, config): super(LogAppWrapper, self).__init__(handler, config) def __call__(self, controller, environ, context): print 'Going to run %s' % context.request.path return self.next_handler(controller, environ, context)
-
injected¶ Whenever the Application Wrapper should be injected.
By default all application wrappers are injected into the wrappers chain, you might want to make so that they are injected or not depending on configuration options.
-
next_handler¶ The next handler in the chain
-
Milestones¶
-
class
tg.configuration.milestones._ConfigMilestoneTracker(name)¶ Tracks actions that need to be performed when a specific configuration point is reached and required options are correctly initialized
-
reach()¶ Marks the milestone as reached.
Runs the registered actions. Calling this method multiple times should lead to nothing.
-
register(action, persist_on_reset=False)¶ Registers an action to be called on milestone completion.
If milestone is already passed action is immediately called
-
Internationalization¶
-
tg.i18n.set_lang(languages, **kwargs)¶ Set the current language(s) used for translations in current call and session.
languages should be a string or a list of strings. First lang will be used as main lang, others as fallbacks.
-
tg.i18n.get_lang(all=True)¶ Return the current i18n languages used
returns
Noneif no supported language is available (no translations are in place) or a list of languages.In case
allparameter isFalseonly the languages for which the application is providing a translation are returned. Otherwise all the languages preferred by the user are returned.
-
tg.i18n.add_fallback(lang, **kwargs)¶ Add a fallback language from which words not matched in other languages will be translated to.
This fallback will be associated with the currently selected language – that is, resetting the language via set_lang() resets the current fallbacks.
This function can be called multiple times to add multiple fallbacks.
-
tg.i18n.set_request_lang(languages, tgl=None)¶ Set the current request language(s) used for translations without touching the session language.
languages should be a string or a list of strings. First lang will be used as main lang, others as fallbacks.
-
tg.i18n.ugettext(value)¶ Mark a string for translation. Returns the localized unicode string of value.
Mark a string to be localized as follows:
_('This should be in lots of languages')
-
tg.i18n.lazy_ugettext(*args, **kwargs)¶ Lazy-evaluated version of the ugettext function
- Mark a string for translation. Returns the localized unicode
string of value.
Mark a string to be localized as follows:
_('This should be in lots of languages')
-
tg.i18n.ungettext(singular, plural, n)¶ Mark a string for translation. Returns the localized unicode string of the pluralized value.
This does a plural-forms lookup of a message id.
singularis used as the message id for purposes of lookup in the catalog, whilenis used to determine which plural form to use. The returned message is a Unicode string.Mark a string to be localized as follows:
ungettext('There is %(num)d file here', 'There are %(num)d files here', n) % {'num': n}
-
tg.i18n.lazy_ungettext(*args, **kwargs)¶ Lazy-evaluated version of the ungettext function
- Mark a string for translation. Returns the localized unicode
string of the pluralized value.
This does a plural-forms lookup of a message id.
singularis used as the message id for purposes of lookup in the catalog, whilenis used to determine which plural form to use. The returned message is a Unicode string.Mark a string to be localized as follows:
ungettext('There is %(num)d file here', 'There are %(num)d files here', n) % {'num': n}
Controller Utilities¶
Helper functions for controller operation.
URL definition and browser redirection are defined here.
-
tg.controllers.util.url(base_url='/', params=None, qualified=False, scheme=None)¶ Generate an absolute URL that’s specific to this application.
The URL function takes a string (base_url) and, appends the SCRIPT_NAME and adds parameters for all of the parameters passed into the params dict.
schemecan be passed in case of aqualifiedurl to create an url with the given scheme.
-
tg.controllers.util.lurl(base_url=None, params=None, **kwargs)¶ Like tg.url but is lazily evaluated.
This is useful when creating global variables as no request is in place.
As without a request it wouldn’t be possible to correctly calculate the url using the SCRIPT_NAME this demands the url resolution to when it is displayed for the first time.
-
tg.controllers.util.redirect(base_url='/', params=None, redirect_with=<class 'tg.exceptions.HTTPFound'>, scheme=None, **kwargs)¶ Generate an HTTP redirect.
The function raises an exception internally, which is handled by the framework. The URL may be either absolute (e.g. http://example.com or /myfile.html) or relative. Relative URLs are automatically converted to absolute URLs. Parameters may be specified, which are appended to the URL. This causes an external redirect via the browser; if the request is POST, the browser will issue GET for the second request.
-
tg.controllers.util.etag_cache(key=None)¶ Use the HTTP Entity Tag cache for Browser side caching
If a “If-None-Match” header is found, and equivilant to
key, then a304HTTP message will be returned with the ETag to tell the browser that it should use its current cache of the page.Otherwise, the ETag header will be added to the response headers.
-
tg.controllers.util.abort(status_code=None, detail='', headers=None, comment=None, passthrough=False, error_handler=False)¶ Aborts the request immediately by returning an HTTP exception
In the event that the status_code is a 300 series error, the detail attribute will be used as the Location header should one not be specified in the headers attribute.
- passthrough
When
Trueinstead of displaying the custom error document for errors or the authentication page for failed authorizations the response will just pass through as is.Set to
"json"to send out the response body in JSON format.- error_handler
When
Trueinstead of immediately abort the request it will create a callable that can be used as@validateerror_handler.A common case is
abort(404, error_handler=True)aserror_handlerfor validation that retrieves objects from database:from formencode.validators import Wrapper @validate({'team': Wrapper(to_python=lambda value: Group.query.find({'group_name': value}).one())}, error_handler=abort(404, error_handler=True)) def view_team(self, team): return dict(team=team)
-
tg.controllers.util.auth_force_logout()¶ Forces user logout if authentication is enabled.
-
tg.controllers.util.auth_force_login(user_name)¶ Forces user login if authentication is enabled.
As TurboGears identifies users by
user_namethe passed parameter should be anything your application declares being theuser_namefield in models.
-
tg.controllers.util.validation_errors_response(*args, **kwargs)¶ Returns a
Responseobject with validation errors.The response will be created with a 412 Precondition Failed status code and errors are reported in JSON format as response body.
Typical usage is as
error_handlerfor JSON based api:@expose('json') @validate({'display_name': validators.NotEmpty(), 'group_name': validators.NotEmpty()}, error_handler=validation_errors_response) def post(self, **params): group = Group(**params) return dict(group=group)
General Utilities¶
-
class
tg.util.bunch.Bunch¶ A dictionary that provides attribute-style access.
-
tg.util.dates.get_fixed_timezone(offset)¶ Returns a tzinfo instance with a fixed offset from UTC.
offsetshould be provided in minutes or as atimedelta.
-
tg.util.dates.parse_datetime(value)¶ Parses a string and return a datetime.datetime.
This function supports time zone offsets. When the input contains one, the output uses a timezone with a fixed offset from UTC.
Raises
ValueErrorif the input isn’t well formatted.
-
tg.util.dates.utctz= <UTC>¶ UTC tzinfo instance
-
tg.util.decorators.no_warn(f, *args, **kwargs)¶ Decorator that suppresses warnings inside the decorated function
-
class
tg.util.files.DottedFileNameFinder¶ this class implements a cache system above the get_dotted_filename function and is designed to be stuffed inside the app_globals.
It exposes a method named get_dotted_filename with the exact same signature as the function of the same name in this module.
The reason is that is uses this function itself and just adds caching mechanism on top.
-
get_dotted_filename(template_name, template_extension='.html')¶ this helper function is designed to search a template or any other file by python module name.
Given a string containing the file/template name passed to the @expose decorator we will return a resource useable as a filename even if the file is in fact inside a zipped egg or in a frozen library.
The actual implementation is a revamp of the Genshi buffet support plugin, but could be used with any kind a file inside a python package.
Parameters: - template_name (str) – the string representation of the template name as it has been given by the user on his @expose decorator. Basically this will be a string in the form of: “myapp.templates.somename”
- template_extension (str) – the extension we excpect the template to have, this MUST be the full extension as returned by the os.path.splitext function. This means it should contain the dot. ie: ‘.html’ This argument is optional and the default value if nothing is provided will be ‘.html’
The
template_nameparameter also accepts a form with explicit extensionmyapp.templates.somename!xhtmlthat will override thetemplate_exstesionargument and will always use.xhtmlas the extension. This is usually convenient in extensions and libraries that expose a template and want to ensure they work even in the case the application using them has a different extension for templates on the same engine.
-
classmethod
lookup(name, extension='.html')¶ Convenience method that permits to quickly get a file by dotted notation.
Creates a
DottedFileNameFinderand uses it to lookup the given file using dotted notation. AsDottedFileNameFinderprovides a lookup cache, using this method actually disables the cache as a new finder is created each time, for this reason if you have recurring lookups it’s better to actually create a dotted filename finder and reuse it.
-
-
tg.util.files.safe_filename(filename)¶ Escapes a filename to ensure is valid and secure.
Filename can then safely be stored on a regular file system and passed to
os.path.join(). The filename returned is an ASCII only string for maximum portability:>>> safe_filename("My cool movie.mov") 'My_cool_movie.mov' >>> safe_filename("../../../etc/passwd") 'etc_passwd' >>> safe_filename(u'i contain cool ümläuts.txt') 'i_contain_cool_umlauts.txt'
The function might return an empty filename. .
-
tg.util.html.script_json_encode(obj, encoder=<tg.jsonify.JSONEncoder object>, **kwargs)¶ Works exactly like
tg.jsonify.encode()but is safe for use in<script>tags.- The following characters are escaped in strings:
<>&'
This makes it safe to embed such strings in any place in HTML with the notable exception of double quoted attributes. In that case single quote your attributes or HTML escape it in addition.
-
class
tg.util.lazystring.LazyString(func, *args, **kwargs)¶ Behaves like a string, but no instance is created until the string is actually used.
Takes a function which should be a string factory and a set of arguments to pass to the factory. Whenever the string is accessed or manipulated the factory is called to create the actual string. This is used mostly by lazy internationalization.
-
tg.util.lazystring.lazify(func)¶ Decorator to return a lazy-evaluated version of the original
Applying decorator to a function it will create a
LazyStringwith the decorated function as factory.
-
tg.util.webtest.test_context(app, url=None, environ=None)¶ Sets up a TurboGears context allowing to use turbogears functions outside of a real request.
Everything inside the with statement will have a TurboGears context available
In case
appis aTGAppinstance (as the one that can be retrieved through configure_new_app hook) a context for such application is set in place.In case of a WebTest application a request to
/_test_varsis performed to setup the test variables.In case
appisNonea fake application is created for which to setup the context.urlparameter is provided to simulate the context as being for that url andenvironparameter is provided to allow changing WSGI environ entries for the context.
-
tg.util.sqlalchemy.dictify(obj)¶ Converts a SQLAlchemy model instance to a dictionary
-
tg.util.sqlalchemy.is_saobject(obj)¶ Checks if the provided object is a SQLAlchemy model instance
-
tg.util.ming.dictify(obj)¶ Converts a Ming model instance to a dictionary
-
tg.util.ming.is_mingobject(obj)¶ Checks if the provided object is a Ming model instance
-
tg.util.misc.unless(func, check=None)¶ Wraps
funcensuring it returns a value different fromcheck.A new function that calls
funcand checks its value is returned. In case func returns a value equal to check aValueErroris raised.A common usage pattern is to join this with
Convertto fail validation when querying objects from the database if they do not exist:Convert(unless(DBSession.query(User).get))
-
class
tg.configuration.utils.DependenciesList(*entries)¶ Manages a list of entries which might depend one from the other.
This powers
AppConfig.register_wrapper()and other features in TurboGears2, making possible to register the wrappers right after other wrappers or at the end of the wrappers chain.Note
This is highly inefficient as it is only meant to run at configuration time, a new implementation will probably be provided based on heapq in the future.
-
DEPENDENCY_HEADS= (False, None, True)¶ - Those are the heads of the dependencies tree
Falsemeans before everything elseNonemeans in the middle.Truemeans after everything else.
-
add(entry, key=None, after=None)¶ Adds an entry to the dependencies list.
Parameters: - entry – Entry that must be added to the list.
- key (str|type|None) – An identifier of the object being inserted.
This is used by later insertions as
afterargument to specify after which object the new one should be inserted. - after (str|type|None|False|True) – After which element this one should be inserted.
This is the
keyof a previously inserted item. In case no item withkeyhas been inserted, the entry will be inserted in normal order of insertion. Also accepts one ofDependenciesList.DEPENDENCY_HEADSas key to add entries at begin or end of the list.
-
replace(key, newvalue)¶ Replaces entry associated to key with a new one.
Parameters: - newvalue – Entry that must replace the previous value.
- key (str|type) – An identifier of the object being inserted.
-
values()¶ Returns all the inserted values without their key as a generator
-
-
class
tg.configuration.utils.GlobalConfigurable¶ Defines a configurable TurboGears object with a global default instance.
GlobalConfigurable are objects which the user can create multiple instances to use in its own application or third party module, but for which TurboGears provides a default instance.
Common examples are
tg.flashand the default JSON encoder for which TurboGears provides default instances of.TGFlashand.JSONEncoderclasses but users can create their own.While user created versions are configured calling the
GlobalConfigurable.configure()method, global versions are configured byAppConfigwhich configures them whenconfig_readymilestone is reached.-
configure(**options)¶ Expected to be implemented by each object to proceed with actualy configuration.
Configure method will receive all the options whose name starts with
CONFIG_NAMESPACE(examplejson.isodateshasjson.namespace).If
CONFIG_OPTIONSis specified options values will be converted withcoerce_config()passingCONFIG_OPTIONSas theconvertersdictionary.
-
-
tg.configuration.utils.coerce_config(configuration, prefix, converters)¶ Extracts a set of options with a common prefix and converts them.
To extract all options starting with
trace_errors.from theconfdictionary and conver them:trace_errors_config = coerce_config(conf, 'trace_errors.', { 'smtp_use_tls': asbool, 'dump_request_size': asint, 'dump_request': asbool, 'dump_local_frames': asbool, 'dump_local_frames_count': asint })
-
tg.configuration.utils.coerce_options(options, converters)¶ Convert some configuration options to expected types.
To replace given options with the converted values in a dictionary you might do:
conf.update(coerce_options(conf, { 'debug': asbool, 'serve_static': asbool, 'auto_reload_templates': asbool }))
-
tg.configuration.utils.get_partial_dict(prefix, dictionary, container_type=<type 'dict'>)¶ Given a dictionary and a prefix, return a Bunch, with just items that start with prefix
The returned dictionary will have ‘prefix.’ stripped so:
get_partial_dict('prefix', {'prefix.xyz':1, 'prefix.zyx':2, 'xy':3})
would return:
{'xyz':1,'zyx':2}