TurboGears Automatic CRUD Generation

Overview

This is a simple extension that provides a basic controller class that can be extended to meet the needs of the developer. The intention is to provide a fast path to data management by allowing the user to define forms and override the data interaction with custom manipulations once the view logic is in place. The name of this extensible class is CrudRestController.

What is CRUD?

CRUD is a set of functions to manipulate the data in a database: create, read, update, delete.

Um, REST?

REST is a methodology for mapping resource manipulation to meaningful URL. For instance if we wanted to edit a user with the ID 3, the URL might look like: /users/3/edit. For a brief discussion on REST, take a look at the microformats entry.

Before We Get Started

Here is the model definition we will be using for this tutorial:

from sqlalchemy import Column, Integer, String, Date, Text, ForeignKey
from sqlalchemy.orm import relation

from moviedemo.model import DeclarativeBase

class Genre(DeclarativeBase):
    __tablename__ = "genres"
    genre_id = Column(Integer, primary_key=True)
    name = Column(String(100))

class Movie(DeclarativeBase):
    __tablename__ = "movies"
    movie_id = Column(Integer, primary_key=True)
    title = Column(String(100), nullable=False)
    description = Column(Text, nullable=True)
    genre_id = Column(Integer, ForeignKey('genres.genre_id'))
    genre = relation('Genre', backref='movies')
    release_date = Column(Date, nullable=True)

EasyCrudRestController

The first thing we want to do is instantiate a EasyCrudRestController. We import the controller from the extension, and then provide it with a model class that it will use for its data manipulation. For this example we will utilize the Movie class.:

from tgext.crud import EasyCrudRestController
from moviedemo.model import DBSession, Movie

class MovieController(EasyCrudRestController):
    model = Movie

class RootController(BaseController):
    movies = MovieController(DBSession)

That will provide a simple and working CRUD controller already configured with some simple views to list, create, edit and delete objects of type Movie.

Customizing EasyCrudRestController

The EasyCrudRestController provides some quick customization tools. Having been thought to quickly prototype parts of your web applications the EasyCrudRestController permits both to tune forms options and to add utility methods on the fly:

class MovieController(EasyCrudRestController):
    model = Movie

    title = "My admin title"

    __form_options__ = {
        '__hide_fields__':['movie_id'],
        '__field_order__':['title', 'description'],
        '__field_widget_types__':{'description':TextArea}
    }

    __table_options__ = { # see Sprox TableBase and Sprox TableFiller
        '__limit_fields__': ['title', 'desc'],
        '__add_fields__': {'computed': None},
        'computed': lambda filler, row: row.some_field * 2
    }

The title option provides a way to customize the title displayed in the titlebar of your browser.

The __form_options__ dictionary will permit to tune the forms configuration. The specified options will be applied to both the form used to create new entities and to edit the existing ones. To have a look at the available options refer to Sprox FormBase

The __table_options__ dictionary will permit to tune the forms configuration. To have a look at the available options refer to Sprox TableBase, Sprox TableFiller, and their parents as well.

Enabling SubString Searches

The CrudRestController provides ready to use search function, when opening the controller index you will see a list of entries and a search box.

By default the search box looks for perfect matches, this is often not the case especially if you are looking in long text entries that the user might not remember, this behavior can be changed by using the substring_filters option.

You can enable substring searches for all the text fields by setting it to True:

class MovieController(EasyCrudRestController):
    model = Movie
    substring_filters = True

    __table_options__ = {
        '__omit_fields__':['movie_id'],
    }

This will permit to search for text inside our movies title and descriptions. If you want to restrict substring searches to only some fields you can specify them explicitly:

class MovieController(EasyCrudRestController):
    model = Movie
    substring_filters = ['description']

    __table_options__ = {
        '__omit_fields__':['movie_id'],
    }

Remembering Previous Values

The default behavior of the CrudRestController is to set fields to the submitted value, if the user submits an empty value the object property gets emptied, there are cases where you might prefer it to keep the previous value when an empty one is provided. This behavior can be enabled using the remember_values option.

This is specially the case with images, you usually prefer to keep the previous image if a new one is not provided instead of deleting it at all.

Suppose we have a Photo model which has an image field using tgext.datahelpers AttachedImage to provide an image field (pease refer to tgext.datahelpers documentation for more details). By default each time the user submits the edit form without specifying a new image we would lose our previous image, to avoid this behavior and just keep our previous image when none is specified we can use the remember_values option:

class PhotoManageController(EasyCrudRestController):
    model = Photo
    remember_values = ['image']

    __table_options__ = {
        '__omit_fields__':['uid'],
        '__xml_fields__' : ['image'],

        'image': lambda filler,row: Markup('<img src="%s"/>' % row.image.thumb_url) if row.image else ''
    }

    __form_options__ = {
        '__field_widget_types__':{'image':FileField},
        '__field_validator_types__' : {'image':FieldStorageUploadConverter},
        '__field_widget_args__': {'image':{'label':'Photo PNG (640x280)'}},
        '__hide_fields__':['uid']
    }

Customizing Pagination

The CrudRestController provides pagination support, by default this is enabled and provides 7 entries per page.

To tune pagination you can set the pagination set of options. To change the number of entries displayed you can set pagination['items_per_page'].

To display 20 items per page you can for example use:

class MovieController(EasyCrudRestController):
    model = Movie
    pagination = {'items_per_page': 20}

To totally disable pagination just set the pagination option to False:

class MovieController(EasyCrudRestController):
    model = Movie
    pagination = False

Custom CrudRestController

The EasyCrudRestController provides a preconfigured CrudRestController but often you will need to deeply customize it for your needs. To do that we can start over with a clean controller and start customizing it:

from tgext.crud import CrudRestController
from moviedemo.model import DBSession, Movie

class MovieController(CrudRestController):
    model = Movie

class RootController(BaseController):
    movies = MovieController(DBSession)

Well that won’t actually get you anywhere, in fact, it will do nothing at all. We need to provide CrudRestController with a set of widgets and datafillers so that it knows how to handle your REST requests. First, lets get all of the Movies to display in a table.

Sprox

Sprox is a library that can help you to generate forms and filler data. It utilizes metadata extracted from the database definitions to provide things like form fields, drop downs, and column header data for view widgets. Sprox is also customizable, so we can go in and modify the way we want our data displayed once we get going with it. Here we define a table widget using Sprox’s sprox.tablebase.TableBase class for our movie table.:

from sprox.tablebase import TableBase

class MovieTable(TableBase):
    __model__ = Movie
    __omit_fields__ = ['genre_id']
movie_table = MovieTable(DBSession)

Filling Our Table With Data

So, now we have our movie_table, but it’s not going to do us much good without data to fill it. Sprox provides a sprox.fillerbase.TableFiller class which will retrieve the relevant data from the database and package it in a dictionary for consumption. This is useful if you are creating JSON. Basically, you can provide CrudRestController with any object that has a get_value function and it will work because of duck typing. Just make certain that your get_value function returns the right data type for the widget you are filling. Here is what the filler would look like instantiated.:

from sprox.fillerbase import TableFiller

class MovieTableFiller(TableFiller):
    __model__ = Movie
movie_table_filler = MovieTableFiller(DBSession)

Putting It All Together

Let’s modify our CrudRestController to utilize our new table. The new RootController would look like this:

from tgext.crud import CrudRestController
from moviedemo.model import DBSession, Movie
from sprox.tablebase import TableBase
from sprox.fillerbase import TableFiller

class MovieTable(TableBase):
    __model__ = Movie
movie_table = MovieTable(DBSession)

class MovieTableFiller(TableFiller):
    __model__ = Movie
movie_table_filler = MovieTableFiller(DBSession)

class MovieController(CrudRestController):
    model = Movie
    table = movie_table
    table_filler = movie_table_filler

class RootController(BaseController):
    movie = MovieController(DBSession)

You can now visit /movies/ and it will display a list of movies.

Forms

One of the nice thing about Sprox table definitions is that they provide you with a set of RESTful links. CrudRestController provides methods for these pages, but you must provide the widgets for the forms. Specifically, we are talking about the edit and new forms. Here is one way you might create a form to add a new record to the database using sprox.formbase.AddRecordForm:

class MovieAddForm(AddRecordForm):
    __model__ = Movie
    __omit_fields__ = ['genre_id', 'movie_id']
movie_add_form = MovieAddForm(DBSession)

Adding this to your movie controller would look make it now look something like this:

class MovieController(CrudRestController):
    model = Movie
    table = movie_table
    table_filler = movie_table_filler
    new_form = movie_add_form

You can now visit /movies/new.

Edit Form

Now we just need to map a form to the edit function so that we can close the loop on our controller. The reason we need separate forms for Add and Edit is due to validation. Sprox will check the database for uniqueness on a “new” form. On an edit form, this is not required since we are updating, not creating.:

from sprox.formbase import EditableForm

class MovieEditForm(EditableForm):
    __model__ = Movie
    __omit_fields__ = ['genre_id', 'movie_id']
movie_edit_form = MovieEditForm(DBSession)

The biggest difference between this form and that of the “new” form is that we have to get data from the database to fill in the form. Here is how we use sprox.formbase.EditFormFiller to do that:

from sprox.fillerbase import EditFormFiller

class MovieEditFiller(EditFormFiller):
    __model__ = Movie
movie_edit_filler = MovieEditFiller(DBSession)

Now it is a simple as adding our filler and form definitions to the MovieController and close the loop on our presentation.

Declarative

If you are interested in brevity, the crud controller may be created in a more declarative manner like this:

from tgext.crud import CrudRestController
from sprox.tablebase import TableBase
from sprox.formbase import EditableForm, AddRecordForm
from sprox.fillerbase import TableFiller, EditFormFiller

class DeclarativeMovieController(CrudRestController):
    model = Movie

    class new_form_type(AddRecordForm):
        __model__ = Movie
        __omit_fields__ = ['genre_id', 'movie_id']

    class edit_form_type(EditableForm):
        __model__ = Movie
        __omit_fields__ = ['genre_id', 'movie_id']

    class edit_filler_type(EditFormFiller):
        __model__ = Movie

    class table_type(TableBase):
        __model__ = Movie
        __omit_fields__ = ['genre_id', 'movie_id']

    class table_filler_type(TableFiller):
        __model__ = Movie

Options reference

The tgext.crud.CrudRestController and tgext.crud.EasyCrudRestController provide a bunch of configuration options that can be changed by subclassing the controller and providing them in a declarative way:

class tgext.crud.CrudRestController(session, menu_items=None)
Initialization options:
 
session

database session

menu_items

Dictionary or mapping type of links to other CRUD sections. This is used to generate the links sidebar of the CRUD interface. Can be specified in the form model_items['lower_model_name'] = ModelClass or model_items['link'] = 'Name'.

Class attributes:
 
title

Title to be used for each page. default: 'Turbogears Admin System'

model

Model this class is associated with.

remember_values

List of model attributes that need to keep previous value when not provided on submission instead of replacing the existing value with an empty one. It’s commonly used with file fields to avoid having to reupload the file again when the model is edited.

keep_params

List of URL parameters that need to be kept around when redirecting between the various pages of the CRUD controller. Can be used to keep around filters or sorting when editing a subset of the models.

filters

Dictionary of filters that must be applied to queries. Those are only “equals to” filters that can be used to limit objects to a subset of entries that have that value. Filters are also applied on POST method to create new object which by default have the value specified by the filter. If filter is callable it will be called to retrieve the filter value.

For example to display a crud that only shows, creates and edits entities owned by logged user through a user_id ForeignKey you can use a filter like:

{'user_id': lambda: request.identity['user'].user_id}
search_fields

Enables searching on some fields, can be True, False or a list of fields for which searching should be enabled.

substring_filters

Enable substring filtering for some fields, by default is disabled. Pass True to enable it on all fields or pass a list of field names to enable it only on some fields.

json_dictify

True or False, enables advanced dictification of retrieved models when providing JSON responses. This also enables JSON encoding of related entities for the returned model.

conditional_update_field

Name of the field used to perform conditional updates when PUT method is used as a REST API. None disables conditional updates (which is the default).

pagination

Dictionary of options for pagination. False disables pagination. By default {'items_per_page': 7} is provided. Currently the only supported option is items_per_page.

response_type

Limit response to a single format, can be: ‘application/json’ or ‘text/html’. By default tgext.crud will detect expected response from Accept header and will provide response content according to the expected one. If you want to avoid HTML access to a plain JSON API you can use this option to limit valid responses to application/json.

provider_type_selector_type

Use a custom provider type selector class. By default the sprox.providerselector.ProviderTypeSelector class will be used to instantiate a provider selector that can select the right provider for ming and sqlalchemy. In case you are not using ming or sqlalchemy or you just want to change the default behavior of the provider selector, you can override this.

resources

A list of CSSSource / JSSource that have to be injected inside CRUD pages when rendering. By default tgext.crud.resources.crud_style and tgext.crud.resources.crud_script are injected.

table

The sprox.tablebase.TableBase Widget instance used to display the table. By default tgext.crud.utils.SortableTableBase is used which permits to sort table by columns.

table_filler

The sprox.fillerbase.TableFiller instance used to retrieve data for the table. If you want to customize how data is retrieved override the TableFiller._do_get_provider_count_and_objs method to return different entities and count. By default tgext.crud.utils.RequestLocalTableFiller is used which keeps track of the numer of entities retrieved during the current request to enable pagination.

edit_form

Form to be used for editing an existing model. By default sprox.formbase.EditForm is used.

edit_filler

sprox.fillerbase.RecordFiller subclass used to load the values for an entity that need to be edited. Override the RecordFiller.get_value method to provide custom values.

new_form

Form that defines how to create a new model. By default sprox.formbase.AddRecordForm is used.

class tgext.crud.EasyCrudRestController(session, menu_items=None)

A CrudRestController that provides a quick way to setup Sprox forms and Table.

Form options are available through the __form_options__ dictionary which can contain any option accepted by sprox FormBase. Options specific to NewForm and EditForm can be provided through __form_new_options__ and __form_edit_options__.

Table options are available through the __table_options__ dictionary which can contain any option accepted by sprox TableBase. Dictionary keys that do not start with __ will be threated as TableFiller attributes apart from __actions__ which is always assigned to the TableFiller.

Usually _options fields will replace the previous values with theirs in case a parent class provided previous values. You can avoid this behaviour and extend the previous values using addopts() instead of dictionaries and lists as the option values.

Class attributes:
 
__form_options__

Sprox options that need to be applied to both EditableForm and AddRecordForm forms

__form_new_options__

Options that need to be applied only to AddRecordForm form

__form_edit_options__

Options that need to be applied only to EditableForm form

__table_options__

Options that need to be applied to TableBase and TableFiller

tgext.crud.addopts(*args, **kwargs)

Specifies options to be added to __form_options__ and __table_options__.

When options are specified using addopts like:

__form_options__ = {
    '__omit_fields__': addopts('name', 'surname')
}

they get added to the current option value instead of replacing it.

In case of lists they always extend the list, so it’s not possible to change the order of the elements.

In case of dictionaries keys already in the option the value of the option will be merged too if it’s a dictionary or a list.

Customizing Crud Operations

We have really been focusing on the View portion of our controller. This is because CrudRestController performs all of the applicable creates, updates, and deletes on your target object for you. This default functionality is provided by sprox.saormprovider.SAORMProvider. This can of course be overridden.

Overriding Crud Operations

CrudRestController extends RestController, which means that any methods available through RestController are also available to CRC.

Method Description Example Method(s) / URL(s)
get_all Display the table widget and its data GET /movies/
new Display new_form GET /movies/new
edit Display edit_form and the containing record’s data GET /movies/1/edit
post Create a new record POST /movies/
put Update an existing record POST /movies/1?_method=PUT
PUT /movies/1
post_delete Delete an existing record POST /movies/1?_method=DELETE
DELETE /movies/1
get_delete Delete Confirmation page Get /movies/1/delete

If you are familiar with RestController you may notice that get_one is missing. There are plans to add this functionality in the near future. Also, you may note the ?_method on some of the URLs. This is basically a hack because existing browsers do not support the PUT and DELETE methods. Just note that if you decide to incorporate a TW in your edit_form description you must provide a HiddenField('_method') in the definition.

Adding Functionality

REST provides consistency across Controller classes and makes it easy to override the functionality of a given RESTful method. For instance, you may want to get an email any time someone adds a movie. Here is what your new controller code would look like:

class MovieController(CrudRestController):

    # (...)

    @expose(inherit=True)
    def post(self, **kw):
        email_info()
        return super(MovieController, self).post(**kw)

You might notice that the function has the @expose decorator. This is required because the expose decoration occurs at the class-level, so that means that when you override the class method, the expose is eliminated. We add it back to the method by adding @expose with the inherit parameter to inherit the behavior from the parent method.

For more details you can refer to the TGController Subclassing documentation.

Overriding Templates

To override the template for a given method, you would simple re-define that method, providing an expose to your own template, while simply returning the value of the super class’s method.:

class MovieController(CrudRestController):

    # (...)

    @expose('movie_demo.templates.my_get_all_template', inherit=True)
    def get_all(self, *args, **kw):
        return super(MovieController, self).get_all(*args, **kw)

Removing Functionality

You can also block-out capabilities of the RestController you do not wish implemented. Simply define the function that you want to block, but do not expose it. Here is how we “delete” the delete functionality.:

class MovieController(CrudRestController):

    # (...)

    def post_delete(self, *args, **kw):
        """This is not allowed."""
        pass