- The version_id_generator parameter of Mapper can now be specified

to rely upon server generated version identifiers, using triggers
or other database-provided versioning features, by passing the value
``False``.  The ORM will use RETURNING when available to immediately
load the new version identifier, else it will emit a second SELECT.
[ticket:2793]
- The ``eager_defaults`` flag of :class:`.Mapper` will now allow the
newly generated default values to be fetched using an inline
RETURNING clause, rather than a second SELECT statement, for backends
that support RETURNING.
- Added a new variant to :meth:`.ValuesBase.returning` called
:meth:`.ValuesBase.return_defaults`; this allows arbitrary columns
to be added to the RETURNING clause of the statement without interfering
with the compilers usual "implicit returning" feature, which is used to
efficiently fetch newly generated primary key values.  For supporting
backends, a dictionary of all fetched values is present at
:attr:`.ResultProxy.returned_defaults`.
- add a glossary entry for RETURNING
- add documentation for version id generation, [ticket:867]

Author: zzzeek

doc/build/changelog/changelog_09.rst

@@ -6,6 +6,37 @@
 .. changelog::
     :version: 0.9.0
 
+    .. change::
+        :tags: feature, orm
+        :tickets: 2793
+
+        The ``version_id_generator`` parameter of ``Mapper`` can now be specified
+        to rely upon server generated version identifiers, using triggers
+        or other database-provided versioning features, by passing the value
+        ``False``.  The ORM will use RETURNING when available to immediately
+        load the new version identifier, else it will emit a second SELECT.
+
+    .. change::
+        :tags: feature, orm
+        :tickets: 2793
+
+        The ``eager_defaults`` flag of :class:`.Mapper` will now allow the
+        newly generated default values to be fetched using an inline
+        RETURNING clause, rather than a second SELECT statement, for backends
+        that support RETURNING.
+
+    .. change::
+        :tags: feature, core
+        :tickets: 2793
+
+        Added a new variant to :meth:`.ValuesBase.returning` called
+        :meth:`.ValuesBase.return_defaults`; this allows arbitrary columns
+        to be added to the RETURNING clause of the statement without interfering
+        with the compilers usual "implicit returning" feature, which is used to
+        efficiently fetch newly generated primary key values.  For supporting
+        backends, a dictionary of all fetched values is present at
+        :attr:`.ResultProxy.returned_defaults`.
+
     .. change::
         :tags: feature
 

doc/build/core/dml.rst

@@ -26,13 +26,11 @@ constructs build on the intermediary :class:`.ValuesBase`.
   :members:
   :inherited-members:
 
-
-.. autoclass:: UpdateBase
+.. autoclass:: sqlalchemy.sql.expression.UpdateBase
   :members:
 
-
-.. autoclass:: ValuesBase
-    :members:
+.. autoclass:: sqlalchemy.sql.expression.ValuesBase
+   :members:
 
 
 

doc/build/core/types.rst

@@ -343,6 +343,8 @@ many decimal places.   Here's a recipe that rounds them down::
                 value = value.quantize(self.quantize)
             return value
 
+.. _custom_guid_type:
+
 Backend-agnostic GUID Type
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 

doc/build/glossary.rst

@@ -521,3 +521,41 @@ Glossary
 
             http://en.wikipedia.org/wiki/Durability_(database_systems)
 
+    RETURNING
+        This is a non-SQL standard clause provided in various forms by
+        certain backends, which provides the service of returning a result
+        set upon execution of an INSERT, UPDATE or DELETE statement.  Any set
+        of columns from the matched rows can be returned, as though they were
+        produced from a SELECT statement.
+
+        The RETURNING clause provides both a dramatic performance boost to
+        common update/select scenarios, including retrieval of inline- or
+        default- generated primary key values and defaults at the moment they
+        were created, as well as a way to get at server-generated
+        default values in an atomic way.
+
+        An example of RETURNING, idiomatic to Postgresql, looks like::
+
+            INSERT INTO user_account (name) VALUES ('new name') RETURNING id, timestamp
+
+        Above, the INSERT statement will provide upon execution a result set
+        which includes the values of the columns ``user_account.id`` and
+        ``user_account.timestamp``, which above should have been generated as default
+        values as they are not included otherwise (but note any series of columns
+        or SQL expressions can be placed into RETURNING, not just default-value columns).
+
+        The backends that currently support
+        RETURNING or a similar construct are Postgresql, SQL Server, Oracle,
+        and Firebird.    The Postgresql and Firebird implementations are generally
+        full featured, whereas the implementations of SQL Server and Oracle
+        have caveats. On SQL Server, the clause is known as "OUTPUT INSERTED"
+        for INSERT and UPDATE statements and "OUTPUT DELETED" for DELETE statements;
+        the key caveat is that triggers are not supported in conjunction with this
+        keyword.  On Oracle, it is known as "RETURNING...INTO", and requires that the
+        value be placed into an OUT paramter, meaning not only is the syntax awkward,
+        but it can also only be used for one row at a time.
+
+        SQLAlchemy's :meth:`.UpdateBase.returning` system provides a layer of abstraction
+        on top of the RETURNING systems of these backends to provide a consistent
+        interface for returning columns.  The ORM also includes many optimizations
+        that make use of RETURNING when available.

doc/build/orm/exceptions.rst

@@ -2,5 +2,4 @@ ORM Exceptions
 ==============
 
 .. automodule:: sqlalchemy.orm.exc
-     
     :members:

doc/build/orm/mapper_config.rst

@@ -1115,6 +1115,202 @@ of these events.
 
 .. autofunction:: reconstructor
 
+
+.. _mapper_version_counter:
+
+Configuring a Version Counter
+=============================
+
+The :class:`.Mapper` supports management of a :term:`version id column`, which
+is a single table column that increments or otherwise updates its value
+each time an ``UPDATE`` to the mapped table occurs.  This value is checked each
+time the ORM emits an ``UPDATE`` or ``DELETE`` against the row to ensure that
+the value held in memory matches the database value.
+
+The purpose of this feature is to detect when two concurrent transactions
+are modifying the same row at roughly the same time, or alternatively to provide
+a guard against the usage of a "stale" row in a system that might be re-using
+data from a previous transaction without refreshing (e.g. if one sets ``expire_on_commit=False``
+with a :class:`.Session`, it is possible to re-use the data from a previous
+transaction).
+
+.. topic:: Concurrent transaction updates
+
+    When detecting concurrent updates within transactions, it is typically the
+    case that the database's transaction isolation level is below the level of
+    :term:`repeatable read`; otherwise, the transaction will not be exposed
+    to a new row value created by a concurrent update which conflicts with
+    the locally updated value.  In this case, the SQLAlchemy versioning
+    feature will typically not be useful for in-transaction conflict detection,
+    though it still can be used for cross-transaction staleness detection.
+
+    The database that enforces repeatable reads will typically either have locked the
+    target row against a concurrent update, or is employing some form
+    of multi version concurrency control such that it will emit an error
+    when the transaction is committed.  SQLAlchemy's version_id_col is an alternative
+    which allows version tracking to occur for specific tables within a transaction
+    that otherwise might not have this isolation level set.
+
+    .. seealso::
+
+        `Repeatable Read Isolation Level <http://www.postgresql.org/docs/9.1/static/transaction-iso.html#XACT-REPEATABLE-READ>`_ - Postgresql's implementation of repeatable read, including a description of the error condition.
+
+Simple Version Counting
+-----------------------
+
+The most straightforward way to track versions is to add an integer column
+to the mapped table, then establish it as the ``version_id_col`` within the
+mapper options::
+
+    class User(Base):
+        __tablename__ = 'user'
+
+        id = Column(Integer, primary_key=True)
+        version_id = Column(Integer, nullable=False)
+        name = Column(String(50), nullable=False)
+
+        __mapper_args__ = {
+            "version_id_col": version_id
+        }
+
+Above, the ``User`` mapping tracks integer versions using the column
+``version_id``.   When an object of type ``User`` is first flushed, the
+``version_id`` column will be given a value of "1".   Then, an UPDATE
+of the table later on will always be emitted in a manner similar to the
+following::
+
+    UPDATE user SET version_id=:version_id, name=:name
+    WHERE user.id = :user_id AND user.version_id = :user_version_id
+    {"name": "new name", "version_id": 2, "user_id": 1, "user_version_id": 1}
+
+The above UPDATE statement is updating the row that not only matches
+``user.id = 1``, it also is requiring that ``user.version_id = 1``, where "1"
+is the last version identifier we've been known to use on this object.
+If a transaction elsewhere has modifed the row independently, this version id
+will no longer match, and the UPDATE statement will report that no rows matched;
+this is the condition that SQLAlchemy tests, that exactly one row matched our
+UPDATE (or DELETE) statement.  If zero rows match, that indicates our version
+of the data is stale, and a :class:`.StaleDataError` is raised.
+
+.. _custom_version_counter:
+
+Custom Version Counters / Types
+-------------------------------
+
+Other kinds of values or counters can be used for versioning.  Common types include
+dates and GUIDs.   When using an alternate type or counter scheme, SQLAlchemy
+provides a hook for this scheme using the ``version_id_generator`` argument,
+which accepts a version generation callable.  This callable is passed the value of the current
+known version, and is expected to return the subsequent version.
+
+For example, if we wanted to track the versioning of our ``User`` class
+using a randomly generated GUID, we could do this (note that some backends
+support a native GUID type, but we illustrate here using a simple string)::
+
+    import uuid
+
+    class User(Base):
+        __tablename__ = 'user'
+
+        id = Column(Integer, primary_key=True)
+        version_uuid = Column(String(32))
+        name = Column(String(50), nullable=False)
+
+        __mapper_args__ = {
+            'version_id_col':version_uuid,
+            'version_id_generator':lambda version: uuid.uuid4().hex
+        }
+
+The persistence engine will call upon ``uuid.uuid4()`` each time a
+``User`` object is subject to an INSERT or an UPDATE.  In this case, our
+version generation function can disregard the incoming value of ``version``,
+as the ``uuid4()`` function
+generates identifiers without any prerequisite value.  If we were using
+a sequential versioning scheme such as numeric or a special character system,
+we could make use of the given ``version`` in order to help determine the
+subsequent value.
+
+.. seealso::
+
+    :ref:`custom_guid_type`
+
+.. _server_side_version_counter:
+
+Server Side Version Counters
+-----------------------------
+
+The ``version_id_generator`` can also be configured to rely upon a value
+that is generated by the database.  In this case, the database would need
+some means of generating new identifiers when a row is subject to an INSERT
+as well as with an UPDATE.   For the UPDATE case, typically an update trigger
+is needed, unless the database in question supports some other native
+version identifier.  The Postgresql database in particular supports a system
+column called `xmin <http://www.postgresql.org/docs/9.1/static/ddl-system-columns.html>`_
+which provides UPDATE versioning.  We can make use
+of the Postgresql ``xmin`` column to version our ``User``
+class as follows::
+
+    class User(Base):
+        __tablename__ = 'user'
+
+        id = Column(Integer, primary_key=True)
+        name = Column(String(50), nullable=False)
+        xmin = Column("xmin", Integer, system=True)
+
+        __mapper_args__ = {
+            'version_id_col': xmin,
+            'version_id_generator': False
+        }
+
+With the above mapping, the ORM will rely upon the ``xmin`` column for
+automatically providing the new value of the version id counter.
+
+.. topic:: creating tables that refer to system columns
+
+    In the above scenario, as ``xmin`` is a system column provided by Postgresql,
+    we use the ``system=True`` argument to mark it as a system-provided
+    column, omitted from the ``CREATE TABLE`` statement.
+
+
+The ORM typically does not actively fetch the values of database-generated
+values when it emits an INSERT or UPDATE, instead leaving these columns as
+"expired" and to be fetched when they are next accessed.  However, when a
+server side version column is used, the ORM needs to actively fetch the newly
+generated value.  This is so that the version counter is set up *before*
+any concurrent transaction may update it again.   This fetching is also
+best done simultaneously within the INSERT or UPDATE statement using :term:`RETURNING`,
+otherwise if emitting a SELECT statement afterwards, there is still a potential
+race condition where the version counter may change before it can be fetched.
+
+When the target database supports RETURNING, an INSERT statement for our ``User`` class will look
+like this::
+
+    INSERT INTO "user" (name) VALUES (%(name)s) RETURNING "user".id, "user".xmin
+    {'name': 'ed'}
+
+Where above, the ORM can acquire any newly generated primary key values along
+with server-generated version identifiers in one statement.   When the backend
+does not support RETURNING, an additional SELECT must be emitted for **every**
+INSERT, which is much less efficient, and also introduces the possibility of
+missed version counters::
+
+    INSERT INTO "user" (name) VALUES (%(name)s) RETURNING "user".id, "user".version_id
+    {'name': 'ed'}
+
+    SELECT "user".version_id AS user_version_id FROM "user" where
+    "user".id = :param_1
+    {"param_1": 1}
+
+It is *strongly recommended* that server side version counters only be used
+when absolutely necessary and only on backends that support :term:`RETURNING`,
+e.g. Postgresql, Oracle, SQL Server (though SQL Server has
+`major caveats <http://blogs.msdn.com/b/sqlprogrammability/archive/2008/07/11/update-with-output-clause-triggers-and-sqlmoreresults.aspx>`_ when triggers are used), Firebird.
+
+.. versionadded:: 0.9.0
+
+    Support for server side version identifier tracking.
+
+
 Class Mapping API
 =================
 

doc/build/static/docs.css

@@ -343,6 +343,21 @@ div.admonition, div.topic, .deprecated, .versionadded, .versionchanged {
     box-shadow: 2px 2px 3px #DFDFDF;
 }
 
+
+div.sidebar {
+    background-color: #FFFFEE;
+    border: 1px solid #DDDDBB;
+    float: right;
+    margin: 0 0 0.5em 1em;
+    padding: 7px 7px 0;
+    width: 40%;
+    font-size:.9em;
+}
+
+p.sidebar-title {
+    font-weight: bold;
+}
+
 /* grrr sphinx changing your document structures, removing classes.... */
 
 .versionadded .versionmodified,

lib/sqlalchemy/engine/base.py

@@ -898,6 +898,12 @@ def _execute_context(self, dialect, constructor,
             elif not context._is_explicit_returning:
                 result.close(_autoclose_connection=False)
                 result._metadata = None
+        elif context.isupdate:
+            if context._is_implicit_returning:
+                context._fetch_implicit_update_returning(result)
+                result.close(_autoclose_connection=False)
+                result._metadata = None
+
         elif result._metadata is None:
             # no results, get rowcount
             # (which requires open cursor on some drivers

lib/sqlalchemy/engine/default.py

@@ -396,6 +396,7 @@ class DefaultExecutionContext(interfaces.ExecutionContext):
     statement = None
     postfetch_cols = None
     prefetch_cols = None
+    returning_cols = None
     _is_implicit_returning = False
     _is_explicit_returning = False
 
@@ -492,6 +493,7 @@ def _init_compiled(cls, dialect, connection, dbapi_connection,
         if self.isinsert or self.isupdate:
             self.postfetch_cols = self.compiled.postfetch
             self.prefetch_cols = self.compiled.prefetch
+            self.returning_cols = self.compiled.returning
             self.__process_defaults()
 
         processors = compiled._bind_processors
@@ -750,6 +752,11 @@ def _fetch_implicit_returning(self, resultproxy):
                 ipk.append(row[c])
 
         self.inserted_primary_key = ipk
+        self.returned_defaults = row
+
+    def _fetch_implicit_update_returning(self, resultproxy):
+        row = resultproxy.fetchone()
+        self.returned_defaults = row
 
     def lastrow_has_defaults(self):
         return (self.isinsert or self.isupdate) and \

lib/sqlalchemy/engine/result.py

@@ -621,6 +621,16 @@ def last_inserted_params(self):
         else:
             return self.context.compiled_parameters[0]
 
+    @property
+    def returned_defaults(self):
+        """Return the values of default columns that were fetched using
+        the ``returned_defaults`` feature.
+
+        .. versionadded:: 0.9.0
+
+        """
+        return self.context.returned_defaults
+
     def lastrow_has_defaults(self):
         """Return ``lastrow_has_defaults()`` from the underlying
         :class:`.ExecutionContext`.