django/docs/topics/db/transactions.txt

=====================
Database transactions
=====================

.. module:: django.db.transaction

Django gives you a few ways to control how database transactions are managed.

Managing database transactions
==============================

Django's default transaction behavior
-------------------------------------

Django's default behavior is to run in autocommit mode. Each query is
immediately committed to the database. :ref:`See below for details
<autocommit-details>`.

..
   Django uses transactions or savepoints automatically to guarantee the
   integrity of ORM operations that require multiple queries, especially
   :ref:`delete() <topics-db-queries-delete>` and :ref:`update()
   <topics-db-queries-update>` queries.

.. versionchanged:: 1.6
    Previous version of Django featured :ref:`a more complicated default
    behavior <transactions-upgrading-from-1.5>`.

.. _tying-transactions-to-http-requests:

Tying transactions to HTTP requests
-----------------------------------

A common way to handle transactions on the web is to wrap each request in a
transaction. Set :setting:`ATOMIC_REQUESTS <DATABASE-ATOMIC_REQUESTS>` to
``True`` in the configuration of each database for which you want to enable
this behavior.

It works like this. When a request starts, Django starts a transaction. If the
response is produced without problems, Django commits the transaction. If the
view function produces an exception, Django rolls back the transaction.
Middleware always runs outside of this transaction.

You may perfom partial commits and rollbacks in your view code, typically with
the :func:`atomic` context manager. However, at the end of the view, either
all the changes will be committed, or none of them.

To disable this behavior for a specific view, you must set the
``transactions_per_request`` attribute of the view function itself to
``False``, like this::

    def my_view(request):
        do_stuff()
    my_view.transactions_per_request = False

.. warning::

    While the simplicity of this transaction model is appealing, it also makes it
    inefficient when traffic increases. Opening a transaction for every view has
    some overhead. The impact on performance depends on the query patterns of your
    application and on how well your database handles locking.

.. admonition:: Per-request transactions and streaming responses

    When a view returns a :class:`~django.http.StreamingHttpResponse`, reading
    the contents of the response will often execute code to generate the
    content. Since the view has already returned, such code runs outside of
    the transaction.

    Generally speaking, it isn't advisable to write to the database while
    generating a streaming response, since there's no sensible way to handle
    errors after starting to send the response.

In practice, this feature simply wraps every view function in the :func:`atomic`
decorator described below.

Note that only the execution of your view in enclosed in the transactions.
Middleware run outside of the transaction, and so does the rendering of
template responses.

.. versionchanged:: 1.6
    Django used to provide this feature via ``TransactionMiddleware``, which is
    now deprecated.

Controlling transactions explicitly
-----------------------------------

.. versionadded:: 1.6

Django provides a single API to control database transactions.

.. function:: atomic(using=None)

    This function creates an atomic block for writes to the database.
    (Atomicity is the defining property of database transactions.)

    When the block completes successfully, the changes are committed to the
    database. When it raises an exception, the changes are rolled back.

    ``atomic`` can be nested. In this case, when an inner block completes
    successfully, its effects can still be rolled back if an exception is
    raised in the outer block at a later point.

    ``atomic`` takes a ``using`` argument which should be the name of a
    database. If this argument isn't provided, Django uses the ``"default"``
    database.

    ``atomic`` is usable both as a `decorator`_::

        from django.db import transaction

        @transaction.atomic
        def viewfunc(request):
            # This code executes inside a transaction.
            do_stuff()

    and as a `context manager`_::

        from django.db import transaction

        def viewfunc(request):
            # This code executes in autocommit mode (Django's default).
            do_stuff()

            with transaction.atomic():
                # This code executes inside a transaction.
                do_more_stuff()

    .. _decorator: http://docs.python.org/glossary.html#term-decorator
    .. _context manager: http://docs.python.org/glossary.html#term-context-manager

    Wrapping ``atomic`` in a try/except block allows for natural handling of
    integrity errors::

        from django.db import IntegrityError, transaction

        @transaction.atomic
        def viewfunc(request):
            do_stuff()

            try:
                with transaction.atomic():
                    do_stuff_that_could_fail()
            except IntegrityError:
                handle_exception()

            do_more_stuff()

    In this example, even if ``do_stuff_that_could_fail()`` causes a database
    error by breaking an integrity constraint, you can execute queries in
    ``do_more_stuff()``, and the changes from ``do_stuff()`` are still there.

    In order to guarantee atomicity, ``atomic`` disables some APIs. Attempting
    to commit, roll back, or change the autocommit state of the database
    connection within an ``atomic`` block will raise an exception.

    ``atomic`` can only be used in autocommit mode. It will raise an exception
    if autocommit is turned off.

    Under the hood, Django's transaction management code:

    - opens a transaction when entering the outermost ``atomic`` block;
    - creates a savepoint when entering an inner ``atomic`` block;
    - releases or rolls back to the savepoint when exiting an inner block;
    - commits or rolls back the transaction when exiting the outermost block.

.. admonition:: Performance considerations

    Open transactions have a performance cost for your database server. To
    minimize this overhead, keep your transactions as short as possible. This
    is especially important of you're using :func:`atomic` in long-running
    processes, outside of Django's request / response cycle.

Autocommit
==========

.. _autocommit-details:

Why Django uses autocommit
--------------------------

In the SQL standards, each SQL query starts a transaction, unless one is
already in progress. Such transactions must then be committed or rolled back.

This isn't always convenient for application developers. To alleviate this
problem, most databases provide an autocommit mode. When autocommit is turned
on, each SQL query is wrapped in its own transaction. In other words, the
transaction is not only automatically started, but also automatically
committed.

:pep:`249`, the Python Database API Specification v2.0, requires autocommit to
be initially turned off. Django overrides this default and turns autocommit
on.

To avoid this, you can :ref:`deactivate the transaction management
<deactivate-transaction-management>`, but it isn't recommended.

.. versionchanged:: 1.6
    Before Django 1.6, autocommit was turned off, and it was emulated by
    forcing a commit after write operations in the ORM.

.. warning::

    If you're using the database API directly — for instance, you're running
    SQL queries with ``cursor.execute()`` — be aware that autocommit is on,
    and consider wrapping your operations in a transaction, with
    :func:`atomic`, to ensure consistency.

.. _deactivate-transaction-management:

Deactivating transaction management
-----------------------------------

You can totally disable Django's transaction management for a given database
by setting :setting:`AUTOCOMMIT <DATABASE-AUTOCOMMIT>` to ``False`` in its
configuration. If you do this, Django won't enable autocommit, and won't
perform any commits. You'll get the regular behavior of the underlying
database library.

This requires you to commit explicitly every transaction, even those started
by Django or by third-party libraries. Thus, this is best used in situations
where you want to run your own transaction-controlling middleware or do
something really strange.

.. versionchanged:: 1.6
    This used to be controlled by the ``TRANSACTIONS_MANAGED`` setting.

Low-level APIs
==============

.. warning::

    Always prefer :func:`atomic` if possible at all. It accounts for the
    idiosyncrasies of each database and prevents invalid operations.

    The low level APIs are only useful if you're implementing your own
    transaction management.

.. _managing-autocommit:

Autocommit
----------

.. versionadded:: 1.6

Django provides a straightforward API to manage the autocommit state of each
database connection, if you need to.

.. function:: get_autocommit(using=None)

.. function:: set_autocommit(using=None, autocommit=True)

These functions take a ``using`` argument which should be the name of a
database. If it isn't provided, Django uses the ``"default"`` database.

Autocommit is initially turned on. If you turn it off, it's your
responsibility to restore it.

Once you turn autocommit off, you get the default behavior of your database
adapter, and Django won't help you. Although that behavior is specified in
:pep:`249`, implementations of adapters aren't always consistent with one
another. Review the documentation of the adapter you're using carefully.

You must ensure that no transaction is active, usually by issuing a
:func:`commit` or a :func:`rollback`, before turning autocommit back on.

:func:`atomic` requires autocommit to be turned on; it will raise an exception
if autocommit is off. Django will also refuse to turn autocommit off when an
:func:`atomic` block is active, because that would break atomicity.

Transactions
------------

A transaction is an atomic set of database queries. Even if your program
crashes, the database guarantees that either all the changes will be applied,
or none of them.

Django doesn't provide an API to start a transaction. The expected way to
start a transaction is to disable autocommit with :func:`set_autocommit`.

Once you're in a transaction, you can choose either to apply the changes
you've performed until this point with :func:`commit`, or to cancel them with
:func:`rollback`.

.. function:: commit(using=None)

.. function:: rollback(using=None)

These functions take a ``using`` argument which should be the name of a
database. If it isn't provided, Django uses the ``"default"`` database.

Django will refuse to commit or to rollback when an :func:`atomic` block is
active, because that would break atomicity.

.. _topics-db-transactions-savepoints:

Savepoints
----------

A savepoint is a marker within a transaction that enables you to roll back
part of a transaction, rather than the full transaction. Savepoints are
available with the SQLite (≥ 3.6.8), PostgreSQL, Oracle and MySQL (when using
the InnoDB storage engine) backends. Other backends provide the savepoint
functions, but they're empty operations -- they don't actually do anything.

Savepoints aren't especially useful if you are using autocommit, the default
behavior of Django. However, once you open a transaction with :func:`atomic`,
you build up a series of database operations awaiting a commit or rollback. If
you issue a rollback, the entire transaction is rolled back. Savepoints
provide the ability to perform a fine-grained rollback, rather than the full
rollback that would be performed by ``transaction.rollback()``.

.. versionchanged:: 1.6

When the :func:`atomic` decorator is nested, it creates a savepoint to allow
partial commit or rollback. You're strongly encouraged to use :func:`atomic`
rather than the functions described below, but they're still part of the
public API, and there's no plan to deprecate them.

Each of these functions takes a ``using`` argument which should be the name of
a database for which the behavior applies.  If no ``using`` argument is
provided then the ``"default"`` database is used.

Savepoints are controlled by three functions in :mod:`django.db.transaction`:

.. function:: savepoint(using=None)

    Creates a new savepoint. This marks a point in the transaction that
    is known to be in a "good" state.

    Returns the savepoint ID (``sid``).

.. function:: savepoint_commit(sid, using=None)

    Releases savepoint ``sid``. The changes performed since the savepoint was
    created become part of the transaction.

.. function:: savepoint_rollback(sid, using=None)

    Rolls back the transaction to savepoint ``sid``.

These functions do nothing if savepoints aren't supported or if the database
is in autocommit mode.

In addition, there's a utility function:

.. function:: clean_savepoints(using=None)

    Resets the counter used to generate unique savepoint IDs.

The following example demonstrates the use of savepoints::

    from django.db import transaction

    # open a transaction
    @transaction.atomic
    def viewfunc(request):

      a.save()
      # transaction now contains a.save()

      sid = transaction.savepoint()

      b.save()
      # transaction now contains a.save() and b.save()

      if want_to_keep_b:
          transaction.savepoint_commit(sid)
          # open transaction still contains a.save() and b.save()
      else:
          transaction.savepoint_rollback(sid)
          # open transaction now contains only a.save()

Database-specific notes
=======================

Savepoints in SQLite
--------------------

While SQLite ≥ 3.6.8 supports savepoints, a flaw in the design of the
:mod:`sqlite3` makes them hardly usable.

When autocommit is enabled, savepoints don't make sense. When it's disabled,
:mod:`sqlite3` commits implicitly before savepoint-related statement. (It
commits before any statement other than ``SELECT``, ``INSERT``, ``UPDATE``,
``DELETE`` and ``REPLACE``.)

As a consequence, savepoints are only usable inside a transaction ie. inside
an :func:`atomic` block.

Transactions in MySQL
---------------------

If you're using MySQL, your tables may or may not support transactions; it
depends on your MySQL version and the table types you're using. (By
"table types," we mean something like "InnoDB" or "MyISAM".) MySQL transaction
peculiarities are outside the scope of this article, but the MySQL site has
`information on MySQL transactions`_.

If your MySQL setup does *not* support transactions, then Django will function
in autocommit mode: Statements will be executed and committed as soon as
they're called. If your MySQL setup *does* support transactions, Django will
handle transactions as explained in this document.

.. _information on MySQL transactions: http://dev.mysql.com/doc/refman/5.0/en/sql-syntax-transactions.html

Handling exceptions within PostgreSQL transactions
--------------------------------------------------

When a call to a PostgreSQL cursor raises an exception (typically
``IntegrityError``), all subsequent SQL in the same transaction will fail with
the error "current transaction is aborted, queries ignored until end of
transaction block". Whilst simple use of ``save()`` is unlikely to raise an
exception in PostgreSQL, there are more advanced usage patterns which
might, such as saving objects with unique fields, saving using the
force_insert/force_update flag, or invoking custom SQL.

There are several ways to recover from this sort of error.

Transaction rollback
~~~~~~~~~~~~~~~~~~~~

The first option is to roll back the entire transaction. For example::

    a.save() # Succeeds, but may be undone by transaction rollback
    try:
        b.save() # Could throw exception
    except IntegrityError:
        transaction.rollback()
    c.save() # Succeeds, but a.save() may have been undone

Calling ``transaction.rollback()`` rolls back the entire transaction. Any
uncommitted database operations will be lost. In this example, the changes
made by ``a.save()`` would be lost, even though that operation raised no error
itself.

Savepoint rollback
~~~~~~~~~~~~~~~~~~

You can use :ref:`savepoints <topics-db-transactions-savepoints>` to control
the extent of a rollback. Before performing a database operation that could
fail, you can set or update the savepoint; that way, if the operation fails,
you can roll back the single offending operation, rather than the entire
transaction. For example::

    a.save() # Succeeds, and never undone by savepoint rollback
    try:
        sid = transaction.savepoint()
        b.save() # Could throw exception
        transaction.savepoint_commit(sid)
    except IntegrityError:
        transaction.savepoint_rollback(sid)
    c.save() # Succeeds, and a.save() is never undone

In this example, ``a.save()`` will not be undone in the case where
``b.save()`` raises an exception.

.. _transactions-upgrading-from-1.5:

Changes from Django 1.5 and earlier
===================================

The features described below were deprecated in Django 1.6 and will be removed
in Django 1.8. They're documented in order to ease the migration to the new
transaction management APIs.

Legacy APIs
-----------

The following functions, defined in ``django.db.transaction``, provided a way
to control transactions on a per-function or per-code-block basis. They could
be used as decorators or as context managers, and they accepted a ``using``
argument, exactly like :func:`atomic`.

.. function:: autocommit

    Enable Django's default autocommit behavior.

    Transactions will be committed as soon as you call ``model.save()``,
    ``model.delete()``, or any other function that writes to the database.

.. function:: commit_on_success

    Use a single transaction for all the work done in a function.

    If the function returns successfully, then Django will commit all work done
    within the function at that point. If the function raises an exception,
    though, Django will roll back the transaction.

.. function:: commit_manually

    Tells Django you'll be managing the transaction on your own.

    Whether you are writing or simply reading from the database, you must
    ``commit()`` or ``rollback()`` explicitly or Django will raise a
    :exc:`TransactionManagementError` exception. This is required when reading
    from the database because ``SELECT`` statements may call functions which
    modify tables, and thus it is impossible to know if any data has been
    modified.

.. _transaction-states:

Transaction states
------------------

The three functions described above relied on a concept called "transaction
states". This mechanisme was deprecated in Django 1.6, but it's still
available until Django 1.8..

At any time, each database connection is in one of these two states:

- **auto mode**: autocommit is enabled;
- **managed mode**: autocommit is disabled.

Django starts in auto mode. ``TransactionMiddleware``,
:func:`commit_on_success` and :func:`commit_manually` activate managed mode;
:func:`autocommit` activates auto mode.

Internally, Django keeps a stack of states. Activations and deactivations must
be balanced.

For example, ``commit_on_success`` switches to managed mode when entering the
block of code it controls; when exiting the block, it commits or rollbacks,
and switches back to auto mode.

So :func:`commit_on_success` really has two effects: it changes the
transaction state and it defines an transaction block. Nesting will give the
expected results in terms of transaction state, but not in terms of
transaction semantics. Most often, the inner block will commit, breaking the
atomicity of the outer block.

:func:`autocommit` and :func:`commit_manually` have similar limitations.

API changes
-----------

Transaction middleware
~~~~~~~~~~~~~~~~~~~~~~

In Django 1.6, ``TransactionMiddleware`` is deprecated and replaced
:setting:`ATOMIC_REQUESTS <DATABASE-ATOMIC_REQUESTS>`. While the general
behavior is the same, there are a few differences.

With the transaction middleware, it was still possible to switch to autocommit
or to commit explicitly in a view. Since :func:`atomic` guarantees atomicity,
this isn't allowed any longer.

To avoid wrapping a particular view in a transaction, instead of::

    @transaction.autocommit
    def my_view(request):
        do_stuff()

you must now use this pattern::

    def my_view(request):
        do_stuff()
    my_view.transactions_per_request = False

The transaction middleware applied not only to view functions, but also to
middleware modules that come after it. For instance, if you used the session
middleware after the transaction middleware, session creation was part of the
transaction. :setting:`ATOMIC_REQUESTS <DATABASE-ATOMIC_REQUESTS>` only
applies to the view itself.

Managing transactions
~~~~~~~~~~~~~~~~~~~~~

Starting with Django 1.6, :func:`atomic` is the only supported API for
defining a transaction. Unlike the deprecated APIs, it's nestable and always
guarantees atomicity.

In most cases, it will be a drop-in replacement for :func:`commit_on_success`.

During the deprecation period, it's possible to use :func:`atomic` within
:func:`autocommit`, :func:`commit_on_success` or :func:`commit_manually`.
However, the reverse is forbidden, because nesting the old decorators /
context managers breaks atomicity.

If you enter :func:`atomic` while you're in managed mode, it will trigger a
commit to start from a clean slate.

Managing autocommit
~~~~~~~~~~~~~~~~~~~

Django 1.6 introduces an explicit :ref:`API for mananging autocommit
<managing-autocommit>`.

To disable autocommit temporarily, instead of::

    with transaction.commit_manually():
        # do stuff

you should now use::

    transaction.set_autocommit(autocommit=False)
    try:
        # do stuff
    finally:
        transaction.set_autocommit(autocommit=True)

To enable autocommit temporarily, instead of::

    with transaction.autocommit():
        # do stuff

you should now use::

    transaction.set_autocommit(autocommit=True)
    try:
        # do stuff
    finally:
        transaction.set_autocommit(autocommit=False)

Disabling transaction management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Instead of setting ``TRANSACTIONS_MANAGED = True``, set the ``AUTOCOMMIT`` key
to ``False`` in the configuration of each database, as explained in :ref
:`deactivate-transaction-management`.

Backwards incompatibilities
---------------------------

Since version 1.6, Django uses database-level autocommit in auto mode.
Previously, it implemented application-level autocommit by triggering a commit
after each ORM write.

As a consequence, each database query (for instance, an ORM read) started a
transaction that lasted until the next ORM write. Such "automatic
transactions" no longer exist in Django 1.6.

There are four known scenarios where this is backwards-incompatible.

Note that managed mode isn't affected at all. This section assumes auto mode.
See the :ref:`description of modes <transaction-states>` above.

Sequences of custom SQL queries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you're executing several :ref:`custom SQL queries <executing-custom-sql>`
in a row, each one now runs in its own transaction, instead of sharing the
same "automatic transaction". If you need to enforce atomicity, you must wrap
the sequence of queries in :func:`commit_on_success`.

To check for this problem, look for calls to ``cursor.execute()``. They're
usually followed by a call to ``transaction.commit_unless_managed``, which
isn't necessary any more and should be removed.

Select for update
~~~~~~~~~~~~~~~~~

If you were relying on "automatic transactions" to provide locking between
:meth:`~django.db.models.query.QuerySet.select_for_update` and a subsequent
write operation — an extremely fragile design, but nonetheless possible — you
must wrap the relevant code in :func:`atomic`.

Using a high isolation level
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you were using the "repeatable read" isolation level or higher, and if you
relied on "automatic transactions" to guarantee consistency between successive
reads, the new behavior might be backwards-incompatible. To enforce
consistency, you must wrap such sequences in :func:`atomic`.

MySQL defaults to "repeatable read" and SQLite to "serializable"; they may be
affected by this problem.

At the "read committed" isolation level or lower, "automatic transactions"
have no effect on the semantics of any sequence of ORM operations.

PostgreSQL and Oracle default to "read committed" and aren't affected, unless
you changed the isolation level.

Using unsupported database features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With triggers, views, or functions, it's possible to make ORM reads result in
database modifications. Django 1.5 and earlier doesn't deal with this case and
it's theoretically possible to observe a different behavior after upgrading to
Django 1.6 or later. In doubt, use :func:`atomic` to enforce integrity.