databases.txt 42 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120
  1. =========
  2. Databases
  3. =========
  4. Django officially supports the following databases:
  5. * :ref:`PostgreSQL <postgresql-notes>`
  6. * :ref:`MariaDB <mariadb-notes>`
  7. * :ref:`MySQL <mysql-notes>`
  8. * :ref:`Oracle <oracle-notes>`
  9. * :ref:`SQLite <sqlite-notes>`
  10. There are also a number of :ref:`database backends provided by third parties
  11. <third-party-notes>`.
  12. Django attempts to support as many features as possible on all database
  13. backends. However, not all database backends are alike, and we've had to make
  14. design decisions on which features to support and which assumptions we can make
  15. safely.
  16. This file describes some of the features that might be relevant to Django
  17. usage. It is not intended as a replacement for server-specific documentation or
  18. reference manuals.
  19. General notes
  20. =============
  21. .. _persistent-database-connections:
  22. Persistent connections
  23. ----------------------
  24. Persistent connections avoid the overhead of re-establishing a connection to
  25. the database in each request. They're controlled by the
  26. :setting:`CONN_MAX_AGE` parameter which defines the maximum lifetime of a
  27. connection. It can be set independently for each database.
  28. The default value is ``0``, preserving the historical behavior of closing the
  29. database connection at the end of each request. To enable persistent
  30. connections, set :setting:`CONN_MAX_AGE` to a positive integer of seconds. For
  31. unlimited persistent connections, set it to ``None``.
  32. Connection management
  33. ~~~~~~~~~~~~~~~~~~~~~
  34. Django opens a connection to the database when it first makes a database
  35. query. It keeps this connection open and reuses it in subsequent requests.
  36. Django closes the connection once it exceeds the maximum age defined by
  37. :setting:`CONN_MAX_AGE` or when it isn't usable any longer.
  38. In detail, Django automatically opens a connection to the database whenever it
  39. needs one and doesn't have one already — either because this is the first
  40. connection, or because the previous connection was closed.
  41. At the beginning of each request, Django closes the connection if it has
  42. reached its maximum age. If your database terminates idle connections after
  43. some time, you should set :setting:`CONN_MAX_AGE` to a lower value, so that
  44. Django doesn't attempt to use a connection that has been terminated by the
  45. database server. (This problem may only affect very low traffic sites.)
  46. At the end of each request, Django closes the connection if it has reached its
  47. maximum age or if it is in an unrecoverable error state. If any database
  48. errors have occurred while processing the requests, Django checks whether the
  49. connection still works, and closes it if it doesn't. Thus, database errors
  50. affect at most one request; if the connection becomes unusable, the next
  51. request gets a fresh connection.
  52. Caveats
  53. ~~~~~~~
  54. Since each thread maintains its own connection, your database must support at
  55. least as many simultaneous connections as you have worker threads.
  56. Sometimes a database won't be accessed by the majority of your views, for
  57. example because it's the database of an external system, or thanks to caching.
  58. In such cases, you should set :setting:`CONN_MAX_AGE` to a low value or even
  59. ``0``, because it doesn't make sense to maintain a connection that's unlikely
  60. to be reused. This will help keep the number of simultaneous connections to
  61. this database small.
  62. The development server creates a new thread for each request it handles,
  63. negating the effect of persistent connections. Don't enable them during
  64. development.
  65. When Django establishes a connection to the database, it sets up appropriate
  66. parameters, depending on the backend being used. If you enable persistent
  67. connections, this setup is no longer repeated every request. If you modify
  68. parameters such as the connection's isolation level or time zone, you should
  69. either restore Django's defaults at the end of each request, force an
  70. appropriate value at the beginning of each request, or disable persistent
  71. connections.
  72. Encoding
  73. --------
  74. Django assumes that all databases use UTF-8 encoding. Using other encodings may
  75. result in unexpected behavior such as "value too long" errors from your
  76. database for data that is valid in Django. See the database specific notes
  77. below for information on how to set up your database correctly.
  78. .. _postgresql-notes:
  79. PostgreSQL notes
  80. ================
  81. Django supports PostgreSQL 10 and higher. `psycopg2`_ 2.5.4 or higher is
  82. required, though the latest release is recommended.
  83. .. _psycopg2: https://www.psycopg.org/
  84. .. _postgresql-connection-settings:
  85. PostgreSQL connection settings
  86. -------------------------------
  87. See :setting:`HOST` for details.
  88. To connect using a service name from the `connection service file`_ and a
  89. password from the `password file`_, you must specify them in the
  90. :setting:`OPTIONS` part of your database configuration in :setting:`DATABASES`:
  91. .. code-block:: python
  92. :caption: settings.py
  93. DATABASES = {
  94. 'default': {
  95. 'ENGINE': 'django.db.backends.postgresql',
  96. 'OPTIONS': {
  97. 'service': 'my_service',
  98. 'passfile': '.my_pgpass',
  99. },
  100. }
  101. }
  102. .. code-block:: text
  103. :caption: .pg_service.conf
  104. [my_service]
  105. host=localhost
  106. user=USER
  107. dbname=NAME
  108. port=5432
  109. .. code-block:: text
  110. :caption: .my_pgpass
  111. localhost:5432:NAME:USER:PASSWORD
  112. .. _connection service file: https://www.postgresql.org/docs/current/libpq-pgservice.html
  113. .. _password file: https://www.postgresql.org/docs/current/libpq-pgpass.html
  114. .. versionchanged:: 4.0
  115. Support for connecting by a service name, and specifying a password file
  116. was added.
  117. Optimizing PostgreSQL's configuration
  118. -------------------------------------
  119. Django needs the following parameters for its database connections:
  120. - ``client_encoding``: ``'UTF8'``,
  121. - ``default_transaction_isolation``: ``'read committed'`` by default,
  122. or the value set in the connection options (see below),
  123. - ``timezone``:
  124. - when :setting:`USE_TZ` is ``True``, ``'UTC'`` by default, or the
  125. :setting:`TIME_ZONE <DATABASE-TIME_ZONE>` value set for the connection,
  126. - when :setting:`USE_TZ` is ``False``, the value of the global
  127. :setting:`TIME_ZONE` setting.
  128. If these parameters already have the correct values, Django won't set them for
  129. every new connection, which improves performance slightly. You can configure
  130. them directly in :file:`postgresql.conf` or more conveniently per database
  131. user with `ALTER ROLE`_.
  132. Django will work just fine without this optimization, but each new connection
  133. will do some additional queries to set these parameters.
  134. .. _ALTER ROLE: https://www.postgresql.org/docs/current/sql-alterrole.html
  135. .. _database-isolation-level:
  136. Isolation level
  137. ---------------
  138. Like PostgreSQL itself, Django defaults to the ``READ COMMITTED`` `isolation
  139. level`_. If you need a higher isolation level such as ``REPEATABLE READ`` or
  140. ``SERIALIZABLE``, set it in the :setting:`OPTIONS` part of your database
  141. configuration in :setting:`DATABASES`::
  142. import psycopg2.extensions
  143. DATABASES = {
  144. # ...
  145. 'OPTIONS': {
  146. 'isolation_level': psycopg2.extensions.ISOLATION_LEVEL_SERIALIZABLE,
  147. },
  148. }
  149. .. note::
  150. Under higher isolation levels, your application should be prepared to
  151. handle exceptions raised on serialization failures. This option is
  152. designed for advanced uses.
  153. .. _isolation level: https://www.postgresql.org/docs/current/transaction-iso.html
  154. Indexes for ``varchar`` and ``text`` columns
  155. --------------------------------------------
  156. When specifying ``db_index=True`` on your model fields, Django typically
  157. outputs a single ``CREATE INDEX`` statement. However, if the database type
  158. for the field is either ``varchar`` or ``text`` (e.g., used by ``CharField``,
  159. ``FileField``, and ``TextField``), then Django will create
  160. an additional index that uses an appropriate `PostgreSQL operator class`_
  161. for the column. The extra index is necessary to correctly perform
  162. lookups that use the ``LIKE`` operator in their SQL, as is done with the
  163. ``contains`` and ``startswith`` lookup types.
  164. .. _PostgreSQL operator class: https://www.postgresql.org/docs/current/indexes-opclass.html
  165. Migration operation for adding extensions
  166. -----------------------------------------
  167. If you need to add a PostgreSQL extension (like ``hstore``, ``postgis``, etc.)
  168. using a migration, use the
  169. :class:`~django.contrib.postgres.operations.CreateExtension` operation.
  170. .. _postgresql-server-side-cursors:
  171. Server-side cursors
  172. -------------------
  173. When using :meth:`QuerySet.iterator()
  174. <django.db.models.query.QuerySet.iterator>`, Django opens a :ref:`server-side
  175. cursor <psycopg2:server-side-cursors>`. By default, PostgreSQL assumes that
  176. only the first 10% of the results of cursor queries will be fetched. The query
  177. planner spends less time planning the query and starts returning results
  178. faster, but this could diminish performance if more than 10% of the results are
  179. retrieved. PostgreSQL's assumptions on the number of rows retrieved for a
  180. cursor query is controlled with the `cursor_tuple_fraction`_ option.
  181. .. _cursor_tuple_fraction: https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-CURSOR-TUPLE-FRACTION
  182. .. _transaction-pooling-server-side-cursors:
  183. Transaction pooling and server-side cursors
  184. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  185. Using a connection pooler in transaction pooling mode (e.g. `PgBouncer`_)
  186. requires disabling server-side cursors for that connection.
  187. Server-side cursors are local to a connection and remain open at the end of a
  188. transaction when :setting:`AUTOCOMMIT <DATABASE-AUTOCOMMIT>` is ``True``. A
  189. subsequent transaction may attempt to fetch more results from a server-side
  190. cursor. In transaction pooling mode, there's no guarantee that subsequent
  191. transactions will use the same connection. If a different connection is used,
  192. an error is raised when the transaction references the server-side cursor,
  193. because server-side cursors are only accessible in the connection in which they
  194. were created.
  195. One solution is to disable server-side cursors for a connection in
  196. :setting:`DATABASES` by setting :setting:`DISABLE_SERVER_SIDE_CURSORS
  197. <DATABASE-DISABLE_SERVER_SIDE_CURSORS>` to ``True``.
  198. To benefit from server-side cursors in transaction pooling mode, you could set
  199. up :doc:`another connection to the database </topics/db/multi-db>` in order to
  200. perform queries that use server-side cursors. This connection needs to either
  201. be directly to the database or to a connection pooler in session pooling mode.
  202. Another option is to wrap each ``QuerySet`` using server-side cursors in an
  203. :func:`~django.db.transaction.atomic` block, because it disables ``autocommit``
  204. for the duration of the transaction. This way, the server-side cursor will only
  205. live for the duration of the transaction.
  206. .. _PgBouncer: https://www.pgbouncer.org/
  207. .. _manually-specified-autoincrement-pk:
  208. Manually-specifying values of auto-incrementing primary keys
  209. ------------------------------------------------------------
  210. Django uses PostgreSQL's `SERIAL data type`_ to store auto-incrementing primary
  211. keys. A ``SERIAL`` column is populated with values from a `sequence`_ that
  212. keeps track of the next available value. Manually assigning a value to an
  213. auto-incrementing field doesn't update the field's sequence, which might later
  214. cause a conflict. For example::
  215. >>> from django.contrib.auth.models import User
  216. >>> User.objects.create(username='alice', pk=1)
  217. <User: alice>
  218. >>> # The sequence hasn't been updated; its next value is 1.
  219. >>> User.objects.create(username='bob')
  220. ...
  221. IntegrityError: duplicate key value violates unique constraint
  222. "auth_user_pkey" DETAIL: Key (id)=(1) already exists.
  223. If you need to specify such values, reset the sequence afterwards to avoid
  224. reusing a value that's already in the table. The :djadmin:`sqlsequencereset`
  225. management command generates the SQL statements to do that.
  226. .. _SERIAL data type: https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-SERIAL
  227. .. _sequence: https://www.postgresql.org/docs/current/sql-createsequence.html
  228. Test database templates
  229. -----------------------
  230. You can use the :setting:`TEST['TEMPLATE'] <TEST_TEMPLATE>` setting to specify
  231. a `template`_ (e.g. ``'template0'``) from which to create a test database.
  232. .. _template: https://www.postgresql.org/docs/current/sql-createdatabase.html
  233. Speeding up test execution with non-durable settings
  234. ----------------------------------------------------
  235. You can speed up test execution times by `configuring PostgreSQL to be
  236. non-durable <https://www.postgresql.org/docs/current/non-durability.html>`_.
  237. .. warning::
  238. This is dangerous: it will make your database more susceptible to data loss
  239. or corruption in the case of a server crash or power loss. Only use this on
  240. a development machine where you can easily restore the entire contents of
  241. all databases in the cluster.
  242. .. _mariadb-notes:
  243. MariaDB notes
  244. =============
  245. Django supports MariaDB 10.2 and higher.
  246. To use MariaDB, use the MySQL backend, which is shared between the two. See the
  247. :ref:`MySQL notes <mysql-notes>` for more details.
  248. .. _mysql-notes:
  249. MySQL notes
  250. ===========
  251. Version support
  252. ---------------
  253. Django supports MySQL 5.7 and higher.
  254. Django's ``inspectdb`` feature uses the ``information_schema`` database, which
  255. contains detailed data on all database schemas.
  256. Django expects the database to support Unicode (UTF-8 encoding) and delegates to
  257. it the task of enforcing transactions and referential integrity. It is important
  258. to be aware of the fact that the two latter ones aren't actually enforced by
  259. MySQL when using the MyISAM storage engine, see the next section.
  260. .. _mysql-storage-engines:
  261. Storage engines
  262. ---------------
  263. MySQL has several `storage engines`_. You can change the default storage engine
  264. in the server configuration.
  265. MySQL's default storage engine is InnoDB_. This engine is fully transactional
  266. and supports foreign key references. It's the recommended choice. However, the
  267. InnoDB autoincrement counter is lost on a MySQL restart because it does not
  268. remember the ``AUTO_INCREMENT`` value, instead recreating it as "max(id)+1".
  269. This may result in an inadvertent reuse of :class:`~django.db.models.AutoField`
  270. values.
  271. The main drawbacks of MyISAM_ are that it doesn't support transactions or
  272. enforce foreign-key constraints.
  273. .. _storage engines: https://dev.mysql.com/doc/refman/en/storage-engines.html
  274. .. _MyISAM: https://dev.mysql.com/doc/refman/en/myisam-storage-engine.html
  275. .. _InnoDB: https://dev.mysql.com/doc/refman/en/innodb-storage-engine.html
  276. .. _mysql-db-api-drivers:
  277. MySQL DB API Drivers
  278. --------------------
  279. MySQL has a couple drivers that implement the Python Database API described in
  280. :pep:`249`:
  281. - `mysqlclient`_ is a native driver. It's **the recommended choice**.
  282. - `MySQL Connector/Python`_ is a pure Python driver from Oracle that does not
  283. require the MySQL client library or any Python modules outside the standard
  284. library.
  285. .. _mysqlclient: https://pypi.org/project/mysqlclient/
  286. .. _MySQL Connector/Python: https://dev.mysql.com/downloads/connector/python/
  287. These drivers are thread-safe and provide connection pooling.
  288. In addition to a DB API driver, Django needs an adapter to access the database
  289. drivers from its ORM. Django provides an adapter for mysqlclient while MySQL
  290. Connector/Python includes `its own`_.
  291. .. _its own: https://dev.mysql.com/doc/connector-python/en/connector-python-django-backend.html
  292. mysqlclient
  293. ~~~~~~~~~~~
  294. Django requires `mysqlclient`_ 1.4.0 or later.
  295. MySQL Connector/Python
  296. ~~~~~~~~~~~~~~~~~~~~~~
  297. MySQL Connector/Python is available from the `download page`_.
  298. The Django adapter is available in versions 1.1.X and later. It may not
  299. support the most recent releases of Django.
  300. .. _download page: https://dev.mysql.com/downloads/connector/python/
  301. .. _mysql-time-zone-definitions:
  302. Time zone definitions
  303. ---------------------
  304. If you plan on using Django's :doc:`timezone support </topics/i18n/timezones>`,
  305. use `mysql_tzinfo_to_sql`_ to load time zone tables into the MySQL database.
  306. This needs to be done just once for your MySQL server, not per database.
  307. .. _mysql_tzinfo_to_sql: https://dev.mysql.com/doc/refman/en/mysql-tzinfo-to-sql.html
  308. Creating your database
  309. ----------------------
  310. You can `create your database`_ using the command-line tools and this SQL::
  311. CREATE DATABASE <dbname> CHARACTER SET utf8;
  312. This ensures all tables and columns will use UTF-8 by default.
  313. .. _create your database: https://dev.mysql.com/doc/refman/en/create-database.html
  314. .. _mysql-collation:
  315. Collation settings
  316. ~~~~~~~~~~~~~~~~~~
  317. The collation setting for a column controls the order in which data is sorted
  318. as well as what strings compare as equal. You can specify the ``db_collation``
  319. parameter to set the collation name of the column for
  320. :attr:`CharField <django.db.models.CharField.db_collation>` and
  321. :attr:`TextField <django.db.models.TextField.db_collation>`.
  322. The collation can also be set on a database-wide level and per-table. This is
  323. `documented thoroughly`_ in the MySQL documentation. In such cases, you must
  324. set the collation by directly manipulating the database settings or tables.
  325. Django doesn't provide an API to change them.
  326. .. _documented thoroughly: https://dev.mysql.com/doc/refman/en/charset.html
  327. By default, with a UTF-8 database, MySQL will use the
  328. ``utf8_general_ci`` collation. This results in all string equality
  329. comparisons being done in a *case-insensitive* manner. That is, ``"Fred"`` and
  330. ``"freD"`` are considered equal at the database level. If you have a unique
  331. constraint on a field, it would be illegal to try to insert both ``"aa"`` and
  332. ``"AA"`` into the same column, since they compare as equal (and, hence,
  333. non-unique) with the default collation. If you want case-sensitive comparisons
  334. on a particular column or table, change the column or table to use the
  335. ``utf8_bin`` collation.
  336. Please note that according to `MySQL Unicode Character Sets`_, comparisons for
  337. the ``utf8_general_ci`` collation are faster, but slightly less correct, than
  338. comparisons for ``utf8_unicode_ci``. If this is acceptable for your application,
  339. you should use ``utf8_general_ci`` because it is faster. If this is not acceptable
  340. (for example, if you require German dictionary order), use ``utf8_unicode_ci``
  341. because it is more accurate.
  342. .. _MySQL Unicode Character Sets: https://dev.mysql.com/doc/refman/en/charset-unicode-sets.html
  343. .. warning::
  344. Model formsets validate unique fields in a case-sensitive manner. Thus when
  345. using a case-insensitive collation, a formset with unique field values that
  346. differ only by case will pass validation, but upon calling ``save()``, an
  347. ``IntegrityError`` will be raised.
  348. .. versionchanged:: 3.2
  349. Support for setting a database collation for the field was added.
  350. Connecting to the database
  351. --------------------------
  352. Refer to the :doc:`settings documentation </ref/settings>`.
  353. Connection settings are used in this order:
  354. #. :setting:`OPTIONS`.
  355. #. :setting:`NAME`, :setting:`USER`, :setting:`PASSWORD`, :setting:`HOST`,
  356. :setting:`PORT`
  357. #. MySQL option files.
  358. In other words, if you set the name of the database in :setting:`OPTIONS`,
  359. this will take precedence over :setting:`NAME`, which would override
  360. anything in a `MySQL option file`_.
  361. Here's a sample configuration which uses a MySQL option file::
  362. # settings.py
  363. DATABASES = {
  364. 'default': {
  365. 'ENGINE': 'django.db.backends.mysql',
  366. 'OPTIONS': {
  367. 'read_default_file': '/path/to/my.cnf',
  368. },
  369. }
  370. }
  371. # my.cnf
  372. [client]
  373. database = NAME
  374. user = USER
  375. password = PASSWORD
  376. default-character-set = utf8
  377. Several other `MySQLdb connection options`_ may be useful, such as ``ssl``,
  378. ``init_command``, and ``sql_mode``.
  379. .. _MySQL option file: https://dev.mysql.com/doc/refman/en/option-files.html
  380. .. _MySQLdb connection options: https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes
  381. .. _mysql-sql-mode:
  382. Setting ``sql_mode``
  383. ~~~~~~~~~~~~~~~~~~~~
  384. From MySQL 5.7 onwards, the default value of the ``sql_mode`` option contains
  385. ``STRICT_TRANS_TABLES``. That option escalates warnings into errors when data
  386. are truncated upon insertion, so Django highly recommends activating a
  387. `strict mode`_ for MySQL to prevent data loss (either ``STRICT_TRANS_TABLES``
  388. or ``STRICT_ALL_TABLES``).
  389. .. _strict mode: https://dev.mysql.com/doc/refman/en/sql-mode.html#sql-mode-strict
  390. If you need to customize the SQL mode, you can set the ``sql_mode`` variable
  391. like other MySQL options: either in a config file or with the entry
  392. ``'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"`` in the
  393. :setting:`OPTIONS` part of your database configuration in :setting:`DATABASES`.
  394. .. _mysql-isolation-level:
  395. Isolation level
  396. ~~~~~~~~~~~~~~~
  397. When running concurrent loads, database transactions from different sessions
  398. (say, separate threads handling different requests) may interact with each
  399. other. These interactions are affected by each session's `transaction isolation
  400. level`_. You can set a connection's isolation level with an
  401. ``'isolation_level'`` entry in the :setting:`OPTIONS` part of your database
  402. configuration in :setting:`DATABASES`. Valid values for
  403. this entry are the four standard isolation levels:
  404. * ``'read uncommitted'``
  405. * ``'read committed'``
  406. * ``'repeatable read'``
  407. * ``'serializable'``
  408. or ``None`` to use the server's configured isolation level. However, Django
  409. works best with and defaults to read committed rather than MySQL's default,
  410. repeatable read. Data loss is possible with repeatable read. In particular,
  411. you may see cases where :meth:`~django.db.models.query.QuerySet.get_or_create`
  412. will raise an :exc:`~django.db.IntegrityError` but the object won't appear in
  413. a subsequent :meth:`~django.db.models.query.QuerySet.get` call.
  414. .. _transaction isolation level: https://dev.mysql.com/doc/refman/en/innodb-transaction-isolation-levels.html
  415. Creating your tables
  416. --------------------
  417. When Django generates the schema, it doesn't specify a storage engine, so
  418. tables will be created with whatever default storage engine your database
  419. server is configured for. The easiest solution is to set your database server's
  420. default storage engine to the desired engine.
  421. If you're using a hosting service and can't change your server's default
  422. storage engine, you have a couple of options.
  423. * After the tables are created, execute an ``ALTER TABLE`` statement to
  424. convert a table to a new storage engine (such as InnoDB)::
  425. ALTER TABLE <tablename> ENGINE=INNODB;
  426. This can be tedious if you have a lot of tables.
  427. * Another option is to use the ``init_command`` option for MySQLdb prior to
  428. creating your tables::
  429. 'OPTIONS': {
  430. 'init_command': 'SET default_storage_engine=INNODB',
  431. }
  432. This sets the default storage engine upon connecting to the database.
  433. After your tables have been created, you should remove this option as it
  434. adds a query that is only needed during table creation to each database
  435. connection.
  436. Table names
  437. -----------
  438. There are `known issues`_ in even the latest versions of MySQL that can cause the
  439. case of a table name to be altered when certain SQL statements are executed
  440. under certain conditions. It is recommended that you use lowercase table
  441. names, if possible, to avoid any problems that might arise from this behavior.
  442. Django uses lowercase table names when it auto-generates table names from
  443. models, so this is mainly a consideration if you are overriding the table name
  444. via the :class:`~django.db.models.Options.db_table` parameter.
  445. .. _known issues: https://bugs.mysql.com/bug.php?id=48875
  446. Savepoints
  447. ----------
  448. Both the Django ORM and MySQL (when using the InnoDB :ref:`storage engine
  449. <mysql-storage-engines>`) support database :ref:`savepoints
  450. <topics-db-transactions-savepoints>`.
  451. If you use the MyISAM storage engine please be aware of the fact that you will
  452. receive database-generated errors if you try to use the :ref:`savepoint-related
  453. methods of the transactions API <topics-db-transactions-savepoints>`. The reason
  454. for this is that detecting the storage engine of a MySQL database/table is an
  455. expensive operation so it was decided it isn't worth to dynamically convert
  456. these methods in no-op's based in the results of such detection.
  457. Notes on specific fields
  458. ------------------------
  459. .. _mysql-character-fields:
  460. Character fields
  461. ~~~~~~~~~~~~~~~~
  462. Any fields that are stored with ``VARCHAR`` column types may have their
  463. ``max_length`` restricted to 255 characters if you are using ``unique=True``
  464. for the field. This affects :class:`~django.db.models.CharField`,
  465. :class:`~django.db.models.SlugField`. See `the MySQL documentation`_ for more
  466. details.
  467. .. _the MySQL documentation: https://dev.mysql.com/doc/refman/en/create-index.html#create-index-column-prefixes
  468. ``TextField`` limitations
  469. ~~~~~~~~~~~~~~~~~~~~~~~~~
  470. MySQL can index only the first N chars of a ``BLOB`` or ``TEXT`` column. Since
  471. ``TextField`` doesn't have a defined length, you can't mark it as
  472. ``unique=True``. MySQL will report: "BLOB/TEXT column '<db_column>' used in key
  473. specification without a key length".
  474. .. _mysql-fractional-seconds:
  475. Fractional seconds support for Time and DateTime fields
  476. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  477. MySQL can store fractional seconds, provided that the column definition
  478. includes a fractional indication (e.g. ``DATETIME(6)``).
  479. Django will not upgrade existing columns to include fractional seconds if the
  480. database server supports it. If you want to enable them on an existing database,
  481. it's up to you to either manually update the column on the target database, by
  482. executing a command like::
  483. ALTER TABLE `your_table` MODIFY `your_datetime_column` DATETIME(6)
  484. or using a :class:`~django.db.migrations.operations.RunSQL` operation in a
  485. :ref:`data migration <data-migrations>`.
  486. ``TIMESTAMP`` columns
  487. ~~~~~~~~~~~~~~~~~~~~~
  488. If you are using a legacy database that contains ``TIMESTAMP`` columns, you must
  489. set :setting:`USE_TZ = False <USE_TZ>` to avoid data corruption.
  490. :djadmin:`inspectdb` maps these columns to
  491. :class:`~django.db.models.DateTimeField` and if you enable timezone support,
  492. both MySQL and Django will attempt to convert the values from UTC to local time.
  493. Row locking with ``QuerySet.select_for_update()``
  494. -------------------------------------------------
  495. MySQL and MariaDB do not support some options to the ``SELECT ... FOR UPDATE``
  496. statement. If ``select_for_update()`` is used with an unsupported option, then
  497. a :exc:`~django.db.NotSupportedError` is raised.
  498. =============== ========= ==========
  499. Option MariaDB MySQL
  500. =============== ========= ==========
  501. ``SKIP LOCKED`` X (≥8.0.1)
  502. ``NOWAIT`` X (≥10.3) X (≥8.0.1)
  503. ``OF`` X (≥8.0.1)
  504. ``NO KEY``
  505. =============== ========= ==========
  506. When using ``select_for_update()`` on MySQL, make sure you filter a queryset
  507. against at least set of fields contained in unique constraints or only against
  508. fields covered by indexes. Otherwise, an exclusive write lock will be acquired
  509. over the full table for the duration of the transaction.
  510. Automatic typecasting can cause unexpected results
  511. --------------------------------------------------
  512. When performing a query on a string type, but with an integer value, MySQL will
  513. coerce the types of all values in the table to an integer before performing the
  514. comparison. If your table contains the values ``'abc'``, ``'def'`` and you
  515. query for ``WHERE mycolumn=0``, both rows will match. Similarly, ``WHERE mycolumn=1``
  516. will match the value ``'abc1'``. Therefore, string type fields included in Django
  517. will always cast the value to a string before using it in a query.
  518. If you implement custom model fields that inherit from
  519. :class:`~django.db.models.Field` directly, are overriding
  520. :meth:`~django.db.models.Field.get_prep_value`, or use
  521. :class:`~django.db.models.expressions.RawSQL`,
  522. :meth:`~django.db.models.query.QuerySet.extra`, or
  523. :meth:`~django.db.models.Manager.raw`, you should ensure that you perform
  524. appropriate typecasting.
  525. .. _sqlite-notes:
  526. SQLite notes
  527. ============
  528. Django supports SQLite 3.9.0 and later.
  529. SQLite_ provides an excellent development alternative for applications that
  530. are predominantly read-only or require a smaller installation footprint. As
  531. with all database servers, though, there are some differences that are
  532. specific to SQLite that you should be aware of.
  533. .. _SQLite: https://www.sqlite.org/
  534. .. _sqlite-string-matching:
  535. Substring matching and case sensitivity
  536. ---------------------------------------
  537. For all SQLite versions, there is some slightly counter-intuitive behavior when
  538. attempting to match some types of strings. These are triggered when using the
  539. :lookup:`iexact` or :lookup:`contains` filters in Querysets. The behavior
  540. splits into two cases:
  541. 1. For substring matching, all matches are done case-insensitively. That is a
  542. filter such as ``filter(name__contains="aa")`` will match a name of ``"Aabb"``.
  543. 2. For strings containing characters outside the ASCII range, all exact string
  544. matches are performed case-sensitively, even when the case-insensitive options
  545. are passed into the query. So the :lookup:`iexact` filter will behave exactly
  546. the same as the :lookup:`exact` filter in these cases.
  547. Some possible workarounds for this are `documented at sqlite.org`_, but they
  548. aren't utilized by the default SQLite backend in Django, as incorporating them
  549. would be fairly difficult to do robustly. Thus, Django exposes the default
  550. SQLite behavior and you should be aware of this when doing case-insensitive or
  551. substring filtering.
  552. .. _documented at sqlite.org: https://www.sqlite.org/faq.html#q18
  553. .. _sqlite-decimal-handling:
  554. Decimal handling
  555. ----------------
  556. SQLite has no real decimal internal type. Decimal values are internally
  557. converted to the ``REAL`` data type (8-byte IEEE floating point number), as
  558. explained in the `SQLite datatypes documentation`__, so they don't support
  559. correctly-rounded decimal floating point arithmetic.
  560. __ https://www.sqlite.org/datatype3.html#storage_classes_and_datatypes
  561. "Database is locked" errors
  562. ---------------------------
  563. SQLite is meant to be a lightweight database, and thus can't support a high
  564. level of concurrency. ``OperationalError: database is locked`` errors indicate
  565. that your application is experiencing more concurrency than ``sqlite`` can
  566. handle in default configuration. This error means that one thread or process has
  567. an exclusive lock on the database connection and another thread timed out
  568. waiting for the lock the be released.
  569. Python's SQLite wrapper has
  570. a default timeout value that determines how long the second thread is allowed to
  571. wait on the lock before it times out and raises the ``OperationalError: database
  572. is locked`` error.
  573. If you're getting this error, you can solve it by:
  574. * Switching to another database backend. At a certain point SQLite becomes
  575. too "lite" for real-world applications, and these sorts of concurrency
  576. errors indicate you've reached that point.
  577. * Rewriting your code to reduce concurrency and ensure that database
  578. transactions are short-lived.
  579. * Increase the default timeout value by setting the ``timeout`` database
  580. option::
  581. 'OPTIONS': {
  582. # ...
  583. 'timeout': 20,
  584. # ...
  585. }
  586. This will make SQLite wait a bit longer before throwing "database is locked"
  587. errors; it won't really do anything to solve them.
  588. ``QuerySet.select_for_update()`` not supported
  589. ----------------------------------------------
  590. SQLite does not support the ``SELECT ... FOR UPDATE`` syntax. Calling it will
  591. have no effect.
  592. "pyformat" parameter style in raw queries not supported
  593. -------------------------------------------------------
  594. For most backends, raw queries (``Manager.raw()`` or ``cursor.execute()``)
  595. can use the "pyformat" parameter style, where placeholders in the query
  596. are given as ``'%(name)s'`` and the parameters are passed as a dictionary
  597. rather than a list. SQLite does not support this.
  598. .. _sqlite-isolation:
  599. Isolation when using ``QuerySet.iterator()``
  600. --------------------------------------------
  601. There are special considerations described in `Isolation In SQLite`_ when
  602. modifying a table while iterating over it using :meth:`.QuerySet.iterator`. If
  603. a row is added, changed, or deleted within the loop, then that row may or may
  604. not appear, or may appear twice, in subsequent results fetched from the
  605. iterator. Your code must handle this.
  606. .. _`Isolation in SQLite`: https://www.sqlite.org/isolation.html
  607. .. _sqlite-json1:
  608. Enabling JSON1 extension on SQLite
  609. ----------------------------------
  610. To use :class:`~django.db.models.JSONField` on SQLite, you need to enable the
  611. `JSON1 extension`_ on Python's :py:mod:`sqlite3` library. If the extension is
  612. not enabled on your installation, a system error (``fields.E180``) will be
  613. raised.
  614. To enable the JSON1 extension you can follow the instruction on
  615. `the wiki page`_.
  616. .. _JSON1 extension: https://www.sqlite.org/json1.html
  617. .. _the wiki page: https://code.djangoproject.com/wiki/JSON1Extension
  618. .. _oracle-notes:
  619. Oracle notes
  620. ============
  621. Django supports `Oracle Database Server`_ versions 19c and higher. Version 7.0
  622. or higher of the `cx_Oracle`_ Python driver is required.
  623. .. _`Oracle Database Server`: https://www.oracle.com/
  624. .. _`cx_Oracle`: https://oracle.github.io/python-cx_Oracle/
  625. In order for the ``python manage.py migrate`` command to work, your Oracle
  626. database user must have privileges to run the following commands:
  627. * CREATE TABLE
  628. * CREATE SEQUENCE
  629. * CREATE PROCEDURE
  630. * CREATE TRIGGER
  631. To run a project's test suite, the user usually needs these *additional*
  632. privileges:
  633. * CREATE USER
  634. * ALTER USER
  635. * DROP USER
  636. * CREATE TABLESPACE
  637. * DROP TABLESPACE
  638. * CREATE SESSION WITH ADMIN OPTION
  639. * CREATE TABLE WITH ADMIN OPTION
  640. * CREATE SEQUENCE WITH ADMIN OPTION
  641. * CREATE PROCEDURE WITH ADMIN OPTION
  642. * CREATE TRIGGER WITH ADMIN OPTION
  643. While the ``RESOURCE`` role has the required ``CREATE TABLE``,
  644. ``CREATE SEQUENCE``, ``CREATE PROCEDURE``, and ``CREATE TRIGGER`` privileges,
  645. and a user granted ``RESOURCE WITH ADMIN OPTION`` can grant ``RESOURCE``, such
  646. a user cannot grant the individual privileges (e.g. ``CREATE TABLE``), and thus
  647. ``RESOURCE WITH ADMIN OPTION`` is not usually sufficient for running tests.
  648. Some test suites also create views or materialized views; to run these, the
  649. user also needs ``CREATE VIEW WITH ADMIN OPTION`` and
  650. ``CREATE MATERIALIZED VIEW WITH ADMIN OPTION`` privileges. In particular, this
  651. is needed for Django's own test suite.
  652. All of these privileges are included in the DBA role, which is appropriate
  653. for use on a private developer's database.
  654. The Oracle database backend uses the ``SYS.DBMS_LOB`` and ``SYS.DBMS_RANDOM``
  655. packages, so your user will require execute permissions on it. It's normally
  656. accessible to all users by default, but in case it is not, you'll need to grant
  657. permissions like so:
  658. .. code-block:: sql
  659. GRANT EXECUTE ON SYS.DBMS_LOB TO user;
  660. GRANT EXECUTE ON SYS.DBMS_RANDOM TO user;
  661. Connecting to the database
  662. --------------------------
  663. To connect using the service name of your Oracle database, your ``settings.py``
  664. file should look something like this::
  665. DATABASES = {
  666. 'default': {
  667. 'ENGINE': 'django.db.backends.oracle',
  668. 'NAME': 'xe',
  669. 'USER': 'a_user',
  670. 'PASSWORD': 'a_password',
  671. 'HOST': '',
  672. 'PORT': '',
  673. }
  674. }
  675. In this case, you should leave both :setting:`HOST` and :setting:`PORT` empty.
  676. However, if you don't use a ``tnsnames.ora`` file or a similar naming method
  677. and want to connect using the SID ("xe" in this example), then fill in both
  678. :setting:`HOST` and :setting:`PORT` like so::
  679. DATABASES = {
  680. 'default': {
  681. 'ENGINE': 'django.db.backends.oracle',
  682. 'NAME': 'xe',
  683. 'USER': 'a_user',
  684. 'PASSWORD': 'a_password',
  685. 'HOST': 'dbprod01ned.mycompany.com',
  686. 'PORT': '1540',
  687. }
  688. }
  689. You should either supply both :setting:`HOST` and :setting:`PORT`, or leave
  690. both as empty strings. Django will use a different connect descriptor depending
  691. on that choice.
  692. Full DSN and Easy Connect
  693. ~~~~~~~~~~~~~~~~~~~~~~~~~
  694. A Full DSN or Easy Connect string can be used in :setting:`NAME` if both
  695. :setting:`HOST` and :setting:`PORT` are empty. This format is required when
  696. using RAC or pluggable databases without ``tnsnames.ora``, for example.
  697. Example of an Easy Connect string::
  698. 'NAME': 'localhost:1521/orclpdb1',
  699. Example of a full DSN string::
  700. 'NAME': (
  701. '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))'
  702. '(CONNECT_DATA=(SERVICE_NAME=orclpdb1)))'
  703. ),
  704. Threaded option
  705. ---------------
  706. If you plan to run Django in a multithreaded environment (e.g. Apache using the
  707. default MPM module on any modern operating system), then you **must** set
  708. the ``threaded`` option of your Oracle database configuration to ``True``::
  709. 'OPTIONS': {
  710. 'threaded': True,
  711. },
  712. Failure to do this may result in crashes and other odd behavior.
  713. INSERT ... RETURNING INTO
  714. -------------------------
  715. By default, the Oracle backend uses a ``RETURNING INTO`` clause to efficiently
  716. retrieve the value of an ``AutoField`` when inserting new rows. This behavior
  717. may result in a ``DatabaseError`` in certain unusual setups, such as when
  718. inserting into a remote table, or into a view with an ``INSTEAD OF`` trigger.
  719. The ``RETURNING INTO`` clause can be disabled by setting the
  720. ``use_returning_into`` option of the database configuration to ``False``::
  721. 'OPTIONS': {
  722. 'use_returning_into': False,
  723. },
  724. In this case, the Oracle backend will use a separate ``SELECT`` query to
  725. retrieve ``AutoField`` values.
  726. Naming issues
  727. -------------
  728. Oracle imposes a name length limit of 30 characters. To accommodate this, the
  729. backend truncates database identifiers to fit, replacing the final four
  730. characters of the truncated name with a repeatable MD5 hash value.
  731. Additionally, the backend turns database identifiers to all-uppercase.
  732. To prevent these transformations (this is usually required only when dealing
  733. with legacy databases or accessing tables which belong to other users), use
  734. a quoted name as the value for ``db_table``::
  735. class LegacyModel(models.Model):
  736. class Meta:
  737. db_table = '"name_left_in_lowercase"'
  738. class ForeignModel(models.Model):
  739. class Meta:
  740. db_table = '"OTHER_USER"."NAME_ONLY_SEEMS_OVER_30"'
  741. Quoted names can also be used with Django's other supported database
  742. backends; except for Oracle, however, the quotes have no effect.
  743. When running ``migrate``, an ``ORA-06552`` error may be encountered if
  744. certain Oracle keywords are used as the name of a model field or the
  745. value of a ``db_column`` option. Django quotes all identifiers used
  746. in queries to prevent most such problems, but this error can still
  747. occur when an Oracle datatype is used as a column name. In
  748. particular, take care to avoid using the names ``date``,
  749. ``timestamp``, ``number`` or ``float`` as a field name.
  750. .. _oracle-null-empty-strings:
  751. NULL and empty strings
  752. ----------------------
  753. Django generally prefers to use the empty string (``''``) rather than
  754. ``NULL``, but Oracle treats both identically. To get around this, the
  755. Oracle backend ignores an explicit ``null`` option on fields that
  756. have the empty string as a possible value and generates DDL as if
  757. ``null=True``. When fetching from the database, it is assumed that
  758. a ``NULL`` value in one of these fields really means the empty
  759. string, and the data is silently converted to reflect this assumption.
  760. ``TextField`` limitations
  761. -------------------------
  762. The Oracle backend stores ``TextFields`` as ``NCLOB`` columns. Oracle imposes
  763. some limitations on the usage of such LOB columns in general:
  764. * LOB columns may not be used as primary keys.
  765. * LOB columns may not be used in indexes.
  766. * LOB columns may not be used in a ``SELECT DISTINCT`` list. This means that
  767. attempting to use the ``QuerySet.distinct`` method on a model that
  768. includes ``TextField`` columns will result in an ``ORA-00932`` error when
  769. run against Oracle. As a workaround, use the ``QuerySet.defer`` method in
  770. conjunction with ``distinct()`` to prevent ``TextField`` columns from being
  771. included in the ``SELECT DISTINCT`` list.
  772. .. _subclassing-database-backends:
  773. Subclassing the built-in database backends
  774. ==========================================
  775. Django comes with built-in database backends. You may subclass an existing
  776. database backends to modify its behavior, features, or configuration.
  777. Consider, for example, that you need to change a single database feature.
  778. First, you have to create a new directory with a ``base`` module in it. For
  779. example::
  780. mysite/
  781. ...
  782. mydbengine/
  783. __init__.py
  784. base.py
  785. The ``base.py`` module must contain a class named ``DatabaseWrapper`` that
  786. subclasses an existing engine from the ``django.db.backends`` module. Here's an
  787. example of subclassing the PostgreSQL engine to change a feature class
  788. ``allows_group_by_selected_pks_on_model``:
  789. .. code-block:: python
  790. :caption: mysite/mydbengine/base.py
  791. from django.db.backends.postgresql import base, features
  792. class DatabaseFeatures(features.DatabaseFeatures):
  793. def allows_group_by_selected_pks_on_model(self, model):
  794. return True
  795. class DatabaseWrapper(base.DatabaseWrapper):
  796. features_class = DatabaseFeatures
  797. Finally, you must specify a :setting:`DATABASE-ENGINE` in your ``settings.py``
  798. file::
  799. DATABASES = {
  800. 'default': {
  801. 'ENGINE': 'mydbengine',
  802. ...
  803. },
  804. }
  805. You can see the current list of database engines by looking in
  806. :source:`django/db/backends`.
  807. .. _third-party-notes:
  808. Using a 3rd-party database backend
  809. ==================================
  810. In addition to the officially supported databases, there are backends provided
  811. by 3rd parties that allow you to use other databases with Django:
  812. * `CockroachDB`_
  813. * `Firebird`_
  814. * `Microsoft SQL Server`_
  815. The Django versions and ORM features supported by these unofficial backends
  816. vary considerably. Queries regarding the specific capabilities of these
  817. unofficial backends, along with any support queries, should be directed to
  818. the support channels provided by each 3rd party project.
  819. .. _CockroachDB: https://pypi.org/project/django-cockroachdb/
  820. .. _Firebird: https://pypi.org/project/django-firebird/
  821. .. _Microsoft SQL Server: https://pypi.org/project/django-mssql-backend/