custom-lookups.txt 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353
  1. ===========================
  2. How to write custom lookups
  3. ===========================
  4. .. currentmodule:: django.db.models
  5. Django offers a wide variety of :ref:`built-in lookups <field-lookups>` for
  6. filtering (for example, ``exact`` and ``icontains``). This documentation
  7. explains how to write custom lookups and how to alter the working of existing
  8. lookups. For the API references of lookups, see the :doc:`/ref/models/lookups`.
  9. A lookup example
  10. ================
  11. Let's start with a small custom lookup. We will write a custom lookup ``ne``
  12. which works opposite to ``exact``. ``Author.objects.filter(name__ne='Jack')``
  13. will translate to the SQL:
  14. .. code-block:: sql
  15. "author"."name" <> 'Jack'
  16. This SQL is backend independent, so we don't need to worry about different
  17. databases.
  18. There are two steps to making this work. Firstly we need to implement the
  19. lookup, then we need to tell Django about it::
  20. from django.db.models import Lookup
  21. class NotEqual(Lookup):
  22. lookup_name = "ne"
  23. def as_sql(self, compiler, connection):
  24. lhs, lhs_params = self.process_lhs(compiler, connection)
  25. rhs, rhs_params = self.process_rhs(compiler, connection)
  26. params = lhs_params + rhs_params
  27. return "%s <> %s" % (lhs, rhs), params
  28. To register the ``NotEqual`` lookup we will need to call ``register_lookup`` on
  29. the field class we want the lookup to be available for. In this case, the lookup
  30. makes sense on all ``Field`` subclasses, so we register it with ``Field``
  31. directly::
  32. from django.db.models import Field
  33. Field.register_lookup(NotEqual)
  34. Lookup registration can also be done using a decorator pattern::
  35. from django.db.models import Field
  36. @Field.register_lookup
  37. class NotEqualLookup(Lookup): ...
  38. We can now use ``foo__ne`` for any field ``foo``. You will need to ensure that
  39. this registration happens before you try to create any querysets using it. You
  40. could place the implementation in a ``models.py`` file, or register the lookup
  41. in the ``ready()`` method of an ``AppConfig``.
  42. Taking a closer look at the implementation, the first required attribute is
  43. ``lookup_name``. This allows the ORM to understand how to interpret ``name__ne``
  44. and use ``NotEqual`` to generate the SQL. By convention, these names are always
  45. lowercase strings containing only letters, but the only hard requirement is
  46. that it must not contain the string ``__``.
  47. We then need to define the ``as_sql`` method. This takes a ``SQLCompiler``
  48. object, called ``compiler``, and the active database connection.
  49. ``SQLCompiler`` objects are not documented, but the only thing we need to know
  50. about them is that they have a ``compile()`` method which returns a tuple
  51. containing an SQL string, and the parameters to be interpolated into that
  52. string. In most cases, you don't need to use it directly and can pass it on to
  53. ``process_lhs()`` and ``process_rhs()``.
  54. A ``Lookup`` works against two values, ``lhs`` and ``rhs``, standing for
  55. left-hand side and right-hand side. The left-hand side is usually a field
  56. reference, but it can be anything implementing the :ref:`query expression API
  57. <query-expression>`. The right-hand is the value given by the user. In the
  58. example ``Author.objects.filter(name__ne='Jack')``, the left-hand side is a
  59. reference to the ``name`` field of the ``Author`` model, and ``'Jack'`` is the
  60. right-hand side.
  61. We call ``process_lhs`` and ``process_rhs`` to convert them into the values we
  62. need for SQL using the ``compiler`` object described before. These methods
  63. return tuples containing some SQL and the parameters to be interpolated into
  64. that SQL, just as we need to return from our ``as_sql`` method. In the above
  65. example, ``process_lhs`` returns ``('"author"."name"', [])`` and
  66. ``process_rhs`` returns ``('"%s"', ['Jack'])``. In this example there were no
  67. parameters for the left hand side, but this would depend on the object we have,
  68. so we still need to include them in the parameters we return.
  69. Finally we combine the parts into an SQL expression with ``<>``, and supply all
  70. the parameters for the query. We then return a tuple containing the generated
  71. SQL string and the parameters.
  72. A transformer example
  73. =====================
  74. The custom lookup above is great, but in some cases you may want to be able to
  75. chain lookups together. For example, let's suppose we are building an
  76. application where we want to make use of the ``abs()`` operator.
  77. We have an ``Experiment`` model which records a start value, end value, and the
  78. change (start - end). We would like to find all experiments where the change
  79. was equal to a certain amount (``Experiment.objects.filter(change__abs=27)``),
  80. or where it did not exceed a certain amount
  81. (``Experiment.objects.filter(change__abs__lt=27)``).
  82. .. note::
  83. This example is somewhat contrived, but it nicely demonstrates the range of
  84. functionality which is possible in a database backend independent manner,
  85. and without duplicating functionality already in Django.
  86. We will start by writing an ``AbsoluteValue`` transformer. This will use the SQL
  87. function ``ABS()`` to transform the value before comparison::
  88. from django.db.models import Transform
  89. class AbsoluteValue(Transform):
  90. lookup_name = "abs"
  91. function = "ABS"
  92. Next, let's register it for ``IntegerField``::
  93. from django.db.models import IntegerField
  94. IntegerField.register_lookup(AbsoluteValue)
  95. We can now run the queries we had before.
  96. ``Experiment.objects.filter(change__abs=27)`` will generate the following SQL:
  97. .. code-block:: sql
  98. SELECT ... WHERE ABS("experiments"."change") = 27
  99. By using ``Transform`` instead of ``Lookup`` it means we are able to chain
  100. further lookups afterward. So
  101. ``Experiment.objects.filter(change__abs__lt=27)`` will generate the following
  102. SQL:
  103. .. code-block:: sql
  104. SELECT ... WHERE ABS("experiments"."change") < 27
  105. Note that in case there is no other lookup specified, Django interprets
  106. ``change__abs=27`` as ``change__abs__exact=27``.
  107. This also allows the result to be used in ``ORDER BY`` and ``DISTINCT ON``
  108. clauses. For example ``Experiment.objects.order_by('change__abs')`` generates:
  109. .. code-block:: sql
  110. SELECT ... ORDER BY ABS("experiments"."change") ASC
  111. And on databases that support distinct on fields (such as PostgreSQL),
  112. ``Experiment.objects.distinct('change__abs')`` generates:
  113. .. code-block:: sql
  114. SELECT ... DISTINCT ON ABS("experiments"."change")
  115. When looking for which lookups are allowable after the ``Transform`` has been
  116. applied, Django uses the ``output_field`` attribute. We didn't need to specify
  117. this here as it didn't change, but supposing we were applying ``AbsoluteValue``
  118. to some field which represents a more complex type (for example a point
  119. relative to an origin, or a complex number) then we may have wanted to specify
  120. that the transform returns a ``FloatField`` type for further lookups. This can
  121. be done by adding an ``output_field`` attribute to the transform::
  122. from django.db.models import FloatField, Transform
  123. class AbsoluteValue(Transform):
  124. lookup_name = "abs"
  125. function = "ABS"
  126. @property
  127. def output_field(self):
  128. return FloatField()
  129. This ensures that further lookups like ``abs__lte`` behave as they would for
  130. a ``FloatField``.
  131. Writing an efficient ``abs__lt`` lookup
  132. =======================================
  133. When using the above written ``abs`` lookup, the SQL produced will not use
  134. indexes efficiently in some cases. In particular, when we use
  135. ``change__abs__lt=27``, this is equivalent to ``change__gt=-27`` AND
  136. ``change__lt=27``. (For the ``lte`` case we could use the SQL ``BETWEEN``).
  137. So we would like ``Experiment.objects.filter(change__abs__lt=27)`` to generate
  138. the following SQL:
  139. .. code-block:: sql
  140. SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27
  141. The implementation is::
  142. from django.db.models import Lookup
  143. class AbsoluteValueLessThan(Lookup):
  144. lookup_name = "lt"
  145. def as_sql(self, compiler, connection):
  146. lhs, lhs_params = compiler.compile(self.lhs.lhs)
  147. rhs, rhs_params = self.process_rhs(compiler, connection)
  148. params = lhs_params + rhs_params + lhs_params + rhs_params
  149. return "%s < %s AND %s > -%s" % (lhs, rhs, lhs, rhs), params
  150. AbsoluteValue.register_lookup(AbsoluteValueLessThan)
  151. There are a couple of notable things going on. First, ``AbsoluteValueLessThan``
  152. isn't calling ``process_lhs()``. Instead it skips the transformation of the
  153. ``lhs`` done by ``AbsoluteValue`` and uses the original ``lhs``. That is, we
  154. want to get ``"experiments"."change"`` not ``ABS("experiments"."change")``.
  155. Referring directly to ``self.lhs.lhs`` is safe as ``AbsoluteValueLessThan``
  156. can be accessed only from the ``AbsoluteValue`` lookup, that is the ``lhs``
  157. is always an instance of ``AbsoluteValue``.
  158. Notice also that as both sides are used multiple times in the query the params
  159. need to contain ``lhs_params`` and ``rhs_params`` multiple times.
  160. The final query does the inversion (``27`` to ``-27``) directly in the
  161. database. The reason for doing this is that if the ``self.rhs`` is something else
  162. than a plain integer value (for example an ``F()`` reference) we can't do the
  163. transformations in Python.
  164. .. note::
  165. In fact, most lookups with ``__abs`` could be implemented as range queries
  166. like this, and on most database backends it is likely to be more sensible to
  167. do so as you can make use of the indexes. However with PostgreSQL you may
  168. want to add an index on ``abs(change)`` which would allow these queries to
  169. be very efficient.
  170. A bilateral transformer example
  171. ===============================
  172. The ``AbsoluteValue`` example we discussed previously is a transformation which
  173. applies to the left-hand side of the lookup. There may be some cases where you
  174. want the transformation to be applied to both the left-hand side and the
  175. right-hand side. For instance, if you want to filter a queryset based on the
  176. equality of the left and right-hand side insensitively to some SQL function.
  177. Let's examine case-insensitive transformations here. This transformation isn't
  178. very useful in practice as Django already comes with a bunch of built-in
  179. case-insensitive lookups, but it will be a nice demonstration of bilateral
  180. transformations in a database-agnostic way.
  181. We define an ``UpperCase`` transformer which uses the SQL function ``UPPER()`` to
  182. transform the values before comparison. We define
  183. :attr:`bilateral = True <django.db.models.Transform.bilateral>` to indicate that
  184. this transformation should apply to both ``lhs`` and ``rhs``::
  185. from django.db.models import Transform
  186. class UpperCase(Transform):
  187. lookup_name = "upper"
  188. function = "UPPER"
  189. bilateral = True
  190. Next, let's register it::
  191. from django.db.models import CharField, TextField
  192. CharField.register_lookup(UpperCase)
  193. TextField.register_lookup(UpperCase)
  194. Now, the queryset ``Author.objects.filter(name__upper="doe")`` will generate a case
  195. insensitive query like this:
  196. .. code-block:: sql
  197. SELECT ... WHERE UPPER("author"."name") = UPPER('doe')
  198. Writing alternative implementations for existing lookups
  199. ========================================================
  200. Sometimes different database vendors require different SQL for the same
  201. operation. For this example we will rewrite a custom implementation for
  202. MySQL for the NotEqual operator. Instead of ``<>`` we will be using ``!=``
  203. operator. (Note that in reality almost all databases support both, including
  204. all the official databases supported by Django).
  205. We can change the behavior on a specific backend by creating a subclass of
  206. ``NotEqual`` with an ``as_mysql`` method::
  207. class MySQLNotEqual(NotEqual):
  208. def as_mysql(self, compiler, connection, **extra_context):
  209. lhs, lhs_params = self.process_lhs(compiler, connection)
  210. rhs, rhs_params = self.process_rhs(compiler, connection)
  211. params = lhs_params + rhs_params
  212. return "%s != %s" % (lhs, rhs), params
  213. Field.register_lookup(MySQLNotEqual)
  214. We can then register it with ``Field``. It takes the place of the original
  215. ``NotEqual`` class as it has the same ``lookup_name``.
  216. When compiling a query, Django first looks for ``as_%s % connection.vendor``
  217. methods, and then falls back to ``as_sql``. The vendor names for the in-built
  218. backends are ``sqlite``, ``postgresql``, ``oracle`` and ``mysql``.
  219. How Django determines the lookups and transforms which are used
  220. ===============================================================
  221. In some cases you may wish to dynamically change which ``Transform`` or
  222. ``Lookup`` is returned based on the name passed in, rather than fixing it. As
  223. an example, you could have a field which stores coordinates or an arbitrary
  224. dimension, and wish to allow a syntax like ``.filter(coords__x7=4)`` to return
  225. the objects where the 7th coordinate has value 4. In order to do this, you
  226. would override ``get_lookup`` with something like::
  227. class CoordinatesField(Field):
  228. def get_lookup(self, lookup_name):
  229. if lookup_name.startswith("x"):
  230. try:
  231. dimension = int(lookup_name.removeprefix("x"))
  232. except ValueError:
  233. pass
  234. else:
  235. return get_coordinate_lookup(dimension)
  236. return super().get_lookup(lookup_name)
  237. You would then define ``get_coordinate_lookup`` appropriately to return a
  238. ``Lookup`` subclass which handles the relevant value of ``dimension``.
  239. There is a similarly named method called ``get_transform()``. ``get_lookup()``
  240. should always return a ``Lookup`` subclass, and ``get_transform()`` a
  241. ``Transform`` subclass. It is important to remember that ``Transform``
  242. objects can be further filtered on, and ``Lookup`` objects cannot.
  243. When filtering, if there is only one lookup name remaining to be resolved, we
  244. will look for a ``Lookup``. If there are multiple names, it will look for a
  245. ``Transform``. In the situation where there is only one name and a ``Lookup``
  246. is not found, we look for a ``Transform`` and then the ``exact`` lookup on that
  247. ``Transform``. All call sequences always end with a ``Lookup``. To clarify:
  248. - ``.filter(myfield__mylookup)`` will call ``myfield.get_lookup('mylookup')``.
  249. - ``.filter(myfield__mytransform__mylookup)`` will call
  250. ``myfield.get_transform('mytransform')``, and then
  251. ``mytransform.get_lookup('mylookup')``.
  252. - ``.filter(myfield__mytransform)`` will first call
  253. ``myfield.get_lookup('mytransform')``, which will fail, so it will fall back
  254. to calling ``myfield.get_transform('mytransform')`` and then
  255. ``mytransform.get_lookup('exact')``.