2
0

unicode.txt 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358
  1. ============
  2. Unicode data
  3. ============
  4. Django supports Unicode data everywhere.
  5. This document tells you what you need to know if you're writing applications
  6. that use data or templates that are encoded in something other than ASCII.
  7. Creating the database
  8. =====================
  9. Make sure your database is configured to be able to store arbitrary string
  10. data. Normally, this means giving it an encoding of UTF-8 or UTF-16. If you use
  11. a more restrictive encoding -- for example, latin1 (iso8859-1) -- you won't be
  12. able to store certain characters in the database, and information will be lost.
  13. * MySQL users, refer to the `MySQL manual`_ for details on how to set or alter
  14. the database character set encoding.
  15. * PostgreSQL users, refer to the `PostgreSQL manual`_ for details on creating
  16. databases with the correct encoding.
  17. * Oracle users, refer to the `Oracle manual`_ for details on how to set
  18. (`section 2`_) or alter (`section 11`_) the database character set encoding.
  19. * SQLite users, there is nothing you need to do. SQLite always uses UTF-8
  20. for internal encoding.
  21. .. _MySQL manual: https://dev.mysql.com/doc/refman/en/charset-database.html
  22. .. _PostgreSQL manual: https://www.postgresql.org/docs/current/multibyte.html#MULTIBYTE-SETTING
  23. .. _Oracle manual: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/index.html
  24. .. _section 2: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/choosing-character-set.html
  25. .. _section 11: https://docs.oracle.com/en/database/oracle/oracle-database/21/nlspg/character-set-migration.html
  26. All of Django's database backends automatically convert strings into
  27. the appropriate encoding for talking to the database. They also automatically
  28. convert strings retrieved from the database into strings. You don't even need
  29. to tell Django what encoding your database uses: that is handled transparently.
  30. For more, see the section "The database API" below.
  31. General string handling
  32. =======================
  33. Whenever you use strings with Django -- e.g., in database lookups, template
  34. rendering or anywhere else -- you have two choices for encoding those strings.
  35. You can use normal strings or bytestrings (starting with a 'b').
  36. .. warning::
  37. A bytestring does not carry any information with it about its encoding.
  38. For that reason, we have to make an assumption, and Django assumes that all
  39. bytestrings are in UTF-8.
  40. If you pass a string to Django that has been encoded in some other format,
  41. things will go wrong in interesting ways. Usually, Django will raise a
  42. ``UnicodeDecodeError`` at some point.
  43. If your code only uses ASCII data, it's safe to use your normal strings,
  44. passing them around at will, because ASCII is a subset of UTF-8.
  45. Don't be fooled into thinking that if your :setting:`DEFAULT_CHARSET` setting is set
  46. to something other than ``'utf-8'`` you can use that other encoding in your
  47. bytestrings! :setting:`DEFAULT_CHARSET` only applies to the strings generated as
  48. the result of template rendering (and email). Django will always assume UTF-8
  49. encoding for internal bytestrings. The reason for this is that the
  50. :setting:`DEFAULT_CHARSET` setting is not actually under your control (if you are the
  51. application developer). It's under the control of the person installing and
  52. using your application -- and if that person chooses a different setting, your
  53. code must still continue to work. Ergo, it cannot rely on that setting.
  54. In most cases when Django is dealing with strings, it will convert them to
  55. strings before doing anything else. So, as a general rule, if you pass
  56. in a bytestring, be prepared to receive a string back in the result.
  57. Translated strings
  58. ------------------
  59. Aside from strings and bytestrings, there's a third type of string-like
  60. object you may encounter when using Django. The framework's
  61. internationalization features introduce the concept of a "lazy translation" --
  62. a string that has been marked as translated but whose actual translation result
  63. isn't determined until the object is used in a string. This feature is useful
  64. in cases where the translation locale is unknown until the string is used, even
  65. though the string might have originally been created when the code was first
  66. imported.
  67. Normally, you won't have to worry about lazy translations. Just be aware that
  68. if you examine an object and it claims to be a
  69. ``django.utils.functional.__proxy__`` object, it is a lazy translation.
  70. Calling ``str()`` with the lazy translation as the argument will generate a
  71. string in the current locale.
  72. For more details about lazy translation objects, refer to the
  73. :doc:`internationalization </topics/i18n/index>` documentation.
  74. Useful utility functions
  75. ------------------------
  76. Because some string operations come up again and again, Django ships with a few
  77. useful functions that should make working with string and bytestring objects
  78. a bit easier.
  79. Conversion functions
  80. ~~~~~~~~~~~~~~~~~~~~
  81. The ``django.utils.encoding`` module contains a few functions that are handy
  82. for converting back and forth between strings and bytestrings.
  83. * ``smart_str(s, encoding='utf-8', strings_only=False, errors='strict')``
  84. converts its input to a string. The ``encoding`` parameter
  85. specifies the input encoding. (For example, Django uses this internally
  86. when processing form input data, which might not be UTF-8 encoded.) The
  87. ``strings_only`` parameter, if set to True, will result in Python
  88. numbers, booleans and ``None`` not being converted to a string (they keep
  89. their original types). The ``errors`` parameter takes any of the values
  90. that are accepted by Python's ``str()`` function for its error
  91. handling.
  92. * ``force_str(s, encoding='utf-8', strings_only=False, errors='strict')`` is
  93. identical to ``smart_str()`` in almost all cases. The difference is when the
  94. first argument is a :ref:`lazy translation <lazy-translations>` instance.
  95. While ``smart_str()`` preserves lazy translations, ``force_str()`` forces
  96. those objects to a string (causing the translation to occur). Normally,
  97. you'll want to use ``smart_str()``. However, ``force_str()`` is useful in
  98. template tags and filters that absolutely *must* have a string to work with,
  99. not just something that can be converted to a string.
  100. * ``smart_bytes(s, encoding='utf-8', strings_only=False, errors='strict')``
  101. is essentially the opposite of ``smart_str()``. It forces the first
  102. argument to a bytestring. The ``strings_only`` parameter has the same
  103. behavior as for ``smart_str()`` and ``force_str()``. This is
  104. slightly different semantics from Python's builtin ``str()`` function,
  105. but the difference is needed in a few places within Django's internals.
  106. Normally, you'll only need to use ``force_str()``. Call it as early as
  107. possible on any input data that might be either a string or a bytestring, and
  108. from then on, you can treat the result as always being a string.
  109. .. _uri-and-iri-handling:
  110. URI and IRI handling
  111. ~~~~~~~~~~~~~~~~~~~~
  112. Web frameworks have to deal with URLs (which are a type of IRI). One
  113. requirement of URLs is that they are encoded using only ASCII characters.
  114. However, in an international environment, you might need to construct a
  115. URL from an :rfc:`IRI <3987>` -- very loosely speaking, a :rfc:`URI <3986>`
  116. that can contain Unicode characters. Use these functions for quoting and
  117. converting an IRI to a URI:
  118. * The :func:`django.utils.encoding.iri_to_uri()` function, which implements the
  119. conversion from IRI to URI as required by :rfc:`3987#section-3.1`.
  120. * The :func:`urllib.parse.quote` and :func:`urllib.parse.quote_plus`
  121. functions from Python's standard library.
  122. These two groups of functions have slightly different purposes, and it's
  123. important to keep them straight. Normally, you would use ``quote()`` on the
  124. individual portions of the IRI or URI path so that any reserved characters
  125. such as '&' or '%' are correctly encoded. Then, you apply ``iri_to_uri()`` to
  126. the full IRI and it converts any non-ASCII characters to the correct encoded
  127. values.
  128. .. note::
  129. Technically, it isn't correct to say that ``iri_to_uri()`` implements the
  130. full algorithm in the IRI specification. It doesn't (yet) perform the
  131. international domain name encoding portion of the algorithm.
  132. The ``iri_to_uri()`` function will not change ASCII characters that are
  133. otherwise permitted in a URL. So, for example, the character '%' is not
  134. further encoded when passed to ``iri_to_uri()``. This means you can pass a
  135. full URL to this function and it will not mess up the query string or anything
  136. like that.
  137. An example might clarify things here:
  138. .. code-block:: pycon
  139. >>> from urllib.parse import quote
  140. >>> from django.utils.encoding import iri_to_uri
  141. >>> quote("Paris & Orléans")
  142. 'Paris%20%26%20Orl%C3%A9ans'
  143. >>> iri_to_uri("/favorites/François/%s" % quote("Paris & Orléans"))
  144. '/favorites/Fran%C3%A7ois/Paris%20%26%20Orl%C3%A9ans'
  145. If you look carefully, you can see that the portion that was generated by
  146. ``quote()`` in the second example was not double-quoted when passed to
  147. ``iri_to_uri()``. This is a very important and useful feature. It means that
  148. you can construct your IRI without worrying about whether it contains
  149. non-ASCII characters and then, right at the end, call ``iri_to_uri()`` on the
  150. result.
  151. Similarly, Django provides :func:`django.utils.encoding.uri_to_iri()` which
  152. implements the conversion from URI to IRI as per :rfc:`3987#section-3.2`.
  153. An example to demonstrate:
  154. .. code-block:: pycon
  155. >>> from django.utils.encoding import uri_to_iri
  156. >>> uri_to_iri("/%E2%99%A5%E2%99%A5/?utf8=%E2%9C%93")
  157. '/♥♥/?utf8=✓'
  158. >>> uri_to_iri("%A9hello%3Fworld")
  159. '%A9hello%3Fworld'
  160. In the first example, the UTF-8 characters are unquoted. In the second, the
  161. percent-encodings remain unchanged because they lie outside the valid UTF-8
  162. range or represent a reserved character.
  163. Both ``iri_to_uri()`` and ``uri_to_iri()`` functions are idempotent, which means the
  164. following is always true::
  165. iri_to_uri(iri_to_uri(some_string)) == iri_to_uri(some_string)
  166. uri_to_iri(uri_to_iri(some_string)) == uri_to_iri(some_string)
  167. So you can safely call it multiple times on the same URI/IRI without risking
  168. double-quoting problems.
  169. Models
  170. ======
  171. Because all strings are returned from the database as ``str`` objects, model
  172. fields that are character based (CharField, TextField, URLField, etc.) will
  173. contain Unicode values when Django retrieves data from the database. This
  174. is *always* the case, even if the data could fit into an ASCII bytestring.
  175. You can pass in bytestrings when creating a model or populating a field, and
  176. Django will convert it to strings when it needs to.
  177. Taking care in ``get_absolute_url()``
  178. -------------------------------------
  179. URLs can only contain ASCII characters. If you're constructing a URL from
  180. pieces of data that might be non-ASCII, be careful to encode the results in a
  181. way that is suitable for a URL. The :func:`~django.urls.reverse` function
  182. handles this for you automatically.
  183. If you're constructing a URL manually (i.e., *not* using the ``reverse()``
  184. function), you'll need to take care of the encoding yourself. In this case,
  185. use the ``iri_to_uri()`` and ``quote()`` functions that were documented
  186. above_. For example::
  187. from urllib.parse import quote
  188. from django.utils.encoding import iri_to_uri
  189. def get_absolute_url(self):
  190. url = "/person/%s/?x=0&y=0" % quote(self.location)
  191. return iri_to_uri(url)
  192. This function returns a correctly encoded URL even if ``self.location`` is
  193. something like "Jack visited Paris & Orléans". (In fact, the ``iri_to_uri()``
  194. call isn't strictly necessary in the above example, because all the
  195. non-ASCII characters would have been removed in quoting in the first line.)
  196. .. _above: `URI and IRI handling`_
  197. Templates
  198. =========
  199. Use strings when creating templates manually::
  200. from django.template import Template
  201. t2 = Template("This is a string template.")
  202. But the common case is to read templates from the filesystem. If your template
  203. files are not stored with a UTF-8 encoding, adjust the :setting:`TEMPLATES`
  204. setting. The built-in :py:mod:`~django.template.backends.django` backend
  205. provides the ``'file_charset'`` option to change the encoding used to read
  206. files from disk.
  207. The :setting:`DEFAULT_CHARSET` setting controls the encoding of rendered templates.
  208. This is set to UTF-8 by default.
  209. Template tags and filters
  210. -------------------------
  211. A couple of tips to remember when writing your own template tags and filters:
  212. * Always return strings from a template tag's ``render()`` method
  213. and from template filters.
  214. * Use ``force_str()`` in preference to ``smart_str()`` in these
  215. places. Tag rendering and filter calls occur as the template is being
  216. rendered, so there is no advantage to postponing the conversion of lazy
  217. translation objects into strings. It's easier to work solely with
  218. strings at that point.
  219. .. _unicode-files:
  220. Files
  221. =====
  222. If you intend to allow users to upload files, you must ensure that the
  223. environment used to run Django is configured to work with non-ASCII file names.
  224. If your environment isn't configured correctly, you'll encounter
  225. ``UnicodeEncodeError`` exceptions when saving files with file names or content
  226. that contains non-ASCII characters.
  227. Filesystem support for UTF-8 file names varies and might depend on the
  228. environment. Check your current configuration in an interactive Python shell by
  229. running::
  230. import sys
  231. sys.getfilesystemencoding()
  232. This should output "UTF-8".
  233. The ``LANG`` environment variable is responsible for setting the expected
  234. encoding on Unix platforms. Consult the documentation for your operating system
  235. and application server for the appropriate syntax and location to set this
  236. variable. See the :doc:`/howto/deployment/wsgi/modwsgi` for examples.
  237. In your development environment, you might need to add a setting to your
  238. ``~.bashrc`` analogous to:
  239. .. code-block:: shell
  240. export LANG="en_US.UTF-8"
  241. Form submission
  242. ===============
  243. HTML form submission is a tricky area. There's no guarantee that the
  244. submission will include encoding information, which means the framework might
  245. have to guess at the encoding of submitted data.
  246. Django adopts a "lazy" approach to decoding form data. The data in an
  247. ``HttpRequest`` object is only decoded when you access it. In fact, most of
  248. the data is not decoded at all. Only the ``HttpRequest.GET`` and
  249. ``HttpRequest.POST`` data structures have any decoding applied to them. Those
  250. two fields will return their members as Unicode data. All other attributes and
  251. methods of ``HttpRequest`` return data exactly as it was submitted by the
  252. client.
  253. By default, the :setting:`DEFAULT_CHARSET` setting is used as the assumed encoding
  254. for form data. If you need to change this for a particular form, you can set
  255. the ``encoding`` attribute on an ``HttpRequest`` instance. For example::
  256. def some_view(request):
  257. # We know that the data must be encoded as KOI8-R (for some reason).
  258. request.encoding = "koi8-r"
  259. ...
  260. You can even change the encoding after having accessed ``request.GET`` or
  261. ``request.POST``, and all subsequent accesses will use the new encoding.
  262. Most developers won't need to worry about changing form encoding, but this is
  263. a useful feature for applications that talk to legacy systems whose encoding
  264. you cannot control.
  265. Django does not decode the data of file uploads, because that data is normally
  266. treated as collections of bytes, rather than strings. Any automatic decoding
  267. there would alter the meaning of the stream of bytes.