Split most box_protocol content and add TODOs

Author: patiencedaur

doc/dev_guide/internals/box_protocol.rst

@@ -1,14 +1,12 @@
 .. _internals-data_persistence:
 
---------------------------------------------------------------------------------
 File formats
---------------------------------------------------------------------------------
+============
 
 .. _internals-wal:
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Data persistence and the WAL file format
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------------------
 
 To maintain data persistence, Tarantool writes each data change request (insert,
 update, delete, replace, upsert) into a write-ahead log (WAL) file in the
@@ -114,9 +112,8 @@ a secondary key, the record in the .xlog file will contain the primary key.
 
 .. _internals-snapshot:
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The snapshot file format
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------
 
 The format of a snapshot .snap file is nearly the same as the format of a WAL .xlog file.
 However, the snapshot header differs: it contains the instance's global unique identifier
@@ -131,3 +128,43 @@ and ``_cluster`` -- will be at the start of the .snap file, before the records o
 any spaces that were created by users.
 
 Secondarily, the .snap file's records are ordered by primary key within space id.
+
+..  _box_protocol-xlog:
+
+Example
+-------
+
+The header of a ``.snap`` or ``.xlog`` file looks like:
+
+..  code-block:: none
+
+    <type>\n                  SNAP\n or XLOG\n
+    <version>\n               currently 0.13\n
+    Server: <server_uuid>\n   where UUID is a 36-byte string
+    VClock: <vclock_map>\n    e.g. {1: 0}\n
+    \n
+
+After the file header come the data tuples.
+Tuples begin with a row marker ``0xd5ba0bab`` and
+the last tuple may be followed by an EOF marker
+``0xd510aded``.
+Thus, between the file header and the EOF marker, there
+may be data tuples that have this form:
+
+..  code-block:: none
+
+    0            3 4                                         17
+    +-------------+========+============+===========+=========+
+    |             |        |            |           |         |
+    | 0xd5ba0bab  | LENGTH | CRC32 PREV | CRC32 CUR | PADDING |
+    |             |        |            |           |         |
+    +-------------+========+============+===========+=========+
+       MP_FIXEXT2    MP_INT     MP_INT       MP_INT      ---
+
+    +============+ +===================================+
+    |            | |                                   |
+    |   HEADER   | |                BODY               |
+    |            | |                                   |
+    +============+ +===================================+
+         MP_MAP                     MP_MAP
+

doc/dev_guide/internals/file_formats.rst

@@ -0,0 +1,52 @@
+Authentication
+==============
+
+..  _box_protocol-authentication:
+
+Greeting message
+----------------
+
+When a client connects to the server instance, the instance responds with
+a 128-byte text greeting message, not in MsgPack format: |br|
+64-byte Greeting text line 1 |br|
+64-byte Greeting text line 2 |br|
+44-byte base64-encoded salt |br|
+20-byte NULL
+
+The greeting contains two 64-byte lines of ASCII text.
+Each line ends with a newline character (:code:`\n`). The first line contains
+the instance version and protocol type. The second line contains up to 44 bytes
+of base64-encoded random string, to use in the authentication packet, and ends
+with up to 23 spaces.
+
+Part of the greeting is a base64-encoded session salt -
+a random string which can be used for authentication. The maximum length of an encoded
+salt (44 bytes) is more than the amount necessary to create the authentication
+message. An excess is reserved for future authentication
+schemas.
+
+Authentication is optional -- if it is skipped, then the session user is ``'guest'``
+(the ``'guest'`` user does not need a password).
+
+If authentication is not skipped, then at any time an authentication packet
+can be prepared using the greeting, the user's name and password,
+and `sha-1 <https://en.wikipedia.org/wiki/SHA-1>`_ functions, as follows.
+
+..  code-block:: none
+
+    PREPARE SCRAMBLE:
+
+        size_of_encoded_salt_in_greeting = 44;
+        size_of_salt_after_base64_decode = 32;
+         /* sha1() will only use the first 20 bytes */
+        size_of_any_sha1_digest = 20;
+        size_of_scramble = 20;
+
+    prepare 'chap-sha1' scramble:
+
+        salt = base64_decode(encoded_salt);
+        step_1 = sha1(password);
+        step_2 = sha1(step_1);
+        step_3 = sha1(first_20_bytes_of_salt, step_2);
+        scramble = xor(step_1, step_3);
+        return scramble;

doc/dev_guide/internals/iproto/authentication.rst

@@ -106,6 +106,10 @@ For example, in a successful response to ``box.space:select()``,
 the Response-Code-Indicator value will be 0 = ``IPROTO_OK`` and the
 array will have all the tuples of the result.
 
+Read the source code file `net_box.c <https://github.com/tarantool/tarantool/blob/master/src/box/lua/net_box.c>`_
+where the function "decode_metadata_optional" is an example of how Tarantool
+itself decodes extra items.
+
 Body
 ~~~~
 
@@ -114,3 +118,130 @@ be absent or be an empty map. Both these states will be interpreted equally.
 Responses will contain the ``<body>`` anyway even for an
 :ref:`IPROTO_PING <box_protocol-ping>` request.
 
+
+..  cssclass:: highlight
+..  parsed-literal::
+
+    # <size>
+    msgpack(:samp:`{{MP_UINT unsigned integer = size(<header>) + size(<body>)}}`)
+    # <header>
+    msgpack({
+        Response-Code-Indicator: IPROTO_OK,
+        IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer, may be 64-bit}}`,
+        IPROTO_SCHEMA_VERSION: :samp:`{{MP_UINT unsigned integer}}`
+    })
+    # <body>
+    msgpack({
+        IPROTO_DATA: :samp:`{{any type}}`
+    })
+
+- For :ref:`IPROTO_PING <box_protocol-ping>` the body will be an empty map.
+
+
+IPROTO_DATA is what we get with net_box and :ref:`Module buffer <buffer-module>`
+so if we were using net_box we could decode with
+:ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked_string>`,
+or we could convert to a string with :samp:`ffi.string({pointer},{length})`.
+The :ref:`pickle.unpack() <pickle-unpack>` function might also be helpful.
+
+Response body
+~~~~~~~~~~~~~
+
+After the :ref:`header <box_protocol-header>`, for a response,
+there will be a body.
+If there was no error, it will contain IPROTO_OK (0x00).
+If there was an error, it will contain an error code other than IPROTO_OK.
+Responses to SQL statements are slightly different and will be described
+in the later section,
+:ref:`Binary protocol -- responses for SQL <box_protocol-sql_protocol>`.
+
+For IPROTO_OK, the header Response-Code-Indicator will be 0 and the body is a 1-item map.
+
+..  cssclass:: highlight
+..  parsed-literal::
+
+    # <size>
+    msgpack(:samp:`{{MP_UINT unsigned integer = size(<header>) + size(<body>)}}`)
+    # <header>
+    msgpack({
+        Response-Code-Indicator: IPROTO_OK,
+        IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer, may be 64-bit}}`,
+        IPROTO_SCHEMA_VERSION: :samp:`{{MP_UINT unsigned integer}}`
+    })
+    # <body>
+    msgpack({
+        IPROTO_DATA: :samp:`{{any type}}`
+    })
+
+Responses for SQL
+-----------------
+
+After the :ref:`header <box_protocol-header>`, for a response to an SQL statement,
+there will be a body that is slightly different from the body for
+:ref:`Binary protocol -- responses if no error and no SQL <box_protocol-responses>`.
+
+If the SQL request is not SELECT or VALUES or PRAGMA, then the response body
+contains only IPROTO_SQL_INFO (0x42). Usually IPROTO_SQL_INFO is a map with only
+one item -- SQL_INFO_ROW_COUNT (0x00) -- which is the number of changed rows.
+
+..  _box_protocol-responses_error:
+
+Responses for errors
+--------------------
+
+For a response other than IPROTO_OK, the header Response-Code-Indicator will be
+``0x8XXX`` and the body will be a 1-item map.
+
+..  cssclass:: highlight
+..  parsed-literal::
+
+    # <size>
+    msgpack(32)
+    # <header>
+    msgpack({
+        Response-Code-Indicator: :samp:`{{0x8XXX}}`,
+        IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer, may be 64-bit}}`,
+        IPROTO_SCHEMA_VERSION: :samp:`{{MP_UINT unsigned integer}}`
+    })
+    # <body>
+    msgpack({
+        IPROTO_ERROR: :samp:`{{MP_STRING string}}`
+    })
+
+where ``0x8XXX`` is the indicator for an error and ``XXX`` is a value in
+`src/box/errcode.h <https://github.com/tarantool/tarantool/blob/master/src/box/errcode.h>`_.
+``src/box/errcode.h`` also has some convenience macros which define hexadecimal
+constants for return codes.
+
+Example: in version 2.4.0 and earlier,
+if this is the fifth message and the request is to create a duplicate
+space with
+``conn:eval([[box.schema.space.create('_space');]])``
+the unsuccessful response will look like this:
+
+..  code-block:: none
+
+    # <size>
+    msgpack(32)
+    # <header>
+    msgpack({
+        Response-Code-Indicator: 0x800a,
+        IPROTO_SYNC: 5,
+        IPROTO_SCHEMA_VERSION: 0x78
+    })
+    # <body>
+    msgpack({
+        IPROTO_ERROR:  "Space '_space' already exists"
+    })
+
+Later in :ref:`Binary protocol -- illustration <box_protocol-illustration>`
+we will show actual byte codes of the response to the IPROTO_EVAL message.
+
+Looking in errcode.h we find that error code 0x0a (decimal 10) is
+ER_SPACE_EXISTS, and the string associated with ER_SPACE_EXISTS is
+"Space '%s' already exists".
+
+Since version :doc:`2.4.1 </release/2.4.1>`, responses for errors have extra information
+following what was described above. This extra information is given via
+MP_ERROR extension type. See details in :ref:`MessagePack extensions
+<msgpack_ext-error>` section.

doc/dev_guide/internals/iproto/format.rst

@@ -35,6 +35,20 @@ This section describes all the iproto keys used to pass request arguments.
         used with SQL within IPROTO_EXECUTE:
     IPROTO_OPTIONS=0x2b
     IPROTO_METADATA=0x32
+    * :samp:`IPROTO_METADATA: {array of column maps}` = array of column maps, with each column map containing
+  at least IPROTO_FIELD_NAME (0x00) and MP_STR, and IPROTO_FIELD_TYPE (0x01) and MP_STR.
+  Additionally, if ``sql_full_metadata`` in the
+  :ref:`_session_settings <box_space-session_settings>` system space
+  is TRUE, then the array will have these additional column maps
+  which correspond to components described in the
+  :ref:`box.execute() <box-sql_if_full_metadata>` section:
+
+..  code-block:: none
+
+    IPROTO_FIELD_COLL (0x02) and MP_STR
+    IPROTO_FIELD_IS_NULLABLE (0x03) and MP_BOOL
+    IPROTO_FIELD_IS_AUTOINCREMENT (0x04) and MP_BOOL
+    IPROTO_FIELD_SPAN (0x05) and MP_STR or MP_NIL
     IPROTO_BIND_METADATA=0x33
     IPROTO_BIND_COUNT=0x34
     IPROTO_SQL_TEXT=0x40
@@ -44,6 +58,23 @@ This section describes all the iproto keys used to pass request arguments.
 
     "No error" constant
     IPROTO_OK=0x00
+For IPROTO_OK, the header Response-Code-Indicator will be 0 and the body is a 1-item map.
+
+..  cssclass:: highlight
+..  parsed-literal::
+
+    # <size>
+    msgpack(:samp:`{{MP_UINT unsigned integer = size(<header>) + size(<body>)}}`)
+    # <header>
+    msgpack({
+        Response-Code-Indicator: IPROTO_OK,
+        IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer, may be 64-bit}}`,
+        IPROTO_SCHEMA_VERSION: :samp:`{{MP_UINT unsigned integer}}`
+    })
+    # <body>
+    msgpack({
+        IPROTO_DATA: :samp:`{{any type}}`
+    })
     
     box.session.sync(), like a clock, must be same for response and request
     IPROTO_SYNC=0x01
@@ -83,7 +114,7 @@ This section describes all the iproto keys used to pass request arguments.
     IPROTO_TIMESTAMP=0x04 Float 64 MP_DOUBLE 8-byte timestamp
     IPROTO_SCHEMA_VERSION=0x05
     Stream transactions
-    IPROTO_STREAM_ID=0x0a
+    IPROTO_STREAM_ID=0x0a Дать ссылку на подраздел про стримы
     IPROTO_TXN_ISOLATION=0x59
     IPROTO_FUNCTION_NAME=0x22
     user name
@@ -105,6 +136,9 @@ This section describes all the iproto keys used to pass request arguments.
     IPROTO_ERROR_24=0x31
 
     IPROTO_CHUNK=0x80
+    If the response is out-of-band, due to use of
+:ref:`box.session.push() <box_session-push>`,
+then the header Response-Code-Indicator will be IPROTO_CHUNK instead of IPROTO_OK.
 
     IPROTO_TIMEOUT=0x56
 
@@ -119,3 +153,11 @@ This section describes all the iproto keys used to pass request arguments.
 
     IPROTO_EVENT_KEY=0x57
     IPROTO_EVENT_DATA=0x58
+
+    IPROTO_SQL_INFO (0x42)
+    Usually IPROTO_SQL_INFO is a map with only one item -- SQL_INFO_ROW_COUNT (0x00) -- which is the number of changed rows
+    The IPROTO_SQL_INFO map may contain a second item -- :samp:`SQL_INFO_AUTO_INCREMENT_IDS
+(0x01)` -- which is the new primary-key value (or values) for an INSERT in a table
+defined with PRIMARY KEY AUTOINCREMENT.
+        SQL_INFO_ROW_COUNT (0x00)
+        SQL_INFO_AUTO_INCREMENT_IDS (0x01)