Document join retry

Author: andreyaksenov

doc/concepts/replication/repl_architecture.rst

@@ -8,62 +8,95 @@ Replication architecture
 Replication mechanism
 ---------------------
 
-A pack of instances which operate on copies of the same databases make up a
-**replica set**. Each instance in a replica set has a role, **master** or
-**replica**.
+.. _replication_overview:
+
+Replication overview
+~~~~~~~~~~~~~~~~~~~~
+
+A pack of instances that operate on copies of the same databases makes up a **replica set**.
+Each instance in a replica set has a role: **master** or **replica**.
 
 A replica gets all updates from the master by continuously fetching and applying
-its :ref:`write ahead log (WAL) <internals-wal>`. Each record in the WAL represents a single
+its :ref:`write-ahead log (WAL) <internals-wal>`. Each record in the WAL represents a single
 Tarantool data-change request such as :ref:`INSERT <box_space-insert>`,
-:ref:`UPDATE <box_space-update>` or :ref:`DELETE <box_space-delete>`, and is assigned
+:ref:`UPDATE <box_space-update>`, or :ref:`DELETE <box_space-delete>`, and is assigned
 a monotonically growing log sequence number (**LSN**). In essence, Tarantool
 replication is **row-based**: each data-change request is fully deterministic
 and operates on a single :ref:`tuple <index-box_tuple>`. However, unlike a classical row-based log, which
 contains entire copies of the changed rows, Tarantool's WAL contains copies of the requests.
 For example, for UPDATE requests, Tarantool only stores the primary key of the row and
-the update operations, to save space.
+the update operations to save space.
+
+.. NOTE::
+
+    `WAL extensions <https://www.tarantool.io/en/enterprise_doc/wal_extensions/>`_ available in Tarantool Enterprise enable you to add auxiliary information to each write-ahead log record.
+    This information might be helpful for implementing a CDC (Change Data Capture) utility that transforms a data replication stream.
 
-Invocations of **stored programs** are not written to the WAL. Instead, records
-of the actual **data-change requests, performed by the Lua code**, are
-written to the WAL. This ensures that possible non-determinism of Lua does not
-cause replication to go out of sync.
+The following are specifics of adding different types of information to the WAL:
 
-Data definition operations on **temporary spaces**, such as creating/dropping, adding
-indexes, truncating, etc., are written to the WAL, since information about
-temporary spaces is stored in non-temporary
-system spaces, such as :ref:`box.space._space <box_space-space>`. Data change
-operations on temporary spaces are not written to the WAL and are not replicated.
+*   Invocations of **stored programs** are not written to the WAL.
+    Instead, records of the actual **data-change requests, performed by the Lua code**, are written to the WAL.
+    This ensures that the possible non-determinism of Lua does not cause replication to go out of sync.
+
+*   Data definition operations on **temporary spaces** (:doc:`created </reference/reference_lua/box_schema/space_create>` with ``temporary = true``), such as creating/dropping, adding indexes, and truncating, are written to the WAL, since information about temporary spaces is stored in non-temporary system spaces, such as :ref:`box.space._space <box_space-space>`.
+
+*   Data change operations on temporary spaces are not written to the WAL and are not replicated.
 
 ..  _replication-local:
 
-Data change operations on **replication-local** spaces
-(spaces :doc:`created </reference/reference_lua/box_schema/space_create>`
-with ``is_local = true``)
-are written to the WAL but are not replicated.
-
-To create a valid initial state, to which WAL changes can be applied, every
-instance of a replica set requires a start set of
-:ref:`checkpoint files <index-box_persistence>`, such as .snap files for memtx
-and .run files for vinyl. A replica joining an existing replica set, chooses an
-existing master and automatically downloads the initial state from it. This is
-called an **initial join**.
-
-When an entire replica set is bootstrapped for the first time, there is no
-master which could provide the initial checkpoint. In such a case, replicas
-connect to each other and elect a master, which then creates the starting set of
-checkpoint files, and distributes it to all the other replicas. This is called
-an **automatic bootstrap** of a replica set.
-
-When a replica contacts a master (there can be many masters) for the first time,
-it becomes part of a replica set. On subsequent occasions, it should always
-contact a master in the same replica set. Once connected to the master, the
-replica requests all changes that happened after the latest local LSN (there
-can be many LSNs -- each master has its own LSN).
-
-Each replica set is identified by a globally unique identifier, called the
-**replica set UUID**. The identifier is created by the master which creates the
-very first checkpoint, and is part of the checkpoint file. It is stored in
-system space :ref:`box.space._schema <box_space-schema>`. For example:
+*    Data change operations on **replication-local** spaces (:doc:`created </reference/reference_lua/box_schema/space_create>` with ``is_local = true``) are written to the WAL but are not replicated.
+
+
+To learn how to enable replication, check the :ref:`Bootstrapping a replica set <replication-setup>` guide.
+
+
+.. _replication_stages:
+
+Replication stages
+~~~~~~~~~~~~~~~~~~
+
+To create a valid initial state, to which WAL changes can be applied, every instance of a replica set requires a start set of :ref:`checkpoint files <index-box_persistence>`, such as ``.snap`` files for memtx and ``.run`` files for vinyl.
+A replica uses the fiber called :ref:`applier <memtx-replication>` to receive the changes from a remote node and apply them to the replica's arena.
+The applier goes through the following stages:
+
+1.  **Connect**
+
+    This is the first stage of connecting a replica to other nodes.
+
+2.  **Auth** (optional)
+
+    This is an optional step performed if the :ref:`URI <index-uri>` contains :ref:`authentication <authentication>` information.
+
+3.  **Bootstrap** (optional)
+
+    When an entire replica set is bootstrapped for the first time, there is no master that could provide the initial checkpoint.
+    In such a case, replicas connect to each other and elect a master.
+    The master creates the starting set of checkpoint files and distributes them to all the other replicas.
+    This is called an **automatic bootstrap** of a replica set.
+
+4.  **Join**
+
+    At this stage, a replica downloads the initial state from the master.
+    If join fails with a non-critical :ref:`error <error_codes>`, for example, ``ER_READONLY``, ``ER_ACCESS_DENIED``, or a network-related issue, an instance tries to find a new master to join.
+
+    .. NOTE::
+
+        On subsequent connections, a replica downloads all changes happened after the latest local LSN (there can be many LSNs – each master has its own LSN).
+
+5.  **Subscribe**
+
+    At this stage, a replica fetches and applies updates from the master's write-ahead log.
+
+You can use the :ref:`box.info.replication[n].upstream.status <box_info_replication>` property to monitor the status of a replica.
+
+
+.. _replication_uuid:
+
+Replica set and instance UUIDs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each replica set is identified by a globally unique identifier, called the **replica set UUID**.
+The identifier is created by the master, which creates the very first checkpoint and is part of the checkpoint file. It is stored in the :ref:`box.space._schema <box_space-schema>` system space, for example:
 
 ..  code-block:: tarantoolsession
 
@@ -79,14 +112,14 @@ joins the replica set. It is called an **instance UUID** and is a globally uniqu
 identifier. The instance UUID is checked to ensure that instances do not join a different
 replica set, e.g. because of a configuration error. A unique instance identifier
 is also necessary to apply rows originating from different masters only once,
-that is, to implement multi-master replication. This is why each row in the write
-ahead log, in addition to its log sequence number, stores the instance identifier
+that is, to implement multi-master replication. This is why each row in the write-ahead log,
+in addition to its log sequence number, stores the instance identifier
 of the instance on which it was created. But using a UUID as such an identifier
-would take too much space in the write ahead log, thus a shorter integer number
+would take too much space in the write-ahead log, thus a shorter integer number
 is assigned to the instance when it joins a replica set. This number is then
-used to refer to the instance in the write ahead log. It is called
-**instance id**. All identifiers are stored in system space
-:ref:`box.space._cluster <box_space-cluster>`. For example:
+used to refer to the instance in the write-ahead log. It is called
+**instance id**. All identifiers are stored in the system space
+:ref:`box.space._cluster <box_space-cluster>`, for example:
 
 ..  code-block:: tarantoolsession
 
@@ -112,12 +145,10 @@ describes the state of replication in regard to each connected peer.
 Here ``vclock`` contains log sequence numbers (827 and 584) for instances with
 instance IDs 1 and 2.
 
-Starting in Tarantool 1.7.7, it is possible for administrators to assign
-the instance UUID and the replica set UUID values, rather than let the system
-generate them -- see the description of the
-:ref:`replicaset_uuid <cfg_replication-replicaset_uuid>` configuration parameter.
+If required, you can explicitly specify the instance and the replica set UUID values rather than letting Tarantool generate them.
+To learn more, see the :ref:`replicaset_uuid <cfg_replication-replicaset_uuid>` configuration parameter description.
+
 
-To learn how to enable replication, check the :ref:`how-to guide <replication-setup>`.
 
 ..  _replication-roles:
 
@@ -137,7 +168,7 @@ be visible on the replicas, but not vice versa.
 A simple two-instance replica set with the master on one machine and the replica
 on a different machine provides two benefits:
 
-*   **failover**, because if the master goes down then the replica can take over,
+*   **failover**, because if the master goes down, then the replica can take over,
     and
 *   **load balancing**, because clients can connect to either the master or the
     replica for read requests.
@@ -168,8 +199,8 @@ order on all replicas (e.g. the DELETE is used to prune expired data),
 a master-master configuration is also safe.
 
 UPDATE operations, however, can easily go out of sync. For example, assignment
-and increment are not commutative, and may yield different results if applied
-in different order on different instances.
+and increment are not commutative and may yield different results if applied
+in a different order on different instances.
 
 More generally, it is only safe to use Tarantool master-master replication if
 all database changes are **commutative**: the end result does not depend on the
@@ -179,15 +210,15 @@ conflict-free replicated data types
 
 ..  _replication-topologies:
 
-Replication topologies: cascade, ring and full mesh
----------------------------------------------------
+Replication topologies: cascade, ring, and full mesh
+----------------------------------------------------
 
 Replication topology is set by the :ref:`replication <cfg_replication-replication>`
-configuration parameter. The recommended topology is a **full mesh**, because it
+configuration parameter. The recommended topology is a **full mesh** because it
 makes potential failover easy.
 
 Some database products offer **cascading replication** topologies: creating a
-replica on a replica. Tarantool does not recommend such setup.
+replica on a replica. Tarantool does not recommend such a setup.
 
 ..  image:: images/no-cascade.svg
     :align: center
@@ -204,14 +235,14 @@ such instances when replication topology changes. Here is how this can happen:
 
 We have a chain of three instances. Instance #1 contains entries for instances
 #1 and #2 in its ``_cluster`` space. Instances #2 and #3 contain entries for
-instances #1, #2 and #3 in their ``_cluster`` spaces.
+instances #1, #2, and #3 in their ``_cluster`` spaces.
 
 ..  image:: images/cascade-problem-2.svg
     :align: center
 
 Now instance #2 is faulty. Instance #3 tries connecting to instance #1 as its
-new master, but the master refuses the connection since it has no entry for
-instance #3.
+new master, but the master refuses the connection since it has no entry, for
+example, #3.
 
 **Ring replication** topology is, however, supported:
 
@@ -244,6 +275,6 @@ Orphan status
 
 During ``box.cfg()``, an instance tries to join all nodes listed
 in :ref:`box.cfg.replication <cfg_replication-replication>`.
-If the instance does not succeed with connecting to the required number of nodes
+If the instance does not succeed in connecting to the required number of nodes
 (see :ref:`bootstrap_strategy <cfg_replication-bootstrap_strategy>`),
 it switches to the :ref:`orphan status <internals-replication-orphan_status>`.