Update the upgrade instructions

doc/book/admin/_includes/1.6-to-2.x-condition.rst

@@ -0,0 +1,7 @@
+To perform a **live** upgrade from Tarantool 1.6 to a more recent version,
+like :doc:`2.8.4 </release/2.8.4>`, :doc:`2.10.1 </release/2.10.1>` and such,
+it is necessary to take an intermediate step by upgrading 1.6 -> 1.10 -> 2.x.
+This is the only way to perform the upgrade without downtime.
+
+However, a direct upgrade of a replica set from 1.6 to 2.x is also possible, but only
+**with downtime**.

doc/book/admin/_includes/script_find_indices.rst

@@ -0,0 +1,54 @@
+local fiber = require('fiber')
+local decimal = require('decimal')
+
+local function isnan(val)
+    return type(val) == 'number' and val ~= val
+end
+
+local function isinf(val)
+    return val == math.huge or val == -math.huge
+end
+
+local function vinyl(id)
+    return box.space[id].engine == 'vinyl'
+end
+
+require_rebuild = {}
+local iters = 0
+for _, v in box.space._index:pairs({512, 0}, {iterator='GE'}) do
+    local id = v[1]
+    iters = iters + 1
+    if iters % 1000 == 0 then
+        fiber.yield()
+    end
+    if vinyl(id) then
+        local format = v[6]
+        local check_fields = {}
+        for _, fmt in pairs(v[6]) do
+            if fmt[2] == 'number' or fmt[2] == 'scalar' then
+                table.insert(check_fields,  fmt[1] + 1)
+            end
+        end
+        local have_decimal = {}
+        local have_nan = {}
+        if #check_fields > 0 then
+            for k, tuple in box.space[id]:pairs() do
+                for _, i in pairs(check_fields) do
+                    iters = iters + 1
+                    if iters % 1000 == 0 then
+                        fiber.yield()
+                    end
+                    have_decimal[i] = have_decimal[i] or
+                                    decimal.is_decimal(tuple[i])
+                    have_nan[i] = have_nan[i] or isnan(tuple[i]) or
+                                isinf(tuple[i])
+                    if have_decimal[i] and have_nan[i] then
+                        table.insert(require_rebuild, v)
+                        goto out
+                    end
+                end
+            end
+        end
+    end
+    ::out::
+end

doc/book/admin/_includes/script_rebuild_indices.rst

@@ -0,0 +1,25 @@
+local log = require('log')
+
+local function rebuild_index(idx)
+    local index_name = idx[3]
+    local space_name = box.space[idx[1]].name
+    log.info("Rebuilding index %s on space %s", index_name, space_name)
+    if (idx[2] == 0) then
+        log.error("Cannot rebuild primary index %s on space %s. Please, "..
+                "recreate the space manually", index_name, space_name)
+        return
+    end
+    log.info("Deleting index %s on space %s", index_name, space_name)
+    local v = box.space._index:delete{idx[1], idx[2]}
+    if v == nil then
+        log.error("Couldn't find index %s on space %s", index_name, space_name)
+        return
+    end
+    log.info("Done")
+    log.info("Creating index %s on space %s", index_name, space_name)
+    box.space._index:insert(v)
+end
+
+for _, idx in pairs(require_rebuild) do
+    rebuild_index(idx)
+end

doc/book/admin/_includes/upgrade_procedure.rst

@@ -0,0 +1,21 @@
+1. Pick any replica in the replica set.
+
+2. Upgrade this replica to the new Tarantool version. See details in
+   :ref:`Upgrading a Tarantool instance <admin-upgrades_instance>`.
+
+3. Make sure the replica connected to the rest of the replica set just fine:
+
+   ..  code-block:: tarantoolsession
+
+       box.info.replication[id].upstream
+       box.info.replication[id].downstream
+
+   The ``status`` field in both outputs should have the value ``follow``.
+
+4. :ref:`Upgrade <admin-upgrades_instance>` all the replicas by repeating steps 1--3
+   until only the master keeps running the old Tarantool version.
+
+5. Make one of the updated replicas the new master.
+   Check that it continues following and being followed by all other replicas.
+
+6. :ref:`Upgrade <admin-upgrades_instance>` the former master.

doc/book/admin/upgrades.rst

@@ -1,53 +1,20 @@
 .. _admin-upgrades:
 
-================================================================================
 Upgrades
-================================================================================
+========
 
 For information about backwards compatibility,
 see the :ref:`compatibility guarantees <compatibility_guarantees>` description.
 
-.. _admin-upgrades_db:
-
---------------------------------------------------------------------------------
-Upgrading a Tarantool database
---------------------------------------------------------------------------------
-
-If you created a database with an older Tarantool version and have now installed
-a newer version, make the request ``box.schema.upgrade()``. This updates
-Tarantool system spaces to match the currently installed version of Tarantool.
-
-For example, here is what happens when you run ``box.schema.upgrade()`` with a
-database created with Tarantool version 1.6.4 to version 1.7.2 (only a small
-part of the output is shown):
-
-.. code-block:: tarantoolsession
-
-   tarantool> box.schema.upgrade()
-   alter index primary on _space set options to {"unique":true}, parts to [[0,"unsigned"]]
-   alter space _schema set options to {}
-   create view _vindex...
-   grant read access to 'public' role for _vindex view
-   set schema version to 1.7.0
-   ---
-   ...
-
 .. _admin-upgrades_instance:
 
---------------------------------------------------------------------------------
-Upgrading a Tarantool instance
---------------------------------------------------------------------------------
-
-Tarantool is backward compatible between two adjacent versions. For example, you
-should have no or little trouble when upgrading from Tarantool 1.6 to 1.7, or
-from Tarantool 1.7 to 2.x. Meanwhile Tarantool 2.x may have incompatible changes
-when migrating from Tarantool 1.6. to 2.x directly.
-
-.. _admin-upgrades_instance_17_to_20:
+Upgrading Tarantool on a standalone instance
+--------------------------------------------
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.7 to 2.x
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This procedure is for upgrading a standalone Tarantool instance in production.
+Notice that this will **always imply a downtime**.
+To upgrade **without downtime**, you need several Tarantool servers running in a
+replication cluster (see :ref:`below <admin-upgrades_replication_cluster>`).
 
 1. Stop the Tarantool server.
 
@@ -58,95 +25,113 @@ How to upgrade from Tarantool 1.7 to 2.x
 3. Update the Tarantool server. See installation instructions at Tarantool
    `download page <http://tarantool.org/download.html>`_.
 
-4. Launch the updated Tarantool server using ``tarantoolctl`` or ``systemctl``.
+After that, make sure to :ref:`finish the upgrade properly <admin-upgrades_db>`.
 
-.. _admin-upgrades_instance_16_to_20:
+.. _admin-upgrades_replication_cluster:
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.6 to 2.x
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Upgrading Tarantool in a replica set with no downtime
+-----------------------------------------------------
 
-The procedure is fully analogous to
-:ref:`upgrading from 1.7 to 2.x <admin-upgrades_instance_17_to_20>`.
+Below are the general instructions for upgrading Tarantool in a replica set.
+Upgrading from some versions can involve certain specifics. You can find
+instructions for individual versions :ref:`in the list below <admin-upgrades_version_specifics>`.
 
-.. _admin-upgrades_instance_16_to_17:
+..  important::
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.6 to 1.7
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    The only way to upgrade Tarantool from version 1.6, 1.7, or 1.9 to 2.x **without downtime** is
+    taking an intermediate step by upgrading to 1.10 and then to 2.x.
 
-This procedure is for upgrading a standalone Tarantool instance in production
-from 1.6.x to 1.7.x. Notice that this will **always imply a downtime**.
-To upgrade **without downtime**, you need several Tarantool servers running in a
-replication cluster (see :ref:`below <admin-upgrades_replication_cluster>`).
+    Before upgrading Tarantool from 1.6 to 2.x, please read about the associated
+    :ref:`caveats <admin-upgrades-1.6-1.10>`.
 
-Tarantool 1.7 has an incompatible :ref:`.snap <internals-snapshot>` and
-:ref:`.xlog <internals-wal>` file format: 1.6 files are
-supported during upgrade, but you won’t be able to return to 1.6 after running
-under 1.7 for a while. It also renames a few configuration parameters, but old
-parameters are supported. The full list of breaking changes is available in
-`release notes for Tarantool 1.7 <https://github.com/tarantool/tarantool/releases>`_.
+Preparations
+~~~~~~~~~~~~
 
-1. Check with application developers whether application files need to be
-   updated due to incompatible changes (see
-   `1.7 release notes <https://github.com/tarantool/tarantool/releases>`_).
-   If yes, back up the old application files.
+#. Check the replica set health by running the following code on every instance:
 
-2. Stop the Tarantool server.
+   ..  code-block:: tarantoolsession
 
-3. Make a copy of all data (see an appropriate hot backup procedure in
-   :ref:`Backups <admin-backups>`) and the package from which the current (old)
-   version was installed (for rollback purposes).
+       box.info.ro -- "false" at least on one instance
+       box.info.status -- should be "running"
+
+   If all instances have ``box.info.ro = true``, this means there are no writable nodes.
+   If you're running Tarantool :doc:`v. 2.10.0 </release/2.10.0>` or later,
+   you can find out the reason by running ``box.info.ro_reason``.
+   If it has the value ``orphan``, the instance doesn't see the rest of the replica set.
+   Similarly, if ``box.info.status`` has the value ``orphan``, the instance doesn't see the rest of the replica set.
+   First resolve the replication issues and only then continue.
 
-4. Update the Tarantool server. See installation instructions at Tarantool
-   `download page <http://tarantool.org/download.html>`_.
+   If you're running Cartridge, you can also check node health in the UI.
 
-5. Update the Tarantool database. Put the request ``box.schema.upgrade()``
-   inside a :doc:`box.once() </reference/reference_lua/box_once>` function in your Tarantool
-   :ref:`initialization file <index-init_label>`.
-   On startup, this will create new system spaces, update data type names (e.g.
-   num -> unsigned, str -> string) and options in Tarantool system spaces.
+#. Make sure each replica connected to the rest of the replica set.
+   Run ``box.info.replication`` and check the output table.
+   For each instance ``id``, there are ``upstream`` and ``downstream`` values.
+   Both of them should have the value ``follow``, except on the instance where you run this code.
+   This means that the replicas are connected and there are no errors in the data flow.
 
-6. Update application files, if needed.
+   The value of the ``lag`` field can be less or equal than ``box.cfg.replication_timeout``,
+   but it can also be moderately larger.
+   For example, if ``box.cfg.replication_timeout`` is 1 second, it's generally OK to have a lag of about 10 seconds.
+   It is up to the user to decide what lag values are fine.
 
-7. Launch the updated Tarantool server using ``tarantoolctl`` or ``systemctl``.
+If the replica set is healthy, proceed to the upgrade.
 
-.. _admin-upgrades_replication_cluster:
+Upgrade procedure
+~~~~~~~~~~~~~~~~~
+
+..  include:: ./_includes/upgrade_procedure.rst
+
+After upgrading the replica set, make sure to run ``box.schema.upgrade()`` on the new master
+as described below in the section ":ref:`Finishing the upgrade <admin-upgrades_db>`".
+There is no need  to run ``box.schema.upgrade()`` on every node:
+changes are propagated to other nodes via the regular replication mechanism.
 
---------------------------------------------------------------------------------
-Upgrading Tarantool in a replication cluster
---------------------------------------------------------------------------------
+Finally, run ``box.snapshot()`` on every node in the replica set
+to make sure that the replicas immediately see the upgraded database state in case of restart.
+
+.. _admin-upgrades_db:
 
-Tarantool 1.7 can work as a :ref:`replica <replication-architecture>`
-for Tarantool 1.6 and vice versa. Replicas
-perform capability negotiation on handshake, and new 1.7 replication features
-are not used with 1.6 replicas. This allows upgrading clustered configurations.
+Finalizing the upgrade
+----------------------
 
-This procedure allows for a rolling upgrade **without downtime** and works for
-any cluster configuration: master-master or master-replica.
+1.  If you created a database with an older Tarantool version and have now installed
+    a newer version, make the request ``box.schema.upgrade()``. This updates
+    Tarantool system spaces to match the currently installed version of Tarantool.
 
-1. Upgrade Tarantool at all replicas (or at any master in a master-master
-   cluster). See details in
-   :ref:`Upgrading a Tarantool instance <admin-upgrades_instance>`.
+    For example, here is what happens when you run ``box.schema.upgrade()`` with a
+    database created with Tarantool version 1.6.4 to version 1.7.2 (only a small
+    part of the output is shown):
 
-2. Verify installation on the replicas:
+    .. code-block:: tarantoolsession
 
-   a. Start Tarantool.
+       tarantool> box.schema.upgrade()
+       alter index primary on _space set options to {"unique":true}, parts to [[0,"unsigned"]]
+       alter space _schema set options to {}
+       create view _vindex...
+       grant read access to 'public' role for _vindex view
+       set schema version to 1.7.0
+       ---
+       ...
+
+    You can also put the request ``box.schema.upgrade()``
+    inside a :doc:`box.once() </reference/reference_lua/box_once>` function in your Tarantool
+    :ref:`initialization file <index-init_label>`.
+    On startup, this will create new system spaces, update data type names (for example,
+    ``num`` -> ``unsigned``, ``str`` -> ``string``) and options in Tarantool system spaces.
 
-   b. Attach to the master and start working as before.
+2.  Update your application files, if needed.
 
-   The master runs the old Tarantool version, which is always compatible with
-   the next major version.
+3.  Launch the updated Tarantool server using ``tarantoolctl``, ``tt``, or ``systemctl``.
 
-3. Upgrade the master. The procedure is similar to upgrading a replica.
 
-4. Verify master installation:
+..  _admin-upgrades_version_specifics:
 
-   a. Start Tarantool with replica configuration to catch up.
+Version specifics
+-----------------
 
-   b. Switch to master mode.
+..  toctree::
+    :maxdepth: 1
 
-5. Upgrade the database on any master node in the cluster. Make the request
-   ``box.schema.upgrade()``. This updates Tarantool system spaces to match the
-   currently installed version of Tarantool. Changes are propagated to other
-   nodes via the regular replication mechanism.
+    upgrades/1.6-1.10
+    upgrades/1.6-2.0-downtime
+    upgrades/2.10.1