Update the upgrade instructions (#3080)

* Add note about caveats when upgrading from 1.6 to 1.10
* Update the upgrading instructions
* Add instructions on checking status before upgrade
* Add instruction on direct upgrade from 1.6 to 2.x
* Add explicit box.schema.upgrade description to the reference
* Bring indents in accordance with the style guide

This PR adapts the following instructions from tarantool/wiki, written by Serge Petrenko:
https://github.com/tarantool/tarantool/wiki/Caveats-when-upgrading-from-tarantool-1.6
https://github.com/tarantool/tarantool/wiki/Fix-wrong-order-of-decimals-and-doubles-in-indices

Resolves #3042
Resolves #1807
Resolves #836

Co-authored-by: Serge Petrenko <[email protected]>
Co-authored-by: Evgeniy Osintsev <[email protected]>

Author: 3 people

doc/book/admin/_includes/1.6-to-2.x-condition.rst

@@ -0,0 +1,12 @@
+Versions later that 1.6 have incompatible :ref:`.snap <internals-snapshot>` and
+:ref:`.xlog <internals-wal>` file formats: 1.6 files are
+supported during upgrade, but you won’t be able to return to 1.6 after running
+under 1.10 or 2.x for a while. A few configuration parameters are also renamed.
+
+To perform a **live** upgrade from Tarantool 1.6 to a more recent version,
+like :doc:`2.8.4 </release/2.8.4>`, :doc:`2.10.1 </release/2.10.1>` and such,
+it is necessary to take an intermediate step by upgrading 1.6 -> **1.10** -> 2.x.
+This is the only way to perform the upgrade **without downtime**.
+
+However, a direct upgrade of a replica set from 1.6 to 2.x is also possible, but only
+**with downtime**.

doc/book/admin/_includes/script_find_indices.rst

@@ -0,0 +1,54 @@
+local fiber = require('fiber')
+local decimal = require('decimal')
+
+local function isnan(val)
+    return type(val) == 'number' and val ~= val
+end
+
+local function isinf(val)
+    return val == math.huge or val == -math.huge
+end
+
+local function vinyl(id)
+    return box.space[id].engine == 'vinyl'
+end
+
+require_rebuild = {}
+local iters = 0
+for _, v in box.space._index:pairs({512, 0}, {iterator='GE'}) do
+    local id = v[1]
+    iters = iters + 1
+    if iters % 1000 == 0 then
+        fiber.yield()
+    end
+    if vinyl(id) then
+        local format = v[6]
+        local check_fields = {}
+        for _, fmt in pairs(v[6]) do
+            if fmt[2] == 'number' or fmt[2] == 'scalar' then
+                table.insert(check_fields,  fmt[1] + 1)
+            end
+        end
+        local have_decimal = {}
+        local have_nan = {}
+        if #check_fields > 0 then
+            for k, tuple in box.space[id]:pairs() do
+                for _, i in pairs(check_fields) do
+                    iters = iters + 1
+                    if iters % 1000 == 0 then
+                        fiber.yield()
+                    end
+                    have_decimal[i] = have_decimal[i] or
+                                    decimal.is_decimal(tuple[i])
+                    have_nan[i] = have_nan[i] or isnan(tuple[i]) or
+                                isinf(tuple[i])
+                    if have_decimal[i] and have_nan[i] then
+                        table.insert(require_rebuild, v)
+                        goto out
+                    end
+                end
+            end
+        end
+    end
+    ::out::
+end

doc/book/admin/_includes/script_rebuild_indices.rst

@@ -0,0 +1,25 @@
+local log = require('log')
+
+local function rebuild_index(idx)
+    local index_name = idx[3]
+    local space_name = box.space[idx[1]].name
+    log.info("Rebuilding index %s on space %s", index_name, space_name)
+    if (idx[2] == 0) then
+        log.error("Cannot rebuild primary index %s on space %s. Please, "..
+                "recreate the space manually", index_name, space_name)
+        return
+    end
+    log.info("Deleting index %s on space %s", index_name, space_name)
+    local v = box.space._index:delete{idx[1], idx[2]}
+    if v == nil then
+        log.error("Couldn't find index %s on space %s", index_name, space_name)
+        return
+    end
+    log.info("Done")
+    log.info("Creating index %s on space %s", index_name, space_name)
+    box.space._index:insert(v)
+end
+
+for _, idx in pairs(require_rebuild) do
+    rebuild_index(idx)
+end

doc/book/admin/_includes/upgrade_procedure.rst

@@ -0,0 +1,46 @@
+1.  Pick any read-only instance in the replica set.
+
+2.  Upgrade this replica to the new Tarantool version. See details in
+    :ref:`Upgrading Tarantool on a standalone instance <admin-upgrades_instance>`.
+    This requires stopping the instance for a while,
+    which won't interrupt the replica set operation.
+    When the upgraded replica is up again, it will synchronize with the other instances in the replica set
+    so that the data are consistent across all the instances.
+
+3.  Make sure the upgraded replica is running and connected to the rest of the replica set just fine.
+    To do this, run ``box.info.replication`` in the instance's console
+    and check the output table for values like ``upstream``, ``downstream``, and ``lag``.
+
+    For each instance ``id``, there are ``upstream`` and ``downstream`` values.
+    Both of them should have the value ``follow``, except on the instance where you run this code.
+    This means that the replicas are connected and there are no errors in the data flow.
+
+    The value of the ``lag`` field can be less or equal than ``box.cfg.replication_timeout``,
+    but it can also be moderately larger.
+    For example, if ``box.cfg.replication_timeout`` is 1 second and the write load on the master is high,
+    it's generally OK to have a lag of about 10 seconds on the master.
+    It is up to the user to decide what lag values are fine.
+
+4.  :ref:`Upgrade <admin-upgrades_instance>` all the read-only instances by repeating steps 1--3
+    until only the master keeps running the old Tarantool version.
+
+5.  Make one of the updated replicas the new master:
+    
+    -   If the replica set is using asynchronous replication without
+        :ref:`RAFT-based leader elections <repl_leader_elect>`,
+        first run ``box.cfg{ read_only = true }`` on the old master
+        and then run ``box.cfg{ read_only = false }`` on the replica that will be the new master.
+
+    -   If the replica set is using :ref:`synchronous replication <repl_sync>` or
+        :ref:`RAFT-based leader elections <repl_leader_elect>`,
+        run ``box.ctl.promote()`` on the new master and then run
+        ``box.cfg{ election_mode = 'voter' }`` on the old master.
+        This will automatically change the ``read_only`` statuses on the instances.
+
+    -   For a Cartridge replica set, it is possible to select the new master in the web UI.
+
+    There is no need to restart the new master.
+
+    Check that the new master continues following and being followed by all other replicas, similarly to step 3.
+
+6.  :ref:`Upgrade <admin-upgrades_instance>` the former master, which is now a read-only instance.

doc/book/admin/upgrades.rst

@@ -1,152 +1,113 @@
-.. _admin-upgrades:
+..  _admin-upgrades:
 
-================================================================================
 Upgrades
-================================================================================
+========
 
 For information about backwards compatibility,
 see the :ref:`compatibility guarantees <compatibility_guarantees>` description.
 
-.. _admin-upgrades_db:
+..  _admin-upgrades_instance:
 
---------------------------------------------------------------------------------
-Upgrading a Tarantool database
---------------------------------------------------------------------------------
+Upgrading Tarantool on a standalone instance
+--------------------------------------------
 
-If you created a database with an older Tarantool version and have now installed
-a newer version, make the request ``box.schema.upgrade()``. This updates
-Tarantool system spaces to match the currently installed version of Tarantool.
-
-For example, here is what happens when you run ``box.schema.upgrade()`` with a
-database created with Tarantool version 1.6.4 to version 1.7.2 (only a small
-part of the output is shown):
-
-.. code-block:: tarantoolsession
-
-   tarantool> box.schema.upgrade()
-   alter index primary on _space set options to {"unique":true}, parts to [[0,"unsigned"]]
-   alter space _schema set options to {}
-   create view _vindex...
-   grant read access to 'public' role for _vindex view
-   set schema version to 1.7.0
-   ---
-   ...
-
-.. _admin-upgrades_instance:
-
---------------------------------------------------------------------------------
-Upgrading a Tarantool instance
---------------------------------------------------------------------------------
-
-Tarantool is backward compatible between two adjacent versions. For example, you
-should have no or little trouble when upgrading from Tarantool 1.6 to 1.7, or
-from Tarantool 1.7 to 2.x. Meanwhile Tarantool 2.x may have incompatible changes
-when migrating from Tarantool 1.6. to 2.x directly.
-
-.. _admin-upgrades_instance_17_to_20:
-
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.7 to 2.x
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-1. Stop the Tarantool server.
-
-2. Make a copy of all data (see an appropriate hot backup procedure in
-   :ref:`Backups <admin-backups>`) and the package from which the current (old)
-   version was installed (for rollback purposes).
-
-3. Update the Tarantool server. See installation instructions at Tarantool
-   `download page <http://tarantool.org/download.html>`_.
+This procedure is for upgrading a standalone Tarantool instance in production.
+Notice that this will **always imply a downtime**.
+To upgrade **without downtime**, you need several Tarantool servers running in a
+replication cluster (see :ref:`below <admin-upgrades_replication_cluster>`).
 
-4. Launch the updated Tarantool server using ``tarantoolctl`` or ``systemctl``.
+#.  Stop the Tarantool server.
 
-.. _admin-upgrades_instance_16_to_20:
+#.  Make a copy of all data (see an appropriate hot backup procedure in
+    :ref:`Backups <admin-backups>`) and the package from which the current (old)
+    version was installed (for rollback purposes).
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.6 to 2.x
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#.  Update the Tarantool server. See installation instructions at Tarantool
+    `download page <http://tarantool.org/download.html>`_.
 
-The procedure is fully analogous to
-:ref:`upgrading from 1.7 to 2.x <admin-upgrades_instance_17_to_20>`.
+#.  Run :ref:`box.schema.upgrade() <box_schema-upgrade>` on the new master.
+    This will update the Tarantool system spaces to match the currently installed version of Tarantool.
+    There is no need  to run ``box.schema.upgrade()`` on every node:
+    changes are propagated to other nodes via the regular replication mechanism.
 
-.. _admin-upgrades_instance_16_to_17:
+#.  Update your application files, if needed.
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-How to upgrade from Tarantool 1.6 to 1.7
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#.  Launch the updated Tarantool server using ``tarantoolctl``, ``tt``, or ``systemctl``.
 
-This procedure is for upgrading a standalone Tarantool instance in production
-from 1.6.x to 1.7.x. Notice that this will **always imply a downtime**.
-To upgrade **without downtime**, you need several Tarantool servers running in a
-replication cluster (see :ref:`below <admin-upgrades_replication_cluster>`).
+..  _admin-upgrades_replication_cluster:
 
-Tarantool 1.7 has an incompatible :ref:`.snap <internals-snapshot>` and
-:ref:`.xlog <internals-wal>` file format: 1.6 files are
-supported during upgrade, but you won’t be able to return to 1.6 after running
-under 1.7 for a while. It also renames a few configuration parameters, but old
-parameters are supported. The full list of breaking changes is available in
-`release notes for Tarantool 1.7 <https://github.com/tarantool/tarantool/releases>`_.
+Upgrading Tarantool in a replica set with no downtime
+-----------------------------------------------------
 
-1. Check with application developers whether application files need to be
-   updated due to incompatible changes (see
-   `1.7 release notes <https://github.com/tarantool/tarantool/releases>`_).
-   If yes, back up the old application files.
+Below are the general instructions for upgrading Tarantool in a replica set.
+Upgrading from some versions can involve certain specifics. You can find
+instructions for individual versions :ref:`in the list below <admin-upgrades_version_specifics>`.
 
-2. Stop the Tarantool server.
+..  important::
 
-3. Make a copy of all data (see an appropriate hot backup procedure in
-   :ref:`Backups <admin-backups>`) and the package from which the current (old)
-   version was installed (for rollback purposes).
+    The only way to upgrade Tarantool from version 1.6, 1.7, or 1.9 to 2.x **without downtime** is
+    taking an intermediate step by upgrading to 1.10 and then to 2.x.
 
-4. Update the Tarantool server. See installation instructions at Tarantool
-   `download page <http://tarantool.org/download.html>`_.
+    Before upgrading Tarantool from 1.6 to 2.x, please read about the associated
+    :ref:`caveats <admin-upgrades-1.6-1.10>`.
 
-5. Update the Tarantool database. Put the request ``box.schema.upgrade()``
-   inside a :doc:`box.once() </reference/reference_lua/box_once>` function in your Tarantool
-   :ref:`initialization file <index-init_label>`.
-   On startup, this will create new system spaces, update data type names (e.g.
-   num -> unsigned, str -> string) and options in Tarantool system spaces.
+Preparations
+~~~~~~~~~~~~
 
-6. Update application files, if needed.
+#.  Check the replica set health by running the following code on every instance:
 
-7. Launch the updated Tarantool server using ``tarantoolctl`` or ``systemctl``.
+    ..  code-block:: tarantoolsession
 
-.. _admin-upgrades_replication_cluster:
+        box.info.ro -- "false" at least on one instance
+        box.info.status -- should be "running"
+ 
+    If all instances have ``box.info.ro = true``, this means there are no writable nodes.
+    If you're running Tarantool :doc:`v. 2.10.0 </release/2.10.0>` or later,
+    you can find out the reason by running ``box.info.ro_reason``.
+    If it has the value ``orphan``, the instance doesn't see the rest of the replica set.
+    Similarly, if ``box.info.status`` has the value ``orphan``, the instance doesn't see the rest of the replica set.
+    First resolve the replication issues and only then continue.
 
---------------------------------------------------------------------------------
-Upgrading Tarantool in a replication cluster
---------------------------------------------------------------------------------
+    If you're running Cartridge, you can also check node health in the UI.
 
-Tarantool 1.7 can work as a :ref:`replica <replication-architecture>`
-for Tarantool 1.6 and vice versa. Replicas
-perform capability negotiation on handshake, and new 1.7 replication features
-are not used with 1.6 replicas. This allows upgrading clustered configurations.
+#.  Make sure each replica is connected to the rest of the replica set.
+    To do this, run ``box.info.replication`` in the instance's console
+    and check the output table for values like ``upstream``, ``downstream``, and ``lag``.
 
-This procedure allows for a rolling upgrade **without downtime** and works for
-any cluster configuration: master-master or master-replica.
+    For each instance ``id``, there are ``upstream`` and ``downstream`` values.
+    Both of them should have the value ``follow``, except on the instance where you run this code.
+    This means that the replicas are connected and there are no errors in the data flow.
 
-1. Upgrade Tarantool at all replicas (or at any master in a master-master
-   cluster). See details in
-   :ref:`Upgrading a Tarantool instance <admin-upgrades_instance>`.
+    The value of the ``lag`` field can be less or equal than ``box.cfg.replication_timeout``,
+    but it can also be moderately larger.
+    For example, if ``box.cfg.replication_timeout`` is 1 second and the write load on the master is high,
+    it's generally OK to have a lag of about 10 seconds on the master.
+    It is up to the user to decide what lag values are fine.
+    
+If the replica set is healthy, proceed to the upgrade.
 
-2. Verify installation on the replicas:
+Upgrade procedure
+~~~~~~~~~~~~~~~~~
 
-   a. Start Tarantool.
+..  include:: ./_includes/upgrade_procedure.rst
 
-   b. Attach to the master and start working as before.
+7.  Run :ref:`box.schema.upgrade() <box_schema-upgrade>` on the new master.
+    This will update the Tarantool system spaces to match the currently installed version of Tarantool.
+    There is no need  to run ``box.schema.upgrade()`` on every node:
+    changes are propagated to other nodes via the regular replication mechanism.
 
-   The master runs the old Tarantool version, which is always compatible with
-   the next major version.
+8.  Run ``box.snapshot()`` on every node in the replica set
+    to make sure that the replicas immediately see the upgraded database state in case of restart.
 
-3. Upgrade the master. The procedure is similar to upgrading a replica.
 
-4. Verify master installation:
+..  _admin-upgrades_version_specifics:
 
-   a. Start Tarantool with replica configuration to catch up.
+Version specifics
+-----------------
 
-   b. Switch to master mode.
+..  toctree::
+    :maxdepth: 1
 
-5. Upgrade the database on any master node in the cluster. Make the request
-   ``box.schema.upgrade()``. This updates Tarantool system spaces to match the
-   currently installed version of Tarantool. Changes are propagated to other
-   nodes via the regular replication mechanism.
+    upgrades/1.6-1.10
+    upgrades/1.6-2.0-downtime
+    upgrades/2.10.1