gh-36982: Make pyproject.toml the source for build dependencies
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes #1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

As noted in
#36951 (comment), the
fact that the dependency declaration of sage-the-library depends on
information contain in sage-the-distro's build folder is problematic for
the separation of concerns. Sage-the-distro should rely on information
of sage-the-library, not the other way around. Here we fix this for the
build requirements by making `src/pyproject.toml` the single source of
thruth. The corresponding `install-requires.txt` are demoted to mere
indicator files (to be removed later). Moreover, this change allows us
to make `pyproject.toml` a static file (no longer generated by
configure), which should help with downstream packaging.

---

Happy new year!

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [ ] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

- #37796
<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #36982
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Gonzalo Tornaría, Matthias Köppe, Tobias Diez

Author: Release Manager

.gitignore

@@ -53,7 +53,6 @@
 
 /src/setup.cfg
 /src/requirements.txt
-/src/pyproject.toml
 /src/Pipfile
 /src/Pipfile.lock
 /Pipfile
@@ -186,6 +185,19 @@ __pycache__/
 build/temp.*/
 build/bin/sage-build-env-config
 
+# Generated files in build
+build/pkgs/cypari/version_requirements.txt
+build/pkgs/cysignals/version_requirements.txt
+build/pkgs/cython/version_requirements.txt
+build/pkgs/gmpy2/version_requirements.txt
+build/pkgs/jupyter_core/version_requirements.txt
+build/pkgs/memory_allocator/version_requirements.txt
+build/pkgs/numpy/version_requirements.txt
+build/pkgs/pkgconfig/version_requirements.txt
+build/pkgs/pplpy/version_requirements.txt
+build/pkgs/setuptools/version_requirements.txt
+build/pkgs/wheel/version_requirements.txt
+
 # Generated files in the top-level source trees
 /pkgs/*/build
 /pkgs/*/dist

Makefile

@@ -175,9 +175,19 @@ bootstrap-clean:
 	rm -rf src/doc/en/reference/spkg/*.rst
 	for a in environment environment-optional src/environment src/environment-dev src/environment-optional; do rm -f $$a.yml $$a-3.[89].yml $$a-3.1[0-9].yml; done
 	rm -f src/Pipfile
-	rm -f src/pyproject.toml
 	rm -f src/requirements.txt
 	rm -f src/setup.cfg
+	rm -f build/pkgs/cypari/version_requirements.txt
+	rm -f build/pkgs/cysignals/version_requirements.txt
+	rm -f build/pkgs/cython/version_requirements.txt
+	rm -f build/pkgs/gmpy2/version_requirements.txt
+	rm -f build/pkgs/jupyter_core/version_requirements.txt
+	rm -f build/pkgs/memory_allocator/version_requirements.txt
+	rm -f build/pkgs/numpy/version_requirements.txt
+	rm -f build/pkgs/pkgconfig/version_requirements.txt
+	rm -f build/pkgs/pplpy/version_requirements.txt
+	rm -f build/pkgs/setuptools/version_requirements.txt
+	rm -f build/pkgs/wheel/version_requirements.txt
 
 # Remove absolutely everything which isn't part of the git repo
 maintainer-clean: distclean bootstrap-clean

bootstrap

@@ -35,6 +35,15 @@ CONFVERSION=$(cat $PKG/package-version.txt)
 
 
 bootstrap () {
+    for pkgname in cypari cysignals cython gmpy2 jupyter_core memory_allocator numpy pkgconfig pplpy setuptools wheel; do
+        # Write the version_requirements.txt files for dependencies declared in pyproject.toml
+        target=build/pkgs/${pkgname}/version_requirements.txt
+        if [ "${BOOTSTRAP_QUIET}" = "no" ]; then
+            echo "bootstrap:$LINENO: installing '"$target"'"
+        fi
+        echo "# Generated by SAGE_ROOT/bootstrap based on src/pyproject.toml; do not edit directly" > $target
+        sage-get-system-packages install-requires ${pkgname} >> $target
+    done
     for a in m4/sage_spkg_configures.m4 m4/sage_spkg_versions.m4 m4/sage_spkg_versions_toml.m4; do
         if [ "${BOOTSTRAP_QUIET}" = "no" ]; then
             echo "bootstrap:$LINENO: installing '"$a"'"
@@ -233,7 +242,18 @@ save () {
 	src/Pipfile \
 	src/pyproject.toml \
 	src/requirements.txt \
-	src/setup.cfg
+	src/setup.cfg \
+	build/pkgs/cypari/version_requirements.txt \
+	build/pkgs/cysignals/version_requirements.txt \
+	build/pkgs/cython/version_requirements.txt \
+	build/pkgs/gmpy2/version_requirements.txt \
+	build/pkgs/jupyter_core/version_requirements.txt \
+	build/pkgs/memory_allocator/version_requirements.txt \
+	build/pkgs/numpy/version_requirements.txt \
+	build/pkgs/pkgconfig/version_requirements.txt \
+	build/pkgs/pplpy/version_requirements.txt \
+	build/pkgs/setuptools/version_requirements.txt \
+	build/pkgs/wheel/version_requirements.txt
 
     # Update version
     echo "$NEWCONFVERSION" >$PKG/package-version.txt

build/bin/sage-get-system-packages

@@ -14,22 +14,25 @@ fi
 
 case "$SYSTEM" in
     install-requires)
-        # Collect version_requirements.txt (falling back to requirements.txt) and output it in the format
+        # Collect from src/pyproject.toml or from version_requirements.txt (falling back to requirements.txt) and output it in the format
         # needed by setup.cfg [options] version_requirements=
         SYSTEM_PACKAGES_FILE_NAMES="version_requirements.txt requirements.txt"
         STRIP_COMMENTS="sed s/#.*//;/^[[:space:]]*$/d;"
+        FROM_PYPROJECT_TOML=1
         COLLECT=
         ;;
     install-requires-toml)
-        # Collect version_requirements.txt (falling back to requirements.txt) and output it in the format
+        # Collect from src/pyproject.toml or from version_requirements.txt (falling back to requirements.txt) and output it in the format
         # needed by pyproject.toml [build-system] requires=
         SYSTEM_PACKAGES_FILE_NAMES="version_requirements.txt requirements.txt"
         STRIP_COMMENTS="sed s/#.*//;/^[[:space:]]*$/d;s/^/'/;s/$/',/;"
+        FROM_PYPROJECT_TOML=1
         COLLECT=
         ;;
     pip)
         SYSTEM_PACKAGES_FILE_NAMES="requirements.txt version_requirements.txt"
         STRIP_COMMENTS='sed s/#.*//;s/[[:space:]]//g;'
+        FROM_PYPROJECT_TOML=1
         COLLECT=echo
         ;;
     *)
@@ -42,11 +45,24 @@ case "$SYSTEM" in
         fi
         SYSTEM_PACKAGES_FILE_NAMES="distros/$SYSTEM.txt"
         STRIP_COMMENTS="sed s/#.*//;s/\${PYTHON_MINOR}/${PYTHON_MINOR}/g"
+        FROM_PYPROJECT_TOML=0
         COLLECT=echo
         ;;
 esac
-for PKG_BASE in $SPKGS; do
 
+for PKG_BASE in $SPKGS; do
+    if [ $FROM_PYPROJECT_TOML -eq 1 ]; then
+        if [ -f "$SAGE_ROOT/src/pyproject.toml" ]; then
+            # Extract from the "requires" block in src/pyproject.toml
+            # Packages are in the format "'sage-conf ~= 10.3b3',"
+            PACKAGE_INFO=$(sed -n '/requires *= *\[/,/^\]/s/^ *'\''\('$PKG_BASE'.*\)'\'',/\1/p' "$SAGE_ROOT/src/pyproject.toml")
+            if [ -n "$PACKAGE_INFO" ]; then
+                echo "$PACKAGE_INFO" | ${STRIP_COMMENTS}
+                continue
+            fi
+        fi
+    fi
+    
     case "$SYSTEM:$ENABLE_SYSTEM_SITE_PACKAGES" in
         install-requires*|pip*)
             # This is output for installation of packages into a Python environment.
@@ -73,12 +89,12 @@ for PKG_BASE in $SPKGS; do
     for NAME in $SYSTEM_PACKAGES_FILE_NAMES; do
         SYSTEM_PACKAGES_FILE="$SAGE_ROOT"/build/pkgs/$PKG_BASE/$NAME
         if [ -f $SYSTEM_PACKAGES_FILE ]; then
-           if [ -z "$COLLECT" ]; then
-              ${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE
-           else
-              $COLLECT $(${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE)
-           fi
-           break
+            if [ -z "$COLLECT" ]; then
+                ${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE
+            else
+                $COLLECT $(${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE)
+            fi
+            break
         fi
     done
 done

build/pkgs/cypari/version_requirements.txt

@@ -640,6 +640,8 @@ and upper bounds).  The constraints are in the format of the
 <https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html>`_
 or `setup.py
 <https://packaging.python.org/discussions/install-requires-vs-requirements/#id5>`_.
+An exception are build time dependencies of Sage library, which should instead
+be declared in the ``requires`` block of ``pyproject.toml``.
 
 Sage uses these version constraints for two purposes:
 

build/pkgs/cysignals/version_requirements.txt

@@ -0,0 +1,26 @@
+[build-system]
+# Minimum requirements for the build system to execute.
+requires = [
+    # 68.1.0 Promote pyproject.toml's [tool.setuptools] out of beta.
+    # 68.1.1 Fix editable install finder handling of nested packages
+    'setuptools >= 68.1.1',
+    # version constraint for macOS Big Sur support (see https://github.com/sagemath/sage/issues/31050)
+    'wheel >=0.36.2',
+    'cypari2 >=2.1.1',
+    'cysignals >=1.10.2',
+    # Exclude 3.0.3 because of https://github.com/cython/cython/issues/5748
+    'cython >=3.0, != 3.0.3, <4.0',
+    'gmpy2 ~=2.1.b999',
+    'jupyter_core >=4.6.3',
+    'memory_allocator',
+    'numpy >=1.19',
+    'pkgconfig',
+    # pplpy 0.8.4 and earlier do not declare dependencies correctly (see https://github.com/sagemath/sage/issues/30922)
+    'pplpy >=0.8.6',
+]
+build-backend = "setuptools.build_meta"
+
+[tool.conda-lock]
+platforms = [
+    'osx-64', 'linux-64', 'linux-aarch64', 'osx-arm64'
+]