Skip to content
This repository was archived by the owner on Feb 13, 2025. It is now read-only.

implement switching using the Stackman submodule #230

Merged
merged 25 commits into from
Jul 16, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
2260bcd
implement switching using the Stackman submodule
kristjanvalur Jan 3, 2020
5537847
update stackman for correct ARM code
kristjanvalur Jan 6, 2020
e4e9cc8
change submodule to https
kristjanvalur Jan 12, 2020
7e82df2
Merge branch 'stackman' of ssh://github.com/kristjanvalur/stackless i…
kristjanvalur Jan 12, 2020
e312dc8
Merge branch 'main-slp' into stackman
Jun 22, 2021
8f36251
Use stackman from https://github.com/stackless-dev/stackman.git
Jun 23, 2021
8bac0cd
Update submodule stackman to the latest version
Jun 23, 2021
d7704ee
Fix #include statements
Jun 23, 2021
af23038
prepare for a merge with main-slp
Jul 9, 2021
504edea
Merge branch 'main-slp' into stackman
Jul 9, 2021
7d85c37
Readd the stackman submodule
Jul 9, 2021
e981179
Adapt the patch to upstream changes.
Jul 9, 2021
7e1bb4a
Change "make clean" to not remove static library files from stackman.
Jul 9, 2021
20657c4
Merge branch 'main-slp' into stackman
Jul 11, 2021
55e8946
Add configure options --with-stackman=dir and --without-stackman
Jul 11, 2021
7772ab7
Update create_source_archive.sh to include Stackman.
Jul 11, 2021
2f8a7e2
Merge branch 'main-slp' into stackman
Jul 11, 2021
8f6a3df
Detect Stackman abiname and link the appropriate libstackman.a
Jul 11, 2021
a20298b
remove superfluous tabs
Jul 11, 2021
553ab3d
Merge remote-tracking branch 'origin/main-slp' into stackman
Jul 13, 2021
c5bea11
Fix the usage of SLP_SAVE_STATE.
Jul 14, 2021
ab0c29a
Integrate Stackman into the windows build infrastructure.
Jul 14, 2021
bb897f8
Add documentation and build instructions.
Jul 14, 2021
0022882
merge main-slp into branch stackman
Jul 15, 2021
ac479bd
Add support for ARM and ARM64
Jul 15, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "Stackless/platf/stackman"]
path = Stackless/stackman
url = https://github.com/stackless-dev/stackman.git
24 changes: 16 additions & 8 deletions Makefile.pre.in
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ CFLAGS_ALIASING=@CFLAGS_ALIASING@

# Special C flags for slp_transfer.c
SLPFLAGS= @SLPFLAGS@
SLP_STACKMAN_LIB= @SLP_STACKMAN_LIB@
SLP_STACKMAN_OBJS= @SLP_STACKMAN_OBJS@

# Machine-dependent subdirectories
MACHDEP= @MACHDEP@
Expand Down Expand Up @@ -348,6 +350,7 @@ PYTHON_OBJS= \
Stackless/module/taskletobject.o \
Stackless/pickling/prickelpit.o \
Stackless/pickling/safe_pickle.o \
$(SLP_STACKMAN_OBJS) \
Python/codecs.o \
Python/compile.o \
Python/coreconfig.o \
Expand Down Expand Up @@ -724,6 +727,17 @@ $(srcdir)/Include/cpython/slp_exttype.h: $(srcdir)/Objects/typeobject.c
$(PYTHON_FOR_REGEN) $(srcdir)/Stackless/Tools/extract_slp_info.py
$(UPDATE_FILE) $(srcdir)/Include/cpython/slp_exttype.h $(srcdir)/Include/cpython/slp_exttype.h.new

$(SLP_STACKMAN_OBJS): $(SLP_STACKMAN_LIB)
cp $(SLP_STACKMAN_LIB) Stackless/platf/libstackman.a
cd Stackless/platf && $(AR) x libstackman.a && rm libstackman.a

Stackless/platf/slp_transfer.o: $(srcdir)/Stackless/platf/slp_transfer.c
$(CC) -c $(PY_CORE_CFLAGS) $(SLPFLAGS) -o $@ $(srcdir)/Stackless/platf/slp_transfer.c

.PHONY: teststackless
teststackless: @DEF_MAKE_RULE@ platform
$(TESTPYTHON) -E $(srcdir)/Stackless/unittests/runAll.py

############################################################################
# Importlib

Expand Down Expand Up @@ -990,9 +1004,6 @@ regen-typeslots:
$(srcdir)/Objects/typeslots.inc.new
$(UPDATE_FILE) $(srcdir)/Objects/typeslots.inc $(srcdir)/Objects/typeslots.inc.new

Stackless/platf/slp_transfer.o: $(srcdir)/Stackless/platf/slp_transfer.c
$(CC) -c $(PY_CORE_CFLAGS) $(SLPFLAGS) -o $@ $(srcdir)/Stackless/platf/slp_transfer.c

############################################################################
# Header files

Expand Down Expand Up @@ -1150,7 +1161,7 @@ TESTPYTHON= $(RUNSHARED) ./$(BUILDPYTHON) $(TESTPYTHONOPTS)
TESTRUNNER= $(TESTPYTHON) $(srcdir)/Tools/scripts/run_tests.py
TESTTIMEOUT= 1200

.PHONY: test testall testuniversal buildbottest pythoninfo teststackless
.PHONY: test testall testuniversal buildbottest pythoninfo

# Run a basic set of regression tests.
# This excludes some tests that are particularly resource-intensive.
Expand All @@ -1171,9 +1182,6 @@ testall: @DEF_MAKE_RULE@ platform
-$(TESTRUNNER) -u all $(TESTOPTS)
$(TESTRUNNER) -u all $(TESTOPTS)

teststackless: @DEF_MAKE_RULE@ platform
$(TESTPYTHON) -E $(srcdir)/Stackless/unittests/runAll.py

# Run the test suite for both architectures in a Universal build on OSX.
# Must be run on an Intel box.
testuniversal: @DEF_MAKE_RULE@ platform
Expand Down Expand Up @@ -1791,7 +1799,7 @@ docclean:
-rm -rf Doc/tools/sphinx Doc/tools/pygments Doc/tools/docutils

clean: pycremoval
find . -name '*.[oa]' -exec rm -f {} ';'
find . -path '*Stackless/stackman/*.a' -prune -o -name '*.[oa]' -exec rm -f {} ';'
find . -name '*.s[ol]' -exec rm -f {} ';'
find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';'
find build -name 'fficonfig.h' -exec rm -f {} ';' || true
Expand Down
5 changes: 5 additions & 0 deletions PCbuild/python.props
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@

<!-- Full path of the resulting python.exe binary -->
<PythonExe Condition="'$(PythonExe)' == ''">$(BuildPath)python$(PyDebugExt).exe</PythonExe>

<!-- Stackless stackman directory. Set this property to 'no', to build without Stackman -->
<stackmanDir Condition="'$(stackmanDir)' == ''">$(PySourcePath)Stackless\stackman\</stackmanDir>
<stackmanDir Condition="'$(stackmanDir)' != 'no' And !HasTrailingSlash($(stackmanDir))">$(stackmanDir)\</stackmanDir>
<stackmanLib Condition="'$(stackmanDir)' != 'no'">stackman.lib;</stackmanLib>
</PropertyGroup>

<PropertyGroup Condition="'$(Platform)'=='ARM'" Label="ArmConfiguration">
Expand Down
33 changes: 15 additions & 18 deletions PCbuild/pythoncore.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,15 @@
<AdditionalIncludeDirectories Condition="$(IncludeExternals)">$(zlibDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
<PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="$(IncludeExternals)">_Py_HAVE_ZLIB;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(stackmanDir)' == 'no'">SLP_NO_STACKMAN;%(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
<Link>
<AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
<AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;$(stackmanLib)%(AdditionalDependencies)</AdditionalDependencies>
<AdditionalLibraryDirectories Condition="'$(stackmanLib)'!='' And '$(Platform)'=='x64'">$(stackmanDir)lib\win_x64;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<AdditionalLibraryDirectories Condition="'$(stackmanLib)'!='' And '$(Platform)'=='Win32'">$(stackmanDir)lib\win_x86;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<AdditionalLibraryDirectories Condition="'$(stackmanLib)'!='' And '$(Platform)'=='ARM'">$(stackmanDir)lib\win_arm;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<AdditionalLibraryDirectories Condition="'$(stackmanLib)'!='' And '$(Platform)'=='ARM64'">$(stackmanDir)lib\win_arm64;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<ImageHasSafeExceptionHandlers Condition="'$(stackmanLib)'!='' And '$(Platform)'=='Win32'">false</ImageHasSafeExceptionHandlers>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kristjanvalur : Currently this is required on x86. Otherwise I get the error message:
stackman.lib(switch_x86_msvc.obj) : error LNK2026: module unsafe for SAFESEH image. [C:\build\stackless38\PCbuild\pythoncore.vcxproj] Creating library C:\build\stackless38\PCbuild\win32\python38_d.lib and object C:\build\stackless38\PCbuild\win32\python38_d.exp C:\build\stackless38\PCbuild\win32\python38_d.dll : fatal error LNK1281: Unable to generate SAFESEH image.[C:\build\stackless38\PCbuild\pythoncore.vcxproj]

</Link>
</ItemDefinitionGroup>
<ItemGroup>
Expand Down Expand Up @@ -454,7 +460,9 @@
<ClCompile Include="..\Python\thread.c" />
<ClCompile Include="..\Python\traceback.c" />
<ClCompile Include="..\Stackless\core\cframeobject.c" />
<ClCompile Include="..\Stackless\platf\slp_transfer.c" />
<ClCompile Include="..\Stackless\platf\slp_transfer.c">
<AdditionalIncludeDirectories Condition="'$(stackmanDir)'!='no'">$(stackmanDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
</ClCompile>
<ClCompile Include="..\Stackless\core\stacklesseval.c" />
<ClCompile Include="..\Stackless\core\stackless_util.c" />
<ClCompile Include="..\Stackless\module\channelobject.c" />
Expand Down Expand Up @@ -486,23 +494,12 @@
</ItemGroup>
<ItemGroup>
<CustomBuild Include="..\Stackless\platf\switch_x64_masm.asm">
<ExcludedFromBuild Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">true</ExcludedFromBuild>
<ExcludedFromBuild Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">true</ExcludedFromBuild>
<ExcludedFromBuild Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">true</ExcludedFromBuild>
<ExcludedFromBuild Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">true</ExcludedFromBuild>
<ExcludedFromBuild Condition="'$(Platform)'!='x64'">true</ExcludedFromBuild>
<ExcludedFromBuild Condition="'$(Platform)'=='x64' And '$(stackmanDir)'!='no'">true</ExcludedFromBuild>
<FileType>Document</FileType>
<Command Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">ml64 /nologo /c /Zi /Fo "$(IntDir)%(Filename) .obj" "%(FullPath)"</Command>
<Message Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">Running ml64</Message>
<Outputs Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">$(IntDir)%(Filename) .obj;%(Outputs)</Outputs>
<Command Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">ml64 /nologo /c /Zi /Fo "$(IntDir)%(Filename) .obj" "%(FullPath)"</Command>
<Message Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">Running ml64</Message>
<Outputs Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">$(IntDir)%(Filename) .obj;%(Outputs)</Outputs>
<Command Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">ml64 /nologo /c /Zi /Fo "$(IntDir)%(Filename) .obj" "%(FullPath)"</Command>
<Message Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">Running ml64</Message>
<Outputs Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">$(IntDir)%(Filename) .obj;%(Outputs)</Outputs>
<Command Condition="'$(Configuration)|$(Platform)'=='Release|x64'">ml64 /nologo /c /Zi /Fo "$(IntDir)%(Filename) .obj" "%(FullPath)"</Command>
<Message Condition="'$(Configuration)|$(Platform)'=='Release|x64'">Running ml64</Message>
<Outputs Condition="'$(Configuration)|$(Platform)'=='Release|x64'">$(IntDir)%(Filename) .obj;%(Outputs)</Outputs>
<Command Condition="'$(Platform)'=='x64'">ml64 /nologo /c /Zi /Fo "$(IntDir)%(Filename) .obj" "%(FullPath)"</Command>
<Message Condition="'$(Platform)'=='x64'">Running ml64</Message>
<Outputs Condition="'$(Platform)'=='x64'">$(IntDir)%(Filename) .obj;%(Outputs)</Outputs>
</CustomBuild>
<None Include="..\Stackless\readme.txt" />
</ItemGroup>
Expand Down
3 changes: 2 additions & 1 deletion Stackless/Tools/create_source_archive.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ if [ $# != 3 ] ; then
exit 1
fi

: ${excludes:=.gitignore .git* .hg* .bzrignore .mention-bot .travis.yml .readthedocs.yml .azure-pipelines}
: ${excludes:=.gitignore .git* .mention-bot .travis.yml .readthedocs.yml .azure-pipelines Stackless/stackman/.git* }

git_python_dir="$1" ; shift
if [ "x$1" = "x-v" ] ; then
Expand Down Expand Up @@ -42,6 +42,7 @@ mkdir "$tmpdir"
cd "$git_python_dir"
git archive --format=tar --prefix="${srcdir}/" "$tag" | \
( cd "${tmpdir}" && tar xf - )
git submodule foreach --recursive "git archive --format=tar --prefix=${srcdir}/\$path/ HEAD | ( cd \"${tmpdir}\" && tar xf - )"
cd "${tmpdir}"
( cd "${srcdir}" && set +f && rm -rf $excludes )

Expand Down
5 changes: 5 additions & 0 deletions Stackless/changelog.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ What's New in Stackless 3.X.X?

*Release date: 20XX-XX-XX*

- https://github.com/stackless-dev/stackless/issues/278
Stackless now uses Stackman https://github.com/stackless-dev/stackman for
stack switching / hard tasklet switching. The legacy stack switching code
is still available. See 'readme.txt' for details.

- https://github.com/stackless-dev/stackless/issues/254
The Stackless version is now "3.8".

Expand Down
11 changes: 11 additions & 0 deletions Stackless/platf/slp_transfer.c
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,16 @@
*/
#define SLP_EVAL /* enable code generation in the included header */

#ifndef SLP_NO_STACKMAN /* defined by configure --without-stackman */
/* First, see if stackman an implementation without external
* assembler, use that if possible
*/
#define STACKMAN_OPTIONAL
#include "stackman/stackman.h"
#endif /* #ifndef SLP_NO_STACKMAN */
#if defined(STACKMAN_PLATFORM) && !defined(STACKMAN_EXTERNAL_ASM)
#include "switch_stackman.h"
#else /* use traditional stackless switching */
#if defined(MS_WIN32) && !defined(MS_WIN64) && defined(_M_IX86)
#include "switch_x86_msvc.h" /* MS Visual Studio on X86 */
#elif defined(MS_WIN64) && defined(_M_X64)
Expand Down Expand Up @@ -74,6 +84,7 @@ Please provide an implementation of the switch_XXX.h
or disable the STACKLESS flag.
**********
#endif
#endif /* use traditional stackless switching */

/* default definitions if not defined in above files */

Expand Down
67 changes: 67 additions & 0 deletions Stackless/platf/switch_stackman.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/*
* this is the internal transfer function, using
* the stackman platform library.
* We create a wrapper callback that employs the existing
* stack macros.
* At some later point in time, the callback could be
* written directly.
*
*/

#define SLP_STACK_REFPLUS 1

#ifdef SLP_EVAL
#define SLP_STACK_MAGIC 0


/* need a special function arount SLP_SAVE_STATE() because
* the macro has a built-in return of 0 or -1. Must catch
* that.
*/
static int slp_stackman_cb_save(void *sp, intptr_t *pdiff)
{
intptr_t diff;
/* first argument must be a pointer to a "stack word", not a void* */
SLP_SAVE_STATE(((intptr_t *)sp), diff);
*pdiff = diff;
return 1;
}

static void *slp_stackman_cb(void *_ctxt, int opcode, void *sp)
{
int *error = (int*)_ctxt;
intptr_t stsizediff;
if (opcode == STACKMAN_OP_SAVE)
{
int ret = slp_stackman_cb_save(sp, &stsizediff);
if (ret == 1) {
/* regular switch */
return (void*)((char*)sp + stsizediff);
}
if (ret == -1)
{
*error = -1;
}
/* error or save only, no change in sp */
return sp;
}
else
{
if (*error != -1)
SLP_RESTORE_STATE();
return sp;
}
}

static int
slp_switch(void)
{
/* this can be on the stack, because if there isn't a switch
* then it is intact. (error or save only) */
int error = 0;
stackman_switch(&slp_stackman_cb, &error);
return error;
}

#include "stackman/stackman_impl.h"
#endif
27 changes: 20 additions & 7 deletions Stackless/readme.txt
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,26 @@ It is the ultimate truth. Read it at https://github.com/stackless-dev/stackless
Building Stackless
------------------

Just follow the build instructions for regular Python. If you define
the C-preprocessor symbol STACKLESS_OFF, you get a Python interpreter
without Stackless. It should behave exactly like the corresponding
version of regular Python. Any difference constitutes a bug.
(Windows 64bit only: if you define STACKLESS_OFF, you also need
to clear the content of the file Stackless\platf\switch_x64_masm.asm:
c:\> echo END > Stackless\platf\switch_x64_masm.asm )
Just follow the build instructions for regular C-Python.

To more easily support new platforms and CPU architectures, Stackless Python
now commonly uses Stackman (https://github.com/stackless-dev/stackman)
for stack switching / hard switching between tasklets. Stackman is a
lightweight library for stack manipulation. The Stackless source archive
already contains a copy of Stackman, which is normally used automatically.
However, the legacy Stackless switching code can still be used without
restrictions. To do this, Stackless must be configured with
`configure --without-stackman` or built under Windows with
`build.bat "/p:stackmanDir=no"`.

It is also possible to use your own version of Stackman instead of the copy
bundled with Stackless. To do this configure Stackless with
`configure --with-stackman=/path/to/your/stackman` or built under Windows
with `build.bat "/p:stackmanDir=X:\path\to\your\stackman"`.

If you define the C-preprocessor symbol STACKLESS_OFF in Include/stackless.h,
you get a Python interpreter without Stackless. It should behave exactly like
the corresponding version of regular Python. Any difference constitutes a bug.

Contributing
------------
Expand Down
1 change: 1 addition & 0 deletions Stackless/stackman
Submodule stackman added at c572d4
Loading