Fortran fpm help_test invokes fpm

[awvwgk]

The help_test currently invokes fpm, this usually triggers a complete rebuild of both debug and release version unconditionally.

fpm/fpm/test/help_test/help_test.f90

Lines 21 to 46 in c485357

	! debug version
	'fpm run -- --version ',& ! verify fpm version being used
	'fpm run -- --help > fpm_scratch_help.txt',&
	'fpm run -- help new >> fpm_scratch_help.txt',&
	'fpm run -- help update >> fpm_scratch_help.txt',&
	'fpm run -- build --help >> fpm_scratch_help.txt',&
	'fpm run -- help run >> fpm_scratch_help.txt',&
	'fpm run -- help test >> fpm_scratch_help.txt',&
	'fpm run -- help runner >> fpm_scratch_help.txt',&
	'fpm run -- help install >> fpm_scratch_help.txt',&
	'fpm run -- help list >> fpm_scratch_help.txt',&
	'fpm run -- help help >> fpm_scratch_help.txt',&
	'fpm run -- --version >> fpm_scratch_help.txt',&
	! release version
	'fpm run --release -- --version ',& ! verify fpm version being used
	'fpm run --release -- --help > fpm_scratch_help3.txt',&
	'fpm run --release -- help new >> fpm_scratch_help3.txt',&
	'fpm run --release -- help update >> fpm_scratch_help3.txt',&
	'fpm run --release -- build --help >> fpm_scratch_help3.txt',&
	'fpm run --release -- help run >> fpm_scratch_help3.txt',&
	'fpm run --release -- help test >> fpm_scratch_help3.txt',&
	'fpm run --release -- help runner >> fpm_scratch_help3.txt',&
	'fpm run --release -- help install >> fpm_scratch_help3.txt',&
	'fpm run --release -- help list >> fpm_scratch_help3.txt',&
	'fpm run --release -- help help >> fpm_scratch_help3.txt',&
	'fpm run --release -- --version >> fpm_scratch_help3.txt',&

It might also invoke another version of fpm or completely fail if no fpm is in the PATH.

[urbanjost]

A lot has changed since it was written, including where a "run" only did a build test if the binary was missing, but it was my understandng that each test was run in an environment that was created on demand and included the fpm command. It is running the help on the latest version built by the fpm command. I have not looked recently at a build log but the last time I looked it displayed the expected version number and did not show a build of fpm itself in the log file. So it has been testing the latest version successfully in the manner expected. Need a better definition of exactly how the test environment is deployed that the tests run in, but what you describe sounds basically what would be desired if the test environment is set up the way I thought; so need some more details, as I believe those conditions are not supposed to occur for the test environment.

[awvwgk]

You can see a new build starting in the testing environment at https://github.com/fortran-lang/fpm/runs/1754249868?check_suite_focus=true#step:12:2331 in the test step.

Also a common case I encountered this was when testing a bootstrapped build with

fpm run -- test

[urbanjost]

It builds the latest version if required and runs the new copy to verify it gets the expected results. It is slower than expected but is doing the intended test. I believe there is an item open to add more actual fpm tests; but testing the actual executable in the CLI and help tests was intentional. I might have misunderstood the exact nature of the test environment? Is it overwriting something? I thought the test environment was created on the fly specifically to do testing like this. What exactly is damaged?

[LKedward]

The rebuild appears to be because the test invokes both the debug and --release configurations. In the CI, the --release version is not built prior to running the tests and hence it is built during execution of help_test.

I would say that the desired behaviour for tests is to test the current build configuration only. i.e. fpm test tests the debug configuration and fpm test --release only tests the --release configuration. The reason for this is that Haskell fpm allows other configurations as specified by --flag and in fact this is what is used in the CI.

The --release version is not directly built at all by the CI since the release version used for binaries is specified explicitly with --flag in order to get static executables:

fpm/.github/workflows/CI.yml

Line 36 in c485357

RELEASE_FLAGS: --flag --static --flag -g --flag -fbacktrace --flag -O3

Hence this test is not actually running at all for the binary release version specified in our CI.

This problem is related to #328 in terms of a test needing information about its context. A possible solution suggested by Brad is to provide environment variables for the build output directory, however this does rely on some specification of the output file structure which has thus far been avoided.

It is worth noting also that cli_test solves a similar problem by providing an alternate entry point within the test executable instead of invoking the main fpm executable..

[urbanjost]

I made the cli_test version before discussions about just what the test environment was. During development of the help_test I made inquiries about whether fpm could be called and at the time that apparently was considered OK. I was under the impression the test environment was a virtual platform created on demand for the tests and had the f-fpm command in it's path.
So exactly what is the test platform? Unit tests etc. are satisfactory for testing procedures and objects (I have used such tests before they had a name) but at some point if you are building an application you want to test the application, both test and production versions; especially since I do not even have several of the environments available that the tests run on. So what is the best way to call the latest version of the program fpm from a test? Even if not done for fpm itself an environment variable giving a path to a built version would seem reasonable. I

PS:
I am particularly surprised the test environment even has h-fpm in it; I have been using f-fpm for months on different platforms just bootstrapping it on machines with just gfortran and git.

[LKedward]

Yes you're right @urbanjost, we do need a way of testing the application end-to-end in addition to unit tests. As you're aware we have previously found bugs with the CLI this way. The issue for discussion is what is the best way to call the executable from a test. For the reasons I mentioned I don't think calling fpm run from within the test is the best option. (The problem isn't with the test environment.)

Since we don't have support for the required path environment variables currently (needs discussion/specification), do you think you would be able to test the same functionality by invoking the test executable like the cli_test?

[everythingfunctional]

I think that fpm needs a facility for integration/end-2-end testing.

1. Tests are specified for a given executable.
2. Tests are run in an isolated directory
3. It must be possible to specify
   a. the runner (i.e. equivalent to fpm run --runner)
   b. command line arguments (i.e. equivalent to fpm run --args)
   c. Any inputs which should be copied into the isolated directory
   d. A way (optionally) of checking the "screen" outputs
   e. A way (optionally) of checking output files

Given that we would like to at least encourage these test be portable (i.e. work for any environment that fpm supports), d and e imply some sort of fpm specific scripting language. Or maybe fpm could carry around a Bash implementation to run these scripts?

What do you guys think?

[awvwgk]

That would be a great addition, with 1, 2, 3a-c the important functionality. 3d and 3e could possibly be implemented outside of fpm by a package used as dev-dependency or by a combination of --runner and --args for a test. A related issue on this is #179.