Cross compile with custom engine.

[fourscience]

This is a placeholder issue to track hover build and cross compile changes discussed in #128.

Proposal:

List of steps.

1. Add custom toolchain and target support.
2. Add support for build flags. (example: glfw gles2 and wayland tags)
3. Add local engine support, therefore providing own artifacts for the embedder.
4. Add sanity and error checking.
5. Refactor.
6. Probably add support for toolkits other then glfw or direct usage of FB, so it can run without a heavy x11 or wayland. (Note: This should be a different change, it's mentioned here to ensure it's known as it affects the build command. An idea would be to add a --driver flags, which tells the build which manager should be used (sdl, glfw, directfb or wayland etc..)

Proposed arguments

- target <arch><sub>-<vendor>-<sys>-<abi>

Description: Describes the target as a triplet.
Format: The triple has the general format <arch><sub>-<vendor>-<sys>-<abi> where:

• arch = x86_64, i386, arm, thumb, mips, etc.
• sub = for ex. on ARM: v5, v6m, v7a, v7m, etc.
• vendor = pc, apple, nvidia, ibm, etc.
• sys = none, linux, win32, darwin, cuda, etc.
• abi = eabi, gnu, android, macho, elf, etc.

Use case: Whenever the target triple is given it can describe the target system and the compiler/linker setup needed. Therefor adding ability to run it on any system.
(Disclaimer: It's up to the developer to make sure he is able to compile to the target and that target is a valid system for flutter to use. Eg: libdl, libpthread, librt, libm must be present for flutter engine.)

Example values: armv7-linux-gnueabi, armv7-linux-gnueabihf, x86_64-linux-eabi
Behavior: By default, if no other options given, it will translate the value given to GOOS, GOARCH, GOARM and validate it (not all permutations are valid), if the base set or host os does not support it should warn the user to specify the docker flag.
If the toolchain arguments is given, this will specify the gcc/g++ (might be clang or other compilers) to use.
An example would be:

• /path/to/toolchain/bin/gcc is used when toolchain is provided but no target is set.
• /path/to/toolchain/bin/${target}-gcc is used when toolchain and target is provided.

- toolchain path/to/toolchain

Description: Describes the CC used by go.
Format: Absolute path.
Use case: Adding custom toolchains for custom target systems.

Example values: /opt/gcc-linaro-4.9.4-2017.01-x86_64_arm-linux-gnueabi
Behavior: By default it does set CC to a gcc/g++/clang etc found in path + /bin/gcc, if target is specified it would suffix the target triple to the gcc binary, as it would be path + /bin/arm-linux-gnueabi-gcc, if the compiler is presented in the environment it might be skipped.
(Disclaimer: if the toolchain path is set, and it is desired to use a sysroot with it, it's up to the developer to either compile the toolchain with support for it, or set the compile flags with-sysroot)

- cflags/cxxflags

Description: Adds CFLAGS/CXXFLAGS to the compilation steps
Format: Any supported flags by the compiler.
Use case: It might be needed if the target needs custom flags, an example would be nacl which requires some flags to use, an other would be the runtime set or std set to use.

Example values: --sysroot=/path/to/sysroot --std=c90
Behavior: By default this should not be used, but custom targets might require this.

- ldflags

Description: Adds LDFLAGS to the linker steps
Format: Any supported flags by the linker.
Use case: It might be needed if the target needs custom flags, as an example setting a custom rpath, or for other use cases where the system would have example glfw already installed and the user would rather use that then the compiled one.

Example values: -rpath,/lib/
Behavior: By default this should not be used, but custom targets might require this.

- local-engine-path /engine/src

Description: Describes the path for the local engine setup for use.
Format: Absolute path.
Use case: When we want to target a custom target, it requires the flutter_engine to be built for that target as well. Therefore it can be linked to the binary.
Example values: /engine/src ${FLUTTER_ROOT}/../engine/src
Behavior: It would use the artifact from this directory, it comes with a pair arguments -local-engine to narrow down which build the user wants, since the flutter tool requires the engine built for the host (host_debug, host_debug_unopt, host_release, host_release_jit) to be present for bundling. It should point only to the engine src, by default as flutter/engine states it's a peer directory to flutter so we could assume that. Otherwise hover might have a new command to support the whole local engine setup. I'll write about it later in the discussion.

- local-engine host_debug_unopt

Description: Describes which artifact directory should be used.
Format: Sub path name for the out directory.
Use case: When we want to target a custom target, it requires the flutter_engine to be built for that target as well. Therefore it can be linked to the binary.
Example values: host_debug_unopt, linux_release_jit
Behavior: It should specify which artifact directory will be used, it contains all the necessary files for the embedder to work. Including the engine library, icu and headers.

- tags gles2

Description: As far as we depend on glfw at the moment it's not mandatory to have this. It would allow to specify build tags for later use cases.
Format: Space separated strings.
Use case: An example use case is when we would like to target a system with specific requirements, like it should use OpenGL ES 2, and use Wayland. This is for now a specific mention for glfw. SDL does this at runtime. Custom drivers might need it or not.
Example values: gles3 wayland
Behavior: It propagates down build tags for dependencies and for our own ones.

- output /path/to/output

Description: A path for the output binary and assets to land in.
Format: Absolute or relative path.
Use case: Since we might have different targets, we might need a different output directory. And don't want to pollute the working dir.
Example values: ../custom-output
Behavior: It should create the given folder and copy the necessary assets (icu, engine) to it, and build the flutter bundle to this folder. This folder should be used as a cache, if it already contains the engine or icu, it matches the artifact directory given then we should not call any copy or recompile step. Recompile steps might be involved when the local engine changes.

Proposed sanity checks.

• We should make sure that only supported variant are passed and let the user know about the miss-configuration. An example would be requesting an host_debug engine, and giving a target triple which does not reflect the host (Running on linux, and using mingw).
• We should make sure that the requested engine exists in the engine path. And let the user know if it's not.
• We should make sure all arguments involving path exists.
• We should make sure that the local engine matches the flutter tool, so it would run with the built flutter assets. Flutter tool has an engine version described in bin/internal/engine.version.
• We should notify the user if he is missing any dependencies (example: for glfw x11 packages)

Proposed refactor on high level.

• We should make sure to have an option to use multiply flutter tools at the same time, therefor i would propose to move all external command calls to it's own.
• We should make sure to separate the actual build steps and arguments parsing, allowing further arguments to added without touching the actual build function. As there might be arguments which does not require any build steps, more like changing environment variables for the current session etc.
• We should make sure that all the cross compile steps are separated from the main build. As one might extend on the other it should not be taken as a single step.
• We should make sure these are callable within code and not just CLI commands.

Other notes:

• Checking out the flutter engine might be a hassle for newcomers, we could add this step for them,
  we could use depot tools, and flutter tool to create an unmanaged local engine, with the revision specified by the tool, making sure about version parity.

I'll extend this in the following days and rewrite some parts, so this is a subject to change by itself and by other inputs from you.

[fourscience]

@jld3103 Can you please make sure that this is labeled correctly?
And could you please do a quick review? I'll revisit this after i have some free time.

[provokateurin]

I'd like to know more about how docker is involved in cross-compiling and how the current targets on desktop are defined then. Do you also plan to make targets like RPI "standard" (so that you don't have to specify the arguments every time)? Also adding something like a "profiles" config file for custom defined targets would be good.

Moving from the current hover structure to the one you're proposing is a lot of effort, but it will be wortwhile.

[fourscience]

Lets separate these for a moment, i'll call it "hover compile".

So running hover build, without any arguments is a sure host compilation requiring no cross compile steps, but it will go trough the hover compile, so we can support local engines.
Calling "hover build linux/arm" would load the "linux/arm" preset, a good list of those are fyne-cross, as they are simple and need only small adjustments (like linux/arm should be split, to armhf and armel). Which would then call hover compile with the preset containing all the above arguments defined.

So flow wise:

• run hover build as normal

• if first argument (OS atm) is given, that would be a preset.
  • if that preset differs from the host we would do the docker step
    * within docker we call hover compile with the preset arguments
  • else we build on the host machine with preset
• else if target is provided and arguments fulfill a hover compile
  • then we call hover compile on host
• else build default host
  Sorry for the style, its md. :)

We can probably create a "build config / presets" with predefined setups, so the users can easily target what they want, and decide if they want to use their host machine or docker to do the build.
That would involve presets for osx, windows, rpi, linux in general and many others later on or custom ones. See the list below for more examples.
As for the docker part goes, it would not involve go-cross, and instead we would provide a base image, and run hover within the docker container :)

by a preset list we could go for something like fyne:

darwin/amd64
darwin/386
linux/amd64
linux/386
linux/arm
linux/armhf
linux/arm64
windows/amd64
windows/386
android/amd64
android/386
android/arm
android/arm64

And could add wasm with more effort to that list, or nacl, tizen etc :)

So an overview:

• Create a base docker image, just like fyne did
• Based on the selected preset, install the compiler and dependencies in an intermediate build container
• Within docker, invoke hover compile (hover compile --target thiscomesfrompreset --cflags ... --ldflags ... etc)

Then hover compile could be used with docker (and for CI) and for dev on host machine as well without docker.

So command wise:
hover build presetName would invoke the docker build only if the host is not as the same as the target, and docker build would invoke hover compile :) (but of course inside a container)

I hope it makes sense.

[provokateurin]

Regarding the docker base image @GeertJohan created an PR for a new docker approach that uses a single image that we define

[provokateurin]

Also I created a PR for AOT that currently only works on linux, but would be needed for your changes I guess.

[fourscience]

Also I created a PR for AOT that currently only works on linux, but would be needed for your changes I guess.

Nice! Ill check it.

Regarding the docker base image @GeertJohan created an PR for a new docker approach that uses a single image that we define

Nice! ill add some comments regarding deps so it could be used by this!

[provokateurin]

Also we are currently using https://github.com/flutter-rs/engine-builds as libflutter_engine sources for AOT (and also could use them for JIT). Adding our predefined targets to the automatic builds would be nice for other users, because instead of compiling the engine themselves they could download the prebuilt engines.

[fourscience]

Also we are currently using https://github.com/flutter-rs/engine-builds as libflutter_engine sources for AOT (and also could use them for JIT). Adding our predefined targets to the automatic builds would be nice for other users, because instead of compiling the engine themselves they could download the prebuilt engines.

For the default set thats fine, i dont want to change the setup for that, just adding the option for users to use local engine :)

non of these arguments should be required, regarding prebuilts, these should be ok.
If the user does not define a local engine path and name then these artifacts should be used.

The flow somewhat similiar to flutter build bundle.

[provokateurin]

@fourscience any news? Since you last wrote a lot of things have changed in hover. We finally have better Docker cross-compiling support. I'd recommend to make yourself familiar with the new hover features to further discuss the progress on this issue.

[fourscience]

@fourscience any news? Since you last wrote a lot of things have changed in hover. We finally have better Docker cross-compiling support. I'd recommend to make yourself familiar with the new hover features to further discuss the progress on this issue.

Hello, yes and no :)
Still making progress in my free time, i saw the changes and actually following up on them, im able to cross compile with a followig setup:

• compiled clang to armv7 eabi (sf)
• compiled binutils to armv7 eabi
• compiled flutter engine for armv7 eabi

same works for darwin

using the cc above with the engine for go
works fine, what i was struggeling with is glfw go has somewhat a restricted protocol setup :)

ill post an update when its somewhat stable
for the following targets:

• rpi
• webos
• tizen

Im still on quite far of having a feature ready setup, at the moment what is left to do besides the basic setup is a more proper configuration for linker, compiler, assembler steps and a pr against go glfw :)

Feel free to close this issue if needed as the changes takes place in hover anyway.

A quick note on this is more of an advanced use case simplified down, as this can be achived by hand, so its somewhat targets developers who intend to use go and flutter on devices which are not common targets like the rpi or smart devices.

I hope when its done it can be used within the docker setup as well with an ease.

I can share a gist if needed with the changes i made for an early preview if anyone interested in it.

[provokateurin]

Could you upload your code on a new branch in your hover fork? Then we could see the progress as you work on it. And this issue should be kept open.

[kero990]

I am not a specific developer, so I am not familiar with the specific implementation methods.

But now Flutter has some third-party arm implementations

For example, this

https://github.com/sony/flutter-elinux

What I want to ask is, is it possible to combine this project and complete the construction of arm/arm64

Or refer to its implementation direction.

This project should appear after all the above conclusions