Altivec/VSX support

[lu-zero]

Since llvm seems to have an almost viable intrinsics support now, is it possible to expose them in rust like the neon ones?

[hlandau]

The altivec intrinsics are already exposed (vec_perm, etc.) AFAICT.

Maybe rename this to "VSX support"?

[lu-zero]

No :) the first part is getting the intrinsics then I'll need to add an adaptation layer in stdsimd to make so they behave the same on le and be.

I'm still far from getting that done. (and the amount of instructions still not mapped is larger than it should)

(yes I ended up writing the support myself...)

[gnzlbg]

So things have changed a bit. I have a branch implementing it in std::arch{powerpc, powerpcle, powerpc64, powerpc64le} following the std::arch RFC (https://github.com/gnzlbg/stdsimd/tree/altivec). The roadmap approximately looks like this:

• get CI running with portable vector types, no intrinsics:
  • powerpc,
  • powerpcle,
  • powerpc64,
  • powerpc64le
• implement one intrinsic and get CI running on:
  • powerpc,
  • powerpcle,
  • powerpc64,
  • powerpc64le
• implement all Altivec and VSX architecture-specific vector types
  • 128x1 vector types
  • all others
• figure out the API to expose Altivec and how to expose Altivec and VSX intrinsics
• actually implement all intrinsics
• automatically verify the intrinsics against the spec
• write an RFC and stabilize

The main blocker is the lack of a quality PowerPC emulator with support for altivec and VSX (qemu is giving us problems here). There are alternatives (rust-lang/stdarch#176 (comment)) but they are outside our control which means that they might be unreliable over time.

The second main blocker is that the altivec and vsx intrinsics are implemented with compiler magic in C, using function overloading (a feature not available in neither C nor Rust). That can be worked around with traits, and we can use sealed traits to avoid exposing those via std::arch - that's the approach I've pursued in the branch - but it is unclear to me if that won't be too controversial to pass the RFC process.

The third main blocker is the lack of an intrinsics specification in machine readable form that could be used for automatic verification. The only document I've found is the OpenPower architecture ISA spec in PDF and the clang / gcc C++ headers. That is, the automatic verification tool is going to either have to parse a PDF, or C++ code. Both formats are not ideal. Maybe we could use rust-bindgen to generate Rust wrappers over the C++ code and parse the Rust code instead, but those headers use vector extensions of the corresponding compilers that rust-bindgen probably cannot handle right now.

In any case, closing this issue requires a non-trivial amount of work. At least a couple of months for somebody working full-time on it.

[lu-zero]

I interacted a bit with the llvm community to see if there is a simple way to extract the information we need from the intrinsics template file.

I hadn't get through it since it involves look at llvm-tblgen and play with getIntrinsicForGCCBuiltin to extract additional information from altivec.h.

In theory we could have OpenPower document the intrinsics the way we put them in rust and have a separate chapter of the Power ISA book for us, since we will need to repeat the little-endian adaptation for the instructions with endianness-bias we might not spare a large amount of time but might be another option.

I'm quite sure @edelsohn can help us regarding having the CI running and decide which direction we can go.

[gnzlbg]

In theory we could have OpenPower document the intrinsics the way we put them in rust and have a separate chapter of the Power ISA book for us,

Does this chapter exist for C already? If so, generating it in xml or similar formats (assuming it is currently in tex or similar) might be enough for us to parse it.

[autun12]

im guessing no one is working on this at the moment?

[lu-zero]

I am, hunting people regarding the documentation took a long time.

[autun12]

Okay so the documentation isn't done then?

[lu-zero]

The agreement is that once the code is done and documented, that documentation would be deemed as reference and have a chapter in the official openpower documentation.

The existing documentation for C is not really machine-parsable and does not provide all the information we need to automatically generate conformity tests.

[autun12]

ahhhh okay. Thanks for explaining

[lu-zero]

I'll send some new code during this weekend :)

[pnkfelix]

Discussed at T-compiler backlog bonanza.

This appears to be correctly categorized as a C-tracking-issue and its other metadata seems correct. Its just stalled due to lack of developer investment, and that's okay!

[clin1234]

Is this bounty still open for reward from BountySource?

[glaubitz]

Doesn't seem so. I can't find it on Bountysource anymore.

[edelsohn]

I don't know where @glaubitz looked. A bounty remains open for this

https://app.bountysource.com/issues/46345753-altivec-vsx-support

however, it's not clear that IBM Power provides the documentation in the easily consumable form that Rust wants.

[lu-zero]

I stopped writing the implementation since it w/out it be not accepted in stable, I do not know if the requirement is now relaxed though.

[workingjubilee]

I don't see why easily-consumable documentation would be necessary for stabilization if we had alternative means of verification.

[edelsohn]

@lu-zero Does this new version of the Power ISA as a JSON document help?
https://github.com/open-power-sdk/PowerISA/blob/main/ISA.json

[lu-zero]

It looks very promising!

[ghost]

@gnzlbg , are you still interested in completing this?

@edelsohn , the effort here seems higher than other PowerPC related bounties. Maybe increase the bounty (note: I'm not after this bounty, as I've not the relevant domain-knowledge to do it alone within a reasonable timeframe).

[lu-zero]

The json doesn't seem useful to generate tests, but if the requirements are relaxed and I can produce something along the lines of neon.spec it should be feasible to progress.

[ghost]

The json doesn't seem useful to generate tests

Is this the remaining task (e.g. implementation is completed, tests need to be provided as per requirements) ?

[lu-zero]

It work stalled since when it started I had been required to find a way to automatically generate tests from machine parsable information for an official source and sadly the openpower documentation did not provide the information in a consumable shape. From what I'm seeing it seems that requirement got relaxed, so I can spend some time in see if the code-generator we have can be reused.

[ghost]

@lu-zero , forgot one thing: it would possibly speed-up things if this ongoing work is within a public repo, thus any interested party can join the efforts.

[lu-zero]

I'll get a branch with some new commits to update the status of altivec and vsx this weekend. The codegenerator for neon would need a full overhaul to support altivec so we'll see :)

[ghost]

@edelsohn , maybe this can be done sooner by:

• Adding an independent(!) task/bounty, kind of:
  • automatically generate tests from machine parsable information

This is something that requires less domain-knowledge, and can be possibly executed faster by any interested dev.

[lu-zero]

One of the most annoying part right now is to write enough tests to validate the bindings and, that I could see, cannot be generated from the json at least easily.

I already opened an issue to ask to tag all the functions that are vsx and power8-only in it. Right now I'm using the json to keep track of what's missing using cargo-public-api and little by little add more intrinsics and makes sure the one present are visible (fixed that last weekend).

The polymorphic nature of the altivec intrinsics makes supporting them about 3-10 times longer to write since I have to wrap the llvm primitive to match each of the type combination, make a trait for each intrinsic, implement it for the supported type combination, test that everything works as intended regarding codegen and behavior.

[ghost]

@lu-zero , I wrote this #42743 (comment) just to try to get some assistance for you. Of course alternatively you could start the 2nd bounty yourself (if it is filed), kind of to justify the increased effort on your side.

[apiraino]

Visited during T-compiler cleanup of open tracking issues on Zulip.

Just to update the records, can someone involved in this work post the current status of this tracking issue?

Thanks

[lu-zero]

Most of it is done, IBM does not seem interested anymore in sponsoring work on this and bountysource went belly up so it even lower priority for me to add the remaining intrinsics...