In plain words
Tensor instructions often accept data only when it is stored and arranged in a particular way. Arknife is a compiler that records those rules, checks them across the whole computation, and inserts conversions when rules clash.
It then chooses data layouts that let instructions work together. On tested x86 and Hexagon workloads, its overall runtime was lower than a competing retargetable compiler, MISAAL; results varied by workload.
The research
Specialized hardware can run tensor computations quickly only when data is arranged in ways each instruction accepts.
A compiler that handles those arrangements across different processors could make kernels easier to retarget while avoiding unnecessary data movement; Arknife tests whether one shared layout model can do both.
Arknife makes operand layouts part of instruction semantics, then reconciles them across a tensor program so one compiler can target different hardware.
For x86 and Hexagon, Arknife and MISAAL ran the same nine-workload sets; Arknife showed geometric-mean runtime speedups of 5.08x and 4.53x, multiplicative averages across each set. On eight TF32 matrix sizes on an A100, Arknife reached 97% of cuBLAS performance, measured as cuBLAS latency divided by Arknife latency.
On Hexagon Add, Arknife also used fewer dynamic instructions to move or rearrange data than Hydride. That diagnostic fits the compiler’s goal of coordinating layouts globally, alongside the runtime results on the tested backends.
The runtime comparisons cover the evaluated workloads on x86, Hexagon and A100, and the layout-handling diagnostic covers one Hexagon Add setting. Individual workload results vary, and the layout model excludes ragged domains and data-dependent access relations.
What the evidence shows
- Make layout part of instructions
Arknife is a retargetable compiler for tensor kernels that treats each instruction’s operand layout as part of its requirements. Arithmetic equivalence alone cannot ensure that data is placed and arranged legally for an instruction.
The compiler combines these requirements across a program, adding target-provided conversions when they conflict and choosing layouts that keep instruction boundaries legal.
- Runtime gains on two targets
The runtime comparison with MISAAL used nine workloads on each of x86 and Hexagon. Arknife’s geometric-mean speedup, a multiplicative average across each workload set, was 5.08x on x86 and 4.53x on Hexagon.
The result supports a runtime advantage over this retargetable-compiler baseline on the tested sets, while individual workload speedups varied.
- Near cuBLAS on tested matrices
On an NVIDIA A100, Arknife reached 97% of cuBLAS performance across eight tested TF32 matrix sizes. The measure divides cuBLAS latency by Arknife latency, so it compares runtime for those matrix-multiplication cases.
This result shows performance close to the vendor library on that GPU task, rather than a result across all GPU kernels.
- Less layout work in one case
The layout-handling diagnostic compared dynamic Move and Convert instructions, which transfer or rearrange data, for Hexagon quantized Add with M=N=1024. Hydride executed 85.8 times as many of these instructions as Arknife in that setting.
This identifies a concrete difference in layout work, but the count itself is not a measure of total runtime.
- Optimization effects vary by target
A control disabling repair-graph optimization found that enabling it reduced dynamic layout handling by 2.18x for Hexagon Add and 1.45x for Hexagon Mul. The same pass left the measured x86 pipelines unchanged.
Its contribution to the diagnostic therefore depends on the target and workloads tested; the control does not establish its effect across every result in the runtime evaluation.
Deep
- Original paper authors
- Likai Ma, Renze Chen, Tianqi Li, Yuhao Luo, Youwei Xiao
- Submitted to arXiv
- License
- arXiv non-exclusive