SIMD Intrinsics: Why Are They Sometimes Slower?

• ⚠️ Hand-written SIMD code can be slower than regular code for small matrices. This is because of setup time and how memory lines up.
• 🧠 Eigen usually uses regular code for small matrices, like 2×2 or 3×3. SIMD does not help here.
• 💊 Compilers can often make code run faster with auto-vectorization than manual SIMD. This happens when data lines up correctly and loops are simple.
• 📊 Tests on small tasks often do not show how SIMD really works in practice. This is because of setup time and how caching works.
• 🛠️ Combining loops, processing data in batches, and aligning memory make SIMD matrix code faster. These methods work better than using raw SIMD instructions.

SIMD (Single Instruction, Multiple Data) promises faster speeds. It does this by working on many data pieces at once. But many developers get confused when their manual SIMD code runs slower than regular code, especially with libraries like Eigen. This article explains why this happens. It also shows what affects SIMD speed, and how to pick the best way to make your code faster: using SIMD or regular code.

What Are SIMD Intrinsics?

SIMD instructions are special CPU commands. They help process data at the same time. Instead of doing one task for each item, SIMD works on bundled data in bigger registers. Examples include 128-bit SSE or 256-bit AVX on x86, or NEON for ARM. Each register can hold many data items, like four 32-bit floats for SSE, or eight for AVX. This lets many calculations happen at once, which is why SIMD seems so fast.

These instructions let developers directly control a processor's vector part. This means they can skip auto-vectorization and choose how data is handled themselves. You might see SIMD instructions used a lot in tasks that need high speed, such as:

• Matrix multiplications
• Dot product calculations
• Physics simulations
• Computer vision filters
• DSP (Digital Signal Processing)

However, this flexibility has a downside. Writing SIMD code means you need to know a lot about how hardware works, how memory lines up, and how fast instructions run. For instance, if you do not know how much a shuffle or blend instruction costs, your code might run slower than you expect, even with vector registers.

People often use SIMD instructions when:

• The compiler doesn’t effectively auto-vectorize certain patterns.
• Fine-grained control over CPU instructions is required.
• Specific performance optimizations are needed for specialized data layouts (e.g., SOA—Structure of Arrays).

If you are not careful, you could spend a week making your code faster and only get a small gain. Or you might even make it slower.

Scalar Code vs SIMD: Theory vs Practice

At first, SIMD seems like it could do a lot. For example, if a regular loop works on one float at a time, using SIMD might mean working on four or eight at once. This sounds like an immediate 4x or 8x speedup.

But in reality, things often do not turn out this way.

Theoretical Advantage

• A scalar 3×3 matrix multiplication processes 9 multiplications and 6 additions manually.
• With SIMD, multiple elements can be multiplied in a single instruction cycle.

This means SIMD should need fewer instructions and run more smoothly.

Why SIMD Under-Delivers in Practice

Even with this promise, several main problems stop SIMD from working well:

• ❌ Data setup like packing/unpacking vectors into proper alignment adds overhead.
• ❌ SIMD operations like shuffle or permute have higher instruction latency.
• ❌ Complex control flow (e.g., if, switch) is hard to express vector-wise.
• ❌ Smaller tasks do not make up for the time spent setting up vectors and moving memory.

Fog (2023) points out that regular operations often run faster. For example, an add instruction might finish in 1 cycle. But the SIMD version might take 3-5 cycles because of the data it needs to move.

Simply put, unless the calculation is big enough and simple enough, like repeating math on long arrays, you might do more work with SIMD than with regular code.

Eigen Matrix Performance and SIMD

Eigen is a fast C++ template library, used a lot for linear algebra. It is known for smart metaprogramming that changes expressions to use SIMD instructions automatically. Eigen balances how much control you have and how complex things are. It offers good speed for many uses right away, but it is not magic.

How Eigen Uses SIMD

• Uses vectorized paths only when beneficial (e.g., working with aligned, sufficiently large data).
• Chooses between scalar and SIMD backends depending on data layout, matrix size, and compiler flags.
• Intrinsically supports AVX, SSE, and NEON—but only under ideal conditions.

Why Small Matrices Slow Down

Most matrix libraries are made to work better as matrices get bigger. But many real-world programs in graphics, robotics, and control systems use small matrices that always stay the same size, like 2×2 or 3×3. For these:

• ✔️ Scalar loops are more tightly optimized.
• ❌ Vectorization setup costs outweigh runtime benefits.
• ✔️ The CPU can keep scalar data hot in registers across the whole computation.

Eigen knows this and often uses regular code for very small matrices. Trying to force them to use SIMD, either with manual SIMD or by setting up code to turn on Eigen's SIMD, can make things slower. This is true unless you are doing many operations at once.

Common Causes of Slower SIMD Performance

Bad SIMD speed often comes from several main reasons. Here is what often goes wrong:

1. Memory Access Misalignment

• SIMD prefers data aligned to 16- or 32-byte boundaries, depending on the vector width.
• A misaligned load may trigger penalties such as split reads across cache lines—or trigger exceptions.
• Scalar code can freely load/unload each element without alignment constraints.

To stop this problem, line up memory by hand. Use special memory tools or compiler hints like __attribute__((aligned(32))) or alignas(32) in C++11.

2. Instruction Overhead

• SIMD isn’t all about raw math—operations like shuffle, unpacklo/hi, and blend add complexity.
• These instructions help rearrange vectors to fit computing layouts but offer no direct arithmetic progress.
• Overuse of them leads to pipelines filled with shuffling more than math (Fog, 2023).

3. Branching and Control Flow

Good SIMD works best with simple, unrolled loops. When you use if-then-else for each item, it breaks the steady way SIMD works. You will often change branches to blend instructions that use masks. But this is also slow and complex.

4. Data Dependencies

In things like prefix sums or time-recursive filters, each calculation needs earlier values. SIMD cannot easily work with these links. This is because vector parts must work together without needing results from each other right away.

5. Cache Misses and Memory Behavior

• SIMD accelerates data throughput—but if data isn’t in L1/L2 cache, it’s like revving an engine without fuel.
• Miss rates are higher with scattered or matrix-transposed access.
• Scalar code often accesses memory more predictably, helping prefetchers stay ahead.

6. Compiler Optimizations Disabled or Blocked

Manual SIMD code often stops the compiler from making other parts of the code faster. But modern compilers can make regular loops use SIMD if:

• Data is aligned
• Control flow is predictable
• Loops have clear bounds

First, look at the output of -O3 -march=native. Do this before you try to use low-level instructions.

Micro-Benchmarking Pitfalls

If you have ever run a tight loop on a 3×3 matrix and wondered why your SIMD code was slow, you have hit the problems of testing small pieces of code.

Why Micro-Benchmarks Mislead

• 🚫 They ignore memory setup cost and attribute slowdowns to computation.
• 🚫 Shorter operations can’t benefit from cache reuse or throughput scaling.
• 🚫 Compiler optimizations like loop unrolling or common subexpression elimination drastically affect small test cases.

For instance, if you time a 2×2 matrix multiplication of floats by hand, you might miss things. These include how caches work, temperature slowing things down, and clock speed changes due to heat.

So, do not make your code faster just because you save 2 µs in made-up examples. Always test within the real program.

Real-World Benchmarking Tips

To really see if SIMD helps:

• ✅ Use realistic matrix sizes and loop lengths. Think hundreds/thousands of iterations, not three.
• ✅ Benchmark full pipelines or hot paths—not isolated operations.
• ✅ Turn on profiling tools:
  • perf for Linux
  • Intel VTune for x86
  • valgrind (specifically callgrind) to see branch hits and how cache works
• ✅ Measure cache hit rates alongside throughput. Tools like perf stat can show why the CPU slows down.

When Not to Use SIMD Intrinsics

Even though SIMD promises speed, it is not always the best choice. Here are important times to avoid using manual SIMD instructions:

• ✋ Code clarity trumps subtle speedups (especially for open-source or team-shared projects).
• 🚫 Very small matrices don’t benefit enough to justify complexity.
• 🧠 Your compiler likely uses better heuristics and instruction scheduling than hand-written SIMD.
• 👨‍💻 If your SIMD version disables auto-vectorization elsewhere, you get a net performance loss.

The 2023 Stack Overflow Developer Survey says that 59% of developers would rather let the compiler make things faster than do it themselves. This is because compilers are getting better at safely using vectorization.

Best Practices for SIMD in Eigen

To make SIMD work well in Eigen and skip common problems:

1. 🔧 Use memory containers that line up: Eigen needs memory to line up correctly for SIMD to turn on. Use EIGEN_MAKE_ALIGNED_OPERATOR_NEW or std::aligned_allocator.

2. 🛠 Do not pack manually: Eigen's API handles how registers are set up better than SIMD arrays you make yourself. Use Eigen types (Vector4f, Matrix3d). Do not use std::array<float, 4>.

3. 🧪 Turn off Eigen vectorization for a bit: Use EIGEN_DONT_VECTORIZE. This makes it easy to compare regular paths to vector paths.

4. 📦 Process small matrices in groups: Group small matrices into tensor-like blocks to use SIMD. Working on 1000 3×3 matrices together uses SIMD better than working on them one by one.

Reliable Paths to Improve SIMD Matrix Performance

If your SIMD code is not making things faster, try these ways to improve it first. They are known to work:

1. Loop Fusion

Put small loops into one bigger loop. This spreads the extra work of the loop structure across more calculations.

2. Loop Unrolling

Unroll short loops yourself. Or use compiler hints like #pragma unroll to help instructions run at the same time.

3. Operational Reordering

Change the order of instructions to stop delays in the pipeline. Move operations that use less memory closer together. This lets the CPU hide how long some tasks take.

4. Batch Processing

Do not run 1000 matrix multiplications one after another. Instead, change the data into a big array and work on many at the same time. This works especially well for very small matrices.

5. Measure, Don’t Assume

Use profiling to find the exact slow spots. You might see that changing one loop to access cache better helps more than switching to AVX-512.

Know When to Optimize

Making code faster takes effort, and you should expect good results. But many people spend too much time making SIMD code faster for only tiny speed gains. When you choose between SIMD and regular processing:

• Start with auto-vectorized, clear code.
• Profile hot paths and memory access patterns.
• Use SIMD intrinsics only when they yield 2×+ speedups (with proof).
• Revisit SIMD after higher-level refactoring fails.

And always keep this in mind: Making code faster without testing is like sailing without a compass.

Want to understand how memory lines up and what slows down SIMD? Read our other guide: Understanding Memory Alignment in Modern CPUs.

Citations

Fog, A. (2023). Instruction tables: Lists of instruction latencies, throughputs and micro-operation breakdowns for Intel, AMD and VIA CPUs. Retrieved from https://www.agner.org/optimize/instruction_tables.pdf

Intel. (2022). Improving Performance with SIMD [Whitepaper]. Intel Developer Zone. Retrieved from https://www.intel.com/content/www/us/en/developer/articles/technical/improving-performance-with-simd.html

Stack Overflow Survey. (2023). Developer Trends Report. StackExchange Inc. Retrieved from https://survey.stackoverflow.co/2023/