Пишу алгоритм FFT на Си для процессора Эльбрус: прямая векторизация
Данная статья является продолжением предыдущей статьи.
Здесь рассматривается вариант прямой векторизации алгоритма FFT.
Перед чтением рекомендуется бегло ознакомиться с предыдущей статьёй.
Промежуточные результаты представлены на графиках в конце каждого раздела реализации.
Итоговые результаты показаны на графиках в разделе Собираем FFT
Исходный код
Исходный код лежит здесь.
Структура кода намеренно сделана как можно проще.
Приоритеты в порядке убывания:
— Скорость выполнения
— Единообразие
— Понятность
Компиляция:
./compile.sh */*.c
Проверка корректности:
./check_all.sh 12 1000 x
Измерение скорости:
taskset 1 ./speed_all_stage.sh 12 10000 x
taskset 1 ./speed_all_fft.sh 12 10000 x
taskset 1 нужен для корректного чтения счётчика тактов (привязка к первому ядру).
Опции компиляции
При компиляции использовались следующие опции lcc:
-Wall -O3 -faligned -ffast-math -march=elbrus-v5
Почему я компилирую для elbrus-v5 написано здесь.
Как измерялось время
Замеры времени делались с помощью функции clock_gettime():
struct timespec t0, t1;
clock_gettime(CLOCK_REALTIME, &t0);
/*** здесь измеряемый код ***/
clock_gettime(CLOCK_REALTIME, &t1);
int usec = (t1.tv_sec - t0.tv_sec)*1000000 + (t1.tv_nsec - t0.tv_nsec)/1000;
Также использовалось чтение счётчика тактов процессора:
uint64_t get_clock_count()
{
uint64_t dst;
#pragma asm_inline
asm ("rrd %%clkr, %0" : "=r" (dst));
return dst;
}
...
uint64_t ticks0 = get_clock_count();
/*** здесь измеряемый код ***/
uint64_t ticks1 = get_clock_count();
uint64_t ticks = ticks1 - ticks0;
Как создавались графики
Графики рисовались в Google Sheets.
Данные для графиков собирались с помощью скрипта data_for_diagram.py, который запускает программу измерения скорости с разными параметрами и парсит её вывод.
Пример запуска скрипта:
taskset 1 ./data_for_diagram.py \
./fft_radix2/fft_radix2_speedtest 4 21 1 1000 -1 > out.txt
taskset 1 нужен для корректного чтения счётчика тактов (привязка к первому ядру).
Содержание:
Пишем векторный Reverse
reverse_radix2_vector2
reverse_radix2_vector4
reverse_radix4_vector2
Пишем векторный Stage
stage_radix2_vector2
stage_radix2_vector4
stage_radix4_vector2
Пишем LastStage
lastStage_radix2
Что такое прямая векторизация
Для ряда операций (сложение, умножение и так далее) в процессоре Эльбрус присутствуют векторные инструкции (SIMD). Векторные инструкции выполняют несколько одинаковых операций одновременно.
В предыдущей статье векторные инструкции использовались просто как набор из нескольких инструкций в одной. Например, вместо двух инструкций умножения float чисел, нужных по алгоритму, применялась одна векторная, выполняющая два умножения (SIMD64).
Проблема такого подхода в том, что нет простого способа ускорить алгоритм через замену инструкции SIMD64 на аналогичную SIMD128. Использование SIMD128 приводило к появлению инструкций перетасовки данных (shuf, perm), занимавших исполнительные юниты.
В этой статье векторные инструкции используются по‑другому.
Здесь с их помощью создаются «векторные» версии алгоритмов из исходных.
Векторные алгоритмы выполняют несколько одинаковых задач одновременно аналогично тому, как векторные инструкции выполняют несколько одинаковых операций одновременно.
Такой подход я и называю «прямой векторизацией».
Для выполнения прямой векторизации нужно заменить обычные инструкции на их векторные версии. Чтобы такое было возможно, в исходной программе должны использоваться инструкции, для которых существуют векторные версии. Обрабатываемые данные также требуется привести в векторную форму, то есть такой формат, который ожидается векторными инструкциями.
Прямая векторизация кода на Си
Пусть, например, алгоритм обрабатывает float числа.
Для выполнения прямой векторизации в 2 или 4 раза нужно заменить в Си‑коде исходного алгоритма тип float на __v2sf или __v4sf соответственно. И компилятор сгенерирует векторизованный код с использованием инструкций SIMD64 или SIMD128 соответственно.
При этом операции «+», «-», «*» будут скомпилированы в векторные версии инструкций.
Обрабатываемые данные также требуется предварительно привести в векторную форму:
там, где было одно
floatчисло, теперь должен быть «вектор»floatчисел (расположенные рядом 2 или 4floatчисла)там, где был массив
floatчисел, теперь должен быть массив «векторов»floatчисел (фактически исходный массив преобразуется в набор перемежённых массивов)
Прямая векторизация алгоритма Stage
Пусть изначально есть следующий код:
void stage_radix2_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef)
{
myComplex *x_in = &data_in[0];
myComplex *y_in = &data_in[1];
myComplex *c_in = coef;
myComplex *out_add = &data_out[0];
myComplex *out_sub = &data_out[data_count/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
myComplex x = x_in[2*i];
myComplex y = y_in[2*i];
myComplex c = c_in[i];
myComplex cy = complex_mul(c, y);
out_add[i] = complex_add(x, cy);
out_sub[i] = complex_sub(x, cy);
}
}
Избавимся от типа myComplex, оставив просто тип float:
void stage_radix2_etalon(int data_count, float *data_in, float *data_out, float *coef)
{
float *x_real_in = &data_in[0];
float *x_imag_in = &data_in[1];
float *y_real_in = &data_in[2];
float *y_imag_in = &data_in[3];
float *c_real_in = &coef[0];
float *c_imag_in = &coef[1];
float *out_add_real = &data_out[0];
float *out_add_imag = &data_out[1];
float *out_sub_real = &data_out[data_count/2 + 0];
float *out_sub_imag = &data_out[data_count/2 + 1];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
float x_real = x_real_in[4*i];
float x_imag = x_imag_in[4*i];
float y_real = y_real_in[4*i];
float y_imag = y_imag_in[4*i];
float c_real = c_real_in[2*i];
float c_imag = c_imag_in[2*i];
float cy_real = c_real*y_real - c_imag*y_imag;
float cy_imag = c_real*y_imag + c_imag*y_real;
out_add_real[2*i] = x_real + cy_real;
out_add_imag[2*i] = x_imag + cy_imag;
out_sub_real[2*i] = x_real - cy_real;
out_sub_imag[2*i] = x_imag - cy_imag;
}
}
Получившийся результат будем называть «базовым алгоритмом Stage» для прямой векторизации.
Если в этом коде заменить все float на __v2sf, получим прямую векторизацию в 2 раза.
Если в этом коде заменить все float на __v4sf, получим прямую векторизацию в 4 раза.
К сожалению, я узнал про этот приём уже после того, как написал весь код.
Поэтому сейчас в коде используются интринсики вместе с типами uint64_t и __v2di там, где в базовом алгоритме стоят операции «+», «-», «*».
В будущем я, скорее всего, исправлю код.
Прямая векторизация полного алгоритма FFT
Алгоритм FFT состоит из одного запуска Reverse и нескольких запусков Stage.
Stage был векторизован в предыдущем разделе.
Reverse векторизуется аналогичным образом.
Объединив векторные Reverse и Stage, получим векторный алгоритм FFT.
Этот алгоритм ожидает на входе и выходе набор перемежённых массивов float чисел.
В результате его работы будут вычислены FFT от каждого из перемежённых массивов.
Решение задачи FFT с помощью векторного FFT
Итак, мы умеем вычислять несколько FFT одновременно.
Как использовать это умение, чтобы вычислить FFT от одного набора комплексных чисел?
Для этого нужно вспомнить, как выглядел рекурсивный алгоритм Кули‑Тьюки:
Вычисляются FFT от каждого из перемежённых массивов комплексных чисел.
Вычисляется итоговый FFT с помощью этих перемежённых FFT.
Первый шаг сделаем с помощью векторного алгоритма FFT.
Векторный FFT ожидает на входе и выходе набор перемежённых массивов float чисел, а алгоритм Кули‑Тьюки ожидает набор перемежённых массивов комплексных чисел.
Поэтому добавим перед этим шагом преобразование исходных данных в векторную форму, а после этого шага — преобразование из векторной формы в исходную.
Схемы преобразования данных в векторную форму
Теперь на вход можно подавать массив комплексных чисел.
Алгоритм будет воспринимать его, как набор перемежённых массивов комплексных чисел.
В результате его работы будут вычислены FFT от каждого из перемежённых массивов.
Второй шаг похож на то, что делает адаптированный для Эльбруса алгоритм Stage из прошлой статьи. Если точнее, на этом шаге выполняются те же действия, что делает самый последний Stage в алгоритме FFT (тот Stage, в котором используется полный набор разных коэффициентов). Поэтому далее второй шаг будем называть «LastStage».
В случае выполнения прямой векторизации в 2 раза (когда получаются 2 перемежённых массива) в качестве LastStage можно использовать любой stage_radix2 из прошлой статьи.
В случае выполнения прямой векторизации в 4 раза (когда получаются 4 перемежённых массива) в качестве LastStage можно использовать любой stage_radix4 из прошлой статьи (или сначала stage_radix2_vector2 и затем любой stage_radix2 из прошлой статьи).
Итого, полный порядок действий выглядит так:
Исходные данные преобразуются в векторную форму.
Выполняется векторный Reverse.
Несколько раз выполняется векторный Stage.
Полученные данные преобразуются из векторной формы в исходную.
Выполняется LastStage.
Преобразования данных в векторную форму и обратно — достаточно простые процедуры. Поэтому в итоговой реализации они сразу включены внутрь Reverse и LastStage соответственно.
Пишем векторный Reverse
reverse_radix2_vector2
1. reverse_radix2_vector2_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix2 вычисляется с помощью цикла.
int reverseNumber_radix2(int number, int bit_count)
{
int answer = 0;
for(int i = 0; i < bit_count; ++i)
{
answer <<= 1;
answer |= number & 1;
number >>= 1;
}
return answer;
}
void reverse_radix2_vector2_etalon(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
for(int64_t i = 0; i < count_out; ++i)
{
int index = reverseNumber_radix2(i, bit_count_out);
myComplex a = data_in[2*i + 0];
myComplex b = data_in[2*i + 1];
data_out[index] = (myComplex2){.real = {a.real, b.real}, .imag = {a.imag, b.imag}};
}
}
2. reverse_radix2_vector2
В процессоре есть инструкция bitrevd, которая производит операцию reverseNumber_radix2 над 64-битным числом. Заменим reverseNumber_radix2() на __builtin_e2k_bitrevd().
Код на Си
void reverse_radix2_vector2(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*)data_in;
__v2di *out = (__v2di*)data_out;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; ++i)
{
int64_t index = __builtin_e2k_bitrevd(i) >> shift_out;
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
out[index] = __builtin_e2k_qppermb(in[i], in[i], mask);
}
}
Основной цикл на ассемблере
.L4793:
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0
}
.L4668:
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
bitrevd,0,sm %b[11], %b[13]
qppermb,1,sm %b[10], %b[10], %r4, %b[12]
addd,2,sm %b[11], 0x1, %b[9]
shrd,3,sm %b[17], %r0, %b[19]
shld,4,sm %b[21], 0x4, %b[20]
stqp,5 %r2, %b[22], %b[18]
movaqp,1 area=0, ind=0, am=1, be=0, %b[0]
}
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
3. reverse_radix2_vector2_x2
Сделаем ручную раскрутку цикла в 2 раза.
Сразу напишем код так, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Код на Си
void reverse_radix2_vector2_x2(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
int64_t delta = (1LL << shift_out) / 16;
__v2di *in = (__v2di*) data_in;
__v2di *out_0 = (__v2di*)&data_out[0];
__v2di *out_1 = (__v2di*)&data_out[count_out/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0, shifted_i = 0; i < count_out; i += 2, shifted_i += 2*delta)
{
int64_t offset = __builtin_e2k_bitrevd(shifted_i);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)out_0 + offset) = __builtin_e2k_qppermb(in[i + 0], in[i + 0], mask);
*(__v2du*)((void*)out_1 + offset) = __builtin_e2k_qppermb(in[i + 1], in[i + 1], mask);
}
}
Основной цикл на ассемблере
.L5010:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
.L4823:
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
bitrevd,0,sm %b[22], %b[13]
qppermb,1,sm %b[10], %b[10], %r5, %b[19]
stqp,2 %r4, %b[17], %b[21]
qppermb,3,sm %b[11], %b[11], %r5, %b[23]
addd,4,sm %b[20], %r0, %b[18]
stqp,5 %r2, %b[17], %b[25]
movaqp,0 area=0, ind=16, am=1, be=0, %b[0]
movaqp,1 area=0, ind=0, am=0, be=0, %b[1]
}
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит два чтения из одного места памяти и запись в два разных места памяти.
4. reverse_radix2_vector2_stream2
Теперь сделаем наоборот: будем читать из двух разных мест, а писать рядом.
Чтение в 2 потока из разных мест памяти, запись в одно место памяти.
Сразу напишем код так, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Код на Си
void reverse_radix2_vector2_stream2(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
int64_t delta = (1LL << shift_out) / 16;
__v2di *in_0 = (__v2di*)&data_in[0];
__v2di *in_1 = (__v2di*)&data_in[count_in/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0, shifted_i = 0; i < count_out/2; ++i, shifted_i += delta)
{
int64_t offset = __builtin_e2k_bitrevd(shifted_i);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[i], in_0[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1[i], in_1[i], mask);
}
}
Основной цикл на ассемблере
.L5212:
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0
}
.L5039:
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
bitrevd,0,sm %b[22], %b[13]
qppermb,1,sm %b[11], %b[11], %r5, %b[23]
stqp,2 %r2, %b[17], %b[25]
qppermb,3,sm %b[10], %b[10], %r5, %b[19]
addd,4,sm %b[20], %r4, %b[18]
stqp,5 %r0, %b[17], %b[21]
movaqp,1 area=0, ind=0, am=1, be=0, %b[1]
movaqp,3 area=0, ind=0, am=1, be=0, %b[0]
}
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix2_vector2_stream4
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream4(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00 = (__v2di*)&data_in[0 * count_in/4];
__v2di *in_01 = (__v2di*)&data_in[1 * count_in/4];
__v2di *in_10 = (__v2di*)&data_in[2 * count_in/4];
__v2di *in_11 = (__v2di*)&data_in[3 * count_in/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/4; ++i)
{
int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[i], in_00[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10[i], in_10[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_01[i], in_01[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_11[i], in_11[i], mask);
}
}
Основной цикл на ассемблере
.L5516:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=0, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=4, abs=16, disp=0
}
.L5255:
{
loop_mode
bitrevd,0,sm %b[4], %b[1]
qppermb,1,sm %b[27], %b[27], %r7, %b[9]
stqp,2 %r2, %b[14], %b[28]
shld,3,sm %b[24], 0x4, %b[12]
qppermb,4,sm %b[22], %b[22], %r7, %b[5]
stqp,5 %r4, %b[14], %b[26]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[19], %b[19], %r7, %b[26]
addd,1,sm %b[2], 0x1, %b[0]
stqp,2 %r0, %b[14], %b[11]
qppermb,3,sm %b[10], %b[10], %r7, %b[24]
shrd,4,sm %b[3], %r6, %b[22]
stqp,5 %r5, %b[14], %b[7]
movaqp,0 area=1, ind=0, am=1, be=0, %b[4]
movaqp,1 area=0, ind=0, am=1, be=0, %b[13]
movaqp,2 area=1, ind=0, am=1, be=0, %b[16]
movaqp,3 area=0, ind=0, am=1, be=0, %b[21]
}
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
6. reverse_radix2_vector2_stream8
Чтение в 8 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream8(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_000 = (__v2di*)&data_in[0 * count_in/8];
__v2di *in_001 = (__v2di*)&data_in[1 * count_in/8];
__v2di *in_010 = (__v2di*)&data_in[2 * count_in/8];
__v2di *in_011 = (__v2di*)&data_in[3 * count_in/8];
__v2di *in_100 = (__v2di*)&data_in[4 * count_in/8];
__v2di *in_101 = (__v2di*)&data_in[5 * count_in/8];
__v2di *in_110 = (__v2di*)&data_in[6 * count_in/8];
__v2di *in_111 = (__v2di*)&data_in[7 * count_in/8];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/8; ++i)
{
int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[i], in_000[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_100[i], in_100[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_010[i], in_010[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_110[i], in_110[i], mask);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_001[i], in_001[i], mask);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_101[i], in_101[i], mask);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_011[i], in_011[i], mask);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_111[i], in_111[i], mask);
}
}
Основной цикл на ассемблере
.L6024:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=3, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=3, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=3, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=3, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=3, abs=24, disp=0
}
.L5591:
{
loop_mode
qppermb,1,sm %b[21], %b[21], %r11, %b[26]
stqp,2 %r2, %b[18], %b[24]
qppermb,4,sm %b[13], %b[13], %r11, %b[25]
stqp,5 %r4, %b[18], %b[23]
}
{
loop_mode
qppermb,1,sm %b[12], %b[12], %r11, %b[24]
stqp,2 %r0, %b[18], %b[26]
shrd,3,sm %b[22], %r12, %b[1]
qppermb,4,sm %b[5], %b[5], %r11, %b[23]
stqp,5 %r5, %b[18], %b[25]
}
{
loop_mode
qppermb,1,sm %b[9], %b[9], %r11, %b[27]
stqp,2 %r6, %b[18], %b[24]
shld,3,sm %b[1], 0x4, %b[16]
qppermb,4,sm %b[8], %b[8], %r11, %b[25]
stqp,5 %r9, %b[18], %b[23]
movaqp,0 area=1, ind=0, am=1, be=0, %b[12]
movaqp,1 area=0, ind=0, am=1, be=0, %b[13]
movaqp,2 area=3, ind=0, am=1, be=0, %b[4]
movaqp,3 area=2, ind=0, am=1, be=0, %b[5]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[15], %b[15], %r11, %b[22]
addd,1,sm %b[2], 0x1, %b[0]
stqp,2 %r7, %b[18], %b[27]
qppermb,3,sm %b[14], %b[14], %r11, %b[21]
bitrevd,4,sm %b[2], %b[20]
stqp,5 %r10, %b[18], %b[25]
movaqp,0 area=3, ind=0, am=1, be=0, %b[1]
movaqp,1 area=2, ind=0, am=1, be=0, %b[8]
movaqp,2 area=1, ind=0, am=1, be=0, %b[9]
movaqp,3 area=0, ind=0, am=1, be=0, %b[17]
}
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
7. reverse_radix2_vector2_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream16(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_0000 = (__v2di*)&data_in[ 0 * count_in/16];
__v2di *in_0001 = (__v2di*)&data_in[ 1 * count_in/16];
__v2di *in_0010 = (__v2di*)&data_in[ 2 * count_in/16];
__v2di *in_0011 = (__v2di*)&data_in[ 3 * count_in/16];
__v2di *in_0100 = (__v2di*)&data_in[ 4 * count_in/16];
__v2di *in_0101 = (__v2di*)&data_in[ 5 * count_in/16];
__v2di *in_0110 = (__v2di*)&data_in[ 6 * count_in/16];
__v2di *in_0111 = (__v2di*)&data_in[ 7 * count_in/16];
__v2di *in_1000 = (__v2di*)&data_in[ 8 * count_in/16];
__v2di *in_1001 = (__v2di*)&data_in[ 9 * count_in/16];
__v2di *in_1010 = (__v2di*)&data_in[10 * count_in/16];
__v2di *in_1011 = (__v2di*)&data_in[11 * count_in/16];
__v2di *in_1100 = (__v2di*)&data_in[12 * count_in/16];
__v2di *in_1101 = (__v2di*)&data_in[13 * count_in/16];
__v2di *in_1110 = (__v2di*)&data_in[14 * count_in/16];
__v2di *in_1111 = (__v2di*)&data_in[15 * count_in/16];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/16; ++i)
{
int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0000[i], in_0000[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1000[i], in_1000[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_0100[i], in_0100[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1100[i], in_1100[i], mask);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_0010[i], in_0010[i], mask);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_1010[i], in_1010[i], mask);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_0110[i], in_0110[i], mask);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_1110[i], in_1110[i], mask);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_0001[i], in_0001[i], mask);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_1001[i], in_1001[i], mask);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_0101[i], in_0101[i], mask);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_1101[i], in_1101[i], mask);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_0011[i], in_0011[i], mask);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_1011[i], in_1011[i], mask);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_0111[i], in_0111[i], mask);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_1111[i], in_1111[i], mask);
}
}
Основной цикл на ассемблере
.L6935:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=2, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=2, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=2, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=2, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=2, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=2, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=2, abs=24, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=2, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=2, abs=28, disp=0
}
.L6163:
{
loop_mode
qppermb,1,sm %b[24], %b[24], %r1, %b[28]
stqp,2 %r2, %b[13], %b[9]
qppermb,4,sm %b[22], %b[22], %r1, %b[27]
stqp,5 %r5, %b[13], %b[4]
}
{
loop_mode
qppermb,1,sm %b[26], %b[26], %r1, %b[30]
stqp,2 %r0, %b[13], %b[28]
qppermb,4,sm %b[25], %b[25], %r1, %b[29]
stqp,5 %r6, %b[13], %b[27]
movaqp,0 area=7, ind=0, am=1, be=0, %b[10]
movaqp,1 area=6, ind=0, am=1, be=0, %b[4]
movaqp,2 area=7, ind=0, am=1, be=0, %b[7]
movaqp,3 area=6, ind=0, am=1, be=0, %b[1]
}
{
loop_mode
qppermb,1,sm %b[23], %b[23], %r1, %b[27]
stqp,2 %r7, %b[13], %b[30]
qppermb,4,sm %b[16], %b[16], %r1, %b[26]
stqp,5 %r10, %b[13], %b[29]
movaqp,0 area=1, ind=0, am=1, be=0, %b[15]
movaqp,1 area=0, ind=0, am=1, be=0, %b[17]
movaqp,2 area=5, ind=0, am=1, be=0, %b[9]
movaqp,3 area=4, ind=0, am=1, be=0, %b[12]
}
{
loop_mode
qppermb,1,sm %b[21], %b[21], %r1, %b[29]
stqp,2 %r9, %b[13], %b[27]
qppermb,4,sm %b[14], %b[14], %r1, %b[28]
stqp,5 %r11, %b[13], %b[26]
movaqp,0 area=5, ind=0, am=1, be=0, %b[16]
movaqp,1 area=4, ind=0, am=1, be=0, %b[19]
movaqp,2 area=1, ind=0, am=1, be=0, %b[20]
movaqp,3 area=0, ind=0, am=1, be=0, %b[22]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[18], %b[18], %r1, %b[27]
stqp,2 %r12, %b[13], %b[29]
bitrevd,3,sm %b[2], %b[25]
qppermb,4,sm %b[11], %b[11], %r1, %b[26]
stqp,5 %r13, %b[13], %b[28]
movaqp,0 area=3, ind=0, am=1, be=0, %b[23]
movaqp,1 area=2, ind=0, am=1, be=0, %b[24]
movaqp,2 area=3, ind=0, am=1, be=0, %b[14]
movaqp,3 area=2, ind=0, am=1, be=0, %b[21]
}
{
loop_mode
qppermb,1,sm %b[6], %b[6], %r1, %b[29]
stqp,2 %r14, %b[13], %b[27]
shrd,3,sm %b[25], %r20, %b[2]
qppermb,4,sm %b[3], %b[3], %r1, %b[28]
stqp,5 %r15, %b[13], %b[26]
}
{
loop_mode
qppermb,1,sm %b[10], %b[10], %r1, %b[6]
stqp,2 %r16, %b[13], %b[29]
shld,3,sm %b[2], 0x4, %b[11]
qppermb,4,sm %b[7], %b[7], %r1, %b[3]
stqp,5 %r17, %b[13], %b[28]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[17], %b[17], %r1, %b[7]
stqp,2 %r18, %b[13], %b[8]
qppermb,3,sm %b[15], %b[15], %r1, %b[2]
stqp,5 %r19, %b[13], %b[5]
}
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
8. reverse_radix2_vector2_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream32(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00000 = (__v2di*)&data_in[ 0 * count_in/32];
__v2di *in_00001 = (__v2di*)&data_in[ 1 * count_in/32];
__v2di *in_00010 = (__v2di*)&data_in[ 2 * count_in/32];
__v2di *in_00011 = (__v2di*)&data_in[ 3 * count_in/32];
__v2di *in_00100 = (__v2di*)&data_in[ 4 * count_in/32];
__v2di *in_00101 = (__v2di*)&data_in[ 5 * count_in/32];
__v2di *in_00110 = (__v2di*)&data_in[ 6 * count_in/32];
__v2di *in_00111 = (__v2di*)&data_in[ 7 * count_in/32];
__v2di *in_01000 = (__v2di*)&data_in[ 8 * count_in/32];
__v2di *in_01001 = (__v2di*)&data_in[ 9 * count_in/32];
__v2di *in_01010 = (__v2di*)&data_in[10 * count_in/32];
__v2di *in_01011 = (__v2di*)&data_in[11 * count_in/32];
__v2di *in_01100 = (__v2di*)&data_in[12 * count_in/32];
__v2di *in_01101 = (__v2di*)&data_in[13 * count_in/32];
__v2di *in_01110 = (__v2di*)&data_in[14 * count_in/32];
__v2di *in_01111 = (__v2di*)&data_in[15 * count_in/32];
__v2di *in_10000 = (__v2di*)&data_in[16 * count_in/32];
__v2di *in_10001 = (__v2di*)&data_in[17 * count_in/32];
__v2di *in_10010 = (__v2di*)&data_in[18 * count_in/32];
__v2di *in_10011 = (__v2di*)&data_in[19 * count_in/32];
__v2di *in_10100 = (__v2di*)&data_in[20 * count_in/32];
__v2di *in_10101 = (__v2di*)&data_in[21 * count_in/32];
__v2di *in_10110 = (__v2di*)&data_in[22 * count_in/32];
__v2di *in_10111 = (__v2di*)&data_in[23 * count_in/32];
__v2di *in_11000 = (__v2di*)&data_in[24 * count_in/32];
__v2di *in_11001 = (__v2di*)&data_in[25 * count_in/32];
__v2di *in_11010 = (__v2di*)&data_in[26 * count_in/32];
__v2di *in_11011 = (__v2di*)&data_in[27 * count_in/32];
__v2di *in_11100 = (__v2di*)&data_in[28 * count_in/32];
__v2di *in_11101 = (__v2di*)&data_in[29 * count_in/32];
__v2di *in_11110 = (__v2di*)&data_in[30 * count_in/32];
__v2di *in_11111 = (__v2di*)&data_in[31 * count_in/32];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32; ++i)
{
int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00000[i], in_00000[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10000[i], in_10000[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_01000[i], in_01000[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_11000[i], in_11000[i], mask);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_00100[i], in_00100[i], mask);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_10100[i], in_10100[i], mask);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_01100[i], in_01100[i], mask);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11100[i], in_11100[i], mask);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_00010[i], in_00010[i], mask);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_10010[i], in_10010[i], mask);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_01010[i], in_01010[i], mask);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_11010[i], in_11010[i], mask);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_00110[i], in_00110[i], mask);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_10110[i], in_10110[i], mask);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_01110[i], in_01110[i], mask);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_11110[i], in_11110[i], mask);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_00001[i], in_00001[i], mask);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_10001[i], in_10001[i], mask);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_01001[i], in_01001[i], mask);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_11001[i], in_11001[i], mask);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_00101[i], in_00101[i], mask);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_10101[i], in_10101[i], mask);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_01101[i], in_01101[i], mask);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_11101[i], in_11101[i], mask);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_00011[i], in_00011[i], mask);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_10011[i], in_10011[i], mask);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_01011[i], in_01011[i], mask);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_11011[i], in_11011[i], mask);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_00111[i], in_00111[i], mask);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_10111[i], in_10111[i], mask);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_01111[i], in_01111[i], mask);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_11111[i], in_11111[i], mask);
}
}
Основной цикл на ассемблере
.L8688:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0
}
.L7202:
{
loop_mode
qppermb,1,sm %b[37], %b[37], %r0, %b[21]
stqp,2 %r2, %b[16], %b[20]
qppermb,4,sm %b[35], %b[35], %r0, %b[18]
stqp,5 %r6, %b[16], %b[18]
}
{
loop_mode
qppermb,1,sm %b[39], %b[39], %r0, %b[20]
stqp,2 %r4, %b[16], %b[21]
qppermb,4,sm %b[38], %b[38], %r0, %b[18]
stqp,5 %r7, %b[16], %b[18]
}
{
loop_mode
qppermb,1,sm %b[23], %b[23], %r0, %b[19]
stqp,2 %r9, %b[16], %b[20]
qppermb,4,sm %b[19], %b[19], %r0, %b[18]
stqp,5 %r11, %b[16], %b[18]
}
{
loop_mode
qppermb,1,sm %b[26], %b[26], %r0, %b[19]
stqp,2 %r10, %b[16], %b[19]
qppermb,4,sm %b[33], %b[33], %r0, %b[18]
stqp,5 %r12, %b[16], %b[18]
}
{
loop_mode
qppermb,1,sm %b[25], %b[25], %r0, %b[19]
stqp,2 %r13, %b[16], %b[19]
qppermb,4,sm %b[32], %b[32], %r0, %b[18]
stqp,5 %r14, %b[16], %b[18]
}
{
loop_mode
qppermb,1,sm %b[31], %b[31], %r0, %b[24]
stqp,2 %r15, %b[16], %b[19]
qppermb,4,sm %b[30], %b[30], %r0, %b[21]
stqp,5 %r16, %b[16], %b[18]
movaqp,0 area=15, ind=0, am=1, be=0, %b[19]
movaqp,1 area=14, ind=0, am=1, be=0, %b[22]
movaqp,2 area=15, ind=0, am=1, be=0, %b[18]
movaqp,3 area=14, ind=0, am=1, be=0, %b[20]
}
{
loop_mode
qppermb,1,sm %b[29], %b[29], %r0, %b[26]
stqp,2 %r17, %b[16], %b[24]
qppermb,4,sm %b[17], %b[17], %r0, %b[25]
stqp,5 %r18, %b[16], %b[21]
movaqp,0 area=13, ind=0, am=1, be=0, %b[21]
movaqp,1 area=12, ind=0, am=1, be=0, %b[24]
movaqp,2 area=13, ind=0, am=1, be=0, %b[17]
movaqp,3 area=12, ind=0, am=1, be=0, %b[23]
}
{
loop_mode
qppermb,1,sm %b[28], %b[28], %r0, %b[26]
stqp,2 %r19, %b[16], %b[26]
qppermb,4,sm %b[27], %b[27], %r0, %b[25]
stqp,5 %r20, %b[16], %b[25]
movaqp,0 area=11, ind=0, am=1, be=0, %b[1]
movaqp,1 area=10, ind=0, am=1, be=0, %b[13]
movaqp,2 area=11, ind=0, am=1, be=0, %b[0]
movaqp,3 area=10, ind=0, am=1, be=0, %b[12]
}
{
loop_mode
qppermb,1,sm %g17, %g17, %r0, %b[28]
stqp,2 %r21, %b[16], %b[26]
qppermb,4,sm %g16, %g16, %r0, %b[27]
stqp,5 %r22, %b[16], %b[25]
movaqp,0 area=9, ind=0, am=1, be=0, %g17
movaqp,1 area=8, ind=0, am=1, be=0, %b[26]
movaqp,2 area=9, ind=0, am=1, be=0, %g16
movaqp,3 area=8, ind=0, am=1, be=0, %b[25]
}
{
loop_mode
qppermb,1,sm %b[15], %b[15], %r0, %b[31]
stqp,2 %r23, %b[16], %b[28]
qppermb,4,sm %b[14], %b[14], %r0, %b[30]
stqp,5 %r24, %b[16], %b[27]
movaqp,0 area=7, ind=0, am=1, be=0, %b[27]
movaqp,1 area=6, ind=0, am=1, be=0, %b[29]
movaqp,2 area=7, ind=0, am=1, be=0, %b[15]
movaqp,3 area=6, ind=0, am=1, be=0, %b[28]
}
{
loop_mode
qppermb,1,sm %b[3], %b[3], %r0, %b[36]
stqp,2 %r25, %b[16], %b[31]
qppermb,4,sm %b[2], %b[2], %r0, %b[33]
stqp,5 %r26, %b[16], %b[30]
movaqp,0 area=1, ind=0, am=1, be=0, %b[32]
movaqp,1 area=0, ind=0, am=1, be=0, %b[34]
movaqp,2 area=5, ind=0, am=1, be=0, %b[30]
movaqp,3 area=4, ind=0, am=1, be=0, %b[31]
}
{
loop_mode
qppermb,1,sm %b[24], %b[24], %r0, %b[3]
stqp,2 %r27, %b[16], %b[36]
qppermb,4,sm %b[23], %b[23], %r0, %b[2]
stqp,5 %r28, %b[16], %b[33]
movaqp,0 area=5, ind=0, am=1, be=0, %b[23]
movaqp,1 area=4, ind=0, am=1, be=0, %b[24]
movaqp,2 area=1, ind=0, am=1, be=0, %b[33]
movaqp,3 area=0, ind=0, am=1, be=0, %b[35]
}
{
loop_mode
addd,0,sm %r5, 0x1, %r5
qppermb,1,sm %b[21], %b[21], %r0, %b[5]
stqp,2 %r29, %b[16], %b[5]
bitrevd,3,sm %r5, %b[38]
qppermb,4,sm %b[17], %b[17], %r0, %b[4]
stqp,5 %r30, %b[16], %b[4]
movaqp,0 area=3, ind=0, am=1, be=0, %b[36]
movaqp,1 area=2, ind=0, am=1, be=0, %b[37]
movaqp,2 area=3, ind=0, am=1, be=0, %b[17]
movaqp,3 area=2, ind=0, am=1, be=0, %b[21]
}
{
loop_mode
qppermb,1,sm %b[22], %b[22], %r0, %b[7]
stqp,2 %r31, %b[16], %b[7]
shrd,3,sm %b[38], %r1, %b[20]
qppermb,4,sm %b[20], %b[20], %r0, %b[6]
stqp,5 %r32, %b[16], %b[6]
}
{
loop_mode
qppermb,1,sm %b[19], %b[19], %r0, %b[9]
stqp,2 %r33, %b[16], %b[9]
shld,3,sm %b[20], 0x4, %b[14]
qppermb,4,sm %b[18], %b[18], %r0, %b[8]
stqp,5 %r34, %b[16], %b[8]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[34], %b[34], %r0, %b[18]
stqp,2 %r35, %b[16], %b[11]
qppermb,3,sm %b[32], %b[32], %r0, %b[16]
stqp,5 %r36, %b[16], %b[10]
}
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Итоги по reverse_radix2_vector2
Победителем можно считать reverse_radix2_vector2_stream16.
При реализации Vector-2 Radix-2 FFT будем использовать его.
reverse_radix2_vector4
1. reverse_radix2_vector4_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix2 вычисляется с помощью цикла.
int reverseNumber_radix2(int number, int bit_count)
{
int answer = 0;
for(int i = 0; i < bit_count; ++i)
{
answer <<= 1;
answer |= number & 1;
number >>= 1;
}
return answer;
}
void reverse_radix2_vector4_etalon(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
for(int64_t i = 0; i < count_out; ++i)
{
int index = reverseNumber_radix2(i, bit_count_out);
myComplex a = data_in[4*i + 0];
myComplex b = data_in[4*i + 1];
myComplex c = data_in[4*i + 2];
myComplex d = data_in[4*i + 3];
data_out[index] = (myComplex4){.real = {a.real, b.real, c.real, d.real}, .imag = {a.imag, b.imag, c.imag, d.imag}};
}
}
2. reverse_radix2_vector4
В процессоре есть инструкция bitrevd, которая производит операцию reverseNumber_radix2 над 64-битным числом. Заменим reverseNumber_radix2() на __builtin_e2k_bitrevd().
Код на Си
void reverse_radix2_vector4(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
int64_t delta = (1LL << shift_out) / 32;
__v2di *in = (__v2di*)data_in;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0, shifted_i = 0; i < count_out; ++i, shifted_i += delta)
{
int64_t offset = __builtin_e2k_bitrevd(shifted_i);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L6535:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
.L6346:
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
bitrevd,0,sm %b[22], %b[13]
qppermb,1,sm %b[11], %b[10], %r5, %b[19]
stqp,2 %r0, %b[17], %b[21]
qppermb,3,sm %b[11], %b[10], %r6, %b[23]
addd,4,sm %b[20], %r4, %b[18]
stqp,5 %r2, %b[17], %b[25]
movaqp,0 area=0, ind=0, am=1, be=0, %b[0]
movaqp,1 area=0, ind=16, am=0, be=0, %b[1]
}
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
3. reverse_radix2_vector4_x2
Сделаем ручную раскрутку цикла в 2 раза.
Код на Си
void reverse_radix2_vector4_x2(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*) data_in;
__v2di *out_0 = (__v2di*)&data_out[0];
__v2di *out_1 = (__v2di*)&data_out[count_out/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; i += 2)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)out_0 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_real);
*(__v2du*)((void*)out_0 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_imag);
*(__v2du*)((void*)out_1 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_real);
*(__v2du*)((void*)out_1 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_imag);
}
}
Основной цикл на ассемблере
.L6884:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32
}
.L6561:
{
loop_mode
bitrevd,0,sm %b[4], %b[1]
qppermb,1,sm %b[25], %b[20], %r7, %b[28]
stqp,2 %r0, %b[7], %b[26]
shld,3,sm %b[22], 0x5, %b[5]
qppermb,4,sm %b[17], %b[12], %r7, %b[27]
stqp,5 %r4, %b[7], %b[24]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[23], %b[18], %r9, %b[24]
addd,1,sm %b[2], 0x2, %b[0]
stqp,2 %r2, %b[7], %b[28]
qppermb,3,sm %b[15], %b[10], %r9, %b[22]
shrd,4,sm %b[3], %r6, %b[20]
stqp,5 %r5, %b[7], %b[27]
movaqp,0 area=0, ind=0, am=1, be=0, %b[12]
movaqp,1 area=0, ind=16, am=0, be=0, %b[17]
movaqp,2 area=0, ind=0, am=1, be=0, %b[4]
movaqp,3 area=0, ind=16, am=0, be=0, %b[9]
}
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит два чтения из одного места памяти и запись в два разных места памяти.
4. reverse_radix2_vector4_stream2
Теперь сделаем наоборот: будем читать из двух разных мест, а писать рядом.
Чтение в 2 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream2(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_0 = (__v2di*)&data_in[0];
__v2di *in_1 = (__v2di*)&data_in[count_in/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/2; ++i)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L7199:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
.L6909:
{
loop_mode
bitrevd,0,sm %b[4], %b[1]
qppermb,1,sm %b[25], %b[20], %r9, %b[28]
stqp,2 %r0, %b[7], %b[26]
shld,3,sm %b[22], 0x5, %b[5]
qppermb,4,sm %b[17], %b[12], %r9, %b[27]
stqp,5 %r5, %b[7], %b[24]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[23], %b[18], %r6, %b[24]
addd,1,sm %b[2], 0x1, %b[0]
stqp,2 %r2, %b[7], %b[28]
qppermb,3,sm %b[15], %b[10], %r6, %b[22]
shrd,4,sm %b[3], %r7, %b[20]
stqp,5 %r4, %b[7], %b[27]
movaqp,0 area=0, ind=0, am=1, be=0, %b[12]
movaqp,1 area=0, ind=16, am=0, be=0, %b[17]
movaqp,2 area=0, ind=0, am=1, be=0, %b[4]
movaqp,3 area=0, ind=16, am=0, be=0, %b[9]
}
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
5. reverse_radix2_vector4_stream4
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream4(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00 = (__v2di*)&data_in[0 * count_in/4];
__v2di *in_01 = (__v2di*)&data_in[1 * count_in/4];
__v2di *in_10 = (__v2di*)&data_in[2 * count_in/4];
__v2di *in_11 = (__v2di*)&data_in[3 * count_in/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/4; ++i)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L7728:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=4, abs=0, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=4, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=0
}
.L7242:
{
loop_mode
qppermb,1,sm %b[20], %b[16], %r13, %b[26]
stqp,2 %r0, %b[21], %b[25]
qppermb,4,sm %b[12], %b[8], %r13, %b[24]
stqp,5 %r5, %b[21], %b[22]
}
{
loop_mode
qppermb,1,sm %b[17], %b[13], %r12, %b[25]
stqp,2 %r2, %b[21], %b[26]
shrd,3,sm %b[19], %r11, %b[1]
qppermb,4,sm %b[9], %b[5], %r12, %b[22]
stqp,5 %r4, %b[21], %b[24]
}
{
loop_mode
qppermb,1,sm %b[17], %b[13], %r13, %b[27]
stqp,2 %r7, %b[21], %b[25]
shld,3,sm %b[1], 0x5, %b[19]
qppermb,4,sm %b[9], %b[5], %r13, %b[24]
stqp,5 %r10, %b[21], %b[22]
movaqp,0 area=0, ind=0, am=1, be=0, %b[12]
movaqp,1 area=0, ind=16, am=0, be=0, %b[16]
movaqp,2 area=0, ind=0, am=1, be=0, %b[4]
movaqp,3 area=0, ind=16, am=0, be=0, %b[8]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[18], %b[14], %r12, %b[23]
addd,1,sm %b[2], 0x1, %b[0]
stqp,2 %r6, %b[21], %b[27]
qppermb,3,sm %b[10], %b[6], %r12, %b[20]
bitrevd,4,sm %b[2], %b[17]
stqp,5 %r9, %b[21], %b[24]
movaqp,0 area=1, ind=0, am=1, be=0, %b[9]
movaqp,1 area=1, ind=16, am=0, be=0, %b[13]
movaqp,2 area=1, ind=0, am=1, be=0, %b[1]
movaqp,3 area=1, ind=16, am=0, be=0, %b[5]
}
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
6. reverse_radix2_vector4_stream8
Чтение в 8 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream8(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_000 = (__v2di*)&data_in[0 * count_in/8];
__v2di *in_001 = (__v2di*)&data_in[1 * count_in/8];
__v2di *in_010 = (__v2di*)&data_in[2 * count_in/8];
__v2di *in_011 = (__v2di*)&data_in[3 * count_in/8];
__v2di *in_100 = (__v2di*)&data_in[4 * count_in/8];
__v2di *in_101 = (__v2di*)&data_in[5 * count_in/8];
__v2di *in_110 = (__v2di*)&data_in[6 * count_in/8];
__v2di *in_111 = (__v2di*)&data_in[7 * count_in/8];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/8; ++i)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L8681:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=3, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=3, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=3, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=3, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=24, disp=0
}
.L7803:
{
loop_mode
qppermb,1,sm %b[16], %b[15], %r20, %b[28]
stqp,2 %r0, %b[9], %b[5]
qppermb,4,sm %b[14], %b[13], %r20, %b[27]
stqp,5 %r6, %b[9], %b[4]
}
{
loop_mode
qppermb,1,sm %b[26], %b[25], %r21, %b[30]
stqp,2 %r2, %b[9], %b[28]
qppermb,4,sm %b[24], %b[23], %r21, %b[29]
stqp,5 %r5, %b[9], %b[27]
movaqp,0 area=3, ind=0, am=1, be=0, %b[5]
movaqp,1 area=3, ind=16, am=0, be=0, %b[8]
movaqp,2 area=3, ind=0, am=1, be=0, %b[1]
movaqp,3 area=3, ind=16, am=0, be=0, %b[4]
}
{
loop_mode
qppermb,1,sm %b[26], %b[25], %r20, %b[28]
stqp,2 %r9, %b[9], %b[30]
qppermb,4,sm %b[24], %b[23], %r20, %b[27]
stqp,5 %r11, %b[9], %b[29]
movaqp,0 area=0, ind=0, am=1, be=0, %b[13]
movaqp,1 area=0, ind=16, am=0, be=0, %b[14]
movaqp,2 area=0, ind=0, am=1, be=0, %b[11]
movaqp,3 area=0, ind=16, am=0, be=0, %b[12]
}
{
loop_mode
qppermb,1,sm %b[22], %b[21], %r20, %b[29]
stqp,2 %r7, %b[9], %b[28]
qppermb,4,sm %b[22], %b[21], %r21, %b[26]
stqp,5 %r10, %b[9], %b[27]
movaqp,0 area=2, ind=0, am=1, be=0, %b[19]
movaqp,1 area=2, ind=16, am=0, be=0, %b[20]
movaqp,2 area=2, ind=0, am=1, be=0, %b[15]
movaqp,3 area=2, ind=16, am=0, be=0, %b[16]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[18], %b[17], %r20, %b[28]
stqp,2 %r12, %b[9], %b[29]
bitrevd,3,sm %b[2], %b[25]
qppermb,4,sm %b[18], %b[17], %r21, %b[27]
stqp,5 %r13, %b[9], %b[26]
movaqp,0 area=1, ind=0, am=1, be=0, %b[23]
movaqp,1 area=1, ind=16, am=0, be=0, %b[24]
movaqp,2 area=1, ind=0, am=1, be=0, %b[21]
movaqp,3 area=1, ind=16, am=0, be=0, %b[22]
}
{
loop_mode
qppermb,1,sm %b[10], %b[7], %r20, %b[29]
stqp,2 %r14, %b[9], %b[28]
shrd,3,sm %b[25], %r22, %b[2]
qppermb,4,sm %b[10], %b[7], %r21, %b[26]
stqp,5 %r15, %b[9], %b[27]
}
{
loop_mode
qppermb,1,sm %b[6], %b[3], %r20, %b[27]
stqp,2 %r16, %b[9], %b[29]
shld,3,sm %b[2], 0x5, %b[7]
qppermb,4,sm %b[6], %b[3], %r21, %b[25]
stqp,5 %r17, %b[9], %b[26]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[14], %b[13], %r21, %b[3]
stqp,2 %r18, %b[9], %b[27]
qppermb,3,sm %b[12], %b[11], %r21, %b[2]
stqp,5 %r19, %b[9], %b[25]
}
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
7. reverse_radix2_vector4_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream16(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_0000 = (__v2di*)&data_in[ 0 * count_in/16];
__v2di *in_0001 = (__v2di*)&data_in[ 1 * count_in/16];
__v2di *in_0010 = (__v2di*)&data_in[ 2 * count_in/16];
__v2di *in_0011 = (__v2di*)&data_in[ 3 * count_in/16];
__v2di *in_0100 = (__v2di*)&data_in[ 4 * count_in/16];
__v2di *in_0101 = (__v2di*)&data_in[ 5 * count_in/16];
__v2di *in_0110 = (__v2di*)&data_in[ 6 * count_in/16];
__v2di *in_0111 = (__v2di*)&data_in[ 7 * count_in/16];
__v2di *in_1000 = (__v2di*)&data_in[ 8 * count_in/16];
__v2di *in_1001 = (__v2di*)&data_in[ 9 * count_in/16];
__v2di *in_1010 = (__v2di*)&data_in[10 * count_in/16];
__v2di *in_1011 = (__v2di*)&data_in[11 * count_in/16];
__v2di *in_1100 = (__v2di*)&data_in[12 * count_in/16];
__v2di *in_1101 = (__v2di*)&data_in[13 * count_in/16];
__v2di *in_1110 = (__v2di*)&data_in[14 * count_in/16];
__v2di *in_1111 = (__v2di*)&data_in[15 * count_in/16];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/16; ++i)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0000[2*i+1], in_0000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0000[2*i+1], in_0000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1000[2*i+1], in_1000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1000[2*i+1], in_1000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_0100[2*i+1], in_0100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_0100[2*i+1], in_0100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_1100[2*i+1], in_1100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_1100[2*i+1], in_1100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_0010[2*i+1], in_0010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_0010[2*i+1], in_0010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_1010[2*i+1], in_1010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_1010[2*i+1], in_1010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_0110[2*i+1], in_0110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_0110[2*i+1], in_0110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_1110[2*i+1], in_1110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_1110[2*i+1], in_1110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_0001[2*i+1], in_0001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_0001[2*i+1], in_0001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_1001[2*i+1], in_1001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_1001[2*i+1], in_1001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_0101[2*i+1], in_0101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_0101[2*i+1], in_0101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_1101[2*i+1], in_1101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_1101[2*i+1], in_1101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_0011[2*i+1], in_0011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_0011[2*i+1], in_0011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_1011[2*i+1], in_1011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_1011[2*i+1], in_1011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_0111[2*i+1], in_0111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_0111[2*i+1], in_0111[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_1111[2*i+1], in_1111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_1111[2*i+1], in_1111[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L10479:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=2, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=2, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=2, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=2, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=2, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=2, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=2, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=2, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=2, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=2, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=2, abs=24, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=2, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=28, disp=0
}
.L8820:
{
loop_mode
qppermb,1,sm %b[34], %b[33], %r36, %b[32]
stqp,2 %r0, %b[4], %b[44]
qppermb,4,sm %b[32], %b[31], %r36, %b[31]
stqp,5 %r6, %b[4], %b[43]
}
{
loop_mode
qppermb,1,sm %b[42], %b[41], %r37, %b[32]
stqp,2 %r2, %b[4], %b[32]
qppermb,4,sm %b[40], %b[39], %r37, %b[31]
stqp,5 %r1, %b[4], %b[31]
}
{
loop_mode
qppermb,1,sm %b[42], %b[41], %r36, %b[32]
stqp,2 %r9, %b[4], %b[32]
qppermb,4,sm %b[40], %b[39], %r36, %b[31]
stqp,5 %r11, %b[4], %b[31]
}
{
loop_mode
qppermb,1,sm %b[38], %b[37], %r36, %b[32]
stqp,2 %r7, %b[4], %b[32]
qppermb,4,sm %b[38], %b[37], %r37, %b[31]
stqp,5 %r10, %b[4], %b[31]
}
{
loop_mode
qppermb,1,sm %b[36], %b[35], %r36, %b[32]
stqp,2 %r12, %b[4], %b[32]
qppermb,4,sm %b[36], %b[35], %r37, %b[31]
stqp,5 %r13, %b[4], %b[31]
}
{
loop_mode
qppermb,1,sm %b[30], %b[29], %r36, %b[30]
stqp,2 %r14, %b[4], %b[32]
qppermb,4,sm %b[30], %b[29], %r37, %b[29]
stqp,5 %r15, %b[4], %b[31]
movaqp,0 area=7, ind=0, am=1, be=0, %b[5]
movaqp,1 area=7, ind=16, am=0, be=0, %b[6]
movaqp,2 area=7, ind=0, am=1, be=0, %b[0]
movaqp,3 area=7, ind=16, am=0, be=0, %b[1]
}
{
loop_mode
qppermb,1,sm %b[26], %b[25], %r36, %b[26]
stqp,2 %r16, %b[4], %b[30]
qppermb,4,sm %b[26], %b[25], %r37, %b[25]
stqp,5 %r17, %b[4], %b[29]
movaqp,0 area=6, ind=0, am=1, be=0, %b[13]
movaqp,1 area=6, ind=16, am=0, be=0, %b[14]
movaqp,2 area=6, ind=0, am=1, be=0, %b[9]
movaqp,3 area=6, ind=16, am=0, be=0, %b[10]
}
{
loop_mode
qppermb,1,sm %b[28], %b[27], %r36, %b[26]
stqp,2 %r18, %b[4], %b[26]
qppermb,4,sm %b[28], %b[27], %r37, %b[25]
stqp,5 %r19, %b[4], %b[25]
movaqp,0 area=5, ind=0, am=1, be=0, %b[21]
movaqp,1 area=5, ind=16, am=0, be=0, %b[22]
movaqp,2 area=5, ind=0, am=1, be=0, %b[17]
movaqp,3 area=5, ind=16, am=0, be=0, %b[18]
}
{
loop_mode
qppermb,1,sm %g16, %g17, %r36, %b[28]
stqp,2 %r20, %b[4], %b[26]
qppermb,4,sm %g16, %g17, %r37, %b[27]
stqp,5 %r21, %b[4], %b[25]
movaqp,0 area=4, ind=0, am=1, be=0, %b[25]
movaqp,1 area=4, ind=16, am=0, be=0, %b[26]
movaqp,2 area=4, ind=0, am=1, be=0, %g17
movaqp,3 area=4, ind=16, am=0, be=0, %g16
}
{
loop_mode
qppermb,1,sm %b[24], %b[23], %r36, %b[30]
stqp,2 %r22, %b[4], %b[28]
qppermb,4,sm %b[24], %b[23], %r37, %b[29]
stqp,5 %r23, %b[4], %b[27]
movaqp,0 area=3, ind=0, am=1, be=0, %b[27]
movaqp,1 area=3, ind=16, am=0, be=0, %b[28]
movaqp,2 area=3, ind=0, am=1, be=0, %b[23]
movaqp,3 area=3, ind=16, am=0, be=0, %b[24]
}
{
loop_mode
qppermb,1,sm %b[20], %b[19], %r36, %b[34]
stqp,2 %r24, %b[4], %b[30]
qppermb,4,sm %b[20], %b[19], %r37, %b[33]
stqp,5 %r25, %b[4], %b[29]
movaqp,0 area=0, ind=0, am=1, be=0, %b[31]
movaqp,1 area=0, ind=16, am=0, be=0, %b[32]
movaqp,2 area=0, ind=0, am=1, be=0, %b[29]
movaqp,3 area=0, ind=16, am=0, be=0, %b[30]
}
{
loop_mode
qppermb,1,sm %b[16], %b[15], %r36, %b[38]
stqp,2 %r26, %b[4], %b[34]
qppermb,4,sm %b[16], %b[15], %r37, %b[37]
stqp,5 %r27, %b[4], %b[33]
movaqp,0 area=2, ind=0, am=1, be=0, %b[35]
movaqp,1 area=2, ind=16, am=0, be=0, %b[36]
movaqp,2 area=2, ind=0, am=1, be=0, %b[33]
movaqp,3 area=2, ind=16, am=0, be=0, %b[34]
}
{
loop_mode
addd,0,sm %r5, 0x1, %r5
qppermb,1,sm %b[12], %b[11], %r36, %b[43]
stqp,2 %r28, %b[4], %b[38]
bitrevd,3,sm %r5, %b[41]
qppermb,4,sm %b[12], %b[11], %r37, %b[42]
stqp,5 %r29, %b[4], %b[37]
movaqp,0 area=1, ind=0, am=1, be=0, %b[39]
movaqp,1 area=1, ind=16, am=0, be=0, %b[40]
movaqp,2 area=1, ind=0, am=1, be=0, %b[37]
movaqp,3 area=1, ind=16, am=0, be=0, %b[38]
}
{
loop_mode
qppermb,1,sm %b[8], %b[7], %r36, %b[43]
stqp,2 %r30, %b[4], %b[43]
shrd,3,sm %b[41], %r38, %b[41]
qppermb,4,sm %b[8], %b[7], %r37, %b[42]
stqp,5 %r31, %b[4], %b[42]
}
{
loop_mode
qppermb,1,sm %b[3], %b[2], %r36, %b[42]
stqp,2 %r32, %b[4], %b[43]
shld,3,sm %b[41], 0x5, %b[2]
qppermb,4,sm %b[3], %b[2], %r37, %b[41]
stqp,5 %r33, %b[4], %b[42]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[32], %b[31], %r37, %b[42]
stqp,2 %r34, %b[4], %b[42]
qppermb,3,sm %b[30], %b[29], %r37, %b[41]
stqp,5 %r35, %b[4], %b[41]
}
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
8. reverse_radix2_vector4_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream32(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00000 = (__v2di*)&data_in[ 0 * count_in/32];
__v2di *in_00001 = (__v2di*)&data_in[ 1 * count_in/32];
__v2di *in_00010 = (__v2di*)&data_in[ 2 * count_in/32];
__v2di *in_00011 = (__v2di*)&data_in[ 3 * count_in/32];
__v2di *in_00100 = (__v2di*)&data_in[ 4 * count_in/32];
__v2di *in_00101 = (__v2di*)&data_in[ 5 * count_in/32];
__v2di *in_00110 = (__v2di*)&data_in[ 6 * count_in/32];
__v2di *in_00111 = (__v2di*)&data_in[ 7 * count_in/32];
__v2di *in_01000 = (__v2di*)&data_in[ 8 * count_in/32];
__v2di *in_01001 = (__v2di*)&data_in[ 9 * count_in/32];
__v2di *in_01010 = (__v2di*)&data_in[10 * count_in/32];
__v2di *in_01011 = (__v2di*)&data_in[11 * count_in/32];
__v2di *in_01100 = (__v2di*)&data_in[12 * count_in/32];
__v2di *in_01101 = (__v2di*)&data_in[13 * count_in/32];
__v2di *in_01110 = (__v2di*)&data_in[14 * count_in/32];
__v2di *in_01111 = (__v2di*)&data_in[15 * count_in/32];
__v2di *in_10000 = (__v2di*)&data_in[16 * count_in/32];
__v2di *in_10001 = (__v2di*)&data_in[17 * count_in/32];
__v2di *in_10010 = (__v2di*)&data_in[18 * count_in/32];
__v2di *in_10011 = (__v2di*)&data_in[19 * count_in/32];
__v2di *in_10100 = (__v2di*)&data_in[20 * count_in/32];
__v2di *in_10101 = (__v2di*)&data_in[21 * count_in/32];
__v2di *in_10110 = (__v2di*)&data_in[22 * count_in/32];
__v2di *in_10111 = (__v2di*)&data_in[23 * count_in/32];
__v2di *in_11000 = (__v2di*)&data_in[24 * count_in/32];
__v2di *in_11001 = (__v2di*)&data_in[25 * count_in/32];
__v2di *in_11010 = (__v2di*)&data_in[26 * count_in/32];
__v2di *in_11011 = (__v2di*)&data_in[27 * count_in/32];
__v2di *in_11100 = (__v2di*)&data_in[28 * count_in/32];
__v2di *in_11101 = (__v2di*)&data_in[29 * count_in/32];
__v2di *in_11110 = (__v2di*)&data_in[30 * count_in/32];
__v2di *in_11111 = (__v2di*)&data_in[31 * count_in/32];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32; ++i)
{
int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out);
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00000[2*i+1], in_00000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00000[2*i+1], in_00000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10000[2*i+1], in_10000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10000[2*i+1], in_10000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01000[2*i+1], in_01000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_01000[2*i+1], in_01000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_11000[2*i+1], in_11000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11000[2*i+1], in_11000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_00100[2*i+1], in_00100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_00100[2*i+1], in_00100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_10100[2*i+1], in_10100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_10100[2*i+1], in_10100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_01100[2*i+1], in_01100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_01100[2*i+1], in_01100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_11100[2*i+1], in_11100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_11100[2*i+1], in_11100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_00010[2*i+1], in_00010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_00010[2*i+1], in_00010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_10010[2*i+1], in_10010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_10010[2*i+1], in_10010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_01010[2*i+1], in_01010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_01010[2*i+1], in_01010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_11010[2*i+1], in_11010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_11010[2*i+1], in_11010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_00110[2*i+1], in_00110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_00110[2*i+1], in_00110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_10110[2*i+1], in_10110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_10110[2*i+1], in_10110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_01110[2*i+1], in_01110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_01110[2*i+1], in_01110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_11110[2*i+1], in_11110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_11110[2*i+1], in_11110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_00001[2*i+1], in_00001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_00001[2*i+1], in_00001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_10001[2*i+1], in_10001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_10001[2*i+1], in_10001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_01001[2*i+1], in_01001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_01001[2*i+1], in_01001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_11001[2*i+1], in_11001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_11001[2*i+1], in_11001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_00101[2*i+1], in_00101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_00101[2*i+1], in_00101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_10101[2*i+1], in_10101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_10101[2*i+1], in_10101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_01101[2*i+1], in_01101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_01101[2*i+1], in_01101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_11101[2*i+1], in_11101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_11101[2*i+1], in_11101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_00011[2*i+1], in_00011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_00011[2*i+1], in_00011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_10011[2*i+1], in_10011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_10011[2*i+1], in_10011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_01011[2*i+1], in_01011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_01011[2*i+1], in_01011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_11011[2*i+1], in_11011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_11011[2*i+1], in_11011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_00111[2*i+1], in_00111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_00111[2*i+1], in_00111[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_10111[2*i+1], in_10111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_10111[2*i+1], in_10111[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_01111[2*i+1], in_01111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_01111[2*i+1], in_01111[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_11111[2*i+1], in_11111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_11111[2*i+1], in_11111[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L13622:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0
}
.L13394:
{
loop_mode
disp %ctpr1, .L13394
bitrevd,0 %r1, %g20
addd,1 %r1, 0x1, %r1
movaqp,0 area=0, ind=16, am=1, be=0, %g17
movaqp,1 area=0, ind=0, am=0, be=0, %g16
movaqp,2 area=0, ind=16, am=1, be=0, %g19
movaqp,3 area=0, ind=0, am=0, be=0, %g18
}
{
loop_mode
disp %ctpr3, .L12302
shrd,0 %g20, %r6, %g20
movaqp,0 area=1, ind=16, am=1, be=0, %g22
movaqp,1 area=1, ind=0, am=0, be=0, %g21
movaqp,2 area=1, ind=16, am=1, be=0, %g24
movaqp,3 area=1, ind=0, am=0, be=0, %g23
}
{
loop_mode
shld,0 %g20, 0x5, %g20
movaqp,0 area=2, ind=16, am=1, be=0, %g26
movaqp,1 area=2, ind=0, am=0, be=0, %g25
movaqp,2 area=2, ind=16, am=1, be=0, %g28
movaqp,3 area=2, ind=0, am=0, be=0, %g27
}
{
loop_mode
addd,0 %r2, %g20, %r7
movaqp,0 area=3, ind=16, am=1, be=0, %g30
movaqp,1 area=3, ind=0, am=0, be=0, %g29
movaqp,2 area=3, ind=16, am=1, be=0, %r4
movaqp,3 area=3, ind=0, am=0, be=0, %g31
}
{
loop_mode
movaqp,0 area=4, ind=16, am=1, be=0, %r10
movaqp,1 area=4, ind=0, am=0, be=0, %r9
movaqp,2 area=4, ind=16, am=1, be=0, %r12
movaqp,3 area=4, ind=0, am=0, be=0, %r11
}
{
loop_mode
qppermb,0 %g17, %g16, %r0, %r17
qppermb,1 %g17, %g16, %r5, %g16
qppermb,3 %g19, %g18, %r0, %g17
qppermb,4 %g19, %g18, %r5, %g18
movaqp,0 area=5, ind=16, am=1, be=0, %r14
movaqp,1 area=5, ind=0, am=0, be=0, %r13
movaqp,2 area=5, ind=16, am=1, be=0, %r16
movaqp,3 area=5, ind=0, am=0, be=0, %r15
}
{
loop_mode
qppermb,0 %g22, %g21, %r0, %r21
qppermb,1 %g22, %g21, %r5, %g21
stqp,2 0x10, %r7, %g16
qppermb,3 %g24, %g23, %r0, %g22
qppermb,4 %g24, %g23, %r5, %g23
stqp,5 %r7, _f16s,_lts0lo 0x30, %g18
movaqp,0 area=6, ind=16, am=1, be=0, %r18
movaqp,1 area=6, ind=0, am=0, be=0, %g19
movaqp,2 area=6, ind=16, am=1, be=0, %r20
movaqp,3 area=6, ind=0, am=0, be=0, %r19
}
{
loop_mode
qppermb,0 %g26, %g25, %r0, %g17
qppermb,1 %g26, %g25, %r5, %g20
stqp,2 %r2, %g20, %r17
qppermb,3 %g28, %g27, %r0, %g25
qppermb,4 %g28, %g27, %r5, %g26
stqp,5 %r7, _f16s,_lts0lo 0x20, %g17
movaqp,0 area=7, ind=16, am=1, be=0, %g18
movaqp,1 area=7, ind=0, am=0, be=0, %g16
movaqp,2 area=7, ind=16, am=1, be=0, %r22
movaqp,3 area=7, ind=0, am=0, be=0, %g24
}
{
loop_mode
qppermb,0 %g30, %g29, %r0, %r24
qppermb,1 %g30, %g29, %r5, %g29
stqp,2 %r7, _f16s,_lts0lo 0x50, %g21
qppermb,3 %r4, %g31, %r0, %g21
qppermb,4 %r4, %g31, %r5, %g23
stqp,5 %r7, _f16s,_lts0hi 0x70, %g23
movaqp,0 area=8, ind=16, am=1, be=0, %g28
movaqp,1 area=8, ind=0, am=0, be=0, %g27
movaqp,2 area=8, ind=16, am=1, be=0, %r23
movaqp,3 area=8, ind=0, am=0, be=0, %r17
}
{
loop_mode
qppermb,0 %r10, %r9, %r0, %g22
qppermb,1 %r10, %r9, %r5, %r9
stqp,2 %r7, _f16s,_lts0lo 0x40, %r21
qppermb,3 %r12, %r11, %r0, %r10
qppermb,4 %r12, %r11, %r5, %r11
stqp,5 %r7, _f16s,_lts0hi 0x60, %g22
movaqp,0 area=9, ind=16, am=1, be=0, %g31
movaqp,1 area=9, ind=0, am=0, be=0, %g30
movaqp,2 area=9, ind=16, am=1, be=0, %r25
movaqp,3 area=9, ind=0, am=0, be=0, %r4
}
{
loop_mode
qppermb,0 %r14, %r13, %r0, %g17
qppermb,1 %r14, %r13, %r5, %g20
stqp,2 %r7, _f16s,_lts0lo 0x80, %g17
qppermb,3 %r16, %r15, %r0, %r13
qppermb,4 %r16, %r15, %r5, %r14
stqp,5 %r7, _f16s,_lts0hi 0x90, %g20
movaqp,0 area=10, ind=16, am=1, be=0, %r21
movaqp,1 area=10, ind=0, am=0, be=0, %r12
movaqp,2 area=10, ind=16, am=1, be=0, %r27
movaqp,3 area=10, ind=0, am=0, be=0, %r26
}
{
loop_mode
qppermb,0 %r18, %g19, %r0, %r28
qppermb,1 %r18, %g19, %r5, %g19
stqp,2 %r7, _f16s,_lts0lo 0xa0, %g25
qppermb,3 %r20, %r19, %r0, %r18
qppermb,4 %r20, %r19, %r5, %r19
stqp,5 %r7, _f16s,_lts0hi 0xb0, %g26
movaqp,0 area=11, ind=16, am=1, be=0, %r16
movaqp,1 area=11, ind=0, am=0, be=0, %r15
movaqp,2 area=11, ind=16, am=1, be=0, %g26
movaqp,3 area=11, ind=0, am=0, be=0, %g25
}
{
loop_mode
qppermb,0 %g18, %g16, %r0, %r30
qppermb,1 %g18, %g16, %r5, %g16
stqp,2 %r7, _f16s,_lts0lo 0xc0, %r24
qppermb,3 %r22, %g24, %r0, %g18
qppermb,4 %r22, %g24, %r5, %g24
stqp,5 %r7, _f16s,_lts0hi 0xd0, %g29
movaqp,0 area=12, ind=16, am=1, be=0, %r20
movaqp,1 area=12, ind=0, am=0, be=0, %g29
movaqp,2 area=12, ind=16, am=1, be=0, %r29
movaqp,3 area=12, ind=0, am=0, be=0, %r24
}
{
loop_mode
qppermb,0 %g28, %g27, %r0, %r32
qppermb,1 %g28, %g27, %r5, %g27
stqp,2 %r7, _f16s,_lts0lo 0xe0, %g21
qppermb,3 %r23, %r17, %r0, %g28
qppermb,4 %r23, %r17, %r5, %r17
stqp,5 %r7, _f16s,_lts0hi 0xf0, %g23
movaqp,0 area=13, ind=16, am=1, be=0, %g23
movaqp,1 area=13, ind=0, am=0, be=0, %g21
movaqp,2 area=13, ind=16, am=1, be=0, %r31
movaqp,3 area=13, ind=0, am=0, be=0, %r22
}
{
loop_mode
qppermb,0 %g31, %g30, %r0, %r34
qppermb,1 %g31, %g30, %r5, %g30
stqp,2 %r7, _f16s,_lts0lo 0x100, %g22
qppermb,3 %r25, %r4, %r0, %g31
qppermb,4 %r25, %r4, %r5, %r4
stqp,5 %r7, _f16s,_lts0hi 0x110, %r9
movaqp,0 area=14, ind=16, am=1, be=0, %r9
movaqp,1 area=14, ind=0, am=0, be=0, %g22
movaqp,2 area=14, ind=16, am=1, be=0, %r33
movaqp,3 area=14, ind=0, am=0, be=0, %r23
}
{
loop_mode
qppermb,0 %r21, %r12, %r0, %r36
qppermb,1 %r21, %r12, %r5, %r12
stqp,2 %r7, _f16s,_lts0lo 0x120, %r10
qppermb,3 %r27, %r26, %r0, %r21
qppermb,4 %r27, %r26, %r5, %r26
stqp,5 %r7, _f16s,_lts0hi 0x130, %r11
movaqp,0 area=15, ind=16, am=1, be=0, %r11
movaqp,1 area=15, ind=0, am=0, be=0, %r10
movaqp,2 area=15, ind=16, am=1, be=0, %r35
movaqp,3 area=15, ind=0, am=0, be=0, %r25
}
{
loop_mode
qppermb,0 %r16, %r15, %r0, %g17
qppermb,1 %r16, %r15, %r5, %g20
stqp,2 %r7, _f16s,_lts0lo 0x140, %g17
qppermb,3 %g26, %g25, %r0, %r15
qppermb,4 %g26, %g25, %r5, %g25
stqp,5 %r7, _f16s,_lts0hi 0x150, %g20
}
{
loop_mode
qppermb,0 %r20, %g29, %r0, %g26
qppermb,1 %r20, %g29, %r5, %g29
stqp,2 %r7, _f16s,_lts0lo 0x160, %r13
qppermb,3 %r29, %r24, %r0, %r13
qppermb,4 %r29, %r24, %r5, %r14
stqp,5 %r7, _f16s,_lts0hi 0x170, %r14
}
{
loop_mode
qppermb,0 %g23, %g21, %r0, %g19
qppermb,1 %g23, %g21, %r5, %g21
stqp,2 %r7, _f16s,_lts0lo 0x180, %r28
qppermb,3 %r31, %r22, %r0, %g23
qppermb,4 %r31, %r22, %r5, %r16
stqp,5 %r7, _f16s,_lts0hi 0x190, %g19
}
{
loop_mode
qppermb,0 %r9, %g22, %r0, %r18
qppermb,1 %r9, %g22, %r5, %g22
stqp,2 %r7, _f16s,_lts0lo 0x1a0, %r18
qppermb,3 %r33, %r23, %r0, %r9
qppermb,4 %r33, %r23, %r5, %r19
stqp,5 %r7, _f16s,_lts0hi 0x1b0, %r19
}
{
loop_mode
qppermb,0 %r11, %r10, %r0, %g16
qppermb,1 %r11, %r10, %r5, %r10
stqp,2 %r7, _f16s,_lts0lo 0x1c0, %r30
qppermb,3 %r35, %r25, %r0, %r11
qppermb,4 %r35, %r25, %r5, %r20
stqp,5 %r7, _f16s,_lts0hi 0x1d0, %g16
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x1e0, %g18
stqp,5 %r7, _f16s,_lts0hi 0x1f0, %g24
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x200, %r32
stqp,5 %r7, _f16s,_lts0hi 0x210, %g27
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x220, %g28
stqp,5 %r7, _f16s,_lts0hi 0x230, %r17
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x240, %r34
stqp,5 %r7, _f16s,_lts0hi 0x250, %g30
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x260, %g31
stqp,5 %r7, _f16s,_lts0hi 0x270, %r4
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x280, %r36
stqp,5 %r7, _f16s,_lts0hi 0x290, %r12
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x2a0, %r21
stqp,5 %r7, _f16s,_lts0hi 0x2b0, %r26
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x2c0, %g17
stqp,5 %r7, _f16s,_lts0hi 0x2d0, %g20
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x2e0, %r15
stqp,5 %r7, _f16s,_lts0hi 0x2f0, %g25
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x300, %g26
stqp,5 %r7, _f16s,_lts0hi 0x310, %g29
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x320, %r13
stqp,5 %r7, _f16s,_lts0hi 0x330, %r14
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x340, %g19
stqp,5 %r7, _f16s,_lts0hi 0x350, %g21
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x360, %g23
stqp,5 %r7, _f16s,_lts0hi 0x370, %r16
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x380, %r18
stqp,5 %r7, _f16s,_lts0hi 0x390, %g22
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x3a0, %r9
stqp,5 %r7, _f16s,_lts0hi 0x3b0, %r19
}
{
loop_mode
stqp,2 %r7, _f16s,_lts0lo 0x3c0, %g16
stqp,5 %r7, _f16s,_lts0hi 0x3d0, %r10
}
{
loop_mode
ct %ctpr1 ? %NOT_LOOP_END
alc alcf=0, alct=1
stqp,2 %r7, _f16s,_lts0lo 0x3e0, %r11
stqp,5 %r7, _f16s,_lts0hi 0x3f0, %r20
}
{
loop_mode
ct %ctpr3
alc alcf=0, alct=1
}
Теоретическая скорость: 128 комплексных чисел за 39 тактов (128/39) = 26.26 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Итоги по reverse_radix2_vector4
Победителем можно считать reverse_radix2_vector4_stream8.
При реализации Vector-4 Radix-2 FFT будем использовать его.
reverse_radix4_vector2
1. reverse_radix4_vector2_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix4 вычисляется с помощью цикла.
int reverseNumber_radix4(int number, int bit_count)
{
int answer = 0;
for(int i = 0; i < bit_count/2; ++i)
{
answer <<= 2;
answer |= number & 3;
number >>= 2;
}
return answer;
}
void reverse_radix4_vector2_etalon(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
for(int64_t i = 0; i < count_out; ++i)
{
int index = reverseNumber_radix4(i, bit_count_out);
myComplex a = data_in[2*i + 0];
myComplex b = data_in[2*i + 1];
data_out[index] = (myComplex2){.real = {a.real, b.real}, .imag = {a.imag, b.imag}};
}
}
2. reverse_radix4_vector2
В процессоре нет готовых инструкций, производящих операцию reverseNumber_radix4.
Поэтому выполним инструкцию bitrevd и переставим соседние биты местами.
Код на Си
void reverse_radix4_vector2(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*)data_in;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset) = __builtin_e2k_qppermb(in[i], in[i], mask);
}
}
Основной цикл на ассемблере
.L4546:
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0
}
.L4397:
{
loop_mode
bitrevd,0,sm %b[10], %b[1]
shr_andd,1,sm %b[3], %r5, %r6, %b[9]
ord,2,sm %b[11], %b[7], %b[12]
shld,3,sm %b[14], 0x4, %b[15]
qppermb,4,sm %b[6], %b[6], %r0, %b[13]
addd,5,sm %b[10], 0x1, %b[8]
movaqp,1 area=0, ind=0, am=1, be=0, %b[0]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
shr_andd,1,sm %b[1], %r4, %r7, %b[3]
stqp,5 %r2, %b[15], %b[13]
}
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
3. reverse_radix4_vector2_x4
Сделаем ручную раскрутку цикла в 4 раза.
Код на Си
void reverse_radix4_vector2_x4(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*) data_in;
__v2di *out_0 = (__v2di*)&data_out[0 * count_out/4];
__v2di *out_1 = (__v2di*)&data_out[1 * count_out/4];
__v2di *out_2 = (__v2di*)&data_out[2 * count_out/4];
__v2di *out_3 = (__v2di*)&data_out[3 * count_out/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; i += 4)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)out_0 + offset) = __builtin_e2k_qppermb(in[i + 0], in[i + 0], mask);
*(__v2du*)((void*)out_1 + offset) = __builtin_e2k_qppermb(in[i + 1], in[i + 1], mask);
*(__v2du*)((void*)out_2 + offset) = __builtin_e2k_qppermb(in[i + 2], in[i + 2], mask);
*(__v2du*)((void*)out_3 + offset) = __builtin_e2k_qppermb(in[i + 3], in[i + 3], mask);
}
}
Основной цикл на ассемблере
.L4888:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32
}
.L4593:
{
loop_mode
qppermb,0,sm %b[11], %b[11], %r7, %b[20]
shr_andd,1,sm %b[26], %r9, %r11, %b[16]
qppermb,3,sm %b[12], %b[12], %r7, %b[23]
shr_andd,4,sm %b[26], %r6, %r10, %b[19]
addd,5,sm %b[17], 0x4, %b[15]
movaqp,0 area=0, ind=16, am=1, be=0, %b[7]
movaqp,1 area=0, ind=0, am=0, be=0, %b[8]
movaqp,2 area=0, ind=16, am=1, be=0, %b[0]
movaqp,3 area=0, ind=0, am=0, be=0, %b[1]
}
{
loop_mode
bitrevd,0,sm %b[17], %b[24]
qppermb,1,sm %b[4], %b[4], %r7, %b[11]
stqp,2 %r5, %b[6], %b[22]
ord,3,sm %b[21], %b[18], %b[27]
qppermb,4,sm %b[5], %b[5], %r7, %b[12]
stqp,5 %r2, %b[6], %b[25]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
stqp,2 %r0, %b[6], %b[13]
shld,4,sm %b[27], 0x4, %b[4]
stqp,5 %r4, %b[6], %b[14]
}
Теоретическая скорость: 8 комплексных чисел за 3 такта (8/3) = 21.33 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит четыре чтения из одного места памяти и запись в четыре разных места памяти.
4. reverse_radix4_vector2_stream4
Теперь сделаем наоборот: будем читать из четырёх разных мест, а писать рядом.
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream4(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_0 = (__v2di*)&data_in[0 * count_in/4];
__v2di *in_1 = (__v2di*)&data_in[1 * count_in/4];
__v2di *in_2 = (__v2di*)&data_in[2 * count_in/4];
__v2di *in_3 = (__v2di*)&data_in[3 * count_in/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/4; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[i], in_0[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1[i], in_1[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_2[i], in_2[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_3[i], in_3[i], mask);
}
}
Основной цикл на ассемблере
.L5211:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=0, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=4, abs=16, disp=0
}
.L4931:
{
loop_mode
qppermb,0,sm %b[11], %b[11], %r7, %b[20]
shr_andd,1,sm %b[26], %r9, %r11, %b[16]
qppermb,3,sm %b[12], %b[12], %r7, %b[23]
shr_andd,4,sm %b[26], %r6, %r10, %b[19]
addd,5,sm %b[17], 0x1, %b[15]
movaqp,0 area=1, ind=0, am=1, be=0, %b[1]
movaqp,1 area=0, ind=0, am=1, be=0, %b[8]
movaqp,2 area=1, ind=0, am=1, be=0, %b[0]
movaqp,3 area=0, ind=0, am=1, be=0, %b[7]
}
{
loop_mode
bitrevd,0,sm %b[17], %b[24]
qppermb,1,sm %b[4], %b[4], %r7, %b[11]
stqp,2 %r0, %b[6], %b[22]
ord,3,sm %b[21], %b[18], %b[27]
qppermb,4,sm %b[5], %b[5], %r7, %b[12]
stqp,5 %r2, %b[6], %b[25]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
stqp,2 %r5, %b[6], %b[13]
shld,4,sm %b[27], 0x4, %b[4]
stqp,5 %r4, %b[6], %b[14]
}
Теоретическая скорость: 8 комплексных чисел за 3 такта (8/3) = 21.33 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix4_vector2_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream16(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00 = (__v2di*)&data_in[ 0 * count_in/16];
__v2di *in_01 = (__v2di*)&data_in[ 1 * count_in/16];
__v2di *in_02 = (__v2di*)&data_in[ 2 * count_in/16];
__v2di *in_03 = (__v2di*)&data_in[ 3 * count_in/16];
__v2di *in_10 = (__v2di*)&data_in[ 4 * count_in/16];
__v2di *in_11 = (__v2di*)&data_in[ 5 * count_in/16];
__v2di *in_12 = (__v2di*)&data_in[ 6 * count_in/16];
__v2di *in_13 = (__v2di*)&data_in[ 7 * count_in/16];
__v2di *in_20 = (__v2di*)&data_in[ 8 * count_in/16];
__v2di *in_21 = (__v2di*)&data_in[ 9 * count_in/16];
__v2di *in_22 = (__v2di*)&data_in[10 * count_in/16];
__v2di *in_23 = (__v2di*)&data_in[11 * count_in/16];
__v2di *in_30 = (__v2di*)&data_in[12 * count_in/16];
__v2di *in_31 = (__v2di*)&data_in[13 * count_in/16];
__v2di *in_32 = (__v2di*)&data_in[14 * count_in/16];
__v2di *in_33 = (__v2di*)&data_in[15 * count_in/16];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/16; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[i], in_00[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10[i], in_10[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_20[i], in_20[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_30[i], in_30[i], mask);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01[i], in_01[i], mask);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_11[i], in_11[i], mask);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_21[i], in_21[i], mask);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_31[i], in_31[i], mask);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_02[i], in_02[i], mask);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_12[i], in_12[i], mask);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_22[i], in_22[i], mask);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_32[i], in_32[i], mask);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_03[i], in_03[i], mask);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_13[i], in_13[i], mask);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_23[i], in_23[i], mask);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_33[i], in_33[i], mask);
}
}
Основной цикл на ассемблере
.L6141:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=2, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=2, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=2, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=2, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=2, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=2, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=2, abs=24, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=2, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=2, abs=28, disp=0
}
.L5350:
{
loop_mode
qppermb,0,sm %b[12], %b[12], %r20, %b[46]
shr_andd,1,sm %b[24], %r21, %r24, %b[1]
stqp,2 %r0, %b[21], %b[4]
qppermb,3,sm %b[17], %b[17], %r20, %b[47]
shr_andd,4,sm %b[24], %r22, %r23, %b[5]
stqp,5 %r2, %b[21], %b[16]
}
{
loop_mode
qppermb,1,sm %b[41], %b[41], %r20, %b[4]
stqp,2 %r6, %b[21], %b[46]
qppermb,4,sm %b[42], %b[42], %r20, %b[9]
stqp,5 %r5, %b[21], %b[47]
}
{
loop_mode
qppermb,1,sm %b[40], %b[40], %r20, %b[24]
stqp,2 %r10, %b[21], %b[6]
qppermb,4,sm %b[39], %b[39], %r20, %b[23]
stqp,5 %r7, %b[21], %b[11]
movaqp,0 area=7, ind=0, am=1, be=0, %b[16]
movaqp,1 area=6, ind=0, am=1, be=0, %b[20]
movaqp,2 area=7, ind=0, am=1, be=0, %b[12]
movaqp,3 area=6, ind=0, am=1, be=0, %b[17]
}
{
loop_mode
qppermb,1,sm %b[38], %b[38], %r20, %b[30]
stqp,2 %r9, %b[21], %b[26]
qppermb,4,sm %b[37], %b[37], %r20, %b[29]
stqp,5 %r11, %b[21], %b[25]
movaqp,0 area=5, ind=0, am=1, be=0, %b[11]
movaqp,1 area=4, ind=0, am=1, be=0, %b[36]
movaqp,2 area=5, ind=0, am=1, be=0, %b[6]
movaqp,3 area=4, ind=0, am=1, be=0, %b[35]
}
{
loop_mode
qppermb,1,sm %b[13], %b[13], %r20, %b[26]
stqp,2 %r12, %b[21], %b[32]
qppermb,4,sm %b[8], %b[8], %r20, %b[25]
stqp,5 %r13, %b[21], %b[31]
movaqp,0 area=3, ind=0, am=1, be=0, %b[39]
movaqp,1 area=2, ind=0, am=1, be=0, %b[40]
movaqp,2 area=3, ind=0, am=1, be=0, %b[37]
movaqp,3 area=2, ind=0, am=1, be=0, %b[38]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[22], %b[22], %r20, %b[42]
stqp,2 %r14, %b[21], %b[28]
ord,3,sm %b[5], %b[1], %b[45]
qppermb,4,sm %b[19], %b[19], %r20, %b[41]
stqp,5 %r15, %b[21], %b[27]
movaqp,0 area=1, ind=0, am=1, be=0, %b[13]
movaqp,1 area=0, ind=0, am=1, be=0, %b[32]
movaqp,2 area=1, ind=0, am=1, be=0, %b[8]
movaqp,3 area=0, ind=0, am=1, be=0, %b[31]
}
{
loop_mode
bitrevd,0,sm %b[2], %b[22]
qppermb,1,sm %b[18], %b[18], %r20, %b[5]
stqp,2 %r16, %b[21], %b[44]
shld,3,sm %b[45], 0x4, %b[19]
qppermb,4,sm %b[14], %b[14], %r20, %b[1]
stqp,5 %r17, %b[21], %b[43]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[33], %b[33], %r20, %b[2]
stqp,2 %r18, %b[21], %b[7]
qppermb,3,sm %b[34], %b[34], %r20, %b[14]
stqp,5 %r19, %b[21], %b[3]
}
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Но можно сделать чтение в 32 потока. Для этого напишем чтение в 64 потока и обработаем сначала одну половину строк в одном цикле, а потом вторую половину строк во втором цикле. В каждом цикле будут использованы 32 потока чтения APB.
6. reverse_radix4_vector2_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream32(int bit_count_in, myComplex *data_in, myComplex2 *data_out)
{
int bit_count_out = bit_count_in - 1;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_000 = (__v2di*)&data_in[ 0 * count_in/64];
__v2di *in_001 = (__v2di*)&data_in[ 1 * count_in/64];
__v2di *in_002 = (__v2di*)&data_in[ 2 * count_in/64];
__v2di *in_003 = (__v2di*)&data_in[ 3 * count_in/64];
__v2di *in_010 = (__v2di*)&data_in[ 4 * count_in/64];
__v2di *in_011 = (__v2di*)&data_in[ 5 * count_in/64];
__v2di *in_012 = (__v2di*)&data_in[ 6 * count_in/64];
__v2di *in_013 = (__v2di*)&data_in[ 7 * count_in/64];
__v2di *in_020 = (__v2di*)&data_in[ 8 * count_in/64];
__v2di *in_021 = (__v2di*)&data_in[ 9 * count_in/64];
__v2di *in_022 = (__v2di*)&data_in[10 * count_in/64];
__v2di *in_023 = (__v2di*)&data_in[11 * count_in/64];
__v2di *in_030 = (__v2di*)&data_in[12 * count_in/64];
__v2di *in_031 = (__v2di*)&data_in[13 * count_in/64];
__v2di *in_032 = (__v2di*)&data_in[14 * count_in/64];
__v2di *in_033 = (__v2di*)&data_in[15 * count_in/64];
__v2di *in_100 = (__v2di*)&data_in[16 * count_in/64];
__v2di *in_101 = (__v2di*)&data_in[17 * count_in/64];
__v2di *in_102 = (__v2di*)&data_in[18 * count_in/64];
__v2di *in_103 = (__v2di*)&data_in[19 * count_in/64];
__v2di *in_110 = (__v2di*)&data_in[20 * count_in/64];
__v2di *in_111 = (__v2di*)&data_in[21 * count_in/64];
__v2di *in_112 = (__v2di*)&data_in[22 * count_in/64];
__v2di *in_113 = (__v2di*)&data_in[23 * count_in/64];
__v2di *in_120 = (__v2di*)&data_in[24 * count_in/64];
__v2di *in_121 = (__v2di*)&data_in[25 * count_in/64];
__v2di *in_122 = (__v2di*)&data_in[26 * count_in/64];
__v2di *in_123 = (__v2di*)&data_in[27 * count_in/64];
__v2di *in_130 = (__v2di*)&data_in[28 * count_in/64];
__v2di *in_131 = (__v2di*)&data_in[29 * count_in/64];
__v2di *in_132 = (__v2di*)&data_in[30 * count_in/64];
__v2di *in_133 = (__v2di*)&data_in[31 * count_in/64];
__v2di *in_200 = (__v2di*)&data_in[32 * count_in/64];
__v2di *in_201 = (__v2di*)&data_in[33 * count_in/64];
__v2di *in_202 = (__v2di*)&data_in[34 * count_in/64];
__v2di *in_203 = (__v2di*)&data_in[35 * count_in/64];
__v2di *in_210 = (__v2di*)&data_in[36 * count_in/64];
__v2di *in_211 = (__v2di*)&data_in[37 * count_in/64];
__v2di *in_212 = (__v2di*)&data_in[38 * count_in/64];
__v2di *in_213 = (__v2di*)&data_in[39 * count_in/64];
__v2di *in_220 = (__v2di*)&data_in[40 * count_in/64];
__v2di *in_221 = (__v2di*)&data_in[41 * count_in/64];
__v2di *in_222 = (__v2di*)&data_in[42 * count_in/64];
__v2di *in_223 = (__v2di*)&data_in[43 * count_in/64];
__v2di *in_230 = (__v2di*)&data_in[44 * count_in/64];
__v2di *in_231 = (__v2di*)&data_in[45 * count_in/64];
__v2di *in_232 = (__v2di*)&data_in[46 * count_in/64];
__v2di *in_233 = (__v2di*)&data_in[47 * count_in/64];
__v2di *in_300 = (__v2di*)&data_in[48 * count_in/64];
__v2di *in_301 = (__v2di*)&data_in[49 * count_in/64];
__v2di *in_302 = (__v2di*)&data_in[50 * count_in/64];
__v2di *in_303 = (__v2di*)&data_in[51 * count_in/64];
__v2di *in_310 = (__v2di*)&data_in[52 * count_in/64];
__v2di *in_311 = (__v2di*)&data_in[53 * count_in/64];
__v2di *in_312 = (__v2di*)&data_in[54 * count_in/64];
__v2di *in_313 = (__v2di*)&data_in[55 * count_in/64];
__v2di *in_320 = (__v2di*)&data_in[56 * count_in/64];
__v2di *in_321 = (__v2di*)&data_in[57 * count_in/64];
__v2di *in_322 = (__v2di*)&data_in[58 * count_in/64];
__v2di *in_323 = (__v2di*)&data_in[59 * count_in/64];
__v2di *in_330 = (__v2di*)&data_in[60 * count_in/64];
__v2di *in_331 = (__v2di*)&data_in[61 * count_in/64];
__v2di *in_332 = (__v2di*)&data_in[62 * count_in/64];
__v2di *in_333 = (__v2di*)&data_in[63 * count_in/64];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32/2; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[i], in_000[i], mask);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_100[i], in_100[i], mask);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_200[i], in_200[i], mask);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_300[i], in_300[i], mask);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_010[i], in_010[i], mask);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_110[i], in_110[i], mask);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_210[i], in_210[i], mask);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_310[i], in_310[i], mask);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_020[i], in_020[i], mask);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_120[i], in_120[i], mask);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_220[i], in_220[i], mask);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_320[i], in_320[i], mask);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_030[i], in_030[i], mask);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_130[i], in_130[i], mask);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_230[i], in_230[i], mask);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_330[i], in_330[i], mask);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_001[i], in_001[i], mask);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_101[i], in_101[i], mask);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_201[i], in_201[i], mask);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_301[i], in_301[i], mask);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_011[i], in_011[i], mask);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_111[i], in_111[i], mask);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_211[i], in_211[i], mask);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_311[i], in_311[i], mask);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_021[i], in_021[i], mask);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_121[i], in_121[i], mask);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_221[i], in_221[i], mask);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_321[i], in_321[i], mask);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_031[i], in_031[i], mask);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_131[i], in_131[i], mask);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_231[i], in_231[i], mask);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_331[i], in_331[i], mask);
}
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32/2; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504};
*(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_002[i], in_002[i], mask);
*(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_102[i], in_102[i], mask);
*(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_202[i], in_202[i], mask);
*(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_302[i], in_302[i], mask);
*(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_012[i], in_012[i], mask);
*(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_112[i], in_112[i], mask);
*(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_212[i], in_212[i], mask);
*(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_312[i], in_312[i], mask);
*(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_022[i], in_022[i], mask);
*(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_122[i], in_122[i], mask);
*(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_222[i], in_222[i], mask);
*(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_322[i], in_322[i], mask);
*(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_032[i], in_032[i], mask);
*(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_132[i], in_132[i], mask);
*(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_232[i], in_232[i], mask);
*(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_332[i], in_332[i], mask);
*(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_003[i], in_003[i], mask);
*(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_103[i], in_103[i], mask);
*(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_203[i], in_203[i], mask);
*(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_303[i], in_303[i], mask);
*(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_013[i], in_013[i], mask);
*(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_113[i], in_113[i], mask);
*(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_213[i], in_213[i], mask);
*(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_313[i], in_313[i], mask);
*(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_023[i], in_023[i], mask);
*(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_123[i], in_123[i], mask);
*(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_223[i], in_223[i], mask);
*(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_323[i], in_323[i], mask);
*(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_033[i], in_033[i], mask);
*(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_133[i], in_133[i], mask);
*(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_233[i], in_233[i], mask);
*(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_333[i], in_333[i], mask);
}
}
Основной цикл на ассемблере
.L9991:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0
}
.L6665:
{
loop_mode
qppermb,0,sm %b[41], %b[41], %r0, %b[58]
shr_andd,1,sm %b[56], %r1, %r38, %b[54]
stqp,2 %r5, %b[34], %b[54]
qppermb,3,sm %b[46], %b[46], %r0, %b[60]
shr_andd,4,sm %b[56], %r4, %r37, %b[56]
stqp,5 %r2, %b[34], %b[58]
}
{
loop_mode
qppermb,1,sm %b[76], %b[76], %r0, %b[1]
stqp,2 %r7, %b[34], %b[58]
qppermb,4,sm %b[77], %b[77], %r0, %b[8]
stqp,5 %r6, %b[34], %b[60]
}
{
loop_mode
qppermb,1,sm %b[65], %b[65], %r0, %b[10]
stqp,2 %r11, %b[34], %b[3]
qppermb,4,sm %b[63], %b[63], %r0, %b[3]
stqp,5 %r9, %b[34], %b[10]
}
{
loop_mode
qppermb,1,sm %b[75], %b[75], %r0, %b[12]
stqp,2 %r10, %b[34], %b[12]
qppermb,4,sm %b[74], %b[74], %r0, %b[5]
stqp,5 %r12, %b[34], %b[5]
}
{
loop_mode
qppermb,1,sm %b[73], %b[73], %r0, %b[14]
stqp,2 %r13, %b[34], %b[14]
qppermb,4,sm %b[72], %b[72], %r0, %b[7]
stqp,5 %r14, %b[34], %b[7]
}
{
loop_mode
qppermb,1,sm %b[71], %b[71], %r0, %b[16]
stqp,2 %r15, %b[34], %b[16]
qppermb,4,sm %b[70], %b[70], %r0, %b[9]
stqp,5 %r16, %b[34], %b[9]
}
{
loop_mode
qppermb,1,sm %b[69], %b[69], %r0, %b[18]
stqp,2 %r17, %b[34], %b[18]
qppermb,4,sm %b[67], %b[67], %r0, %b[11]
stqp,5 %r18, %b[34], %b[11]
movaqp,0 area=15, ind=0, am=1, be=0, %b[60]
movaqp,1 area=14, ind=0, am=1, be=0, %b[64]
movaqp,2 area=15, ind=0, am=1, be=0, %b[58]
movaqp,3 area=14, ind=0, am=1, be=0, %b[62]
}
{
loop_mode
qppermb,1,sm %b[68], %b[68], %r0, %b[36]
stqp,2 %r19, %b[34], %b[20]
qppermb,4,sm %b[61], %b[61], %r0, %b[31]
stqp,5 %r20, %b[34], %b[13]
movaqp,0 area=13, ind=0, am=1, be=0, %b[63]
movaqp,1 area=12, ind=0, am=1, be=0, %b[20]
movaqp,2 area=13, ind=0, am=1, be=0, %b[61]
movaqp,3 area=12, ind=0, am=1, be=0, %b[13]
}
{
loop_mode
qppermb,1,sm %b[59], %b[59], %r0, %b[53]
stqp,2 %r21, %b[34], %b[38]
qppermb,4,sm %b[57], %b[57], %r0, %b[49]
stqp,5 %r22, %b[34], %b[33]
movaqp,0 area=11, ind=0, am=1, be=0, %b[38]
movaqp,1 area=10, ind=0, am=1, be=0, %b[46]
movaqp,2 area=11, ind=0, am=1, be=0, %b[33]
movaqp,3 area=10, ind=0, am=1, be=0, %b[41]
}
{
loop_mode
qppermb,1,sm %b[48], %b[48], %r0, %b[48]
stqp,2 %r23, %b[34], %b[55]
qppermb,4,sm %b[43], %b[43], %r0, %b[43]
stqp,5 %r24, %b[34], %b[51]
movaqp,0 area=9, ind=0, am=1, be=0, %b[57]
movaqp,1 area=8, ind=0, am=1, be=0, %b[66]
movaqp,2 area=9, ind=0, am=1, be=0, %b[55]
movaqp,3 area=8, ind=0, am=1, be=0, %b[59]
}
{
loop_mode
qppermb,1,sm %b[40], %b[40], %r0, %b[40]
stqp,2 %r25, %b[34], %b[50]
qppermb,4,sm %b[35], %b[35], %r0, %b[35]
stqp,5 %r26, %b[34], %b[45]
movaqp,0 area=7, ind=0, am=1, be=0, %b[67]
movaqp,1 area=6, ind=0, am=1, be=0, %b[69]
movaqp,2 area=7, ind=0, am=1, be=0, %b[65]
movaqp,3 area=6, ind=0, am=1, be=0, %b[68]
}
{
loop_mode
qppermb,1,sm %b[22], %b[22], %r0, %b[22]
stqp,2 %r27, %b[34], %b[42]
qppermb,4,sm %b[15], %b[15], %r0, %b[15]
stqp,5 %r28, %b[34], %b[37]
movaqp,0 area=5, ind=0, am=1, be=0, %b[71]
movaqp,1 area=4, ind=0, am=1, be=0, %b[73]
movaqp,2 area=5, ind=0, am=1, be=0, %b[70]
movaqp,3 area=4, ind=0, am=1, be=0, %b[72]
}
{
loop_mode
qppermb,1,sm %b[63], %b[63], %r0, %b[24]
stqp,2 %r29, %b[34], %b[24]
qppermb,4,sm %b[61], %b[61], %r0, %b[17]
stqp,5 %r30, %b[34], %b[17]
movaqp,0 area=3, ind=0, am=1, be=0, %b[74]
movaqp,1 area=2, ind=0, am=1, be=0, %b[75]
movaqp,2 area=3, ind=0, am=1, be=0, %b[61]
movaqp,3 area=2, ind=0, am=1, be=0, %b[63]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[64], %b[64], %r0, %b[28]
stqp,2 %r31, %b[34], %b[28]
ord,3,sm %b[56], %b[54], %b[54]
qppermb,4,sm %b[62], %b[62], %r0, %b[21]
stqp,5 %r32, %b[34], %b[21]
movaqp,0 area=1, ind=0, am=1, be=0, %b[42]
movaqp,1 area=0, ind=0, am=1, be=0, %b[50]
movaqp,2 area=1, ind=0, am=1, be=0, %b[37]
movaqp,3 area=0, ind=0, am=1, be=0, %b[45]
}
{
loop_mode
bitrevd,0,sm %b[2], %b[54]
qppermb,1,sm %b[60], %b[60], %r0, %b[25]
stqp,2 %r33, %b[34], %b[32]
shld,3,sm %b[54], 0x4, %b[32]
qppermb,4,sm %b[58], %b[58], %r0, %b[2]
stqp,5 %r34, %b[34], %b[25]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[47], %b[47], %r0, %b[52]
stqp,2 %r35, %b[34], %b[29]
qppermb,3,sm %b[52], %b[52], %r0, %b[56]
stqp,5 %r36, %b[34], %b[6]
}
...
.L9337:
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0
}
.L7373:
{
loop_mode
qppermb,0,sm %b[41], %b[41], %r0, %b[58]
shr_andd,1,sm %b[56], %r1, %r38, %b[54]
stqp,2 %r5, %b[34], %b[54]
qppermb,3,sm %b[46], %b[46], %r0, %b[60]
shr_andd,4,sm %b[56], %r4, %r37, %b[56]
stqp,5 %r2, %b[34], %b[58]
}
{
loop_mode
qppermb,1,sm %b[76], %b[76], %r0, %b[1]
stqp,2 %r7, %b[34], %b[58]
qppermb,4,sm %b[77], %b[77], %r0, %b[8]
stqp,5 %r6, %b[34], %b[60]
}
{
loop_mode
qppermb,1,sm %b[65], %b[65], %r0, %b[10]
stqp,2 %r11, %b[34], %b[3]
qppermb,4,sm %b[63], %b[63], %r0, %b[3]
stqp,5 %r9, %b[34], %b[10]
}
{
loop_mode
qppermb,1,sm %b[75], %b[75], %r0, %b[12]
stqp,2 %r10, %b[34], %b[12]
qppermb,4,sm %b[74], %b[74], %r0, %b[5]
stqp,5 %r12, %b[34], %b[5]
}
{
loop_mode
qppermb,1,sm %b[73], %b[73], %r0, %b[14]
stqp,2 %r13, %b[34], %b[14]
qppermb,4,sm %b[72], %b[72], %r0, %b[7]
stqp,5 %r14, %b[34], %b[7]
}
{
loop_mode
qppermb,1,sm %b[71], %b[71], %r0, %b[16]
stqp,2 %r15, %b[34], %b[16]
qppermb,4,sm %b[70], %b[70], %r0, %b[9]
stqp,5 %r16, %b[34], %b[9]
}
{
loop_mode
qppermb,1,sm %b[69], %b[69], %r0, %b[18]
stqp,2 %r17, %b[34], %b[18]
qppermb,4,sm %b[67], %b[67], %r0, %b[11]
stqp,5 %r18, %b[34], %b[11]
movaqp,0 area=15, ind=0, am=1, be=0, %b[60]
movaqp,1 area=14, ind=0, am=1, be=0, %b[64]
movaqp,2 area=15, ind=0, am=1, be=0, %b[58]
movaqp,3 area=14, ind=0, am=1, be=0, %b[62]
}
{
loop_mode
qppermb,1,sm %b[68], %b[68], %r0, %b[36]
stqp,2 %r19, %b[34], %b[20]
qppermb,4,sm %b[61], %b[61], %r0, %b[31]
stqp,5 %r20, %b[34], %b[13]
movaqp,0 area=13, ind=0, am=1, be=0, %b[63]
movaqp,1 area=12, ind=0, am=1, be=0, %b[20]
movaqp,2 area=13, ind=0, am=1, be=0, %b[61]
movaqp,3 area=12, ind=0, am=1, be=0, %b[13]
}
{
loop_mode
qppermb,1,sm %b[59], %b[59], %r0, %b[53]
stqp,2 %r21, %b[34], %b[38]
qppermb,4,sm %b[57], %b[57], %r0, %b[49]
stqp,5 %r22, %b[34], %b[33]
movaqp,0 area=11, ind=0, am=1, be=0, %b[38]
movaqp,1 area=10, ind=0, am=1, be=0, %b[46]
movaqp,2 area=11, ind=0, am=1, be=0, %b[33]
movaqp,3 area=10, ind=0, am=1, be=0, %b[41]
}
{
loop_mode
qppermb,1,sm %b[48], %b[48], %r0, %b[48]
stqp,2 %r23, %b[34], %b[55]
qppermb,4,sm %b[43], %b[43], %r0, %b[43]
stqp,5 %r24, %b[34], %b[51]
movaqp,0 area=9, ind=0, am=1, be=0, %b[57]
movaqp,1 area=8, ind=0, am=1, be=0, %b[66]
movaqp,2 area=9, ind=0, am=1, be=0, %b[55]
movaqp,3 area=8, ind=0, am=1, be=0, %b[59]
}
{
loop_mode
qppermb,1,sm %b[40], %b[40], %r0, %b[40]
stqp,2 %r25, %b[34], %b[50]
qppermb,4,sm %b[35], %b[35], %r0, %b[35]
stqp,5 %r26, %b[34], %b[45]
movaqp,0 area=7, ind=0, am=1, be=0, %b[67]
movaqp,1 area=6, ind=0, am=1, be=0, %b[69]
movaqp,2 area=7, ind=0, am=1, be=0, %b[65]
movaqp,3 area=6, ind=0, am=1, be=0, %b[68]
}
{
loop_mode
qppermb,1,sm %b[22], %b[22], %r0, %b[22]
stqp,2 %r27, %b[34], %b[42]
qppermb,4,sm %b[15], %b[15], %r0, %b[15]
stqp,5 %r28, %b[34], %b[37]
movaqp,0 area=5, ind=0, am=1, be=0, %b[71]
movaqp,1 area=4, ind=0, am=1, be=0, %b[73]
movaqp,2 area=5, ind=0, am=1, be=0, %b[70]
movaqp,3 area=4, ind=0, am=1, be=0, %b[72]
}
{
loop_mode
qppermb,1,sm %b[63], %b[63], %r0, %b[24]
stqp,2 %r29, %b[34], %b[24]
qppermb,4,sm %b[61], %b[61], %r0, %b[17]
stqp,5 %r30, %b[34], %b[17]
movaqp,0 area=3, ind=0, am=1, be=0, %b[74]
movaqp,1 area=2, ind=0, am=1, be=0, %b[75]
movaqp,2 area=3, ind=0, am=1, be=0, %b[61]
movaqp,3 area=2, ind=0, am=1, be=0, %b[63]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[64], %b[64], %r0, %b[28]
stqp,2 %r31, %b[34], %b[28]
ord,3,sm %b[56], %b[54], %b[54]
qppermb,4,sm %b[62], %b[62], %r0, %b[21]
stqp,5 %r32, %b[34], %b[21]
movaqp,0 area=1, ind=0, am=1, be=0, %b[42]
movaqp,1 area=0, ind=0, am=1, be=0, %b[50]
movaqp,2 area=1, ind=0, am=1, be=0, %b[37]
movaqp,3 area=0, ind=0, am=1, be=0, %b[45]
}
{
loop_mode
bitrevd,0,sm %b[2], %b[54]
qppermb,1,sm %b[60], %b[60], %r0, %b[25]
stqp,2 %r33, %b[34], %b[32]
shld,3,sm %b[54], 0x4, %b[32]
qppermb,4,sm %b[58], %b[58], %r0, %b[2]
stqp,5 %r34, %b[34], %b[25]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[47], %b[47], %r0, %b[52]
stqp,2 %r35, %b[34], %b[29]
qppermb,3,sm %b[52], %b[52], %r0, %b[56]
stqp,5 %r36, %b[34], %b[6]
}
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
Итоги по reverse_radix4_vector2
Победителем можно считать reverse_radix4_vector2_stream16.
При реализации Vector-2 Radix-4 FFT будем использовать его.
reverse_radix4_vector4
1. reverse_radix4_vector4_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix4 вычисляется с помощью цикла.
int reverseNumber_radix4(int number, int bit_count)
{
int answer = 0;
for(int i = 0; i < bit_count/2; ++i)
{
answer <<= 2;
answer |= number & 3;
number >>= 2;
}
return answer;
}
void reverse_radix4_vector4_etalon(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
for(int64_t i = 0; i < count_out; ++i)
{
int index = reverseNumber_radix4(i, bit_count_out);
myComplex a = data_in[4*i + 0];
myComplex b = data_in[4*i + 1];
myComplex c = data_in[4*i + 2];
myComplex d = data_in[4*i + 3];
data_out[index] = (myComplex4){.real = {a.real, b.real, c.real, d.real}, .imag = {a.imag, b.imag, c.imag, d.imag}};
}
}
2. reverse_radix4_vector4
В процессоре нет готовых инструкций, производящих операцию reverseNumber_radix4.
Поэтому выполним инструкцию bitrevd и переставим соседние биты местами.
Код на Си
void reverse_radix4_vector4(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*)data_in;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L6561:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
.L6351:
{
loop_mode
qppermb,0,sm %b[15], %b[12], %r4, %b[19]
shr_andd,1,sm %b[14], %r6, %r10, %b[0]
qppermb,3,sm %b[15], %b[12], %r5, %b[20]
shr_andd,4,sm %b[16], %r7, %r9, %b[5]
ord,5,sm %b[7], %b[4], %b[18]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
bitrevd,0,sm %b[3], %b[12]
stqp,2 %r0, %b[17], %b[19]
addd,3,sm %b[3], 0x1, %b[1]
shld,4,sm %b[18], 0x5, %b[15]
stqp,5 %r2, %b[17], %b[20]
movaqp,0 area=0, ind=0, am=1, be=0, %b[4]
movaqp,1 area=0, ind=16, am=0, be=0, %b[7]
}
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
3. reverse_radix4_vector4_x4
Сделаем ручную раскрутку цикла в 4 раза.
Код на Си
void reverse_radix4_vector4_x4(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
// int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in = (__v2di*) data_in;
__v2di *out_0 = (__v2di*)&data_out[0 * count_out/4];
__v2di *out_1 = (__v2di*)&data_out[1 * count_out/4];
__v2di *out_2 = (__v2di*)&data_out[2 * count_out/4];
__v2di *out_3 = (__v2di*)&data_out[3 * count_out/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out; i += 4)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)out_0 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_real);
*(__v2du*)((void*)out_0 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_imag);
*(__v2du*)((void*)out_1 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_real);
*(__v2du*)((void*)out_1 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_imag);
*(__v2du*)((void*)out_2 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+2)+1], in[2*(i+2)+0], mask_real);
*(__v2du*)((void*)out_2 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+2)+1], in[2*(i+2)+0], mask_imag);
*(__v2du*)((void*)out_3 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+3)+1], in[2*(i+3)+0], mask_real);
*(__v2du*)((void*)out_3 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+3)+1], in[2*(i+3)+0], mask_imag);
}
}
Основной цикл на ассемблере
.L7192:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=96
}
.L6608:
{
loop_mode
qppermb,0,sm %b[23], %b[9], %r11, %b[35]
stqp,2 %r0, %b[33], %b[22]
qppermb,3,sm %b[23], %b[9], %r13, %b[36]
shr_andd,4,sm %b[12], %r14, %r16, %b[1]
stqp,5 %r2, %b[33], %b[25]
}
{
loop_mode
ord,0,sm %b[17], %b[3], %b[23]
qppermb,1,sm %b[15], %b[10], %r11, %b[37]
stqp,2 %r4, %b[33], %b[35]
qppermb,4,sm %b[28], %b[20], %r13, %b[38]
stqp,5 %r10, %b[33], %b[36]
movaqp,0 area=1, ind=0, am=1, be=0, %b[14]
movaqp,1 area=1, ind=16, am=0, be=0, %b[22]
movaqp,2 area=1, ind=0, am=1, be=0, %b[4]
movaqp,3 area=1, ind=16, am=0, be=0, %b[9]
}
{
loop_mode
shld,0,sm %b[23], 0x5, %b[31]
qppermb,1,sm %b[28], %b[20], %r11, %b[36]
stqp,2 %r6, %b[33], %b[37]
addd,3,sm %b[2], 0x4, %b[0]
qppermb,4,sm %b[15], %b[10], %r13, %b[35]
stqp,5 %r9, %b[33], %b[38]
movaqp,0 area=0, ind=0, am=1, be=0, %b[25]
movaqp,1 area=0, ind=16, am=0, be=0, %b[30]
movaqp,2 area=0, ind=0, am=1, be=0, %b[3]
movaqp,3 area=0, ind=16, am=0, be=0, %b[17]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[34], %b[29], %r11, %b[20]
shr_andd,1,sm %b[12], %r12, %r15, %b[15]
stqp,2 %r5, %b[33], %b[36]
qppermb,3,sm %b[34], %b[29], %r13, %b[23]
bitrevd,4,sm %b[2], %b[10]
stqp,5 %r7, %b[33], %b[35]
}
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале и замедление в конце графика.
Здесь происходит четыре чтения из одного места памяти и запись в четыре разных места памяти.
4. reverse_radix4_vector4_stream4
Теперь сделаем наоборот: будем читать из четырёх разных мест, а писать рядом.
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream4(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_0 = (__v2di*)&data_in[0 * count_in/4];
__v2di *in_1 = (__v2di*)&data_in[1 * count_in/4];
__v2di *in_2 = (__v2di*)&data_in[2 * count_in/4];
__v2di *in_3 = (__v2di*)&data_in[3 * count_in/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/4; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_2[2*i+1], in_2[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_2[2*i+1], in_2[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_3[2*i+1], in_3[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_3[2*i+1], in_3[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L7740:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=4, abs=0, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=4, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=0
}
.L7235:
{
loop_mode
qppermb,0,sm %b[23], %b[9], %r11, %b[35]
stqp,2 %r0, %b[33], %b[22]
qppermb,3,sm %b[23], %b[9], %r13, %b[36]
shr_andd,4,sm %b[12], %r14, %r16, %b[1]
stqp,5 %r2, %b[33], %b[25]
}
{
loop_mode
ord,0,sm %b[17], %b[3], %b[23]
qppermb,1,sm %b[15], %b[10], %r11, %b[37]
stqp,2 %r5, %b[33], %b[35]
qppermb,4,sm %b[28], %b[20], %r13, %b[38]
stqp,5 %r4, %b[33], %b[36]
movaqp,0 area=1, ind=0, am=1, be=0, %b[14]
movaqp,1 area=1, ind=16, am=0, be=0, %b[22]
movaqp,2 area=1, ind=0, am=1, be=0, %b[4]
movaqp,3 area=1, ind=16, am=0, be=0, %b[9]
}
{
loop_mode
shld,0,sm %b[23], 0x5, %b[31]
qppermb,1,sm %b[28], %b[20], %r11, %b[36]
stqp,2 %r10, %b[33], %b[37]
addd,3,sm %b[2], 0x1, %b[0]
qppermb,4,sm %b[15], %b[10], %r13, %b[35]
stqp,5 %r6, %b[33], %b[38]
movaqp,0 area=0, ind=0, am=1, be=0, %b[25]
movaqp,1 area=0, ind=16, am=0, be=0, %b[30]
movaqp,2 area=0, ind=0, am=1, be=0, %b[3]
movaqp,3 area=0, ind=16, am=0, be=0, %b[17]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[34], %b[29], %r11, %b[20]
shr_andd,1,sm %b[12], %r12, %r15, %b[15]
stqp,2 %r7, %b[33], %b[36]
qppermb,3,sm %b[34], %b[29], %r13, %b[23]
bitrevd,4,sm %b[2], %b[10]
stqp,5 %r9, %b[33], %b[35]
}
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix4_vector4_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream16(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_00 = (__v2di*)&data_in[ 0 * count_in/16];
__v2di *in_01 = (__v2di*)&data_in[ 1 * count_in/16];
__v2di *in_02 = (__v2di*)&data_in[ 2 * count_in/16];
__v2di *in_03 = (__v2di*)&data_in[ 3 * count_in/16];
__v2di *in_10 = (__v2di*)&data_in[ 4 * count_in/16];
__v2di *in_11 = (__v2di*)&data_in[ 5 * count_in/16];
__v2di *in_12 = (__v2di*)&data_in[ 6 * count_in/16];
__v2di *in_13 = (__v2di*)&data_in[ 7 * count_in/16];
__v2di *in_20 = (__v2di*)&data_in[ 8 * count_in/16];
__v2di *in_21 = (__v2di*)&data_in[ 9 * count_in/16];
__v2di *in_22 = (__v2di*)&data_in[10 * count_in/16];
__v2di *in_23 = (__v2di*)&data_in[11 * count_in/16];
__v2di *in_30 = (__v2di*)&data_in[12 * count_in/16];
__v2di *in_31 = (__v2di*)&data_in[13 * count_in/16];
__v2di *in_32 = (__v2di*)&data_in[14 * count_in/16];
__v2di *in_33 = (__v2di*)&data_in[15 * count_in/16];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/16; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_20[2*i+1], in_20[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_20[2*i+1], in_20[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_30[2*i+1], in_30[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_30[2*i+1], in_30[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_21[2*i+1], in_21[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_21[2*i+1], in_21[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_31[2*i+1], in_31[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_31[2*i+1], in_31[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_02[2*i+1], in_02[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_02[2*i+1], in_02[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_12[2*i+1], in_12[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_12[2*i+1], in_12[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_22[2*i+1], in_22[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_22[2*i+1], in_22[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_32[2*i+1], in_32[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_32[2*i+1], in_32[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_03[2*i+1], in_03[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_03[2*i+1], in_03[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_13[2*i+1], in_13[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_13[2*i+1], in_13[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_23[2*i+1], in_23[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_23[2*i+1], in_23[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_33[2*i+1], in_33[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_33[2*i+1], in_33[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L9557:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=2, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=2, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=2, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=2, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=2, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=2, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=2, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=2, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=2, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=2, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=2, abs=24, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=2, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=28, disp=0
}
.L7879:
{
loop_mode
qppermb,0,sm %b[20], %b[15], %r36, %b[73]
shr_andd,1,sm %b[69], %r37, %r40, %b[69]
stqp,2 %r0, %b[4], %b[71]
qppermb,3,sm %b[20], %b[15], %r35, %b[72]
shr_andd,4,sm %b[69], %r38, %r39, %b[71]
stqp,5 %r2, %b[4], %b[72]
}
{
loop_mode
qppermb,1,sm %b[28], %b[23], %r36, %b[72]
stqp,2 %r5, %b[4], %b[73]
qppermb,4,sm %b[67], %b[64], %r35, %b[74]
stqp,5 %r1, %b[4], %b[72]
}
{
loop_mode
qppermb,1,sm %b[67], %b[64], %r36, %b[72]
stqp,2 %r10, %b[4], %b[72]
qppermb,4,sm %b[28], %b[23], %r35, %b[67]
stqp,5 %r6, %b[4], %b[74]
}
{
loop_mode
qppermb,1,sm %b[63], %b[60], %r36, %b[67]
stqp,2 %r7, %b[4], %b[72]
qppermb,4,sm %b[63], %b[60], %r35, %b[72]
stqp,5 %r9, %b[4], %b[67]
}
{
loop_mode
qppermb,1,sm %b[36], %b[31], %r36, %b[67]
stqp,2 %r12, %b[4], %b[67]
qppermb,4,sm %b[36], %b[31], %r35, %b[72]
stqp,5 %r11, %b[4], %b[72]
}
{
loop_mode
qppermb,1,sm %b[59], %b[56], %r36, %b[67]
stqp,2 %r14, %b[4], %b[67]
qppermb,4,sm %b[59], %b[56], %r35, %b[72]
stqp,5 %r13, %b[4], %b[72]
}
{
loop_mode
qppermb,1,sm %b[44], %b[39], %r36, %b[67]
stqp,2 %r16, %b[4], %b[67]
qppermb,4,sm %b[44], %b[39], %r35, %b[72]
stqp,5 %r15, %b[4], %b[72]
movaqp,0 area=7, ind=0, am=1, be=0, %b[7]
movaqp,1 area=7, ind=16, am=0, be=0, %b[12]
movaqp,2 area=7, ind=0, am=1, be=0, %b[1]
movaqp,3 area=7, ind=16, am=0, be=0, %b[6]
}
{
loop_mode
qppermb,1,sm %b[55], %b[52], %r36, %b[67]
stqp,2 %r18, %b[4], %b[67]
qppermb,4,sm %b[55], %b[52], %r35, %b[72]
stqp,5 %r17, %b[4], %b[72]
movaqp,0 area=6, ind=0, am=1, be=0, %b[23]
movaqp,1 area=6, ind=16, am=0, be=0, %b[28]
movaqp,2 area=6, ind=0, am=1, be=0, %b[15]
movaqp,3 area=6, ind=16, am=0, be=0, %b[20]
}
{
loop_mode
qppermb,1,sm %b[51], %b[47], %r36, %b[67]
stqp,2 %r20, %b[4], %b[67]
qppermb,4,sm %b[51], %b[47], %r35, %b[72]
stqp,5 %r19, %b[4], %b[72]
movaqp,0 area=5, ind=0, am=1, be=0, %b[39]
movaqp,1 area=5, ind=16, am=0, be=0, %b[44]
movaqp,2 area=5, ind=0, am=1, be=0, %b[31]
movaqp,3 area=5, ind=16, am=0, be=0, %b[36]
}
{
loop_mode
qppermb,1,sm %b[48], %b[43], %r36, %b[67]
stqp,2 %r22, %b[4], %b[67]
qppermb,4,sm %b[48], %b[43], %r35, %b[72]
stqp,5 %r21, %b[4], %b[72]
movaqp,0 area=4, ind=0, am=1, be=0, %b[48]
movaqp,1 area=4, ind=16, am=0, be=0, %b[51]
movaqp,2 area=4, ind=0, am=1, be=0, %b[43]
movaqp,3 area=4, ind=16, am=0, be=0, %b[47]
}
{
loop_mode
qppermb,1,sm %b[40], %b[35], %r36, %b[67]
stqp,2 %r24, %b[4], %b[67]
qppermb,4,sm %b[40], %b[35], %r35, %b[72]
stqp,5 %r23, %b[4], %b[72]
movaqp,0 area=3, ind=0, am=1, be=0, %b[52]
movaqp,1 area=3, ind=16, am=0, be=0, %b[55]
movaqp,2 area=3, ind=0, am=1, be=0, %b[35]
movaqp,3 area=3, ind=16, am=0, be=0, %b[40]
}
{
loop_mode
qppermb,1,sm %b[32], %b[27], %r36, %b[67]
stqp,2 %r26, %b[4], %b[67]
qppermb,4,sm %b[32], %b[27], %r35, %b[72]
stqp,5 %r25, %b[4], %b[72]
movaqp,0 area=2, ind=0, am=1, be=0, %b[56]
movaqp,1 area=2, ind=16, am=0, be=0, %b[59]
movaqp,2 area=2, ind=0, am=1, be=0, %b[27]
movaqp,3 area=2, ind=16, am=0, be=0, %b[32]
}
{
loop_mode
qppermb,1,sm %b[24], %b[19], %r36, %b[67]
stqp,2 %r28, %b[4], %b[67]
qppermb,4,sm %b[24], %b[19], %r35, %b[72]
stqp,5 %r27, %b[4], %b[72]
movaqp,0 area=1, ind=0, am=1, be=0, %b[60]
movaqp,1 area=1, ind=16, am=0, be=0, %b[63]
movaqp,2 area=1, ind=0, am=1, be=0, %b[19]
movaqp,3 area=1, ind=16, am=0, be=0, %b[24]
}
{
loop_mode
addd,0,sm %b[2], 0x1, %b[0]
qppermb,1,sm %b[16], %b[11], %r36, %b[69]
stqp,2 %r30, %b[4], %b[67]
ord,3,sm %b[71], %b[69], %b[67]
qppermb,4,sm %b[16], %b[11], %r35, %b[71]
stqp,5 %r29, %b[4], %b[72]
movaqp,0 area=0, ind=0, am=1, be=0, %b[64]
movaqp,1 area=0, ind=16, am=0, be=0, %b[68]
movaqp,2 area=0, ind=0, am=1, be=0, %b[11]
movaqp,3 area=0, ind=16, am=0, be=0, %b[16]
}
{
loop_mode
bitrevd,0,sm %b[2], %b[67]
qppermb,1,sm %b[10], %b[5], %r36, %b[69]
stqp,2 %r32, %b[4], %b[69]
shld,3,sm %b[67], 0x5, %b[2]
qppermb,4,sm %b[10], %b[5], %r35, %b[71]
stqp,5 %r31, %b[4], %b[71]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qppermb,0,sm %b[70], %b[66], %r36, %b[69]
stqp,2 %r34, %b[4], %b[69]
qppermb,3,sm %b[70], %b[66], %r35, %b[70]
stqp,5 %r33, %b[4], %b[71]
}
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Но можно сделать чтение в 32 потока. Для этого напишем чтение в 64 потока и обработаем сначала одну половину строк в одном цикле, а потом вторую половину строк во втором цикле. В каждом цикле будут использованы 32 потока чтения APB.
6. reverse_radix4_vector4_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream32(int bit_count_in, myComplex *data_in, myComplex4 *data_out)
{
int bit_count_out = bit_count_in - 2;
int count_in = 1 << bit_count_in;
int count_out = 1 << bit_count_out;
int shift_out = 64 - bit_count_out;
__v2di *in_000 = (__v2di*)&data_in[ 0 * count_in/64];
__v2di *in_001 = (__v2di*)&data_in[ 1 * count_in/64];
__v2di *in_002 = (__v2di*)&data_in[ 2 * count_in/64];
__v2di *in_003 = (__v2di*)&data_in[ 3 * count_in/64];
__v2di *in_010 = (__v2di*)&data_in[ 4 * count_in/64];
__v2di *in_011 = (__v2di*)&data_in[ 5 * count_in/64];
__v2di *in_012 = (__v2di*)&data_in[ 6 * count_in/64];
__v2di *in_013 = (__v2di*)&data_in[ 7 * count_in/64];
__v2di *in_020 = (__v2di*)&data_in[ 8 * count_in/64];
__v2di *in_021 = (__v2di*)&data_in[ 9 * count_in/64];
__v2di *in_022 = (__v2di*)&data_in[10 * count_in/64];
__v2di *in_023 = (__v2di*)&data_in[11 * count_in/64];
__v2di *in_030 = (__v2di*)&data_in[12 * count_in/64];
__v2di *in_031 = (__v2di*)&data_in[13 * count_in/64];
__v2di *in_032 = (__v2di*)&data_in[14 * count_in/64];
__v2di *in_033 = (__v2di*)&data_in[15 * count_in/64];
__v2di *in_100 = (__v2di*)&data_in[16 * count_in/64];
__v2di *in_101 = (__v2di*)&data_in[17 * count_in/64];
__v2di *in_102 = (__v2di*)&data_in[18 * count_in/64];
__v2di *in_103 = (__v2di*)&data_in[19 * count_in/64];
__v2di *in_110 = (__v2di*)&data_in[20 * count_in/64];
__v2di *in_111 = (__v2di*)&data_in[21 * count_in/64];
__v2di *in_112 = (__v2di*)&data_in[22 * count_in/64];
__v2di *in_113 = (__v2di*)&data_in[23 * count_in/64];
__v2di *in_120 = (__v2di*)&data_in[24 * count_in/64];
__v2di *in_121 = (__v2di*)&data_in[25 * count_in/64];
__v2di *in_122 = (__v2di*)&data_in[26 * count_in/64];
__v2di *in_123 = (__v2di*)&data_in[27 * count_in/64];
__v2di *in_130 = (__v2di*)&data_in[28 * count_in/64];
__v2di *in_131 = (__v2di*)&data_in[29 * count_in/64];
__v2di *in_132 = (__v2di*)&data_in[30 * count_in/64];
__v2di *in_133 = (__v2di*)&data_in[31 * count_in/64];
__v2di *in_200 = (__v2di*)&data_in[32 * count_in/64];
__v2di *in_201 = (__v2di*)&data_in[33 * count_in/64];
__v2di *in_202 = (__v2di*)&data_in[34 * count_in/64];
__v2di *in_203 = (__v2di*)&data_in[35 * count_in/64];
__v2di *in_210 = (__v2di*)&data_in[36 * count_in/64];
__v2di *in_211 = (__v2di*)&data_in[37 * count_in/64];
__v2di *in_212 = (__v2di*)&data_in[38 * count_in/64];
__v2di *in_213 = (__v2di*)&data_in[39 * count_in/64];
__v2di *in_220 = (__v2di*)&data_in[40 * count_in/64];
__v2di *in_221 = (__v2di*)&data_in[41 * count_in/64];
__v2di *in_222 = (__v2di*)&data_in[42 * count_in/64];
__v2di *in_223 = (__v2di*)&data_in[43 * count_in/64];
__v2di *in_230 = (__v2di*)&data_in[44 * count_in/64];
__v2di *in_231 = (__v2di*)&data_in[45 * count_in/64];
__v2di *in_232 = (__v2di*)&data_in[46 * count_in/64];
__v2di *in_233 = (__v2di*)&data_in[47 * count_in/64];
__v2di *in_300 = (__v2di*)&data_in[48 * count_in/64];
__v2di *in_301 = (__v2di*)&data_in[49 * count_in/64];
__v2di *in_302 = (__v2di*)&data_in[50 * count_in/64];
__v2di *in_303 = (__v2di*)&data_in[51 * count_in/64];
__v2di *in_310 = (__v2di*)&data_in[52 * count_in/64];
__v2di *in_311 = (__v2di*)&data_in[53 * count_in/64];
__v2di *in_312 = (__v2di*)&data_in[54 * count_in/64];
__v2di *in_313 = (__v2di*)&data_in[55 * count_in/64];
__v2di *in_320 = (__v2di*)&data_in[56 * count_in/64];
__v2di *in_321 = (__v2di*)&data_in[57 * count_in/64];
__v2di *in_322 = (__v2di*)&data_in[58 * count_in/64];
__v2di *in_323 = (__v2di*)&data_in[59 * count_in/64];
__v2di *in_330 = (__v2di*)&data_in[60 * count_in/64];
__v2di *in_331 = (__v2di*)&data_in[61 * count_in/64];
__v2di *in_332 = (__v2di*)&data_in[62 * count_in/64];
__v2di *in_333 = (__v2di*)&data_in[63 * count_in/64];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32/2; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_200[2*i+1], in_200[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_200[2*i+1], in_200[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_300[2*i+1], in_300[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_300[2*i+1], in_300[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_210[2*i+1], in_210[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_210[2*i+1], in_210[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_310[2*i+1], in_310[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_310[2*i+1], in_310[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_020[2*i+1], in_020[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_020[2*i+1], in_020[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_120[2*i+1], in_120[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_120[2*i+1], in_120[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_220[2*i+1], in_220[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_220[2*i+1], in_220[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_320[2*i+1], in_320[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_320[2*i+1], in_320[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_030[2*i+1], in_030[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_030[2*i+1], in_030[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_130[2*i+1], in_130[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_130[2*i+1], in_130[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_230[2*i+1], in_230[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_230[2*i+1], in_230[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_330[2*i+1], in_330[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_330[2*i+1], in_330[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_201[2*i+1], in_201[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_201[2*i+1], in_201[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_301[2*i+1], in_301[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_301[2*i+1], in_301[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_211[2*i+1], in_211[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_211[2*i+1], in_211[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_311[2*i+1], in_311[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_311[2*i+1], in_311[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_021[2*i+1], in_021[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_021[2*i+1], in_021[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_121[2*i+1], in_121[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_121[2*i+1], in_121[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_221[2*i+1], in_221[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_221[2*i+1], in_221[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_321[2*i+1], in_321[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_321[2*i+1], in_321[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_031[2*i+1], in_031[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_031[2*i+1], in_031[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_131[2*i+1], in_131[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_131[2*i+1], in_131[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_231[2*i+1], in_231[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_231[2*i+1], in_231[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_331[2*i+1], in_331[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_331[2*i+1], in_331[2*i+0], mask_imag);
}
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < count_out/32/2; ++i)
{
uint64_t rev = __builtin_e2k_bitrevd(i);
int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555));
__v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110};
__v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514};
*(__v2du*)((void*)data_out + offset + 64*16) = __builtin_e2k_qppermb(in_002[2*i+1], in_002[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 65*16) = __builtin_e2k_qppermb(in_002[2*i+1], in_002[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 66*16) = __builtin_e2k_qppermb(in_102[2*i+1], in_102[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 67*16) = __builtin_e2k_qppermb(in_102[2*i+1], in_102[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 68*16) = __builtin_e2k_qppermb(in_202[2*i+1], in_202[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 69*16) = __builtin_e2k_qppermb(in_202[2*i+1], in_202[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 70*16) = __builtin_e2k_qppermb(in_302[2*i+1], in_302[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 71*16) = __builtin_e2k_qppermb(in_302[2*i+1], in_302[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 72*16) = __builtin_e2k_qppermb(in_012[2*i+1], in_012[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 73*16) = __builtin_e2k_qppermb(in_012[2*i+1], in_012[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 74*16) = __builtin_e2k_qppermb(in_112[2*i+1], in_112[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 75*16) = __builtin_e2k_qppermb(in_112[2*i+1], in_112[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 76*16) = __builtin_e2k_qppermb(in_212[2*i+1], in_212[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 77*16) = __builtin_e2k_qppermb(in_212[2*i+1], in_212[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 78*16) = __builtin_e2k_qppermb(in_312[2*i+1], in_312[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 79*16) = __builtin_e2k_qppermb(in_312[2*i+1], in_312[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 80*16) = __builtin_e2k_qppermb(in_022[2*i+1], in_022[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 81*16) = __builtin_e2k_qppermb(in_022[2*i+1], in_022[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 82*16) = __builtin_e2k_qppermb(in_122[2*i+1], in_122[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 83*16) = __builtin_e2k_qppermb(in_122[2*i+1], in_122[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 84*16) = __builtin_e2k_qppermb(in_222[2*i+1], in_222[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 85*16) = __builtin_e2k_qppermb(in_222[2*i+1], in_222[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 86*16) = __builtin_e2k_qppermb(in_322[2*i+1], in_322[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 87*16) = __builtin_e2k_qppermb(in_322[2*i+1], in_322[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 88*16) = __builtin_e2k_qppermb(in_032[2*i+1], in_032[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 89*16) = __builtin_e2k_qppermb(in_032[2*i+1], in_032[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 90*16) = __builtin_e2k_qppermb(in_132[2*i+1], in_132[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 91*16) = __builtin_e2k_qppermb(in_132[2*i+1], in_132[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 92*16) = __builtin_e2k_qppermb(in_232[2*i+1], in_232[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 93*16) = __builtin_e2k_qppermb(in_232[2*i+1], in_232[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 94*16) = __builtin_e2k_qppermb(in_332[2*i+1], in_332[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 95*16) = __builtin_e2k_qppermb(in_332[2*i+1], in_332[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 96*16) = __builtin_e2k_qppermb(in_003[2*i+1], in_003[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 97*16) = __builtin_e2k_qppermb(in_003[2*i+1], in_003[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 98*16) = __builtin_e2k_qppermb(in_103[2*i+1], in_103[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 99*16) = __builtin_e2k_qppermb(in_103[2*i+1], in_103[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 100*16) = __builtin_e2k_qppermb(in_203[2*i+1], in_203[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 101*16) = __builtin_e2k_qppermb(in_203[2*i+1], in_203[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 102*16) = __builtin_e2k_qppermb(in_303[2*i+1], in_303[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 103*16) = __builtin_e2k_qppermb(in_303[2*i+1], in_303[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 104*16) = __builtin_e2k_qppermb(in_013[2*i+1], in_013[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 105*16) = __builtin_e2k_qppermb(in_013[2*i+1], in_013[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 106*16) = __builtin_e2k_qppermb(in_113[2*i+1], in_113[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 107*16) = __builtin_e2k_qppermb(in_113[2*i+1], in_113[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 108*16) = __builtin_e2k_qppermb(in_213[2*i+1], in_213[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 109*16) = __builtin_e2k_qppermb(in_213[2*i+1], in_213[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 110*16) = __builtin_e2k_qppermb(in_313[2*i+1], in_313[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 111*16) = __builtin_e2k_qppermb(in_313[2*i+1], in_313[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 112*16) = __builtin_e2k_qppermb(in_023[2*i+1], in_023[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 113*16) = __builtin_e2k_qppermb(in_023[2*i+1], in_023[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 114*16) = __builtin_e2k_qppermb(in_123[2*i+1], in_123[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 115*16) = __builtin_e2k_qppermb(in_123[2*i+1], in_123[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 116*16) = __builtin_e2k_qppermb(in_223[2*i+1], in_223[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 117*16) = __builtin_e2k_qppermb(in_223[2*i+1], in_223[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 118*16) = __builtin_e2k_qppermb(in_323[2*i+1], in_323[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 119*16) = __builtin_e2k_qppermb(in_323[2*i+1], in_323[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 120*16) = __builtin_e2k_qppermb(in_033[2*i+1], in_033[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 121*16) = __builtin_e2k_qppermb(in_033[2*i+1], in_033[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 122*16) = __builtin_e2k_qppermb(in_133[2*i+1], in_133[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 123*16) = __builtin_e2k_qppermb(in_133[2*i+1], in_133[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 124*16) = __builtin_e2k_qppermb(in_233[2*i+1], in_233[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 125*16) = __builtin_e2k_qppermb(in_233[2*i+1], in_233[2*i+0], mask_imag);
*(__v2du*)((void*)data_out + offset + 126*16) = __builtin_e2k_qppermb(in_333[2*i+1], in_333[2*i+0], mask_real);
*(__v2du*)((void*)data_out + offset + 127*16) = __builtin_e2k_qppermb(in_333[2*i+1], in_333[2*i+0], mask_imag);
}
}
Основной цикл на ассемблере
.L15871:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0
}
.L15388:
{
loop_mode
disp %ctpr1, .L15388
bitrevd,0 %r0, %g16
addd,1 %r0, 0x1, %r0
movaqp,0 area=0, ind=16, am=1, be=0, %g18
movaqp,1 area=0, ind=0, am=0, be=0, %g17
movaqp,2 area=0, ind=16, am=1, be=0, %g20
movaqp,3 area=0, ind=0, am=0, be=0, %g19
}
{
loop_mode
disp %ctpr3, .L13219
shrd,0 %g16, %r7, %g25
shrd,1 %g16, %r6, %g16
movaqp,0 area=1, ind=16, am=1, be=0, %g22
movaqp,1 area=1, ind=0, am=0, be=0, %g21
movaqp,2 area=1, ind=16, am=1, be=0, %g24
movaqp,3 area=1, ind=0, am=0, be=0, %g23
}
{
loop_mode
andd,0 %g25, _f64,_lts0 0xaaaaaaaaaaaaaaaa, %g25
andd,1 %g16, _f64,_lts2 0x5555555555555555, %g16
movaqp,0 area=2, ind=16, am=1, be=0, %g27
movaqp,1 area=2, ind=0, am=0, be=0, %g26
movaqp,2 area=2, ind=16, am=1, be=0, %g29
movaqp,3 area=2, ind=0, am=0, be=0, %g28
}
{
loop_mode
ord,0 %g25, %g16, %g16
movaqp,0 area=3, ind=16, am=1, be=0, %g31
movaqp,1 area=3, ind=0, am=0, be=0, %g30
movaqp,2 area=3, ind=16, am=1, be=0, %r9
movaqp,3 area=3, ind=0, am=0, be=0, %r1
}
{
loop_mode
shld,0 %g16, 0x5, %g16
movaqp,0 area=4, ind=16, am=1, be=0, %r10
movaqp,1 area=4, ind=0, am=0, be=0, %g25
movaqp,2 area=4, ind=16, am=1, be=0, %r12
movaqp,3 area=4, ind=0, am=0, be=0, %r11
}
{
loop_mode
qppermb,0 %g18, %g17, %r4, %r17
qppermb,1 %g18, %g17, %r5, %g17
addd,2 %r2, %g16, %g16
qppermb,3 %g20, %g19, %r4, %g18
qppermb,4 %g20, %g19, %r5, %g19
movaqp,0 area=5, ind=16, am=1, be=0, %r14
movaqp,1 area=5, ind=0, am=0, be=0, %r13
movaqp,2 area=5, ind=16, am=1, be=0, %r16
movaqp,3 area=5, ind=0, am=0, be=0, %r15
}
{
loop_mode
qppermb,0 %g22, %g21, %r4, %r21
qppermb,1 %g22, %g21, %r5, %g21
stqp,2 %g16, _f16s,_lts0lo 0x410, %g17
qppermb,3 %g24, %g23, %r4, %g22
qppermb,4 %g24, %g23, %r5, %g23
movaqp,0 area=6, ind=16, am=1, be=0, %r18
movaqp,1 area=6, ind=0, am=0, be=0, %g20
movaqp,2 area=6, ind=16, am=1, be=0, %r20
movaqp,3 area=6, ind=0, am=0, be=0, %r19
}
{
loop_mode
qppermb,0 %g27, %g26, %r4, %r24
qppermb,1 %g27, %g26, %r5, %g26
stqp,2 %g16, _f16s,_lts0lo 0x400, %r17
qppermb,3 %g29, %g28, %r4, %g27
qppermb,4 %g29, %g28, %r5, %g28
stqp,5 %g16, _f16s,_lts0hi 0x430, %g19
movaqp,0 area=7, ind=16, am=1, be=0, %g24
movaqp,1 area=7, ind=0, am=0, be=0, %g17
movaqp,2 area=7, ind=16, am=1, be=0, %r23
movaqp,3 area=7, ind=0, am=0, be=0, %r22
}
{
loop_mode
qppermb,0 %g31, %g30, %r4, %r26
qppermb,1 %g31, %g30, %r5, %g30
stqp,2 %g16, _f16s,_lts0lo 0x420, %g18
qppermb,3 %r9, %r1, %r4, %g31
qppermb,4 %r9, %r1, %r5, %r1
stqp,5 %g16, _f16s,_lts0hi 0x470, %g23
movaqp,0 area=8, ind=16, am=1, be=0, %g29
movaqp,1 area=8, ind=0, am=0, be=0, %g19
movaqp,2 area=8, ind=16, am=1, be=0, %r25
movaqp,3 area=8, ind=0, am=0, be=0, %r17
}
{
loop_mode
qppermb,0 %r10, %g25, %r4, %r28
qppermb,1 %r10, %g25, %r5, %g25
stqp,2 %g16, _f16s,_lts0lo 0x440, %r21
qppermb,3 %r12, %r11, %r4, %r10
qppermb,4 %r12, %r11, %r5, %r11
stqp,5 %g16, _f16s,_lts0hi 0x450, %g21
movaqp,0 area=9, ind=16, am=1, be=0, %g23
movaqp,1 area=9, ind=0, am=0, be=0, %g18
movaqp,2 area=9, ind=16, am=1, be=0, %r27
movaqp,3 area=9, ind=0, am=0, be=0, %r9
}
{
loop_mode
qppermb,0 %r14, %r13, %r4, %r30
qppermb,1 %r14, %r13, %r5, %r13
stqp,2 %g16, _f16s,_lts0lo 0x460, %g22
qppermb,3 %r16, %r15, %r4, %r14
qppermb,4 %r16, %r15, %r5, %r15
stqp,5 %g16, _f16s,_lts0hi 0x490, %g26
movaqp,0 area=10, ind=16, am=1, be=0, %r12
movaqp,1 area=10, ind=0, am=0, be=0, %g21
movaqp,2 area=10, ind=16, am=1, be=0, %r29
movaqp,3 area=10, ind=0, am=0, be=0, %r21
}
{
loop_mode
qppermb,0 %r18, %g20, %r4, %r32
qppermb,1 %r18, %g20, %r5, %g20
stqp,2 %g16, _f16s,_lts0lo 0x480, %r24
qppermb,3 %r20, %r19, %r4, %r18
qppermb,4 %r20, %r19, %r5, %r19
stqp,5 %g16, _f16s,_lts0hi 0x4b0, %g28
movaqp,0 area=11, ind=16, am=1, be=0, %g26
movaqp,1 area=11, ind=0, am=0, be=0, %g22
movaqp,2 area=11, ind=16, am=1, be=0, %r31
movaqp,3 area=11, ind=0, am=0, be=0, %r16
}
{
loop_mode
qppermb,0 %g24, %g17, %r4, %r34
qppermb,1 %g24, %g17, %r5, %g17
stqp,2 %g16, _f16s,_lts0lo 0x4a0, %g27
qppermb,3 %r23, %r22, %r4, %g24
qppermb,4 %r23, %r22, %r5, %r22
stqp,5 %g16, _f16s,_lts0hi 0x4d0, %g30
movaqp,0 area=12, ind=16, am=1, be=0, %r20
movaqp,1 area=12, ind=0, am=0, be=0, %g28
movaqp,2 area=12, ind=16, am=1, be=0, %r33
movaqp,3 area=12, ind=0, am=0, be=0, %r24
}
{
loop_mode
qppermb,0 %g29, %g19, %r4, %r36
qppermb,1 %g29, %g19, %r5, %g19
stqp,2 %g16, _f16s,_lts0lo 0x4c0, %r26
qppermb,3 %r25, %r17, %r4, %g29
qppermb,4 %r25, %r17, %r5, %r17
stqp,5 %g16, _f16s,_lts0hi 0x4f0, %r1
movaqp,0 area=13, ind=16, am=1, be=0, %g30
movaqp,1 area=13, ind=0, am=0, be=0, %g27
movaqp,2 area=13, ind=16, am=1, be=0, %r35
movaqp,3 area=13, ind=0, am=0, be=0, %r23
}
{
loop_mode
qppermb,0 %g23, %g18, %r4, %r38
qppermb,1 %g23, %g18, %r5, %g18
stqp,2 %g16, _f16s,_lts0lo 0x4e0, %g31
qppermb,3 %r27, %r9, %r4, %g23
qppermb,4 %r27, %r9, %r5, %r9
stqp,5 %g16, _f16s,_lts0hi 0x510, %g25
movaqp,0 area=14, ind=16, am=1, be=0, %r25
movaqp,1 area=14, ind=0, am=0, be=0, %r1
movaqp,2 area=14, ind=16, am=1, be=0, %r37
movaqp,3 area=14, ind=0, am=0, be=0, %r26
}
{
loop_mode
qppermb,0 %r12, %g21, %r4, %r40
qppermb,1 %r12, %g21, %r5, %g21
stqp,2 %g16, _f16s,_lts0lo 0x500, %r28
qppermb,3 %r29, %r21, %r4, %r12
qppermb,4 %r29, %r21, %r5, %r21
stqp,5 %g16, _f16s,_lts0hi 0x530, %r11
movaqp,0 area=15, ind=16, am=1, be=0, %g31
movaqp,1 area=15, ind=0, am=0, be=0, %g25
movaqp,2 area=15, ind=16, am=1, be=0, %r39
movaqp,3 area=15, ind=0, am=0, be=0, %r27
}
{
loop_mode
qppermb,0 %g26, %g22, %r4, %r11
qppermb,1 %g26, %g22, %r5, %g22
stqp,2 %g16, _f16s,_lts0lo 0x520, %r10
qppermb,3 %r31, %r16, %r4, %g26
qppermb,4 %r31, %r16, %r5, %r16
stqp,5 %g16, _f16s,_lts0hi 0x550, %r13
}
{
loop_mode
qppermb,0 %r20, %g28, %r4, %r10
qppermb,1 %r20, %g28, %r5, %g28
stqp,2 %g16, _f16s,_lts0lo 0x540, %r30
qppermb,3 %r33, %r24, %r4, %r13
qppermb,4 %r33, %r24, %r5, %r20
stqp,5 %g16, _f16s,_lts0hi 0x570, %r15
}
{
loop_mode
qppermb,0 %g30, %g27, %r4, %r15
qppermb,1 %g30, %g27, %r5, %g27
stqp,2 %g16, _f16s,_lts0lo 0x560, %r14
qppermb,3 %r35, %r23, %r4, %g30
qppermb,4 %r35, %r23, %r5, %r23
stqp,5 %g16, _f16s,_lts0hi 0x590, %g20
}
{
loop_mode
qppermb,0 %r25, %r1, %r4, %g20
qppermb,1 %r25, %r1, %r5, %r1
stqp,2 %g16, _f16s,_lts0lo 0x580, %r32
qppermb,3 %r37, %r26, %r4, %r14
qppermb,4 %r37, %r26, %r5, %r24
stqp,5 %g16, _f16s,_lts0hi 0x5b0, %r19
}
{
loop_mode
qppermb,0 %g31, %g25, %r4, %r19
qppermb,1 %g31, %g25, %r5, %g25
stqp,2 %g16, _f16s,_lts0lo 0x5a0, %r18
qppermb,3 %r39, %r27, %r4, %g31
qppermb,4 %r39, %r27, %r5, %r25
stqp,5 %g16, _f16s,_lts0hi 0x5d0, %g17
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x5c0, %r34
stqp,5 %g16, _f16s,_lts0hi 0x5f0, %r22
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x5e0, %g24
stqp,5 %g16, _f16s,_lts0hi 0x610, %g19
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x600, %r36
stqp,5 %g16, _f16s,_lts0hi 0x630, %r17
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x620, %g29
stqp,5 %g16, _f16s,_lts0hi 0x650, %g18
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x640, %r38
stqp,5 %g16, _f16s,_lts0hi 0x670, %r9
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x660, %g23
stqp,5 %g16, _f16s,_lts0hi 0x690, %g21
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x680, %r40
stqp,5 %g16, _f16s,_lts0hi 0x6b0, %r21
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x6a0, %r12
stqp,5 %g16, _f16s,_lts0hi 0x6d0, %g22
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x6c0, %r11
stqp,5 %g16, _f16s,_lts0hi 0x6f0, %r16
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x6e0, %g26
stqp,5 %g16, _f16s,_lts0hi 0x710, %g28
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x700, %r10
stqp,5 %g16, _f16s,_lts0hi 0x730, %r20
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x720, %r13
stqp,5 %g16, _f16s,_lts0hi 0x750, %g27
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x740, %r15
stqp,5 %g16, _f16s,_lts0hi 0x770, %r23
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x760, %g30
stqp,5 %g16, _f16s,_lts0hi 0x790, %r1
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x780, %g20
stqp,5 %g16, _f16s,_lts0hi 0x7b0, %r24
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x7a0, %r14
stqp,5 %g16, _f16s,_lts0hi 0x7d0, %g25
}
{
loop_mode
stqp,2 %g16, _f16s,_lts0lo 0x7c0, %r19
stqp,5 %g16, _f16s,_lts0hi 0x7f0, %r25
}
{
loop_mode
ct %ctpr1 ? %NOT_LOOP_END
alc alcf=0, alct=1
stqp,2 %g16, _f16s,_lts0lo 0x7e0, %g31
}
{
loop_mode
ct %ctpr3
alc alcf=0, alct=1
}
.L16599:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0
}
.L15607:
{
loop_mode
disp %ctpr1, .L15607
bitrevd,0 %r0, %g16
addd,1 %r0, 0x1, %r0
movaqp,0 area=0, ind=16, am=1, be=0, %g18
movaqp,1 area=0, ind=0, am=0, be=0, %g17
movaqp,2 area=0, ind=16, am=1, be=0, %g20
movaqp,3 area=0, ind=0, am=0, be=0, %g19
}
{
loop_mode
disp %ctpr3, .L15094
shrd,0 %g16, %r7, %g25
shrd,1 %g16, %r6, %g16
movaqp,0 area=1, ind=16, am=1, be=0, %g22
movaqp,1 area=1, ind=0, am=0, be=0, %g21
movaqp,2 area=1, ind=16, am=1, be=0, %g24
movaqp,3 area=1, ind=0, am=0, be=0, %g23
}
{
loop_mode
andd,0 %g25, _f64,_lts0 0xaaaaaaaaaaaaaaaa, %g25
andd,1 %g16, _f64,_lts2 0x5555555555555555, %g16
movaqp,0 area=2, ind=16, am=1, be=0, %g27
movaqp,1 area=2, ind=0, am=0, be=0, %g26
movaqp,2 area=2, ind=16, am=1, be=0, %g29
movaqp,3 area=2, ind=0, am=0, be=0, %g28
}
{
loop_mode
ord,0 %g25, %g16, %g16
movaqp,0 area=3, ind=16, am=1, be=0, %g31
movaqp,1 area=3, ind=0, am=0, be=0, %g30
movaqp,2 area=3, ind=16, am=1, be=0, %r14
movaqp,3 area=3, ind=0, am=0, be=0, %r11
}
{
loop_mode
shld,0 %g16, 0x5, %g16
movaqp,0 area=4, ind=16, am=1, be=0, %r18
movaqp,1 area=4, ind=0, am=0, be=0, %g25
movaqp,2 area=4, ind=16, am=1, be=0, %r20
movaqp,3 area=4, ind=0, am=0, be=0, %r19
}
{
loop_mode
qppermb,0 %g18, %g17, %r4, %r31
qppermb,1 %g18, %g17, %r5, %g17
addd,2 %r2, %g16, %g20
qppermb,3 %g20, %g19, %r4, %g18
qppermb,4 %g20, %g19, %r5, %g19
movaqp,0 area=5, ind=16, am=1, be=0, %r25
movaqp,1 area=5, ind=0, am=0, be=0, %r23
movaqp,2 area=5, ind=16, am=1, be=0, %r29
movaqp,3 area=5, ind=0, am=0, be=0, %r28
}
{
loop_mode
qppermb,0 %g22, %g21, %r4, %r50
qppermb,1 %g22, %g21, %r5, %g21
stqp,2 %r2, %g16, %r31
qppermb,3 %g24, %g23, %r4, %g22
qppermb,4 %g24, %g23, %r5, %g23
movaqp,0 area=6, ind=16, am=1, be=0, %r37
movaqp,1 area=6, ind=0, am=0, be=0, %r34
movaqp,2 area=6, ind=16, am=1, be=0, %r46
movaqp,3 area=6, ind=0, am=0, be=0, %r39
}
{
loop_mode
qppermb,0 %g27, %g26, %r4, %r56
qppermb,1 %g27, %g26, %r5, %g26
stqp,2 0x10, %g20, %g17
qppermb,3 %g29, %g28, %r4, %g27
qppermb,4 %g29, %g28, %r5, %g28
stqp,5 %g20, _f16s,_lts0lo 0x30, %g19
movaqp,0 area=7, ind=16, am=1, be=0, %g24
movaqp,1 area=7, ind=0, am=0, be=0, %g16
movaqp,2 area=7, ind=16, am=1, be=0, %r53
movaqp,3 area=7, ind=0, am=0, be=0, %r31
}
{
loop_mode
qppermb,0 %g31, %g30, %r4, %r58
qppermb,1 %g31, %g30, %r5, %g30
stqp,2 %g20, _f16s,_lts0lo 0x20, %g18
qppermb,3 %r14, %r11, %r4, %g31
qppermb,4 %r14, %r11, %r5, %r11
stqp,5 %g20, _f16s,_lts0hi 0x70, %g23
movaqp,0 area=8, ind=16, am=1, be=0, %g19
movaqp,1 area=8, ind=0, am=0, be=0, %g17
movaqp,2 area=8, ind=16, am=1, be=0, %r57
movaqp,3 area=8, ind=0, am=0, be=0, %g29
}
{
loop_mode
qppermb,0 %r18, %g25, %r4, %r61
qppermb,1 %r18, %g25, %r5, %g25
stqp,2 %g20, _f16s,_lts0lo 0x40, %r50
qppermb,3 %r20, %r19, %r4, %r18
qppermb,4 %r20, %r19, %r5, %r19
stqp,5 %g20, _f16s,_lts0hi 0x50, %g21
movaqp,0 area=9, ind=16, am=1, be=0, %g23
movaqp,1 area=9, ind=0, am=0, be=0, %g18
movaqp,2 area=9, ind=16, am=1, be=0, %r59
movaqp,3 area=9, ind=0, am=0, be=0, %r14
}
{
loop_mode
qppermb,0 %r25, %r23, %r4, %r63
qppermb,1 %r25, %r23, %r5, %r23
stqp,2 %g20, _f16s,_lts0lo 0x60, %g22
qppermb,3 %r29, %r28, %r4, %r25
qppermb,4 %r29, %r28, %r5, %r28
stqp,5 %g20, _f16s,_lts0hi 0x90, %g26
movaqp,0 area=10, ind=16, am=1, be=0, %r20
movaqp,1 area=10, ind=0, am=0, be=0, %g21
movaqp,2 area=10, ind=16, am=1, be=0, %r62
movaqp,3 area=10, ind=0, am=0, be=0, %r50
}
{
loop_mode
qppermb,0 %r37, %r34, %r4, %b[1]
qppermb,1 %r37, %r34, %r5, %r34
stqp,2 %g20, _f16s,_lts0lo 0x80, %r56
qppermb,3 %r46, %r39, %r4, %r37
qppermb,4 %r46, %r39, %r5, %r39
stqp,5 %g20, _f16s,_lts0hi 0xb0, %g28
movaqp,0 area=11, ind=16, am=1, be=0, %g26
movaqp,1 area=11, ind=0, am=0, be=0, %g22
movaqp,2 area=11, ind=16, am=1, be=0, %b[0]
movaqp,3 area=11, ind=0, am=0, be=0, %r29
}
{
loop_mode
qppermb,0 %g24, %g16, %r4, %b[3]
qppermb,1 %g24, %g16, %r5, %g16
stqp,2 %g20, _f16s,_lts0lo 0xa0, %g27
qppermb,3 %r53, %r31, %r4, %g24
qppermb,4 %r53, %r31, %r5, %r31
stqp,5 %g20, _f16s,_lts0hi 0xd0, %g30
movaqp,0 area=12, ind=16, am=1, be=0, %r46
movaqp,1 area=12, ind=0, am=0, be=0, %g28
movaqp,2 area=12, ind=16, am=1, be=0, %b[2]
movaqp,3 area=12, ind=0, am=0, be=0, %r56
}
{
loop_mode
qppermb,0 %g19, %g17, %r4, %b[5]
qppermb,1 %g19, %g17, %r5, %g17
stqp,2 %g20, _f16s,_lts0lo 0xc0, %r58
qppermb,3 %r57, %g29, %r4, %g19
qppermb,4 %r57, %g29, %r5, %g29
stqp,5 %g20, _f16s,_lts0hi 0xf0, %r11
movaqp,0 area=13, ind=16, am=1, be=0, %g30
movaqp,1 area=13, ind=0, am=0, be=0, %g27
movaqp,2 area=13, ind=16, am=1, be=0, %b[4]
movaqp,3 area=13, ind=0, am=0, be=0, %r53
}
{
loop_mode
qppermb,0 %g23, %g18, %r4, %b[8]
qppermb,1 %g23, %g18, %r5, %g18
stqp,2 %g20, _f16s,_lts0lo 0xe0, %g31
qppermb,3 %r59, %r14, %r4, %g23
qppermb,4 %r59, %r14, %r5, %r14
stqp,5 %g20, _f16s,_lts0hi 0x110, %g25
movaqp,0 area=14, ind=16, am=1, be=0, %r57
movaqp,1 area=14, ind=0, am=0, be=0, %r11
movaqp,2 area=14, ind=16, am=1, be=0, %b[6]
movaqp,3 area=14, ind=0, am=0, be=0, %r58
}
{
loop_mode
qppermb,0 %r20, %g21, %r4, %b[10]
qppermb,1 %r20, %g21, %r5, %g21
stqp,2 %g20, _f16s,_lts0lo 0x100, %r61
qppermb,3 %r62, %r50, %r4, %r20
qppermb,4 %r62, %r50, %r5, %r50
stqp,5 %g20, _f16s,_lts0hi 0x130, %r19
movaqp,0 area=15, ind=16, am=1, be=0, %g31
movaqp,1 area=15, ind=0, am=0, be=0, %g25
movaqp,2 area=15, ind=16, am=1, be=0, %b[9]
movaqp,3 area=15, ind=0, am=0, be=0, %r59
}
{
loop_mode
qppermb,0 %g26, %g22, %r4, %r19
qppermb,1 %g26, %g22, %r5, %g22
stqp,2 %g20, _f16s,_lts0lo 0x120, %r18
qppermb,3 %b[0], %r29, %r4, %g26
qppermb,4 %b[0], %r29, %r5, %r29
stqp,5 %g20, _f16s,_lts0hi 0x150, %r23
}
{
loop_mode
qppermb,0 %r46, %g28, %r4, %r18
qppermb,1 %r46, %g28, %r5, %g28
stqp,2 %g20, _f16s,_lts0lo 0x140, %r63
qppermb,3 %b[2], %r56, %r4, %r23
qppermb,4 %b[2], %r56, %r5, %r46
stqp,5 %g20, _f16s,_lts0hi 0x170, %r28
}
{
loop_mode
qppermb,0 %g30, %g27, %r4, %r28
qppermb,1 %g30, %g27, %r5, %g27
stqp,2 %g20, _f16s,_lts0lo 0x160, %r25
qppermb,3 %b[4], %r53, %r4, %g30
qppermb,4 %b[4], %r53, %r5, %r53
stqp,5 %g20, _f16s,_lts0hi 0x190, %r34
}
{
loop_mode
qppermb,0 %r57, %r11, %r4, %r25
qppermb,1 %r57, %r11, %r5, %r11
stqp,2 %g20, _f16s,_lts0lo 0x180, %b[1]
qppermb,3 %b[6], %r58, %r4, %r34
qppermb,4 %b[6], %r58, %r5, %r56
stqp,5 %g20, _f16s,_lts0hi 0x1b0, %r39
}
{
loop_mode
qppermb,0 %g31, %g25, %r4, %r39
qppermb,1 %g31, %g25, %r5, %g25
stqp,2 %g20, _f16s,_lts0lo 0x1a0, %r37
qppermb,3 %b[9], %r59, %r4, %g31
qppermb,4 %b[9], %r59, %r5, %r57
stqp,5 %g20, _f16s,_lts0hi 0x1d0, %g16
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x1c0, %b[3]
stqp,5 %g20, _f16s,_lts0hi 0x1f0, %r31
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x1e0, %g24
stqp,5 %g20, _f16s,_lts0hi 0x210, %g17
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x200, %b[5]
stqp,5 %g20, _f16s,_lts0hi 0x230, %g29
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x220, %g19
stqp,5 %g20, _f16s,_lts0hi 0x250, %g18
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x240, %b[8]
stqp,5 %g20, _f16s,_lts0hi 0x270, %r14
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x260, %g23
stqp,5 %g20, _f16s,_lts0hi 0x290, %g21
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x280, %b[10]
stqp,5 %g20, _f16s,_lts0hi 0x2b0, %r50
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x2a0, %r20
stqp,5 %g20, _f16s,_lts0hi 0x2d0, %g22
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x2c0, %r19
stqp,5 %g20, _f16s,_lts0hi 0x2f0, %r29
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x2e0, %g26
stqp,5 %g20, _f16s,_lts0hi 0x310, %g28
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x300, %r18
stqp,5 %g20, _f16s,_lts0hi 0x330, %r46
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x320, %r23
stqp,5 %g20, _f16s,_lts0hi 0x350, %g27
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x340, %r28
stqp,5 %g20, _f16s,_lts0hi 0x370, %r53
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x360, %g30
stqp,5 %g20, _f16s,_lts0hi 0x390, %r11
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x380, %r25
stqp,5 %g20, _f16s,_lts0hi 0x3b0, %r56
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x3a0, %r34
stqp,5 %g20, _f16s,_lts0hi 0x3d0, %g25
}
{
loop_mode
stqp,2 %g20, _f16s,_lts0lo 0x3c0, %r39
stqp,5 %g20, _f16s,_lts0hi 0x3f0, %r57
}
{
loop_mode
ct %ctpr1 ? %NOT_LOOP_END
alc alcf=0, alct=1
stqp,2 %g20, _f16s,_lts0lo 0x3e0, %g31
}
{
loop_mode
ct %ctpr3
alc alcf=0, alct=1
}
Теоретическая скорость: 128 комплексных чисел за 40 тактов (128/40) = 25.6 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
Итоги по reverse_radix4_vector4
Победителем можно считать reverse_radix4_vector4_stream16.
При реализации Vector-4 Radix-4 FFT будем использовать его.
Пишем векторный Stage
stage_radix2_vector2
1. stage_radix2_vector2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix2_vector2_etalon(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coef)
{
myComplex2 *x_in = &data_in[0];
myComplex2 *y_in = &data_in[1];
myComplex2 *c_in = coef;
myComplex2 *out_add = &data_out[0];
myComplex2 *out_sub = &data_out[data_count/2];
#pragma ivdep
#pragma unroll(1)
// #pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
myComplex2 x = x_in[2*i];
myComplex2 y = y_in[2*i];
myComplex2 c = c_in[i];
myComplex2 cy = complex2_mul(c, y);
out_add[i] = complex2_add(x, cy);
out_sub[i] = complex2_sub(x, cy);
}
}
Основной цикл на ассемблере
.L917:
{
fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0
}
.L546:
{
loop_mode
pfmuls,0,sm %b[67], %b[6], %b[37]
pfsubs,1,sm %b[46], %b[24], %b[57]
staad,2 %b[61], %aad1[ %aasti3 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[55], %b[13], %b[43], %b[14]
pfadds,4,sm %b[46], %b[24], %b[58]
staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=16, am=0, be=0, %b[0]
movad,1 area=0, ind=24, am=0, be=0, %b[1]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfmuls,0,sm %b[67], %b[7], %b[62]
pfsubs,1,sm %b[35], %b[56], %b[69]
staad,2 %b[73], %aad1[ %aasti3 ]
incr,2 %aaincr3
pfmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46]
pfadds,4,sm %b[35], %b[56], %b[70]
staad,5 %b[74], %aad2[ %aasti4 ]
incr,5 %aaincr3
movad,0 area=0, ind=0, am=0, be=0, %b[13]
movad,1 area=0, ind=8, am=1, be=0, %b[24]
movad,2 area=0, ind=8, am=1, be=0, %b[61]
movad,3 area=0, ind=0, am=0, be=0, %b[43]
}
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
2. stage_radix2_vector2
Прямая векторизация базового алгоритма в 2 раза с помощью инструкций SIMD64.
Код на Си
void stage_radix2_vector2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coef)
{
uint64_t *x_real_in = (uint64_t*)&data_in[0].real;
uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag;
uint64_t *y_real_in = (uint64_t*)&data_in[1].real;
uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag;
uint64_t *c_real_in = (uint64_t*)&coef[0].real;
uint64_t *c_imag_in = (uint64_t*)&coef[0].imag;
uint64_t *out_add_real = (uint64_t*)&data_out[0].real;
uint64_t *out_add_imag = (uint64_t*)&data_out[0].imag;
uint64_t *out_sub_real = (uint64_t*)&data_out[data_count/2].real;
uint64_t *out_sub_imag = (uint64_t*)&data_out[data_count/2].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
uint64_t x_real = x_real_in[4*i];
uint64_t x_imag = x_imag_in[4*i];
uint64_t y_real = y_real_in[4*i];
uint64_t y_imag = y_imag_in[4*i];
uint64_t c_real = c_real_in[2*i];
uint64_t c_imag = c_imag_in[2*i];
uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag));
uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real));
out_add_real[2*i] = __builtin_e2k_pfadds(x_real, cy_real);
out_add_imag[2*i] = __builtin_e2k_pfadds(x_imag, cy_imag);
out_sub_real[2*i] = __builtin_e2k_pfsubs(x_real, cy_real);
out_sub_imag[2*i] = __builtin_e2k_pfsubs(x_imag, cy_imag);
}
}
Основной цикл на ассемблере
.L1379:
{
fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0
}
.L1117:
{
loop_mode
pfmuls,0,sm %b[67], %b[6], %b[37]
pfsubs,1,sm %b[46], %b[24], %b[57]
staad,2 %b[61], %aad1[ %aasti3 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[55], %b[13], %b[43], %b[14]
pfadds,4,sm %b[46], %b[24], %b[58]
staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=16, am=0, be=0, %b[0]
movad,1 area=0, ind=24, am=0, be=0, %b[1]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfmuls,0,sm %b[67], %b[7], %b[62]
pfsubs,1,sm %b[35], %b[56], %b[69]
staad,2 %b[73], %aad1[ %aasti3 ]
incr,2 %aaincr3
pfmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46]
pfadds,4,sm %b[35], %b[56], %b[70]
staad,5 %b[74], %aad2[ %aasti4 ]
incr,5 %aaincr3
movad,0 area=0, ind=0, am=0, be=0, %b[13]
movad,1 area=0, ind=8, am=1, be=0, %b[24]
movad,2 area=0, ind=8, am=1, be=0, %b[61]
movad,3 area=0, ind=0, am=0, be=0, %b[43]
}
Код на ассемблере получился такой же, как в эталонном варианте, то есть компилятор самостоятельно векторизовал эталонный вариант в 2 раза.
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Из консольного вывода от разных размеров массива видно, что эталонный вариант всегда выполняется на 40–50 тактов быстрее.
Я не смог понять, с чем это связано (особенности расположения кода в памяти?).
Итоги по stage_radix2_vector2
stage_radix2_vector4
1. stage_radix2_vector4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix2_vector4_etalon(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coef)
{
myComplex4 *x_in = &data_in[0];
myComplex4 *y_in = &data_in[1];
myComplex4 *c_in = coef;
myComplex4 *out_add = &data_out[0];
myComplex4 *out_sub = &data_out[data_count/2];
#pragma ivdep
#pragma unroll(1)
// #pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
myComplex4 x = x_in[2*i];
myComplex4 y = y_in[2*i];
myComplex4 c = c_in[i];
myComplex4 cy = complex4_mul(c, y);
out_add[i] = complex4_add(x, cy);
out_sub[i] = complex4_sub(x, cy);
}
}
Основной цикл на ассемблере
.L1531:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=0, disp=16
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=32
}
.L926:
{
loop_mode
pfmuls,0,sm %b[77], %b[39], %b[22]
pfsubs,1,sm %b[36], %b[69], %b[52]
staad,2 %b[54], %aad1[ %aasti3 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[51], %b[43], %b[72], %b[7]
pfadds,4,sm %b[36], %b[69], %b[60]
staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ]
movad,2 area=1, ind=16, am=0, be=0, %b[0]
movad,3 area=0, ind=0, am=0, be=0, %b[1]
}
{
loop_mode
pfmuls,0,sm %b[5], %b[4], %b[62]
pfsubs,1,sm %b[23], %b[63], %b[66]
staad,2 %b[68], %aad1[ %aasti3 ]
pfmul_adds,3,sm %b[44], %b[8], %b[78], %b[54]
pfadds,4,sm %b[23], %b[63], %b[69]
staad,5 %b[71], %aad2[ %aasti4 ]
movad,0 area=1, ind=0, am=0, be=0, %b[36]
movad,1 area=1, ind=8, am=1, be=0, %b[43]
movad,3 area=1, ind=8, am=0, be=0, %b[51]
}
{
loop_mode
pfmuls,0,sm %b[77], %b[55], %b[68]
pfsubs,1,sm %b[35], %b[11], %b[71]
staad,2 %b[73], %aad1[ %aasti3 + _f32s,_lts0 0x18 ]
pfmul_rsubs,3,sm %b[49], %b[57], %b[24], %b[63]
pfadds,4,sm %b[35], %b[11], %b[72]
staad,5 %b[74], %aad2[ %aasti4 + _f32s,_lts0 0x18 ]
movad,0 area=0, ind=16, am=0, be=0, %b[8]
movad,1 area=0, ind=24, am=0, be=0, %b[23]
movad,3 area=1, ind=0, am=0, be=0, %b[44]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfmuls,0,sm %b[5], %b[48], %b[74]
pfsubs,1,sm %b[20], %b[58], %b[77]
staad,2 %b[79], %aad1[ %aasti3 + _f32s,_lts0 0x10 ]
incr,2 %aaincr3
pfmul_rsubs,3,sm %b[42], %b[50], %b[64], %b[57]
pfadds,4,sm %b[20], %b[58], %b[78]
staad,5 %b[80], %aad2[ %aasti4 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
movad,0 area=0, ind=0, am=0, be=0, %b[11]
movad,1 area=0, ind=8, am=1, be=0, %b[24]
movad,2 area=1, ind=24, am=1, be=0, %b[35]
movad,3 area=0, ind=8, am=1, be=0, %b[73]
}
Теоретическая скорость: 8 комплексных чисел за 4 такта (8/4) = 16 Байт/такт
Замеры скорости
2. stage_radix2_vector4
Прямая векторизация базового алгоритма в 4 раза с помощью инструкций SIMD128.
Код на Си
void stage_radix2_vector4(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coef)
{
__v2di *x_real_in = (__v2di*)&data_in[0].real;
__v2di *x_imag_in = (__v2di*)&data_in[0].imag;
__v2di *y_real_in = (__v2di*)&data_in[1].real;
__v2di *y_imag_in = (__v2di*)&data_in[1].imag;
__v2di *c_real_in = (__v2di*)&coef[0].real;
__v2di *c_imag_in = (__v2di*)&coef[0].imag;
__v2di *out_add_real = (__v2di*)&data_out[0].real;
__v2di *out_add_imag = (__v2di*)&data_out[0].imag;
__v2di *out_sub_real = (__v2di*)&data_out[data_count/2].real;
__v2di *out_sub_imag = (__v2di*)&data_out[data_count/2].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
__v2di x_real = x_real_in[4*i];
__v2di x_imag = x_imag_in[4*i];
__v2di y_real = y_real_in[4*i];
__v2di y_imag = y_imag_in[4*i];
__v2di c_real = c_real_in[2*i];
__v2di c_imag = c_imag_in[2*i];
__v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag));
__v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real));
out_add_real[2*i] = __builtin_e2k_qpfadds(x_real, cy_real);
out_add_imag[2*i] = __builtin_e2k_qpfadds(x_imag, cy_imag);
out_sub_real[2*i] = __builtin_e2k_qpfsubs(x_real, cy_real);
out_sub_imag[2*i] = __builtin_e2k_qpfsubs(x_imag, cy_imag);
}
}
Основной цикл на ассемблере
.L1875:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L1577:
{
loop_mode
qpfmuls,0,sm %b[43], %b[52], %b[45]
qpfsubs,1,sm %b[46], %b[24], %b[60]
staaqp,2 %b[64], %aad1[ %aasti3 + _f32s,_lts0 0x10 ]
qpfmul_adds,3,sm %b[12], %b[13], %b[51], %b[14]
qpfadds,4,sm %b[46], %b[24], %b[63]
staaqp,5 %b[67], %aad2[ %aasti4 + _f32s,_lts0 0x10 ]
movaqp,0 area=1, ind=16, am=1, be=0, %b[37]
movaqp,1 area=1, ind=0, am=0, be=0, %b[0]
movaqp,3 area=0, ind=16, am=0, be=0, %b[1]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfmuls,0,sm %b[43], %b[7], %b[64]
qpfsubs,1,sm %b[35], %b[61], %b[67]
staaqp,2 %b[71], %aad1[ %aasti3 ]
incr,2 %aaincr3
qpfmul_rsubs,3,sm %b[12], %b[58], %b[70], %b[51]
qpfadds,4,sm %b[35], %b[61], %b[72]
staaqp,5 %b[76], %aad2[ %aasti4 ]
incr,5 %aaincr3
movaqp,0 area=0, ind=0, am=0, be=0, %b[13]
movaqp,1 area=0, ind=16, am=1, be=0, %b[24]
movaqp,3 area=0, ind=0, am=1, be=0, %b[46]
}
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Компилятор не сумел векторизовать вариант stage_radix2_vector4_etalon в 4 раза.
До этого он успешно векторизовал вариант stage_radix2_vector2_etalon в 2 раза.
Итоги по stage_radix2_vector4
stage_radix4_vector2
Один проход по stage_radix4_vector2 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_vector2 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_vector2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix4_vector2_etalon(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE)
{
myComplex2 *x_in = &data_in[0];
myComplex2 *y_in = &data_in[1];
myComplex2 *z_in = &data_in[2];
myComplex2 *w_in = &data_in[3];
myComplex2 *c_in = coefC;
myComplex2 *d_in = coefD;
myComplex2 *e_in = coefE;
myComplex2 *out_0 = &data_out[0*data_count/4];
myComplex2 *out_1 = &data_out[1*data_count/4];
myComplex2 *out_2 = &data_out[2*data_count/4];
myComplex2 *out_3 = &data_out[3*data_count/4];
#pragma ivdep
#pragma unroll(1)
// #pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
myComplex2 x = x_in[4*i];
myComplex2 y = y_in[4*i];
myComplex2 z = z_in[4*i];
myComplex2 w = w_in[4*i];
myComplex2 c = c_in[i];
myComplex2 d = d_in[i];
myComplex2 e = e_in[i];
myComplex2 cy = complex2_mul(c, y);
myComplex2 dz = complex2_mul(d, z);
myComplex2 ew = complex2_mul(e, w);
myComplex2 add02 = complex2_add( x, dz);
myComplex2 sub02 = complex2_sub( x, dz);
myComplex2 add13 = complex2_add(cy, ew);
myComplex2 sub13 = complex2_sub(cy, ew);
myComplex2 sub13i = complex2_mul_i(sub13);
out_0[i] = complex2_add(add02, add13);
out_1[i] = complex2_sub(sub02, sub13i);
out_2[i] = complex2_sub(add02, add13);
out_3[i] = complex2_add(sub02, sub13i);
}
}
Основной цикл на ассемблере
.L2695:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L1896:
{
loop_mode
pfmul_adds,0,sm %b[48], %b[70], %b[42], %b[0]
pfsub_rsubs,1,sm %b[37], %b[73], %b[59], %b[4]
pfsub_adds,2,sm %b[37], %b[73], %b[59], %b[1]
pfmuls,4,sm %b[63], %b[62], %b[40]
pfsubs,5,sm %b[78], %b[74], %b[57]
}
{
loop_mode
pfmul_rsubs,0,sm %b[48], %b[64], %b[83], %b[70]
pfadd_adds,1,sm %b[25], %b[82], %b[90], %b[77]
pfadd_rsubs,2,sm %b[25], %b[82], %b[90], %b[73]
pfmuls,4,sm %b[63], %b[68], %b[81]
pfadds,5,sm %b[78], %b[74], %b[88]
movad,0 area=2, ind=0, am=0, be=0, %b[42]
movad,1 area=2, ind=8, am=1, be=0, %b[59]
movad,2 area=1, ind=0, am=0, be=0, %b[37]
movad,3 area=0, ind=16, am=0, be=0, %b[58]
}
{
loop_mode
pfmul_rsubs,0,sm %b[69], %b[13], %b[84], %b[74]
pfmul_rsubs,1,sm %b[45], %b[34], %b[85], %b[78]
staad,2 %b[49], %aad3[ %aasti7 ]
pfmuls,3,sm %b[14], %b[53], %b[82]
pfmuls,4,sm %b[54], %b[20], %b[83]
staad,5 %b[26], %aad1[ %aasti5 ]
movad,0 area=1, ind=0, am=0, be=0, %b[63]
movad,1 area=0, ind=8, am=0, be=0, %b[25]
movad,2 area=1, ind=8, am=1, be=0, %b[48]
movad,3 area=0, ind=24, am=0, be=0, %b[64]
}
{
loop_mode
pfmul_adds,0,sm %b[67], %b[53], %b[86], %b[34]
pfmul_adds,1,sm %b[45], %b[22], %b[87], %b[69]
staad,2 %b[10], %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
pfmuls,3,sm %b[12], %b[9], %b[84]
pfmuls,4,sm %b[54], %b[32], %b[85]
staad,5 %b[7], %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=0, am=0, be=0, %b[13]
movad,1 area=0, ind=24, am=0, be=0, %b[49]
movad,2 area=0, ind=0, am=0, be=0, %b[26]
movad,3 area=0, ind=8, am=1, be=0, %b[14]
}
{
loop_mode
pfsub_rsubs,0,sm %b[23], %b[80], %b[55], %b[45]
pfsub_adds,1,sm %b[23], %b[80], %b[55], %b[22]
staad,2 %b[6], %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
incr,2 %aaincr3
pfsubs,4,sm %b[2], %b[38], %b[53]
staad,5 %b[3], %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
incr,5 %aaincr3
movad,0 area=1, ind=8, am=1, be=0, %b[10]
movad,1 area=0, ind=16, am=1, be=0, %b[7]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfadd_adds,0,sm %b[35], %b[71], %b[56], %b[6]
pfadd_rsubs,1,sm %b[35], %b[71], %b[56], %b[3]
staad,2 %b[79], %aad4[ %aasti8 ]
incr,2 %aaincr3
pfadds,4,sm %b[38], %b[2], %b[54]
staad,5 %b[75], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
2. stage_radix4_vector2
Прямая векторизация базового алгоритма в 2 раза с помощью инструкций SIMD64.
Код на Си
void stage_radix4_vector2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE)
{
uint64_t *x_real_in = (uint64_t*)&data_in[0].real;
uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag;
uint64_t *y_real_in = (uint64_t*)&data_in[1].real;
uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag;
uint64_t *z_real_in = (uint64_t*)&data_in[2].real;
uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag;
uint64_t *w_real_in = (uint64_t*)&data_in[3].real;
uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag;
uint64_t *c_real_in = (uint64_t*)&coefC[0].real;
uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag;
uint64_t *d_real_in = (uint64_t*)&coefD[0].real;
uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag;
uint64_t *e_real_in = (uint64_t*)&coefE[0].real;
uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag;
uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real;
uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag;
uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real;
uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag;
uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real;
uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag;
uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real;
uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
uint64_t x_real = x_real_in[8*i];
uint64_t x_imag = x_imag_in[8*i];
uint64_t y_real = y_real_in[8*i];
uint64_t y_imag = y_imag_in[8*i];
uint64_t z_real = z_real_in[8*i];
uint64_t z_imag = z_imag_in[8*i];
uint64_t w_real = w_real_in[8*i];
uint64_t w_imag = w_imag_in[8*i];
uint64_t c_real = c_real_in[2*i];
uint64_t c_imag = c_imag_in[2*i];
uint64_t d_real = d_real_in[2*i];
uint64_t d_imag = d_imag_in[2*i];
uint64_t e_real = e_real_in[2*i];
uint64_t e_imag = e_imag_in[2*i];
uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag));
uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real));
uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag));
uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real));
uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag));
uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real));
uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real);
uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag);
uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real);
uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag);
uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real);
uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag);
uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real);
// uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag);
// uint64_t sub13i_real = -sub13_imag;
uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag);
uint64_t sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L3659:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L3151:
{
loop_mode
pfmul_adds,0,sm %b[48], %b[70], %b[42], %b[0]
pfsub_rsubs,1,sm %b[37], %b[73], %b[59], %b[4]
pfsub_adds,2,sm %b[37], %b[73], %b[59], %b[1]
pfmuls,4,sm %b[63], %b[62], %b[40]
pfsubs,5,sm %b[78], %b[74], %b[57]
}
{
loop_mode
pfmul_rsubs,0,sm %b[48], %b[64], %b[83], %b[70]
pfadd_adds,1,sm %b[25], %b[82], %b[90], %b[77]
pfadd_rsubs,2,sm %b[25], %b[82], %b[90], %b[73]
pfmuls,4,sm %b[63], %b[68], %b[81]
pfadds,5,sm %b[78], %b[74], %b[88]
movad,0 area=2, ind=0, am=0, be=0, %b[42]
movad,1 area=2, ind=8, am=1, be=0, %b[59]
movad,2 area=1, ind=0, am=0, be=0, %b[37]
movad,3 area=0, ind=16, am=0, be=0, %b[58]
}
{
loop_mode
pfmul_rsubs,0,sm %b[69], %b[13], %b[84], %b[74]
pfmul_rsubs,1,sm %b[45], %b[34], %b[85], %b[78]
staad,2 %b[49], %aad3[ %aasti7 ]
pfmuls,3,sm %b[14], %b[53], %b[82]
pfmuls,4,sm %b[54], %b[20], %b[83]
staad,5 %b[26], %aad1[ %aasti5 ]
movad,0 area=1, ind=0, am=0, be=0, %b[63]
movad,1 area=0, ind=8, am=0, be=0, %b[25]
movad,2 area=1, ind=8, am=1, be=0, %b[48]
movad,3 area=0, ind=24, am=0, be=0, %b[64]
}
{
loop_mode
pfmul_adds,0,sm %b[67], %b[53], %b[86], %b[34]
pfmul_adds,1,sm %b[45], %b[22], %b[87], %b[69]
staad,2 %b[10], %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
pfmuls,3,sm %b[12], %b[9], %b[84]
pfmuls,4,sm %b[54], %b[32], %b[85]
staad,5 %b[7], %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=0, am=0, be=0, %b[13]
movad,1 area=0, ind=24, am=0, be=0, %b[49]
movad,2 area=0, ind=0, am=0, be=0, %b[26]
movad,3 area=0, ind=8, am=1, be=0, %b[14]
}
{
loop_mode
pfsub_rsubs,0,sm %b[23], %b[80], %b[55], %b[45]
pfsub_adds,1,sm %b[23], %b[80], %b[55], %b[22]
staad,2 %b[6], %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
incr,2 %aaincr3
pfsubs,4,sm %b[2], %b[38], %b[53]
staad,5 %b[3], %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
incr,5 %aaincr3
movad,0 area=1, ind=8, am=1, be=0, %b[10]
movad,1 area=0, ind=16, am=1, be=0, %b[7]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfadd_adds,0,sm %b[35], %b[71], %b[56], %b[6]
pfadd_rsubs,1,sm %b[35], %b[71], %b[56], %b[3]
staad,2 %b[79], %aad4[ %aasti8 ]
incr,2 %aaincr3
pfadds,4,sm %b[38], %b[2], %b[54]
staad,5 %b[75], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Код на ассемблере получился такой же, как в эталонном варианте, то есть компилятор самостоятельно векторизовал эталонный вариант в 2 раза.
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
3. stage_radix4_vector2_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector2_unroll2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE)
{
uint64_t *x_real_in = (uint64_t*)&data_in[0].real;
uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag;
uint64_t *y_real_in = (uint64_t*)&data_in[1].real;
uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag;
uint64_t *z_real_in = (uint64_t*)&data_in[2].real;
uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag;
uint64_t *w_real_in = (uint64_t*)&data_in[3].real;
uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag;
uint64_t *c_real_in = (uint64_t*)&coefC[0].real;
uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag;
uint64_t *d_real_in = (uint64_t*)&coefD[0].real;
uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag;
uint64_t *e_real_in = (uint64_t*)&coefE[0].real;
uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag;
uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real;
uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag;
uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real;
uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag;
uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real;
uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag;
uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real;
uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(2)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
uint64_t x_real = x_real_in[8*i];
uint64_t x_imag = x_imag_in[8*i];
uint64_t y_real = y_real_in[8*i];
uint64_t y_imag = y_imag_in[8*i];
uint64_t z_real = z_real_in[8*i];
uint64_t z_imag = z_imag_in[8*i];
uint64_t w_real = w_real_in[8*i];
uint64_t w_imag = w_imag_in[8*i];
uint64_t c_real = c_real_in[2*i];
uint64_t c_imag = c_imag_in[2*i];
uint64_t d_real = d_real_in[2*i];
uint64_t d_imag = d_imag_in[2*i];
uint64_t e_real = e_real_in[2*i];
uint64_t e_imag = e_imag_in[2*i];
uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag));
uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real));
uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag));
uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real));
uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag));
uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real));
uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real);
uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag);
uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real);
uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag);
uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real);
uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag);
uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real);
// uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag);
// uint64_t sub13i_real = -sub13_imag;
uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag);
uint64_t sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L4871:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=16
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=16, disp=96
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=3, abs=24, disp=0
}
.L4115:
{
loop_mode
pfadd_adds,0,sm %b[56], %b[48], %b[97], %b[91]
pfadd_rsubs,1,sm %b[56], %b[48], %b[97], %b[88]
pfsubs,2,sm %b[83], %b[73], %b[98]
pfmul_adds,3,sm %b[35], %b[49], %b[91], %b[0]
pfmuls,4,sm %b[88], %b[4], %b[1]
pfmuls,5,sm %b[44], %b[72], %b[95]
}
{
loop_mode
pfmul_rsubs,0,sm %b[35], %b[29], %g18, %b[11]
pfmul_adds,1,sm %b[11], %b[63], %b[96], %b[72]
pfadds,2,sm %b[73], %b[83], %b[83]
pfmul_adds,3,sm %b[30], %b[72], %b[94], %b[29]
pfmuls,4,sm %b[36], %b[47], %g18
pfadds,5,sm %g16, %g17, %b[73]
}
{
loop_mode
pfmul_rsubs,0,sm %b[68], %b[14], %b[24], %g17
pfmul_rsubs,1,sm %b[45], %b[34], %b[3], %g16
pfsubs,2,sm %g16, %g17, %b[96]
pfmul_adds,3,sm %b[18], %b[60], %b[92], %b[3]
pfmuls,4,sm %b[10], %b[61], %b[94]
pfsubs,5,sm %b[5], %b[31], %b[92]
}
{
loop_mode
pfmul_adds,0,sm %b[45], %b[6], %b[71], %b[71]
pfmul_adds,1,sm %b[68], %b[41], %b[85], %b[81]
staad,2 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
pfmul_rsubs,3,sm %b[18], %b[67], %b[93], %b[24]
pfadds,4,sm %b[31], %b[5], %b[85]
staad,5 %b[81], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
movad,0 area=3, ind=0, am=0, be=0, %b[14]
movad,1 area=3, ind=8, am=1, be=0, %b[18]
movad,2 area=3, ind=0, am=0, be=0, %b[5]
movad,3 area=3, ind=8, am=0, be=0, %b[6]
}
{
loop_mode
pfsub_rsubs,0,sm %b[19], %b[13], %b[98], %b[84]
pfsub_adds,1,sm %b[19], %b[13], %b[98], %b[79]
staad,2 %b[84], %aad4[ %aasti8 + _f32s,_lts0 0x18 ]
pfmul_rsubs,3,sm %b[30], %b[64], %b[95], %b[35]
pfsubs,4,sm %b[37], %b[26], %b[93]
staad,5 %b[79], %aad2[ %aasti6 + _f32s,_lts0 0x18 ]
movad,0 area=2, ind=24, am=0, be=0, %b[86]
movad,1 area=1, ind=16, am=0, be=0, %b[30]
movad,2 area=3, ind=16, am=0, be=0, %b[31]
movad,3 area=3, ind=24, am=1, be=0, %b[34]
}
{
loop_mode
pfadd_adds,0,sm %b[25], %b[2], %b[83], %b[82]
pfadd_rsubs,1,sm %b[25], %b[2], %b[83], %b[77]
staad,2 %b[82], %aad3[ %aasti7 + _f32s,_lts0 0x18 ]
pfmuls,3,sm %g19, %b[12], %b[83]
pfadds,4,sm %b[37], %b[26], %b[95]
staad,5 %b[77], %aad1[ %aasti5 + _f32s,_lts0 0x18 ]
movad,0 area=2, ind=16, am=0, be=0, %b[41]
movad,1 area=2, ind=0, am=0, be=0, %b[26]
movad,2 area=2, ind=24, am=0, be=0, %b[37]
movad,3 area=0, ind=8, am=0, be=0, %g19
}
{
loop_mode
pfsub_rsubs,0,sm %b[25], %b[2], %b[96], %b[80]
pfsub_adds,1,sm %b[25], %b[2], %b[96], %b[75]
staad,2 %b[80], %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
pfmul_rsubs,3,sm %b[9], %b[40], %b[94], %b[44]
pfmuls,4,sm %b[10], %b[40], %b[94]
staad,5 %b[75], %aad2[ %aasti6 + _f32s,_lts0 0x10 ]
movad,0 area=2, ind=8, am=1, be=0, %b[40]
movad,1 area=1, ind=24, am=0, be=0, %b[2]
movad,2 area=2, ind=0, am=0, be=0, %b[25]
movad,3 area=2, ind=16, am=0, be=0, %b[10]
}
{
loop_mode
pfadd_adds,0,sm %b[19], %b[13], %b[73], %b[78]
pfadd_rsubs,1,sm %b[19], %b[13], %b[73], %b[73]
staad,2 %b[78], %aad3[ %aasti7 ]
pfmuls,4,sm %b[36], %b[27], %b[89]
staad,5 %b[89], %aad1[ %aasti5 ]
movad,0 area=1, ind=8, am=1, be=0, %b[19]
movad,1 area=1, ind=0, am=0, be=0, %b[13]
movad,2 area=2, ind=8, am=1, be=0, %b[45]
movad,3 area=1, ind=0, am=0, be=0, %b[36]
}
{
loop_mode
pfsub_rsubs,0,sm %b[54], %b[46], %b[92], %b[76]
pfsub_adds,1,sm %b[54], %b[46], %b[92], %b[87]
staad,2 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
pfmuls,4,sm %b[42], %b[62], %b[92]
staad,5 %b[87], %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=8, am=0, be=0, %b[49]
movad,1 area=0, ind=0, am=0, be=0, %b[48]
movad,2 area=1, ind=8, am=0, be=0, %b[57]
movad,3 area=1, ind=24, am=0, be=0, %b[56]
}
{
loop_mode
pfadd_adds,0,sm %b[55], %b[72], %b[85], %b[74]
pfadd_rsubs,1,sm %b[55], %b[72], %b[85], %b[85]
staad,2 %b[90], %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
incr,2 %aaincr3
pfmuls,3,sm %b[86], %b[30], %b[67]
pfmuls,4,sm %b[20], %b[65], %b[90]
staad,5 %b[74], %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
incr,5 %aaincr3
movad,0 area=0, ind=24, am=1, be=0, %b[68]
movad,1 area=0, ind=16, am=0, be=0, %b[60]
movad,2 area=1, ind=16, am=1, be=0, %b[63]
movad,3 area=0, ind=0, am=1, be=0, %b[64]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfsub_rsubs,0,sm %b[55], %b[72], %b[93], %b[88]
pfsub_adds,1,sm %b[55], %b[72], %b[93], %b[72]
staad,2 %b[91], %aad4[ %aasti8 ]
incr,2 %aaincr3
pfmuls,3,sm %g19, %b[37], %b[20]
pfmuls,4,sm %b[20], %b[58], %b[91]
staad,5 %b[88], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
4. stage_radix4_vector2_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector2_unroll3(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE)
{
uint64_t *x_real_in = (uint64_t*)&data_in[0].real;
uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag;
uint64_t *y_real_in = (uint64_t*)&data_in[1].real;
uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag;
uint64_t *z_real_in = (uint64_t*)&data_in[2].real;
uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag;
uint64_t *w_real_in = (uint64_t*)&data_in[3].real;
uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag;
uint64_t *c_real_in = (uint64_t*)&coefC[0].real;
uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag;
uint64_t *d_real_in = (uint64_t*)&coefD[0].real;
uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag;
uint64_t *e_real_in = (uint64_t*)&coefE[0].real;
uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag;
uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real;
uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag;
uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real;
uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag;
uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real;
uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag;
uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real;
uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(3)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
uint64_t x_real = x_real_in[8*i];
uint64_t x_imag = x_imag_in[8*i];
uint64_t y_real = y_real_in[8*i];
uint64_t y_imag = y_imag_in[8*i];
uint64_t z_real = z_real_in[8*i];
uint64_t z_imag = z_imag_in[8*i];
uint64_t w_real = w_real_in[8*i];
uint64_t w_imag = w_imag_in[8*i];
uint64_t c_real = c_real_in[2*i];
uint64_t c_imag = c_imag_in[2*i];
uint64_t d_real = d_real_in[2*i];
uint64_t d_imag = d_imag_in[2*i];
uint64_t e_real = e_real_in[2*i];
uint64_t e_imag = e_imag_in[2*i];
uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag));
uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real));
uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag));
uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real));
uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag));
uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real));
uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real);
uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag);
uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real);
uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag);
uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real);
uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag);
uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real);
// uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag);
// uint64_t sub13i_real = -sub13_imag;
uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag);
uint64_t sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L6432:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=2, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=16, disp=32
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=3, abs=24, disp=32
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=32
}
.L4979:
{
loop_mode
pfsub_rsubs,0,sm %b[40], %g19, %g20, %g21
pfadd_adds,1,sm %b[77], %g16, %g17, %g18
pfsubs,2,sm %g22, %g23, %g24
pfmuls,3,sm %b[6], %b[32], %g27
pfmuls,4,sm %b[56], %b[65], %g26
pfmuls,5,sm %b[72], %b[48], %g25
}
{
loop_mode
pfadd_adds,0,sm %b[68], %g28, %g29, %g31
pfadd_rsubs,1,sm %b[68], %g28, %g29, %g30
pfadds,2,sm %g23, %g22, %r0
pfmuls,3,sm %b[55], %b[61], %r3
pfmuls,4,sm %b[56], %b[102], %r2
pfmuls,5,sm %b[52], %b[99], %r1
}
{
loop_mode
pfsub_rsubs,0,sm %b[68], %g28, %r4, %r7
pfsub_adds,1,sm %b[68], %g28, %r4, %r5
pfmuls,2,sm %b[20], %b[78], %r8
pfmuls,3,sm %b[18], %b[90], %r11
pfmuls,4,sm %b[51], %b[95], %r10
pfmuls,5,sm %b[41], %b[83], %r9
}
{
loop_mode
pfadd_adds,0,sm %b[40], %g19, %r12, %r14
pfadd_rsubs,1,sm %b[40], %g19, %r12, %r13
pfmuls,2,sm %b[35], %b[112], %r18
pfmuls,3,sm %b[55], %b[116], %r20
pfmuls,4,sm %b[35], %b[113], %r19
pfadds,5,sm %r15, %r16, %r17
}
{
loop_mode
pfsub_rsubs,0,sm %b[77], %g16, %g24, %g24
pfsub_adds,1,sm %b[77], %g16, %g24, %r22
staad,2 %b[119], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
pfmuls,3,sm %b[70], %b[104], %b[0]
pfsubs,4,sm %r23, %r24, %r25
staad,5 %r21, %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
}
{
loop_mode
pfadd_adds,0,sm %b[14], %b[103], %r0, %b[118]
pfadd_rsubs,1,sm %b[14], %b[103], %r0, %r27
staad,2 %b[118], %aad4[ %aasti8 + _f32s,_lts0 0x18 ]
pfmul_adds,3,sm %b[23], %b[107], %r1, %b[20]
pfadds,4,sm %r24, %r23, %r0
staad,5 %r26, %aad2[ %aasti6 + _f32s,_lts0 0x18 ]
movad,0 area=5, ind=0, am=0, be=0, %b[1]
movad,1 area=5, ind=8, am=1, be=0, %b[14]
}
{
loop_mode
pfmul_adds,0,sm %b[69], %b[106], %g25, %r24
pfmul_rsubs,1,sm %b[7], %b[92], %r8, %g16
staad,2 %r29, %aad3[ %aasti7 + _f32s,_lts0 0x18 ]
pfmul_adds,3,sm %b[45], %b[65], %r2, %r23
pfsubs,4,sm %r15, %r16, %r1
staad,5 %r28, %aad1[ %aasti5 + _f32s,_lts0 0x18 ]
movad,0 area=4, ind=0, am=0, be=0, %b[40]
movad,1 area=4, ind=16, am=0, be=0, %b[41]
movad,2 area=5, ind=0, am=0, be=0, %b[7]
movad,3 area=5, ind=8, am=1, be=0, %b[35]
}
{
loop_mode
pfmul_rsubs,0,sm %b[45], %b[102], %g26, %r16
pfmul_rsubs,1,sm %b[19], %b[113], %r18, %g19
staad,2 %r31, %aad3[ %aasti7 + _f32s,_lts0 0x28 ]
pfmul_rsubs,3,sm %b[23], %b[99], %r34, %b[55]
pfsubs,4,sm %r32, %r33, %g25
staad,5 %r30, %aad1[ %aasti5 + _f32s,_lts0 0x28 ]
movad,0 area=4, ind=8, am=1, be=0, %b[51]
movad,1 area=4, ind=24, am=0, be=0, %b[52]
movad,2 area=4, ind=0, am=0, be=0, %b[23]
movad,3 area=4, ind=8, am=1, be=0, %b[45]
}
{
loop_mode
pfmul_adds,0,sm %b[29], %b[89], %r10, %g23
pfmul_adds,1,sm %b[13], %b[86], %r9, %g22
staad,2 %r36, %aad2[ %aasti6 + _f32s,_lts1 0x10 ]
pfmul_rsubs,3,sm %b[11], %b[81], %r37, %r33
pfadds,4,sm %r32, %r33, %g17
staad,5 %r35, %aad2[ %aasti6 + _f32s,_lts0 0x20 ]
movad,0 area=3, ind=0, am=0, be=0, %b[56]
movad,1 area=3, ind=16, am=0, be=0, %b[65]
movad,2 area=3, ind=0, am=0, be=0, %b[13]
movad,3 area=3, ind=8, am=0, be=0, %b[29]
}
{
loop_mode
pfmul_rsubs,0,sm %b[69], %b[48], %b[2], %r15
pfmul_adds,1,sm %b[19], %b[112], %r19, %g28
staad,2 %r39, %aad1[ %aasti5 ]
pfmul_rsubs,3,sm %b[27], %b[93], %r41, %r32
pfsubs,4,sm %r40, %b[80], %g20
staad,5 %r38, %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
movad,0 area=3, ind=8, am=1, be=0, %b[2]
movad,1 area=3, ind=24, am=0, be=0, %b[68]
movad,2 area=3, ind=16, am=0, be=0, %b[19]
movad,3 area=3, ind=24, am=1, be=0, %b[48]
}
{
loop_mode
pfsub_rsubs,0,sm %b[28], %b[57], %r25, %b[117]
pfsub_adds,1,sm %b[28], %b[57], %r25, %r21
staad,2 %g21, %aad3[ %aasti7 ]
pfmul_adds,3,sm %b[60], %b[117], %g27, %b[78]
pfmuls,4,sm %b[6], %b[117], %g18
staad,5 %g18, %aad4[ %aasti8 + _f32s,_lts0 0x20 ]
movad,0 area=2, ind=0, am=0, be=0, %b[69]
movad,1 area=2, ind=8, am=0, be=0, %b[6]
movad,2 area=2, ind=8, am=0, be=0, %b[72]
movad,3 area=2, ind=16, am=0, be=0, %b[77]
}
{
loop_mode
pfadd_adds,0,sm %b[98], %b[22], %r0, %b[116]
pfadd_rsubs,1,sm %b[98], %b[22], %r0, %r26
staad,2 %g31, %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[44], %b[116], %r3, %r40
pfadds,4,sm %b[80], %r40, %g29
staad,5 %g30, %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
movad,0 area=2, ind=16, am=0, be=0, %b[89]
movad,1 area=2, ind=24, am=1, be=0, %b[83]
movad,2 area=2, ind=0, am=1, be=0, %b[86]
movad,3 area=2, ind=24, am=0, be=0, %b[80]
}
{
loop_mode
pfsub_rsubs,0,sm %b[98], %b[22], %r1, %r29
pfsub_adds,1,sm %b[98], %b[22], %r1, %r28
staad,2 %r7, %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[5], %b[76], %r11, %b[99]
pfsubs,4,sm %r42, %r43, %r4
staad,5 %r5, %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
movad,0 area=1, ind=8, am=0, be=0, %b[92]
movad,1 area=1, ind=0, am=0, be=0, %b[22]
movad,2 area=1, ind=16, am=0, be=0, %b[98]
movad,3 area=1, ind=0, am=0, be=0, %b[95]
}
{
loop_mode
pfsub_rsubs,0,sm %b[12], %b[101], %g25, %r31
pfsub_adds,1,sm %b[12], %b[101], %g25, %r30
staad,2 %r14, %aad4[ %aasti8 ]
pfmul_rsubs,3,sm %b[44], %b[61], %r20, %r43
pfadds,4,sm %r42, %r43, %r12
staad,5 %r13, %aad2[ %aasti6 ]
movad,0 area=1, ind=16, am=0, be=0, %b[44]
movad,1 area=1, ind=24, am=1, be=0, %b[102]
movad,2 area=1, ind=24, am=0, be=0, %b[61]
movad,3 area=1, ind=8, am=1, be=0, %b[103]
}
{
loop_mode
pfadd_rsubs,0,sm %b[28], %b[57], %r17, %r36
pfadd_rsubs,1,sm %b[75], %g16, %g17, %r35
staad,2 %g24, %aad3[ %aasti7 + _f32s,_lts0 0x20 ]
incr,2 %aaincr3
pfmul_rsubs,3,sm %b[60], %b[32], %g18, %r42
pfmuls,4,sm %b[37], %b[82], %r37
staad,5 %r22, %aad1[ %aasti5 + _f32s,_lts0 0x20 ]
incr,5 %aaincr3
movad,0 area=0, ind=8, am=0, be=0, %b[60]
movad,1 area=0, ind=0, am=0, be=0, %b[32]
movad,2 area=0, ind=0, am=0, be=0, %b[107]
movad,3 area=0, ind=8, am=0, be=0, %b[106]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfsub_adds,0,sm %b[38], %g19, %g20, %r39
pfadd_adds,1,sm %b[28], %b[57], %r17, %r38
staad,2 %b[118], %aad4[ %aasti8 + _f32s,_lts0 0x28 ]
incr,2 %aaincr3
pfmuls,3,sm %b[47], %b[85], %r41
pfmuls,4,sm %b[50], %b[105], %r34
staad,5 %r27, %aad2[ %aasti6 + _f32s,_lts0 0x28 ]
incr,5 %aaincr3
movad,0 area=0, ind=24, am=1, be=0, %b[113]
movad,1 area=0, ind=16, am=0, be=0, %b[28]
movad,2 area=0, ind=24, am=1, be=0, %b[112]
movad,3 area=0, ind=16, am=0, be=0, %b[57]
}
Теоретическая скорость: 24 комплексных числа за 16 тактов (24/16) = 12 Байт/такт
Двойная теоретическая скорость: 24 Байт/такт
Замеры скорости
Итоги по stage_radix4_vector2
stage_radix4_vector4
Один проход по stage_radix4_vector4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_vector4 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_vector4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix4_vector4_etalon(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE)
{
myComplex4 *x_in = &data_in[0];
myComplex4 *y_in = &data_in[1];
myComplex4 *z_in = &data_in[2];
myComplex4 *w_in = &data_in[3];
myComplex4 *c_in = coefC;
myComplex4 *d_in = coefD;
myComplex4 *e_in = coefE;
myComplex4 *out_0 = &data_out[0*data_count/4];
myComplex4 *out_1 = &data_out[1*data_count/4];
myComplex4 *out_2 = &data_out[2*data_count/4];
myComplex4 *out_3 = &data_out[3*data_count/4];
#pragma ivdep
#pragma unroll(1)
// #pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
myComplex4 x = x_in[4*i];
myComplex4 y = y_in[4*i];
myComplex4 z = z_in[4*i];
myComplex4 w = w_in[4*i];
myComplex4 c = c_in[i];
myComplex4 d = d_in[i];
myComplex4 e = e_in[i];
myComplex4 cy = complex4_mul(c, y);
myComplex4 dz = complex4_mul(d, z);
myComplex4 ew = complex4_mul(e, w);
myComplex4 add02 = complex4_add( x, dz);
myComplex4 sub02 = complex4_sub( x, dz);
myComplex4 add13 = complex4_add(cy, ew);
myComplex4 sub13 = complex4_sub(cy, ew);
myComplex4 sub13i = complex4_mul_i(sub13);
out_0[i] = complex4_add(add02, add13);
out_1[i] = complex4_sub(sub02, sub13i);
out_2[i] = complex4_sub(add02, add13);
out_3[i] = complex4_add(sub02, sub13i);
}
}
Основной цикл на ассемблере
.L4263:
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=16
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=16, disp=96
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=3, abs=24, disp=0
}
.L3418:
{
loop_mode
pfsub_rsubs,0,sm %b[68], %b[38], %b[93], %b[98]
pfsub_adds,1,sm %b[68], %b[38], %b[93], %b[93]
pfsubs,2,sm %b[87], %b[77], %b[102]
pfmul_adds,3,sm %b[19], %b[59], %b[98], %b[0]
pfmuls,4,sm %b[47], %b[53], %b[1]
pfmuls,5,sm %b[26], %b[11], %b[100]
}
{
loop_mode
pfmul_adds,0,sm %b[22], %b[32], %b[91], %b[26]
pfmul_adds,1,sm %b[18], %b[55], %b[97], %b[32]
pfadds,2,sm %b[87], %b[77], %b[91]
pfmul_adds,3,sm %b[14], %b[45], %b[99], %b[38]
pfmuls,4,sm %b[26], %b[15], %b[87]
pfadds,5,sm %g16, %g17, %b[77]
}
{
loop_mode
pfmul_rsubs,0,sm %b[22], %b[72], %b[6], %g17
pfmul_rsubs,1,sm %b[18], %b[76], %b[3], %g16
pfsubs,2,sm %g16, %g17, %b[99]
pfmul_adds,3,sm %b[10], %b[23], %b[101], %b[23]
pfadds,4,sm %b[34], %b[28], %b[76]
pfsubs,5,sm %b[25], %b[40], %b[97]
movad,0 area=3, ind=8, am=1, be=0, %b[6]
movad,1 area=3, ind=0, am=0, be=0, %b[18]
movad,2 area=3, ind=0, am=0, be=0, %b[3]
movad,3 area=3, ind=16, am=0, be=0, %b[22]
}
{
loop_mode
pfmul_rsubs,0,sm %b[10], %b[71], %b[56], %b[75]
pfmul_rsubs,1,sm %b[14], %b[75], %b[31], %b[85]
staad,2 %b[90], %aad3[ %aasti7 + _f32s,_lts0 0x18 ]
pfmul_rsubs,3,sm %b[19], %b[51], %b[95], %b[31]
pfadds,4,sm %b[40], %b[25], %b[90]
staad,5 %b[85], %aad1[ %aasti5 + _f32s,_lts0 0x18 ]
movad,0 area=2, ind=0, am=0, be=0, %b[14]
movad,1 area=2, ind=8, am=0, be=0, %b[10]
movad,2 area=2, ind=24, am=0, be=0, %b[19]
movad,3 area=0, ind=8, am=0, be=0, %b[25]
}
{
loop_mode
pfsub_rsubs,0,sm %b[39], %b[2], %b[102], %b[88]
pfsub_adds,1,sm %b[39], %b[2], %b[102], %b[83]
staad,2 %b[89], %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
pfmul_adds,3,sm %b[7], %b[15], %b[100], %b[40]
pfmuls,4,sm %b[83], %b[70], %b[89]
staad,5 %b[88], %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
movad,0 area=2, ind=16, am=0, be=0, %b[45]
movad,1 area=2, ind=24, am=1, be=0, %b[44]
movad,2 area=3, ind=8, am=0, be=0, %b[15]
movad,3 area=1, ind=24, am=0, be=0, %b[41]
}
{
loop_mode
pfadd_adds,0,sm %b[52], %b[33], %b[91], %b[87]
pfadd_rsubs,1,sm %b[52], %b[33], %b[91], %b[86]
staad,2 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
pfmul_rsubs,3,sm %b[7], %b[11], %b[87], %b[34]
pfsubs,4,sm %b[28], %b[34], %b[91]
staad,5 %b[81], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
movad,0 area=1, ind=16, am=0, be=0, %b[11]
movad,1 area=1, ind=0, am=0, be=0, %b[7]
movad,2 area=2, ind=16, am=0, be=0, %b[28]
movad,3 area=0, ind=0, am=1, be=0, %b[81]
}
{
loop_mode
pfsub_rsubs,0,sm %b[65], %b[42], %b[99], %b[84]
pfsub_adds,1,sm %b[65], %b[42], %b[99], %b[79]
staad,2 %b[84], %aad4[ %aasti8 ]
pfmuls,4,sm %b[47], %b[74], %b[95]
staad,5 %b[79], %aad2[ %aasti6 ]
movad,0 area=1, ind=24, am=0, be=0, %b[55]
movad,1 area=1, ind=8, am=1, be=0, %b[47]
movad,2 area=3, ind=24, am=1, be=0, %b[56]
movad,3 area=1, ind=16, am=0, be=0, %b[51]
}
{
loop_mode
pfadd_adds,0,sm %b[66], %b[36], %b[77], %b[82]
pfadd_rsubs,1,sm %b[66], %b[36], %b[77], %b[77]
staad,2 %b[96], %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
pfmuls,4,sm %b[58], %b[49], %b[96]
staad,5 %b[82], %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=0, am=0, be=0, %b[60]
movad,1 area=0, ind=16, am=0, be=0, %b[59]
movad,2 area=2, ind=8, am=1, be=0, %b[67]
movad,3 area=2, ind=0, am=0, be=0, %b[68]
}
{
loop_mode
pfsub_rsubs,0,sm %b[52], %b[33], %b[97], %b[94]
pfsub_adds,1,sm %b[52], %b[33], %b[97], %b[80]
staad,2 %b[94], %aad2[ %aasti6 + _f32s,_lts1 0x10 ]
pfmuls,3,sm %b[25], %b[19], %b[52]
pfmuls,4,sm %b[46], %b[73], %b[97]
staad,5 %b[80], %aad2[ %aasti6 + _f32s,_lts0 0x18 ]
incr,5 %aaincr3
movad,0 area=0, ind=8, am=1, be=0, %b[46]
movad,1 area=0, ind=24, am=0, be=0, %b[33]
movad,2 area=1, ind=8, am=1, be=0, %b[71]
movad,3 area=1, ind=0, am=0, be=0, %b[72]
}
{
loop_mode
pfadd_rsubs,0,sm %b[65], %b[42], %b[76], %b[92]
pfadd_rsubs,1,sm %b[39], %b[2], %b[90], %b[78]
staad,2 %b[92], %aad4[ %aasti8 + _f32s,_lts1 0x10 ]
pfmuls,3,sm %b[44], %b[41], %b[27]
pfmuls,4,sm %b[27], %b[69], %b[99]
staad,5 %b[78], %aad4[ %aasti8 + _f32s,_lts0 0x18 ]
incr,5 %aaincr3
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfadd_adds,0,sm %b[65], %b[42], %b[76], %b[90]
pfadd_adds,1,sm %b[39], %b[2], %b[90], %b[76]
staad,2 %b[98], %aad3[ %aasti7 ]
incr,2 %aaincr3
pfmuls,3,sm %b[81], %b[28], %b[2]
pfmuls,4,sm %b[58], %b[57], %b[93]
staad,5 %b[93], %aad1[ %aasti5 ]
incr,5 %aaincr3
}
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
2. stage_radix4_vector4
Прямая векторизация базового алгоритма в 4 раза с помощью инструкций SIMD128.
Код на Си
void stage_radix4_vector4(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE)
{
__v2di *x_real_in = (__v2di*)&data_in[0].real;
__v2di *x_imag_in = (__v2di*)&data_in[0].imag;
__v2di *y_real_in = (__v2di*)&data_in[1].real;
__v2di *y_imag_in = (__v2di*)&data_in[1].imag;
__v2di *z_real_in = (__v2di*)&data_in[2].real;
__v2di *z_imag_in = (__v2di*)&data_in[2].imag;
__v2di *w_real_in = (__v2di*)&data_in[3].real;
__v2di *w_imag_in = (__v2di*)&data_in[3].imag;
__v2di *c_real_in = (__v2di*)&coefC[0].real;
__v2di *c_imag_in = (__v2di*)&coefC[0].imag;
__v2di *d_real_in = (__v2di*)&coefD[0].real;
__v2di *d_imag_in = (__v2di*)&coefD[0].imag;
__v2di *e_real_in = (__v2di*)&coefE[0].real;
__v2di *e_imag_in = (__v2di*)&coefE[0].imag;
__v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real;
__v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag;
__v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real;
__v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag;
__v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real;
__v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag;
__v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real;
__v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
__v2di x_real = x_real_in[8*i];
__v2di x_imag = x_imag_in[8*i];
__v2di y_real = y_real_in[8*i];
__v2di y_imag = y_imag_in[8*i];
__v2di z_real = z_real_in[8*i];
__v2di z_imag = z_imag_in[8*i];
__v2di w_real = w_real_in[8*i];
__v2di w_imag = w_imag_in[8*i];
__v2di c_real = c_real_in[2*i];
__v2di c_imag = c_imag_in[2*i];
__v2di d_real = d_real_in[2*i];
__v2di d_imag = d_imag_in[2*i];
__v2di e_real = e_real_in[2*i];
__v2di e_imag = e_imag_in[2*i];
__v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag));
__v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real));
__v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag));
__v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real));
__v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag));
__v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real));
__v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real);
__v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag);
__v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real);
__v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag);
__v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real);
__v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag);
__v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real);
// __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag);
// __v2di sub13i_real = -sub13_imag;
__v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag);
__v2di sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L5045:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=96
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0
}
.L4371:
{
loop_mode
qpfmul_adds,0,sm %b[6], %b[69], %b[57], %b[1]
qpfsub_rsubs,1,sm %b[30], %b[74], %b[58], %b[47]
qpfsub_adds,2,sm %b[30], %b[74], %b[58], %b[5]
qpfmuls,4,sm %b[62], %b[61], %b[55]
qpfsubs,5,sm %b[77], %b[73], %b[56]
movaqp,1 area=3, ind=0, am=0, be=0, %b[0]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[6], %b[63], %b[84], %b[69]
qpfadd_adds,1,sm %b[29], %b[81], %b[89], %b[78]
qpfadd_rsubs,2,sm %b[29], %b[81], %b[89], %b[74]
qpfmuls,4,sm %b[62], %b[67], %b[82]
qpfadds,5,sm %b[77], %b[73], %b[87]
movaqp,0 area=3, ind=16, am=1, be=0, %b[58]
movaqp,1 area=2, ind=0, am=0, be=0, %b[30]
movaqp,3 area=1, ind=0, am=0, be=0, %b[57]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[36], %b[17], %b[83], %b[73]
qpfmul_rsubs,1,sm %b[14], %b[70], %b[86], %b[77]
staaqp,2 %b[46], %aad3[ %aasti7 ]
qpfmuls,3,sm %b[18], %b[50], %b[81]
qpfmuls,4,sm %b[42], %b[35], %b[84]
staaqp,5 %b[41], %aad1[ %aasti5 ]
movaqp,0 area=1, ind=0, am=0, be=0, %b[62]
movaqp,1 area=1, ind=16, am=1, be=0, %b[29]
movaqp,2 area=2, ind=0, am=0, be=0, %b[6]
movaqp,3 area=1, ind=16, am=1, be=0, %b[63]
}
{
loop_mode
qpfmul_adds,0,sm %b[34], %b[50], %b[85], %b[41]
qpfmul_adds,1,sm %b[14], %b[37], %b[88], %b[70]
staaqp,2 %b[53], %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
qpfmuls,3,sm %b[16], %b[13], %b[83]
qpfmuls,4,sm %b[42], %b[68], %b[86]
staaqp,5 %b[11], %aad2[ %aasti6 + _f32s,_lts0 0x10 ]
movaqp,0 area=0, ind=16, am=1, be=0, %b[18]
movaqp,1 area=0, ind=0, am=0, be=0, %b[17]
movaqp,2 area=2, ind=16, am=1, be=0, %b[36]
movaqp,3 area=0, ind=16, am=0, be=0, %b[46]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[27], %b[79], %b[52], %b[42]
qpfsub_adds,1,sm %b[27], %b[79], %b[52], %b[37]
staaqp,2 %b[49], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
incr,2 %aaincr3
qpfsubs,4,sm %b[3], %b[45], %b[50]
staaqp,5 %b[7], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
movaqp,1 area=2, ind=16, am=1, be=0, %b[14]
movaqp,3 area=0, ind=0, am=1, be=0, %b[11]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfadd_adds,0,sm %b[28], %b[72], %b[54], %b[49]
qpfadd_rsubs,1,sm %b[28], %b[72], %b[54], %b[7]
staaqp,2 %b[80], %aad4[ %aasti8 ]
incr,2 %aaincr3
qpfadds,4,sm %b[45], %b[3], %b[52]
staaqp,5 %b[76], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Теоретическая скорость: 16 комплексных чисел за 6 тактов (16/6) = 21.33 Байт/такт
Двойная теоретическая скорость: 42.67 Байт/такт
Замеры скорости
3. stage_radix4_vector4_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector4_unroll2(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE)
{
__v2di *x_real_in = (__v2di*)&data_in[0].real;
__v2di *x_imag_in = (__v2di*)&data_in[0].imag;
__v2di *y_real_in = (__v2di*)&data_in[1].real;
__v2di *y_imag_in = (__v2di*)&data_in[1].imag;
__v2di *z_real_in = (__v2di*)&data_in[2].real;
__v2di *z_imag_in = (__v2di*)&data_in[2].imag;
__v2di *w_real_in = (__v2di*)&data_in[3].real;
__v2di *w_imag_in = (__v2di*)&data_in[3].imag;
__v2di *c_real_in = (__v2di*)&coefC[0].real;
__v2di *c_imag_in = (__v2di*)&coefC[0].imag;
__v2di *d_real_in = (__v2di*)&coefD[0].real;
__v2di *d_imag_in = (__v2di*)&coefD[0].imag;
__v2di *e_real_in = (__v2di*)&coefE[0].real;
__v2di *e_imag_in = (__v2di*)&coefE[0].imag;
__v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real;
__v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag;
__v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real;
__v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag;
__v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real;
__v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag;
__v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real;
__v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(2)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
__v2di x_real = x_real_in[8*i];
__v2di x_imag = x_imag_in[8*i];
__v2di y_real = y_real_in[8*i];
__v2di y_imag = y_imag_in[8*i];
__v2di z_real = z_real_in[8*i];
__v2di z_imag = z_imag_in[8*i];
__v2di w_real = w_real_in[8*i];
__v2di w_imag = w_imag_in[8*i];
__v2di c_real = c_real_in[2*i];
__v2di c_imag = c_imag_in[2*i];
__v2di d_real = d_real_in[2*i];
__v2di d_imag = d_imag_in[2*i];
__v2di e_real = e_real_in[2*i];
__v2di e_imag = e_imag_in[2*i];
__v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag));
__v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real));
__v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag));
__v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real));
__v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag));
__v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real));
__v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real);
__v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag);
__v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real);
__v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag);
__v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real);
__v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag);
__v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real);
// __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag);
// __v2di sub13i_real = -sub13_imag;
__v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag);
__v2di sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L6226:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=192
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=224
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=32
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=32
}
.L5153:
{
loop_mode
qpfadd_adds,0,sm %b[32], %b[21], %b[94], %b[65]
qpfadd_rsubs,1,sm %b[32], %b[21], %b[94], %b[88]
qpfsubs,2,sm %b[61], %b[65], %b[92]
qpfmuls,3,sm %b[15], %b[40], %b[0]
qpfmuls,4,sm %b[91], %b[48], %b[1]
qpfmuls,5,sm %b[17], %b[9], %b[90]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[22], %b[60], %g16, %b[21]
qpfmul_adds,1,sm %b[8], %b[64], %b[95], %b[22]
qpfsubs,2,sm %b[81], %b[70], %b[94]
qpfmul_adds,3,sm %b[22], %b[57], %b[83], %b[17]
qpfmuls,4,sm %b[35], %b[55], %g16
qpfsubs,5,sm %b[16], %b[10], %b[83]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[14], %b[51], %b[3], %b[8]
qpfmul_rsubs,1,sm %b[46], %b[52], %b[96], %b[14]
qpfadds,2,sm %b[81], %b[70], %b[84]
qpfmul_rsubs,3,sm %b[8], %b[38], %b[84], %b[3]
qpfmuls,4,sm %b[53], %b[50], %b[70]
qpfmuls,5,sm %b[91], %b[49], %b[81]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[13], %b[9], %b[2], %b[68]
qpfmul_rsubs,1,sm %b[56], %b[33], %b[93], %b[79]
staaqp,2 %b[82], %aad3[ %aasti7 ]
qpfmul_adds,3,sm %b[56], %b[68], %b[89], %b[2]
qpfadds,4,sm %b[16], %b[10], %b[82]
staaqp,5 %b[79], %aad1[ %aasti5 ]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[29], %b[5], %b[92], %b[80]
qpfsub_adds,1,sm %b[29], %b[5], %b[92], %b[77]
staaqp,2 %b[80], %aad2[ %aasti6 + _f32s,_lts1 0x10 ]
qpfmul_adds,3,sm %b[13], %b[42], %b[90], %b[16]
qpfsubs,4,sm %b[18], %b[4], %b[90]
staaqp,5 %b[77], %aad1[ %aasti5 + _f32s,_lts0 0x30 ]
movaqp,0 area=6, ind=0, am=0, be=0, %b[10]
movaqp,1 area=6, ind=16, am=1, be=0, %b[89]
movaqp,2 area=6, ind=0, am=0, be=0, %b[9]
movaqp,3 area=6, ind=16, am=1, be=0, %b[13]
}
{
loop_mode
qpfadd_rsubs,0,sm %b[47], %b[24], %b[87], %b[78]
qpfsub_adds,1,sm %b[30], %b[19], %b[94], %b[75]
staaqp,2 %b[75], %aad3[ %aasti7 + _f32s,_lts1 0x30 ]
qpfmuls,3,sm %g17, %b[66], %b[91]
qpfadds,4,sm %b[4], %b[18], %b[92]
staaqp,5 %b[78], %aad2[ %aasti6 + _f32s,_lts0 0x20 ]
movaqp,0 area=5, ind=0, am=0, be=0, %b[4]
movaqp,1 area=5, ind=16, am=1, be=0, %b[32]
movaqp,2 area=5, ind=0, am=0, be=0, %b[18]
movaqp,3 area=5, ind=16, am=1, be=0, %b[33]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[30], %b[19], %b[94], %b[73]
qpfadd_rsubs,1,sm %b[41], %b[23], %b[84], %b[76]
staaqp,2 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
qpfmuls,4,sm %b[34], %b[36], %b[93]
staaqp,5 %b[73], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
movaqp,0 area=4, ind=0, am=0, be=0, %b[42]
movaqp,1 area=4, ind=16, am=1, be=0, %b[51]
movaqp,2 area=3, ind=16, am=0, be=0, %b[38]
movaqp,3 area=1, ind=16, am=0, be=0, %b[46]
}
{
loop_mode
qpfadd_adds,0,sm %b[47], %b[24], %b[87], %b[74]
qpfsub_adds,1,sm %b[47], %b[24], %b[83], %b[71]
staaqp,2 %b[74], %aad3[ %aasti7 + _f32s,_lts1 0x10 ]
qpfmuls,3,sm %g17, %b[31], %b[87]
qpfmuls,4,sm %b[53], %b[67], %b[94]
staaqp,5 %b[71], %aad4[ %aasti8 + _f32s,_lts0 0x20 ]
movaqp,0 area=3, ind=0, am=0, be=0, %b[56]
movaqp,1 area=3, ind=16, am=1, be=0, %b[53]
movaqp,2 area=4, ind=0, am=0, be=0, %b[52]
movaqp,3 area=4, ind=16, am=1, be=0, %g17
}
{
loop_mode
qpfsub_rsubs,0,sm %b[47], %b[24], %b[83], %b[72]
qpfadd_adds,1,sm %b[41], %b[23], %b[84], %b[69]
staaqp,2 %b[72], %aad4[ %aasti8 ]
qpfmul_adds,3,sm %b[12], %b[48], %b[81], %b[57]
qpfmuls,4,sm %b[35], %b[58], %b[81]
staaqp,5 %b[69], %aad2[ %aasti6 ]
movaqp,0 area=2, ind=0, am=0, be=0, %b[35]
movaqp,1 area=2, ind=16, am=1, be=0, %b[24]
movaqp,2 area=1, ind=0, am=1, be=0, %b[47]
movaqp,3 area=0, ind=0, am=0, be=0, %b[48]
}
{
loop_mode
qpfadd_adds,0,sm %b[29], %b[5], %b[82], %b[70]
qpfadd_rsubs,1,sm %b[29], %b[5], %b[82], %b[67]
staaqp,2 %b[85], %aad3[ %aasti7 + _f32s,_lts0 0x20 ]
incr,2 %aaincr3
qpfmul_adds,3,sm %b[44], %b[67], %b[70], %b[61]
qpfmuls,4,sm %b[34], %b[62], %b[82]
staaqp,5 %b[86], %aad1[ %aasti5 + _f32s,_lts0 0x20 ]
incr,5 %aaincr3
movaqp,0 area=1, ind=16, am=1, be=0, %b[60]
movaqp,1 area=1, ind=0, am=0, be=0, %b[34]
movaqp,2 area=3, ind=0, am=1, be=0, %b[5]
movaqp,3 area=2, ind=0, am=0, be=0, %b[29]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfsub_rsubs,0,sm %b[41], %b[23], %b[90], %b[83]
qpfsub_adds,1,sm %b[41], %b[23], %b[90], %b[84]
staaqp,2 %b[65], %aad4[ %aasti8 + _f32s,_lts0 0x30 ]
incr,2 %aaincr3
qpfadds,4,sm %b[63], %b[59], %b[85]
staaqp,5 %b[88], %aad2[ %aasti6 + _f32s,_lts0 0x30 ]
incr,5 %aaincr3
movaqp,0 area=0, ind=16, am=1, be=0, %b[41]
movaqp,1 area=0, ind=0, am=0, be=0, %b[23]
movaqp,2 area=2, ind=16, am=1, be=0, %b[64]
movaqp,3 area=0, ind=16, am=1, be=0, %b[65]
}
Теоретическая скорость: 32 комплексных числа за 11 тактов (32/11) = 23.27 Байт/такт
Двойная теоретическая скорость: 46.55 Байт/такт
Замеры скорости
4. stage_radix4_vector4_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector4_unroll3(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE)
{
__v2di *x_real_in = (__v2di*)&data_in[0].real;
__v2di *x_imag_in = (__v2di*)&data_in[0].imag;
__v2di *y_real_in = (__v2di*)&data_in[1].real;
__v2di *y_imag_in = (__v2di*)&data_in[1].imag;
__v2di *z_real_in = (__v2di*)&data_in[2].real;
__v2di *z_imag_in = (__v2di*)&data_in[2].imag;
__v2di *w_real_in = (__v2di*)&data_in[3].real;
__v2di *w_imag_in = (__v2di*)&data_in[3].imag;
__v2di *c_real_in = (__v2di*)&coefC[0].real;
__v2di *c_imag_in = (__v2di*)&coefC[0].imag;
__v2di *d_real_in = (__v2di*)&coefD[0].real;
__v2di *d_imag_in = (__v2di*)&coefD[0].imag;
__v2di *e_real_in = (__v2di*)&coefE[0].real;
__v2di *e_imag_in = (__v2di*)&coefE[0].imag;
__v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real;
__v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag;
__v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real;
__v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag;
__v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real;
__v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag;
__v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real;
__v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(3)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
__v2di x_real = x_real_in[8*i];
__v2di x_imag = x_imag_in[8*i];
__v2di y_real = y_real_in[8*i];
__v2di y_imag = y_imag_in[8*i];
__v2di z_real = z_real_in[8*i];
__v2di z_imag = z_imag_in[8*i];
__v2di w_real = w_real_in[8*i];
__v2di w_imag = w_imag_in[8*i];
__v2di c_real = c_real_in[2*i];
__v2di c_imag = c_imag_in[2*i];
__v2di d_real = d_real_in[2*i];
__v2di d_imag = d_imag_in[2*i];
__v2di e_real = e_real_in[2*i];
__v2di e_imag = e_imag_in[2*i];
__v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag));
__v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real));
__v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag));
__v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real));
__v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag));
__v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real));
__v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real);
__v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag);
__v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real);
__v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag);
__v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real);
__v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag);
__v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real);
// __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag);
// __v2di sub13i_real = -sub13_imag;
__v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag);
__v2di sub13i_imag = sub13_real;
out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real);
out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag);
out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real);
out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag);
out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real);
out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag);
out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real);
out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag);
}
}
Основной цикл на ассемблере
.L7982:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=96
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=128
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=160
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=192
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=224
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=8, disp=256
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=288
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=10, disp=320
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=352
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=32
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=24, disp=64
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=24, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=28, disp=32
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=28, disp=64
}
.L6334:
{
loop_mode
qpfsub_rsubs,0,sm %b[105], %b[23], %g16, %g18
qpfsub_adds,1,sm %b[105], %b[23], %g16, %g17
qpfsubs,2,sm %g19, %g20, %g21
qpfmuls,3,sm %b[38], %b[113], %g24
qpfmuls,4,sm %b[46], %b[81], %g23
qpfmuls,5,sm %b[38], %b[64], %g22
}
{
loop_mode
qpfadd_adds,0,sm %b[102], %b[49], %g25, %g27
qpfadd_rsubs,1,sm %b[102], %b[49], %g25, %g26
qpfadds,2,sm %g19, %g20, %g28
qpfmuls,3,sm %b[45], %b[109], %g31
qpfmuls,4,sm %b[42], %b[31], %g30
qpfmuls,5,sm %b[90], %b[94], %g29
}
{
loop_mode
qpfsub_rsubs,0,sm %b[30], %r0, %r1, %r3
qpfsub_adds,1,sm %b[30], %r0, %r1, %r2
qpfmuls,2,sm %b[90], %b[97], %r7
qpfmuls,3,sm %b[48], %b[104], %b[0]
qpfmuls,4,sm %b[82], %b[18], %r5
qpfmuls,5,sm %b[19], %b[76], %r4
}
{
loop_mode
qpfadd_adds,0,sm %b[63], %r9, %r10, %r12
qpfadd_rsubs,1,sm %b[63], %r9, %r10, %r11
qpfmuls,2,sm %b[13], %b[93], %r14
qpfmuls,3,sm %b[11], %b[85], %r16
qpfmuls,4,sm %b[82], %b[70], %r15
qpfadds,5,sm %b[118], %b[119], %r13
}
{
loop_mode
qpfsub_rsubs,0,sm %b[63], %r9, %g21, %g21
qpfsub_adds,1,sm %b[63], %r9, %g21, %r19
staaqp,2 %r18, %aad3[ %aasti7 ]
qpfmuls,3,sm %b[17], %b[73], %b[118]
qpfsubs,4,sm %b[119], %b[118], %r20
staaqp,5 %r17, %aad1[ %aasti5 ]
}
{
loop_mode
qpfadd_adds,0,sm %b[30], %r0, %g28, %g28
qpfadd_rsubs,1,sm %b[30], %r0, %g28, %r23
staaqp,2 %r22, %aad2[ %aasti6 + _f32s,_lts1 0x10 ]
qpfmul_rsubs,3,sm %b[86], %b[97], %g29, %b[19]
qpfsubs,4,sm %b[116], %r24, %r25
staaqp,5 %r21, %aad1[ %aasti5 + _f32s,_lts0 0x30 ]
movaqp,0 area=10, ind=0, am=0, be=0, %b[1]
movaqp,1 area=10, ind=16, am=1, be=0, %b[13]
}
{
loop_mode
qpfmul_adds,0,sm %b[12], %b[113], %g22, %b[49]
qpfmul_adds,1,sm %b[86], %b[94], %r7, %b[45]
staaqp,2 %r27, %aad3[ %aasti7 + _f32s,_lts1 0x30 ]
qpfmul_rsubs,3,sm %b[27], %b[35], %g30, %r24
qpfadds,4,sm %b[116], %r24, %b[113]
staaqp,5 %r26, %aad2[ %aasti6 + _f32s,_lts0 0x20 ]
movaqp,0 area=9, ind=0, am=0, be=0, %b[30]
movaqp,1 area=9, ind=16, am=1, be=0, %b[46]
movaqp,2 area=9, ind=0, am=0, be=0, %b[23]
movaqp,3 area=9, ind=16, am=1, be=0, %b[38]
}
{
loop_mode
qpfmul_adds,0,sm %b[34], %b[106], %r30, %b[117]
qpfmul_adds,1,sm %b[41], %b[114], %g23, %b[116]
staaqp,2 %b[117], %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
qpfmul_rsubs,3,sm %b[22], %b[109], %r31, %b[114]
qpfsubs,4,sm %r29, %b[37], %g22
staaqp,5 %r28, %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
movaqp,0 area=8, ind=0, am=0, be=0, %b[82]
movaqp,1 area=8, ind=16, am=1, be=0, %b[86]
movaqp,2 area=8, ind=0, am=0, be=0, %b[63]
movaqp,3 area=8, ind=16, am=1, be=0, %b[76]
}
{
loop_mode
qpfmul_rsubs,0,sm %b[7], %b[75], %r4, %g20
qpfmul_rsubs,1,sm %b[8], %b[87], %r14, %g19
staaqp,2 %r33, %aad3[ %aasti7 + _f32s,_lts1 0x10 ]
qpfmul_rsubs,3,sm %b[34], %b[110], %b[2], %b[35]
qpfmuls,4,sm %b[42], %b[35], %g23
staaqp,5 %r32, %aad4[ %aasti8 + _f32s,_lts0 0x20 ]
movaqp,0 area=7, ind=0, am=0, be=0, %b[2]
movaqp,1 area=7, ind=16, am=1, be=0, %b[7]
movaqp,2 area=7, ind=0, am=0, be=0, %b[8]
movaqp,3 area=7, ind=16, am=1, be=0, %b[34]
}
{
loop_mode
qpfmul_adds,0,sm %b[69], %b[70], %r5, %r9
qpfmul_rsubs,1,sm %b[69], %b[18], %r15, %r0
staaqp,2 %r35, %aad4[ %aasti8 ]
qpfmul_rsubs,3,sm %b[41], %b[81], %r36, %r29
qpfadds,4,sm %r29, %b[37], %g29
staaqp,5 %r34, %aad2[ %aasti6 ]
movaqp,0 area=6, ind=0, am=0, be=0, %b[37]
movaqp,1 area=6, ind=16, am=1, be=0, %b[42]
movaqp,2 area=6, ind=0, am=0, be=0, %b[18]
movaqp,3 area=6, ind=16, am=1, be=0, %b[41]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[54], %b[77], %r20, %r18
qpfsub_adds,1,sm %b[54], %b[77], %r20, %r17
staaqp,2 %g18, %aad3[ %aasti7 + _f32s,_lts0 0x20 ]
qpfmul_rsubs,3,sm %b[12], %b[64], %g24, %b[75]
qpfsubs,4,sm %r37, %r38, %g16
staaqp,5 %g17, %aad1[ %aasti5 + _f32s,_lts0 0x20 ]
movaqp,0 area=5, ind=16, am=1, be=0, %b[64]
movaqp,1 area=5, ind=0, am=0, be=0, %b[12]
movaqp,2 area=5, ind=16, am=1, be=0, %b[70]
movaqp,3 area=5, ind=0, am=0, be=0, %b[69]
}
{
loop_mode
qpfadd_rsubs,0,sm %b[60], %b[51], %r13, %r22
qpfsub_adds,1,sm %b[100], %b[47], %r25, %r21
staaqp,2 %g27, %aad4[ %aasti8 + _f32s,_lts0 0x30 ]
qpfmul_adds,3,sm %b[22], %b[55], %g31, %r38
qpfadds,4,sm %r38, %r37, %g25
staaqp,5 %g26, %aad2[ %aasti6 + _f32s,_lts0 0x30 ]
movaqp,0 area=4, ind=16, am=1, be=0, %b[55]
movaqp,1 area=4, ind=0, am=0, be=0, %b[22]
movaqp,2 area=4, ind=16, am=1, be=0, %b[87]
movaqp,3 area=4, ind=0, am=0, be=0, %b[81]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[100], %b[47], %r25, %r27
qpfadd_rsubs,1,sm %b[103], %b[21], %b[113], %r26
staaqp,2 %r3, %aad3[ %aasti7 + _f32s,_lts0 0x40 ]
qpfmul_adds,3,sm %b[27], %b[31], %g23, %r37
qpfsubs,4,sm %b[115], %r39, %r1
staaqp,5 %r2, %aad1[ %aasti5 + _f32s,_lts0 0x40 ]
movaqp,0 area=3, ind=0, am=0, be=0, %b[93]
movaqp,1 area=3, ind=16, am=1, be=0, %b[90]
movaqp,2 area=3, ind=0, am=0, be=0, %b[31]
movaqp,3 area=3, ind=16, am=1, be=0, %b[27]
}
{
loop_mode
qpfadd_adds,0,sm %b[60], %b[51], %r13, %b[115]
qpfsub_adds,1,sm %b[60], %b[51], %g22, %r28
staaqp,2 %r12, %aad4[ %aasti8 + _f32s,_lts0 0x50 ]
qpfmul_adds,3,sm %b[6], %b[91], %r16, %r39
qpfadds,4,sm %r39, %b[115], %r10
staaqp,5 %r11, %aad2[ %aasti6 + _f32s,_lts0 0x50 ]
movaqp,0 area=2, ind=0, am=0, be=0, %b[97]
movaqp,1 area=2, ind=16, am=1, be=0, %b[94]
movaqp,2 area=2, ind=0, am=0, be=0, %b[105]
movaqp,3 area=1, ind=16, am=0, be=0, %b[102]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[60], %b[51], %g22, %r33
qpfadd_adds,1,sm %b[103], %b[21], %b[113], %r32
staaqp,2 %g21, %aad3[ %aasti7 + _f32s,_lts0 0x50 ]
incr,2 %aaincr3
qpfmul_adds,3,sm %b[5], %b[74], %b[118], %b[113]
qpfmuls,4,sm %b[43], %b[53], %r31
staaqp,5 %r19, %aad1[ %aasti5 + _f32s,_lts0 0x50 ]
incr,5 %aaincr3
movaqp,0 area=1, ind=16, am=1, be=0, %b[109]
movaqp,1 area=1, ind=0, am=0, be=0, %b[60]
movaqp,2 area=2, ind=16, am=1, be=0, %b[51]
movaqp,3 area=1, ind=0, am=1, be=0, %b[106]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfadd_adds,0,sm %b[54], %b[77], %g29, %r35
qpfadd_rsubs,1,sm %b[54], %b[77], %g29, %r34
staaqp,2 %g28, %aad4[ %aasti8 + _f32s,_lts0 0x40 ]
incr,2 %aaincr3
qpfmuls,3,sm %b[44], %b[112], %r36
qpfmuls,4,sm %b[48], %b[108], %r30
staaqp,5 %r23, %aad2[ %aasti6 + _f32s,_lts0 0x40 ]
incr,5 %aaincr3
movaqp,0 area=0, ind=16, am=1, be=0, %b[54]
movaqp,1 area=0, ind=0, am=0, be=0, %b[48]
movaqp,2 area=0, ind=0, am=0, be=0, %b[77]
movaqp,3 area=0, ind=16, am=1, be=0, %b[110]
}
Теоретическая скорость: 48 комплексных чисел за 16 тактов (48/16) = 24 Байт/такт
Двойная теоретическая скорость: 48 Байт/такт
Замеры скорости
Компилятор не сумел векторизовать вариант stage_radix4_vector4_etalon в 4 раза.
До этого он успешно векторизовал вариант stage_radix4_vector2_etalon в 2 раза.
Итоги по stage_radix4_vector4
Пишем LastStage
В случае выполнения прямой векторизации в 2 раза нужно использовать lastStage_radix2.
В случае выполнения прямой векторизации в 4 раза нужно использовать lastStage_radix4 (или сначала stage_radix2_vector2 и затем lastStage_radix2).
lastStage_radix2
1. lastStage_radix2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void lastStage_radix2_etalon(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef)
{
float *x_real_in = &data_in[0].real[0];
float *x_imag_in = &data_in[0].imag[0];
float *y_real_in = &data_in[0].real[1];
float *y_imag_in = &data_in[0].imag[1];
float *c_real_in = &coef[0].real;
float *c_imag_in = &coef[0].imag;
float *out_add_real = &data_out[0].real;
float *out_add_imag = &data_out[0].imag;
float *out_sub_real = &data_out[data_count/2].real;
float *out_sub_imag = &data_out[data_count/2].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
float x_real = x_real_in[4*i];
float x_imag = x_imag_in[4*i];
float y_real = y_real_in[4*i];
float y_imag = y_imag_in[4*i];
float c_real = c_real_in[2*i];
float c_imag = c_imag_in[2*i];
float cy_real = c_real*y_real - c_imag*y_imag;
float cy_imag = c_real*y_imag + c_imag*y_real;
out_add_real[2*i] = x_real + cy_real;
out_add_imag[2*i] = x_imag + cy_imag;
out_sub_real[2*i] = x_real - cy_real;
out_sub_imag[2*i] = x_imag - cy_imag;
}
}
Основной цикл на ассемблере
.L479:
{
fapb ct=1, dcd=0, fmt=3, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=5, abs=0, disp=0
}
.L182:
{
loop_mode
fmuls,0,sm %b[67], %b[6], %b[37]
fsubs,1,sm %b[46], %b[24], %b[58]
staaw,2 %b[62], %aad1[ %aasti3 + _f32s,_lts0 0x4 ]
fmul_adds,3,sm %b[55], %b[13], %b[43], %b[14]
fadds,4,sm %b[46], %b[24], %b[57]
staaw,5 %b[61], %aad2[ %aasti4 + _f32s,_lts0 0x4 ]
movaw,0 area=0, ind=4, am=0, be=0, %b[0]
movaw,1 area=0, ind=12, am=0, be=0, %b[1]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
fmuls,0,sm %b[67], %b[7], %b[62]
fsubs,1,sm %b[35], %b[56], %b[70]
staaw,2 %b[74], %aad1[ %aasti3 ]
incr,2 %aaincr3
fmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46]
fadds,4,sm %b[35], %b[56], %b[69]
staaw,5 %b[73], %aad2[ %aasti4 ]
incr,5 %aaincr3
movaw,0 area=0, ind=0, am=0, be=0, %b[13]
movaw,1 area=0, ind=8, am=1, be=0, %b[24]
movaw,2 area=0, ind=4, am=1, be=0, %b[61]
movaw,3 area=0, ind=0, am=0, be=0, %b[43]
}
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
2. lastStage_radix2_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix2_etalon_unroll2(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef)
{
float *x_real_in = &data_in[0].real[0];
float *x_imag_in = &data_in[0].imag[0];
float *y_real_in = &data_in[0].real[1];
float *y_imag_in = &data_in[0].imag[1];
float *c_real_in = &coef[0].real;
float *c_imag_in = &coef[0].imag;
float *out_add_real = &data_out[0].real;
float *out_add_imag = &data_out[0].imag;
float *out_sub_real = &data_out[data_count/2].real;
float *out_sub_imag = &data_out[data_count/2].imag;
#pragma ivdep
#pragma unroll(2)
#pragma prefetch
for(int64_t i = 0; i < data_count/2; ++i)
{
float x_real = x_real_in[4*i];
float x_imag = x_imag_in[4*i];
float y_real = y_real_in[4*i];
float y_imag = y_imag_in[4*i];
float c_real = c_real_in[2*i];
float c_imag = c_imag_in[2*i];
float cy_real = c_real*y_real - c_imag*y_imag;
float cy_imag = c_real*y_imag + c_imag*y_real;
out_add_real[2*i] = x_real + cy_real;
out_add_imag[2*i] = x_imag + cy_imag;
out_sub_real[2*i] = x_real - cy_real;
out_sub_imag[2*i] = x_imag - cy_imag;
}
}
Основной цикл на ассемблере
.L1357:
{
fapb ct=0, dcd=0, fmt=3, mrng=12, d=0, incr=1, ind=1, asz=4, abs=0, disp=20
fapb dpl=0, dcd=0, fmt=3, mrng=20, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L525:
{
loop_mode
pfmuls,0,sm %b[56], %b[16], %b[26]
insfd,1,sm %b[52], %r8, %b[47], %b[1]
pfmuls,2,sm %b[56], %b[13], %b[31]
pshufb,3,sm %b[19], %b[9], %r0, %b[57]
insfd,4,sm %b[55], %r8, %b[37], %b[0]
pfadds,5,sm %b[42], %b[43], %b[10]
}
{
loop_mode
pfmul_rsubs,0,sm %b[5], %b[15], %b[28], %b[42]
insfd,1,sm %b[27], %r8, %b[22], %b[32]
pfmul_adds,2,sm %b[5], %b[18], %b[33], %b[37]
pshufb,4,sm %b[12], %b[23], %r0, %b[55]
staad,5 %b[57], %aad1[ %aasti3 + _f32s,_lts0 0x8 ]
movad,0 area=1, ind=0, am=1, be=0, %b[43]
movad,1 area=1, ind=8, am=0, be=0, %b[48]
movaw,3 area=0, ind=12, am=0, be=0, %b[47]
}
{
loop_mode
pfsubs,0,sm %b[8], %b[46], %b[5]
insfd,1,sm %b[19], %r8, %b[9], %b[58]
pfsubs,2,sm %b[40], %b[41], %b[15]
insfd,4,sm %b[12], %r8, %b[23], %b[56]
staad,5 %b[55], %aad2[ %aasti4 + _f32s,_lts0 0x8 ]
movaw,0 area=0, ind=8, am=0, be=0, %b[22]
movaw,1 area=0, ind=0, am=0, be=0, %b[28]
movaw,2 area=0, ind=8, am=0, be=0, %b[18]
movaw,3 area=0, ind=4, am=0, be=0, %b[27]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfadds,0,sm %b[8], %b[46], %b[19]
insfd,1,sm %b[30], %r8, %b[29], %b[9]
staad,2 %b[58], %aad1[ %aasti3 ]
incr,2 %aaincr3
pshufb,3,sm %b[50], %b[45], %r0, %b[52]
insfd,4,sm %b[24], %r8, %b[49], %b[12]
staad,5 %b[56], %aad2[ %aasti4 ]
incr,5 %aaincr3
movaw,1 area=0, ind=4, am=1, be=0, %b[23]
movaw,2 area=0, ind=0, am=1, be=0, %b[33]
movaw,3 area=0, ind=16, am=0, be=0, %b[51]
}
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Замеры скорости
3. lastStage_radix2_simd64
Модифицированный stage_radix2_simd64 для работы с векторными данными.
Код на Си
void lastStage_radix2_simd64(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef)
{
uint64_t *xy0_real_in = (uint64_t*)&data_in[0].real;
uint64_t *xy0_imag_in = (uint64_t*)&data_in[0].imag;
uint64_t *xy1_real_in = (uint64_t*)&data_in[1].real;
uint64_t *xy1_imag_in = (uint64_t*)&data_in[1].imag;
uint64_t *c0_in = (uint64_t*)&coef[0];
uint64_t *c1_in = (uint64_t*)&coef[1];
uint64_t *out_add0 = (uint64_t*)&data_out[0];
uint64_t *out_add1 = (uint64_t*)&data_out[1];
uint64_t *out_sub0 = (uint64_t*)&data_out[data_count/2 + 0];
uint64_t *out_sub1 = (uint64_t*)&data_out[data_count/2 + 1];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
uint64_t xy0_real = xy0_real_in[4*i];
uint64_t xy0_imag = xy0_imag_in[4*i];
uint64_t xy1_real = xy1_real_in[4*i];
uint64_t xy1_imag = xy1_imag_in[4*i];
uint64_t c0 = c0_in[2*i];
uint64_t c1 = c1_in[2*i];
uint64_t x_real = __builtin_e2k_pshufb(xy1_real, xy0_real, 0x0B0A090803020100);
uint64_t x_imag = __builtin_e2k_pshufb(xy1_imag, xy0_imag, 0x0B0A090803020100);
uint64_t y_real = __builtin_e2k_pshufb(xy1_real, xy0_real, 0x0F0E0D0C07060504);
uint64_t y_imag = __builtin_e2k_pshufb(xy1_imag, xy0_imag, 0x0F0E0D0C07060504);
uint64_t c_real = __builtin_e2k_pshufb(c1, c0, 0x0B0A090803020100);
uint64_t c_imag = __builtin_e2k_pshufb(c1, c0, 0x0F0E0D0C07060504);
uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag));
uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real));
uint64_t add_real = __builtin_e2k_pfadds(x_real, cy_real);
uint64_t add_imag = __builtin_e2k_pfadds(x_imag, cy_imag);
uint64_t sub_real = __builtin_e2k_pfsubs(x_real, cy_real);
uint64_t sub_imag = __builtin_e2k_pfsubs(x_imag, cy_imag);
out_add0[2*i] = __builtin_e2k_pshufb(add_imag, add_real, 0x0B0A090803020100);
out_add1[2*i] = __builtin_e2k_pshufb(add_imag, add_real, 0x0F0E0D0C07060504);
out_sub0[2*i] = __builtin_e2k_pshufb(sub_imag, sub_real, 0x0B0A090803020100);
out_sub1[2*i] = __builtin_e2k_pshufb(sub_imag, sub_real, 0x0F0E0D0C07060504);
}
}
Основной цикл на ассемблере
.L1929:
{
fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0
}
.L1613:
{
loop_mode
pfadds,0,sm %b[26], %b[50], %b[68]
pshufb,1,sm %b[58], %b[61], %r6, %b[1]
pfadds,2,sm %b[10], %b[49], %b[9]
pshufb,3,sm %b[43], %b[44], %r5, %b[0]
pshufb,4,sm %b[71], %b[57], %r6, %b[72]
pfsubs,5,sm %b[10], %b[49], %b[69]
}
{
loop_mode
pshufb,0,sm %b[53], %b[54], %r5, %b[10]
pfmul_adds,1,sm %b[34], %b[66], %b[33], %b[43]
pfmul_rsubs,2,sm %b[34], %b[23], %b[67], %b[44]
pshufb,3,sm %b[60], %b[63], %r5, %b[26]
pshufb,4,sm %b[71], %b[57], %r5, %b[73]
staad,5 %b[72], %aad1[ %aasti3 + _f32s,_lts0 0x8 ]
movad,0 area=0, ind=0, am=0, be=0, %b[50]
movad,1 area=0, ind=16, am=0, be=0, %b[49]
}
{
loop_mode
pfmuls,0,sm %b[1], %b[13], %b[23]
pshufb,1,sm %b[11], %b[70], %r6, %b[71]
pfsubs,2,sm %b[24], %b[48], %b[53]
pshufb,4,sm %b[11], %b[70], %r5, %b[72]
staad,5 %b[73], %aad1[ %aasti3 ]
incr,5 %aaincr3
movad,0 area=0, ind=8, am=1, be=0, %b[34]
movad,1 area=0, ind=24, am=0, be=0, %b[33]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pshufb,0,sm %b[51], %b[52], %r6, %b[11]
pfmuls,1,sm %b[7], %b[62], %b[63]
staad,2 %b[71], %aad2[ %aasti4 + _f32s,_lts0 0x8 ]
pshufb,4,sm %b[41], %b[42], %r6, %b[60]
staad,5 %b[72], %aad2[ %aasti4 ]
incr,5 %aaincr3
movad,2 area=0, ind=0, am=0, be=0, %b[57]
movad,3 area=0, ind=8, am=1, be=0, %b[54]
}
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Замеры скорости
4. lastStage_radix2_simd128_noConj
Модифицированный stage_radix2_simd128_noConj для работы с векторными данными.
Код на Си
void lastStage_radix2_simd128_noConj(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef)
{
__v2di *xy0_in = (__v2di*)&data_in[0];
__v2di *xy1_in = (__v2di*)&data_in[1];
__v2di *c_in = (__v2di*)coef;
__v2di *out_add = (__v2di*)&data_out[0];
__v2di *out_sub = (__v2di*)&data_out[data_count/2];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
__v2di xy0 = xy0_in[2*i];
__v2di xy1 = xy1_in[2*i];
__v2di c = c_in[i];
__v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0B0A090803020100, 0x0B0A090803020100});
__v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C07060504, 0x0F0E0D0C07060504});
__v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di cy_real = __builtin_e2k_qpfmuls( c, y);
__v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y);
__v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real);
__v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag);
__v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
out_add[i] = __builtin_e2k_qpfadds(x, cy);
out_sub[i] = __builtin_e2k_qpfsubs(x, cy);
}
}
Основной цикл на ассемблере
.L2215:
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0
}
.L1954:
{
loop_mode
qpfmul_hsubs,0,sm %b[25], %b[28], %r9, %b[16]
qpfmul_hadds,1,sm %b[27], %b[28], %r9, %b[1]
qpfsubs,2,sm %b[14], %b[42], %b[37]
qpshufb,3,sm %b[35], %b[36], %r6, %b[0]
qppermb,4,sm %b[11], %b[26], %r7, %b[38]
staaqp,5 %b[43], %aad1[ %aasti3 ]
incr,5 %aaincr0
movaqp,0 area=0, ind=0, am=0, be=0, %b[30]
movaqp,1 area=0, ind=16, am=1, be=0, %b[29]
movaqp,3 area=0, ind=0, am=1, be=0, %b[19]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpshufb,0,sm %b[33], %b[34], %r0, %b[26]
qpshufb,1,sm %b[23], %b[23], %r5, %b[25]
qpfadds,2,sm %b[14], %b[42], %b[11]
staaqp,5 %b[17], %aad2[ %aasti4 ]
incr,5 %aaincr0
}
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Итоги по lastStage_radix2
При реализации Vector-2 FFT будем использовать lastStage_radix2_simd128_noConj.
lastStage_radix4
Один проход по lastStage_radix4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость lastStage_radix4 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. lastStage_radix4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void lastStage_radix4_etalon(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE)
{
float *x_real_in = &data_in[0].real[0];
float *x_imag_in = &data_in[0].imag[0];
float *y_real_in = &data_in[0].real[1];
float *y_imag_in = &data_in[0].imag[1];
float *z_real_in = &data_in[0].real[2];
float *z_imag_in = &data_in[0].imag[2];
float *w_real_in = &data_in[0].real[3];
float *w_imag_in = &data_in[0].imag[3];
float *c_real_in = &coefC[0].real;
float *c_imag_in = &coefC[0].imag;
float *d_real_in = &coefD[0].real;
float *d_imag_in = &coefD[0].imag;
float *e_real_in = &coefE[0].real;
float *e_imag_in = &coefE[0].imag;
float *out_0_real = &data_out[0*data_count/4].real;
float *out_0_imag = &data_out[0*data_count/4].imag;
float *out_1_real = &data_out[1*data_count/4].real;
float *out_1_imag = &data_out[1*data_count/4].imag;
float *out_2_real = &data_out[2*data_count/4].real;
float *out_2_imag = &data_out[2*data_count/4].imag;
float *out_3_real = &data_out[3*data_count/4].real;
float *out_3_imag = &data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
float x_real = x_real_in[8*i];
float x_imag = x_imag_in[8*i];
float y_real = y_real_in[8*i];
float y_imag = y_imag_in[8*i];
float z_real = z_real_in[8*i];
float z_imag = z_imag_in[8*i];
float w_real = w_real_in[8*i];
float w_imag = w_imag_in[8*i];
float c_real = c_real_in[2*i];
float c_imag = c_imag_in[2*i];
float d_real = d_real_in[2*i];
float d_imag = d_imag_in[2*i];
float e_real = e_real_in[2*i];
float e_imag = e_imag_in[2*i];
float cy_real = c_real*y_real - c_imag*y_imag;
float cy_imag = c_real*y_imag + c_imag*y_real;
float dz_real = d_real*z_real - d_imag*z_imag;
float dz_imag = d_real*z_imag + d_imag*z_real;
float ew_real = e_real*w_real - e_imag*w_imag;
float ew_imag = e_real*w_imag + e_imag*w_real;
float add02_real = x_real + dz_real;
float add02_imag = x_imag + dz_imag;
float sub02_real = x_real - dz_real;
float sub02_imag = x_imag - dz_imag;
float add13_real = cy_real + ew_real;
float add13_imag = cy_imag + ew_imag;
float sub13_real = cy_real - ew_real;
float sub13_imag = cy_imag - ew_imag;
float sub13i_real = -sub13_imag;
float sub13i_imag = sub13_real;
out_0_real[2*i] = add02_real + add13_real;
out_0_imag[2*i] = add02_imag + add13_imag;
out_1_real[2*i] = sub02_real - sub13i_real;
out_1_imag[2*i] = sub02_imag - sub13i_imag;
out_2_real[2*i] = add02_real - add13_real;
out_2_imag[2*i] = add02_imag - add13_imag;
out_3_real[2*i] = sub02_real + sub13i_real;
out_3_imag[2*i] = sub02_imag + sub13i_imag;
}
}
Основной цикл на ассемблере
.L973:
{
fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=4, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0
}
{
fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=3, asz=3, abs=8, disp=0
}
{
fapb ct=1, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L394:
{
loop_mode
fmul_adds,0,sm %b[66], %b[72], %b[54], %b[1]
fsub_rsubs,1,sm %b[12], %b[71], %b[55], %b[5]
fsub_adds,2,sm %b[12], %b[71], %b[55], %b[45]
fmuls,4,sm %b[59], %b[58], %b[52]
fsubs,5,sm %b[79], %b[75], %b[53]
movaw,3 area=0, ind=16, am=0, be=0, %b[0]
}
{
loop_mode
fmul_rsubs,0,sm %b[66], %b[60], %b[82], %b[71]
fadd_adds,1,sm %b[24], %b[83], %b[90], %b[72]
fadd_rsubs,2,sm %b[24], %b[83], %b[90], %b[76]
fmuls,4,sm %b[59], %b[70], %b[80]
fadds,5,sm %b[79], %b[75], %b[88]
movaw,1 area=2, ind=4, am=0, be=0, %b[55]
movaw,2 area=0, ind=0, am=0, be=0, %b[12]
movaw,3 area=0, ind=12, am=0, be=0, %b[54]
}
{
loop_mode
fmul_rsubs,0,sm %b[42], %b[17], %b[84], %b[75]
fmul_rsubs,1,sm %b[32], %b[67], %b[85], %b[79]
staaw,2 %b[36], %aad3[ %aasti7 ]
fmuls,3,sm %b[29], %b[46], %b[82]
fmuls,4,sm %b[35], %b[23], %b[83]
staaw,5 %b[39], %aad1[ %aasti5 ]
movaw,0 area=2, ind=0, am=1, be=0, %b[60]
movaw,1 area=1, ind=0, am=0, be=0, %b[24]
movaw,2 area=0, ind=8, am=0, be=0, %b[59]
movaw,3 area=0, ind=28, am=0, be=0, %b[66]
}
{
loop_mode
fmul_adds,0,sm %b[40], %b[46], %b[86], %b[39]
fmul_adds,1,sm %b[32], %b[25], %b[87], %b[67]
staaw,2 %b[11], %aad4[ %aasti8 + _f32s,_lts0 0x4 ]
fmuls,3,sm %b[27], %b[13], %b[84]
fmuls,4,sm %b[35], %b[65], %b[85]
staaw,5 %b[51], %aad2[ %aasti6 + _f32s,_lts0 0x4 ]
movaw,0 area=1, ind=4, am=1, be=0, %b[29]
movaw,1 area=0, ind=0, am=0, be=0, %b[36]
movaw,2 area=0, ind=24, am=0, be=0, %b[17]
movaw,3 area=0, ind=20, am=0, be=0, %b[42]
}
{
loop_mode
fsub_rsubs,0,sm %b[22], %b[81], %b[48], %b[32]
fsub_adds,1,sm %b[22], %b[81], %b[48], %b[35]
staaw,2 %b[7], %aad3[ %aasti7 + _f32s,_lts0 0x4 ]
incr,2 %aaincr3
fsubs,4,sm %b[3], %b[43], %b[46]
staaw,5 %b[47], %aad1[ %aasti5 + _f32s,_lts0 0x4 ]
incr,5 %aaincr3
movaw,1 area=0, ind=4, am=1, be=0, %b[25]
movaw,3 area=0, ind=4, am=1, be=0, %b[11]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
fadd_adds,0,sm %b[10], %b[69], %b[50], %b[7]
fadd_rsubs,1,sm %b[10], %b[69], %b[50], %b[47]
staaw,2 %b[74], %aad4[ %aasti8 ]
incr,2 %aaincr3
fadds,4,sm %b[43], %b[3], %b[48]
staaw,5 %b[78], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Теоретическая скорость: 4 комплексных числа за 6 тактов (4/6) = 5.33 Байт/такт
Двойная теоретическая скорость: 10.67 Байт/такт
Замеры скорости
2. lastStage_radix4_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_etalon_unroll2(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE)
{
float *x_real_in = &data_in[0].real[0];
float *x_imag_in = &data_in[0].imag[0];
float *y_real_in = &data_in[0].real[1];
float *y_imag_in = &data_in[0].imag[1];
float *z_real_in = &data_in[0].real[2];
float *z_imag_in = &data_in[0].imag[2];
float *w_real_in = &data_in[0].real[3];
float *w_imag_in = &data_in[0].imag[3];
float *c_real_in = &coefC[0].real;
float *c_imag_in = &coefC[0].imag;
float *d_real_in = &coefD[0].real;
float *d_imag_in = &coefD[0].imag;
float *e_real_in = &coefE[0].real;
float *e_imag_in = &coefE[0].imag;
float *out_0_real = &data_out[0*data_count/4].real;
float *out_0_imag = &data_out[0*data_count/4].imag;
float *out_1_real = &data_out[1*data_count/4].real;
float *out_1_imag = &data_out[1*data_count/4].imag;
float *out_2_real = &data_out[2*data_count/4].real;
float *out_2_imag = &data_out[2*data_count/4].imag;
float *out_3_real = &data_out[3*data_count/4].real;
float *out_3_imag = &data_out[3*data_count/4].imag;
#pragma ivdep
#pragma unroll(2)
#pragma prefetch
for(int64_t i = 0; i < data_count/4; ++i)
{
float x_real = x_real_in[8*i];
float x_imag = x_imag_in[8*i];
float y_real = y_real_in[8*i];
float y_imag = y_imag_in[8*i];
float z_real = z_real_in[8*i];
float z_imag = z_imag_in[8*i];
float w_real = w_real_in[8*i];
float w_imag = w_imag_in[8*i];
float c_real = c_real_in[2*i];
float c_imag = c_imag_in[2*i];
float d_real = d_real_in[2*i];
float d_imag = d_imag_in[2*i];
float e_real = e_real_in[2*i];
float e_imag = e_imag_in[2*i];
float cy_real = c_real*y_real - c_imag*y_imag;
float cy_imag = c_real*y_imag + c_imag*y_real;
float dz_real = d_real*z_real - d_imag*z_imag;
float dz_imag = d_real*z_imag + d_imag*z_real;
float ew_real = e_real*w_real - e_imag*w_imag;
float ew_imag = e_real*w_imag + e_imag*w_real;
float add02_real = x_real + dz_real;
float add02_imag = x_imag + dz_imag;
float sub02_real = x_real - dz_real;
float sub02_imag = x_imag - dz_imag;
float add13_real = cy_real + ew_real;
float add13_imag = cy_imag + ew_imag;
float sub13_real = cy_real - ew_real;
float sub13_imag = cy_imag - ew_imag;
float sub13i_real = -sub13_imag;
float sub13i_imag = sub13_real;
out_0_real[2*i] = add02_real + add13_real;
out_0_imag[2*i] = add02_imag + add13_imag;
out_1_real[2*i] = sub02_real - sub13i_real;
out_1_imag[2*i] = sub02_imag - sub13i_imag;
out_2_real[2*i] = add02_real - add13_real;
out_2_imag[2*i] = add02_imag - add13_imag;
out_3_real[2*i] = sub02_real + sub13i_real;
out_3_imag[2*i] = sub02_imag + sub13i_imag;
}
}
Основной цикл на ассемблере
.L2296:
{
fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0
}
.L1081:
{
loop_mode
pfmul_rsubs,0,sm %g17, %g18, %g19, %b[62]
pshufb,1,sm %b[90], %b[87], %r0, %g16
pfmul_adds,2,sm %g17, %b[101], %b[117], %b[73]
insfd,3,sm %b[107], %r7, %b[98], %b[27]
insfd,4,sm %b[49], %r7, %b[97], %b[49]
pfmuls,5,sm %b[91], %b[29], %b[117]
movad,0 area=1, ind=0, am=0, be=0, %b[1]
movad,1 area=1, ind=8, am=1, be=0, %b[0]
}
{
loop_mode
pfsub_adds,0,sm %b[118], %b[64], %b[108], %b[85]
pfmuls,1,sm %b[115], %b[99], %g19
pfsub_adds,2,sm %b[119], %b[75], %g20, %b[88]
insfd,3,sm %b[32], %r7, %b[37], %b[92]
insfd,4,sm %b[78], %r7, %b[68], %b[81]
pfadds,5,sm %b[28], %b[33], %b[91]
movad,0 area=2, ind=0, am=0, be=0, %b[78]
movad,1 area=2, ind=8, am=1, be=0, %b[68]
movad,2 area=1, ind=0, am=0, be=0, %b[37]
movad,3 area=1, ind=8, am=1, be=0, %b[32]
}
{
loop_mode
pfadd_rsubs,0,sm %b[118], %b[64], %b[93], %b[98]
pfsubs,1,sm %b[55], %b[44], %b[106]
pfadd_rsubs,2,sm %b[119], %b[75], %b[113], %b[102]
insfd,3,sm %b[50], %r7, %b[61], %b[97]
insfd,4,sm %b[15], %r7, %b[14], %g18
pfmuls,5,sm %b[114], %b[81], %b[101]
movaw,0 area=0, ind=0, am=0, be=0, %b[15]
movaw,1 area=0, ind=16, am=0, be=0, %b[61]
movaw,2 area=0, ind=0, am=0, be=0, %b[50]
movaw,3 area=0, ind=16, am=0, be=0, %b[14]
}
{
loop_mode
pfsub_rsubs,0,sm %b[118], %b[64], %b[108], %b[107]
pfadds,1,sm %b[44], %b[55], %b[111]
pfsub_rsubs,2,sm %b[119], %b[75], %g20, %b[108]
insfd,3,sm %b[6], %r7, %b[7], %g22
pshufb,4,sm %b[104], %b[100], %r0, %g21
pfmuls,5,sm %b[115], %g18, %b[115]
movaw,0 area=0, ind=8, am=0, be=0, %b[6]
movaw,1 area=0, ind=24, am=0, be=0, %b[55]
movaw,2 area=0, ind=8, am=0, be=0, %b[7]
movaw,3 area=0, ind=24, am=0, be=0, %b[44]
}
{
loop_mode
pfadd_adds,0,sm %b[118], %b[64], %b[93], %b[75]
pshufb,1,sm %b[38], %b[43], %r0, %b[113]
pfadd_adds,2,sm %b[119], %b[75], %b[113], %b[84]
insfd,3,sm %b[74], %r7, %b[84], %g24
pshufb,4,sm %b[110], %b[109], %r0, %g23
pfsubs,5,sm %b[28], %b[33], %g20
movaw,0 area=0, ind=20, am=0, be=0, %b[33]
movaw,1 area=0, ind=4, am=0, be=0, %b[64]
movaw,2 area=0, ind=20, am=0, be=0, %b[28]
movaw,3 area=0, ind=4, am=0, be=0, %b[74]
}
{
loop_mode
pfmul_adds,0,sm %g22, %b[94], %b[103], %b[40]
pshufb,1,sm %b[2], %b[3], %r0, %b[112]
staad,2 %g16, %aad1[ %aasti5 + _f32s,_lts0 0x8 ]
insfd,3,sm %b[40], %r7, %b[45], %g17
pshufb,4,sm %b[86], %b[77], %r0, %g16
staad,5 %g23, %aad3[ %aasti7 + _f32s,_lts0 0x8 ]
movaw,0 area=0, ind=12, am=1, be=0, %b[94]
movaw,1 area=0, ind=28, am=0, be=0, %b[93]
movaw,2 area=0, ind=12, am=1, be=0, %b[103]
movaw,3 area=0, ind=28, am=0, be=0, %b[45]
}
{
loop_mode
pfmul_adds,0,sm %g24, %b[51], %b[117], %b[51]
pshufb,1,sm %b[70], %b[80], %r0, %b[87]
staad,2 %g21, %aad2[ %aasti6 + _f32s,_lts0 0x8 ]
insfd,3,sm %b[90], %r7, %b[87], %b[117]
insfd,4,sm %b[110], %r7, %b[109], %g16
staad,5 %g16, %aad4[ %aasti8 + _f32s,_lts0 0x8 ]
}
{
loop_mode
pfmul_rsubs,0,sm %g24, %b[29], %b[116], %b[29]
pfmuls,1,sm %b[114], %b[92], %b[77]
staad,2 %b[117], %aad1[ %aasti5 ]
incr,2 %aaincr3
insfd,3,sm %b[86], %r7, %b[77], %b[114]
insfd,4,sm %b[104], %r7, %b[100], %b[116]
staad,5 %g16, %aad3[ %aasti7 ]
incr,5 %aaincr3
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
pfmul_rsubs,0,sm %g22, %b[83], %b[79], %b[24]
pfmuls,1,sm %b[89], %b[49], %b[114]
staad,2 %b[114], %aad4[ %aasti8 ]
incr,2 %aaincr3
insfd,3,sm %b[60], %r7, %b[25], %b[116]
insfd,4,sm %b[24], %r7, %b[71], %b[117]
staad,5 %b[116], %aad2[ %aasti6 ]
incr,5 %aaincr3
}
Теоретическая скорость: 8 комплексных чисел за 9 тактов (8/9) = 7.11 Байт/такт
Двойная теоретическая скорость: 14.22 Байт/такт
Замеры скорости
3. lastStage_radix4_simd128_noConj
Модифицированный stage_radix4_simd128_noConj для работы с векторными данными.
Код на Си
void lastStage_radix4_simd128_noConj(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE)
{
__v2di *xyzw0_real_in = (__v2di*)&data_in[0].real;
__v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag;
__v2di *xyzw1_real_in = (__v2di*)&data_in[1].real;
__v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag;
__v2di *c_in = (__v2di*)coefC;
__v2di *d_in = (__v2di*)coefD;
__v2di *e_in = (__v2di*)coefE;
__v2di *out_0 = (__v2di*)&data_out[0*data_count/4];
__v2di *out_1 = (__v2di*)&data_out[1*data_count/4];
__v2di *out_2 = (__v2di*)&data_out[2*data_count/4];
__v2di *out_3 = (__v2di*)&data_out[3*data_count/4];
#pragma ivdep
#pragma unroll(1)
#pragma prefetch
for(int64_t i = 0; i < data_count/8; ++i)
{
__v2di xyzw0_real = xyzw0_real_in[4*i];
__v2di xyzw0_imag = xyzw0_imag_in[4*i];
__v2di xyzw1_real = xyzw1_real_in[4*i];
__v2di xyzw1_imag = xyzw1_imag_in[4*i];
__v2di c = c_in[i];
__v2di d = d_in[i];
__v2di e = e_in[i];
__v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di cy_real = __builtin_e2k_qpfmuls( c, y);
__v2di dz_real = __builtin_e2k_qpfmuls( d, z);
__v2di ew_real = __builtin_e2k_qpfmuls( e, w);
__v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y);
__v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z);
__v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w);
__v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real);
__v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real);
__v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real);
__v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag);
__v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag);
__v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag);
__v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di add02 = __builtin_e2k_qpfadds( x, dz);
__v2di sub02 = __builtin_e2k_qpfsubs( x, dz);
__v2di add13 = __builtin_e2k_qpfadds(cy, ew);
__v2di sub13 = __builtin_e2k_qpfsubs(cy, ew);
//__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63});
//__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31});
out_0[i] = __builtin_e2k_qpfadds(add02, add13);
out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i);
out_2[i] = __builtin_e2k_qpfsubs(add02, add13);
out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i);
}
}
Основной цикл на ассемблере
.L2983:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0
}
.L2350:
{
loop_mode
qpfmul_hsubs,0,sm %b[21], %b[49], %r14, %b[4]
qpshufb,1,sm %b[21], %b[21], %r10, %b[11]
qpfsub_rsubs,2,sm %b[42], %b[56], %b[54], %b[1]
qppermb,4,sm %b[24], %b[25], %r12, %b[9]
qpfadds,5,sm %b[48], %b[45], %b[0]
}
{
loop_mode
qpfmul_hadds,0,sm %b[11], %b[49], %r14, %b[21]
qppermb,1,sm %b[51], %b[55], %r12, %b[33]
qpfsub_adds,2,sm %b[42], %b[56], %b[54], %b[12]
qpshufb,4,sm %b[46], %b[46], %r10, %b[32]
qpfsubs,5,sm %b[48], %b[45], %b[16]
}
{
loop_mode
qppermb,1,sm %b[26], %b[27], %r0, %b[34]
qpshufb,3,sm %b[39], %b[39], %r10, %b[48]
qppermb,4,sm %b[53], %b[57], %r0, %b[45]
staaqp,5 %b[37], %aad4[ %aasti8 ]
incr,5 %aaincr0
movaqp,1 area=1, ind=0, am=1, be=0, %b[42]
movaqp,3 area=1, ind=0, am=1, be=0, %b[11]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[41], %b[33], %r14, %b[27]
qppermb,1,sm %b[23], %b[6], %r9, %b[54]
qpfmul_hadds,2,sm %b[50], %b[33], %r14, %b[26]
qpshufb,3,sm %b[52], %b[59], %r11, %b[53]
qpshufb,4,sm %b[43], %b[58], %r11, %b[49]
staaqp,5 %b[22], %aad2[ %aasti6 ]
incr,5 %aaincr0
movaqp,1 area=2, ind=0, am=1, be=0, %b[37]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[46], %b[9], %r14, %b[6]
qpshufb,1,sm %b[18], %b[18], %r10, %b[60]
qpfadd_adds,2,sm %b[40], %b[54], %b[2], %b[33]
qpshufb,3,sm %b[52], %b[59], %r13, %b[23]
qpshufb,4,sm %b[43], %b[58], %r13, %b[22]
staaqp,5 %b[3], %aad3[ %aasti7 ]
incr,5 %aaincr0
movaqp,0 area=0, ind=0, am=0, be=0, %b[57]
movaqp,1 area=0, ind=16, am=1, be=0, %b[56]
movaqp,2 area=0, ind=0, am=0, be=0, %b[50]
movaqp,3 area=0, ind=16, am=1, be=0, %b[41]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfmul_hadds,0,sm %b[32], %b[9], %r14, %b[3]
qpxor,1,sm %b[60], %r7, %b[52]
qpfadd_rsubs,2,sm %b[40], %b[54], %b[2], %b[18]
qppermb,3,sm %b[30], %b[31], %r9, %b[43]
qppermb,4,sm %b[7], %b[10], %r9, %b[46]
staaqp,5 %b[14], %aad1[ %aasti5 ]
incr,5 %aaincr0
}
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
4. lastStage_radix4_simd128_noConj_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_simd128_noConj_unroll2(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE)
{
__v2di *xyzw0_real_in = (__v2di*)&data_in[0].real;
__v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag;
__v2di *xyzw1_real_in = (__v2di*)&data_in[1].real;
__v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag;
__v2di *c_in = (__v2di*)coefC;
__v2di *d_in = (__v2di*)coefD;
__v2di *e_in = (__v2di*)coefE;
__v2di *out_0 = (__v2di*)&data_out[0*data_count/4];
__v2di *out_1 = (__v2di*)&data_out[1*data_count/4];
__v2di *out_2 = (__v2di*)&data_out[2*data_count/4];
__v2di *out_3 = (__v2di*)&data_out[3*data_count/4];
#pragma ivdep
#pragma unroll(2)
#pragma prefetch
for(int64_t i = 0; i < data_count/8; ++i)
{
__v2di xyzw0_real = xyzw0_real_in[4*i];
__v2di xyzw0_imag = xyzw0_imag_in[4*i];
__v2di xyzw1_real = xyzw1_real_in[4*i];
__v2di xyzw1_imag = xyzw1_imag_in[4*i];
__v2di c = c_in[i];
__v2di d = d_in[i];
__v2di e = e_in[i];
__v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di cy_real = __builtin_e2k_qpfmuls( c, y);
__v2di dz_real = __builtin_e2k_qpfmuls( d, z);
__v2di ew_real = __builtin_e2k_qpfmuls( e, w);
__v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y);
__v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z);
__v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w);
__v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real);
__v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real);
__v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real);
__v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag);
__v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag);
__v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag);
__v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di add02 = __builtin_e2k_qpfadds( x, dz);
__v2di sub02 = __builtin_e2k_qpfsubs( x, dz);
__v2di add13 = __builtin_e2k_qpfadds(cy, ew);
__v2di sub13 = __builtin_e2k_qpfsubs(cy, ew);
//__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63});
//__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31});
out_0[i] = __builtin_e2k_qpfadds(add02, add13);
out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i);
out_2[i] = __builtin_e2k_qpfsubs(add02, add13);
out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i);
}
}
Основной цикл на ассемблере
.L3893:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=96
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=4, abs=16, disp=0
}
{
fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0
}
.L3037:
{
loop_mode
qpfsub_adds,0,sm %b[16], %b[71], %b[66], %b[57]
qpshufb,1,sm %b[32], %b[32], %r14, %b[71]
qpfsub_rsubs,2,sm %b[16], %b[71], %b[66], %b[64]
qppermb,3,sm %b[38], %b[42], %r11, %b[0]
qppermb,4,sm %b[73], %b[76], %r11, %b[1]
qpfadds,5,sm %b[64], %b[57], %b[66]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[32], %b[83], %r15, %b[74]
qpshufb,1,sm %b[10], %b[10], %r14, %b[73]
qpfmul_hadds,2,sm %b[71], %b[83], %r15, %b[71]
qpshufb,4,sm %b[21], %b[25], %r9, %b[32]
qpfadds,5,sm %b[1], %b[0], %b[16]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[31], %b[75], %r15, %b[38]
qppermb,1,sm %b[69], %b[34], %r13, %b[31]
qpfmul_hadds,2,sm %b[72], %b[75], %r15, %b[34]
qpshufb,3,sm %b[19], %b[19], %r14, %b[54]
qppermb,4,sm %b[54], %b[55], %r13, %b[42]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[10], %b[65], %r15, %g17
qpshufb,1,sm %b[26], %b[26], %r14, %b[65]
qpfmul_hadds,2,sm %b[73], %b[65], %r15, %g16
qppermb,3,sm %g16, %g17, %r11, %b[55]
qppermb,4,sm %b[62], %b[63], %r11, %b[62]
qpfsubs,5,sm %b[3], %b[2], %b[63]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[41], %b[82], %r15, %b[61]
qppermb,1,sm %b[80], %b[78], %r11, %b[72]
qpfmul_hadds,2,sm %b[61], %b[82], %r15, %b[60]
qpshufb,3,sm %b[43], %b[46], %r9, %b[67]
qppermb,4,sm %b[60], %b[67], %r11, %b[69]
qpfsubs,5,sm %b[62], %b[55], %b[2]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[26], %b[35], %r15, %b[65]
qppermb,1,sm %b[58], %b[70], %r13, %b[3]
qpfmul_hadds,2,sm %b[65], %b[35], %r15, %b[58]
qpshufb,3,sm %b[29], %b[29], %r14, %b[70]
qppermb,4,sm %b[67], %b[32], %r12, %b[73]
staaqp,5 %b[77], %aad4[ %aasti8 ]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[19], %b[44], %r15, %b[76]
qppermb,1,sm %b[56], %b[68], %r13, %b[10]
qpfadd_adds,2,sm %b[7], %b[72], %b[66], %b[75]
qpshufb,3,sm %b[49], %b[53], %r0, %b[68]
qpshufb,4,sm %b[11], %b[52], %r0, %b[56]
staaqp,5 %b[79], %aad2[ %aasti6 ]
}
{
loop_mode
qpfmul_hadds,0,sm %b[54], %b[44], %r15, %b[78]
qpshufb,1,sm %b[4], %b[4], %r14, %b[79]
qpfadd_rsubs,2,sm %b[7], %b[72], %b[66], %b[77]
qpshufb,3,sm %b[23], %b[27], %r0, %b[66]
qpshufb,4,sm %b[45], %b[48], %r0, %b[54]
staaqp,5 %b[81], %aad2[ %aasti6 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
movaqp,0 area=3, ind=0, am=0, be=0, %b[4]
movaqp,1 area=1, ind=0, am=0, be=0, %b[23]
movaqp,3 area=1, ind=0, am=0, be=0, %b[19]
}
{
loop_mode
qpfadd_adds,0,sm %b[14], %b[69], %b[18], %g18
qpxor,1,sm %b[79], %r10, %b[82]
qpfadd_rsubs,2,sm %b[14], %b[69], %b[18], %b[79]
qppermb,3,sm %b[56], %b[68], %r12, %b[80]
qppermb,4,sm %b[54], %b[66], %r12, %b[81]
staaqp,5 %g18, %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
movaqp,0 area=3, ind=16, am=1, be=0, %b[27]
movaqp,1 area=2, ind=16, am=0, be=0, %b[26]
movaqp,3 area=2, ind=16, am=0, be=0, %b[18]
}
{
loop_mode
qpfsub_adds,0,sm %b[7], %b[72], %b[82], %g19
qpshufb,1,sm %b[63], %b[63], %r14, %b[59]
staaqp,2 %g19, %aad1[ %aasti5 ]
qpshufb,3,sm %b[49], %b[53], %r9, %b[53]
qpshufb,4,sm %b[11], %b[52], %r9, %b[52]
staaqp,5 %b[59], %aad3[ %aasti7 ]
movaqp,0 area=2, ind=0, am=1, be=0, %b[35]
movaqp,1 area=1, ind=16, am=1, be=0, %b[44]
movaqp,2 area=2, ind=0, am=1, be=0, %b[11]
movaqp,3 area=1, ind=16, am=1, be=0, %b[41]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfsub_rsubs,0,sm %b[7], %b[72], %b[82], %b[57]
qpxor,1,sm %b[59], %r10, %b[64]
staaqp,2 %b[57], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
incr,2 %aaincr3
qpshufb,3,sm %b[39], %b[39], %r14, %b[59]
qppermb,4,sm %b[52], %b[53], %r12, %b[63]
staaqp,5 %b[64], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
movaqp,0 area=0, ind=0, am=0, be=0, %b[49]
movaqp,1 area=0, ind=16, am=1, be=0, %b[48]
movaqp,2 area=0, ind=0, am=0, be=0, %b[45]
movaqp,3 area=0, ind=16, am=1, be=0, %b[7]
}
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
5. lastStage_radix4_simd128_noConj_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_simd128_noConj_unroll3(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE)
{
__v2di *xyzw0_real_in = (__v2di*)&data_in[0].real;
__v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag;
__v2di *xyzw1_real_in = (__v2di*)&data_in[1].real;
__v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag;
__v2di *c_in = (__v2di*)coefC;
__v2di *d_in = (__v2di*)coefD;
__v2di *e_in = (__v2di*)coefE;
__v2di *out_0 = (__v2di*)&data_out[0*data_count/4];
__v2di *out_1 = (__v2di*)&data_out[1*data_count/4];
__v2di *out_2 = (__v2di*)&data_out[2*data_count/4];
__v2di *out_3 = (__v2di*)&data_out[3*data_count/4];
#pragma ivdep
#pragma unroll(3)
#pragma prefetch
for(int64_t i = 0; i < data_count/8; ++i)
{
__v2di xyzw0_real = xyzw0_real_in[4*i];
__v2di xyzw0_imag = xyzw0_imag_in[4*i];
__v2di xyzw1_real = xyzw1_real_in[4*i];
__v2di xyzw1_imag = xyzw1_imag_in[4*i];
__v2di c = c_in[i];
__v2di d = d_in[i];
__v2di e = e_in[i];
__v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100});
__v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908});
__v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908});
__v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C});
__v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di cy_real = __builtin_e2k_qpfmuls( c, y);
__v2di dz_real = __builtin_e2k_qpfmuls( d, z);
__v2di ew_real = __builtin_e2k_qpfmuls( e, w);
__v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y);
__v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z);
__v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w);
__v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real);
__v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real);
__v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real);
__v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag);
__v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag);
__v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag);
__v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C});
__v2di add02 = __builtin_e2k_qpfadds( x, dz);
__v2di sub02 = __builtin_e2k_qpfsubs( x, dz);
__v2di add13 = __builtin_e2k_qpfadds(cy, ew);
__v2di sub13 = __builtin_e2k_qpfsubs(cy, ew);
//__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63});
//__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C});
__v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31});
out_0[i] = __builtin_e2k_qpfadds(add02, add13);
out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i);
out_2[i] = __builtin_e2k_qpfsubs(add02, add13);
out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i);
}
}
Основной цикл на ассемблере
.L5077:
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=12, disp=0
}
{
fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=16, disp=0
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=4, asz=3, abs=16, disp=32
}
{
fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=3, asz=3, abs=24, disp=32
fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=32
}
.L3947:
{
loop_mode
qpfmul_hadds,0,sm %b[93], %b[76], %r15, %b[75]
qpshufb,1,sm %b[85], %b[85], %r13, %b[91]
qpfadd_adds,2,sm %b[32], %b[87], %b[75], %b[76]
qppermb,3,sm %b[18], %b[17], %r10, %b[86]
qppermb,4,sm %b[91], %b[86], %r10, %b[85]
qpfsubs,5,sm %b[88], %b[89], %b[0]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[52], %b[94], %r15, %b[93]
qpxor,1,sm %b[91], %r12, %b[105]
qpfadd_adds,2,sm %b[80], %b[77], %b[83], %b[91]
qppermb,3,sm %b[44], %b[69], %r11, %b[103]
qppermb,4,sm %b[22], %b[27], %r11, %b[104]
qpfsubs,5,sm %b[85], %b[86], %b[83]
}
{
loop_mode
qpfmul_hadds,0,sm %b[97], %b[94], %r15, %b[85]
qpxor,1,sm %b[102], %r12, %b[97]
qpfsub_adds,2,sm %b[80], %b[77], %b[105], %b[82]
qppermb,3,sm %b[101], %b[100], %r10, %b[94]
qppermb,4,sm %b[82], %b[81], %r10, %b[86]
qpfadds,5,sm %b[85], %b[86], %b[81]
}
{
loop_mode
qpfmul_hadds,0,sm %b[95], %b[72], %r15, %b[100]
qpshufb,1,sm %b[2], %b[2], %r13, %b[101]
qpfsub_adds,2,sm %b[73], %b[78], %b[97], %b[95]
qpshufb,3,sm %b[39], %b[65], %r0, %b[1]
qpshufb,4,sm %b[68], %b[60], %r0, %b[2]
qpfsubs,5,sm %b[86], %b[94], %b[72]
movaqp,1 area=4, ind=16, am=0, be=0, %b[8]
movaqp,3 area=5, ind=0, am=1, be=0, %b[7]
}
{
loop_mode
qpfsub_rsubs,0,sm %b[80], %b[77], %b[105], %b[78]
qpxor,1,sm %b[101], %r12, %b[88]
qpfsub_rsubs,2,sm %b[73], %b[78], %b[97], %b[77]
qpshufb,3,sm %b[99], %b[98], %r0, %b[18]
qppermb,4,sm %b[2], %b[1], %r11, %b[80]
qpfadds,5,sm %b[88], %b[89], %b[73]
movaqp,0 area=5, ind=0, am=1, be=0, %b[17]
movaqp,1 area=1, ind=0, am=0, be=0, %b[23]
movaqp,3 area=1, ind=0, am=0, be=0, %b[26]
}
{
loop_mode
qpfsub_adds,0,sm %b[32], %b[87], %b[88], %b[87]
qpshufb,1,sm %b[61], %b[61], %r13, %b[89]
qpfsub_rsubs,2,sm %b[32], %b[87], %b[88], %b[86]
qpshufb,3,sm %b[99], %b[98], %r9, %b[39]
qpshufb,4,sm %b[96], %b[92], %r9, %b[45]
qpfadds,5,sm %b[86], %b[94], %b[88]
movaqp,0 area=4, ind=0, am=1, be=0, %b[31]
movaqp,1 area=1, ind=16, am=1, be=0, %b[32]
movaqp,2 area=4, ind=0, am=1, be=0, %b[40]
movaqp,3 area=1, ind=16, am=1, be=0, %b[36]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[61], %b[104], %r15, %b[84]
qpshufb,1,sm %b[35], %b[35], %r13, %b[94]
qpfmul_hadds,2,sm %b[89], %b[104], %r15, %b[89]
qpshufb,3,sm %b[96], %b[92], %r0, %b[60]
qppermb,4,sm %b[45], %b[39], %r11, %b[92]
staaqp,5 %b[84], %aad4[ %aasti8 ]
movaqp,0 area=3, ind=0, am=0, be=0, %b[52]
movaqp,1 area=3, ind=16, am=1, be=0, %b[57]
movaqp,2 area=3, ind=0, am=0, be=0, %b[46]
movaqp,3 area=3, ind=16, am=1, be=0, %b[51]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[35], %b[103], %r15, %b[98]
qpshufb,1,sm %b[56], %b[56], %r13, %b[103]
qpfmul_hadds,2,sm %b[94], %b[103], %r15, %b[99]
qpshufb,3,sm %b[42], %b[42], %r13, %b[101]
qppermb,4,sm %b[60], %b[18], %r11, %b[102]
staaqp,5 %b[90], %aad2[ %aasti6 ]
movaqp,0 area=2, ind=16, am=1, be=0, %b[90]
movaqp,1 area=2, ind=0, am=0, be=0, %b[96]
movaqp,2 area=2, ind=16, am=1, be=0, %b[94]
movaqp,3 area=2, ind=0, am=0, be=0, %b[97]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[56], %b[80], %r15, %b[79]
qppermb,1,sm %b[11], %b[12], %r11, %b[103]
qpfmul_hadds,2,sm %b[103], %b[80], %r15, %b[80]
qpshufb,3,sm %b[7], %b[7], %r13, %b[65]
qpshufb,4,sm %b[8], %b[8], %r13, %b[68]
staaqp,5 %b[79], %aad2[ %aasti6 + _f32s,_lts0 0x20 ]
movaqp,0 area=0, ind=16, am=1, be=0, %b[56]
movaqp,1 area=0, ind=0, am=0, be=0, %b[61]
movaqp,2 area=0, ind=16, am=1, be=0, %b[64]
movaqp,3 area=0, ind=0, am=0, be=0, %b[35]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[10], %b[103], %r15, %b[13]
qppermb,1,sm %b[47], %b[41], %r14, %b[70]
qpfmul_hadds,2,sm %b[70], %b[103], %r15, %b[14]
qpshufb,3,sm %b[26], %b[23], %r9, %b[10]
qppermb,4,sm %b[13], %b[14], %r14, %b[74]
staaqp,5 %b[74], %aad2[ %aasti6 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
}
{
loop_mode
qpfmul_hsubs,0,sm %b[9], %b[92], %r15, %b[41]
qppermb,1,sm %b[85], %b[93], %r10, %b[76]
qpfmul_hadds,2,sm %b[67], %b[92], %r15, %b[47]
qpshufb,3,sm %b[36], %b[32], %r9, %b[9]
qpshufb,4,sm %b[37], %b[63], %r9, %b[67]
staaqp,5 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x20 ]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[42], %b[102], %r15, %b[3]
qppermb,1,sm %b[4], %b[3], %r14, %b[71]
qpfmul_hadds,2,sm %b[101], %b[102], %r15, %b[4]
qpshufb,3,sm %b[66], %b[58], %r9, %b[42]
qppermb,4,sm %b[75], %b[71], %r10, %b[75]
staaqp,5 %b[91], %aad4[ %aasti8 + _f32s,_lts0 0x10 ]
incr,5 %aaincr3
}
{
loop_mode
qpfmul_hsubs,0,sm %b[21], %b[70], %r15, %g16
qppermb,1,sm %b[100], %g16, %r10, %b[85]
qpfadd_adds,2,sm %b[71], %b[76], %b[88], %b[82]
qpshufb,3,sm %b[28], %b[25], %r0, %b[25]
qpshufb,4,sm %b[55], %b[55], %r13, %b[91]
staaqp,5 %b[82], %aad1[ %aasti5 + _f32s,_lts0 0x10 ]
}
{
loop_mode
qpfmul_hsubs,0,sm %b[55], %b[74], %r15, %b[69]
qppermb,1,sm %b[62], %b[20], %r14, %b[28]
qpfadd_rsubs,2,sm %b[71], %b[76], %b[88], %b[88]
qpshufb,3,sm %b[38], %b[34], %r0, %b[20]
qppermb,4,sm %b[44], %b[69], %r14, %b[92]
staaqp,5 %b[95], %aad1[ %aasti5 ]
}
{
loop_mode
qpfadd_rsubs,0,sm %b[30], %b[85], %b[73], %b[77]
qppermb,1,sm %b[24], %b[29], %r14, %b[78]
staaqp,2 %b[78], %aad3[ %aasti7 + _f32s,_lts0 0x10 ]
qpshufb,3,sm %b[21], %b[21], %r13, %b[93]
qpshufb,4,sm %b[50], %b[50], %r13, %b[95]
staaqp,5 %b[77], %aad3[ %aasti7 ]
}
{
loop_mode
alc alcf=1, alct=1
abn abnf=1, abnt=1
ct %ctpr1 ? %NOT_LOOP_END
qpfadd_rsubs,0,sm %b[78], %b[75], %b[81], %b[72]
qpshufb,1,sm %b[72], %b[72], %r13, %b[100]
staaqp,2 %b[87], %aad1[ %aasti5 + _f32s,_lts0 0x20 ]
incr,2 %aaincr3
qppermb,3,sm %b[49], %b[43], %r10, %b[87]
qppermb,4,sm %b[6], %b[5], %r10, %b[86]
staaqp,5 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x20 ]
incr,5 %aaincr3
}
Теоретическая скорость: 24 комплексных числа за 16 тактов (24/16) = 12 Байт/такт
Двойная теоретическая скорость: 24 Байт/такт
Замеры скорости
Итоги по lastStage_radix4
При реализации Vector-4 FFT будем использовать lastStage_radix4_simd128_noConj_unroll2.
Собираем FFT
fft_radix2_vector2
Собираем fft_radix2_vector2 из reverse_radix2_vector2_stream16, всех вариантов stage_radix2_vector2 и lastStage_radix2_simd128_noConj.
fft_radix2_vector4
Собираем fft_radix2_vector4 из reverse_radix2_vector4_stream8, всех вариантов stage_radix2_vector4 и lastStage_radix4_simd128_noConj_unroll2.
fft_radix4_vector2
Собираем fft_radix4_vector2 из reverse_radix4_vector2_stream16, всех вариантов stage_radix4_vector2 и lastStage_radix2_simd128_noConj.
fft_radix4_vector4
Собираем fft_radix4_vector4 из reverse_radix4_vector4_stream16, всех вариантов stage_radix4_vector4 и lastStage_radix4_simd128_noConj_unroll2.
Дальнейшие улучшения имеющегося кода
Что изменить в текущем коде:
Перемежить коэффициенты.
Сейчас коэффициенты c/d/e размещаются в нескольких массивах.
Нужно разместить их в одном массиве, перемежив между собой.
Это повысит эффективность чтения памяти.
Скорость вне кэша должна вырасти.Заменить типы на векторные для упрощения внешнего вида кода.
Сейчас везде используются интринсики вместе с типамиuint64_tи__v2di.
Нужно заменитьuint64_tи__v2diна__v2sfи__v4sfсоответственно.
Тогда можно будет заменить интринсики на операции «+», «-», «*».
Код станет выглядеть менее громоздко.
Дальнейшие направления создания кода
Что делать дальше (в дополнение к указанному в предыдущей статье):
Рассмотреть все Stage в качестве LastStage.
Сейчас в качестве LastStage рассмотрены только некоторые Stage из прошлой статьи. Для полноты картины нужно рассмотреть остальные варианты Stage. Небольшое ускорение внутри кэша и замедление вне кэша ожидается от вариантов"_readConjSwap".Оптимизация для малых размеров входных данных (как в EML).
На графиках скорости FFT можно заметить выделяющийся угол у версии из EML.
Скорее всего, в EML сделана оптимизация для малых длин массивов (до 64 элементов). Оптимизация может состоять в хранении промежуточных результатов вычислений (между последовательными Stage) прямо в регистрах, а не в памяти. Если размеры входных данных достаточно малы, то такие данные можно целиком разместить в регистрах. В этом случае этап Reverse не нужен. Вместо него нужно просто переименовать регистры в первом Stage.
Заключение
Прямая векторизация проще для написания, понятнее для чтения и работает быстрее, чем тот код, что был написан в прошлой статье. Однако, нельзя сказать, что та работа была сделана напрасно. Ведь LastStage написан на основе кода из прошлой статьи.
Судя по графикам, версия из EML аналогична fft_radix2_vector4 с добавлением перемежения коэффициентов и оптимизации для малых размеров входных данных.
Пока я писал этот код, возникло ощущение, что было бы удобно создать классы myComplex, myComplex2 и myComplex4. Но сейчас думаю, что если заменить myComplex на родной float complex, то классы уже и не нужны.
KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.