compiler – Performance analysis, tools and experiments

Home→Tags compiler

cpu2017

Posted on June 6, 2024 by mevJune 6, 2024

I have reached the point of diminishing returns for Phoronix tests. Reached 240 workloads analyzed and another ~30+ workloads that are skipped, mostly as GPU centric tests. These 270+ tests fully cover the 56 Phoronix benchmark articles so far this year. It has also become increasingly rare for a new article to reference an uncharacterized Phoronix test. When this happens, I will add to my analysis but I don’t see as much point in going through other, often obsolete workload examples. So I expect this to slowly creep up but not that quickly.

SPEC CPU is an interesting workload both as a workload and as a study of performance counters. There were three general issues that kept me from jumping full-bore into adding SPEC CPU until now:

The suite is expensive, ~$1000
My point is to characterize it as a workload, not to create hardware measurements. There are both detailed reporting rules and an emphasis on publicizing SPEC numbers to measure/compare hardware. While I would like the code to be somewhat optimized, I am also not trying for the absolute highest scores. So I will refrain from creating specific numbers with my “estimates” and use compiler options that I generally find without searching for optimal
SPEC CPU is a good measure of processor, memory and compiler. So for these measurements, I created config files with AMD AOCC compiler suite.

SPEC CPU has both rate benchmarks and speed benchmarks. The rate benchmarks maximize throughput, running multiple copies typically one per logical core. The speed benchmarks minimize latency, sometimes one copy but now also using OpenMP as it makes sense. I have concentrated first on the rate benchmarks. Looking at their profiles, I see some commonality between them and occasional variation with many Phoronix benchmarks.

benchmark	status	elapsed	on_cpu	inblock	onblock	page-fault	context-switch	IPC	GHz	retire-rate	frontend-stall	backend-stall	spec-stall	retire-ucode	retire-fastpath	float-density	frontend-latency	frontend-bandwidth	opcache-miss	icache-miss	backend-cpu	backend-memory	amd-l2-miss	amd-l2-density	spec-branch	spec-pipeline	branch-miss	branch-density	branch-cond	branch-ind	smt-contention
500.perlbench_r	CPU	1272.531	15.75	0.02	542.18	389.437	10.721	1.58	4.10	33.3	19.7	44.7	2.3	0.1	25.2	16.803	8.8	6.2	5.6	42.5	3.1	30.8	11.62	15.736	1.7	0.1	0.63	184.569	132.594	12.996	23.9
502.gcc_r	CPU	1280.597	15.70	0.00	4845.61	26600.971	10.948	0.60	4.39	12.7	25.6	59.8	1.9	0.0	9.6	7.943	12.6	6.9	15.2	29.7	3.1	42.5	29.37	60.271	1.4	0.0	1.99	222.751	170.047	5.594	23.8
503.bwaves_r	CPU	4632.332	15.85	0.00	4.72	565.445	11.053	0.13	4.47	2.3	1.4	96.3	0.1	0.0	2.2	260.237	1.2	0.2	9.7	6.5	6.8	86.5	49.20	132.988	0.0	0.0	1.38	19.172	14.975	1.398	3.0
505.mcf_r	CPU	706.488	15.73	0.00	15.39	1495.202	10.497	0.98	4.06	19.9	26.9	43.9	9.3	0.0	16.3	0.687	13.1	8.9	0.4	16.7	4.3	31.6	16.33	53.401	7.6	0.0	4.98	169.000	147.312	0.017	18.2
507.cactuBSSN_r	CPU	609.922	15.69	0.00	495.30	1135.260	13.491	0.23	4.21	4.3	4.8	90.9	0.1	0.0	4.1	55.472	3.6	1.1	47.7	41.8	9.1	78.5	10.76	476.467	0.0	0.0	0.64	48.787	33.403	3.734	3.6
508.namd_r	CPU	705.983	15.81	0.00	45.01	239.918	11.385	1.83	3.74	51.3	4.8	40.9	3.0	0.0	31.3	395.622	2.2	0.8	0.2	18.8	17.2	7.7	1.35	63.010	1.8	0.0	4.40	26.587	24.258	0.021	39.1
510.parest_r	CPU	4550.517	15.80	0.00	16.88	213.125	10.974	0.29	4.41	5.8	4.2	89.7	0.3	0.0	4.8	338.504	1.9	1.6	1.4	23.3	6.8	67.4	34.77	93.116	0.2	0.0	1.59	106.362	90.964	3.249	17.3
511.povray_r	CPU	1207.797	15.60	0.00	1449.68	63.683	10.402	1.81	3.78	52.4	6.6	38.9	2.1	0.3	31.4	244.832	2.5	1.5	3.7	20.9	11.2	12.4	0.06	63.187	1.1	0.2	0.25	157.689	109.665	10.900	39.5
519.lbm_r	CPU	1416.599	15.53	0.00	7.13	256.969	10.588	0.26	4.42	4.7	2.1	93.2	0.0	0.0	4.5	51.354	1.1	1.0	15.3	6.0	2.2	88.9	24.72	172.647	0.0	0.0	0.08	138.203	137.728	0.026	2.3
520.omnetpp_r	CPU	2024.583	15.83	0.00	870.95	107.720	9.994	0.28	4.58	5.6	9.6	81.9	3.0	0.0	4.9	16.375	4.4	3.9	2.3	28.4	4.8	66.7	44.62	72.990	2.5	0.1	3.02	196.963	143.767	11.785	12.8
521.wrf_r	CPU	1953.561	15.79	0.00	1454.42	198.479	12.169	0.43	4.35	7.8	7.9	83.8	0.5	0.0	7.4	278.072	6.2	1.2	3.1	27.3	15.4	63.5	26.15	77.326	0.4	0.0	0.90	113.379	77.677	13.945	5.9
523.xalancbmk_r	CPU	716.422	15.72	0.00	8293.00	930.346	10.489	0.77	4.32	19.0	8.9	71.1	0.9	0.1	12.2	34.341	3.6	2.2	4.0	20.7	3.0	42.9	15.96	75.229	0.5	0.1	0.34	267.659	234.583	7.449	35.4
525.x264_r	CPU	427.195	15.00	0.00	16419.20	1583.996	12.872	1.59	3.69	38.0	21.0	38.6	2.5	0.1	27.2	189.560	11.2	3.9	28.6	42.4	11.4	16.3	6.16	37.180	1.8	0.0	2.76	66.296	48.490	3.845	28.0
526.blender_r	CPU	1003.543	15.62	0.00	655.15	1981.576	11.564	1.08	3.88	24.9	6.6	62.3	6.2	0.0	17.9	397.608	3.7	1.1	1.0	28.3	13.6	31.2	15.56	31.225	4.5	0.0	2.09	134.627	120.414	1.256	28.1
527.cam4_r	CPU	1312.508	15.81	7.50	935.89	3656.413	10.919	0.84	4.07	16.4	16.0	66.9	0.7	0.0	14.5	189.588	9.3	4.9	8.0	25.9	19.5	40.0	20.78	62.695	0.6	0.0	1.15	124.383	89.321	8.772	11.1
531.deepsjeng_r	CPU	811.223	15.78	0.00	37.06	718.782	10.181	1.42	4.08	30.2	29.4	34.9	5.5	0.0	23.6	21.274	15.0	7.9	17.7	16.4	3.7	23.6	4.85	23.537	4.2	0.1	3.99	123.840	97.520	0.907	21.9
538.imagick_r	CPU	371.223	14.33	0.00	1329.38	1026.201	10.757	2.19	3.63	56.0	11.1	25.2	7.7	0.0	33.8	149.855	3.8	2.9	0.1	14.4	8.3	6.9	6.13	11.978	4.6	0.0	0.89	182.276	175.161	0.187	39.6
541.leela_r	CPU	1062.974	15.76	0.00	144.03	156.446	10.467	1.05	4.07	23.6	50.0	12.7	13.7	0.0	18.2	81.302	27.5	11.2	2.6	8.3	3.0	6.8	4.14	18.021	10.5	0.1	12.17	141.333	118.833	0.187	22.7
544.nab_r	CPU	581.066	15.53	0.00	23.20	301.419	11.174	1.27	3.77	36.9	8.7	52.1	2.3	0.1	25.5	318.530	5.0	1.1	1.4	15.1	22.8	13.3	4.94	52.578	1.5	0.0	1.30	83.289	72.175	1.774	30.8
548.exchange2_r	CPU	557.819	15.77	0.00	23.76	71.047	10.670	1.89	3.99	46.4	36.9	14.2	2.6	0.0	32.3	126.182	12.0	13.6	1.6	17.7	4.6	5.3	0.76	0.827	1.8	0.0	1.30	165.361	157.689	1.022	30.3
549.fotonik3d_r	CPU	4829.371	15.91	0.00	32.94	144.718	11.510	0.11	4.54	2.0	1.9	96.1	0.1	0.0	1.9	286.386	1.3	0.5	6.8	12.8	2.7	91.8	44.60	137.645	0.0	0.0	0.45	36.518	33.907	0.471	1.7
554.roms_r	CPU	2913.604	15.82	0.00	8.12	600.297	11.852	0.16	4.45	2.8	2.3	94.9	0.1	0.0	2.8	129.670	1.8	0.4	4.4	21.7	7.4	85.6	36.36	196.455	0.0	0.0	0.43	76.566	57.327	6.664	2.0
557.xz_r	CPU	1413.262	15.76	0.00	11.48	1363.376	9.484	0.77	4.41	19.4	9.5	64.8	6.3	0.0	12.6	21.356	4.5	1.7	1.0	17.8	2.2	39.8	30.11	23.253	4.1	0.0	4.53	115.607	104.940	1.340	35.2

The on_cpu values are high. This is very much a test of a CPU-dominated workload. There are not as many delays waiting for I/O, networking, graphics or other parts of the system. So there is an intensity to the mix that isn’t always as present with a more generic set of applications. Correspondingly the “GHz” values as a number of clock cycles per second are also high.
Most of the floating point benchmarks are dominated by backend stalls. On my 7840 processor, the memory subsystem more often becomes a limiter.

I have gone through fprate and am in process of working through intrate. While I have run the intspeed and fpspeed benchmarks, those are lower on my list to characterize. This sets me up for two later exercises to follow (a) after zen5 processors are available, I can use the benchmarks to see how the workloads compare on a zen5 vs zen4 core and (b) I am thinking of a “clustering” exercise to look for similarities between both phoronix and SPEC CPU.

graphics-magick sharpen, compiler improvements

Posted on March 19, 2024 by mevMarch 19, 2024

The following Phoronix Article – https://www.phoronix.com/review/nvidia-gh200-compilers compares GCC 13.2 with Clang 17.0.2 on an ARM platform. On the discussions attached the improvement for graphics-magick sharpen benchmark particularly stand out. So I thought I would see if I could see a similar improvement and using performance tools could spot likely areas contributing to the difference.

My system has Ubuntu 22.04 system compiler or gcc 11.4 and also aocc 4.1 based on clang 16.0.3 so not exactly the same but close enough. I forced a rebuild by reinstalling the test and setting environment variables, e.g.

export CC=/opt/AMD/aocc-compiler-4.1.0/bin/clang
export CXX=/opt/AMD/aocc-compiler-4.1.0/bin/clang++
export CFLAGS="-O3 -march=native"
export CXXFLAGS="-O3 -march=native"

With these differences, I see the following with gcc 11.4

    Operation: Sharpen:
        107
        108
        108

    Average: 108 Iterations Per Minute
    Deviation: 0.54%

and the following differences with clang 16.0

    Operation: Sharpen:
        177
        178
        178

    Average: 178 Iterations Per Minute
    Deviation: 0.32%

So overall a 1.65x speedup. Noy quite the 2x speedup seen on the AArch64 system but close enough given different compilers.

Here is what my topdown profile shows for gcc

Here is the comparison point with clang

Interestingly the total runtime is close to the same (time-bound test?) but we definitely have dropped backend stalls in favor of retiring a higher percentage of instructions.

Here is what the metrics show for gcc

elapsed              196.669
on_cpu               0.912          # 14.59 / 16 cores
utime                2847.753
stime                22.595
nvcsw                7799           # 21.59%
nivcsw               28324          # 78.41%
inblock              72             # 0.37/sec
onblock              12832          # 65.25/sec
cpu-clock            2870386800612  # 2870.387 seconds
task-clock           2870418438024  # 2870.418 seconds
page faults          8219671        # 2863.579/sec
context switches     36937          # 12.868/sec
cpu migrations       252            # 0.088/sec
major page faults    3              # 0.001/sec
minor page faults    8219668        # 2863.578/sec
alignment faults     0              # 0.000/sec
emulation faults     0              # 0.000/sec
branches             1140203600927  # 59.971 branches per 1000 inst
branch misses        9706629710     # 0.85% branch miss
conditional          1123801038031  # 59.108 conditional branches per 1000 inst
indirect             79758239       # 0.004 indirect branches per 1000 inst
cpu-cycles           12006859596451 # 3.83 GHz
instructions         18841741204039 # 1.57 IPC
slots                24015874376394 #
retiring             6873291214752  # 28.6% (44.9%)
-- ucode             776914684      #     0.0%
-- fastpath          6872514300068  #    28.6%
frontend             280560426894   #  1.2% ( 1.8%) low
-- latency           204333739230   #     0.9%
-- bandwidth         76226687664    #     0.3%
backend              8106573021445  # 33.8% (52.9%)
-- cpu               7904629941195  #    32.9%
-- memory            201943080250   #     0.8%
speculation          52507444606    #  0.2% ( 0.3%) low
-- branch mispredict 52421287288    #     0.2%
-- pipeline restart  86157318       #     0.0%
smt-contention       8702915928072  # 36.2% ( 0.0%)
cpu-cycles           12008757786517 # 3.84 GHz
instructions         18832540244485 # 1.57 IPC
instructions         6279648919771  # 2.124 l2 access per 1000 inst
l2 hit from l1       7173349879     # 20.29% l2 miss
l2 miss from l1      704685663      #
l2 hit from l2 pf    4162164156     #
l3 hit from l2 pf    1757001598     #
l3 miss from l2 pf   244954442      #
instructions         6277843164084  # 351.548 float per 1000 inst
float 512            57             # 0.000 AVX-512 per 1000 inst
float 256            584            # 0.000 AVX-256 per 1000 inst
float 128            2206965520855  # 351.548 AVX-128 per 1000 inst
float MMX            0              # 0.000 MMX per 1000 inst
float scalar         0              # 0.000 scalar per 1000 inst
instructions         18950819136221 #
opcache              2107351794817  # 111.201 opcache per 1000 inst
opcache miss         9523189344     #  0.5% opcache miss rate
l1 dTLB miss         902958198      # 0.048 L1 dTLB per 1000 inst
l2 dTLB miss         68055690       # 0.004 L2 dTLB per 1000 inst
instructions         18892305597227 #
icache               18578037535    # 0.983 icache per 1000 inst
icache miss          1477678165     #  8.0% icache miss rate
l1 iTLB miss         8626682        # 0.000 L1 iTLB per 1000 inst
l2 iTLB miss         0              # 0.000 L2 iTLB per 1000 inst
tlb flush            34816          # 0.000 TLB flush per 1000 inst

Here is what they show for clang

elapsed              198.605
on_cpu               0.910          # 14.55 / 16 cores
utime                2846.489
stime                43.933
nvcsw                10817          # 26.18%
nivcsw               30507          # 73.82%
inblock              8              # 0.04/sec
onblock              12904          # 64.97/sec
cpu-clock            2890592540363  # 2890.593 seconds
task-clock           2890613273288  # 2890.613 seconds
page faults          13446401       # 4651.747/sec
context switches     42134          # 14.576/sec
cpu migrations       320            # 0.111/sec
major page faults    51             # 0.018/sec
minor page faults    13446350       # 4651.729/sec
alignment faults     0              # 0.000/sec
emulation faults     0              # 0.000/sec
branches             1895985554702  # 85.338 branches per 1000 inst
branch misses        16208435546    # 0.85% branch miss
conditional          1856679790302  # 83.569 conditional branches per 1000 inst
indirect             162624101      # 0.007 indirect branches per 1000 inst
cpu-cycles           11802414391046 # 3.76 GHz
instructions         22167963494891 # 1.88 IPC
slots                23606223963606 #
retiring             7476214292393  # 31.7% (52.1%)
-- ucode             3459130581     #     0.0%
-- fastpath          7472755161812  #    31.7%
frontend             577593637926   #  2.4% ( 4.0%) low
-- latency           362394713874   #     1.5%
-- bandwidth         215198924052   #     0.9%
backend              6205319253065  # 26.3% (43.3%)
-- cpu               5685432163067  #    24.1%
-- memory            519887089998   #     2.2%
speculation          83292194787    #  0.4% ( 0.6%) low
-- branch mispredict 83160520795    #     0.4%
-- pipeline restart  131673992      #     0.0%
smt-contention       9263789209330  # 39.2% ( 0.0%)
cpu-cycles           11818914678350 # 3.74 GHz
instructions         22211450935976 # 1.88 IPC
instructions         7404943446705  # 2.939 l2 access per 1000 inst
l2 hit from l1       11586386135    # 19.79% l2 miss
l2 miss from l1      1111590991     #
l2 hit from l2 pf    6979543347     #
l3 hit from l2 pf    2793906722     #
l3 miss from l2 pf   399941912      #
instructions         7400104673984  # 491.708 float per 1000 inst
float 512            72             # 0.000 AVX-512 per 1000 inst
float 256            668            # 0.000 AVX-256 per 1000 inst
float 128            3638689694804  # 491.708 AVX-128 per 1000 inst
float MMX            0              # 0.000 MMX per 1000 inst
float scalar         0              # 0.000 scalar per 1000 inst
instructions         22251978896991 #
opcache              3428837218389  # 154.091 opcache per 1000 inst
opcache miss         16257852042    #  0.5% opcache miss rate
l1 dTLB miss         1527716103     # 0.069 L1 dTLB per 1000 inst
l2 dTLB miss         108536720      # 0.005 L2 dTLB per 1000 inst
instructions         22248633347533 #
icache               35471913129    # 1.594 icache per 1000 inst
icache miss          1971706421     #  5.6% icache miss rate
l1 iTLB miss         9490954        # 0.000 L1 iTLB per 1000 inst
l2 iTLB miss         0              # 0.000 L2 iTLB per 1000 inst
tlb flush            71325          # 0.000 TLB flush per 1000 inst

Looking with a rough comparison I notice:

User time is almost identical so likely some time-bounded loop
There are more instructions overall, and particularly AVX-128 has gone from 351 per thousand to 491 per thousand. The number of branches has also gone up
IPC has gone from 1.57 to 1.88.

Based on this my likely guess is some greater vectorization to tighten the core loop. This indirectly results in more branches (smaller loop). CPU stalls still contribute most to backend stalls but have gone down while number of vector instructions have gone up.

There may be other more direct ways to compare compiler options and results, but this is at least an indirect way to view the effects looking at the overall performance characterization.

Performance analysis, tools and experiments

An eclectic collection

Tag Archives: compiler

cpu2017

graphics-magick sharpen, compiler improvements