{"id":2486,"date":"2024-06-08T07:45:54","date_gmt":"2024-06-08T07:45:54","guid":{"rendered":"https:\/\/mvermeulen.org\/perf\/?p=2486"},"modified":"2024-06-08T07:48:01","modified_gmt":"2024-06-08T07:48:01","slug":"clustering","status":"publish","type":"post","link":"https:\/\/mvermeulen.org\/perf\/2024\/06\/08\/clustering\/","title":{"rendered":"clustering"},"content":{"rendered":"\n

Following are affinity clusters of similar benchmarks based on on_cpu (# of cores used) and topdown metrics (retirement, fronted stall, backend stall, speculation)<\/p>\n\n\n\n

Cluster 0 (16 entries): 500.perlbench_r 508.namd_r 511.povray_r 525.x264_r 544.nab_r brl-cad c-ray john-the-ripper namd openssl povray qe quicksilver specfem3d svt-hevc vvenc\nCluster 1 (23 entries): 503.bwaves_r 507.cactuBSSN_r 510.parest_r 519.lbm_r 520.omnetpp_r 521.wrf_r 549.fotonik3d_r 554.roms_r ai-benchmark cloverleaf easywave kripke libxsmm minibud\ne mt-dgemm ncnn oidn onednn openvino stream tensorflow xmrig y-cruncher\nCluster 2 (4 entries): 541.leela_r compress-7zip m-queens n-queens\nCluster 3 (5 entries): aobench compress-lz4 gnupg lammps lzbench\nCluster 4 (7 entries): 538.imagick_r 548.exchange2_r avifenc helsing hmmer rays1bench uvg266\nCluster 5 (6 entries): build-eigen build-python compress-gzip hadoop inkscape tscp\nCluster 6 (8 entries): 505.mcf_r 531.deepsjeng_r appleseed asmfish blender primesieve stockfish v-ray\nCluster 7 (14 entries): build2 build-erlang build-ffmpeg build-gcc build-gdb build-gem5 build-godot build-imagemagick build-linux-kernel build-llvm build-mesa build-mplayer build-php\n build-wasmer\nCluster 8 (10 entries): apache cassandra compilebench ctx-clock dbench fast-cli ipc-benchmark memcached sqlite wireguard\nCluster 9 (8 entries): cp2k graphics-magick qmcpack rodinia smallpt stargate vpxenc x264\nCluster 10 (6 entries): blosc clickhouse core-latency daphne gimp mbw\nCluster 11 (6 entries): arrayfire dragonflydb nginx pgbench pjsip rbenchmark\nCluster 12 (5 entries): espeak phpbench pybench securemark smhasher\nCluster 13 (7 entries): apache-iotdb encode-wavpack fftw gcrypt java-scimark2 polybench-c synthmark\nCluster 14 (12 entries): 523.xalancbmk_r 526.blender_r 527.cam4_r 557.xz_r embree graph500 lczero openvkl ospray-studio quadray sysbench tensorflow-lite\nCluster 15 (10 entries): amg askap heffte hpcg incompact3d onnx openfoam parboil pytorch ramspeed\nCluster 16 (9 entries): compress-zstd cpp-perf-bench draco indigobench jpegxl jpegxl-decode polyhedron scimark2 z3\nCluster 17 (12 entries): clomp darktable deepsparse deepspeech ffte himeno llama.cpp llamafile lulesh ngspice npb palabos\nCluster 18 (6 entries): 502.gcc_r build-nodejs ebizzy faiss minife mnn\nCluster 19 (5 entries): botan cachebench glibc-bench gnuradio nettle\nCluster 20 (8 entries): blake2 build-apache build-clash octave-benchmark openjpeg selenium tungsten vkpeak\nCluster 21 (5 entries): mpcbench node-octane openscad pyperformance rav1e\nCluster 22 (7 entries): bork byte libreoffice numpy perl-benchmark rsvg sudokut\nCluster 23 (9 entries): compress-rar dacapobench duckdb ffmpeg gegl node-web-tooling pyhpc renaissance spark-tpcds\nCluster 24 (11 entries): aircrack-ng astcenc basis coremark cpuminer-opt java-jmh kvazaar mrbayes quantlib toybrot webp2\nCluster 25 (6 entries): dav1d opencv ospray schbench svt-av1 svt-vp9\nCluster 26 (9 entries): cryptopp dolfyn encode-flac gmpbench mutex nwchem rnnoise simdjson spark\nCluster 27 (6 entries): cockroach hackbench rocksdb scylladb speedb stress-ng\nCluster 28 (13 entries): aom-av1 financebench gpaw gromacs liquid-dsp neat openradioss pennant rawtherapee srsran tnn whisper.cpp x265\nCluster 29 (11 entries): bullet compress-pbzip2 crafty cython-bench encode-mp3 encode-opus etcpak fhourstones git libraw webp\n<\/code><\/pre>\n\n\n\n
Experimented some and settled on the following approach for clustering.<\/p>\n\n\n\n
Attributes of interest<\/h2>\n\n\n\n
First question was “clustering based on what”.  Following is a more complete set of metrics that range anywhere from the amount of I\/O to floating density to counter-based instrumentation.  These also have rather different ranges though can be normalized using the mean and standard deviation. I experimented first using a set of 10 metrics (on_cpu, retire, frontend, backend, speculation, IPC, GHz, float-density, branch-density, smt-contention) before settling on just the first five.<\/p>\n\n\n\n
Some of these metrics are correlated and why I figured it wouldn’t add value to use both AMD and Intel. Some such as I\/O metrics are very orthogonal and likely make sense in broader context but I don’t have enough study to clearly characterize.<\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
metric<\/th>count<\/th>min<\/th>max<\/th>median<\/th>mean<\/th>stddev<\/th>\n<\/tr>\n<\/thead>\n
elapsed<\/td>247<\/td>3.53<\/td>8.76e+03<\/td>532<\/td>1.15e+03<\/td>1.53e+03<\/td>\n<\/tr>\n
on_cpu<\/td>247<\/td>0<\/td>15.9<\/td>6.34<\/td>6.68<\/td>5.42<\/td>\n<\/tr>\n
inblock<\/td>247<\/td>0<\/td>4.46e+05<\/td>0<\/td>3.65e+03<\/td>3.92e+04<\/td>\n<\/tr>\n
onblock<\/td>247<\/td>0.46<\/td>8.97e+05<\/td>131<\/td>1.57e+04<\/td>7.91e+04<\/td>\n<\/tr>\n
page-fault<\/td>247<\/td>2.6<\/td>1.31e+05<\/td>2.1e+03<\/td>1.17e+04<\/td>2.1e+04<\/td>\n<\/tr>\n
context-switch<\/td>247<\/td>2.88<\/td>4.77e+04<\/td>66<\/td>2.49e+03<\/td>7.58e+03<\/td>\n<\/tr>\n
IPC<\/td>247<\/td>0.01<\/td>4.59<\/td>1.51<\/td>1.64<\/td>0.888<\/td>\n<\/tr>\n
GHz<\/td>247<\/td>0<\/td>5.23<\/td>1.87<\/td>1.82<\/td>1.4<\/td>\n<\/tr>\n
retire-rate<\/td>247<\/td>0.7<\/td>76.2<\/td>29.2<\/td>32.1<\/td>16<\/td>\n<\/tr>\n
frontend-stall<\/td>247<\/td>0.1<\/td>72.8<\/td>16.3<\/td>21.7<\/td>18<\/td>\n<\/tr>\n
backend-stall<\/td>247<\/td>4.1<\/td>97.4<\/td>38.2<\/td>42.9<\/td>21.9<\/td>\n<\/tr>\n
spec-stall<\/td>247<\/td>0<\/td>21.3<\/td>2<\/td>3.28<\/td>3.78<\/td>\n<\/tr>\n
retire-ucode<\/td>247<\/td>0<\/td>1.2<\/td>0<\/td>0.0769<\/td>0.141<\/td>\n<\/tr>\n
retire-fastpath<\/td>247<\/td>0.7<\/td>76.2<\/td>25.5<\/td>27.7<\/td>14.5<\/td>\n<\/tr>\n
float-density<\/td>247<\/td>0.016<\/td>676<\/td>62.9<\/td>129<\/td>151<\/td>\n<\/tr>\n
frontend-latency<\/td>247<\/td>0.1<\/td>57.8<\/td>8.4<\/td>13.4<\/td>13.6<\/td>\n<\/tr>\n
frontend-bandwidth<\/td>247<\/td>0<\/td>31.7<\/td>4.8<\/td>5.83<\/td>4.8<\/td>\n<\/tr>\n
opcache-miss<\/td>247<\/td>0<\/td>65.9<\/td>6.1<\/td>13.4<\/td>15.1<\/td>\n<\/tr>\n
icache-miss<\/td>247<\/td>0.1<\/td>69<\/td>13.4<\/td>16.8<\/td>11.5<\/td>\n<\/tr>\n
backend-cpu<\/td>247<\/td>0.7<\/td>64<\/td>9.4<\/td>12.8<\/td>11.4<\/td>\n<\/tr>\n
backend-memory<\/td>247<\/td>0.2<\/td>95.9<\/td>19.9<\/td>24.6<\/td>18.1<\/td>\n<\/tr>\n
amd-l2-miss<\/td>247<\/td>0.05<\/td>67.5<\/td>17.1<\/td>18.3<\/td>13<\/td>\n<\/tr>\n
amd-l2-density<\/td>247<\/td>0.022<\/td>4.29e+04<\/td>35.1<\/td>229<\/td>2.72e+03<\/td>\n<\/tr>\n
spec-branch<\/td>247<\/td>0<\/td>21.2<\/td>1.7<\/td>2.82<\/td>3.6<\/td>\n<\/tr>\n
spec-pipeline<\/td>247<\/td>0<\/td>2<\/td>0<\/td>0.113<\/td>0.249<\/td>\n<\/tr>\n
branch-miss<\/td>247<\/td>0<\/td>14.8<\/td>1.85<\/td>2.72<\/td>2.99<\/td>\n<\/tr>\n
branch-density<\/td>247<\/td>4.9<\/td>317<\/td>128<\/td>130<\/td>61.8<\/td>\n<\/tr>\n
branch-cond<\/td>247<\/td>4.5<\/td>311<\/td>92.3<\/td>98.3<\/td>48.8<\/td>\n<\/tr>\n
branch-ind<\/td>247<\/td>0.003<\/td>28.7<\/td>2.85<\/td>4.44<\/td>5.12<\/td>\n<\/tr>\n
smt-contention<\/td>247<\/td>0<\/td>48.4<\/td>9.6<\/td>12.7<\/td>13.3<\/td>\n<\/tr>\n
elapsed<\/td>238<\/td>16<\/td>1.69e+04<\/td>655<\/td>1.66e+03<\/td>2.5e+03<\/td>\n<\/tr>\n
on_cpu<\/td>238<\/td>0<\/td>15.8<\/td>6.77<\/td>7<\/td>5.54<\/td>\n<\/tr>\n
inblock<\/td>238<\/td>0.01<\/td>3.86e+05<\/td>65.8<\/td>6.31e+03<\/td>3.54e+04<\/td>\n<\/tr>\n
onblock<\/td>238<\/td>0.37<\/td>6.35e+05<\/td>19.9<\/td>1.1e+04<\/td>5.18e+04<\/td>\n<\/tr>\n
page-fault<\/td>238<\/td>4.07<\/td>1.21e+05<\/td>1.68e+03<\/td>1.06e+04<\/td>2.02e+04<\/td>\n<\/tr>\n
context-switch<\/td>238<\/td>2.23<\/td>6.44e+04<\/td>62.4<\/td>2.49e+03<\/td>8.92e+03<\/td>\n<\/tr>\n
IPC<\/td>238<\/td>0.01<\/td>5.53<\/td>1.89<\/td>2.02<\/td>1.01<\/td>\n<\/tr>\n
GHz<\/td>238<\/td>0<\/td>3.04<\/td>1.32<\/td>1.2<\/td>0.88<\/td>\n<\/tr>\n
retire-rate<\/td>238<\/td>3.7<\/td>87.3<\/td>43.2<\/td>43.4<\/td>15.4<\/td>\n<\/tr>\n
frontend-stall<\/td>238<\/td>0.5<\/td>52<\/td>15.9<\/td>17.8<\/td>11<\/td>\n<\/tr>\n
backend-stall<\/td>238<\/td>1.3<\/td>95.3<\/td>26.2<\/td>30.9<\/td>20.2<\/td>\n<\/tr>\n
spec-stall<\/td>238<\/td>0<\/td>46.7<\/td>6.1<\/td>8.38<\/td>8.3<\/td>\n<\/tr>\n
retire-ucode<\/td>238<\/td>0<\/td>16.7<\/td>2.9<\/td>3.26<\/td>2.26<\/td>\n<\/tr>\n
retire-fastpath<\/td>238<\/td>2.4<\/td>83.2<\/td>39.6<\/td>40.2<\/td>14.9<\/td>\n<\/tr>\n
frontend-latency<\/td>238<\/td>0.3<\/td>39.2<\/td>8.6<\/td>9.57<\/td>6.73<\/td>\n<\/tr>\n
frontend-bandwidth<\/td>238<\/td>0.1<\/td>25.8<\/td>7.3<\/td>8.2<\/td>5.84<\/td>\n<\/tr>\n
backend-cpu<\/td>238<\/td>0.6<\/td>76.1<\/td>11.5<\/td>15.5<\/td>12.4<\/td>\n<\/tr>\n
backend-memory<\/td>238<\/td>0.3<\/td>89.9<\/td>10.7<\/td>15.4<\/td>16.1<\/td>\n<\/tr>\n
l1-stall<\/td>238<\/td>0<\/td>29.6<\/td>4.1<\/td>5.53<\/td>5.66<\/td>\n<\/tr>\n
l2-stall<\/td>238<\/td>0<\/td>58.8<\/td>7.1<\/td>7.84<\/td>8.18<\/td>\n<\/tr>\n
l3-stall<\/td>238<\/td>0<\/td>35<\/td>2.3<\/td>3.98<\/td>5.42<\/td>\n<\/tr>\n
dram-stall<\/td>238<\/td>0<\/td>86.9<\/td>3.3<\/td>7.46<\/td>11.7<\/td>\n<\/tr>\n
store-stall<\/td>238<\/td>0<\/td>28.3<\/td>0.8<\/td>1.7<\/td>3.11<\/td>\n<\/tr>\n
intel-l2-miss<\/td>238<\/td>0.35<\/td>92.7<\/td>29.1<\/td>30.1<\/td>18.2<\/td>\n<\/tr>\n
intel-l2-density<\/td>238<\/td>0.019<\/td>2.12e+04<\/td>21.9<\/td>126<\/td>1.37e+03<\/td>\n<\/tr>\n
spec-branch<\/td>238<\/td>0<\/td>46.7<\/td>5.7<\/td>7.95<\/td>8.3<\/td>\n<\/tr>\n
spec-pipeline<\/td>238<\/td>0<\/td>6.2<\/td>0.3<\/td>0.427<\/td>0.652<\/td>\n<\/tr>\n
branch-miss<\/td>238<\/td>0<\/td>20.4<\/td>0.83<\/td>1.57<\/td>2.32<\/td>\n<\/tr>\n
branch-density<\/td>238<\/td>6.24<\/td>320<\/td>129<\/td>129<\/td>61<\/td>\n<\/tr>\n
branch-cond<\/td>238<\/td>6.24<\/td>320<\/td>129<\/td>129<\/td>61<\/td>\n<\/tr>\n
branch-ind<\/td>238<\/td>0.027<\/td>83<\/td>20.8<\/td>22.1<\/td>17.4<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
Clustering algorithm<\/h2>\n\n\n\n
After a web search, I ended up with a variation of Lloyd’s Algorithm<\/a>. This seems relatively straightforward and settles fairly quickly on a set of clusters. As a summary, this goes through the following steps:<\/p>\n\n\n\n