I copied Stream from https://www.cs.virginia.edu/stream/ and put a copy in https://github.com/cycletourist/perf. This suggested the following compilation flags

/opt/AMD/aocc-compiler-4.1.0/bin/clang -O2 -fopenmp -mcmodel=large -ffp-contract=fast -fnt-store stream.c -DSTREAM_TYPE=double -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=100 -o stream

On my system with a Ryzen 7 7800X3D this results in the following performance:

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           44965.3     0.035895     0.035583     0.041095
Scale:          44902.1     0.035803     0.035633     0.040071
Add:            44214.4     0.054599     0.054281     0.057224
Triad:          44659.5     0.054155     0.053740     0.062816

The question is what is the sensitivity of various alternatives for compiling/running stream and can I do better than roughly ~45k MB/s?

Number of threads

The first dimension to try is the number of concurrent threads. By default, we run on all cores so 16 threads (8 cores by 2-way hyperthreading). However, there are only two memory channels on the processor, so perhaps if we limit threads we get less contention? Using likwid-topology we see the following

********************************************************************************
Cache Topology
********************************************************************************
Level:			1
Size:			32 kB
Cache groups:		( 0 8 ) ( 1 9 ) ( 2 10 ) ( 3 11 ) ( 4 12 ) ( 5 13 ) ( 6 14 ) ( 7 15 )
--------------------------------------------------------------------------------
Level:			2
Size:			1 MB
Cache groups:		( 0 8 ) ( 1 9 ) ( 2 10 ) ( 3 11 ) ( 4 12 ) ( 5 13 ) ( 6 14 ) ( 7 15 )
--------------------------------------------------------------------------------
Level:			3
Size:			96 MB
Cache groups:		( 0 8 1 9 2 10 3 11 4 12 5 13 6 14 7 15 )
--------------------------------------------------------------------------------

So other than avoiding the same core which share L1/L2, we try lower numbers of threads. Using 8 copies (one for each core) and “taskset -c 0,1,2,3,4,5,7 we are slightly higher

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           45801.8     0.035042     0.034933     0.036696
Scale:          45887.3     0.034946     0.034868     0.035780
Add:            44956.4     0.053540     0.053385     0.054156
Triad:          45478.8     0.052950     0.052772     0.053537

Using 4 copies and “taskset -c 0,2,4,6” we are higher still

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           47930.1     0.033463     0.033382     0.034021
Scale:          47935.9     0.033456     0.033378     0.033836
Add:            47367.2     0.050786     0.050668     0.051453
Triad:          47514.5     0.050662     0.050511     0.053086

Using 2 copies and “taskset -c 0,4” we are even higher

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           49953.0     0.032219     0.032030     0.032577
Scale:          50139.6     0.032096     0.031911     0.032346
Add:            49384.7     0.048934     0.048598     0.049323
Triad:          49297.6     0.049014     0.048684     0.049403

Using 1 copy and “taskset -c 0” we are lower again

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           45256.7     0.035486     0.035354     0.035758
Scale:          45941.4     0.034933     0.034827     0.035466
Add:            45713.4     0.052640     0.052501     0.053054
Triad:          45691.7     0.052665     0.052526     0.053191

For completeness, we try using 3 copies and “taskset -c 0,2,4” and are also slightly lower than the two copy run

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           49209.8     0.032582     0.032514     0.032765
Scale:          49169.0     0.032624     0.032541     0.032854
Add:            48532.8     0.049577     0.049451     0.049879
Triad:          48672.6     0.049475     0.049309     0.049913

So looks like for this processor it runs fastest with two threads, one for each memory channel.

Compiler options

The next dimension to try is the compiler and compiler options. Here we expect we have a recommended AOCC compiler and options, so don’t expect removing them to add performance – but useful to see anyways.

Running with “gcc -O2” instead of aocc results in slower performance

gcc -O2 -fopenmp -mcmodel=large stream.c -DSTREAM_TYPE=double -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=100 -o stream

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           38927.6     0.041432     0.041102     0.041713
Scale:          31689.5     0.050793     0.050490     0.051133
Add:            34335.7     0.070263     0.069898     0.070816
Triad:          34257.9     0.070204     0.070057     0.070589

The -fnt-store option uses a non-temporal store. This keeps the processor from keeping entries in the caches. This makes sense for stream since we are streaming through memory much larger than the cache and it otherwise gets polluted where cache entries conflict with new fetches from memory. Removing the -fnt-store option results in numbers close to the gcc numbers

/opt/AMD/aocc-compiler-4.1.0/bin/clang -O2 -fopenmp -mcmodel=large -ffp-contract=fast stream.c -DSTREAM_TYPE=double -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=100 -o stream

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           38575.4     0.041629     0.041477     0.041940
Scale:          31698.9     0.050659     0.050475     0.051251
Add:            34480.2     0.069831     0.069605     0.070409
Triad:          34473.8     0.069811     0.069618     0.070233

The -ffp-contract=fast option allows different core for contractions (e.g. multiply and add) so should primarily focus on “triad” which has those options. We essentially see no difference removing this option, so that may be a “don’t care”

/opt/AMD/aocc-compiler-4.1.0/bin/clang -O2 -fopenmp -mcmodel=large -fnt-store stream.c -DSTREAM_TYPE=double -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=100 -o stream

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           49973.1     0.032147     0.032017     0.032620
Scale:          50160.2     0.032049     0.031898     0.033233
Add:            49402.9     0.048854     0.048580     0.050982
Triad:          49385.9     0.048923     0.048597     0.049789

The -O3 option performance set of optimizations such as vectorization and loop optimizations. While beneficial to some codes, this slows down slightly for stream suggesting these are not helpful

/opt/AMD/aocc-compiler-4.1.0/bin/clang -O3 -fopenmp -mcmodel=large -ffp-contract=fast -fnt-store stream.c -DSTREAM_TYPE=double -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=100 -o stream

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           49810.3     0.032276     0.032122     0.032575
Scale:          49892.5     0.032225     0.032069     0.032478
Add:            48962.7     0.049157     0.049017     0.049637
Triad:          49026.6     0.049232     0.048953     0.049644

So overall, we use the options given.

Another point of comparison is the Intel compiler (icx). Compiling with options -axCORE-AVX2 -O3 -qopenmp -qopt-streaming-stores results in slightly lower performance. Will check this with an Intel system as well.

Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           47829.7     0.033554     0.033452     0.033797
Scale:          31703.3     0.050603     0.050468     0.050823
Add:            34460.0     0.069887     0.069646     0.070249
Triad:          34514.5     0.069775     0.069536     0.070086

Memory configuration

The other alternative we do not change is the memory configuration. This system has the following memory – https://www.gskill.com/product/165/374/1648545408/F5-5600J3636D32GX2-TZ5RK-F5-5600J3636D32GA2-TZ5RK. Each DIMM is 32 GB and there are four dimms. This thread suggests that DDR5 with 32GB are dual rank and DDR5 with 8GB are single rank. Also the processor specification suggests that 2x2R can run at DDR5-5200 and 4x2R runs at DDR5-3600.

I haven’t done the experiments but there is a suggestion that if this processor is paired with 16GB of RAM in 2 DIMMs that perhaps we can see faster stream performance than the current 128GB of RAM in 4 DIMMs. There would of course be a tradeoff on other workloads with a larger working set size.

Versions of Stream

Another variable would be the specific version of stream. As a comparison, I tried running the phoronix-test-suite copy of the stream benchmark and got the following results

Stream 2013-01-17:
    pts/stream-1.3.4 [Type: Copy]
    Test 1 of 4
    Estimated Trial Run Count:    5                      
    Estimated Test Run-Time:      4 Minutes              
    Estimated Time To Completion: 16 Minutes [15:04 CST] 
        Started Run 1 @ 14:48:35
        Started Run 2 @ 14:50:18
        Started Run 3 @ 14:52:02
        Started Run 4 @ 14:53:45
        Started Run 5 @ 14:55:28

    Type: Copy:
        44593
        44619.9
        44697.6
        44632.3
        44658.9

    Average: 44640.3 MB/s
    Deviation: 0.09%

Stream 2013-01-17:
    pts/stream-1.3.4 [Type: Scale]
    Test 2 of 4
    Estimated Trial Run Count:    5                      
    Estimated Test Run-Time:      9 Minutes              
    Estimated Time To Completion: 26 Minutes [15:22 CST] 
        Utilizing Data From Shared Cache @ 14:57:12

    Type: Scale:
        28951.6
        28930.4
        28951.2
        28969.1
        28933.6

    Average: 28947.2 MB/s
    Deviation: 0.05%

Stream 2013-01-17:
    pts/stream-1.3.4 [Type: Triad]
    Test 3 of 4
    Estimated Trial Run Count:    5                      
    Estimated Test Run-Time:      9 Minutes              
    Estimated Time To Completion: 17 Minutes [15:13 CST] 
        Utilizing Data From Shared Cache @ 14:57:14

    Type: Triad:
        32139.4
        32127.3
        32150.8
        32161.8
        32152.7

    Average: 32146.4 MB/s
    Deviation: 0.04%

Stream 2013-01-17:
    pts/stream-1.3.4 [Type: Add]
    Test 4 of 4
    Estimated Trial Run Count:    5                     
    Estimated Time To Completion: 9 Minutes [15:05 CST] 
        Utilizing Data From Shared Cache @ 14:57:16

    Type: Add:
        32140.2
        32135.8
        32107.5
        32110.3
        32120.3

    Average: 32122.8 MB/s
    Deviation: 0.05%

These are substantially slower. A peek at the installed directory and run logs suggests several reasons

• Sixteen threads are run instead of two
• The compiler is gcc and compiler options are “-mcmodel=medium -O3 -march=native -fopenmp”
• We run an array size of 402653184 elements instead of 100000000 elements (though I suspect this doesn’t have as much effect)

The other remaining variable would be to find the older version of stream I previously ran ~five years ago. Will find this and also compare.