{"id":157,"date":"2024-01-02T00:13:08","date_gmt":"2024-01-02T00:13:08","guid":{"rendered":"https:\/\/mvermeulen.org\/perf\/?page_id=157"},"modified":"2024-01-02T01:07:01","modified_gmt":"2024-01-02T01:07:01","slug":"tensorflow","status":"publish","type":"page","link":"https:\/\/mvermeulen.org\/perf\/workloads\/phoronix\/tensorflow\/","title":{"rendered":"tensorflow"},"content":{"rendered":"\n<p>Phoronix tensorflow with resnet50 batch size 64<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"960\" src=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/01\/amdtopdown-4.png\" alt=\"\" class=\"wp-image-175\" srcset=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/01\/amdtopdown-4.png 1280w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/01\/amdtopdown-4-1024x768.png 1024w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/01\/amdtopdown-4-768x576.png 768w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/figure>\n\n\n\n<p>AMD metrics shows a workload with a lot of memory operations including L2 misses combined with cpu core operations with a fair amount of floating point.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elapsed              863.009\non_cpu               0.905          # 14.49 \/ 16 cores\nutime                12411.355\nstime                91.535\nnvcsw                3282710        # 91.85%\nnivcsw               291211         # 8.15%\ninblock              5720\nonblock              1200\ncpu-clock            12516157413304 # 12516.157 seconds\ntask-clock           12518177989728 # 12518.178 seconds\npage faults          11064521       # 883.876\/sec\ncontext switches     3578045        # 285.828\/sec\ncpu migrations       541658         # 43.270\/sec\nmajor page faults    42             # 0.003\/sec\nminor page faults    11064479       # 883.873\/sec\nalignment faults     0              # 0.000\/sec\nemulation faults     0              # 0.000\/sec\nbranches             584022412717   # 20.641 branches per 1000 inst\nbranch misses        5377049822     # 0.92% branch miss\nconditional          536426064913   # 18.959 conditional branches per 1000 inst\nindirect             3308982806     # 0.117 indirect branches per 1000 inst\ncpu-cycles           53588811756063 # 3.87 GHz\ninstructions         28290781703108 # 0.53 IPC\nslots                107133025023402 #\nretiring             9524494731791  #  8.9% (10.4%)\n-- ucode             997841396      #     0.0%\n-- fastpath          9523496890395  #     8.9%\nfrontend             1938764873769  #  1.8% ( 2.1%)\n-- latency           1594472883906  #     1.5%\n-- bandwidth         344291989863   #     0.3%\nbackend              79921103165442 # 74.6% (87.4%)\n-- cpu               34642755177137 #    32.3%\n-- memory            45278347988305 #    42.3%\nspeculation          45476223603    #  0.0% ( 0.0%)\n-- branch mispredict 38988923174    #     0.0%\n-- pipeline restart  6487300429     #     0.0%\nsmt-contention       15702947676761 # 14.7% ( 0.0%)\ncpu-cycles           53578106185765 # 3.87 GHz\ninstructions         28287157572549 # 0.53 IPC\ninstructions         9432751214605  # 134.834 l2 access per 1000 inst\nl2 hit from l1       928033007528   # 11.50% l2 miss\nl2 miss from l1      52635401579    #\nl2 hit from l2 pf    250202870552   #\nl3 hit from l2 pf    47147988792    #\nl3 miss from l2 pf   46468462841    #\ninstructions         9428428106186  # 37.224 float per 1000 inst\nfloat 512            68             # 0.000 AVX-512 per 1000 inst\nfloat 256            787391345      # 0.084 AVX-256 per 1000 inst\nfloat 128            350180974896   # 37.141 AVX-128 per 1000 inst\nfloat MMX            0              # 0.000 MMX per 1000 inst\nfloat scalar         31             # 0.000 scalar per 1000 inst\n<\/code><\/pre>\n\n\n\n<p>Intel metrics shows less time on the cores and also a longer overall runtime.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elapsed              1665.194\non_cpu               0.621          # 9.94 \/ 16 cores\nutime                16471.812\nstime                74.065\nnvcsw                2445852        # 86.38%\nnivcsw               385496         # 13.62%\ninblock              6928\nonblock              1168\ncpu-clock            16522570247592 # 16522.570 seconds\ntask-clock           16525566278147 # 16525.566 seconds\npage faults          9772881        # 591.379\/sec\ncontext switches     2839550        # 171.828\/sec\ncpu migrations       742592         # 44.936\/sec\nmajor page faults    51             # 0.003\/sec\nminor page faults    9772830        # 591.376\/sec\nalignment faults     0              # 0.000\/sec\nemulation faults     0              # 0.000\/sec\nbranches             1315588911649  # 22.142 branches per 1000 inst\nbranch misses        4086514516     # 0.31% branch miss\nconditional          1315588926369  # 22.142 conditional branches per 1000 inst\nindirect             470231770021   # 7.914 indirect branches per 1000 inst\nslots                81158674362062 #\nretiring             30632591230285 # 37.7% (37.7%)\n-- ucode             1473030581277  #     1.8%\n-- fastpath          29159560649008 #    35.9%\nfrontend             9540960332730  # 11.8% (11.8%)\n-- latency           8398912584891  #    10.3%\n-- bandwidth         1142047747839  #     1.4%\nbackend              40569942688186 # 50.0% (50.0%)\n-- cpu               14005646405876 #    17.3%\n-- memory            26564296282310 #    32.7%\nspeculation          1651850357581  #  2.0% ( 2.0%)\n-- branch mispredict 1271607577543  #     1.6%\n-- pipeline restart  380242780038   #     0.5%\nsmt-contention       0              #  0.0% ( 0.0%)\ncpu-cycles           54996415778281 # 2.07 GHz\ninstructions         77038363659819 # 1.40 IPC\nl2 access            1914216855761  # 62.810 l2 access per 1000 inst\nl2 miss              344524930940   # 18.00% l2 miss<\/code><\/pre>\n\n\n\n<p>The process tree time mostly concentrated in python code with a large number of parallel threads started, more than the # of cores.  Here is a summary of processes started.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>470 processes\n\t105 tf_cnn_benchmar      422684.85  2544.84\n\t 91 python3              372951.14  2240.24\n\t 38 vulkaninfo               0.76     1.14\n\t  6 php                      0.19     0.14\n\t  4 vulkani:disk$0           0.08     0.12\n\t  2 llvmpipe-0               0.04     0.06\n\t  2 llvmpipe-1               0.04     0.06\n\t  2 llvmpipe-10              0.04     0.06\n\t  2 llvmpipe-11              0.04     0.06\n\t  2 llvmpipe-12              0.04     0.06\n\t  2 llvmpipe-13              0.04     0.06\n\t  2 llvmpipe-14              0.04     0.06\n\t  2 llvmpipe-15              0.04     0.06\n\t  2 llvmpipe-2               0.04     0.06\n\t  2 llvmpipe-3               0.04     0.06\n\t  2 llvmpipe-4               0.04     0.06\n\t  2 llvmpipe-5               0.04     0.06\n\t  2 llvmpipe-6               0.04     0.06\n\t  2 llvmpipe-7               0.04     0.06\n\t  2 llvmpipe-8               0.04     0.06\n\t  2 llvmpipe-9               0.04     0.06\n\t  6 clang                    0.03     0.04\n\t  1 lspci                    0.00     0.04\n\t 88 sh                       0.00     0.00\n\t 12 gcc                      0.00     0.00\n\t 11 gsettings                0.00     0.00\n\t  9 stty                     0.00     0.00\n\t  8 stat                     0.00     0.00\n\t  8 systemd-detect-          0.00     0.00\n\t  6 llvm-link                0.00     0.00\n\t  5 glxinfo                  0.00     0.00\n\t  5 phoronix-test-s          0.00     0.00\n\t  3 gmain                    0.00     0.00\n\t  3 tensorflow               0.00     0.00\n\t  2 dconf worker             0.00     0.00\n\t  2 grep                     0.00     0.00\n\t  2 lscpu                    0.00     0.00\n\t  2 setterm                  0.00     0.00\n\t  2 uname                    0.00     0.00\n\t  2 which                    0.00     0.00\n\t  1 cc                       0.00     0.00\n\t  1 date                     0.00     0.00\n\t  1 dirname                  0.00     0.00\n\t  1 dmesg                    0.00     0.00\n\t  1 dmidecode                0.00     0.00\n\t  1 ifconfig                 0.00     0.00\n\t  1 ip                       0.00     0.00\n\t  1 lsmod                    0.00     0.00\n\t  1 mktemp                   0.00     0.00\n\t  1 ps                       0.00     0.00\n\t  1 readlink                 0.00     0.00\n\t  1 realpath                 0.00     0.00\n\t  1 sed                      0.00     0.00\n\t  1 sort                     0.00     0.00\n\t  1 systemctl                0.00     0.00\n\t  1 template.sh              0.00     0.00\n\t  1 wc                       0.00     0.00\n0 processes running\n75 maximum processes<\/code><\/pre>\n\n\n\n<p>The process tree shows the bulk of time spent in a parallel block with many more threads started than cores<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>      1915428) tensorflow start=291.21 finish=574.01\n        1915429) tf_cnn_benchmar start=291.22 finish=573.63\n          1915430) python3 start=291.34 finish=573.62\n          1915431) python3 start=291.34 finish=573.62\n          1915432) python3 start=291.34 finish=573.62\n          1915433) python3 start=291.34 finish=573.62\n          1915434) python3 start=291.34 finish=573.62\n          1915435) python3 start=291.34 finish=573.62\n          1915436) python3 start=291.34 finish=573.62\n          1915437) python3 start=291.34 finish=573.62\n          1915438) python3 start=291.34 finish=573.62\n          1915439) python3 start=291.34 finish=573.62\n          1915440) python3 start=291.34 finish=573.62\n          1915441) python3 start=291.34 finish=573.62\n          1915442) python3 start=291.34 finish=573.62\n          1915443) python3 start=291.35 finish=573.62\n          1915444) python3 start=291.35 finish=573.62\n          1915445) python3 start=291.78 finish=573.63\n          1915446) python3 start=291.78 finish=573.63\n          1915447) python3 start=291.78 finish=573.63\n          1915448) python3 start=291.78 finish=573.63\n          1915449) python3 start=291.78 finish=573.63\n          1915450) python3 start=291.78 finish=573.63\n          1915451) python3 start=291.78 finish=573.63\n          1915452) python3 start=291.78 finish=573.63\n          1915453) python3 start=291.78 finish=573.63\n          1915454) python3 start=291.78 finish=573.63\n          1915455) python3 start=291.78 finish=573.63\n          1915456) python3 start=291.78 finish=573.63\n          1915457) python3 start=291.78 finish=573.63\n          1915458) python3 start=291.79 finish=573.63\n          1915459) python3 start=291.79 finish=573.63\n          1915460) tf_cnn_benchmar start=292.10 finish=573.63\n          1915461) tf_cnn_benchmar start=292.10 finish=573.63\n          1915462) tf_cnn_benchmar start=292.10 finish=573.63\n          1915463) tf_cnn_benchmar start=292.10 finish=573.63\n          1915464) tf_cnn_benchmar start=292.10 finish=573.63\n          1915465) tf_cnn_benchmar start=292.10 finish=573.63\n          1915466) tf_cnn_benchmar start=292.10 finish=573.63\n          1915467) tf_cnn_benchmar start=292.10 finish=573.63\n          1915468) tf_cnn_benchmar start=292.10 finish=573.63\n          1915469) tf_cnn_benchmar start=292.10 finish=573.63\n          1915470) tf_cnn_benchmar start=292.10 finish=573.63\n          1915471) tf_cnn_benchmar start=292.10 finish=573.63\n          1915472) tf_cnn_benchmar start=292.10 finish=573.63\n          1915473) tf_cnn_benchmar start=292.10 finish=573.63\n          1915474) tf_cnn_benchmar start=292.10 finish=573.63\n          1915475) tf_cnn_benchmar start=292.10 finish=573.63\n          1915476) tf_cnn_benchmar start=292.10 finish=573.63\n          1915477) tf_cnn_benchmar start=292.10 finish=573.63\n          1915478) tf_cnn_benchmar start=292.10 finish=573.63\n          1915479) tf_cnn_benchmar start=292.10 finish=573.63\n          1915480) tf_cnn_benchmar start=292.10 finish=573.63\n          1915481) tf_cnn_benchmar start=292.11 finish=573.63\n          1915482) tf_cnn_benchmar start=292.11 finish=573.63\n          1915483) tf_cnn_benchmar start=292.11 finish=573.63\n          1915484) tf_cnn_benchmar start=292.11 finish=573.63\n          1915485) tf_cnn_benchmar start=292.11 finish=573.63\n          1915486) tf_cnn_benchmar start=292.11 finish=573.63\n          1915487) tf_cnn_benchmar start=292.11 finish=573.63\n          1915488) tf_cnn_benchmar start=292.11 finish=573.63\n          1915489) tf_cnn_benchmar start=292.11 finish=573.63\n          1915490) tf_cnn_benchmar start=292.11 finish=573.63\n          1915491) tf_cnn_benchmar start=292.11 finish=573.63\n          1915492) tf_cnn_benchmar start=292.11 finish=573.63\n          1915493) tf_cnn_benchmar start=293.26 finish=293.26\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Phoronix tensorflow with resnet50 batch size 64 AMD metrics shows a workload with a lot of memory operations including L2 misses combined with cpu core operations with a fair amount of floating point. Intel metrics shows less time on the <span class=\"excerpt-dots\">&hellip;<\/span> <a class=\"more-link\" href=\"https:\/\/mvermeulen.org\/perf\/workloads\/phoronix\/tensorflow\/\"><span class=\"more-msg\">Continue reading &rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":58,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-157","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/comments?post=157"}],"version-history":[{"count":2,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/157\/revisions"}],"predecessor-version":[{"id":176,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/157\/revisions\/176"}],"up":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/58"}],"wp:attachment":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/media?parent=157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}