{"id":2151,"date":"2024-03-22T10:14:41","date_gmt":"2024-03-22T10:14:41","guid":{"rendered":"https:\/\/mvermeulen.org\/perf\/?page_id=2151"},"modified":"2024-03-23T09:52:38","modified_gmt":"2024-03-23T09:52:38","slug":"hadoop","status":"publish","type":"page","link":"https:\/\/mvermeulen.org\/perf\/workloads\/phoronix\/hadoop\/","title":{"rendered":"hadoop"},"content":{"rendered":"\n<p>Apache Hadoop using a built-in benchmark. The test itself has multiple configurations, picked one slightly smaller one with operations for open, create, delete, rename and file status. These are  done with 50 threads and 100,000 files. These seem to be spiky in terms of how much on-core is used.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"960\" src=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/systemtime-39.png\" alt=\"\" class=\"wp-image-2159\" srcset=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/systemtime-39.png 1280w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/systemtime-39-1024x768.png 1024w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/systemtime-39-768x576.png 768w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/figure>\n\n\n\n<p>All the operations appear to have similar profiles.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"960\" src=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/amdtopdown-41.png\" alt=\"\" class=\"wp-image-2158\" srcset=\"https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/amdtopdown-41.png 1280w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/amdtopdown-41-1024x768.png 1024w, https:\/\/mvermeulen.org\/perf\/wp-content\/uploads\/sites\/7\/2024\/03\/amdtopdown-41-768x576.png 768w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/figure>\n\n\n\n<p>AMD profile shows on average only one core. There is a reasonable branch misprediction ratio and higher icache misses contributing to front end stalls.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elapsed              1645.269\non_cpu               0.066          # 1.06 \/ 16 cores\nutime                1483.955\nstime                266.041\nnvcsw                19302655       # 97.72%\nnivcsw               449656         # 2.28%\ninblock              0              # 0.00\/sec\nonblock              12071720       # 7337.23\/sec\ncpu-clock            2033061393428  # 2033.061 seconds\ntask-clock           2040938903340  # 2040.939 seconds\npage faults          32715886       # 16029.821\/sec\ncontext switches     21041669       # 10309.799\/sec\ncpu migrations       4409281        # 2160.418\/sec\nmajor page faults    13             # 0.006\/sec\nminor page faults    32693761       # 16018.981\/sec\nalignment faults     0              # 0.000\/sec\nemulation faults     0              # 0.000\/sec\nbranches             2440557014096  # 192.814 branches per 1000 inst\nbranch misses        70292043999    # 2.88% branch miss\nconditional          1985101834707  # 156.831 conditional branches per 1000 inst\nindirect             88675993406    # 7.006 indirect branches per 1000 inst\ncpu-cycles           7283153577899  # 0.27 GHz\ninstructions         12733302691242 # 1.75 IPC\nslots                15117443150724 #\nretiring             4093571277912  # 27.1% (28.8%)\n-- ucode             18565784593    #     0.1%\n-- fastpath          4075005493319  #    27.0%\nfrontend             5668074910837  # 37.5% (39.9%)\n-- latency           4187468977962  #    27.7%\n-- bandwidth         1480605932875  #     9.8%\nbackend              3252934091472  # 21.5% (22.9%)\n-- cpu               440157329922   #     2.9%\n-- memory            2812776761550  #    18.6%\nspeculation          1178546967447  #  7.8% ( 8.3%)\n-- branch mispredict 1162848752037  #     7.7%\n-- pipeline restart  15698215410    #     0.1%\nsmt-contention       922950531585   #  6.1% ( 0.0%)\ncpu-cycles           6723415939481  # 0.27 GHz\ninstructions         11687210661615 # 1.74 IPC\ninstructions         3949493254292  # 35.068 l2 access per 1000 inst\nl2 hit from l1       117662844689   # 23.41% l2 miss\nl2 miss from l1      22225189608    #\nl2 hit from l2 pf    10639199506    #\nl3 hit from l2 pf    7690790023     #\nl3 miss from l2 pf   2506757202     #\ninstructions         3939984934674  # 12.396 float per 1000 inst\nfloat 512            5876           # 0.000 AVX-512 per 1000 inst\nfloat 256            580696         # 0.000 AVX-256 per 1000 inst\nfloat 128            48840297285    # 12.396 AVX-128 per 1000 inst\nfloat MMX            0              # 0.000 MMX per 1000 inst\nfloat scalar         26670          # 0.000 scalar per 1000 inst\ninstructions         14677205691183 #\nopcache              2833928335429  # 193.084 opcache per 1000 inst\nopcache miss         423825305666   # 15.0% opcache miss rate\nl1 dTLB miss         43954457210    # 2.995 L1 dTLB per 1000 inst\nl2 dTLB miss         2729867433     # 0.186 L2 dTLB per 1000 inst\ninstructions         11454228858121 #\nicache               618852003376   # 54.028 icache per 1000 inst\nicache miss          163637198109   # 26.4% icache miss rate\nl1 iTLB miss         10599527518    # 0.925 L1 iTLB per 1000 inst\nl2 iTLB miss         0              # 0.000 L2 iTLB per 1000 inst\ntlb flush            4179118        # 0.000 TLB flush per 1000 inst\n<\/code><\/pre>\n\n\n\n<p>Intel metrics show L2 memory has heaviest usage.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elapsed              1908.392\non_cpu               0.073          # 1.17 \/ 16 cores\nutime                1967.641\nstime                272.647\nnvcsw                19875397       # 93.34%\nnivcsw               1417344        # 6.66%\ninblock              5600           # 2.93\/sec\nonblock              12147032       # 6365.06\/sec\ncpu-clock            2609535955733  # 2609.536 seconds\ntask-clock           2618682297029  # 2618.682 seconds\npage faults          32849851       # 12544.420\/sec\ncontext switches     22854844       # 8727.612\/sec\ncpu migrations       6394302        # 2441.801\/sec\nmajor page faults    32             # 0.012\/sec\nminor page faults    32829180       # 12536.526\/sec\nalignment faults     0              # 0.000\/sec\nemulation faults     0              # 0.000\/sec\nbranches             3342701443304  # 197.833 branches per 1000 inst\nbranch misses        63340712362    # 1.89% branch miss\nconditional          3342703351624  # 197.833 conditional branches per 1000 inst\nindirect             418999339481   # 24.798 indirect branches per 1000 inst\nslots                30031057052438 #\nretiring             11866028127417 # 39.5% (39.5%)\n-- ucode             546411586624   #     1.8%\n-- fastpath          11319616540793 #    37.7%\nfrontend             7206824202511  # 24.0% (24.0%)\n-- latency           3954378356567  #    13.2%\n-- bandwidth         3252445845944  #    10.8%\nbackend              6063507461747  # 20.2% (20.2%)\n-- cpu               4004372975049  #    13.3%\n-- memory            2059134486698  #     6.9%\nspeculation          5089708599852  # 16.9% (16.9%) high\n-- branch mispredict 4965012099294  #    16.5%\n-- pipeline restart  124696500558   #     0.4%\nsmt-contention       0              #  0.0% ( 0.0%)\ncpu-cycles           8613657887438  # 0.28 GHz\ninstructions         20456692773872 # 2.37 IPC\nl2 access            296465887800   # 22.784 l2 access per 1000 inst\nl2 miss              89267010975    # 30.11% l2 miss\ncpu-cycles           5466044200874  # 18.1% memory latency\nload stalls          924169128515   #  1.0% l1 bound\nl1 miss              871316263366   #  9.0% l2 bound\nl2 miss              379565201362   #  4.1% l3 bound\nl3 miss              157380003321   #  2.9% dram bound\nstore_stalls         67401023726    #  1.2% store bound\n<\/code><\/pre>\n\n\n\n<p>Process profile shows a large number of processes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>61181 processes\n\t351 org.apache.hado       5364.25   856.42\n\t1172 java                  3719.09   430.50\n\t147 qtp1346799731-3       2700.20   471.70\n\t586 Finalizer             1859.53   215.18\n\t585 Common-Cleaner        1859.43   215.11\n\t105 qtp1174248013-3       1854.74   304.29\n\t585 Logging-Cleaner       1851.69   212.62\n\t195 Hadoop-Metrics-       1842.68   207.92\n\t 75 StatsDaemon-41        1209.12   215.25\n\t 75 StatsDaemon-18        1209.05   215.16\n\t 75 StatsDaemon-42        1208.61   215.17\n\t 75 StatsDaemon-46        1208.29   215.30\n\t 75 StatsDaemon-40        1207.93   215.20\n\t 75 StatsDaemon-25        1207.88   215.12\n\t 75 StatsDaemon-33        1207.70   215.15\n\t 75 StatsDaemon-49        1207.69   215.28\n\t 75 StatsDaemon-47        1207.65   215.32\n\t 75 StatsDaemon-45        1207.45   215.30\n\t 75 StatsDaemon-28        1207.42   215.09\n\t 75 StatsDaemon-29        1207.30   215.12\n\t 75 StatsDaemon-32        1207.24   215.12\n\t 75 StatsDaemon-20        1207.21   215.13\n\t 75 StatsDaemon-44        1207.19   215.22\n\t 75 StatsDaemon-39        1207.18   215.14\n\t 75 StatsDaemon-23        1207.11   215.19\n\t 75 StatsDaemon-34        1207.11   215.12\n\t 75 StatsDaemon-24        1207.04   215.20\n\t 75 StatsDaemon-15        1207.01   215.17\n\t 75 StatsDaemon-26        1206.99   215.08\n\t 75 StatsDaemon-30        1206.82   215.08\n\t 75 StatsDaemon-38        1206.80   215.10\n\t 75 StatsDaemon-37        1206.78   215.20\n\t 75 StatsDaemon-27        1206.73   215.14\n\t 75 StatsDaemon-43        1206.66   215.24\n\t 75 StatsDaemon-19        1206.60   215.13\n\t 75 StatsDaemon-17        1206.59   215.10\n\t 75 StatsDaemon-31        1206.56   215.06\n\t 75 StatsDaemon-36        1206.21   215.09\n\t 75 StatsDaemon-48        1206.00   215.18\n\t 75 StatsDaemon-16        1205.83   215.10\n\t 75 StatsDaemon-21        1205.73   215.08\n\t 75 StatsDaemon-35        1205.61   215.19\n\t 75 StatsDaemon-14        1205.53   215.03\n\t 75 StatsDaemon-22        1204.71   215.14\n\t 75 StatsDaemon-13        1204.13   214.80\n\t 75 StatsDaemon-12        1202.89   214.86\n\t 75 StatsDaemon-5         1202.57   214.51\n\t 75 StatsDaemon-8         1202.54   214.61\n\t 75 StatsDaemon-7         1202.25   214.63\n\t 75 StatsDaemon-6         1201.79   214.50\n\t 75 StatsDaemon-11        1201.59   214.77\n\t 75 StatsDaemon-9         1201.59   214.57\n\t 75 StatsDaemon-3         1201.18   214.60\n\t 75 StatsDaemon-1         1200.91   214.36\n\t 75 StatsDaemon-4         1200.80   214.50\n\t 75 StatsDaemon-10        1200.07   214.47\n\t 75 StatsDaemon-0         1199.48   214.04\n\t 75 StatsDaemon-2         1199.35   214.31\n\t 63 qtp1346799731-2       1157.37   202.20\n\t 78 Session-HouseKe        979.42   138.82\n\t115 ForkJoinPool.co        856.56    69.77\n\t 45 qtp1174248013-2        794.97   130.41\n\t 39 AsyncAppender-D        693.12   116.80\n\t 39 pool-2-thread-1        690.09   116.47\n\t 39 pool-1-thread-1        689.93   116.45\n\t 39 DatanodeAdminMo        684.76   116.29\n\t 39 MarkedDeleteBlo        684.73   116.29\n\t 39 RedundancyMonit        684.66   116.29\n\t 39 pool-7-thread-1        684.19   116.07\n\t 39 CacheReplicatio        684.12   116.03\n\t 39 FSEditLogAsync         683.11   115.62\n\t 75 qtp2134157671-3        549.43    41.05\n\t 75 qtp2134157671-4        549.04    41.05\n\t585 Thread-0               457.67    59.84\n\t 78 Thread-1               420.35    37.35\n\t 78 shutdown-hook-0        419.47    37.11\n\t 39 Timer-0                290.16    23.46\n\t 39 Timer-1                290.16    23.46\n\t 39 Timer-2                290.16    23.46\n\t 39 Thread&#91;Thread-2        290.16    23.44\n\t 39 NodeLabelManage        290.12    23.39\n\t 39 AttributeNodeLa        290.12    23.38\n\t 39 ActivitiesManag        290.11    23.37\n\t 39 ResourceManager        290.09    23.36\n\t 39 SchedulerEventD        290.09    23.36\n\t 39 Timer-3                290.00    23.23\n\t 39 ApplicationMast        289.48    22.86\n\t 39 pool-5-thread-1        287.88    22.16\n\t 39 Thread&#91;Thread-1        287.87    22.15\n\t 39 pool-4-thread-1        287.80    22.10\n\t 24 qtp715289224-35        178.80    14.18\n\t 24 qtp715289224-36        178.79    14.17\n\t 24 qtp715289224-37        178.78    14.17\n\t 24 qtp715289224-38        178.77    14.16\n\t 24 qtp715289224-39        178.77    14.16\n\t 24 qtp715289224-40        178.77    14.16\n\t 24 qtp715289224-41        178.76    14.16\n\t 24 qtp715289224-42        178.75    14.16\n\t 24 qtp715289224-47        178.70    14.15\n\t 24 qtp715289224-48        178.69    14.16\n\t 39 ForkJoinPool-1-        153.67    14.38\n\t 39 Thread-23              149.70    13.89\n\t 39 fsimage_0000000        135.15    12.18\n\t  3 qtp726408598-27         40.72     5.71\n\t  3 qtp726408598-28         40.72     5.71\n\t  3 qtp726408598-29         40.72     5.71\n\t  3 qtp726408598-30         40.72     5.71\n\t  3 qtp726408598-31         40.72     5.71\n\t  3 qtp726408598-32         40.72     5.71\n\t  3 qtp726408598-33         40.72     5.71\n\t  3 qtp726408598-34         40.72     5.71\n\t  3 qtp726408598-37         40.72     5.71\n\t  3 qtp726408598-38         40.72     5.71\n\t 68 clinfo                  19.51     6.65\n\t7720 bash                     2.61     2.24\n\t 38 vulkaninfo               1.52     1.52\n\t312 ssh                      0.55     0.00\n\t  6 glxinfo:gdrv0            0.19     0.01\n\t  6 glxinfo:gl0              0.19     0.01\n\t  4 vulkani:disk$0           0.16     0.16\n\t  6 php                      0.14     0.59\n\t  2 glxinfo                  0.09     0.01\n\t  2 glxinfo:cs0              0.09     0.01\n\t  2 glxinfo:disk$0           0.09     0.01\n\t  2 glxinfo:sh0              0.09     0.01\n\t  2 glxinfo:shlo0            0.09     0.01\n\t  2 llvmpipe-0               0.08     0.08\n\t  2 llvmpipe-1               0.08     0.08\n\t  2 llvmpipe-10              0.08     0.08\n\t  2 llvmpipe-11              0.08     0.08\n\t  2 llvmpipe-12              0.08     0.08\n\t  2 llvmpipe-13              0.08     0.08\n\t  2 llvmpipe-14              0.08     0.08\n\t  2 llvmpipe-15              0.08     0.08\n\t  2 llvmpipe-2               0.08     0.08\n\t  2 llvmpipe-3               0.08     0.08\n\t  2 llvmpipe-4               0.08     0.08\n\t  2 llvmpipe-5               0.08     0.08\n\t  2 llvmpipe-6               0.08     0.08\n\t  2 llvmpipe-7               0.08     0.08\n\t  2 llvmpipe-8               0.08     0.08\n\t  2 llvmpipe-9               0.08     0.08\n\t  6 clang                    0.04     0.08\n\t  3 rocminfo                 0.03     0.00\n\t118 ps                       0.01     0.72\n\t  1 lspci                    0.01     0.02\n\t586 G1 Main Marker           0.01     0.00\n\t586 G1 Young RemSet          0.01     0.00\n\t586 VM Periodic Tas          0.01     0.00\n\t4084 C2 CompilerThre          0.00 16655.19\n\t2431 C1 CompilerThre          0.00  8550.02\n\t586 GC Thread#0              0.00  1859.56\n\t586 G1 Conc#0                0.00  1859.55\n\t586 Reference Handl          0.00  1859.55\n\t586 Service Thread           0.00  1859.53\n\t586 Signal Dispatch          0.00  1859.52\n\t586 Sweeper thread           0.00  1859.47\n\t586 VM Thread                0.00  1859.44\n\t585 GC Thread#1              0.00  1859.38\n\t585 GC Thread#2              0.00  1859.35\n\t585 GC Thread#3              0.00  1859.35\n\t585 GC Thread#4              0.00  1859.34\n\t585 GC Thread#5              0.00  1859.34\n\t585 GC Thread#6              0.00  1859.33\n\t585 GC Thread#7              0.00  1859.33\n\t585 GC Thread#9              0.00  1859.32\n\t585 GC Thread#8              0.00  1859.31\n\t585 process reaper           0.00  1859.30\n\t586 G1 Refine#0              0.00  1856.61\n\t117 G1 Conc#1                0.00  1113.65\n\t117 G1 Conc#2                0.00  1113.63\n\t 78 GC Thread#10             0.00   983.92\n\t 78 GC Thread#11             0.00   983.92\n\t 78 GC Thread#12             0.00   983.92\n\t 78 SSL Certificate          0.00   822.85\n\t 39 GcTimeMonitor o          0.00   693.13\n\t 39 SIGTERM handler          0.00   290.79\n\t 39 Pending Contain          0.00   290.12\n\t 12 G1 Refine#1              0.00   214.25\n\t  9 G1 Refine#2              0.00   157.67\n\t 39 Reconstruction           0.00   153.28\n\t 39 FSImageSaver fo          0.00   124.47\n\t  5 G1 Refine#3              0.00    92.45\n\t  3 G1 Refine#4              0.00    61.56\n\t  3 G1 Refine#5              0.00    61.56\n\t  3 G1 Refine#6              0.00    61.56\n\t  3 G1 Refine#7              0.00    61.56\n\t  3 G1 Refine#8              0.00    61.56\n\t  3 G1 Refine#9              0.00    61.56\n\t  2 G1 Refine#10             0.00    40.91\n\t11272 dirname                  0.00     0.00\n\t6279 IPC Server hand          0.00     0.00\n\t1209 sysctl                   0.00     0.00\n\t975 basename                 0.00     0.00\n\t469 sed                      0.00     0.00\n\t351 rm                       0.00     0.00\n\t273 sleep                    0.00     0.00\n\t195 IPC Server Resp          0.00     0.00\n\t195 IPC Server idle          0.00     0.00\n\t195 IPC Server list          0.00     0.00\n\t195 Ping Checker fo          0.00     0.00\n\t195 Socket Reader #          0.00     0.00\n\t195 mv                       0.00     0.00\n\t158 uname                    0.00     0.00\n\t156 hostname                 0.00     0.00\n\t156 touch                    0.00     0.00\n\t117 cat                      0.00     0.00\n\t 90 sh                       0.00     0.00\n\t 78 cut                      0.00     0.00\n\t 78 hadoop                   0.00     0.00\n\t 78 tr                       0.00     0.00\n\t 39 Block report pr          0.00     0.00\n\t 39 Node Removal Ti          0.00     0.00\n\t 39 RM Event dispat          0.00     0.00\n\t 39 RM StateStore d          0.00     0.00\n\t 39 Timer for 'Name          0.00     0.00\n\t 39 Timer for 'Reso          0.00     0.00\n\t 39 df                       0.00     0.00\n\t 39 renice                   0.00     0.00\n\t 12 gcc                      0.00     0.00\n\t 11 gsettings                0.00     0.00\n\t  8 stat                     0.00     0.00\n\t  8 systemd-detect-          0.00     0.00\n\t  6 llvm-link                0.00     0.00\n\t  5 phoronix-test-s          0.00     0.00\n\t  3 gmain                    0.00     0.00\n\t  2 dconf worker             0.00     0.00\n\t  2 lscpu                    0.00     0.00\n\t  2 which                    0.00     0.00\n\t  2 xset                     0.00     0.00\n\t  1 cc                       0.00     0.00\n\t  1 date                     0.00     0.00\n\t  1 dmesg                    0.00     0.00\n\t  1 dmidecode                0.00     0.00\n\t  1 grep                     0.00     0.00\n\t  1 ifconfig                 0.00     0.00\n\t  1 ip                       0.00     0.00\n\t  1 lsmod                    0.00     0.00\n\t  1 mktemp                   0.00     0.00\n\t  1 qdbus                    0.00     0.00\n\t  1 readlink                 0.00     0.00\n\t  1 realpath                 0.00     0.00\n\t  1 sort                     0.00     0.00\n\t  1 stty                     0.00     0.00\n\t  1 systemctl                0.00     0.00\n\t  1 template.sh              0.00     0.00\n\t  1 wc                       0.00     0.00\n\t  1 xrandr                   0.00     0.00\n1 processes running\n396 maximum processes\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Apache Hadoop using a built-in benchmark. The test itself has multiple configurations, picked one slightly smaller one with operations for open, create, delete, rename and file status. These are done with 50 threads and 100,000 files. These seem to be <span class=\"excerpt-dots\">&hellip;<\/span> <a class=\"more-link\" href=\"https:\/\/mvermeulen.org\/perf\/workloads\/phoronix\/hadoop\/\"><span class=\"more-msg\">Continue reading &rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":58,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-2151","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/2151","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/comments?post=2151"}],"version-history":[{"count":2,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/2151\/revisions"}],"predecessor-version":[{"id":2160,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/2151\/revisions\/2160"}],"up":[{"embeddable":true,"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/pages\/58"}],"wp:attachment":[{"href":"https:\/\/mvermeulen.org\/perf\/wp-json\/wp\/v2\/media?parent=2151"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}