Benchmark: TurboPFor Integer Compression  - Skylake i7-6700 3.4GHz

TurboPFor: IcApp Integer Compression Benchmark - Skylake i7-6700 3.4GHz

<img src="https://5a71de9c-a-62cb3a1a-s-sites.googlegroups.com/site/powturbo/skylake.JPG" />


[Inverted index testdata](https://github.com/ot/partitioned_elias_fano/tree/master/test/test_data)

docs: Document Ids

		./icapp test_collection.docs

file: max bits histogram:
00: 0.002%
01:## 1.7%
02:# 0.7%
03: 0.4%
04: 0.2%
05: 0.2%
06:# 0.5%
07:# 1.4%
08:## 1.6%
09:#### 3.6%
10:###### 6.4%
11:######## 8.1%
12:################# 17%
13:############################################# 45%
14:############# 13%
<pre>
 E MB/s size ratio D MB/s function (integer size=32 bits) BOLD=Pareto
 1038.82 3017800 21.93% 2661.63 vszenc32 TurboVSimple zigzag
 1139.82 3147043 22.87% 4950.83 p4nzenc128v32 TurboPForV zigzag
 1076.60 3147043 22.87% 3642.05 p4nzenc32 TurboPFor zigzag
 964.09 3186374 23.15% 2922.15 p4nzzenc128v32 TurboPFor zzag/delta
 887.67 3190622 23.18% 1954.18 bitshuffleZ+lz Transpose+zigzag+lz
 1088.78 3228329 23.46% 2693.93 tpnibbleZ+lz Transpose+zigzag+lz
 1122.80 3239377 23.54% 5056.32 p4nd1enc128v32 TurboPForV delta1
 1075.09 3239377 23.54% 3680.03 p4nd1enc32 TurboPFor delta1
 1331.20 3246518 23.59% 6489.07 p4nzenc256v32 TurboPFor256 zigzag
 1034.37 3277695 23.81% 2621.08 tpbyteZ+lz Transpose+zigzag+lz
 1135.96 3357179 24.39% 5334.62 p4ndenc128v32 TurboPForV delta
 1075.59 3357179 24.39% 3644.94 p4ndenc32 TurboPFor delta
 1305.82 3423417 24.87% 6158.08 p4nd1enc256v32 TurboPFor256 delta1
 1346.84 3515568 25.54% 1550.80 vbddenc32 TurboVByte zzag delt
 1311.54 3528896 25.64% 6330.87 p4ndenc256v32 TurboPFor256 delta
 920.32 3531916 25.66% 1058.47 bvzzenc32 bitio zigzag/delta
 1094.85 3673776 26.69% 3244.53 tpnibble+lz Transpose+lz
 1079.98 3676707 26.71% 3193.34 tpbyte+lz Transpose+lz
 981.41 3701682 26.90% 2531.42 tpnibbleX+lz Transpose+xor+lz
 983.37 3738695 27.16% 2506.98 tpbyteX+lz Transpose+xor+lz
 8699.94 3925193 28.52% 11144.38 v8nd1enc128v32 TByte+TPackV delta1
 9571.15 3945921 28.67% 11644.09 v8ndenc128v32 TByte+TPackV delta
 2644.76 3948726 28.69% 2783.84 vbzenc32 TurboVByte zigzag
 7443.65 4014799 29.17% 9577.81 v8nzenc128v32 TByte+TPackV zigzag
11010.65 4173952 30.33% 12030.87 v8ndenc256v32 TByte+TPackV delta
10286.48 4161012 30.23% 11723.43 v8nd1enc256v32 TByte+TPackV delta1
 3054.44 4180490 30.37% 3489.68 vbd1enc32 TurboVByte delta1
 3230.82 4181163 30.38% 3716.80 vbdenc32 TurboVByte delta
 8755.29 4215033 30.63% 10769.41 v8nzenc256v32 TByte+TPackV zigzag
 867.36 4228629 30.72% 1893.95 bitshuffleX+lz Transpose+xor+lz
 970.41 4260129 30.95% 2366.86 bitshuffle+lz Transpose+lz
 1062.80 4329918 31.46% 1271.79 bvzenc32 bitio zigzag
 8738.61 4406850 32.02% 10031.57 bitnzpack128v32 TurboPackV zigzag
 4255.82 4406850 32.02% 5017.61 bitnzpack32 TurboPack zigzag
11063.76 4724718 34.33% 10735.81 v8zenc32 TurboByte zigzag
 3235.38 4724718 34.33% 4183.38 streamvbyte zzag StreamVByte zigzag
11926.61 4819041 35.01% 11644.09 bitnzpack256v32 TurboPack256 zigzag
12041.39 4889190 35.52% 12223.19 v8d1enc32 TurboByte delta1
12661.74 4889479 35.53% 12850.90 v8denc32 TurboByte delta
10490.33 4889479 35.53% 14101.75 streamvbyte delt StreamVByte delta
 385.16 5050551 36.70% 423.20 SPDP SPDP Floating Point
 1931.69 5265005 38.25% 8086.55 fpxenc32 TurboFloat XOR
 1350.93 5265005 38.25% 1524.01 fpfcmenc32 TurboFloat FCM
 997.49 5310026 38.58% 1184.65 fpgenc32 bitio TurboGorilla
 1657.63 5552401 40.34% 11926.61 p4nenc128v32 TurboPForV
 1578.90 5552401 40.34% 7583.09 p4nenc32 TurboPFor
 1883.32 5626136 40.88% 7616.66 FastPFor FastPFor
 45.75 5626576 40.88% 4884.07 SimdOptPFor FastPFor SIMD
 2247.07 5626832 40.88% 13818.59 SimdFastPFor FastPFor SIMD
 1903.38 5638279 40.97% 13045.79 p4nenc256v32 TurboPFor256
 8291.15 5653941 41.08% 15343.71 v8nenc128v32 TurboByte+TbPackV
18400.15 5661003 41.13% 15275.60 bitnpack128v32 TurboPackV
10458.44 5661003 41.13% 10263.47 bitnpack32 TurboPack
 9331.06 5745570 41.75% 15429.72 v8nenc256v32 TurboByte+TbPackV
19115.71 5748842 41.77% 15464.40 bitnpack256v32 TurboPack256
 2062.54 6226583 45.24% 6671.50 vsenc32 TurboVSimple
 777.85 6577162 47.79% 1149.34 fpdfcmenc32 TurboFloat DFCM
 784.99 6658332 48.38% 1150.49 fp2dfcmenc32 TurboFloat DFCM 2D
 2628.09 6682141 48.55% 4124.46 vbenc32 TurboVByte scalar
 3850.95 6705452 48.72% 6507.48 maskeydvbyte MasedVByte SIMD
10949.33 6790511 49.34% 11527.06 bitnd1pack128v32 TurboPackV delta1
 6674.74 6790511 49.34% 8212.00 bitnd1pack32 TurboPack delta1
12169.15 6816223 49.52% 11854.70 bitndpack128v32 TurboPackV delta
 7004.23 6816223 49.52% 9001.51 bitndpack32 TurboPack delta
11498.17 7511530 54.58% 14442.09 v8enc32 TurboByte SIMD
10769.41 7511530 54.58% 14321.86 streamvbyte StreamVByte SIMD
13681.22 8062094 58.58% 10710.75 bitnd1pack256v32 TurboPack256 delta1
14262.50 8076334 58.68% 10854.35 bitndpack256v32 TurboPack256 delta
 475.40 9044107 65.71% 2907.33 lz lz
 1427.58 10833105 78.71% 5124.09 srlez32 TurboRLE32 ESC zzag
 1681.94 13712109 99.63% 5869.22 srlex32 TurboRLE32 ESC xor
 1869.00 13763312 100.00% 17532.88 srle32 TurboRLE32 ESC
 298.29 13763312 100.00% 17532.88 trle TurboRLE
 216.18 13763312 100.00% 17532.88 trlex TurboRLE xor
 200.00 13763312 100.00% 17532.88 trlez TurboRLE zigzag
17334.15 13763312 100.00% 17510.58 memcpy memcpy
 9151.14 13763312 100.00% 11595.04 tpenc Byte transpose
 8891.03 13763312 100.00% 11054.87 tp4enc Nibble transpose
 4135.61 13763312 100.00% 4112.13 bitshuffle Bit transpose
 * : external library
</pre>

freqs: Term Frequencies

		./icapp test_collection.freqs

file: max bits histogram:
00: 0.001%
01:########################### 27%
02:############ 12%
03:###### 6.0%
04:### 3.1%
05:## 1.6%
06:# 1.0%
07:# 1.0%
08:# 0.9%
09:## 1.9%
10:### 3.2%
11:#### 4.1%
12:######## 8.5%
13:####################### 23%
14:###### 6.5%
<pre>
 E MB/s size ratio D MB/s function (integer size=32 bits)
 91.61 1692992 12.30% 4615.46 SimdOptPFor FastPFor SIMD
 1204.45 1722849 12.52% 4061.17 vsenc32 TurboVSimple
 1402.56 1730336 12.57% 8397.38 p4nenc128v32 TurboPForV
 1339.76 1730336 12.57% 4979.49 p4nenc32 TurboPFor
 1343.42 1778796 12.92% 6106.17 FastPFor FastPFor
 1447.85 1779168 12.93% 12169.15 SimdFastPFor FastPFor SIMD
 1662.23 1780534 12.94% 12603.76 p4nenc256v32 TurboPFor256
 1701.06 1907713 13.86% 3908.92 tpnibble+lz Transpose+lz
 1084.24 1922544 13.97% 2792.88 vszenc32 TurboVSimple zigzag
 1435.32 1955549 14.21% 2904.88 tpnibbleX+lz Transpose+xor+lz
 1023.67 1958057 14.23% 2334.74 bitshuffleX+lz Transpose+xor+lz
 1194.83 1962095 14.26% 3022.91 bitshuffle+lz Transpose+lz
 1185.37 1996059 14.50% 5303.78 p4nzenc128v32 TurboPForV zigzag
 1126.29 1996059 14.50% 3584.19 p4nzenc32 TurboPFor zigzag
 1364.32 2087485 15.17% 7105.47 p4nzenc256v32 TurboPFor256 zigzag
 1081.60 2130744 15.48% 4695.77 fpxenc32 TurboFloat XOR
 877.76 2130744 15.48% 1330.30 fpfcmenc32 TurboFloat FCM
 1326.33 2145784 15.59% 2916.57 tpnibbleZ+lz Transpose+zigzag+lz
 1293.42 2219153 16.12% 3770.77 tpbyte+lz Transpose+lz
 1015.97 2259515 16.42% 1055.63 bvzenc32 bitio zigzag
 1003.67 2300817 16.72% 2364.83 bitshuffleZ+lz Transpose+zigzag+lz
 1128.69 2381528 17.30% 2856.05 tpbyteX+lz Transpose+xor+lz
 1021.17 2398296 17.43% 2984.24 p4nzzenc128v32 TurboPFor zzag/delta
 1137.28 2407925 17.50% 2936.48 tpbyteZ+lz Transpose+zigzag+lz
 1004.84 2764046 20.08% 1203.82 bvzzenc32 bitio zigzag/delta
16483.00 2784387 20.23% 17312.33 v8nenc128v32 TurboByte+TbPackV
17488.32 2804702 20.38% 17118.54 bitnpack128v32 TurboPackV
 9268.22 2804702 20.38% 10522.40 bitnpack32 TurboPack
 1670.91 2883341 20.95% 1931.15 vbddenc32 TurboVByte zazg delt
 1110.66 3029119 22.01% 1114.80 fpgenc32 bitio TurboGorilla
 8271.22 3146929 22.86% 10202.60 v8nzenc128v32 TByte+TPackV zigzag
20948.71 3150687 22.89% 16764.07 v8nenc256v32 TurboByte+TbPackV
22824.72 3189005 23.17% 16764.07 bitnpack256v32 TurboPack256
 8591.33 3189535 23.17% 10340.57 bitnzpack128v32 TurboPackV zigzag
 4135.61 3189535 23.17% 5150.94 bitnzpack32 TurboPack zigzag
11546.40 3451880 25.08% 11365.24 vbenc32 TurboVByte scalar
 8661.61 3457754 25.12% 16846.15 maskeydvbyte MasedVByte SIMD
 5340.82 3475286 25.25% 5265.23 vbzenc32 TurboVByte zigzag
10611.65 3502875 25.45% 12959.80 v8nzenc256v32 TByte+TPackV zigzag
11733.42 3589870 26.08% 13349.47 bitnzpack256v32 TurboPack256 zigzag
 810.18 3925934 28.52% 3365.11 lz lz
14087.31 4307847 31.30% 16562.34 v8enc32 TurboByte SIMD
11517.41 4307847 31.30% 16825.56 streamvbyte StreamVByte SIMD
11099.44 4323652 31.41% 10744.19 v8zenc32 TurboByte zigzag
 4017.31 4323652 31.41% 4233.56 streamvbyte zzag StreamVByte zigzag
 1048.79 5742202 41.72% 5785.33 p4ndenc128v32 TurboPForV delta
 1001.91 5742202 41.72% 3535.40 p4ndenc32 TurboPFor delta
 1258.30 5773355 41.95% 7098.15 p4ndenc256v32 TurboPFor256 delta
11793.75 7227089 52.51% 12432.98 v8denc32 TurboByte delta
10271.12 7227089 52.51% 12062.49 streamvbyte delt StreamVByte delta
 8705.44 7231250 52.54% 11703.49 v8ndenc256v32 TByte+TPackV delta
 8077.06 7234307 52.56% 10949.33 v8ndenc128v32 TByte+TPackV delta
 1408.88 7343213 53.35% 1508.14 vbdenc32 TurboVByte delta
 1277.10 10319134 74.98% 5386.81 srle32 TurboRLE32 ESC
 735.30 10350497 75.20% 1170.55 fpdfcmenc32 TurboFloat DFCM
 731.12 10690707 77.68% 1161.46 fp2dfcmenc32 TurboFloat DFCM 2D
 1124.82 10963622 79.66% 4061.17 srlex32 TurboRLE32 ESC xor
 853.54 10981857 79.79% 4564.94 p4nd1enc128v32 TurboPForV delta1
 840.30 10981857 79.79% 3156.72 p4nd1enc32 TurboPFor delta1
 1024.06 11014587 80.03% 5551.96 p4nd1enc256v32 TurboPFor256 delta1
 1348.15 11113500 80.75% 4623.21 srlez32 TurboRLE32 ESC zzag
 169.10 11621965 84.44% 191.26 SPDP SPDP Floating Point
 7451.71 11667495 84.77% 9261.98 v8nd1enc128v32 TByte+TPackV delta1
 8029.93 11691692 84.95% 9712.99 v8nd1enc256v32 TByte+TPackV delta1
10016.96 11716644 85.13% 10434.65 v8d1enc32 TurboByte delta1
 1137.46 13329244 96.85% 1150.97 vbd1enc32 TurboVByte delta1
11614.60 13727224 99.74% 10458.44 bitndpack128v32 TurboPackV delta
 8464.52 13727224 99.74% 9054.81 bitndpack32 TurboPack delta
11527.06 13747047 99.88% 9465.82 bitndpack256v32 TurboPack256 delta
16542.43 13763304 100.00% 16116.28 memcpy memcpy
 412.58 13763304 100.00% 16116.28 trle TurboRLE
 246.88 13763304 100.00% 16116.28 trlez TurboRLE zigzag
 361.16 13763304 100.00% 16097.43 trlex TurboRLE xor
10719.08 13763304 100.00% 11663.82 tpenc Byte transpose
 8931.41 13763304 100.00% 11001.84 tp4enc Nibble transpose
 4247.93 13763304 100.00% 5147.08 bitshuffle Bit transpose
11693.55 13776743 100.10% 8954.65 bitnd1pack256v32 TurboPack256 delta1
11374.63 13790184 100.20% 10450.50 bitnd1pack128v32 TurboPackV delta1
 8251.38 13790184 100.20% 8925.62 bitnd1pack32 TurboPack delta1
</pre>
Synthetic data: zipfian distribution

		./icapp -a1.5 -m0 -M255 -n100M ZIPF

 bits histogram:
00:######################################## 40%
01:############## 14%
02:############# 13%
03:########## 10%
04:######## 7.8%
05:###### 5.7%
06:#### 4.1%
07:### 2.9%
08:## 2.1%
<pre>
 E MB/s size ratio D MB/s function (integer size=32 bits)
 2368.80 62939886 15.73% 10950.80 p4nenc256v32 TurboPFor256
 1327.02 63392759 15.85% 7702.23 p4nenc128v32 TurboPForV
 1321.43 63392759 15.85% 4556.85 p4nenc32 TurboPFor
 66.39 65060504 16.27% 3077.49 SimdOptPFor FastPFor SIMD
 606.96 73459928 18.36% 5349.24 FastPFor FastPFor
 631.90 73469416 18.37% 10197.58 SimdFastPFor FastPFor SIMD
 1010.83 76345141 19.09% 2866.89 vsenc32 TurboVSimple
 1745.84 79163645 19.79% 3332.50 tpnibble+lz Transpose+lz
 1464.36 80509600 20.13% 2494.28 tpnibbleX+lz Transpose+xor+lz
 1097.09 82870974 20.72% 2541.51 bitshuffle+lz Transpose+lz
 975.88 83384370 20.85% 2073.53 bitshuffleX+lz Transpose+xor+lz
 1321.55 85243365 21.31% 6370.95 p4nzenc256v32 TurboPFor256 zigzag
 1178.72 85546946 21.39% 4991.20 p4nzenc128v32 TurboPForV zigzag
 1104.59 85546946 21.39% 3735.66 p4nzenc32 TurboPFor zigzag
 1334.30 88589767 22.15% 2936.28 tpbyte+lz Transpose+lz
 902.66 89283071 22.32% 2240.92 vszenc32 TurboVSimple zigzag
 1010.73 94657651 23.66% 4570.28 fpxenc32 TurboFloat XOR
 823.29 94657651 23.66% 1348.86 fpfcmenc32 TurboFloat FCM
 1284.95 95923933 23.98% 2332.20 tpbyteX+lz Transpose+xor+lz
 1376.65 97552451 24.39% 2321.01 tpnibbleZ+lz Transpose+zigzag+lz
 1011.41 98127316 24.53% 2129.66 bitshuffleZ+lz Transpose+zigzag+lz
 888.50 98371167 24.59% 926.12 bvzenc32 bitio zigzag
17298.05 99910930 24.98% 12408.10 v8nenc128v32 TurboByte+TbPackV
17356.59 99910930 24.98% 12362.47 bitnpack128v32 TurboPackV
11693.51 99910930 24.98% 10137.62 bitnpack32 TurboPack
17057.57 100332929 25.08% 11994.72 v8nenc256v32 TurboByte+TbPackV
17076.50 100332929 25.08% 11170.06 bitnpack256v32 TurboPack256
 1013.86 100628462 25.16% 3083.11 p4nzzenc128v32 TurboPFor zzag/delta
11191.32 101015650 25.25% 10332.45 vbenc32 TurboVByte scalar
 6689.75 102074663 25.52% 9524.04 maskeydvbyte MasedVByte SIMD
 1161.18 103144618 25.79% 1263.64 fpgenc32 bitio TurboGorilla
 1164.71 105990559 26.50% 2202.97 tpbyteZ+lz Transpose+zigzag+lz
 3857.24 106284616 26.57% 4009.10 vbzenc32 TurboVByte zigzag
 9820.77 112368050 28.09% 10469.01 bitnzpack128v32 TurboPackV zigzag
 9686.17 112368050 28.09% 10426.44 v8nzenc128v32 TByte+TPackV zigzag
 4189.84 112368050 28.09% 5317.24 bitnzpack32 TurboPack zigzag
12560.05 112825409 28.21% 11054.92 v8nzenc256v32 TByte+TPackV zigzag
12680.30 112825409 28.21% 11036.61 bitnzpack256v32 TurboPack256 zigzag
 1650.88 116367689 29.09% 1882.50 vbddenc32 TurboVByte zazg delt
 832.48 119294130 29.82% 973.89 bvzzenc32 bitio zigzag/delta
13107.45 125000000 31.25% 12375.09 v8enc32 TurboByte SIMD
11186.62 125000000 31.25% 12123.78 streamvbyte StreamVByte SIMD
10956.20 128705458 32.18% 10696.90 v8zenc32 TurboByte zigzag
 3208.47 128705458 32.18% 3850.37 streamvbyte zzag StreamVByte zigzag
 656.49 140353625 35.09% 2416.76 lz lz
 1002.48 231440944 57.86% 5208.47 p4ndenc128v32 TurboPForV delta
 959.54 231440944 57.86% 3770.10 p4ndenc32 TurboPFor delta
 1180.67 231486347 57.87% 5985.96 p4ndenc256v32 TurboPFor256 delta
 488.03 239203583 59.80% 2701.28 trle TurboRLE
10108.16 245851337 61.46% 10481.08 v8denc32 TurboByte delta
 9441.09 245851337 61.46% 10096.17 streamvbyte delt StreamVByte delta
 8420.70 246241959 61.56% 10398.52 v8ndenc256v32 TByte+TPackV delta
 7895.00 246632587 61.66% 9510.45 v8ndenc128v32 TByte+TPackV delta
 1142.53 262025557 65.51% 1198.36 vbdenc32 TurboVByte delta
 436.08 263233919 65.81% 1532.21 trlex TurboRLE xor
 315.02 263233919 65.81% 1053.77 trlez TurboRLE zigzag
 1085.73 291716271 72.93% 5591.59 p4nd1enc256v32 TurboPFor256 delta1
 918.79 291995716 73.00% 4686.75 p4nd1enc128v32 TurboPForV delta1
 897.86 291995716 73.00% 3595.60 p4nd1enc32 TurboPFor delta1
 9353.88 304132355 76.03% 9687.11 v8d1enc32 TurboByte delta1
 7881.46 304522974 76.13% 8913.05 v8nd1enc256v32 TByte+TPackV delta1
 7441.72 304913602 76.23% 8818.73 v8nd1enc128v32 TByte+TPackV delta1
 1026.88 339718007 84.93% 1063.92 vbd1enc32 TurboVByte delta1
 168.67 374829697 93.71% 189.64 SPDP SPDP Floating Point
 1374.41 384560608 96.14% 9134.30 srle32 TurboRLE32 ESC
 709.62 387715234 96.93% 1186.37 fpdfcmenc32 TurboFloat DFCM
 710.10 388257392 97.06% 1175.15 fp2dfcmenc32 TurboFloat DFCM 2D
 1174.42 392073698 98.02% 5061.88 srlex32 TurboRLE32 ESC xor
 1433.31 393849411 98.46% 5286.18 srlez32 TurboRLE32 ESC zzag
14002.66 400000000 100.00% 14043.46 memcpy memcpy
 3876.57 400000000 100.00% 3789.60 bitshuffle Bit transpose
 8518.79 400000000 100.00% 9106.43 tpenc Byte transpose
 8304.44 400000000 100.00% 9598.08 tp4enc Nibble transpose
 9278.16 400390622 100.10% 7273.65 bitnd1pack256v32 TurboPack256 delta1
 9303.19 400390622 100.10% 7575.18 bitndpack256v32 TurboPack256 delta
 7950.55 400781247 100.20% 7644.97 bitnd1pack32 TurboPack delta1
 8069.89 400781247 100.20% 7650.09 bitndpack32 TurboPack delta
 9284.62 400781247 100.20% 8515.17 bitnd1pack128v32 TurboPackV delta1
 9294.98 400781247 100.20% 8564.58 bitndpack128v32 TurboPackV delta
</pre>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz #47

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz #47

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions