Skip to content

Commit 29ef75d

Browse files
TocarIPcopybara-github
TocarIP
authored and
copybara-github
committed
Add simd codepath to ascii skip.
We are already using intrinsics and complier generates significantly worse code when I modify code to make it vectorizable. Looking at prod profiles we are averaging ~30 bytes per call, so AVX (32-byte) SIMD is not needed. Makes this ~30% faster for common case. name B/s B/s vs base BM_IsInterchangeValid7BitAsciiX 12.95Gi ± 0% 12.67Gi ± 0% -2.14% (p=0.002 n=6) BM_IsInterchangeValid7BitAsciiTab 3.167Gi ± 0% 3.133Gi ± 0% -1.07% (p=0.002 n=6) BM_IsInterchangeValid7BitAsciiMix 8.820Gi ± 1% 8.763Gi ± 1% -0.64% (p=0.041 n=6) BM_CordIsInterchangeValid 451.3Mi ± 1% 452.5Mi ± 1% ~ (p=0.485 n=6) BM_CordIsStructurallyValid 1.156Gi ± 5% 1.122Gi ± 2% -2.90% (p=0.002 n=6) BM_ToLowerHackString/1 [all_ascii_short] 364.0Mi ± 0% 349.3Mi ± 0% -4.04% (p=0.002 n=6) BM_ToLowerHackString/2 [all_ascii_long ] 1.077Gi ± 1% 1.109Gi ± 1% +2.91% (p=0.002 n=6) BM_ToLowerHackString/3 [UTF8_short ] 97.03Mi ± 3% 98.56Mi ± 0% +1.57% (p=0.002 n=6) BM_ToLowerHackString/4 [UTF8_long ] 286.2Mi ± 1% 292.5Mi ± 2% +2.20% (p=0.015 n=6) BM_ToLowerHackBuf/1 [all_ascii_short] 524.0Mi ± 0% 493.5Mi ± 0% -5.82% (p=0.002 n=6) BM_ToLowerHackBuf/2 [all_ascii_long ] 1.919Gi ± 0% 1.852Gi ± 0% -3.52% (p=0.002 n=6) BM_ToLowerHackBuf/3 [UTF8_short ] 134.2Mi ± 0% 129.1Mi ± 0% -3.80% (p=0.002 n=6) BM_ToLowerHackBuf/4 [UTF8_long ] 373.5Mi ± 0% 371.2Mi ± 2% -0.60% (p=0.015 n=6) BM_CoerceToStructurallyValid/4/0 1.837Gi ± 51% 1.686Gi ± 49% -8.19% (p=0.041 n=6) BM_CoerceToStructurallyValid/4/1 1.869Gi ± 51% 1.687Gi ± 49% -9.77% (p=0.041 n=6) BM_CoerceToStructurallyValid/4/8 1.367Gi ± 61% 1.276Gi ± 34% ~ (p=0.699 n=6) BM_CoerceToStructurallyValid/4/64 240.6Mi ± 70% 291.4Mi ± 64% ~ (p=0.180 n=6) BM_CoerceToStructurallyValid/4/100 230.2Mi ± 42% 256.4Mi ± 39% ~ (p=0.589 n=6) BM_CoerceToStructurallyValid/8/0 1.491Gi ± 95% 2.755Gi ± 50% ~ (p=0.818 n=6) BM_CoerceToStructurallyValid/8/1 2.907Gi ± 54% 1.471Gi ± 87% ~ (p=0.394 n=6) BM_CoerceToStructurallyValid/8/8 1.214Gi ± 140% 1.133Gi ± 143% ~ (p=0.937 n=6) BM_CoerceToStructurallyValid/8/64 303.5Mi ± 38% 286.2Mi ± 36% ~ (p=0.818 n=6) BM_CoerceToStructurallyValid/8/100 237.3Mi ± 23% 222.9Mi ± 28% ~ (p=0.394 n=6) BM_CoerceToStructurallyValid/64/0 4.240Gi ± 5% 4.225Gi ± 7% ~ (p=0.699 n=6) BM_CoerceToStructurallyValid/64/1 3.018Gi ± 40% 3.612Gi ± 33% ~ (p=0.699 n=6) BM_CoerceToStructurallyValid/64/8 1.030Gi ± 16% 1.137Gi ± 35% ~ (p=0.310 n=6) BM_CoerceToStructurallyValid/64/64 261.1Mi ± 16% 224.6Mi ± 19% ~ (p=0.065 n=6) BM_CoerceToStructurallyValid/64/100 198.9Mi ± 18% 182.9Mi ± 14% ~ (p=0.485 n=6) BM_CoerceToStructurallyValid/512/0 5.608Gi ± 3% 5.660Gi ± 2% ~ (p=0.093 n=6) BM_CoerceToStructurallyValid/512/1 3.847Gi ± 15% 3.651Gi ± 26% ~ (p=0.310 n=6) BM_CoerceToStructurallyValid/512/8 1.153Gi ± 17% 1.068Gi ± 37% ~ (p=0.818 n=6) BM_CoerceToStructurallyValid/512/64 235.0Mi ± 4% 221.8Mi ± 5% -5.60% (p=0.009 n=6) BM_CoerceToStructurallyValid/512/100 184.6Mi ± 3% 172.5Mi ± 3% -6.59% (p=0.002 n=6) BM_CoerceToStructurallyValid/4k/0 5.770Gi ± 0% 6.027Gi ± 1% +4.45% (p=0.002 n=6) BM_CoerceToStructurallyValid/4k/1 3.538Gi ± 7% 3.535Gi ± 4% ~ (p=0.589 n=6) BM_CoerceToStructurallyValid/4k/8 1078.9Mi ± 11% 913.8Mi ± 8% -15.31% (p=0.002 n=6) BM_CoerceToStructurallyValid/4k/64 219.1Mi ± 2% 209.7Mi ± 2% -4.29% (p=0.002 n=6) BM_CoerceToStructurallyValid/4k/100 180.4Mi ± 1% 171.6Mi ± 1% -4.92% (p=0.002 n=6) BM_CoerceToStructurallyValid/32k/0 5.512Gi ± 0% 5.797Gi ± 0% +5.17% (p=0.002 n=6) BM_CoerceToStructurallyValid/32k/1 3.137Gi ± 2% 3.092Gi ± 3% ~ (p=0.240 n=6) BM_CoerceToStructurallyValid/32k/8 736.1Mi ± 2% 717.4Mi ± 2% -2.55% (p=0.004 n=6) BM_CoerceToStructurallyValid/32k/64 214.8Mi ± 1% 205.1Mi ± 1% -4.48% (p=0.002 n=6) BM_CoerceToStructurallyValid/32k/100 178.5Mi ± 0% 170.8Mi ± 2% -4.32% (p=0.002 n=6) BM_CoerceToStructurallyValid/256k/0 5.030Gi ± 1% 5.268Gi ± 1% +4.74% (p=0.002 n=6) BM_CoerceToStructurallyValid/256k/1 2.644Gi ± 1% 2.625Gi ± 1% ~ (p=0.240 n=6) BM_CoerceToStructurallyValid/256k/8 688.9Mi ± 0% 677.9Mi ± 0% -1.61% (p=0.002 n=6) BM_CoerceToStructurallyValid/256k/64 212.2Mi ± 0% 203.3Mi ± 0% -4.19% (p=0.002 n=6) BM_CoerceToStructurallyValid/256k/100 177.8Mi ± 1% 169.8Mi ± 0% -4.52% (p=0.002 n=6) BM_CoerceToStructurallyValid/1M/0 5.034Gi ± 0% 5.252Gi ± 1% +4.33% (p=0.002 n=6) BM_CoerceToStructurallyValid/1M/1 2.605Gi ± 0% 2.587Gi ± 1% -0.69% (p=0.026 n=6) BM_CoerceToStructurallyValid/1M/8 689.1Mi ± 0% 675.6Mi ± 0% -1.95% (p=0.002 n=6) BM_CoerceToStructurallyValid/1M/64 212.4Mi ± 1% 203.3Mi ± 0% -4.30% (p=0.002 n=6) BM_CoerceToStructurallyValid/1M/100 177.6Mi ± 0% 169.8Mi ± 0% -4.35% (p=0.002 n=6) BM_UTF8Valid<IsValidUTF8>/64 1.228Gi ± 9% 1.268Gi ± 7% ~ (p=0.394 n=6) BM_UTF8Valid<IsValidUTF8>/512 1.538Gi ± 1% 1.545Gi ± 1% ~ (p=0.310 n=6) BM_UTF8Valid<IsValidUTF8>/4k 1.587Gi ± 0% 1.586Gi ± 0% ~ (p=0.589 n=6) BM_UTF8Valid<IsValidUTF8>/32k 1.595Gi ± 0% 1.592Gi ± 0% -0.22% (p=0.009 n=6) BM_UTF8Valid<IsValidUTF8>/3 287.5Mi ± 13% 272.1Mi ± 16% ~ (p=0.937 n=6) BM_UTF8Valid<IsValidUTF8>/6 485.2Mi ± 15% 514.6Mi ± 14% ~ (p=0.937 n=6) BM_UTF8Valid<IsValidUTF8>/9 914.2Mi ± 3% 914.3Mi ± 6% ~ (p=0.394 n=6) BM_UTF8Valid<IsValidUTF8>/12 1016.8Mi ± 38% 1043.8Mi ± 38% ~ (p=0.937 n=6) BM_UTF8Valid<IsValidUTF8>/15 825.5Mi ± 15% 719.7Mi ± 27% ~ (p=0.589 n=6) BM_UTF8Valid<IsValidUTF8>/18 1107.7Mi ± 15% 1017.7Mi ± 12% ~ (p=0.240 n=6) BM_UTF8Valid<IsValidUTF8>/21 1.242Gi ± 3% 1.234Gi ± 4% ~ (p=0.310 n=6) BM_UTF8Valid<IsValidUTF8>/24 925.0Mi ± 11% 945.7Mi ± 12% ~ (p=0.485 n=6) BM_UTF8Valid<SpanUTF8>/64 1.230Gi ± 6% 1.175Gi ± 8% ~ (p=0.132 n=6) BM_UTF8Valid<SpanUTF8>/512 1.502Gi ± 1% 1.438Gi ± 1% -4.29% (p=0.002 n=6) BM_UTF8Valid<SpanUTF8>/4k 1.554Gi ± 0% 1.492Gi ± 0% -4.01% (p=0.002 n=6) BM_UTF8Valid<SpanUTF8>/32k 1.563Gi ± 0% 1.496Gi ± 0% -4.26% (p=0.002 n=6) BM_UTF8Valid<SpanUTF8>/3 240.3Mi ± 18% 261.5Mi ± 10% ~ (p=0.240 n=6) BM_UTF8Valid<SpanUTF8>/6 365.8Mi ± 28% 439.6Mi ± 5% ~ (p=0.065 n=6) BM_UTF8Valid<SpanUTF8>/9 908.2Mi ± 19% 880.1Mi ± 4% ~ (p=0.093 n=6) BM_UTF8Valid<SpanUTF8>/12 577.2Mi ± 17% 644.3Mi ± 101% +11.63% (p=0.015 n=6) BM_UTF8Valid<SpanUTF8>/15 680.8Mi ± 17% 689.6Mi ± 30% ~ (p=0.180 n=6) BM_UTF8Valid<SpanUTF8>/18 838.3Mi ± 38% 869.0Mi ± 24% ~ (p=0.699 n=6) BM_UTF8Valid<SpanUTF8>/21 1.237Gi ± 7% 1.174Gi ± 4% ~ (p=0.132 n=6) BM_UTF8Valid<SpanUTF8>/24 937.4Mi ± 55% 965.6Mi ± 11% ~ (p=0.589 n=6) BM_FrenchValid<IsValidUTF8>/64 4.750Gi ± 14% 4.369Gi ± 16% ~ (p=0.240 n=6) BM_FrenchValid<IsValidUTF8>/512 5.551Gi ± 2% 5.606Gi ± 4% ~ (p=0.699 n=6) BM_FrenchValid<IsValidUTF8>/4k 5.631Gi ± 1% 5.659Gi ± 1% ~ (p=0.240 n=6) BM_FrenchValid<IsValidUTF8>/32k 5.700Gi ± 0% 5.719Gi ± 0% +0.33% (p=0.002 n=6) BM_FrenchValid<IsValidUTF8>/3 1081.8Mi ± 29% 1014.9Mi ± 17% ~ (p=0.589 n=6) BM_FrenchValid<IsValidUTF8>/6 1.231Gi ± 45% 1.442Gi ± 32% ~ (p=0.818 n=6) BM_FrenchValid<IsValidUTF8>/9 1.078Gi ± 32% 1.081Gi ± 30% ~ (p=0.937 n=6) BM_FrenchValid<IsValidUTF8>/12 1.132Gi ± 285% 1.073Gi ± 265% ~ (p=0.818 n=6) BM_FrenchValid<IsValidUTF8>/15 1.219Gi ± 79% 1.101Gi ± 137% ~ (p=0.132 n=6) BM_FrenchValid<IsValidUTF8>/18 1.744Gi ± 69% 2.757Gi ± 166% ~ (p=0.180 n=6) BM_FrenchValid<IsValidUTF8>/21 2.105Gi ± 61% 2.569Gi ± 112% ~ (p=0.699 n=6) BM_FrenchValid<IsValidUTF8>/24 2.402Gi ± 33% 2.480Gi ± 23% ~ (p=0.818 n=6) BM_FrenchValid<SpanUTF8>/64 4.230Gi ± 14% 3.991Gi ± 22% ~ (p=0.818 n=6) BM_FrenchValid<SpanUTF8>/512 5.474Gi ± 2% 5.079Gi ± 4% -7.23% (p=0.002 n=6) BM_FrenchValid<SpanUTF8>/4k 5.496Gi ± 1% 5.338Gi ± 1% -2.87% (p=0.002 n=6) BM_FrenchValid<SpanUTF8>/32k 5.582Gi ± 0% 5.405Gi ± 1% -3.18% (p=0.002 n=6) BM_FrenchValid<SpanUTF8>/3 825.4Mi ± 21% 989.2Mi ± 40% ~ (p=0.818 n=6) BM_FrenchValid<SpanUTF8>/6 1246.0Mi ± 15% 859.4Mi ± 32% -31.03% (p=0.009 n=6) BM_FrenchValid<SpanUTF8>/9 1.270Gi ± 189% 1.012Gi ± 222% -20.34% (p=0.026 n=6) BM_FrenchValid<SpanUTF8>/12 1342.7Mi ± 99% 1020.0Mi ± 10% -24.03% (p=0.002 n=6) BM_FrenchValid<SpanUTF8>/15 2.043Gi ± 41% 1.468Gi ± 91% ~ (p=0.310 n=6) BM_FrenchValid<SpanUTF8>/18 2.254Gi ± 39% 2.064Gi ± 49% ~ (p=0.699 n=6) BM_FrenchValid<SpanUTF8>/21 2.022Gi ± 13% 1.964Gi ± 28% ~ (p=0.589 n=6) BM_FrenchValid<SpanUTF8>/24 1.962Gi ± 9% 2.094Gi ± 33% ~ (p=0.485 n=6) BM_ASCIIValid<IsValidUTF8>/64 14.95Gi ± 6% 17.43Gi ± 0% +16.56% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/512 18.99Gi ± 1% 30.74Gi ± 1% +61.87% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/4k 18.40Gi ± 0% 27.96Gi ± 0% +51.97% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/32k 19.53Gi ± 2% 30.12Gi ± 0% +54.17% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/3 1.224Gi ± 0% 1.090Gi ± 0% -11.00% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/6 1.783Gi ± 1% 1.636Gi ± 0% -8.25% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/9 4.519Gi ± 0% 3.675Gi ± 0% -18.66% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/12 4.357Gi ± 0% 3.566Gi ± 0% -18.15% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/15 4.085Gi ± 1% 3.505Gi ± 0% -14.21% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/18 6.534Gi ± 0% 7.338Gi ± 0% +12.31% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/21 5.721Gi ± 0% 6.234Gi ± 3% +8.96% (p=0.002 n=6) BM_ASCIIValid<IsValidUTF8>/24 8.714Gi ± 0% 8.707Gi ± 0% ~ (p=0.394 n=6) BM_ASCIIValid<SpanUTF8>/64 14.45Gi ± 3% 14.95Gi ± 0% +3.45% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/512 18.67Gi ± 15% 23.29Gi ± 1% +24.74% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/4k 18.91Gi ± 0% 24.88Gi ± 0% +31.55% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/32k 22.13Gi ± 2% 25.99Gi ± 1% +17.42% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/3 1002.3Mi ± 2% 913.5Mi ± 0% -8.86% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/6 1.509Gi ± 0% 1.402Gi ± 0% -7.10% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/9 3.674Gi ± 0% 2.675Gi ± 0% -27.19% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/12 3.566Gi ± 0% 2.804Gi ± 0% -21.36% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/15 3.309Gi ± 3% 2.888Gi ± 0% -12.71% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/18 5.880Gi ± 1% 5.875Gi ± 0% ~ (p=0.310 n=6) BM_ASCIIValid<SpanUTF8>/21 5.035Gi ± 3% 5.204Gi ± 1% +3.36% (p=0.002 n=6) BM_ASCIIValid<SpanUTF8>/24 8.708Gi ± 0% 7.839Gi ± 0% -9.98% (p=0.002 n=6) geomean 1.517Gi 1.505Gi -0.80% PiperOrigin-RevId: 927448522
1 parent 2f37915 commit 29ef75d

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

‎third_party/utf8_range/utf8_range.c‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,13 @@
2121
#include <stdint.h>
2222
#include <string.h>
2323

24+
#if defined(__SSE4_1__)
25+
#include <emmintrin.h>
26+
#include <smmintrin.h>
27+
#elif defined(__ARM_NEON) && defined(__ARM_64BIT_STATE)
28+
#include <arm_neon.h>
29+
#endif
30+
2431
#if defined(__GNUC__)
2532
#define FORCE_INLINE_ATTR __attribute__((always_inline)) inline
2633
#elif defined(_MSC_VER)
@@ -159,6 +166,20 @@ static inline int utf8_range_CodepointSkipBackwards(int32_t codepoint_word) {
159166
*/
160167
static inline const char* utf8_range_SkipAscii(const char* data,
161168
const char* end) {
169+
#if defined(__SSE4_1__)
170+
__m128i mask = _mm_set1_epi8((char)0x80);
171+
while (16 <= end - data) {
172+
__m128i v = _mm_loadu_si128((const __m128i*)data);
173+
if (!_mm_testz_si128(v, mask)) break;
174+
data += 16;
175+
}
176+
#elif defined(__ARM_NEON) && defined(__ARM_64BIT_STATE)
177+
while (16 <= end - data) {
178+
uint8x16_t v = vld1q_u8((const uint8_t*)data);
179+
if (vmaxvq_u8(v) >= 0x80) break;
180+
data += 16;
181+
}
182+
#endif
162183
while (8 <= end - data &&
163184
(utf8_range_UnalignedLoad64(data) & 0x8080808080808080) == 0) {
164185
data += 8;

0 commit comments

Comments
 (0)