You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+33-22Lines changed: 33 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,11 +3,10 @@
3
3
Most strings online are in unicode using the UTF-8 encoding. Validating strings
4
4
quickly before accepting them is important.
5
5
6
-
7
-
8
6
## How to use is_utf8
9
7
10
-
This is a simple one-source file library to validate UTF-8 strings at high speeds using SIMD instructions. It works on all platforms (ARM, x64).
8
+
This is a simple one-source file library to validate UTF-8 strings at high
9
+
speeds using SIMD instructions. It works on all platforms (ARM, x64).
11
10
12
11
Build and link `is_utf8.cpp` with your project. Code usage:
13
12
@@ -20,30 +19,43 @@ Build and link `is_utf8.cpp` with your project. Code usage:
20
19
21
20
It should be able to validate strings using less than 1 cycle per input byte.
22
21
23
-
Real-word usage
24
-
------
25
-
26
-
This C++ library is part of the JavaScript package [utf-8-validate](https://github.com/websockets/utf-8-validate). The utf-8-validate package is routinely downloaded more than [a million times per week](https://www.npmjs.com/package/utf-8-validate).
22
+
## Real-word usage
27
23
28
-
If you are using Node JS (19.4.0 or better), you already have access to this function as `buffer.isUtf8(...)`.
24
+
This C++ library is part of the JavaScript package
25
+
[utf-8-validate](https://github.com/websockets/utf-8-validate). The
26
+
utf-8-validate package is routinely downloaded more than
27
+
[a million times per week](https://www.npmjs.com/package/utf-8-validate).
29
28
29
+
If you are using Node JS (19.4.0 or better), you already have access to this
30
+
function as `buffer.isUtf8(...)`.
30
31
32
+
## Requirements
31
33
32
-
Requirements
33
-
-------
34
-
35
-
- C++11 compatible compiler. We support LLVM clang, GCC, Visual Studio. (Our optional benchmark tool requires C++17.)
34
+
- C++11 compatible compiler. We support LLVM clang, GCC, Visual Studio. (Our
35
+
optional benchmark tool requires C++17.)
36
36
- For high speed, you should have a recent 64-bit system (e.g., ARM or x64).
37
37
- If you rely on CMake, you should use a recent CMake (at least 3.15).
38
-
- AVX-512 support require a processor with AVX512-VBMI2 (Ice Lake or better) and a recent compiler (GCC 8 or better, Visual Studio 2019 or better, LLVM clang 6 or better). You need a correspondingly recent assembler such as gas (2.30+) or nasm (2.14+): recent compilers usually come with recent assemblers. If you mix a recent compiler with an incompatible/old assembler (e.g., when using a recent compiler with an old Linux distribution), you may get errors at build time because the compiler produces instructions that the assembler does not recognize: you should update your assembler to match your compiler (e.g., upgrade binutils to version 2.30 or better under Linux) or use an older compiler matching the capabilities of your assembler.
38
+
- AVX-512 support require a processor with AVX512-VBMI2 (Ice Lake or better) and
39
+
a recent compiler (GCC 8 or better, Visual Studio 2019 or better, LLVM clang 6
40
+
or better). You need a correspondingly recent assembler such as gas (2.30+) or
41
+
nasm (2.14+): recent compilers usually come with recent assemblers. If you mix
42
+
a recent compiler with an incompatible/old assembler (e.g., when using a
43
+
recent compiler with an old Linux distribution), you may get errors at build
44
+
time because the compiler produces instructions that the assembler does not
45
+
recognize: you should update your assembler to match your compiler (e.g.,
46
+
upgrade binutils to version 2.30 or better under Linux) or use an older
47
+
compiler matching the capabilities of your assembler.
39
48
40
49
## Reference
41
50
42
-
- John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021
51
+
- John Keiser, Daniel Lemire,
52
+
[Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090),
53
+
Software: Practice & Experience 51 (5), 2021
43
54
44
-
###Want more?
55
+
## Want more?
45
56
46
-
If you want a wide range of fast Unicode function for production use, you can rely on the simdutf library. It is as simple as the following:
57
+
If you want a wide range of fast Unicode function for production use, you can
58
+
rely on the simdutf library. It is as simple as the following:
47
59
48
60
```C++
49
61
#include"simdutf.cpp"
@@ -64,12 +76,11 @@ int main(int argc, char *argv[]) {
64
76
65
77
See https://github.com/simdutf/
66
78
67
-
68
79
## License
69
80
70
-
This library is distributed under the terms of any of the following
71
-
licenses, at your option:
81
+
This library is distributed under the terms of any of the following licenses, at
0 commit comments