Skip to content

Commit 4eb49f2

Browse files
daipomdaipom
committed
README: init
Signed-off-by: Daijiro Fukuda <fukuda@clear-code.com>
1 parent 905cbf2 commit 4eb49f2

1 file changed

Lines changed: 262 additions & 13 deletions

File tree

‎README.md‎

Lines changed: 262 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,287 @@
1-
# Fluent::TailChecker
1+
# tailcheck
22

3-
TODO: Delete this and the text below, and describe your gem
3+
`tailcheck` is a command for [Fluentd](https://www.fluentd.org/).
44

5-
Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/fluent/tail_checker`. To experiment with that code, run `bin/console` for an interactive prompt.
5+
This command checks whether [in_tail](https://docs.fluentd.org/input/tail) plugin is collecting logs properly.
6+
For example, this command verifies whether a known critical log missing issue is occurring.
7+
8+
Known critical log missing issues:
9+
10+
* https://github.com/fluent/fluentd/issues/3614
11+
* In case `follow_inodes false` (default setting), collection of a file may stop and continue to stop after log rotation.
12+
* Fixed since Fluentd v1.16.3 (fluent-package v5.0.2, td-agent v4.5.2).
13+
* https://github.com/fluent/fluentd/issues/4190
14+
* In case `follow_inodes true`, collection of a file may stop and continue to stop after log rotation.
15+
* Fixed since Fluentd v1.16.2 (fluent-package v5.0.0, td-agent v4.5.1).
16+
17+
This command allows you to check whether these issues are occurring on your Fluentd.
18+
19+
## Requirements
20+
21+
| tailcheck | fluentd | td-agent | fluent-package |
22+
|--------------|---------|----------|----------------|
23+
| all versions | >= v1.0 | >= 3.1.1 | >= 5.0.0 |
624

725
## Installation
826

9-
TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
27+
### fluent-package
28+
29+
#### RPM/DEB (Linux)
30+
31+
```console
32+
$ sudo fluent-gem install fluent-tail_checker
33+
```
34+
35+
Then, you can use the command as follows.
36+
37+
```console
38+
$ /opt/fluent/bin/tailcheck --help
39+
```
40+
41+
#### .msi (Windows)
42+
43+
`Fluent Package Command Prompt` with Administrator privilege:
44+
45+
```console
46+
$ fluent-gem install fluent-tail_checker
47+
```
48+
49+
Then, you can use the command as follows.
50+
51+
```console
52+
$ tailcheck --help
53+
```
54+
55+
### For td-agent v4
56+
57+
#### RPM/DEB (Linux)
58+
59+
```console
60+
$ sudo td-agent-gem install fluent-tail_checker
61+
```
62+
63+
Then, you can use the command as follows.
64+
65+
```console
66+
$ /opt/td-agent/bin/tailcheck --help
67+
```
68+
69+
#### .msi (Windows)
1070

11-
Install the gem and add to the application's Gemfile by executing:
71+
`Td-agent Command Prompt` with Administrator privilege:
1272

13-
```bash
14-
bundle add UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
73+
```console
74+
$ td-agent-gem install fluent-tail_checker
1575
```
1676

17-
If bundler is not being used to manage dependencies, install the gem by executing:
77+
Then, you can use the command as follows.
1878

19-
```bash
20-
gem install UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
79+
```console
80+
$ tailcheck --help
81+
```
82+
83+
### For td-agent v3
84+
85+
#### RPM/DEB (Linux)
86+
87+
```console
88+
$ sudo td-agent-gem install fluent-tail_checker
89+
```
90+
91+
Then, you can use the command as follows.
92+
93+
```console
94+
$ /opt/td-agent/embedded/lib/ruby/gems/2.4.0/bin/tailcheck --help
95+
```
96+
97+
#### .msi (Windows)
98+
99+
`Td-agent Command Prompt` with Administrator privilege:
100+
101+
```console
102+
$ fluent-gem install fluent-tail_checker
103+
```
104+
105+
Then, you can use the command as follows.
106+
107+
```console
108+
$ tailcheck --help
109+
```
110+
111+
### For your Ruby environment
112+
113+
```console
114+
$ gem install fluent-tail_checker
115+
```
116+
117+
Then, you can use the command as follows.
118+
119+
```console
120+
$ tailcheck --help
21121
```
22122

23123
## Usage
24124

25-
TODO: Write usage instructions here
125+
### Help
126+
127+
```console
128+
$ tailcheck --help
129+
```
130+
131+
### Version
132+
133+
```console
134+
$ tailcheck --version
135+
```
136+
137+
### Check pos files
138+
139+
You can specify a path of a pos file to check:
140+
141+
```console
142+
$ tailcheck /var/log/td-agent/pos/secure
143+
```
144+
145+
You can specify multiple paths:
146+
147+
```console
148+
$ tailcheck /var/log/td-agent/pos/secure /var/log/td-agent/pos/message
149+
```
150+
151+
You can use wildcards:
152+
153+
```console
154+
$ tailcheck /var/log/td-agent/pos/*
155+
```
156+
157+
If you use [follow_inodes](https://docs.fluentd.org/input/tail#follow_inodes), then you must specify `--follow_inodes` option.
158+
(If you have both settings that use `follow_inodes` and those that do not, please run the command separately.)
159+
160+
```console
161+
$ tailcheck --follow_inodes /var/log/td-agent/pos/secure
162+
```
163+
164+
### Result example
165+
166+
#### No anomalies found
167+
168+
```
169+
Check /path/to/pos.
170+
Checked duplicated pos for 2 PosEntries.
171+
Checked collection ratio of 2 files.
172+
173+
All check completed.
174+
There is no anomalies.
175+
```
176+
177+
This means:
178+
179+
* The command has checked `/path/to/pos`.
180+
* The command has checked 2 entries for duplication check.
181+
* The command has checked 2 files for collection ratio check.
182+
* There is no anomalies.
183+
184+
#### Pos duplication found
185+
186+
```
187+
Check /path/to/pos.
188+
Checked duplicated pos for 3 PosEntries.
189+
Duplicated PosEntries are found. This is a known log loss issue that was fixed in Fluentd v1.16.3.
190+
Duplicated keys:
191+
/test/foo.log
192+
Checked collection ratio of 0 files.
193+
194+
All check completed.
195+
Some anomalies are found. Please check whether there is any log loss.
196+
```
197+
198+
In this case, some keys of the pos entries are duplicated.
199+
It is a known log missing issue that was fixed in Fluentd v1.16.3.
200+
201+
> * https://github.com/fluent/fluentd/issues/3614
202+
> * In case `follow_inodes false` (default setting), collection of a file may stop and continue to stop after log rotation.
203+
> * Fixed since Fluentd v1.16.3 (fluent-package v5.0.2, td-agent v4.5.2).
204+
205+
So, you should check whether there is any log missing, and consider updating, especially, if your version is before Fluentd v1.16.3 (fluent-package v5.0.2, td-agent v4.5.2).
206+
207+
#### Too low collection ratio file found
208+
209+
```
210+
Check /path/to/pos.
211+
Checked duplicated pos for 2 PosEntries.
212+
Checked collection ratio of 2 files.
213+
Collection ratio of some files are too low. Collection of those files may have stopped or may not be keeping up.
214+
Filepaths with too low collection ratio (threshold: 0.8):
215+
/test/bar.log (ratio: 0.7)
216+
/test/foo.log (ratio: 0.7)
217+
218+
All check completed.
219+
Some anomalies are found. Please check whether there is any log loss.
220+
```
221+
222+
In this case, collection ratio of some target files are too low.
223+
Collection of those files may have stopped or may not be keeping up.
224+
225+
> /test/bar.log (ratio: 0.7)
226+
227+
This means that only 70% of the data of the file is collected for the filesize.
228+
If it is not keeping up temporarily, then it is no problem.
229+
If this is always the case, or if collection has stopped completely, then log missing may occur.
230+
231+
Especially, if the `in_tail` uses [follow_inodes](https://docs.fluentd.org/input/tail#follow_inodes), there is a known log missing issue that was fixed in Fluentd v1.16.2.
232+
233+
> * https://github.com/fluent/fluentd/issues/4190
234+
> * In case `follow_inodes true`, collection of a file may stop and continue to stop after log rotation.
235+
> * Fixed since Fluentd v1.16.2 (fluent-package v5.0.0, td-agent v4.5.1).
236+
237+
If this issue is occurring, this too low collection ratio is detected.
238+
In that case, please consider updating if your version is before Fluentd v1.16.2 (td-agent v4.5.1).
26239

27240
## Development
28241

29-
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
242+
### Quick start
243+
244+
```console
245+
$ bundle
246+
$ bundle exec exe/tailcheck --version
247+
$ bundle exec exe/tailcheck --help
248+
$ bundle exec exe/tailcheck test/data/pos_duplicate_unwatched_path
249+
```
250+
251+
### Unit test
252+
253+
```console
254+
$ bundle exec rake test
255+
```
256+
257+
### Package test
258+
259+
Need `Vagrant`.
260+
261+
```console
262+
$ vagrant status
263+
$ vagrant up {id}
264+
$ vagrant snapshot save {id} init
265+
$ vagrant ssh {id} -- /vagrant/test/script/{test-script}
266+
$ vagrant snapshot restore {id} init
267+
```
268+
269+
Example:
270+
271+
```console
272+
$ vagrant up centos-7
273+
$ vagrant snapshot save centos-7 init
274+
$ vagrant ssh centos-7 -- /vagrant/test/script/td-agent-v3.1.1.el7-test.bash
275+
$ vagrant snapshot restore centos-7 init
276+
```
277+
278+
### Install and release
30279

31280
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
32281

33282
## Contributing
34283

35-
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/fluent-tail_checker.
284+
Bug reports and pull requests are welcome on GitHub at https://github.com/clear-code/fluent-tail_checker.
36285

37286
## Copyright
38287

0 commit comments

Comments
 (0)