-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathsyslog.txt
More file actions
170 lines (170 loc) · 9.66 KB
/
syslog.txt
File metadata and controls
170 lines (170 loc) · 9.66 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
2021-02-07 06:12:41,073 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-42-178.ec2.internal/172.31.42.178:8032
2021-02-07 06:12:41,150 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-42-178.ec2.internal/172.31.42.178:10200
2021-02-07 06:12:41,185 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-42-178.ec2.internal/172.31.42.178:8032
2021-02-07 06:12:41,185 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-42-178.ec2.internal/172.31.42.178:10200
2021-02-07 06:12:43,710 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-02-07 06:12:43,968 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2021-02-07 06:12:43,980 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 3fb854bbfdabadafad1fa2cca072658fa097fd67]
2021-02-07 06:12:44,021 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 1
2021-02-07 06:12:44,068 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:8
2021-02-07 06:12:44,177 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1612675665477_0007
2021-02-07 06:12:44,397 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2021-02-07 06:12:44,398 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2021-02-07 06:12:44,402 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
2021-02-07 06:12:44,403 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Adding resource type - name = vcores, units = , type = COUNTABLE
2021-02-07 06:12:44,440 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1612675665477_0007
2021-02-07 06:12:44,472 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-42-178.ec2.internal:20888/proxy/application_1612675665477_0007/
2021-02-07 06:12:44,474 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1612675665477_0007
2021-02-07 06:12:50,564 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1612675665477_0007 running in uber mode : false
2021-02-07 06:12:50,565 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-02-07 06:12:59,625 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2021-02-07 06:13:05,654 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 0%
2021-02-07 06:13:06,660 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 0%
2021-02-07 06:13:07,664 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-02-07 06:13:12,690 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 33%
2021-02-07 06:13:13,695 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-02-07 06:13:14,703 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1612675665477_0007 completed successfully
2021-02-07 06:13:14,777 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 55
File System Counters
FILE: Number of bytes read=32807
FILE: Number of bytes written=2487273
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=680
HDFS: Number of bytes written=0
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
S3: Number of bytes read=110064
S3: Number of bytes written=52
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=8
Launched reduce tasks=3
Data-local map tasks=8
Total time spent by all maps in occupied slots (ms)=7391904
Total time spent by all reduces in occupied slots (ms)=3330240
Total time spent by all map tasks (ms)=76999
Total time spent by all reduce tasks (ms)=17345
Total vcore-milliseconds taken by all map tasks=76999
Total vcore-milliseconds taken by all reduce tasks=17345
Total megabyte-milliseconds taken by all map tasks=236540928
Total megabyte-milliseconds taken by all reduce tasks=106567680
Map-Reduce Framework
Map input records=10000
Map output records=10000
Map output bytes=80000
Map output materialized bytes=41177
Input split bytes=680
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=41177
Reduce input records=10000
Reduce output records=4
Spilled Records=20000
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=3720
CPU time spent (ms)=17960
Physical memory (bytes) snapshot=6812106752
Virtual memory (bytes) snapshot=59679277056
Total committed heap usage (bytes)=6206521344
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=110064
File Output Format Counters
Bytes Written=52
2021-02-07 06:13:14,800 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-42-178.ec2.internal/172.31.42.178:8032
2021-02-07 06:13:14,800 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-42-178.ec2.internal/172.31.42.178:10200
2021-02-07 06:13:14,811 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-42-178.ec2.internal/172.31.42.178:8032
2021-02-07 06:13:14,811 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-42-178.ec2.internal/172.31.42.178:10200
2021-02-07 06:13:14,903 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-02-07 06:13:15,065 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 3
2021-02-07 06:13:15,092 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:11
2021-02-07 06:13:15,121 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1612675665477_0008
2021-02-07 06:13:15,139 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1612675665477_0008
2021-02-07 06:13:15,158 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-42-178.ec2.internal:20888/proxy/application_1612675665477_0008/
2021-02-07 06:13:15,158 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1612675665477_0008
2021-02-07 06:13:21,224 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1612675665477_0008 running in uber mode : false
2021-02-07 06:13:21,224 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-02-07 06:13:30,276 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2021-02-07 06:13:36,300 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 0%
2021-02-07 06:13:38,308 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 0%
2021-02-07 06:13:46,335 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-02-07 06:13:47,338 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 33%
2021-02-07 06:13:53,356 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-02-07 06:13:54,363 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1612675665477_0008 completed successfully
2021-02-07 06:13:54,386 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 55
File System Counters
FILE: Number of bytes read=119
FILE: Number of bytes written=3072050
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=979
HDFS: Number of bytes written=0
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
S3: Number of bytes read=112
S3: Number of bytes written=16
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=11
Launched reduce tasks=3
Data-local map tasks=11
Total time spent by all maps in occupied slots (ms)=9805920
Total time spent by all reduces in occupied slots (ms)=3730176
Total time spent by all map tasks (ms)=102145
Total time spent by all reduce tasks (ms)=19428
Total vcore-milliseconds taken by all map tasks=102145
Total vcore-milliseconds taken by all reduce tasks=19428
Total megabyte-milliseconds taken by all map tasks=313789440
Total megabyte-milliseconds taken by all reduce tasks=119365632
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=68
Map output materialized bytes=604
Input split bytes=979
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=604
Reduce input records=4
Reduce output records=1
Spilled Records=8
Shuffled Maps =33
Failed Shuffles=0
Merged Map outputs=33
GC time elapsed (ms)=4797
CPU time spent (ms)=19800
Physical memory (bytes) snapshot=8568197120
Virtual memory (bytes) snapshot=73596473344
Total committed heap usage (bytes)=7629438976
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=112
File Output Format Counters
Bytes Written=16