Access analysis (tools)

[kikuomax]

Develop tools to analyze access logs on top of the data warehouse.

• Within a specific time period,
  • Where did the traffic to my site come from?
  • How often did the traffic from a specific origin (referer) happen?
  • How many attentions did a specific page on my site get?

Originally posted by @kikuomax in #30 (comment)

[kikuomax]

We should experiment analysis queries before the free trial credits expire.

[kikuomax]

Findings from preliminary analysis,

• Most of the visitors without the referer
• A lot of internal traffic
• A lot of suspicious visitors trying to exploit misconfigurations

[kikuomax]

Since running queries on Redshift is expensive, I am going to save analysis results somewhere. Where am I going to store analysis results?

• In CSV files in an S3 bucket?
• In DynamoDB table?

[kikuomax]

DynamoDB may be preferable because it can export items to CSV files.

[kikuomax]

DynamoDB may be preferable because it can export items to CSV files.

But we still have to transform data to get desired representations.