willthefarmer.github.io/index.html at main · WillTheFarmer/willthefarmer.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Apache & NGINX log files to fully-normalized Relational Database</title>
    <meta name="description" content="Parses and Imports HTTP Access & Error logs into MySQL Database designed to normalize data and track log files for unlimited Domains.">
    <meta name="keywords" content="apache, NGINX, log parser, MySQL, MariaDB, Python, ECharts, LogFormat, ErrorLogFormat, Data Normalization, files-to-mysql, mysql-to-echarts">
    <meta name="robots" content="index, follow">
    <meta name="geo.placename" content="Adamsville, NY">
    <meta name="geo.region" content="US-NY">
    <link rel="canonical" href="https://willthefarmer.github.io/" />
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.7.1/css/all.min.css" />
    <link href="css/styles.css" rel="stylesheet" />
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js"></script>
    <link rel="icon" type="image/png" href="assets/farm-cow.png" />
  </head>

  <body>

    <!-- Footer on Top-->
    <footer class="py-2 bg-dark text-center">
      <span class="byline">
        <div class="container">
          <a class="fa fa-home" href="https://farmfreshsoftware.com"></a>&emsp;
          <a class="fab fa-github" href="https://github.com/WillTheFarmer"></a>&emsp;
          <a class="fab fa-youtube" href="https://youtube.com/@FarmFreshSoftware"></a>&emsp;
          <a class="fab fa-x-twitter" href="https://x.com/@FarmFreshNode"></a>&emsp;
          <a class="fas fa-map-marker-alt"
            href="https://www.google.com/maps/place/Farm+Fresh+Software/@43.3335322,-73.4567446,1172m/data=!3m2!1e3!4b1!4m6!3m5!1s0x89e079946b7ce0bd:0xfa195132f9820c1e!8m2!3d43.3335322!4d-73.4567446!16s%2Fg%2F11frs2fr35?authuser=0&hl=en&entry=ttu&g_ep=EgoyMDI0MTIwMy4wIKXMDSoASAFQAw%3D%3D">
          </a>&emsp;
          <a class="fa fa-envelope" href="MAILTO:inquiry@farmfreshsoftware.com?subject=Website Comment"></a>
        </div>
      </span>
    </footer>

    <div class="container">

      <div class="row align-items-center">
        <div class="col-md-3">
          <img src="assets/Apache_Python_MySQL.png" alt="...">
          <img src="assets/Nginx_Logo.png" alt="...">
        </div>

        <div class="col-md-9">
          <h3>Apache & NGINX log files consolidated into a fully Normalized Database</h3>

          <p>Proper Python & NGINX formats released with Factory Method implementation

          </p>

          <h5>Python handles File Processing & MySQL or MariaDB handles Data Processing</h5>

          <p><a href="https://github.com/WillTheFarmer/files-to-mysql">files-to-mysql</a>
            consists of two Python Modules & one Database Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.
          </p>

          <h6>Quickly convert a folder of HTTP log files into a normalized database. Process unlimited files, from multiple domains, from multiple servers with complete Data Lineage. Small codebase & simple setup.</h6>

          <p>Runs on Windows, Linux and MacOS. Database runs on MySQL and MariaDB.

          </p>

          <p>multiple LogFormats & ErrorLogFormats can be loaded and multiple Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

          </p>

          <p><b>IP Geolocation data integration</b> using <a href="https://pypi.org/project/geoip2/">MaxMind GeoIP2</a> Python API provides IP country, subdivision, city, system organization,
            network and coordinates information stored and normalized into 6 Database Schema tables.
          </p>

          <p><b>User-Agent data integration</b> using <a href="https://pypi.org/project/user-agents/">User-Agents</a> provides browser, device and operating system information stored and normalized into 11 Database Schema tables.

          </p>

          <p>Database Schema designed to accommodate unlimited Servers & Domains. Step-by-step guide for easy installation.

          </p>

          <p>Screenshots of Process Messages and <a href="index.html#Benchmarks">Benchmarks</a> for MariaDB version 11.8 and MySQL version 9.2.

          </p>
        </div>
      </div>

      <div class="row align-items-center">
        <div class="col-md-3">
          <img src="assets/MySQL_Node_Express_Apache_Echarts.png" alt="...">
        </div>
        <div class="col-md-9">
          <h3>HTTP Log Management and Visualization</h3>

          <h5>Express handles Data Querying & ECharts handles Data Visualization of MySQL or MariaDB Schema</h5>

          <p><a href="https://github.com/willthefarmer/mysql-to-echarts">mysql-to-echarts</a> is a Web interface that consists of Express.js web application
            frameworks with Drill Down Capability & Apache ECharts frameworks for Log Data Visualization in charts, reports & data analysis
            interfaces of MySQL or MariaDB databases created by <a href="https://github.com/willthefarmer/mysql-http-logs">mysql-http-logs</a>
            and <a href="https://github.com/willthefarmer/mysql-system-logs">mysql-system-logs</a>
            and populated by <a href="https://github.com/willthefarmer/files-to-mysql">files-to-mysql</a>.
          </p>

          <p><a href="https://github.com/WillTheFarmer/mysql-to-echarts">mysql-to-echarts</a>
            has Drill Down Capability & Apache ECharts frameworks for Log Data Visualization in charts, reports & data analysis
            interfaces.
          </p>

          <p>All feedback welcomed!

          </p>

        </div>
      </div>
    </div>

    <div class="text-center">
      <h3>Entity Relationship Diagram of Database Schema tables</h3>

      <p>Database Schema currently has 55 Tables, 1040 Columns, 190 Indexes, 85 Views,
        12 Stored Procedures and 90 Functions to process 6 Access log formats & 3 Error log formats.
         Database normalization at work!
      </p>

      <img src="./assets/entity_relationship_diagram.png" class="img-fluid" alt="...">
    </div>

    <div class="row text-center" id='Benchmarks'>

      <h3>Benchmark Execution Times & Process Messages for HTTP logs - multiple LogFormats, ErrorLogFormats and Stored Procedures processed</h3>

      <div class="col" id='Maria-Benchmark'>
        <a name='Maria-Benchmark'></a>
        <h3>MariaDB version 11.8</h3>
        <img class="mx-auto" style="width: 92%;" src="./assets/benchmark-mariadb-118.png">
      </div>

      <div class="col" id='MySQL-Benchmark'>
        <a name='MySQL-Benchmark'></a>
        <h3>MySQL version 9.2</h3>
        <img class="mx-auto" style="width: 92%;" src="./assets/benchmark-mysql-92.png">
      </div>

      <p>Benchmarks executed on Windows 11 | 13th Gen Intel(R) Core(TM) i7-13700T 1.40 GHz | 96GB DDR5-5600 SODIMM | WD_BLACK 1TB SN850X NVMe SSD - Up to 7,300 MB/s

      </p>

    </div>

  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js" integrity="sha384-YvpcrYf0tY3lHB60NNkmXc5s9fDVZLESaAA55NDzOxhy9GkcIdslK1eN7N6jIeHz" crossorigin="anonymous"></script>
  </body>
</html>