Skip to content

store table data cleaning and standardization #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ritik574-coder merged 1 commit into from
May 22, 2026
Merged

store table data cleaning and standardization #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion explore_database/employees/employees.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -856,7 +856,7 @@ WHERE PATINDEX('%@%@%', TRIM(LOWER(email))) > 0
--############################## EMPLOYEE CLEAN DATA ##########################################
--#############################################################################################

SELECT TOP (1000)
SELECT
[employee_id]

,CASE
Expand Down
6 changes: 3 additions & 3 deletions explore_database/reviews/reviews.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
--#############################################################################################
--#################################### RETURNS DATA ###########################################
--#################################### REVIEW DATA ############################################
--#############################################################################################

--=============================================================================================
--================================== return table overview ====================================
--================================== review table overview ====================================
--=============================================================================================
SELECT
[review_id]
Expand Down Expand Up @@ -435,7 +435,7 @@ SELECT
FROM bronze.reviews ;

--#############################################################################################
--############################## RETURNS CLEAN DATA ###########################################
--############################## REVIEW CLEAN DATA ############################################
--#############################################################################################
SELECT
review_id
Expand Down
243 changes: 241 additions & 2 deletions explore_database/stores/stores.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
SELECT TOP (1000) [store_id]
--#############################################################################################
--#################################### STORES DATA ############################################
--#############################################################################################

--=============================================================================================
--================================== stores table overview ====================================
--=============================================================================================

SELECT TOP (1000)
[store_id]
,[store_name]
,[store_type]
,[address]
Expand All @@ -18,4 +27,234 @@ SELECT TOP (1000) [store_id]
,[is_active]
,[has_parking]
,[has_cafe]
FROM .[bronze].[stores]
FROM .[bronze].[stores]

@coderabbitai coderabbitai Bot May 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read-only check: find leading-dot schema/table references that are likely invalid in T-SQL.
rg -nP --type-add 'sql:*.sql' --type sql '^\s*(FROM|JOIN)\s+\.\[[^]]+\]\.\[[^]]+\]'

Repository: Ritik574-coder/dbt_learning_project

Length of output: 292

Fix invalid T-SQL table reference syntax (FROM .[bronze].[stores]).

The leading dot before [bronze] in FROM .[bronze].[stores] is not valid multipart identifier syntax for SQL Server and can fail execution.

Also present in:

  • explore_database/stores/stores.sql:259 (FROM .[bronze].[stores])
  • explore_database/transactions/transactions.sql:62 (FROM .[bronze].[sales_transactions])
Proposed fix 
-FROM .[bronze].[stores]
+FROM [bronze].[stores]
...
-FROM .[bronze].[stores]
+FROM [bronze].[stores]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
FROM .[bronze].[stores]
FROM [bronze].[stores]
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@explore_database/stores/stores.sql` at line 30, Remove the stray leading dot
from the multipart table references; replace occurrences like "FROM
.[bronze].[stores]" and "FROM .[bronze].[sales_transactions]" with valid
identifiers such as "FROM [bronze].[stores]" (or the full three-part name
"[database].[bronze].[stores]" if a database qualifier is intended) so the T‑SQL
multipart identifier is syntactically correct; update the instances in the
stores SQL and transactions SQL (the lines containing ".\[bronze\].\[stores\]"
and ".\[bronze\].\[sales_transactions\]") accordingly.


--=============================================================================================
--================================== store_id column cleaning =================================
--=============================================================================================
-- store_id data overview
SELECT
store_id
FROM bronze.stores ;

-- store_id data type check
SELECT
store_id
FROM bronze.stores
WHERE TRY_CONVERT(INT, store_id) IS NULL ;

-- store_id data profiling
SELECT
store_id
FROM bronze.stores
WHERE store_id IS NULL
OR store_id < 1 ;

-- store_id duplicate chck
SELECT
*
FROM
(
SELECT
store_id,
ROW_NUMBER() OVER(PARTITION BY store_id ORDER BY store_id DESC) as flag
FROM bronze.stores
)t WHERE flag != 1 ;

--=============================================================================================
--================================== store_name column cleaning ===============================
--=============================================================================================
-- store_name data overview
SELECT
store_name
FROM bronze.stores ;

-- store_name data profiling
SELECT
store_name
FROM bronze.stores
WHERE store_name IS NULL
OR store_name != store_name

@coderabbitai coderabbitai Bot May 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Replace self-comparison predicates in profiling filters.

store_name != store_name and store_type != store_type are ineffective for non-NULL rows, so these checks won’t catch dirty values as intended.

Proposed fix 
-WHERE store_name IS NULL
-   OR store_name != store_name
+WHERE store_name IS NULL
+   OR store_name != TRIM(store_name)
    OR store_name != dbo.TitleCase(store_name)
    OR TRIM(store_name) = ''
    OR LEN(TRIM(store_name)) < 3 ;

-WHERE store_type IS NULL 
-   OR store_type != store_type
+WHERE store_type IS NULL 
+   OR store_type != TRIM(store_type)
    OR store_type != dbo.TitleCase(store_type)
    OR TRIM(store_type) = ''
    OR LEN(TRIM(store_type)) < 3 ;

Also applies to: 121-121

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@explore_database/stores/stores.sql` at line 77, Replace the ineffective
self-comparisons used in the profiling filters — change "store_name !=
store_name" and "store_type != store_type" to explicit NULL/empty checks (e.g.,
"store_name IS NULL OR store_name = ''" and "store_type IS NULL OR store_type =
''") or, if you intend to compare to another column, use "IS DISTINCT FROM";
update occurrences for the symbols store_name and store_type (also replace the
same predicates at the other location referenced around lines 121).

OR store_name != dbo.TitleCase(store_name)
OR TRIM(store_name) = ''
OR LEN(TRIM(store_name)) < 3 ;

-- duplicate check in store_name
SELECT
*
FROM
(
SELECT
store_name,
ROW_NUMBER() OVER(PARTITION BY store_name ORDER BY store_name DESC) as flag
FROM bronze.stores
)t WHERE flag != 1 ;

-- store_name cleaning and standardization
WITH clean_store_name AS
(
SELECT
CASE
WHEN store_name IS NULL OR LEN(TRIM(store_name)) < 3 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(store_name))
END as store_name
FROM bronze.stores
)
SELECT
*
FROM clean_store_name
WHERE store_name = 'Unknown';

--=============================================================================================
--================================== store_type column cleaning ===============================
--=============================================================================================
-- store_type data overview
SELECT
store_type
FROM bronze.stores ;

-- store_type data profiling
SELECT
store_type
FROM bronze.stores
WHERE store_type IS NULL
OR store_type != store_type
OR store_type != dbo.TitleCase(store_type)
OR TRIM(store_type) = ''
OR LEN(TRIM(store_type)) < 3 ;

-- store type count
SELECT
store_type ,
COUNT(*) type_count,
CAST(ROUND(COUNT(*)*100/SUM(COUNT(*)) OVER(), 2)as nvarchar) + '%' as percentages
FROM bronze.stores
GROUP BY store_type
ORDER BY type_count DESC ;

-- store_type cleaning and standardization
WITH clean_store_type AS
(
SELECT
CASE
WHEN store_type IS NULL OR LEN(TRIM(store_type)) < 3 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(store_type))
END as store_type
FROM bronze.stores
)
SELECT
*
FROM clean_store_type
WHERE store_type = 'Unknown';

--=============================================================================================
--================================== address column cleaning ==================================
--=============================================================================================
-- address data overview
SELECT
address
FROM bronze.stores ;

-- address column data profiling
SELECT
address
FROM bronze.stores
WHERE address IS NULL
OR address != TRIM(address)
OR address != TRIM(dbo.TitleCase(address))
OR TRIM(address) = ''
OR LEN(address) < 5 ;


SELECT
address
from bronze.stores
where len(address) = 12


-- address cleaning and standardization
WITH clean_address AS

@chatgpt-codex-connector chatgpt-codex-connector Bot May 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Terminate prior statement before clean_address CTE

Add a statement terminator before this CTE. The preceding debug query (... where len(address) = 12) is not ended with ;, and in SQL Server a WITH CTE in the middle of a batch must follow a terminated statement. As written, this section raises a syntax error near WITH, which prevents the rest of the stores cleaning script (including the final clean-data select) from running.

Useful? React with 👍 / 👎.

(
SELECT
CASE
WHEN address IS NULL OR TRIM(address) = '' OR LEN(address) < 5 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(address))
END address
FROM bronze.stores
)
SELECT
*
FROM clean_address
WHERE address = 'Unknown' ;

--=============================================================================================
--===================================== city column cleaning ==================================
--=============================================================================================
-- city data overview
SELECT
city
FROM bronze.stores ;

--#############################################################################################
--############################## STORES CLEAN DATA ############################################
--#############################################################################################

SELECT
store_id
,store_name
,store_type
,address
,city
,state
,state_full
,zip_code
,country
,region
,district
,phone
,manager_name
,opened_date
,sq_footage
,num_employees
,annual_rent_usd
,is_active
,has_parking
,has_cafe
FROM
(
SELECT
store_id

,CASE
WHEN store_name IS NULL OR LEN(TRIM(store_name)) < 3 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(store_name))
END as store_name

,CASE
WHEN store_type IS NULL OR LEN(TRIM(store_type)) < 3 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(store_type))
END as store_type

,CASE
WHEN address IS NULL OR TRIM(address) = '' OR LEN(address) < 5 THEN 'Unknown'
ELSE TRIM(dbo.TitleCase(address))
END address

,[city]
,[state]
,[state_full]
,[zip_code]
,[country]
,[region]
,[district]
,[phone]
,[manager_name]
,[opened_date]
,[sq_footage]
,[num_employees]
,[annual_rent_usd]
,[is_active]
,[has_parking]
,[has_cafe]
FROM .[bronze].[stores]
)t ;