Skip to content

PUBSUB_ISSUES_RESOLVED.md

Latest commit

 

History

History
156 lines (122 loc) · 4.95 KB

PUBSUB_ISSUES_RESOLVED.md

File metadata and controls

156 lines (122 loc) · 4.95 KB

Pub/Sub Issues - RESOLVED ✅

Problem Summary

The orchestrator service was failing with the error:

❌ Error publishing to orchestrator-topic: Error: 5 NOT_FOUND: Topic not found

This was happening because:

  1. Missing Topics: The required Pub/Sub topics didn't exist in the Firebase emulator
  2. Wrong Connection: Services were trying to connect to production Google Cloud Pub/Sub instead of the local emulator
  3. Missing Configuration: No automatic emulator detection in the service code

Root Causes Identified

1. Missing Pub/Sub Topics

  • orchestrator-topic (the main one causing the error)
  • category-extraction-topic
  • product-extraction-topic
  • product-enrichment-topic
  • sustainability-enrichment-topic
  • ai-logging-topic

2. Services Not Using Emulator

The following services were hardcoded to use production Pub/Sub:

  • orchestrator-service/pubsub-handler.js
  • orchestrator-service/index.js
  • category-extraction-service/index.js
  • product-extraction-service/index.js
  • libs/shared/interfaces/cloudProvider.js

3. No Environment Detection

Services didn't automatically detect when running in the Firebase emulator environment.

Solutions Implemented

1. Created Pub/Sub Management Scripts

  • setup-pubsub-topics.js - Creates all required topics and subscriptions
  • test-pubsub-connection.js - Tests end-to-end Pub/Sub functionality
  • cleanup-pubsub-topics.js - Removes topics (for cleanup/reset)
  • verify-emulator-config.js - Verifies all services are properly configured

2. Fixed Service Configuration

Updated all services to automatically detect emulator environment:

// Auto-detect emulator environment
const isEmulator = process.env.FUNCTIONS_EMULATOR === 'true' || process.env.NODE_ENV === 'development';
const projectId = process.env.FIREBASE_PROJECT_ID || 'your-project-id';

// Initialize Pub/Sub client with emulator support
let pubsub;
if (isEmulator) {
  // Use emulator
  process.env.PUBSUB_EMULATOR_HOST = 'localhost:8085';
  console.log('🔧 Using Pub/Sub emulator at localhost:8085');
  pubsub = new PubSub({
    projectId: projectId
  });
} else {
  // Use production
  pubsub = new PubSub({
    projectId: projectId
  });
}

3. Created Required Topics

All required topics are now created in the Firebase emulator:

  • ✅ orchestrator-topic
  • ✅ category-extraction-topic
  • ✅ product-extraction-topic
  • ✅ product-enrichment-topic
  • ✅ sustainability-enrichment-topic
  • ✅ ai-logging-topic

Plus corresponding subscriptions for each topic.

Files Modified

Core Service Files

  1. orchestrator-service/pubsub-handler.js - Main Pub/Sub handler
  2. orchestrator-service/index.js - Main orchestrator service
  3. category-extraction-service/index.js - Category extraction service
  4. product-extraction-service/index.js - Product extraction service
  5. libs/shared/interfaces/cloudProvider.js - Shared cloud provider interface

New Management Scripts

  1. setup-pubsub-topics.js - Topic creation script
  2. test-pubsub-connection.js - Connection testing script
  3. cleanup-pubsub-topics.js - Cleanup script
  4. verify-emulator-config.js - Configuration verification script
  5. PUBSUB_MANAGEMENT.md - Management documentation

How to Use

1. Setup Topics (First Time)

cd esg_microservices_platform/firebase-scripts/local
node setup-pubsub-topics.js

2. Test Connection

node test-pubsub-connection.js

3. Verify Configuration

node verify-emulator-config.js

4. Cleanup (if needed)

node cleanup-pubsub-topics.js

Current Status

✅ ALL ISSUES RESOLVED

  • All required Pub/Sub topics exist
  • All services are configured to use the emulator
  • End-to-end Pub/Sub communication is working
  • Automatic emulator detection is implemented
  • Comprehensive management scripts are available

Next Steps

  1. Restart your orchestrator service - It should now work without errors
  2. Test the pipeline - The Pub/Sub messaging should work correctly
  3. Use management scripts - For future setup, testing, or cleanup needs

Prevention

The automatic emulator detection ensures that:

  • Services automatically use the emulator when running locally
  • Services automatically use production when deployed
  • No manual environment variable configuration is needed
  • The same code works in both environments

Troubleshooting

If you encounter issues again:

  1. Check emulator status: Ensure Firebase emulator is running with Pub/Sub support
  2. Verify topics exist: Run node test-pubsub-connection.js
  3. Check service logs: Look for emulator detection messages
  4. Recreate topics: Run node setup-pubsub-topics.js if needed

Notes

  • The emulator data is ephemeral and will be lost when the emulator is stopped
  • Always test your setup with the test script before running your services
  • The automatic detection makes the setup much more robust and user-friendly