Compass - Nimbly Documentation

Reports 2.0 App Backend

13 min read

Section 1: Nimbly API Reports System

1. Overview

The Nimbly API Reports system is a comprehensive solution for managing audit reports within the Nimbly application. It enables users to create, retrieve, update, submit, and delete reports through a set of well-defined API endpoints. The system supports both regular reports and [team audits](../Schedule/Team Audit/TeamAuditOverview.md), with various validation mechanisms to ensure data integrity.

The codebase follows a clean architecture pattern with clear separation of concerns:

  • Routes define API endpoints
  • Controllers handle request/response processing
  • Usecases contain the business logic
  • Repositories manage data persistence

2. API Endpoints Overview

The Reports API exposes several endpoints to manage reports. Here’s a summary of all available endpoints:

EndpointHTTP MethodDescription
/lite/startPOSTCreates a new report in Firebase
/:siteID/:firebaseReportIDGETRetrieves a report and its summary from Firebase
/:mongoReportIDGETRetrieves a [team audit](../Schedule/Team Audit/TeamAuditOverview.md) report and its summary
/lite/:siteID/:firebaseReportID/updatePUTUpdates an existing report in Firebase
/lite/:siteID/:firebaseReportID/update-team-auditPATCHUpdates a [team audit](../Schedule/Team Audit/TeamAuditOverview.md) report
/:firebaseReportID/submitPUTSubmits a report in Firebase
/lite/:siteID/:firebaseReportID/submitPOSTSubmits a report in Firebase (V2)
/lite/:siteID/:firebaseReportID/submit-team-auditPOSTSubmits a [team audit](../Schedule/Team Audit/TeamAuditOverview.md) report
/:siteID/:firebaseReportIDDELETEDeletes a report from Firebase

All endpoints are secured with an authentication middleware, ensuring that only authenticated users can access the API.

3. System Architecture

The Nimbly Reports API follows a clean architecture pattern with the following components:

flowchart TB
    Router["Router (report.app.router.ts)"] --> |"Handle HTTP requests"| Controller["Controller (report.app.controller.ts)"]
    Controller --> |"Execute business logic"| Usecase["Usecase (report.app.usecase.ts)"]
    Usecase --> |"Persist data"| Repositories["Repositories"]
    Repositories --> MongoDB[("MongoDB")]
    Repositories --> Firebase[("Firebase")]
    
    subgraph Repositories["Repository Layer"]
        ReportRepo["Report Repository"]
        SiteRepo["Site Repository"]
        QuestionnaireRepo["Questionnaire Repository"]
        UserRepo["User Repository"]
        FileMetadataRepo["File Metadata Repository"]
        OrgRepo["Organization Repository"]
    end

The system implements a clean separation of concerns:

  1. Router Layer: Defines API endpoints and routes requests to the appropriate controller methods
  2. Controller Layer: Processes HTTP requests and responses, handles parameter validation
  3. Usecase Layer: Contains the business logic for report operations
  4. Repository Layer: Provides data access and persistence mechanisms for different entities

This architecture ensures maintainability, testability, and scalability of the codebase.

4. Key Data Models

The system uses several key data models to represent reports and related entities:

4.1. Report Model

The core data structure for reports includes:

// Report model used in MongoDB
interface Report {
  // Core identifiers
  reportID: string;
  scheduleID: string;
  siteID: string;
  auditorID: string;
  organizationID: string;
  questionnaireID: string;
  
  // Status and metadata
  status: string; // 'draft' | 'complete'
  checkInAt: Date;
  checkOutAt: Date | null;
  submittedAt: Date | null;
  scheduledAt: Date;
  dueAt: Date;
  
  // Content
  questions: Question[];
  sections: ReportSectionIndex[] | null;
  signatures: ReportSignature[];
  emailTargets: EmailTarget[];
  
  // Flags and settings
  hasDeadline: boolean;
  startTime: number;
  endTime: number;
  isMakeUp: boolean;
  makeUpReason: string;
  isAdhoc: boolean;
  
  // Progress tracking
  totalQuestions: number;
  totalAnsweredQuestions: number;
  questionnaireProgress: Record<string, boolean>;
}

4.2. Firebase Report Model

A parallel model is used for Firebase storage:

// Report_Firebase model
interface Report_Firebase {
  // Core identifiers
  site: string;
  auditor: string;
  questionnaire: string;
  scheduleKey: string;
  
  // Timestamps (as ISO strings)
  datetimeIn: string;
  datetimeOut: string;
  datetimeSubmitted: string;
  datetimeUpdated: string;
  datetimeScheduled: string;
  dueDate: string;
  
  // Content
  questions: Question[];
  sections: ReportSectionIndex_Firebase[];
  signatures?: ReportSignature[];
  selfieSignatures?: SelfieSignature[];
  emailTargets: EmailTarget[];
  
  // Status and flags
  status: string; // 'draft' | 'complete'
  isMakeUp: boolean;
  makeUpReason: string;
  isAdhoc: boolean;
  
  // Settings
  hasDeadline: boolean;
  startTime: number;
  endTime: number;
  
  // Progress tracking
  totalQuestions: number;
  totalAnsweredQuestions: number;
  questionnaireProgress: Record<string, boolean>;
}

5. Report Creation Flow

The report creation process is initiated when a user triggers the /lite/start endpoint. This flow creates a new report in both Firebase and MongoDB.

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant Usecase
    participant Repositories
    participant MongoDB
    participant Firebase
    
    Client->>Router: POST /lite/start
    Router->>Controller: createReportFirebase()
    Controller->>Usecase: start(context, payload.data)
    
    Usecase->>Repositories: findByScheduledIDAndDate(scheduleID, scheduledDate)
    Usecase->>Repositories: findByID(organizationID)
    Usecase->>Repositories: findByID(scheduleID)
    
    alt Report already exists
        Usecase-->>Controller: Throw "Report Already Exists" error
    end
    
    alt Schedule not found or disabled
        Usecase-->>Controller: Throw "Schedule Not Found" error
    end
    
    Usecase->>Repositories: findByID(siteID, organizationID)
    Usecase->>Repositories: findByID(questionnaireIndexID, organizationID)
    
    alt Site not found or disabled
        Usecase-->>Controller: Throw "Site not found" error
    end
    
    alt Questionnaire Index not found or disabled
        Usecase-->>Controller: Throw "Questionnaire Index not found" error
    end
    
    Usecase->>Repositories: findOne(ctx, latest, organizationID)
    
    alt Questionnaire not found or disabled
        Usecase-->>Controller: Throw "Questionnaire not found" error
    end
    
    Usecase->>Usecase: Generate report payload
    Usecase->>Usecase: Generate MongoDB report
    Usecase->>Usecase: Generate Firebase report summary
    
    Usecase->>Repositories: upsertOne(mongoReportID, mongoReport)
    Usecase->>Repositories: upsertOneToSummary(reportSummary)
    Usecase->>Repositories: updateReportFirebase(report, organizationID, firebaseReportID, summary)
    
    Usecase-->>Controller: Return report data
    Controller-->>Router: Return response
    Router-->>Client: Return created report data

6. Report Retrieval Flow

The system provides two main endpoints for retrieving reports:

  1. GET /:siteID/:firebaseReportID - Retrieves a regular report by its Firebase ID
  2. GET /:mongoReportID - Retrieves a team audit report by its MongoDB ID

6.1. Regular Report Retrieval

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant Usecase
    participant Repositories
    participant MongoDB
    participant Firebase
    
    Client->>Router: GET /:siteID/:firebaseReportID
    Router->>Controller: getReportAndSummaryFirebase()
    Controller->>Usecase: get(context, firebaseReportID, siteID)
    
    Usecase->>Repositories: findByID(mongoReportID)
    Usecase->>Repositories: findMongoReportSummaryByID(mongoReportID)
    
    alt Report has questionnaire
        Usecase->>Repositories: findOne(ctx, report.questionnaire, organizationID)
    end
    
    Repositories-->>Usecase: Return report and summary data
    Usecase-->>Controller: Return report, summary, and questionnaire details
    Controller-->>Router: Return response
    Router-->>Client: Return report data

7. Report Update Flow

The system provides two endpoints for updating reports:

  1. PUT /lite/:siteID/:firebaseReportID/update - Updates a regular report
  2. PATCH /lite/:siteID/:firebaseReportID/update-team-audit - Updates a team audit report

The report update process involves calculating progress metrics, handling email targets, and updating both MongoDB and Firebase databases:

// From report.app.usecase.ts - update method
public async update(
  ctx: Context<UserAuth>,
  siteID: string,
  firebaseReportID: string,
  data: Partial<Report_Firebase>,
): Promise<UsecaseReturn<Report_Firebase>> {
  const { organizationID } = ctx.user;
 
  // Get the reports and site
  const mongoReportID = `${siteID}_${firebaseReportID}`;
  const [reportMongo, reportSummaryMongo, site] = await Promise.all([
    this.reportRepo.findByID(mongoReportID),
    this.reportRepo.findMongoReportSummaryByID(mongoReportID),
    this.siteRepo.findByID(siteID, organizationID),
  ]);
  const report = reportFirebaseMap(reportMongo);
  const summary = reportSummaryFirebaseMap(reportSummaryMongo);
 
  if (!report) {
    log.info(`Report not found, reportID: ${firebaseReportID}`);
    return {
      data: data as Report_Firebase,
    };
  }
 
  // Validation and data processing...
 
  // Get questionnaire progress and total answered questions count
  const questionnaireProgress = getQuestionnaireProgress(
    updateReportPayload.questions,
    updateReportPayload.questionnaireProgress,
  );
 
  const totalAnsweredQuestions = getTotalQuestionsAnswered(updateReportPayload.questions);
 
  updateReportPayload['totalAnsweredQuestions'] = totalAnsweredQuestions;
  updateReportPayload['questionnaireProgress'] = questionnaireProgress;
 
  // Update MongoDB and Firebase
  await Promise.all([
    this.reportRepo.upsertOne(mongoReportID, mongoReportUpdatePayload),
    this.reportRepo.updateReportFirebase({
      report: updateReportPayload,
      organizationID: organizationID,
      reportFirebaseID: firebaseReportID,
      summary: updateSummaryPayload,
    }),
  ]);
 
  return {
    data: updateReportPayload,
  };
}

8. Report Submission Flow

The report submission process is critical as it finalizes the report and makes it available for viewing and analysis. When a report is submitted, its status changes from “draft” to “complete”, and it triggers subsequent processing (including the reportQueueHandler described in Section 2).

Section 2: Report Queue Handler Implementation

1. Overview

The reportQueueHandler is a critical Firebase Cloud Function in the Nimbly audit application. Its primary responsibility is to process completed audit reports submitted by users, generate PDF/Excel reports, distribute notifications through multiple channels (email, push, WhatsApp), and perform various post-processing tasks.

This function is triggered when a new entry is added to the Firebase Realtime Database path /queue/reportCompleted/{queueKey}. It serves as a comprehensive asynchronous processing pipeline that transforms raw audit data into finalized reports with notifications to relevant stakeholders.

flowchart TB
    A[Function Called] --> B{Authentication Check}
    B -->|Success| C[Initialize MongoDB]
    B -->|Failure| Z[Unauthorized Error]
    
    C --> D[Validate Report]
    D -->|Invalid| Y[Validation Error]
    D -->|Valid| E[Setup API Clients]
    
    E --> F[Fetch Resources]
    F --> G[Calculate Scores]
    
    G --> H{Image Detection Enabled?}
    H -->|Yes| I[Process Object Detection]
    H -->|No| J[Generate Report]
    I --> J
    
    J --> K[Update Report with Download URL]
    K --> L{Issue Created?}
    
    L -->|No| M[Create Issues]
    L -->|Yes| N[Make Report Summary]
    M --> N
    
    N --> O[Update Report State]
    O --> P[Create Report Queue]
    P --> Q[Return URL]

2. Libraries and Dependencies

The reportQueueHandler utilizes numerous libraries to accomplish its complex tasks:

LibraryPurpose
@sendgrid/helpers/classes/mailEmail formatting and structure definition
@sendgrid/mailEmail delivery service
firebase-adminFirebase administration SDK for database, storage access
firebase-functionsCloud Functions framework for handling triggers
lodash/cloneDeepDeep cloning of objects
moment-timezoneDate/time handling with timezone support
mongooseMongoDB ODM (Object-Document Mapper)
@nimbly-technologies/nimbly-commonShared models and types
@nimbly-technologies/nimbly-accessBackend access utilities
@nimbly-technologies/entity-nodeEntity repositories

3. Function Signature and Parameters

export async function reportQueueHandler(
  snapshot?: functions.database.DataSnapshot,
  context?: functions.EventContext,
  retryCount = 2
) { /* ... */ }
ParameterTypeDescription
snapshotfunctions.database.DataSnapshotReport snapshot from queue docs, used to validate report status
contextfunctions.EventContextFirebase context, used to retrieve queueKey
retryCountNumberNumber of retries when function call fails (default: 2)

4. Implementation Flow

4.1. Function Initialization and MongoDB Connection

await initMongoose(functions.config().mongodb.url);
 
// init repositories
const siteRepo: ISiteRepository = new SiteMongo(connection.useDb('nimbly'));
const siteRepoV2 = new SiteMongo(connection.useDb('nimbly'));
const siteScheduleRepo: ISiteScheduleIndexRepository = new SiteScheduleMongo(connection.useDb('nimbly'));
const orgRepo: IOrganizationRepository = new OrganizationMongo(connection.useDb('nimbly'));
// ... additional repositories

The function begins by establishing a connection to MongoDB and initializing various repositories for data access. The repositories provide an abstraction layer over MongoDB collections, making it easier to perform CRUD operations on specific entities.

4.2. Data Extraction from Queue

const { queueKey } = context.params;
console.info(`[DEBUG] queKey: ${queueKey})`);
 
const config = functions.config();
// Create API instances for internal and external services
const api = createAPI(config.env.internal_cloud_api_url || 'https://asia-east2-nimbly-api-261017.cloudfunctions.net');
const internalAPI = createAPI(
  config.env.internal_cloud_api_url || 'https://asia-east2-nimbly-api-261017.cloudfunctions.net'
);
// ... additional API clients
 
const value: Classes.ReportQueueData = snapshot.val();
const { reportKey, siteKey, organizationKey } = value;
let { report } = value;

The function extracts necessary data from the queue entry, including report details, site information, and organization identification. It also initializes API clients for communication with other services.

4.3. Error Handling Setup

const handleError = async (error: Error, updates: Partial<Classes.Report>) => {
  if (retryCount === 0) {
    console.log(`[DEBUG] Max Retry for :
    emailTargets: ${updates.emailTargets},
    isGenerated: ${updates.isGenerated},
    isIssueCreated: ${updates.isIssueCreated},
    isScheduleAdjusted: ${updates.isScheduleAdjusted}`);
    throw error;
  }
  console.error(error.message);
  console.log(`[DEBUG] Retrying...${retryCount} of 3`);
  return await reportQueueHandler(snapshot, context, retryCount - 1);
};

A robust error handling mechanism is established to manage failures during processing. The function implements a retry mechanism with a configurable retry count.

4.4. Image Detection Processing

if (!report.isJourney) {
  try {
    // Perform Object detection on images
    const objectDetectionProcessor = new ObjectDetectionVertexAiProcessor();
    if (organizationMongo?.useImageDetection) {
      await objectDetectionProcessor.processReport(authorizationHeader, report);
    }
 
    // Update the report in firebase
    await db.ref(`/report`).child(organizationKey).child(siteKey).child(reportKey).update(report);
    console.log('[DEBUG] Updated firebase report with object detection results');
    
    // Update report in MongoDB
    report.questions.forEach((questionFb, index) => {
      const questionMongo = mongoReport.questions[index];
      // If the question is not using image detection, skip it
      if (
        !questionFb.useImageDetection ||
        !questionFb.imageDetectionKeywords ||
        questionFb.imageDetectionKeywords.length === 0 ||
        !questionFb.photos ||
        questionFb.photos.length === 0
      )
        return;
      questionMongo.imageDetectionScore = questionFb.imageDetectionScore;
      questionMongo.imageDetectionResult = questionFb.imageDetectionResult;
      questionMongo.imageDetectionScoresForPhotos = questionFb.imageDetectionScoresForPhotos;
    });
 
    await reportRepositoryMongo.upsertOne(mongoReport);
  } catch (error) {
    console.log('[ERROR] - error when performing object detection', error.message);
  }
}

For reports with images, the function performs object detection using the Google Vertex AI service if enabled for the organization, updating both Firebase and MongoDB with the detection results.

4.5. Report Generation

const retryGenerateReport = async (retry: number = 2) => {
  try {
    generatedReport = await generateReport(
      report,
      { key: organizationKey, value: organization },
      { key: siteKey, value: siteFirebase },
      reportKey,
      questionnaire,
      questionnaireIndex,
      auditorInCharge,
      organizationSKUs,
      flag,
      scoreV1,
      score.total,
      score.raw,
      false,
      departmentID,
      branding,
      answerColumnFlag,
      isAutoFailEnabled,
      scoreDetails,
      totalFailedQuestions,
      reportFailed,
      useImageDetectionConfigFlag
    );
  } catch (error) {
    if (retry > 0) {
      return await retryGenerateReport(retry - 1);
    }
 
    throw error;
  }
};
 
await retryGenerateReport();

The report generation process creates a PDF or Excel file based on the collected data and organizational settings. The process includes a retry mechanism for resilience.

4.6. Multi-Channel Notification Delivery

// Email notifications
if (userEmailTargets.length > 0 && report.site === siteKey && allowAutoGenerateReport && emailNotificationsEnabled) {
  const emailTriesRemaining = 2;
  try {
    console.log(`[DEBUG] sending report:complete email to ${message.to}`);
    if (message.cc && Array.isArray(message.cc)) {
      message.cc.forEach(ccEmail => {
        console.log(`[DEBUG] sending report:complete email to ${ccEmail}`);
      });
    }
    await sendEmail(emailTriesRemaining);
  } catch (error) {
    console.error('[ERROR] Error sending email:', error);
  }
}
 
// Push notifications
if (recipientsPushNotificationUids && recipientsPushNotificationUids.length > 0 && pushNotificationEnabled) {
  for (const auditor of recipientsPushNotificationUids) {
    try {
      console.log(`[DEBUG] sending push notification to ${auditor}`);
      await sendNotification(auditor, site.name, 'Your report was just generated. Check your email!');
    } catch (error) {
      console.error('[ERROR] Error sending notification:', error);
    }
  }
}
 
// WhatsApp notifications
if (wtspNotificationEnabled && userWhatsappTargets.length > 0) {
  const whatsappULR =
    config.api.whatsapp_url || 'https://asia-east2-nimbly-api-261017.cloudfunctions.net/api-whatsapp';
  const whatsappAPI = createAPI(whatsappULR);
  const whatsappNotificationPromises: Promise<string>[] = [];
 
  userWhatsappTargets.forEach(member => {
    if (!member) {
      console.error(`[DEBUG][Whatsapp] User ${member} not found`);
      return;
    }
    const userID = member;
    console.log(`[DEBUG] sending whatsapp notification to ${userID}`);
    const requestBody = {
      notificationType: 'report-generated',
      userID: userID,
      questionnaire: questionnaireTitle,
      fileUrl: url
    };
    whatsappNotificationPromises.push(
      whatsappAPI.post('/report-notification', requestBody, {
        headers: {
          authorization: authorizationHeader
        }
      })
    );
  });
  try {
    await Promise.all(whatsappNotificationPromises);
  } catch (error) {
    console.error(`[DEBUG] Error Sending request to whatsapp api for report generated, error:${error}`);
  }
}

The function delivers notifications across multiple channels based on configured preferences and availability.

5. Report Finalization and Queue Management

// Clean report data and update references
report.status = 'complete';
report.failStatus = reportFailed;
 
if (report.failStatus === null) delete report.failStatus;
 
await reportRef.update(report);
console.log(`[DEBUG] report updated`);
 
try {
  // eslint-disable-next-line no-shadow
  const data: any = {
    ...report,
    grade: report?.grade || '',
    // mark for handler to process report as whole
    isWholeReport: true
  };
 
  console.log(`[DEBUG] create queue`);
  const reportQueueRepository = new ReportQueueRepository();
  await reportQueueRepository.create(organizationKey, siteKey, reportKey, data);
} catch (err) {
  console.log(`[DEBUG] queue err: ${err}`);
}
 
// Remove the original queue entry to prevent reprocessing
await snapshot.ref.remove();

The function finalizes the report processing by updating the report status in the database, creating a new queue entry for potential downstream processing, and removing the original queue entry to prevent duplicate processing.

6. Error Handling and Resilience

The function implements several resilience patterns:

  1. Retry mechanisms for critical operations like report generation and email sending
  2. Error logging for better diagnostics
  3. Graceful failure for non-critical components
  4. Memory usage tracking to prevent resource exhaustion
flowchart TD
    A[Operation Start] --> B{Try Operation}
    B -->|Success| C[Continue Processing]
    B -->|Failure| D{Retry Count > 0?}
    D -->|Yes| E[Decrement Retry Count]
    E --> B
    D -->|No| F[Log Max Retries]
    F --> G[Throw Error]

7. Performance Considerations

7.1. Memory Management

const used = process.memoryUsage().heapUsed / 1024 / 1024;
console.log(`[INFO] Memory Used: ~${used.toFixed(0)} MB`);

The function tracks memory usage to ensure it stays within Cloud Function limits (2GB as specified in configuration).

7.2. Timeout Configuration

module.exports = functions
  .runWith({ memory: '2GB', timeoutSeconds: 540 })
  .database.instance(functions.config().env.FIREBASE_DATABASE_INSTANCE)
  .ref('/queue/reportCompleted/{queueKey}')
  .onCreate((snapshot, context) => reportQueueHandler(snapshot, context));

The function is configured with a 540-second timeout (9 minutes), allowing substantial processing time.

8. Data Flow Diagram

flowchart LR
    A[Mobile App] -->|Submit Report| B[Firebase DB]
    B -->|Trigger| C[reportQueueHandler]
    C -->|Fetch Data| D[MongoDB]
    C -->|Fetch Data| E[Firebase DB]
    C -->|Upload| F[Firebase Storage]
    C -->|Send| G[SendGrid Email]
    C -->|Notify| H[Push Notification]
    C -->|Message| I[WhatsApp API]
    C -->|Create| J[Attachment Gallery]
    C -->|Analyze| K[Vertex AI]
    C -->|Create| L[Issue Reports]

Integration Between API Reports System and Report Queue Handler

The integration between these two components forms a complete workflow:

  1. The API Reports System provides endpoints for user-facing operations (create, retrieve, update, delete reports)
  2. When a report is submitted via the API, a new entry is created in the Firebase queue
  3. This queue entry triggers the Report Queue Handler function
  4. The Report Queue Handler processes the report, generates PDFs/Excel files, and sends notifications
  5. The original report data in both MongoDB and Firebase is updated with processing results

This architecture enables asynchronous processing of reports, improving scalability and user experience by not blocking the API response while heavy processing occurs.

Conclusion

The Reports Backend is a comprehensive system that manages the entire report lifecycle:

  1. Creation & Management: Handled by the API Reports System with clean architecture
  2. Processing & Distribution: Managed by the Report Queue Handler with resilient processing
  3. Storage & Retrieval: Dual-database approach with MongoDB and Firebase

Together, these components provide a robust foundation for the reporting features of the Nimbly application, balancing immediate user interactions with asynchronous background processing.

Graph View

Backlinks