Gallery Backend

1. Overview

This document provides a comprehensive overview of the API Attachment Gallery service, which manages attachments from various sources like [issues](../Issue Tracker/IssueTrackerOverview.md) and reports. The documentation traces the function flows from their endpoints, explains implementation details, and provides visualizations of the system architecture. The API Attachment Gallery is a service that manages the storage, retrieval, and processing of attachments within the Nimbly platform. It provides endpoints for creating, finding, filtering, downloading, and searching attachments across different departments, sites, and organizations. The service integrates with Firebase Storage for file storage and uses MongoDB for metadata storage.

The architecture follows a clean architecture pattern with clear separation between:

Routes: Define API endpoints and handle HTTP requests/responses
Controllers: Process input, call use cases, and format responses
Use Cases: Contain business logic and orchestrate domain operations
Repositories: Handle data access and persistence

2. Libraries and Dependencies

The API Attachment Gallery relies on the following main dependencies:

Category	Dependencies
Core Framework	express, cors, morgan
Authentication	jsonwebtoken, @nimbly-technologies/nimbly-backend-utils (AuthMiddleware)
Database	mongoose, mongodb
File Storage	firebase-admin, archiver, jszip
Date/Time	moment, moment-timezone
Utilities	lodash, uuid, sharp
Nimbly Packages	@nimbly-technologies/nimbly-common, @nimbly-technologies/entity-node, @nimbly-technologies/nimbly-access, @nimbly-technologies/nimbly-backend-utils
Logging & Tracing	@nimbly-technologies/nimbly-backend-utils (log), tracer
Testing	jest, supertest, chai

3. API Endpoints

The API Attachment Gallery exposes the following endpoints:

Method	Endpoint	Description	Controller Function	Validator
GET	/gallery/ping	Health check endpoint	N/A	None
GET	/gallery/:id	Get a single attachment by ID	findOne	getOne
POST	/gallery	Create a new attachment	create	None
POST	/gallery/report-attachments	Create attachments from report	createAttachmentsFromReport	createFromReport
GET	/gallery/filter	Find attachments with pagination	findAttachmentsByPagination	filter
GET	/gallery/search	Search attachments by text	searchAttachmentGallery	filter
GET	/gallery/download	Download attachments as ZIP	downloadAttachments	download
GET	/gallery/getTodaysCount	Get count of attachments created today	getTodaysCount	None

All endpoints require authentication using JWT tokens, which is enforced by the authMiddleware.expressHandler() middleware.

4. Controllers and Usecase Functions

4.1. AttachmentGalleryController

The AttachmentGalleryController class handles HTTP requests and delegates business logic to the usecase layer. It’s responsible for extracting data from requests, calling the appropriate usecase functions, and formatting responses.

export default class AttachmentGalleryController {
    public constructor(private attachmentGalleryUsecase: IAttachmentGalleryUsecases) {}
    
    // Controller methods...
}

4.1.1. Controller Functions

Function	Description	Parameters	Return Type
`findOne`	Retrieves a single attachment by ID	`context: Context<UserAuth>, payload: { params: { id: string } }`	`Promise<{ data: PopulatedAttachmentGallery, error: null } \| { data: null, error: string }>`
`create`	Creates a new attachment	`context: Context<UserAuth>, payload: { data: CreateAttachmentGallery }`	`Promise<{ data: string, error: null } \| { data: null, error: string }>`
`createAttachmentsFromReport`	Creates attachments from a report	`context: Context<UserAuth>, payload: { data: CreateAttachmentGalleryFromReport }`	`Promise<{ data: string, error: null } \| { data: null, error: string }>`
`findAttachmentsByPagination`	Retrieves paginated attachments with filters	`context: Context<UserAuth>, payload: { query: PaginationOptions & AttachmentGalleryQueryOptions }`	`Promise<{ data: FilterAttachmentsResult, error: null } \| { data: null, error: string }>`
`searchAttachmentGallery`	Searches attachments by text	`context: Context<UserAuth>, payload: { query: PaginationOptions & AttachmentGalleryQueryOptions }`	`Promise<{ data: SearchAttachmentResult, error: null } \| { data: null, error: string }>`
`downloadAttachments`	Downloads attachments as a ZIP file	`req: Request, res: Response, next: NextFunction`	Direct response stream
`getTodaysCount`	Gets count of attachments created today	`context: Context<UserAuth>`	`Promise<{ data: { count: number }, error: null } \| { data: null, error: string }>`

4.1.2. Controller Implementation Details

Error Handling Pattern

Each controller method follows a consistent pattern for error handling:

public findOne = async ({ context, payload }: FindOneParams) => {
    try {
        const { id } = payload.params;
        const result = await this.attachmentGalleryUsecase.findOne(context, id);
        return response(result.data, null);
    } catch (error) {
        log.error(error);
        return response(null, error);
    }
};

Query Processing

The controller processes query parameters and converts them to strongly-typed objects:

public findAttachmentsByPagination = async ({ context, payload }: FindAllPaginateParams) => {
    try {
        const queryOptions = new AttachmentGalleryQueryOptions(payload.query);
        if (!payload.query.sortDirections) queryOptions.sortDirections = getDefaultSorting(payload.query.groupBy);
 
        const result = await this.attachmentGalleryUsecase.paginatedAttachments(context, queryOptions);
        return response(result.data, null);
    } catch (error) {
        log.error(error);
        return response(error.message, errors.INVALID);
    }
};

Direct Response Handling

For file downloads, the controller handles the response directly:

public downloadAttachments = async (req: Request, res: Response, next) => {
    try {
        const parameter: DownloadAttachmentParams = {
            context: req.headers.internal ? { ...(res.locals || {}), internal: req.headers.internal } : res.locals,
            payload: {
                data: { ...req.body },
                query: { ...req.query },
                params: { ...req.params },
            },
        } as any;
 
        const { payload, context } = parameter;
        const queryParams = new AttachmentDownloadQuery(payload.query);
        this.attachmentGalleryUsecase.downloadAttachmentV2(context as any, queryParams, res);
    } catch (error) {
        log.error(error);
        res.status(400).send(error.message);
    }
};

4.2. AttachmentGalleryUsecases

The AttachmentGalleryUsecases class implements the business logic of the service. It orchestrates operations across multiple repositories and applies domain rules.

export class AttachmentGalleryUsecases implements IAttachmentGalleryUsecases {
    // Constructor with multiple repositories injected
    public constructor(
        attachmentGalleryRepo: IAttachmentGalleryRepository,
        issueRepository: IIssueRepository,
        userRepository: IUserRepository,
        issueTrackerSettingsRepository: IIssueTrackerSettingsRepository,
        issueReportRepository: IIssueReportRepository,
        questionnaireIndexRepository: IQuestionnaireIndexRepository,
        siteRepository: ISiteRepository,
        roleRepository: DatabaseAccess,
        organizationRepository: IOrganizationRepository,
        departmentIndexRepository: IDepartmentIndexRepository
    ) {
        // Initialize repositories
    }
    
    // Usecase methods...
}

4.2.1. Usecase Functions

Function	Description	Parameters	Return Type
`findOne`	Retrieves and populates a single attachment	`ctx: Context<UserAuth>, id: string`	`Promise<UsecaseReturn<PopulatedAttachmentGallery>>`
`create`	Creates new attachments from issue data	`ctx: Context<UserAuth>, data: CreateAttachmentGallery`	`Promise<UsecaseReturn<string>>`
`createFromReport`	Creates attachments from report data	`ctx: Context<UserAuth>, data: CreateAttachmentGalleryFromReport`	`Promise<UsecaseReturn<string>>`
`baseQuery`	Builds base query filters based on user context	`ctx: Context<UserAuth>, queryOptions: AttachmentGalleryQueryOptions`	`Promise<{ defaultQuery: MongoQuery<AttachmentGallery> }>`
`paginatedAttachments`	Retrieves paginated attachments with filtering	`ctx: Context<UserAuth>, queryOptions: AttachmentGalleryQueryOptions`	`Promise<UsecaseReturn<FilterAttachmentsResult>>`
`searchAttachments`	Searches attachments based on text query	`ctx: Context<UserAuth>, queryOptions: AttachmentGalleryQueryOptions`	`Promise<UsecaseReturn<SearchAttachmentResult>>`
`downloadAttachments`	Downloads attachments as ZIP (v1)	`ctx: Context<UserAuth>, queryParams: AttachmentDownloadQuery, res: Response`	`Promise<any>`
`downloadAttachmentV2`	Downloads attachments as ZIP with streaming (v2)	`ctx: Context<UserAuth>, queryParams: AttachmentDownloadQuery, res: Response`	`Promise<any>`
`getTodaysCount`	Gets count of attachments created today	`ctx: Context<UserAuth>`	`Promise<UsecaseReturn<{ count: number }>>`
`applyVisibilityFilter`	Creates filters based on user’s data visibility permissions	`ctx: Context<UserAuth>`	`Promise<DataVisibilityQuery>`

4.2.2. Usecase Implementation Details

findOne Function

The findOne function retrieves an attachment by ID, enriches it with additional data, and formats dates:

public async findOne(ctx: Context<UserAuth>, id: string): Promise<UsecaseReturn<PopulatedAttachmentGallery>> {
    const { organizationID } = ctx.user;
    const result = await this.attachmentGalleryRepo.findOne(organizationID, id);
    if (!result) {
        throw new ErrorCode(errors.NOT_FOUND, 'Attachment not found');
    }
 
    const { offset, siteID } = result;
    const site = await this.siteRepository.findByID(siteID, organizationID);
    const populatedResult = new PopulatedAttachmentGallery(result);
 
    // check if attachment has offset else get from site level, default Asia/Jakarta
    const timeZoneOffset = offset || site?.utcOffset || 420;
 
    // check if signed url is not 404
    populatedResult.signedUrl = await getSignedUrl(populatedResult.attachmentPath);
 
    // format the submitted date
    populatedResult.submittedDate = moment
        .utc(populatedResult.submittedDate)
        .utcOffset(timeZoneOffset)
        .format('DD MMM YYYY, hh:mm:ss A');
    return {
        data: populatedResult,
    };
}

create Function

The create function processes attachment data and handles sequence ID generation:

public async create(ctx: Context<UserAuth>, data: CreateAttachmentGallery): Promise<UsecaseReturn<any>> {
    try {
        const { organizationID } = ctx.user;
        const { attachments, submittedDate, ...basePayload } = data;
        basePayload.sequenceID = data.sequenceID; // Initialize with the incoming value
 
        // Handle sequence ID logic
        const isProblematicSequenceID = !data.sequenceID || (typeof data.sequenceID === 'string' && data.sequenceID.includes('undefined'));
        if (!isProblematicSequenceID) {
            log.debug('[AttachmentGallery.create] SequenceID is valid.');
        } else if (!data.issueID) {
            log.warn(`[AttachmentGallery.create] Missing issueID; cannot update problematic sequenceID '${basePayload.sequenceID}'.`);
        } else {
            await this.updateSequenceIDFromIssue(data.issueID, basePayload, data.sequenceID);
        }
 
        // Process each attachment
        const createPromiseArray = [] as any;
        for (const attachment of attachments) {
            const payload: AttachmentGallery = { ...basePayload, organizationID } as any;
            // Set attachment properties
            payload.origin = data.origin ?? 'issue';
            payload.attachmentPath = attachment.path;
            payload.attachmentType = attachment.fileType;
            payload.fileName = attachment.fileName;
            payload.offset = moment.parseZone(submittedDate).utcOffset();
            payload.submittedDate = new Date(submittedDate);
            // Get signed URL
            payload.signedUrl = await getSignedUrl(attachment.path);
            payload.fileSize = Number(attachment.size);
            // Generate thumbnail path
            const filename = path.basename(payload.attachmentPath);
            const dirname = path.dirname(payload.attachmentPath);
            payload.thumbnailPath = path.join(dirname, `thumb_${filename.split('.')[0]}.webp`);
            // Add to creation array
            createPromiseArray.push(this.attachmentGalleryRepo.create(payload));
        }
        return { data: 'DONE' };
    } catch (error) {
        log.info(`[DEBUG] - error while creating attachment from issue comments - ${error}`);
        throw new ErrorCode(errors.INTERNAL, error);
    }
}

baseQuery Function

The baseQuery function builds query filters based on user context and query options:

public async baseQuery(ctx: Context<UserAuth>, queryOptions: AttachmentGalleryQueryOptions) {
    const { organizationID, userID, role } = ctx.user;
 
    // Start with organization filter
    const defaultQuery: MongoQuery<AttachmentGallery> = {
        organizationID,
    };
 
    // Get organization timezone
    const organizationInfo = await this.organizationRepository.findByID(organizationID);
    const {
        schedule: { timezone },
    } = organizationInfo!;
 
    // Get user role and apply visibility filters
    const userRole = await this.roleRepository.getRoleForUser(userID, organizationID);
    const userBaseRole = userRole?.origin || role;
    const dataVisibilityQuery = await this.applyVisibilityFilter(ctx);
 
    // Handle user-related issues
    const qUserIDs = [userID];
    const relatedIssueIDs = await this.issueRepository.findIssuesRelatedToUser(ctx as any, qUserIDs, {
        issueID: 1,
    });
    const issueIDs = relatedIssueIDs.map((issue) => issue.issueID);
 
    // Define "own issue" conditions
    const ownIssue = [
        { createdBy: { $in: qUserIDs } },
        { auditorID: { $in: qUserIDs } },
        { issueID: { $in: issueIDs } },
    ];
 
    // Apply visibility filters
    if (dataVisibilityQuery.departmentSites) {
        // Department-site visibility logic
    }
    if (dataVisibilityQuery.departments) {
        // Department visibility logic
    }
    if (dataVisibilityQuery.sites) {
        // Site visibility logic
    }
 
    // Apply query option filters
    if (queryOptions.source.length) {
        defaultQuery.origin = { $in: queryOptions.source };
    }
    if (queryOptions.type.length) {
        defaultQuery.attachmentType = { $in: queryOptions.type };
    }
    // More filter conditions...
 
    return {
        defaultQuery,
    };
}

downloadAttachmentV2 Function

The downloadAttachmentV2 function implements streaming ZIP creation for downloading attachments:

public async downloadAttachmentV2(
    ctx: Context<UserAuth>,
    queryParams: AttachmentDownloadQuery,
    res: Response
): Promise<any> {
    try {
        const startTime = Date.now();
        const { organizationID } = ctx.user;
        const defaultQuery: MongoQuery<AttachmentGallery> = { organizationID };
        log.info(`Memory usage before zip creation: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(0)} MB`);
 
        // Validate attachment count
        if (queryParams.ids.length > 300) {
            throw new ErrorCode(errors.CONFLICT, 'Downloadable attachments Limit Exceeded');
        }
 
        // Build query filters
        defaultQuery.$or = [];
        let filters = [] as any;
        filters.push({
            _id: {
                $in: queryParams.ids.map((id) => mongoose.Types.ObjectId(id)),
            },
        });
        // Apply date filters
        // Apply site filters
 
        // Execute query
        let attachments;
        if (queryParams?.isFailedAttachment) {
            attachments = await this.attachmentGalleryRepo.findFailed(defaultQuery);
        } else {
            attachments = await this.attachmentGalleryRepo.find(defaultQuery);
        }
 
        // Set up streaming ZIP creation
        const fbBucket = admin.storage().bucket();
        const archive = archiver('zip', { zlib: { level: 0 } });
        archive.setMaxListeners(0);
        archive.pipe(res);
 
        // Add files to the ZIP
        let totalAttachments = 0;
        for (const attachment of attachments) {
            const { date, siteID, data } = attachment;
            totalAttachments += data.length;
            for (const d of data) {
                // Process each file
                const path = d.attachmentPath!;
                const sanitisedPath = path.startsWith('/') ? path.substring(1) : path;
                const fileName = d.fileName;
                const folderPath = `Attachments/${date}/${siteID}`;
                
                // Add file to archive with proper error handling
                archive.append(
                    fbBucket.file(sanitisedPath).createReadStream()
                        .on('error', (error) => {
                            log.error(`error while downloading file from gc storage, ${error}`);
                            throw new ErrorCode(errors.INVALID, error.message);
                        }),
                    { name: `${folderPath}/${fileName}` }
                ).on('error', (error) => {
                    log.error(`error while appending stream to archive, ${error}`);
                    throw new ErrorCode(errors.INVALID, error.message);
                });
            }
        }
 
        // Finalize the archive
        await archive.finalize().catch((error) => {
            log.error(`error at finalizing archive, ${error}`);
            res.end();
        });
 
        // Log performance metrics
        log.info(`total attachments : ${totalAttachments}`);
        log.info(`Memory usage After zip creation: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(0)} MB`);
        log.info(`total time taken: ${(Date.now() - startTime) / 1000}`);
    } catch (error) {
        log.error(`[ERROR] - ${error}`);
        res.status(500).send({
            message: error.message,
        });
    }
}

applyVisibilityFilter Function

The applyVisibilityFilter function restricts data access based on user permissions:

public async applyVisibilityFilter(ctx: Context<UserAuth>): Promise<DataVisibilityQuery> {
    const { userID, organizationID } = ctx.user;
    // Get user role
    const role = await this.roleRepository.getRoleForUser(userID, organizationID);
    ctx.user.role = role?.origin || ctx.user.role;
 
    // Extract resource permissions
    const resourceMap = role.resources?.length
        ? role.resources.reduce((acc, curr) => {
              acc[curr['resource']] = curr;
              return acc;
          }, {})
        : {};
 
    // Determine visibility level
    const dataVisibility = getDataVisibility(resourceMap, ctx.user.role);
    const dataVisibilityQuery: DataVisibilityQuery = {};
    
    // If user has full access, return empty query (no restrictions)
    if (dataVisibility === DataVisibility.ALL) return dataVisibilityQuery;
 
    // Get departments and sites user has access to
    const deptIndexes = await this.departmentIndexRepository.findByUserID(ctx, ctx.user.userID);
    const deptIDs: string[] = [];
    const siteIDs: string[] = [];
    let deptSites: DepartmentSiteQuery[] = [];
 
    // Build department and site lists
    for (const dept of deptIndexes) {
        deptIDs.push(dept.departmentID);
        for (const site of dept.sites) {
            siteIDs.push(site.id);
            deptSites.push({ siteID: site.id, departmentID: dept.departmentID });
        }
    }
 
    // Apply filters based on visibility level
    if (dataVisibility === DataVisibility.SPECIFIC_DEPT) {
        dataVisibilityQuery.departments = deptIDs;
    }
 
    // Get sites user is part of
    const allowedSites = await this.siteRepository.findByQuery(
        {
            organizationID: ctx.user.organizationID,
            $or: [
                { team: ctx.user.userID },
                { 'supervisors.userID': ctx.user.userID },
            ],
        },
        { siteID: 1 },
    );
    const allowedSiteIDs = allowedSites.map((site) => site.siteID);
 
    if (dataVisibility === DataVisibility.SPECIFIC_SITE) {
        dataVisibilityQuery.sites = allowedSiteIDs;
    }
 
    if (dataVisibility === DataVisibility.SPECIFIC_DEPT_SITE) {
        deptSites = deptSites.filter((deptSite) => allowedSiteIDs.includes(deptSite.siteID));
        dataVisibilityQuery.departmentSites = deptSites;
    }
 
    return dataVisibilityQuery;
}

5. Function Traces

5.1. Authentication Flow

sequenceDiagram
    participant Client
    participant Router
    participant AuthMiddleware
    participant Controller
    participant UseCase
    participant Repository
    participant Database

    Client->>Router: Request with JWT Token
    Router->>AuthMiddleware: expressHandler()
    AuthMiddleware->>AuthMiddleware: Verify JWT token
    AuthMiddleware->>AuthMiddleware: Decode user info from token
    AuthMiddleware->>Router: Add user context to request
    Router->>Controller: Call endpoint handler with auth context
    Controller->>UseCase: Pass auth context to use case
    UseCase->>Repository: Query with organization filter
    Repository->>Database: Execute query with organization filter
    Database->>Repository: Return data
    Repository->>UseCase: Return results
    UseCase->>Controller: Return processed data
    Controller->>Client: Return response

Authentication is handled through a JWT middleware that extracts and validates the token, then adds the authenticated user’s context to the request. This context includes:

organizationID: The organization the user belongs to
userID: The user’s unique identifier
role: The user’s role in the organization

5.2. Find Attachment Flow

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant UseCase
    participant Repository
    participant FirebaseStorage
    participant MongoDB

    Client->>Router: GET /gallery/:id
    Router->>Controller: findOne(context, id)
    Controller->>UseCase: findOne(context, id)
    UseCase->>Repository: findOne(organizationID, id)
    Repository->>MongoDB: Aggregate query with findOneAggregator
    MongoDB->>Repository: Return attachment document
    UseCase->>FirebaseStorage: getSignedUrl(attachmentPath)
    FirebaseStorage->>UseCase: Return signed URL
    UseCase->>UseCase: Format submittedDate with timezone
    UseCase->>Controller: Return PopulatedAttachmentGallery
    Controller->>Client: Return formatted response

The find attachment flow retrieves a single attachment by ID, enriches it with a signed URL for accessing the file in Firebase Storage, and formats the submission date based on the site’s timezone.

5.3. Create Attachment Flow

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant UseCase
    participant Repository
    participant FirebaseStorage
    participant MongoDB

    Client->>Router: POST /gallery
    Router->>Controller: create(context, payload)
    Controller->>UseCase: create(context, payload)
    UseCase->>UseCase: Process sequence ID
    UseCase->>UseCase: Process each attachment
    UseCase->>FirebaseStorage: getSignedUrl(attachment.path)
    FirebaseStorage->>UseCase: Return signed URL
    UseCase->>UseCase: Generate thumbnail path
    UseCase->>Repository: create(attachmentData)
    Repository->>MongoDB: Insert document
    UseCase->>Controller: Return success
    Controller->>Client: Return response

The create attachment flow processes incoming attachments, enriches them with metadata, and stores this information in MongoDB. It handles sequence IDs for issue tracking and generates paths for thumbnails.

5.4. Download Attachments Flow

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant UseCase
    participant Repository
    participant MongoDB
    participant FirebaseStorage
    participant ZipStream

    Client->>Router: GET /gallery/download
    Router->>Controller: downloadAttachments(req, res)
    Controller->>UseCase: downloadAttachmentV2(context, queryParams, res)
    UseCase->>Repository: find(query)
    Repository->>MongoDB: Execute query
    MongoDB->>Repository: Return matching attachments
    UseCase->>UseCase: Group attachments by date and site
    UseCase->>ZipStream: Create zip archive
    loop For Each Attachment
        UseCase->>FirebaseStorage: Get file stream
        FirebaseStorage->>ZipStream: Append file to archive with path
    end
    ZipStream->>Client: Stream zip file directly to response

The download attachments flow queries for attachments based on filters, retrieves the files from Firebase Storage, and streams them as a ZIP archive directly to the client, maintaining a folder structure based on date and site.

5.5. Search Attachments Flow

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant UseCase
    participant Repository
    participant MongoDB

    Client->>Router: GET /gallery/search
    Router->>Controller: searchAttachmentGallery(context, payload)
    Controller->>Controller: Create AttachmentGalleryQueryOptions
    Controller->>UseCase: searchAttachments(context, queryOptions)
    UseCase->>UseCase: baseQuery(context, queryOptions)
    UseCase->>UseCase: applyVisibilityFilter(context)
    UseCase->>Repository: findAttachmentsBySearch(defaultQuery, search, sortDirections)
    Repository->>Repository: Create searchAttachmentsAggregator
    Repository->>MongoDB: Execute aggregation pipeline
    MongoDB->>Repository: Return question, category, and issue matches
    Repository->>UseCase: Return search results
    UseCase->>Controller: Return formatted results
    Controller->>Client: Return response

The search attachments flow processes a search query and applies it across questions, categories, and issue IDs, returning matches grouped by these criteria.

5.6. Filter Attachments Flow

sequenceDiagram
    participant Client
    participant Router
    participant Controller
    participant UseCase
    participant Repository
    participant MongoDB

    Client->>Router: GET /gallery/filter
    Router->>Controller: findAttachmentsByPagination(context, payload)
    Controller->>Controller: Create AttachmentGalleryQueryOptions
    Controller->>UseCase: paginatedAttachments(context, queryOptions)
    UseCase->>UseCase: baseQuery(context, queryOptions)
    UseCase->>UseCase: applyVisibilityFilter(context)
    UseCase->>Repository: findAttachmentsByPagination(query, options, pagination, sort)
    Repository->>Repository: Create findPaginateAggregator
    Repository->>Repository: Create findCountAggregator
    Repository->>Repository: Create findGroupByCountAggregator
    Repository->>MongoDB: Execute aggregation pipelines
    MongoDB->>Repository: Return paginated results and counts
    Repository->>UseCase: Return paginated data
    UseCase->>UseCase: Group results by specified criteria
    UseCase->>Controller: Return grouped and paginated results
    Controller->>Client: Return formatted response

The filter attachments flow applies various filters (by date, site, department, auditor, etc.) and returns paginated results, optionally grouped by specified criteria.

6. Important Implementation Details

6.1. Pagination Implementation

The pagination implementation uses MongoDB’s aggregation framework to efficiently query and paginate large datasets. The key components include:

findPaginateAggregator: Builds an aggregation pipeline that:
- Filters documents based on query criteria
- Sorts results according to specified sort directions
- Skips documents for pagination
- Limits the number of returned documents

// Example from findPaginate.aggregator.ts
export const findPaginateAggregator = (
    query: MongoQuery<AttachmentGallery>,
    queryOptions: AttachmentGalleryQueryOptions,
    paginationOption: PaginationOptions,
    sortDirections: string,
) => {
    const { page, limit } = paginationOption;
    const skip = (page - 1) * limit;
    
    const aggregator: any[] = [
        { $match: query },
        { $sort: parseSort(sortDirections) },
        { $skip: skip },
        { $limit: limit }
    ];
    
    return aggregator;
};

findCountAggregator: Creates a separate aggregation to count the total number of documents matching the query, used for calculating total pages.
findGroupByCountAggregator: Builds an aggregation that groups documents by a specified field and counts the number of documents in each group.

This approach separates concerns and allows for efficient querying even with large datasets.

6.2. Data Visibility Filtering

The service implements a sophisticated visibility filtering system that restricts users to see only attachments from sites and departments they have access to:

public async applyVisibilityFilter(ctx: Context<UserAuth>): Promise<DataVisibilityQuery> {
    const { userID, organizationID } = ctx.user;
    const role = await this.roleRepository.getRoleForUser(userID, organizationID);
    ctx.user.role = role?.origin || ctx.user.role;
 
    const resourceMap = role.resources?.length
        ? role.resources.reduce((acc, curr) => {
              acc[curr['resource']] = curr;
              return acc;
          }, {})
        : {};
 
    const dataVisibility = getDataVisibility(resourceMap, ctx.user.role);
 
    const dataVisibilityQuery: DataVisibilityQuery = {};
    if (dataVisibility === DataVisibility.ALL) return dataVisibilityQuery;
 
    // Logic to restrict visibility based on user's departments and sites
    // ...
 
    return dataVisibilityQuery;
}

The visibility filter is applied based on:

User’s role and permissions
Departments the user belongs to
Sites the user has access to
Custom department-site combinations

This ensures data security and proper access control across the system.

6.3. Today’s Count Optimization

The service includes an optimized implementation for counting attachments created on the current day:

public async countTodaysAttachments(organizationID: string, startDate: Date, endDate: Date): Promise<number> {
    try {
        // Create a simple query that only filters by organization and date range
        const query = {
            organizationID,
            submittedDate: {
                $gte: startDate,
                $lt: endDate,
            },
        };
 
        // Use the countDocuments method which is optimized for counting
        const count = await this.model.countDocuments(query);
        return count;
    } catch (error) {
        log.error(`Error counting today's attachments: ${error}`);
        return 0;
    }
}

This optimization:

Uses MongoDB’s countDocuments instead of fetching and counting results
Applies minimal filtering to improve performance
Works with the organization’s timezone to ensure accurate day boundaries

6.4. Attachment Processing

The service handles various attachment types and sources, with special processing for:

Thumbnails: Automatically generates thumbnail paths for images

payload.thumbnailPath = path.join(dirname, `thumb_${filename.split('.')[0]}.webp`);

Signed URLs: Creates temporary signed URLs for secure access to attachments
```
payload.signedUrl = await getSignedUrl(attachment.path);
```
File Size: Retrieves and stores the file size for reporting and filtering
```
payload.fileSize = Number(attachment.size);
```
Timezone Handling: Preserves the original timezone offset for correct date display
```
payload.offset = moment.parseZone(submittedDate).utcOffset();
```

7. Repository Pattern Implementation

The service implements the repository pattern to abstract data access:

classDiagram
    class IAttachmentGalleryRepository {
        <<interface>>
        +create(data: AttachmentGallery): Promise<void>
        +findOne(organizationID: string, id: string): Promise<PopulatedAttachmentGallery | null>
        +findAll(organizationID: string): Promise<AttachmentGallery[]>
        +find(filterQuery: MongoQuery<AttachmentGallery>): Promise<DownloadQueryResponse[]>
        +findAttachmentsByPagination(query, queryOptions, paginationOption, sortDirections): Promise<PaginationResult>
        +findAttachmentsBySearch(queryOptions, querySearch, sortDirections): Promise<SearchAttachmentsQueryResponse>
        +countTodaysAttachments(organizationID, startDate, endDate): Promise<number>
    }
    
    class AttachmentGalleryRepo {
        -model: AttachentGalleryModel
        -reportModel: Collection
        +constructor(conn: Connection)
        +create(data: AttachmentGallery): Promise<void>
        +findOne(organizationID: string, id: string): Promise<PopulatedAttachmentGallery | null>
        +findAll(organizationID: string): Promise<AttachmentGallery[]>
        +find(filterQuery: MongoQuery<AttachmentGallery>): Promise<DownloadQueryResponse[]>
        +findAttachmentsByPagination(query, queryOptions, paginationOption, sortDirections): Promise<PaginationResult>
        +findAttachmentsBySearch(queryOptions, querySearch, sortDirections): Promise<SearchAttachmentsQueryResponse>
        +countTodaysAttachments(organizationID, startDate, endDate): Promise<number>
    }
    
    IAttachmentGalleryRepository <|.. AttachmentGalleryRepo

Key aspects of the repository implementation:

Interface Definition: Clear contract defined in attachmentGallery.repository.d.ts
MongoDB Implementation: Concrete implementation using Mongoose in attachmentGallery.repository.ts
Aggregation Separation: Complex MongoDB aggregations are separated into their own files
Dependency Injection: Repository instances are created and injected through the domain factory

This pattern provides:

Clean separation of concerns
Testability through mocking
Flexibility to change data sources
Clear API contract for data access

8. Database Schema

The AttachmentGallery collection schema includes the following key fields:

Field	Type	Description
`_id`	ObjectId	Unique identifier for the attachment
`organizationID`	String	Organization that owns the attachment
`attachmentPath`	String	Path to the file in Firebase Storage
`attachmentType`	String	Type of attachment (image, video, etc.)
`thumbnailPath`	String	Path to the thumbnail image
`fileName`	String	Original file name
`fileSize`	Number	Size of the file in bytes
`signedUrl`	String	Temporary URL for accessing the file
`submittedDate`	Date	When the attachment was submitted
`offset`	Number	Timezone offset in minutes
`siteID`	String	Site associated with the attachment
`deptID`	String	Department associated with the attachment
`auditorID`	String	User who created the attachment
`issueID`	String	Related issue ID (if applicable)
`sequenceID`	String	Sequence number for tracking
`reportID`	String	Related report ID (if applicable)
`questionnaireID`	String	Related questionnaire ID (if applicable)
`question`	String	Question text (for report attachments)
`answer`	String	Answer text (for report attachments)
`category`	String	Category of the attachment
`origin`	String	Source of the attachment (issue, report)

9. Error Handling Strategy

The service implements a consistent error handling pattern:

Controller Level: Catches exceptions and formats responses

try {
    const result = await this.attachmentGalleryUsecase.findOne(context, id);
    return response(result.data, null);
} catch (error) {
    log.error(error);
    return response(null, error);
}

Use Case Level: Throws typed errors with error codes

if (!result) {
    throw new ErrorCode(errors.NOT_FOUND, 'Attachment not found');
}

Repository Level: Logs errors and rethrows or returns empty results

try {
    const count = await this.model.countDocuments(query);
    return count;
} catch (error) {
    log.error(`Error counting today's attachments: ${error}`);
    return 0;
}

Middleware Level: Handles global errors and formatting

This multi-layered approach ensures:

Consistent error responses
Detailed logging for debugging
Type-safe error handling
Clean separation of concerns

10. File Management and Storage

The service uses Firebase Storage for file storage and implements several file management strategies:

Download Implementation:
- V1 (JSZip): Uses JSZip to create in-memory ZIP archives
- V2 (Archiver): Uses Archiver for streaming ZIP creation to reduce memory usage
File Size Limits:
- Restricts downloads to 300 attachments maximum
- Implements memory monitoring during ZIP creation
Folder Structure:
- Organizes files by date and site in download archives
- Maintains a consistent path structure in Firebase Storage
Signed URLs:
- Generates temporary signed URLs for secure file access
- Caches URLs to reduce Firebase API calls
Error Handling:
- Gracefully continues if individual files fail during download
- Provides detailed error messages for file access issues

11. Performance Considerations

The service implements several performance optimizations:

Memory Management:
- Streaming ZIP creation to avoid memory issues with large downloads
- Memory usage logging and monitoring
```
log.info(`Memory usage before zip creation: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(0)} MB`);
```
Database Optimizations:
- Efficient MongoDB aggregation pipelines
- Selective field projection to reduce data transfer
- Indexes on frequently queried fields
Query Optimization:
- Specialized counting queries for better performance
- Separate aggregation pipelines for different query types
Caching:
- Caching of expensive operations
- Reuse of common queries
Response Streaming:
- Direct streaming of large responses to clients
- Chunked transfer encoding for downloads

Compass - Nimbly Documentation

Explorer