Skip to main content

Wally: SAM.GOV - Unauthorised Workflow

1. Purpose​

This workflow automates the discovery and monitoring of procurement opportunities from SAM.gov (System for Award Management) related to accessibility compliance. It searches for opportunities using accessibility keywords, scrapes procurement data, stores results in Google Sheets, and sends daily email reports to stakeholders via cron schedule. The workflow runs automatically at 6:30 PM IST daily to send email reports.

2. Trigger​

Type: Cron (Scheduled)

Details:

The workflow runs automatically at 6:30 PM IST (18:30) every day. This uses predefined accessibility keywords: Section 508, WCAG, EN 301 549, and digital accessibility.

Schedule Time: 6:30 PM IST (18:30) daily

Timezone: Asia/Kolkata

3. Step-by-Step Process​

Step 1: Schedule Trigger - 6:30 PM IST​

What happens here: The workflow starts automatically at 6:30 PM IST (18:30) every day. This is the scheduled entry point of the workflow.

What input it uses: No input required - scheduled trigger.

Why this step is needed: Ensures the workflow runs consistently on a daily schedule at 6:30 PM IST without manual intervention, allowing for automated procurement opportunity discovery.

Step 2: Generate Current Date​

What happens here: Generates the current date in ISO format (YYYY-MM-DD) to be used for sheet naming and email reporting.

What input it uses: System date/time.

What output it produces: Date string in format "YYYY-MM-DD".

Why this step is needed: Ensures each execution creates uniquely named Google Sheets and includes the date in email reports for tracking purposes.

Step 3: Search keyword1​

What happens here: Defines accessibility-related keywords and constructs SAM.gov search URLs for each keyword. Creates filtered search queries targeting 8(a) Set-Aside and Sole Source opportunities.

What input it uses: Hardcoded keywords array: ["Section 508", "WCAG", "EN 301 549", "digital accessibility"]

What output it produces: Individual items for each keyword containing:

  • Keyword name
  • SAM.gov search URL with filters
  • Date range parameters (postedFrom, postedTo)
  • Response due date filter (nextMonth)

Why this step is needed: Prepares structured search queries for SAM.gov that target accessibility-related procurement opportunities with specific filters for active opportunities and 8(a) business set-asides.

Search Filters Applied:

  • Status: Active opportunities only
  • Response Due: Next month
  • Set-Aside: 8(a) Set-Aside and 8(a) Sole Source
  • Sort: By modified date (newest first)

Step 4: Return encoded url1​

What happens here: URL-encodes the SAM.gov search URLs to ensure they can be safely passed to the scraping service.

What input it uses: SAM.gov search URLs from the previous step.

What output it produces: URL-encoded versions of the search URLs along with the original keyword.

Why this step is needed: Ensures URLs are properly formatted for HTTP requests and prevents errors from special characters in the search query parameters.

Step 5: Scrapping bee1​

What happens here: Uses ScrapingBee API to scrape the SAM.gov search results pages. Renders JavaScript and waits for page content to load before extracting HTML.

What input it uses:

  • Encoded SAM.gov search URL
  • ScrapingBee API key
  • Render JavaScript: enabled
  • Wait time: 2000ms

What output it produces: Raw HTML content from the SAM.gov search results page.

Why this step is needed: SAM.gov requires JavaScript rendering to display search results. ScrapingBee handles this and bypasses anti-scraping measures, allowing reliable data extraction.

API Endpoint: https://app.scrapingbee.com/api/v1

Method: GET

Authentication: API Key in URL parameter

Step 6: HTML1​

What happens here: Extracts specific HTML content from the scraped pages using CSS selectors. Targets headings, paragraphs, and published date fields.

What input it uses: HTML content from ScrapingBee.

What output it produces: Array of extracted text content including titles, descriptions, and published dates.

Why this step is needed: Isolates relevant content from the full HTML page, making it easier to parse procurement opportunity details in the next step.

CSS Selectors Used:

  • Headings: h1, h2, h3, h4, h5, h6
  • Paragraphs: p
  • Published Date: div.sds-field.sds-field--stacked:has(div.sds-field__name:contains("Published Date")) div.sds-field__value

Step 7: Extract required data1​

What happens here: Parses the extracted HTML content to identify and extract key procurement opportunity details including title, notice ID, objective, publication date, and SAM.gov URL.

What input it uses: Array of text content from HTML extraction.

What output it produces: Structured data objects containing:

  • Title (first meaningful line with 3+ words)
  • Notice ID (extracted from "Notice ID:" pattern)
  • Objective (extracted from "Purpose" or "Objective" sections)
  • Publication date (formatted as "Mon DD, YYYY")
  • SAM.gov URL (constructed from workspace path)

Why this step is needed: Transforms unstructured HTML text into structured data that can be stored in Google Sheets and used for reporting.

Data Extraction Logic:

  • Filters out common SAM.gov boilerplate text
  • Extracts SAM.gov workspace paths from brackets
  • Constructs full URLs from paths
  • Uses regex patterns to find dates and IDs

Step 8: Check Data Exists​

What happens here: Checks whether any meaningful procurement data was found by examining the extracted results. Determines if the workflow should proceed with data storage or send a "no data found" notification.

What input it uses: Extracted procurement data from previous step.

What output it produces: Object containing:

  • Results array
  • hasData boolean flag (true if valid opportunities found)

Why this step is needed: Prevents unnecessary processing when no opportunities are found and enables conditional routing to appropriate notification paths.

Step 9: If​

What happens here: Conditional branching based on whether procurement data was found.

What input it uses: hasData boolean flag from previous step.

What output it produces:

  • True path: Continues to data cleaning and storage if data exists
  • False path: Sends "no data found" email notification

Why this step is needed: Ensures appropriate handling for both scenarios - when opportunities are found and when searches return no results.

Step 10: Data cleaning1​

What happens here: Cleans and normalizes the extracted procurement data. Removes unwanted text, extracts links, and formats fields for Google Sheets storage.

What input it uses: Raw extracted data from previous steps.

What output it produces: Cleaned data objects with:

  • Cleaned title (newlines removed, workspace paths removed)
  • Notice ID
  • Cleaned objective (newlines removed)
  • Full SAM.gov link
  • Keyword
  • Publication date

Why this step is needed: Ensures data quality and consistency before storing in Google Sheets, making reports more readable and professional.

Step 11: Format Data for Sheets​

What happens here: Further data cleaning and formatting, ensuring all fields are properly structured for Google Sheets append operation.

What input it uses: Cleaned data from previous step.

What output it produces: Final formatted data ready for Google Sheets.

Why this step is needed: Provides additional data validation and ensures compatibility with Google Sheets column mapping.

Step 12: Create sheet1​

What happens here: Creates a new Google Sheet with a date-based name for storing the day's procurement opportunities.

What input it uses:

  • Current date from Generate Current Date
  • Google Sheets document ID (parent spreadsheet)

What output it produces: New sheet within the parent Google Sheets document with:

  • Sheet name: "n8n-sheet YYYY-MM-DD FinalSheet-testing..."
  • Sheet ID for reference

Why this step is needed: Organizes procurement data by date, making it easy to track daily discoveries and maintain historical records.

Google Sheets Document ID: 1aVbkfuEb8qNDNXZi8odnuP3JKso1bNaZVSxoxXPk4Fw

Step 13: Merge2​

What happens here: Merges data from two input streams: the cleaned procurement data and the newly created sheet information.

What input it uses:

  • Cleaned procurement data
  • Sheet creation response (sheet ID, spreadsheet ID)

What output it produces: Combined data objects ready for Google Sheets append operation.

Why this step is needed: Combines procurement opportunity data with sheet metadata so data can be written to the correct sheet.

Step 14: Merge Sheet Data​

What happens here: Merges all input items into a single consolidated object, combining sheet metadata with procurement data.

What input it uses: Multiple items from Merge2.

What output it produces: Single merged object with all fields combined.

Why this step is needed: Ensures data is in the correct format for the Google Sheets append operation.

Step 15: Append or update row in sheet1​

What happens here: Writes the cleaned procurement opportunity data to the newly created Google Sheet. Uses auto-mapping to match data fields to sheet columns.

What input it uses:

  • Sheet name (from created sheet)
  • Cleaned procurement data
  • Spreadsheet ID

What output it produces: Confirmation of data written to Google Sheets.

Why this step is needed: Persists the discovered procurement opportunities in a structured format for review and analysis.

Operation: Append or Update

Column Mapping: Auto-mapped based on data fields (title, notice_id, objective, link, keyword, publication_date)

Step 16: Delete rows or columns from sheet1​

What happens here: Removes the first 6 columns from the sheet that contain metadata (spreadsheetId, sheetId, etc.) that aren't needed in the final report.

What input it uses: Sheet name and spreadsheet ID.

What output it produces: Sheet with cleaned columns containing only procurement data.

Why this step is needed: Removes technical metadata columns, keeping only the relevant procurement opportunity information for cleaner reports.

Columns Deleted: 6 columns (metadata fields)

Step 17: Merge3​

What happens here: Merges the sheet information with date information for email reporting.

What input it uses:

  • Sheet metadata (spreadsheet ID, sheet ID)
  • Date information

What output it produces: Combined object with all information needed for email notification.

Why this step is needed: Prepares all necessary data (sheet link, date) for the email notification step.

Step 18: Add Date to First Item​

What happens here: Adds the current date to the first item in the data stream for email template use.

What input it uses: Merged data from previous steps.

What output it produces: Data object with date field added.

Why this step is needed: Ensures the email template has access to the date for personalized reporting.

Step 19: Merge All Data for Email​

What happens here: Final merge of all data fields (date, spreadsheet ID, sheet ID) into a single object for email template rendering.

What input it uses: Data from previous merge operations.

What output it produces: Complete data object with all fields needed for email.

Why this step is needed: Consolidates all information into a single object that the email template can access.

Step 20: Send an email based on a template1​

What happens here: Sends an HTML-formatted email notification to stakeholders with a link to the Google Sheet containing the day's procurement opportunities.

What input it uses:

  • Date
  • Spreadsheet ID
  • Sheet ID
  • Email template

What output it produces: Email delivery confirmation.

Why this step is needed: Notifies team members when new procurement opportunities are discovered and provides direct access to the detailed data.

Email Service: AWS SES

From Email: support@wallyax.com

To Email: vmoola@fleetstudio.com

Email Template: HTML template with Google Sheets link.

Sample Email Template (illustrative): Email Template with Sheet Link

Sample Google Sheet: The email includes a link to the Google Sheet. Example sheet link format: https://docs.google.com/spreadsheets/d/{{spreadsheetId}}/edit?gid={{sheetId}}

Step 21: Send an email based on a template (No Data Path)​

What happens here: Sends a notification email when no procurement opportunities are found for the searched keywords.

What input it uses: Date and keyword list.

What output it produces: Email delivery confirmation.

Why this step is needed: Keeps stakeholders informed even when no opportunities are found, ensuring transparency in the monitoring process.

Email Service: AWS SES

From Email: support@wallyax.com

To Email: vmoola@fleetstudio.com

Email Template: HTML template indicating no data found.

Sample Email Template (illustrative): Email Template No Data

4. Workflow Logic Summary​

The workflow executes the following sequence from start to finish:

  1. Triggers automatically at 6:30 PM IST daily via schedule trigger
  2. Gets current date for sheet naming and reporting
  3. Defines keywords for accessibility-related searches (Section 508, WCAG, EN 301 549, digital accessibility)
  4. Creates SAM.gov search URLs with filters for active opportunities and 8(a) set-asides
  5. Encodes URLs for safe HTTP transmission
  6. Scrapes SAM.gov pages using ScrapingBee to handle JavaScript rendering
  7. Extracts HTML content using CSS selectors
  8. Parses procurement data (title, notice ID, objective, publication date, links)
  9. Checks if data exists to determine next steps
  10. Conditional branching:
    • If data found: Cleans data → Creates Google Sheet → Appends data → Deletes metadata columns → Sends success email with sheet link
    • If no data: Sends "no data found" notification email
  11. Sends email notifications to stakeholders with results

This automated process ensures consistent monitoring of SAM.gov for accessibility-related procurement opportunities and timely notification to stakeholders.

5. Diagram / Screenshot​

Wally SAM.GOV Unauthorised Workflow Diagram

The workflow diagram shows the complete automation flow from scheduled trigger through keyword processing, SAM.gov scraping, data extraction, Google Sheets storage, and email notification processes.

6. Dependencies​

This workflow depends on several external APIs, third-party services, and cloud platforms to function properly.

External APIs​

ScrapingBee API

  • Endpoint: https://app.scrapingbee.com/api/v1
  • Method: GET
  • Purpose: Web scraping service that handles JavaScript rendering and bypasses anti-scraping measures
  • Requires Auth: Yes (API Key in URL parameter)
  • API Key: Stored in n8n credentials
  • Usage: Scrapes SAM.gov search results pages that require JavaScript rendering
  • Configuration:
    • Render JavaScript: enabled
    • Wait time: 2000ms
    • Accept header: text/html

SAM.gov Website

  • URL: https://sam.gov/search/?index=opp&page=1&pageSize=25&sort=-modifiedDate&sfm%5BsimpleSearch%5D%5BkeywordTags%5D%5B0%5D%5Bkey%5D=accessibility&sfm%5BsimpleSearch%5D%5BkeywordTags%5D%5B0%5D%5Bvalue%5D=${encodedKeyword}&sfm%5BsimpleSearch%5D%5BkeywordRadio%5D=ALL&sfm%5Bstatus%5D%5Bis_active%5D=true&sfm%5Bstatus%5D%5Bis_inactive%5D=false&sfm%5Bdates%5D%5BresponseDue%5D%5BresponseDueSelect%5D=${DateOffersDue}&sfm%5BsetAside%5D%5B0%5D%5Bkey%5D=8A&sfm%5BsetAside%5D%5B0%5D%5Bvalue%5D=8(a)%20Set-Aside%20(FAR%2019.8)&sfm%5BsetAside%5D%5B1%5D%5Bkey%5D=8AN&sfm%5BsetAside%5D%5B1%5D%5Bvalue%5D=8(a)%20Sole%20Source%20(FAR%2019.8)
  • Purpose: Source of procurement opportunity data with pre-configured search filters
  • Access: Public (no authentication required)
  • Usage: Searches for accessibility-related procurement opportunities with specific filters:
    • Index: Opportunities (index=opp)
    • Sort: By modified date, newest first (sort=-modifiedDate)
    • Status: Active opportunities only (is_active=true, is_inactive=false)
    • Response Due: Next month (responseDueSelect=nextMonth)
    • Set-Aside: 8(a) Set-Aside and 8(a) Sole Source opportunities
    • Keyword Search: Searches within accessibility keyword tags
  • Note: Requires JavaScript rendering to display search results. The URL includes encoded query parameters for filtering opportunities.

Third-Party Services​

Google Sheets API

  • Purpose: Cloud-based spreadsheet storage and management
  • Usage: Stores discovered procurement opportunities in organized, date-based sheets
  • Configuration: OAuth2 authentication via n8n credentials
  • Document ID: 1aVbkfuEb8qNDNXZi8odnuP3JKso1bNaZVSxoxXPk4Fw
  • Operations Used:
    • Create sheet
    • Append or update rows
    • Delete columns
  • Credentials: Google Sheets OAuth2 API (stored in n8n)

AWS SES (Simple Email Service)

  • Purpose: Email delivery service for sending notifications
  • Usage: Sends HTML-formatted email reports to stakeholders
  • Configuration: AWS credentials stored in n8n
  • From Email: support@wallyax.com
  • To Email: vmoola@fleetstudio.com
  • Credentials: AWS account credentials (stored in n8n)

n8n Instance​

Workflow URL: https://agent.fsgarage.in/workflow/gXzJLXOJBNzsXUMx

Workflow Name: Wally: SAM.GOV - Unauthorised

Workflow ID: gXzJLXOJBNzsXUMx

Email: technology@fleetstudio.com

Password:

n8n Instance URL: https://agent.fsgarage.in

GitHub Repository​

Repository: https://github.com/TeamFleetStudio/Agentic-AI-POC

Description: Source code repository for the Agentic AI POC project, including workflow configurations and related automation scripts.