Skip to main content

Automated Keywords Fetch - SAM.gov Workflow

1. Purpose​

This workflow fetches keywords from external sources (typically the RFP Ranking AI system), searches SAM.gov for procurement opportunities matching those keywords, and returns structured procurement data via webhook response. It serves as an API endpoint for the RFP Ranking AI system to discover accessibility-related procurement opportunities from SAM.gov.

2. Trigger​

Type: Webhook

Details:

The workflow is triggered via HTTP POST request to a webhook endpoint with keywords passed as query parameters.

Webhook Endpoint: /fetch-keywords

Webhook ID: 76b51f54-ab55-4ce4-8916-3a92933eb20e

Method: POST

Query Parameters: keywords (comma-separated list of keywords to search)

Example Request:

POST /fetch-keywords?keywords=Section%20508,WCAG,accessibility

3. Step-by-Step Process​

Step 1: Webhook​

What happens here: Receives HTTP POST requests with keywords as query parameters from the RFP Ranking AI system backend. The workflow automatically fetches and adds keywords based on website text passed from the backend system. This is an automated process that extracts relevant keywords from the provided website content and uses them to search SAM.gov. The final response from this workflow (processed in Step 10: Respond to Webhook) will be stored by the Respond to Webhook node and made available to the calling backend system.

What input it uses: Query parameters containing comma-separated keywords. The backend system sends website text, and keywords are automatically extracted and added to the search process.

What output it produces: Webhook request data with query parameters containing the extracted keywords.

Why this step is needed: Provides the entry point for the workflow, allowing the RFP Ranking AI system to trigger SAM.gov searches with keywords automatically extracted from website content. The automated keyword extraction and search process enables seamless integration with the backend system. The response from the "Respond to Webhook" node (Step 10) is stored and can be retrieved by the calling backend system for further processing.

Step 2: Parse Keywords from Query​

What happens here: Parses the keywords query parameter, splits comma-separated values, trims whitespace, and creates a clean array of keywords.

What input it uses: Query parameters from webhook request (keywords parameter).

What output it produces: Array of cleaned keywords ready for processing.

Why this step is needed: Converts the query parameter string into a structured array format that can be processed by subsequent workflow steps.

Step 3: Search keyword​

What happens here: Constructs SAM.gov search URLs for each keyword. Creates filtered search queries targeting 8(a) Set-Aside and Sole Source opportunities with specific date ranges and filters.

What input it uses: Keywords array from previous step.

What output it produces: Individual items for each keyword containing:

  • Keyword name
  • SAM.gov search URL with filters
  • Date range parameters (postedFrom, postedTo)
  • Response due date filter (nextMonth)

Why this step is needed: Prepares structured search queries for SAM.gov that target accessibility-related procurement opportunities with specific filters for active opportunities and 8(a) business set-asides.

Search Filters Applied:

  • Status: Active opportunities only
  • Response Due: Next month
  • Set-Aside: 8(a) Set-Aside and 8(a) Sole Source
  • Sort: By modified date (newest first)

Step 4: Return encoded url​

What happens here: URL-encodes the SAM.gov search URLs to ensure they can be safely passed to the scraping service.

What input it uses: SAM.gov search URLs from the previous step.

What output it produces: URL-encoded versions of the search URLs along with the original keyword.

Why this step is needed: Ensures URLs are properly formatted for HTTP requests and prevents errors from special characters in the search query parameters.

Step 5: Limit Results to Last 5​

What happens here: Limits the number of search URLs to process, taking only the first 5 items from the array to optimize response time and processing efficiency.

What input it uses: Array of encoded URLs and keywords.

What output it produces: First 5 items from the input array.

Why this step is needed: Optimizes workflow performance by limiting the number of SAM.gov pages to scrape, ensuring faster response times for the webhook caller.

Step 6: Scrapping bee​

What happens here: Uses ScrapingBee API to scrape the SAM.gov search results pages. Renders JavaScript and waits for page content to load before extracting HTML.

What input it uses:

  • Encoded SAM.gov search URL
  • ScrapingBee API key
  • Render JavaScript: enabled
  • Wait time: 2000ms

What output it produces: Raw HTML content from the SAM.gov search results page.

Why this step is needed: SAM.gov requires JavaScript rendering to display search results. ScrapingBee handles this and bypasses anti-scraping measures, allowing reliable data extraction.

API Endpoint: https://app.scrapingbee.com/api/v1

Method: GET

Authentication: API Key in URL parameter

Step 7: HTML​

What happens here: Extracts specific HTML content from the scraped pages using CSS selectors. Targets headings, paragraphs, and published date fields.

What input it uses: HTML content from ScrapingBee.

What output it produces: Array of extracted text content including titles, descriptions, and published dates.

Why this step is needed: Isolates relevant content from the full HTML page, making it easier to parse procurement opportunity details in the next step.

CSS Selectors Used:

  • Headings: h1, h2, h3, h4, h5, h6
  • Paragraphs: p
  • Published Date: div.sds-field.sds-field--stacked:has(div.sds-field__name:contains("Published Date")) div.sds-field__value

Step 8: Extract required data​

What happens here: Parses the extracted HTML content to identify and extract key procurement opportunity details including title, notice ID, objective, publication date, and SAM.gov URL.

What input it uses: Array of text content from HTML extraction.

What output it produces: Structured data objects containing:

  • Title (first meaningful line with 3+ words)
  • Notice ID (extracted from "Notice ID:" pattern)
  • Objective (extracted from "Purpose" or "Objective" sections)
  • Publication date (formatted as "Mon DD, YYYY")
  • SAM.gov URL (constructed from workspace path)
  • Links array

Why this step is needed: Transforms unstructured HTML text into structured data that can be returned to the webhook caller in JSON format.

Data Extraction Logic:

  • Filters out common SAM.gov boilerplate text
  • Extracts SAM.gov workspace paths from brackets
  • Constructs full URLs from paths
  • Uses regex patterns to find dates and IDs

Step 9: Check Data Exists and Prepare Response​

What happens here: Validates whether any meaningful procurement data was found and structures the response data for the webhook response.

What input it uses: Extracted procurement data from previous step.

What output it produces: Object containing:

  • Results array with all extracted opportunities
  • hasData boolean flag (true if valid opportunities found)

Why this step is needed: Ensures data quality before responding and provides metadata about whether opportunities were found.

Step 10: Respond to Webhook​

What happens here: Returns the procurement search results as JSON response to the webhook caller (RFP Ranking AI system). The response is stored by this node and made available to the calling backend system.

What input it uses: Processed procurement data with validation results.

What output it produces: HTTP response with JSON data containing procurement opportunities. The response is stored and can be retrieved by the backend system.

Why this step is needed: Provides API-like functionality, allowing the RFP Ranking AI system to retrieve procurement data programmatically for ranking and analysis. The stored response enables the backend to process the results asynchronously.

Response Format: JSON object with results array and hasData boolean flag

Response Storage: The response from this node is stored and can be accessed by the calling backend system for further processing.

4. Workflow Logic Summary​

The workflow executes the following sequence from start to finish:

  1. Receives webhook request with custom keywords from RFP Ranking AI system
  2. Parses keywords from query parameters into a clean array
  3. Creates SAM.gov search URLs for each keyword with filters for active opportunities and 8(a) set-asides
  4. Encodes URLs for safe HTTP transmission
  5. Limits to first 5 keywords for optimized processing
  6. Scrapes SAM.gov pages using ScrapingBee to handle JavaScript rendering
  7. Extracts HTML content using CSS selectors
  8. Parses procurement data (title, notice ID, objective, publication date, links)
  9. Validates data and prepares response structure
  10. Returns JSON response with procurement opportunities to the webhook caller

This automated process enables the RFP Ranking AI system to discover and retrieve accessibility-related procurement opportunities from SAM.gov on-demand, supporting automated RFP ranking and analysis workflows.

5. Diagram / Screenshot​

Automated Keywords Fetch SAM.gov Workflow Diagram

The workflow diagram shows the complete automation flow from webhook trigger through keyword parsing, SAM.gov search URL construction, scraping, data extraction, and JSON response delivery.

6. Dependencies​

This workflow depends on several external APIs and third-party services to function properly.

External APIs​

ScrapingBee API

  • Endpoint: https://app.scrapingbee.com/api/v1
  • Method: GET
  • Purpose: Web scraping service that handles JavaScript rendering and bypasses anti-scraping measures
  • Requires Auth: Yes (API Key in URL parameter)
  • API Key: Stored in n8n credentials
  • Usage: Scrapes SAM.gov search results pages that require JavaScript rendering
  • Configuration:
    • Render JavaScript: enabled
    • Wait time: 2000ms
    • Accept header: text/html

SAM.gov Website

  • URL: https://sam.gov/search/?index=opp&page=1&pageSize=25&sort=-modifiedDate&sfm%5BsimpleSearch%5D%5BkeywordTags%5D%5B0%5D%5Bkey%5D=accessibility&sfm%5BsimpleSearch%5D%5BkeywordTags%5D%5B0%5D%5Bvalue%5D=${encodedKeyword}&sfm%5BsimpleSearch%5D%5BkeywordRadio%5D=ALL&sfm%5Bstatus%5D%5Bis_active%5D=true&sfm%5Bstatus%5D%5Bis_inactive%5D=false&sfm%5Bdates%5D%5BresponseDue%5D%5BresponseDueSelect%5D=${DateOffersDue}&sfm%5BsetAside%5D%5B0%5D%5Bkey%5D=8A&sfm%5BsetAside%5D%5B0%5D%5Bvalue%5D=8(a)%20Set-Aside%20(FAR%2019.8)&sfm%5BsetAside%5D%5B1%5D%5Bkey%5D=8AN&sfm%5BsetAside%5D%5B1%5D%5Bvalue%5D=8(a)%20Sole%20Source%20(FAR%2019.8)
  • Purpose: Source of procurement opportunity data with pre-configured search filters
  • Access: Public (no authentication required)
  • Usage: Searches for accessibility-related procurement opportunities with specific filters:
    • Index: Opportunities (index=opp)
    • Sort: By modified date, newest first (sort=-modifiedDate)
    • Status: Active opportunities only (is_active=true, is_inactive=false)
    • Response Due: Next month (responseDueSelect=nextMonth)
    • Set-Aside: 8(a) Set-Aside and 8(a) Sole Source opportunities
    • Keyword Search: Searches within accessibility keyword tags, where ${encodedKeyword} is dynamically replaced with the keyword extracted from the backend website text
  • Note: Requires JavaScript rendering to display search results. The URL includes encoded query parameters for filtering opportunities. The keyword parameter is automatically populated from the website text provided by the backend system.

Backend Server APIs​

RFP Ranking AI System

  • Purpose: External system that triggers this workflow via webhook
  • Usage: Sends keyword requests and receives procurement opportunity data
  • Integration: Webhook-based integration for on-demand procurement data retrieval

n8n Instance​

Workflow URL: https://agent.fsgarage.in/workflow/6rfOQrAfnMUf9FLK

Workflow Name: Automated keywords fetch - sam.gov

Workflow ID: 6rfOQrAfnMUf9FLK

Email: technology@fleetstudio.com

Password:

n8n Instance URL: https://agent.fsgarage.in

GitHub Repository​

Repository: https://github.com/TeamFleetStudio/Agentic-AI-POC

Description: Source code repository for the Agentic AI POC project, including workflow configurations and related automation scripts.