Accuracy
How accurate is the solicitation reading?
We picked 30 real, open SAM.gov solicitations across 13 trades, read every posted file by hand, wrote down the answers to the questions that decide a bid, and scored OppBeacon against them. Then we fixed the biggest causes of misses and ran it again, and finally tested it, with no further changes, on 20 fresh solicitations it had never seen.
Results (2026-11-03)
| Field | Dev set: before fixes | Dev set: after fixes | Fresh set (20, untuned) | Fresh set: wrong / missed / extra |
|---|---|---|---|---|
| Warns when the package is incomplete | 7/8 (87.5%) | 8/8 (100%) | 4/4 (100%) | 0 / 0 / 5 |
| Set-aside / eligibility | 29/29 (100%) | 29/29 (100%) | 19/19 (100%) | 0 / 0 / 0 |
| NAICS code | 30/30 (100%) | 30/30 (100%) | 20/20 (100%) | 0 / 0 / 0 |
| Size standard | 18/19 (94.7%) | 18/19 (94.7%) | 10/11 (90.9%) | 0 / 1 / 0 |
| How and where to submit | 21/23 (91.3%) | 23/23 (100%) · 1 extra | 16/17 (94.1%) | 1 / 0 / 0 |
| Response deadline — date | 29/30 (96.7%) | 30/30 (100%) | 19/20 (95%) | 1 / 0 / 0 |
| Response deadline — time | 27/29 (93.1%) | 29/29 (100%) | 19/20 (95%) | 0 / 1 / 0 |
| Response deadline — time zone | 26/29 (89.7%) | 29/29 (100%) | 19/20 (95%) | 0 / 1 / 0 |
| Required submission items (recall) | 67/81 (82.7%) | 79/81 (97.5%) | 54/57 (94.7%) | 0 / 3 / 0 |
| Site visit | 12/12 (100%) | 12/12 (100%) | 8/9 (88.9%) | 1 / 0 / 0 |
| Questions deadline | 18/20 (90%) | 19/20 (95%) | 11/13 (84.6%) | 1 / 1 / 0 |
Critical misses (a wrong or missing set-aside, NAICS, deadline date or submission address; a missed bid bond, limitations-on-subcontracting certificate or IEE representation; a missed mandatory site visit; or no warning on an incomplete package): dev set 5 before → 0 after fixes; fresh set 2 of 20.
Fresh-set critical misses: submission method (wrong); response deadline (wrong). Both came from one notice whose solicitation is a scanned image: OppBeacon read an older extension date from the Q&A file (it did flag that SAM.gov shows a different date) and did not pick up the email address from the scan. The weakest field on the fresh set is the “documents withheld” warning: it raised 5 false alarms on complete packages, so treat that warning as a prompt to check, not a fact.
What we fixed
- Files attached inside a PDF (the paperclip kind) are now opened and read, like any other attachment.
- Dates written without a year, or offer deadlines without a time or time zone, are checked against the SAM.gov response-date field; a disagreement is shown, not hidden.
- “Submit by email to the Contract Specialist” now resolves to the actual address in the documents, and every required recipient is listed.
- A bid bond, a limitations-on-subcontracting certificate or an Indian Economic Enterprise representation found in the documents is added to the submission checklist with its quote and page.
- Fewer false “documents withheld” warnings, and a new one when the notice mentions a file (“a new SF 1449 has been uploaded”) that isn’t actually posted.
How we scored
- Correct means the same value as the documents (for set-asides, the same category; for the deadline, the same day, then time and zone separately). “Extra” means OppBeacon reported something the documents don’t contain.
- Required items are scored on recall only: our hand list is the items a solicitation explicitly asks for, not every possible form.
- Five notices post their documents only on PIEE; for those only the SAM.gov fields and the “incomplete package” warning are scored.
- The “now” column is measured on the same 30 notices we used to find the fixes, so it is optimistic. Ten of them were re-read with the updated instructions; the other twenty keep their first reading plus the rule checks. The fresh-set column is the honest number: 20 different solicitations (similar trade mix), answer keys written from the documents before looking at OppBeacon’s output, and no tuning afterwards.
- After the first run we re-checked every disagreement against the documents and corrected seven answers in our own key where OppBeacon had been right or our key was incomplete — before making any fix. On the fresh set we corrected only an encoding slip in our own key (“no set-aside” written as blank) and a scorer bug that read “Tier 1 SDVOSB … priority to WOSB” as WOSB; OppBeacon’s answers were not changed.