Test it on a crane specification that is public.
The Clearwater Dam crane section is a public document. Hand it to any vendor, including us, and see whether the software finds these without being told where to look.
- That the job is a rebuild of an existing crane, from the speeds table on page 5.
- The ASME NOG-1 sentence on page 9, raised as a question and not priced in silence.
- The load test at 125 percent on page 9, and the test weights of 200,000 and 250,000 pounds on page 23.
- The capacity calculations for the building on page 6, with a professional engineer's stamp.
- The lines that cannot apply to this crane: the cab receptacle on page 19, and the two cranes on one runway on page 12.
Score it the way you would score a new hire. Did it find the item? Did it cite the page? Did it ask the question, or quietly pick a side?