How We Measure AI Recommendations for Property Management Software

Published at /ai-shortlist-index/methodology/. Source of truth for how every number on the Index is produced. Every published figure traces back to a raw run file under index-engine/runs/<edition>/; nothing is hand-edited after capture.

The firewall: Paying WebPossible never affects Index placement. The Index measures what AI engines say. Shortlist clients buy the work that earns a position, not the position.

1. What the Index measures

For each launch category we ask four AI answer engines the questions a property manager actually asks when choosing software, three times each on separate days, and record which vendors the engines name, in what order, with what framing, and which web pages they cite while doing it. The Index is a measurement of engine output, not an editorial opinion about which product is best.

Launch categories (three): resident benefits packages, multifamily access control, maintenance coordination and vendor networks. A category is added only when at least ten viable vendors exist and prompts show real buyer demand. Additions are logged in the changelog.

2. Prompt design and counts

Each category has exactly 25 prompts across six types:

Type	Count	What it captures
best / shortlist	7	open "what's the best…" buying questions
alternatives	5	"alternatives to <named vendor>"
for-segment	4	SFR, small multifamily, student, affordable, etc.
integration	3	"…that integrates with <PMS>"
problem-led	4	the operational pain, no product words
comparison	2	"<vendor A> vs <vendor B>"

Prompts are written the way a PM asks: first person, one question, a real portfolio context, US and unpersonalized. They are frozen per edition and versioned as a set (current: Prompt Set v1.0, approved 2026-09-02). Changing a prompt's text creates a new prompt id and a changelog entry; retired prompts stay in the file with a retired edition. The full library is public at /ai-shortlist-index/prompt-library/. Cite as "AI Shortlist Index Prompt Set v1.x, WebPossible." Design rules: docs/prompt-design.md.

Seed-vendor exclusion

alternatives and comparison prompts name vendors because a buyer would. A vendor named in the prompt text is recorded when the engine mentions it, but that mention does not count toward the vendor's mention rate, position score, consistency or rank for that prompt. Otherwise "alternatives to X" would hand X a free mention every run. The mention is stored with seeded: true so the exclusion is auditable.

3. Engines and exact capture method

ChatGPT is captured from its consumer web app. The other three engines are queried through documented APIs or a licensed data provider. Each engine's row below states which, and every stored run records its own collection method, so no reader has to guess which is which.

Engine	Capture	What is stored
ChatGPT	The consumer web app at chatgpt.com, signed in to an account used only for the Index, one temporary chat per prompt, driven by browser automation in a headed session. Default model selection; whether to search the web is left to ChatGPT. Account memory, custom instructions and chat-history search are off.	answer text, every link in the answer, the untouched capture, the capture time. The web app does not disclose which model answered, so no model id, token count or price is recorded, and the run costs nothing.
Perplexity	Perplexity API, chat completions, sonar model (the consumer default search model).	answer text, citations, search_results (url, title, date), usage, cost
Gemini	Gemini API with Google Search grounding enabled (2026-09: gemini-3.8-flash).	answer text, grounding chunks (redirect URIs resolved to final URLs), grounding supports, web search queries, usage, cost
Google AI Overviews	US, non-personalized, desktop Google SERP for the prompt text, captured through a licensed third-party SERP data provider (Apify Google Search Results Scraper). AI Overview text and its cited links are extracted; whether an AI Overview appeared at all is recorded.	AIO text and sources, organic top results, aio_present flag, cost

The Google AI Overviews engine has no first-party fallback. If the provider fails, the engine ships as degraded for that edition.

Why ChatGPT is captured from the app. The app is what buyers use, and it is the only way to read the consumer product rather than an API approximation of it. The trade-offs are stated rather than hidden: the app names no model, so a model change can move a ranking with nothing else having changed; capture runs on one operator's machine rather than on a schedule; and OpenAI's terms cover automated use of its services, a risk carried by the Index, never by a vendor it measures. Answers collected this way are written into the same run files as every other engine, each carrying collection: chatgpt_web_app.

What is removed from a captured answer. ChatGPT renders each source as a chip: a short label naming the site, sometimes followed by a +N counter. Those labels name sources, not vendors the answer recommended, so they are removed before the text is stored and counted. The untouched capture is kept in the same run file, so any removal can be checked. Nothing else is edited.

Personalization. A run is invalid if the answer shows the account behind it: advice addressed to the operator's own business, or a location no prompt supplied. Every capture is checked for both, and a contaminated set is discarded and re-run rather than published. This is not hypothetical: the first resident-benefits capture of 2026-09 was discarded because ChatGPT's chat-history search had leaked the operator's client work into 4 of 25 answers.

Degraded, never dropped. Adapters are modular. If an engine has no credential, its API breaks, or a call fails, the edition still publishes with that engine marked degraded and the reason stated. Degraded engines are excluded from the denominators (a vendor's blended mention rate averages only engines that actually answered). Failed calls are stored as failure records; they never become silent zeros.

4. Run counts and windows

• 3 runs per prompt per engine, each in a fresh session (no shared conversation state; each run has its own session_id).
• Runs are spread across at least 2 days: the scheduler runs on the 2nd, 4th and 6th of each month at 14:00 UTC, one run index per day. ChatGPT is the exception, because a browser session cannot run unattended on a server: its three runs are captured by hand on three days at least two days apart, and imported with the capture time each answer was collected.
• Launch volume per edition: 3 runs × 25 prompts × 4 engines × 3 categories = 900 responses.
• Engine defaults are used for temperature and other sampling parameters.
• A verification step checks every prompt×engine cell has 3 runs, on ≥2 distinct days, in distinct sessions, before an edition may publish.

Provenance and retention

Every response is stored as runs/<YYYY-MM>/<category>/<engine>/<prompt_id>_run<n>.json containing: prompt id and exact text, prompt set version, engine and model identifier, timestamp (UTC), session id, answer text, every cited/consulted URL, token usage, cost in USD, and the complete raw API response. A ChatGPT run carries the same fields, with no model id, no token count and a cost of 0.00, plus the untouched capture and the name of the capture file it came from. Files are written once and never modified. A re-run writes the next run index; an existing file is skipped. Raw responses are retained indefinitely as the dispute audit trail. Per-engine cost is logged to runs/<YYYY-MM>/costs.csv.

5. Extraction rules

Extraction is deterministic code, re-runnable against the raw files.

• Named. A vendor is named when its canonical name or a listed alias appears in the answer text. Matching folds diacritics, tolerates a space or hyphen between name tokens ("ButterflyMX" / "Butterfly MX"), accepts possessives, and ignores markdown emphasis around the name. Names that are also common English words (DOOR, Latch, Level, Alloy, Meld…) match case-sensitively, and a sentence-initial common-word name used as a verb ("Latch the gate…") is not counted.
• Position. Ordinal of the vendor's first mention among all vendors named in the answer (1 = named first).
• Sentiment. recommended / neutral / cautioned, from recommendation and caution cue words within a 180-character window of the first mention. This is a coarse framing signal, not a review score, and is displayed as such.
• Context snippet. ~240 characters around the first mention, stored for audit.
• Cited sources. Every cited or consulted URL is normalized (tracking parameters such as utm_source removed), classified by page type (vendor site, review platform, listicle, trade publication, directory, forum, association, WebPossible-operated site, other), and linked to the vendor(s) whose domain it is.
• Unknown names. Capitalized names in list, heading or bold positions that match no known vendor, candidate, or property-management-system name go to a human review queue (review/<edition>/unknown-vendors.yaml). A name enters data/vendors.yaml only after a human confirms it, with a source_url and last_verified date. An edition cannot publish while review items are pending.
• Accuracy gate. Extraction is validated against a hand-labeled set of ≥50 responses; it must reach ≥95% agreement on named vendors before metrics publish. The labeled set is never edited to raise the score. Additionally, 50 responses per edition are spot-checked by hand.

6. Metric definitions

All metrics are per vendor, per category, per edition. Seeded mentions are excluded (see §2). Degraded engines are excluded from denominators, not counted as zero.

Metric	Definition
Mention rate (per engine)	Runs in which the vendor was named ÷ ok runs for that engine, over prompts where the vendor is not seeded.
Mention rate (blended)	Mean of per-engine mention rates across engines that returned ok runs.
Shortlist rank	Ordinal by blended mention rate (ties: better position score, then wider engine spread, then name). Vendors never named have no rank.
Position score	Mean first-mention position across runs where the vendor was named (1.0 = always first). Lower is better.
Consistency	Share of prompt×engine cells (with ≥2 runs) where all runs agreed (all named or none named). 1.0 = fully consistent; low values mean the engine flips between runs. Displayed, never hidden.
Engine spread	Number of ok engines (0–4) with a non-zero mention rate.
Source dependency	Top cited domains on the runs where the vendor was named — the pages its mentions ride on.
Movement	Prior-edition rank minus current rank (positive = moved up). Null when there is no prior edition or the vendor was unranked before.

Source Map (per category). Top 20 cited URLs and top 20 cited domains by citation count across ok runs, each with page type, the engines that cited it, and the vendors it supports. When webpossible.com or a WebPossible Media site appears, it is flagged own_network and disclosed inline. Known pay-to-play listicles are flagged, not hidden.

7. Known limitations

• Non-determinism. Engines give different answers to the same prompt. Three runs and the consistency metric make the variance visible; they do not remove it.
• Mixed instruments. ChatGPT is read from the consumer app; the other three are read through APIs, which are the closest repeatable proxy for their apps but are not identical to them (different system prompts, memory, personalization). The ChatGPT column therefore answers a slightly different question from the other three, and no claim of parity is made in either direction.
• No personalization. Runs use no account history or location beyond "US", and the ChatGPT account runs with memory, custom instructions and chat-history search off. Real users get personalized answers; the Index measures the unpersonalized baseline. Every ChatGPT capture is checked for answers that address the operator's own business or assume a location no prompt supplied; a contaminated set is discarded and re-run.
• US only. Prompts, location signals and SERP captures are US English.
• Model drift. Engines change models without notice. The model identifier returned by each API is stored per run and shown per edition; month-to-month movement may reflect a model change rather than anything a vendor did. ChatGPT is the blind spot: the app names no model, so a ChatGPT row can move because OpenAI changed the default, with nothing in the run file to show it.
• Extraction is lexical. A vendor mentioned only by a nickname we have not listed as an alias is missed until the review queue surfaces it. Sentiment is cue-based.
• Google AI Overviews are third-party captured and may not appear for every query; absence is recorded as "no AI Overview", not as a zero mention.

8. Attribution and terms

Every result is labeled with the instrument that produced it: the ChatGPT web app for ChatGPT, vendor APIs or a licensed data provider for the rest. Attribution lines carried on this page:

• OpenAI: Answers captured from the ChatGPT web app at chatgpt.com by automated browser capture, one temporary chat per prompt, no personalization. ChatGPT and OpenAI are trademarks of OpenAI. OpenAI's terms govern automated use of its services; that risk sits with WebPossible, never with a vendor the Index measures. Runs published before 2026-09-19 came from the OpenAI API (Responses endpoint, web search tool) and are labeled as such in the run files.
• Perplexity: Answers and citations retrieved via the Perplexity API (sonar). Perplexity is a trademark of Perplexity AI, Inc.
• Google: Gemini answers and grounding metadata retrieved via the Gemini API with Google Search grounding. Google AI Overview text captured from public Google Search results pages via a third-party data provider (Apify). Google, Gemini and Google Search are trademarks of Google LLC.
• Apify: SERP data captured with the Apify Google Search Results Scraper under Apify's terms of service.

Vendor names are used nominatively. Logos appear only with permission via a claimed row. Engine terms are reviewed each edition; if a provider's terms change in a way that bars publication, that engine ships as degraded and the change is logged.

9. Re-check and disputes

Any vendor may request one re-run of its category's prompt set per edition through "Request a re-check" on its row. Re-checks follow the same protocol (3 runs, separate days, all engines) and are published regardless of outcome, alongside the original. Entity corrections (name, aliases, category membership, website) are handled in-edition after verification against a source URL. We respond to disputes on the record, with data, on the policy page. No vendor can pay for placement, removal, or a favorable re-check.

10. Changelog policy

Every change that can move a number is logged at /ai-shortlist-index/changelog/ with the edition it takes effect: prompt additions, retirements or wording changes (new prompt set version), engine or model changes, capture-method changes, metric definition changes, vendor additions/merges/retirements, category additions, and price-table updates used for cost logging. Editions are immutable once published; corrections are published as amendments with the original preserved.

Engine and version record (2026-09)

Engine	Adapter	Model / provider
ChatGPT	chatgpt.com web app, temporary chat, browser capture	not disclosed by the app
Perplexity	Perplexity chat completions	sonar
Gemini	Gemini API + Google Search grounding	gemini-3.8-flash
Google AI Overviews	Apify Google Search Results Scraper	US desktop SERP