Published at /ai-shortlist-index/methodology/. Source of truth for how every number on the Index is produced. Every published figure traces back to a raw run file under index-engine/runs/<edition>/; nothing is hand-edited after capture.

The firewall: Paying WebPossible never affects Index placement. The Index measures what AI engines say. Clients buy the work that earns a position, not the position.

1. What the Index measures

For each launch category we ask five AI answer engines the questions a property manager actually asks when choosing software, three times each on separate days, and record which vendors the engines name, in what order, with what framing, and which web pages they cite while doing it. The Index is a measurement of engine output, not an editorial opinion about which product is best.

Launch categories (three): resident benefits packages, multifamily access control, maintenance coordination and vendor networks. A category is added only when at least ten viable vendors exist and prompts show real buyer demand. Additions are logged in the changelog.

2. Prompt design and counts

Each category has exactly 25 prompts across six types:

Type	Count	What it captures
best / shortlist	7	open "what's the best…" buying questions
alternatives	5	"alternatives to `<named vendor>`"
for-segment	4	SFR, small multifamily, student, affordable, etc.
integration	3	"…that integrates with `<PMS>`"
problem-led	4	the operational pain, no product words
comparison	2	"`<vendor A>` vs `<vendor B>`"

Prompts are written the way a PM asks: first person, one question, a real portfolio context, US and unpersonalized. They are frozen per edition and versioned as a set (current: Prompt Set v1.0, approved 2026-09-02). Changing a prompt's text creates a new prompt id and a changelog entry; retired prompts stay in the file with a retired edition. The full library is public at /ai-shortlist-index/prompt-library/. Cite as "AI Shortlist Index Prompt Set v1.x, WebPossible." Design rules: docs/prompt-design.md.

Seed-vendor exclusion

alternatives and comparison prompts name vendors because a buyer would. A vendor named in the prompt text is recorded when the engine mentions it, but that mention does not count toward the vendor's mention rate, position score, consistency or rank for that prompt. Otherwise "alternatives to X" would hand X a free mention every run. The mention is stored with seeded: true so the exclusion is auditable.

3. Engines and exact capture method

ChatGPT is captured from its consumer web app. The other four engines are queried through documented APIs or a licensed data provider. Each engine's row below states which, and every stored run records its own collection method, so no reader has to guess which is which.

Engine	Capture	What is stored
ChatGPT	The consumer web app at chatgpt.com, signed in to the operator's ChatGPT Plus account (not an account reserved for the Index), account memory off (checked 2026-10-04), one temporary chat per prompt set to Unpersonalized, which ignores memory, plugins and custom instructions, driven by browser automation in a headed session that no other process uses. Default model selection; whether to search the web is left to ChatGPT.	answer text, every link in the answer, the untouched capture, the capture time. Captures from October 2026 on also store the conversation record the page itself loads, which names the model that answered (`gpt-5-6-thinking` on every capture of 2–3 October 2026). The September captures have no model id. No token count or price is recorded, and the run costs nothing.
Claude	Anthropic Messages API with the web search tool set to an approximate US location, Claude Haiku 5.5 at medium effort, one request per prompt with the prompt text alone. Added in October 2026. The API is not the claude.ai app, so a buyer using the app may see a different answer.	answer text, the links shown in the answer, the search results the model read, the model id, usage, cost, the capture time
Perplexity	Perplexity API, chat completions, `sonar` model (the consumer default search model).	answer text, `citations`, `search_results` (url, title, date), usage, cost
Gemini	Gemini API with Google Search grounding enabled (2026-09: `gemini-3.8-flash`).	answer text, grounding chunks (redirect URIs resolved to final URLs), grounding supports, web search queries, usage, cost
Google AI Overviews	US, non-personalized, desktop Google SERP for the prompt text, captured through a licensed third-party SERP data provider (Apify Google Search Results Scraper). AI Overview text and its cited links are extracted; whether an AI Overview appeared at all is recorded.	AIO text and sources, organic top results, `aio_present` flag, cost

The Google AI Overviews engine has no first-party fallback. If the provider fails, the engine ships as degraded for that edition.

Why ChatGPT is captured from the app. The app is what buyers use, and it is the only way to read the consumer product rather than an API approximation of it. The trade-offs are stated rather than hidden: the model is read from the conversation record only since October 2026, and a model change can still move a ranking with nothing else having changed; capture runs on one operator's machine rather than on a schedule; and OpenAI's terms cover automated use of its services, a risk carried by the Index, never by a vendor it measures. Answers collected this way are written into the same run files as every other engine, each carrying collection: chatgpt_web_app.

What is removed from a captured answer. ChatGPT renders each source as a chip: a short label naming the site, sometimes followed by a +N counter. Those labels name sources, not vendors the answer recommended, so they are removed before the text is stored and counted. The untouched capture is kept in the same run file, so any removal can be checked. Nothing else is edited.

Personalization. A run is invalid if the answer shows the account behind it, such as advice addressed to the operator's own business. Every capture is checked for it, and a contaminated set is discarded and re-run rather than published. Location is different. The app passes the model an approximate location derived from the connection, and no setting removes it. The 2026-09 captures were made from Michigan, and 32 of the 425 answers captured on 2–3 October 2026 mention Michigan or Detroit, mostly where the prompt leaves the place open: 6 of 25 pest control answers recommended local exterminators, and 5 of 25 EV charging answers pointed to Michigan utility rebates. Those answers are kept and flagged in the category notes rather than discarded, because there is no location-free version of the app to compare against. Local service providers are not ranked. This is not hypothetical: the first resident-benefits capture of 2026-09 was discarded because ChatGPT's chat-history search had leaked the operator's client work into 4 of 25 answers.

Degraded, never dropped. Adapters are modular. If an engine has no credential, its API breaks, or a call fails, the edition still publishes with that engine marked degraded and the reason stated. Degraded engines are excluded from the denominators (a vendor's blended mention rate averages only engines that actually answered). Failed calls are stored as failure records; they never become silent zeros.

4. Run counts and windows

3 runs per prompt per engine, each in a fresh session (no shared conversation state; each run has its own session_id).
Runs are spread across at least 2 days: the scheduler runs on the 2nd, 4th and 6th of each month at 14:00 UTC, one run index per day. ChatGPT is the exception, because a browser session cannot run unattended on a server: its three runs are captured by hand on three days at least two days apart, and imported with the capture time each answer was collected.
Full volume per edition: 3 runs × 25 prompts × 5 engines × 20 categories = 7,500 responses (the launch plan, three categories, was 900).
Engine defaults are used for temperature and other sampling parameters.
A verification step checks every prompt×engine cell has 3 runs, on ≥2 distinct days, in distinct sessions, before an edition may publish.

Provenance and retention

Every response is stored as runs/<YYYY-MM>/<category>/<engine>/<prompt_id>_run<n>.json containing: prompt id and exact text, prompt set version, engine and model identifier, timestamp (UTC), session id, answer text, every cited/consulted URL, token usage, cost in USD, and the complete raw API response. A ChatGPT run carries the same fields, with no token count and a cost of 0.00, plus the untouched capture and the name of the capture file it came from; the model id is present from the October 2026 captures on. Files are written once and never modified. A re-run writes the next run index; an existing file is skipped. Raw responses are retained indefinitely as the dispute audit trail. Per-engine cost is logged to runs/<YYYY-MM>/costs.csv.

5. Extraction rules

Extraction is deterministic code, re-runnable against the raw files.

Named. A vendor is named when its canonical name or a listed alias appears in the answer text. Matching folds diacritics, tolerates a space or hyphen between name tokens ("ButterflyMX" / "Butterfly MX"), accepts possessives, and ignores markdown emphasis around the name. Names that are also common English words (DOOR, Latch, Level, Alloy, Meld…) match case-sensitively, and a sentence-initial common-word name used as a verb ("Latch the gate…") is not counted.
Position. Ordinal of the vendor's first mention among all vendors named in the answer (1 = named first).
Sentiment. recommended / neutral / cautioned, from recommendation and caution cue words within a 180-character window of the first mention. This is a coarse framing signal, not a review score, and is displayed as such.
Context snippet. ~240 characters around the first mention, stored for audit.
Cited sources. Every cited or consulted URL is normalized (tracking parameters such as utm_source removed), classified by page type (vendor site, review platform, listicle, trade publication, directory, forum, association, WebPossible-operated site, other), and linked to the vendor(s) whose domain it is.
Not ranked. Property management systems (AppFolio, Yardi, RealPage, Entrata, Buildium, MRI and their peers) and their built-in modules are not ranked: the Index measures the vendors a property manager buys around its system of record. Nor are local service providers, or hardware brands that appear only as compatible hardware under another platform. An answer that names only those counts as naming no vendor.
Unknown names. Capitalized names in list, heading, bold or table-row positions (app captures carry tables as tab-separated rows) that match no known vendor, candidate, or property-management-system name go to a human review queue (review/<edition>/unknown-vendors.yaml). A name enters data/vendors.yaml only after a human confirms it, with a source_url and last_verified date. An edition cannot publish while review items are pending.
Accuracy gate. Extraction is validated against a hand-labeled set of ≥50 responses; it must reach ≥95% agreement on named vendors before metrics publish. The labeled set is never edited to raise the score. On 2026-10-03, six labels were corrected because the companies those samples name had just joined the vendor database; the samples' text was not changed. Additionally, 50 responses per edition are spot-checked by hand.

6. Metric definitions

All metrics are per vendor, per category, per edition. Seeded mentions are excluded (see §2). Degraded engines are excluded from denominators, not counted as zero.

Metric	Definition
Mention rate (per engine)	Runs in which the vendor was named ÷ ok runs for that engine, over prompts where the vendor is not seeded.
Mention rate (blended)	Mean of per-engine mention rates across engines that returned ok runs.
Shortlist rank	Ordinal by blended mention rate (ties: better position score, then wider engine spread, then name). Vendors never named have no rank.
Position score	Mean first-mention position across runs where the vendor was named (1.0 = always first). Lower is better.
Consistency	Share of prompt×engine cells (with ≥2 runs) where all runs agreed (all named or none named). 1.0 = fully consistent; low values mean the engine flips between runs. Displayed, never hidden.
Engine spread	Number of ok engines (0–5) with a non-zero mention rate.
Source dependency	Top cited domains on the runs where the vendor was named — the pages its mentions ride on.
Movement	Prior-edition rank minus current rank (positive = moved up). Null when there is no prior edition or the vendor was unranked before.

Source Map (per category). Top 20 cited URLs and top 20 cited domains by citation count across ok runs, each with page type, the engines that cited it, and the vendors it supports. When webpossible.com or another site WebPossible operates appears, it is flagged own_network and disclosed inline. Known pay-to-play listicles are flagged, not hidden.

7. Known limitations

Non-determinism. Engines give different answers to the same prompt. Three runs and the consistency metric make the variance visible; they do not remove it.
Mixed instruments. ChatGPT is read from the consumer app; the other four are read through APIs, which are the closest repeatable proxy for their apps but are not identical to them (different system prompts, memory, personalization). The ChatGPT column therefore answers a slightly different question from the other four, and no claim of parity is made in either direction.
No personalization, but not no location. Runs use no account history, and the ChatGPT chats run Unpersonalized (memory, plugins and custom instructions ignored). Real users get personalized answers; the Index measures the unpersonalized baseline. Every ChatGPT capture is checked for answers that address the operator's own business; a contaminated set is discarded and re-run. The app's approximate location cannot be turned off, so ChatGPT answers carry the capture location (Michigan in 2026-09) where a question depends on place.
US only. Prompts and SERP captures are US English, and every ChatGPT capture so far has come from one US state.
Model drift. Engines change models without notice. The model identifier returned by each API is stored per run and shown per edition; month-to-month movement may reflect a model change rather than anything a vendor did. ChatGPT was the blind spot until October 2026: the September captures name no model. Captures since then record the model from the conversation record.
Extraction is lexical. A vendor mentioned only by a nickname we have not listed as an alias is missed until the review queue surfaces it. Sentiment is cue-based.
Google AI Overviews are third-party captured and may not appear for every query; absence is recorded as "no AI Overview", not as a zero mention.

8. Attribution and terms

Every result is labeled with the instrument that produced it: the ChatGPT web app for ChatGPT, vendor APIs or a licensed data provider for the rest. Attribution lines carried on this page:

OpenAI: Answers captured from the ChatGPT web app at chatgpt.com by automated browser capture, one temporary chat per prompt, no personalization. ChatGPT and OpenAI are trademarks of OpenAI. OpenAI's terms govern automated use of its services; that risk sits with WebPossible, never with a vendor the Index measures. Runs published before 2026-09-19 came from the OpenAI API (Responses endpoint, web search tool) and are labeled as such in the run files.
Anthropic: Claude answers and citations retrieved via the Anthropic Messages API with the web search tool. Claude and Anthropic are trademarks of Anthropic, PBC.
Perplexity: Answers and citations retrieved via the Perplexity API (sonar). Perplexity is a trademark of Perplexity AI, Inc.
Google: Gemini answers and grounding metadata retrieved via the Gemini API with Google Search grounding. Google AI Overview text captured from public Google Search results pages via a third-party data provider (Apify). Google, Gemini and Google Search are trademarks of Google LLC.
Apify: SERP data captured with the Apify Google Search Results Scraper under Apify's terms of service.

Vendor names are used nominatively. Logos appear only with permission via a claimed row. Engine terms are reviewed each edition; if a provider's terms change in a way that bars publication, that engine ships as degraded and the change is logged.

9. Re-check and disputes

Any vendor may request one re-run of its category's prompt set per edition through "Request a re-check" on its row. Re-checks follow the same protocol (3 runs, separate days, all engines) and are published regardless of outcome, alongside the original. Entity corrections (name, aliases, category membership, website) are handled in-edition after verification against a source URL. We respond to disputes on the record, with data, on the policy page. No vendor can pay for placement, removal, or a favorable re-check.

10. Changelog policy

Every change that can move a number is logged at /ai-shortlist-index/changelog/ with the edition it takes effect: prompt additions, retirements or wording changes (new prompt set version), engine or model changes, capture-method changes, metric definition changes, vendor additions/merges/retirements, category additions, and price-table updates used for cost logging. Editions are immutable once published; corrections are published as amendments with the original preserved.

Engine and version record (2026-09)

Engine	Adapter	Model / provider
ChatGPT	chatgpt.com web app, temporary chat (Unpersonalized), browser capture	September captures: not recorded; 2–3 October 2026 captures: gpt-5-6-thinking
Perplexity	Perplexity chat completions	sonar
Gemini	Gemini API + Google Search grounding	gemini-3.8-flash
Google AI Overviews	Apify Google Search Results Scraper	US desktop SERP

How We Measure AI Recommendations for Property Management Software