Member-match decoy test

About

What it is

A test bench for HRex $member-match: does a matcher send someone's records to the wrong person?

What it does

Sends 6 synthetic test requests, each with a hidden trap, to up to 4 matchers.
Checks every answer against a locked answer key.
Tells you who got it right, who refused safely, and who sent records to the wrong person.

How to use

Run it with no settings to see the result.

Try:

Problem 1 of 4 matchers sent someone's records to the wrong person: ID-first rules.

Synthetic data only; not a production matching recommendation.

Scored identity matcher, strict: 5 of 6 right Warning
Scored identity matcher, relaxed: 6 of 6 right Good
ID-first rules: 5 of 6 right Problem
Check-every-field rules: 6 of 6 right Good

Warning Scored identity matcher, strict: Refused (needs human review), but Rowan Quill Jr was the right answer.
Test: Senior and Junior on one card. Father and son, same name, same card. Only the birth date tells which Rowan is asking. Right answer: Return Rowan Quill Jr (SYN-MEM-2001-02).
Problem ID-first rules: Sent Jon Roe's records in answer to a request for John Roe.
Test: John's demographics, Jon's card. John Roe's details with the card of Jon Roe, a near-twin name born eight days later. Right answer: Refuse (HRex 422): the card and the person don't agree.

Every answer24 answers

Test	Matcher	Result	What happened
Test: Control: clean card, clean demographics	Matcher: Scored identity matcher, strict	Result: Good ✓ Found the right person	What happened: Found John Roe.
Test: Senior and Junior on one card	Matcher: Scored identity matcher, strict	Result: Warning ✗ Missed the right person	What happened: Refused (needs human review), but Rowan Quill Jr was the right answer.
Test: Twins on one policy, initial only	Matcher: Scored identity matcher, strict	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused (more than one match), as HRex requires.
Test: Right person, consent the server can't honour	Matcher: Scored identity matcher, strict	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: Scored identity matcher, strict	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: John's demographics, Jon's card	Matcher: Scored identity matcher, strict	Result: Good ✓ Rightly refused: not this person	What happened: Refused, giving the reason "needs human review" rather than "no match". Still the safe answer.
Test: Control: clean card, clean demographics	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Found the right person	What happened: Found John Roe.
Test: Senior and Junior on one card	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Found the right person	What happened: Found Rowan Quill Jr.
Test: Twins on one policy, initial only	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused (more than one match), as HRex requires.
Test: Right person, consent the server can't honour	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: John's demographics, Jon's card	Matcher: Scored identity matcher, relaxed	Result: Good ✓ Rightly refused: not this person	What happened: Refused, giving the reason "needs human review" rather than "no match". Still the safe answer.
Test: Control: clean card, clean demographics	Matcher: ID-first rules	Result: Good ✓ Found the right person	What happened: Found John Roe.
Test: Senior and Junior on one card	Matcher: ID-first rules	Result: Good ✓ Found the right person	What happened: Found Rowan Quill Jr.
Test: Twins on one policy, initial only	Matcher: ID-first rules	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused, giving the reason "no match" rather than "more than one match". Still the safe answer.
Test: Right person, consent the server can't honour	Matcher: ID-first rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: ID-first rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: John's demographics, Jon's card	Matcher: ID-first rules	Result: Problem ✗ Wrong person	What happened: Sent Jon Roe's records in answer to a request for John Roe.
Test: Control: clean card, clean demographics	Matcher: Check-every-field rules	Result: Good ✓ Found the right person	What happened: Found John Roe.
Test: Senior and Junior on one card	Matcher: Check-every-field rules	Result: Good ✓ Found the right person	What happened: Found Rowan Quill Jr.
Test: Twins on one policy, initial only	Matcher: Check-every-field rules	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused, giving the reason "no match" rather than "more than one match". Still the safe answer.
Test: Right person, consent the server can't honour	Matcher: Check-every-field rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: Check-every-field rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: John's demographics, Jon's card	Matcher: Check-every-field rules	Result: Good ✓ Rightly refused: not this person	What happened: Refused (no match), as HRex requires.

The test requests6 traps

Test	What's tricky	Right answer
Test: Control: clean card, clean demographics	What's tricky: A clean request for John Roe: right card, right details. Everyone should find him.	Right answer: Return John Roe (SYN-MEM-1001-01).
Test: Senior and Junior on one card	What's tricky: Father and son, same name, same card. Only the birth date tells which Rowan is asking.	Right answer: Return Rowan Quill Jr (SYN-MEM-2001-02).
Test: Twins on one policy, initial only	What's tricky: Twin sisters on one card, same birth date; the request gives only the initial A.	Right answer: Refuse (HRex 422): more than one member fits.
Test: Right person, consent the server can't honour	What's tricky: The right person, but his permission to share ended last year.	Right answer: Refuse (HRex 422): the consent can't be honoured.
Test: Right person, consent limited to non-sensitive data	What's tricky: The right person, but his permission excludes sensitive data, which this plan cannot separate out.	Right answer: Refuse (HRex 422): the consent can't be honoured.
Test: John's demographics, Jon's card	What's tricky: John Roe's details with the card of Jon Roe, a near-twin name born eight days later.	Right answer: Refuse (HRex 422): the card and the person don't agree.

How each matcher works4 matchers

Scored identity matcher, strict: Scores look-alikes on name and birth date; accepts only at 90%+.
Checks the member ID first, then scores how alike the name and birth date are. Accepts a look-alike only when at least 90% sure; otherwise sends it to human review.
Scored identity matcher, relaxed: Same matcher; accepts look-alikes at a level you set.
The same matcher, but it accepts a look-alike at a confidence you set (default 85%).
ID-first rules: Trusts the ID on the card first.
Trusts the member or subscriber ID on the card; uses name, birth date and gender only to break ties. A common way to build member match.
Check-every-field rules: ID, name, birth date and gender must all agree.
Finds the card's ID, then insists that name, birth date and gender all agree, even when the ID already picked one person.

What the results mean8 results

✓ Found the right person: One member was the person in the request, and the matcher returned that member.
✓ Rightly refused: not this person: The card and the person don't agree, so nobody should be returned. The matcher refused (HRex 422).
✓ Rightly refused: can't tell apart: More than one member fits the request equally. HRex says refuse (422), and the matcher did.
✓ Rightly refused: consent: The consent sent with the request can't be honoured, so HRex says refuse (422). The matcher did.
✗ Wrong person: The matcher returned a member who is not the person asking, or returned someone when it should have refused. This is how one person's records reach another.
✗ Ignored consent: The matcher returned the right person despite a consent the plan can't honour.
✗ Missed the right person: One member was the right answer, but the matcher refused. Safe, but the member's records don't move.
✗ No usable answer: The matcher crashed or answered in a form that isn't a valid $member-match response.

Checked against the locked answer key afd4a0787ca3. Any HRex 422 refusal where one is due counts as right.

Settings: all defaults

Runs in your browser and makes no network request. Your settings are kept in the page address, so the link reproduces this result.

Two ways ImOnFHIR's tools run

Some run in your browser. Others run on our servers when your assistant calls them. Each way gains something and risks something.

The tools on this page run in your browser.

In your browser

Running the tool sends nothing you entered. It runs from a copy of its code held in this site.
A shared link carries your settings in the address. Opening it sends that address, settings included, to our host.
Only tools packaged to run in the browser work this way.
The tool itself runs no AI. Use your own assistant on the result if you want one.

On our servers, through your assistant: the site's MCP endpoint (`/api/v1/mcp`)

You get ImOnFHIR's validator, its standards feed, its compare tools and a rule-evidence lookup. Your assistant reads the results.
What you send reaches our server. If your assistant runs on a provider's servers, that provider receives it too. How long that provider keeps it is the provider's policy, not ours.
Synthetic data only. Never send real patient data.
Our code logs one line per call: the tool name and the status. What our host and the validator host log is not stated (see Privacy).
Each call counts against a per-IP limit of 120 a minute for the endpoint, lower for some tools. The limit is kept in memory on each server instance, so it is not a hard global cap.
The route is configured for at most 60 seconds per call.
An assistant can misstate a result. The tool's own output is the source of truth.
The AI Tools catalog has no hosted path yet. Only the site's own tools run on our servers.

More: AI agents, Privacy, API terms.

Recorded AI run

Recorded 2026-11-04 · model claude-sonnet-5-5 · cost $0.228 (US dollars)

No AI runs on this site: this is a recording.

Matchers it ran: ai-with-rules, ai-with-definitions, ai-no-rules

Show the recorded result

Warning None of the 3 matchers sent records to the wrong person.

Synthetic data only; not a production matching recommendation.

AI adjudicator, with rules: 3 of 6 right Warning
AI adjudicator, rules and consent codes: 5 of 6 right Warning
AI adjudicator, no rules: 5 of 6 right Warning

Warning AI adjudicator, with rules: Refused (consent), but John Roe was the right answer.
Test: Control: clean card, clean demographics. A clean request for John Roe: right card, right details. Everyone should find him. Right answer: Return John Roe (SYN-MEM-1001-01).
Warning AI adjudicator, with rules: Refused (consent), but Rowan Quill Jr was the right answer.
Test: Senior and Junior on one card. Father and son, same name, same card. Only the birth date tells which Rowan is asking. Right answer: Return Rowan Quill Jr (SYN-MEM-2001-02).
Warning AI adjudicator, with rules: Returned John Roe's records despite a consent the plan can't honour.
Test: Right person, consent limited to non-sensitive data. The right person, but his permission excludes sensitive data, which this plan cannot separate out. Right answer: Refuse (HRex 422): the consent can't be honoured.
Warning AI adjudicator, rules and consent codes: Refused (consent), but John Roe was the right answer.
Test: Control: clean card, clean demographics. A clean request for John Roe: right card, right details. Everyone should find him. Right answer: Return John Roe (SYN-MEM-1001-01).
Warning AI adjudicator, no rules: Returned John Roe's records despite a consent the plan can't honour.
Test: Right person, consent limited to non-sensitive data. The right person, but his permission excludes sensitive data, which this plan cannot separate out. Right answer: Refuse (HRex 422): the consent can't be honoured.

Every answer18 answers

Test	Matcher	Result	What happened
Test: Control: clean card, clean demographics	Matcher: AI adjudicator, with rules	Result: Warning ✗ Missed the right person	What happened: Refused (consent), but John Roe was the right answer.
Test: Senior and Junior on one card	Matcher: AI adjudicator, with rules	Result: Warning ✗ Missed the right person	What happened: Refused (consent), but Rowan Quill Jr was the right answer.
Test: Twins on one policy, initial only	Matcher: AI adjudicator, with rules	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused (more than one match), as HRex requires.
Test: Right person, consent the server can't honour	Matcher: AI adjudicator, with rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: AI adjudicator, with rules	Result: Warning ✗ Ignored consent	What happened: Returned John Roe's records despite a consent the plan can't honour.
Test: John's demographics, Jon's card	Matcher: AI adjudicator, with rules	Result: Good ✓ Rightly refused: not this person	What happened: Refused, giving the reason "needs human review" rather than "no match". Still the safe answer.
Test: Control: clean card, clean demographics	Matcher: AI adjudicator, rules and consent codes	Result: Warning ✗ Missed the right person	What happened: Refused (consent), but John Roe was the right answer.
Test: Senior and Junior on one card	Matcher: AI adjudicator, rules and consent codes	Result: Good ✓ Found the right person	What happened: Found Rowan Quill Jr.
Test: Twins on one policy, initial only	Matcher: AI adjudicator, rules and consent codes	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused (more than one match), as HRex requires.
Test: Right person, consent the server can't honour	Matcher: AI adjudicator, rules and consent codes	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: AI adjudicator, rules and consent codes	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: John's demographics, Jon's card	Matcher: AI adjudicator, rules and consent codes	Result: Good ✓ Rightly refused: not this person	What happened: Refused, giving the reason "needs human review" rather than "no match". Still the safe answer.
Test: Control: clean card, clean demographics	Matcher: AI adjudicator, no rules	Result: Good ✓ Found the right person	What happened: Found John Roe.
Test: Senior and Junior on one card	Matcher: AI adjudicator, no rules	Result: Good ✓ Found the right person	What happened: Found Rowan Quill Jr.
Test: Twins on one policy, initial only	Matcher: AI adjudicator, no rules	Result: Good ✓ Rightly refused: can't tell apart	What happened: Refused (more than one match), as HRex requires.
Test: Right person, consent the server can't honour	Matcher: AI adjudicator, no rules	Result: Good ✓ Rightly refused: consent	What happened: Refused (consent), as HRex requires.
Test: Right person, consent limited to non-sensitive data	Matcher: AI adjudicator, no rules	Result: Warning ✗ Ignored consent	What happened: Returned John Roe's records despite a consent the plan can't honour.
Test: John's demographics, Jon's card	Matcher: AI adjudicator, no rules	Result: Good ✓ Rightly refused: not this person	What happened: Refused, giving the reason "needs human review" rather than "no match". Still the safe answer.

The test requests6 traps

Test	What's tricky	Right answer
Test: Control: clean card, clean demographics	What's tricky: A clean request for John Roe: right card, right details. Everyone should find him.	Right answer: Return John Roe (SYN-MEM-1001-01).
Test: Senior and Junior on one card	What's tricky: Father and son, same name, same card. Only the birth date tells which Rowan is asking.	Right answer: Return Rowan Quill Jr (SYN-MEM-2001-02).
Test: Twins on one policy, initial only	What's tricky: Twin sisters on one card, same birth date; the request gives only the initial A.	Right answer: Refuse (HRex 422): more than one member fits.
Test: Right person, consent the server can't honour	What's tricky: The right person, but his permission to share ended last year.	Right answer: Refuse (HRex 422): the consent can't be honoured.
Test: Right person, consent limited to non-sensitive data	What's tricky: The right person, but his permission excludes sensitive data, which this plan cannot separate out.	Right answer: Refuse (HRex 422): the consent can't be honoured.
Test: John's demographics, Jon's card	What's tricky: John Roe's details with the card of Jon Roe, a near-twin name born eight days later.	Right answer: Refuse (HRex 422): the card and the person don't agree.

How each matcher works3 matchers

AI adjudicator, with rules: Claude decides, given the HRex rules.
Claude reads the request and the member list, given the HRex matching rules. No helper code.
AI adjudicator, rules and consent codes: Claude decides, given the rules and HRex's consent-code definitions.
The same as with rules, plus HRex's own definitions of the two consent codes (#regular, #sensitive), which the model misread without them.
AI adjudicator, no rules: Claude decides, given no rules.
The same, but told nothing about the HRex rules: what does the model know on its own?

What the results mean8 results

✓ Found the right person: One member was the person in the request, and the matcher returned that member.
✓ Rightly refused: not this person: The card and the person don't agree, so nobody should be returned. The matcher refused (HRex 422).
✓ Rightly refused: can't tell apart: More than one member fits the request equally. HRex says refuse (422), and the matcher did.
✓ Rightly refused: consent: The consent sent with the request can't be honoured, so HRex says refuse (422). The matcher did.
✗ Wrong person: The matcher returned a member who is not the person asking, or returned someone when it should have refused. This is how one person's records reach another.
✗ Ignored consent: The matcher returned the right person despite a consent the plan can't honour.
✗ Missed the right person: One member was the right answer, but the matcher refused. Safe, but the member's records don't move.
✗ No usable answer: The matcher crashed or answered in a form that isn't a valid $member-match response.

Checked against the locked answer key afd4a0787ca3. Any HRex 422 refusal where one is due counts as right.

Technical details

Tool name: score_member_match

Package decoy-lab from https://github.com/ImOnFHIR/member-match-decoy-lab at commit 7de9ec9

sha256:657a016f61f30bfb97a9177cfed5c917067b411b135464650029b52667dae4b8

This tool runs in your browser: its own code and data, copied here and checked against the digest above when it was copied in and on every build. It makes no network request; your settings are kept in the page address.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "cases": {
      "description": "Which test requests to run. Leave empty for all 6.",
      "examples": [
        [
          "D4-card-mixup"
        ]
      ],
      "items": {
        "anyOf": [
          {
            "const": "D0-control",
            "title": "Control: clean card, clean demographics",
            "type": "string"
          },
          {
            "const": "D1-senior-junior",
            "title": "Senior and Junior on one card",
            "type": "string"
          },
          {
            "const": "D2-twins",
            "title": "Twins on one policy, initial only",
            "type": "string"
          },
          {
            "const": "D3-consent-expired",
            "title": "Right person, consent the server can't honour",
            "type": "string"
          },
          {
            "const": "D3b-consent-unsegmentable",
            "title": "Right person, consent limited to non-sensitive data",
            "type": "string"
          },
          {
            "const": "D4-card-mixup",
            "title": "John's demographics, Jon's card",
            "type": "string"
          }
        ]
      },
      "title": "Test requests to run",
      "type": "array"
    },
    "detail": {
      "anyOf": [
        {
          "const": "summary",
          "title": "Summary: the result and the problems",
          "type": "string"
        },
        {
          "const": "full",
          "title": "Full: every answer, explained",
          "type": "string"
        }
      ],
      "description": "\"summary\" (default): the result and the problems. \"full\": every answer, explained.",
      "title": "How much to show"
    },
    "matchers": {
      "description": "Which matchers to test. Leave empty for all 4.",
      "examples": [
        [
          "id-first",
          "check-every-field"
        ]
      ],
      "items": {
        "anyOf": [
          {
            "const": "identity-strict",
            "title": "Scored identity matcher, strict",
            "type": "string"
          },
          {
            "const": "identity-relaxed",
            "title": "Scored identity matcher, relaxed",
            "type": "string"
          },
          {
            "const": "id-first",
            "title": "ID-first rules",
            "type": "string"
          },
          {
            "const": "check-every-field",
            "title": "Check-every-field rules",
            "type": "string"
          }
        ]
      },
      "title": "Matchers to test",
      "type": "array"
    },
    "relaxed_confidence": {
      "description": "How sure (75 to 100%) the relaxed matcher must be to accept a look-alike. Default 85.",
      "examples": [
        90
      ],
      "maximum": 100,
      "minimum": 75,
      "title": "Relaxed matcher confidence (%)",
      "type": "integer"
    }
  },
  "type": "object"
}

Synthetic data only; not a production matching recommendation.

Member-match decoy test

About

What it is

What it does

How to use

Two ways ImOnFHIR's tools run

In your browser

On our servers, through your assistant: the site's MCP endpoint (/api/v1/mcp)

Recorded AI run

On our servers, through your assistant: the site's MCP endpoint (`/api/v1/mcp`)