coulomb/identity-canon
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Complete IDENTITY-WP-0003 corpus backfill and model refinement

Browse Source 
Backfill all 23 research source notes with terminology extracts, modeling
assumptions, conflicts, canonical mappings, and references. Refresh terminology
artifacts, refine the conceptual model with explicit scenario paths, reconcile
canon surfaces and open questions, and mark the workplan finished.
This commit is contained in:
Bernd Worsch
2026-06-21 20:22:20 +02:00
parent 790a2f2041
commit 1c1b5c9bc6
32 changed files with 2676 additions and 623 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
research/entity-resolution-privacy/deterministic-vs-probabilistic-matching.md
View File

@@ -1,50 +1,109 @@
# Deterministic Vs Probabilistic Matching
# Deterministic vs Probabilistic Matching
## Source Type
TODO: Identify whether this is a standard, specification, product documentation, academic concept, architecture pattern, or implementation reference.
Academic and industry practice. Entity resolution, record linkage, and
duplicate detection literature plus operational MDM patterns.
## Domain
TODO: Classify the source domain.
Entity resolution, record linkage, duplicate detection, and account matching
confidence.
## Why This Source Matters
Deterministic and probabilistic matching patterns for entity resolution and identity linking.
Entity resolution literature distinguishes deterministic keys from probabilistic
matching — the foundation for weak vs. strong synonymity modeling.
## Key Concepts
TODO:
- concept 1
- concept 2
- concept 3
- **Deterministic matching**: records match when agreed key fields are equal
(exact email, government ID, OIDC `iss`+`sub`).
- **Probabilistic matching**: match score from weighted field similarity
(Fellegi-Sunter, Jaro-Winkler, ML classifiers).
- **Record linkage**: identifying records across datasets referring to same
entity.
- **Blocking**: reduce comparison space by bucketing on partial keys.
- **Match threshold**: score above which records are linked or flagged.
- **False positive / false negative tradeoff**: precision vs. recall in linking.
- **Golden record / survivor**: MDM pattern selecting canonical merged record.
- **Non-destructive link**: associate records without merge (preferred in modern MDM).
- **Human review queue**: ambiguous matches escalated for operator decision.
- **Master data management (MDM)**: operational discipline around entity resolution.
## Relevant Terminology
TODO: Extract terms used by the source and note their source-specific meaning.
| Term | Source meaning |
| --- | --- |
| Deterministic match | Equality on predefined key fields. |
| Probabilistic match | Scored similarity above threshold. |
| Record linkage | Cross-dataset entity correspondence. |
| Blocking key | Partial key for candidate pair generation. |
| Match score | Confidence metric for probable same entity. |
| Duplicate | Records hypothesized to refer to same entity. |
| Merge | Combine records into one (destructive). |
| Link | Associate records preserving sources (non-destructive). |
| Survivor record | Chosen primary after merge. |
| Quarantine | Hold ambiguous matches for review. |
## Modeling Assumptions
TODO: Capture assumptions made by the source about users, accounts, organizations, tenants, groups, roles, identities, relationships, or credentials.
- **Same entity is hypothesis until verified** in probabilistic approaches.
- **Deterministic rules are domain-specific** (no universal golden key).
- **Merge destroys provenance** unless carefully audited — increasingly avoided.
- **Confidence is continuous or banded** (weak/medium/strong).
- **Source system identity must be preserved** for compliance and undo.
- **Human review is part of high-assurance linking.**
- **Privacy regulations constrain** which fields can be matched.
## Identity-Canon Implications
TODO: Explain what this source suggests for the canonical identity model.
- Deterministic match maps to **strong Synonymity Assertion** when keys are
authoritative (S13).
- Probabilistic match maps to **weak Synonymity Assertion** with confidence
score and method (S12).
- **Link without merge** is the canonical preferred pattern (**P7**).
- **Match score** maps to confidence/strength on Synonymity Assertion.
- **Blocking/method** maps to Evidence Source metadata.
- **Quarantine** maps to Lifecycle State `proposed` on assertion.
- **Golden record** is downstream MDM pattern; canon should not require merge.
- **Human review** maps to Evidence Source (operator decision).
## Terminology Conflicts
TODO: Record where this source uses terms differently from other sources.
- **Duplicate vs. Synonymity**: duplicates imply merge; synonymity allows coexistence.
- **Match vs. Link**: industry uses interchangeably; canon distinguishes strength.
- **Entity vs. Actor**: resolution literature says entity; canon prefers Actor target.
- **Identity vs. Record**: matching is between records, not persons directly.
- **Deterministic vs. Strong**: deterministic can still be wrong if key is shared
(shared email).
## Candidate Canonical Mappings
TODO: Map source-specific concepts to identity-canon candidate concepts.
| Entity resolution concept | Candidate canonical concept |
| --- | --- |
| Deterministic match | Strong Synonymity Assertion |
| Probabilistic match | Weak Synonymity Assertion |
| Match score | Confidence / strength metadata |
| Link (non-destructive) | Synonymity Assertion |
| Merge | Downstream anti-pattern (avoid) |
| Blocking key | Evidence Source method |
| Review queue | Lifecycle State `proposed` |
| Source record ID | Identifier |
| Golden record | Downstream projection only |
| False positive handling | Revocation / supersession of assertion |
## Open Questions
TODO:
- Question 1
- Question 2
- What confidence bands (weak/medium/strong) should canon standardize?
- Which deterministic keys are authoritative per source family (OIDC iss+sub,
persistent SAML NameID, verified email)?
- Should probabilistic matchers be required to store feature-level Evidence Source?
- How should shared-attribute false positives (family email) be classified?
## References
TODO: Add canonical URLs, RFC/spec identifiers, documentation links, and citation notes.
- Fellegi-Sunter model (1969) — foundational probabilistic record linkage
- Christen, "Data Matching" (2012) — entity resolution textbook
- NIST SP 800-63A evidence requirements — https://pages.nist.gov/800-63-4/sp800-63A.html
- MDM Institute duplicate management practices — industry reference

98
research/entity-resolution-privacy/gdpr-pseudonymization.md
View File

@@ -1,50 +1,114 @@
# Gdpr Pseudonymization
# GDPR Pseudonymization and Privacy
## Source Type
TODO: Identify whether this is a standard, specification, product documentation, academic concept, architecture pattern, or implementation reference.
Regulatory guidance. EU GDPR (Regulation 2016/679) Article 4(5) and Recital 26;
EDPB guidance on identifiability, anonymization, and data subject rights.
## Domain
TODO: Classify the source domain.
Privacy regulation, pseudonymization, identifiability, data minimization, and
lawful basis for identity processing.
## Why This Source Matters
GDPR-relevant pseudonymization, anonymization, data minimization, and identity linkage considerations.
GDPR pseudonymization and identifiability concepts affect how canonical models
should represent privacy-limited links, scoped identifiers, and correlation risk.
## Key Concepts
TODO:
- concept 1
- concept 2
- concept 3
- **Personal data**: information relating to identified or identifiable natural
person.
- **Identifiable person**: can be identified directly or indirectly by reasonable
means.
- **Pseudonymization (Art. 4(5))**: processing personal data so it cannot be
attributed to a subject without additional information kept separately.
- **Anonymization**: irreversible de-identification; data no longer personal.
- **Data subject**: identified or identifiable natural person.
- **Controller / Processor**: roles responsible for processing personal data.
- **Purpose limitation**: data used for specified, explicit, legitimate purposes.
- **Data minimization**: adequate, relevant, limited to necessary.
- **Right of access / erasure**: data subject rights affecting linked records.
- **Additional information**: key held separately to re-identify pseudonymous data.
## Relevant Terminology
TODO: Extract terms used by the source and note their source-specific meaning.
| Term | Source meaning |
| --- | --- |
| Personal data | Data about identifiable natural person. |
| Pseudonymization | Reversible de-identification with separate key. |
| Anonymization | Irreversible; no longer personal data (if effective). |
| Data subject | Natural person the data relates to. |
| Identifiable | Reasonably linkable to person. |
| Additional information | Re-identification key stored separately. |
| Controller | Determines purposes and means of processing. |
| Processing | Any operation on personal data. |
| Erasure | Delete personal data (right to be forgotten). |
| Profiling | Automated evaluation of personal aspects. |
## Modeling Assumptions
TODO: Capture assumptions made by the source about users, accounts, organizations, tenants, groups, roles, identities, relationships, or credentials.
- **Pseudonymization is not anonymization**; data may remain personal.
- **Separate storage of additional information** is required for pseudonymization.
- **Scope and access control on keys** determine correlation risk.
- **Linking pseudonymous records across purposes** may increase identifiability.
- **Legal basis and purpose** govern whether linking is permissible.
- **Erasure requests** may require breaking links or deleting assertions.
- **Regulatory role (controller)** is organizational, not purely technical.
## Identity-Canon Implications
TODO: Explain what this source suggests for the canonical identity model.
- **Pseudonymous Identifier** and **Scoped Identifier** map to pseudonymization
techniques (pairwise sub, hashed email, internal IDs).
- **Privacy-limited Synonymity Assertion** must record privacy classification
and scope (S14).
- **Additional information** (re-identification key) maps to separately secured
**Evidence Source** or **Credential** with strict Scope access.
- **Data subject** maps to **Natural Person** with privacy rights overlay
(downstream policy, not canon legal advice).
- **Erasure** maps to Lifecycle State transitions: revoke assertions, sever
bindings, archive with legal exceptions noted downstream.
- Pairwise OIDC, tenant-local subjects, and restricted persona links are
technical pseudonymization patterns aligned with GDPR concepts.
- Reinforces visibility of privacy constraints on relationships (**P8**, S14 checks).
## Terminology Conflicts
TODO: Record where this source uses terms differently from other sources.
- **Pseudonym vs. Pseudonymization**: pseudonym is identifier; pseudonymization
is processing technique.
- **Anonymous vs. Pseudonymous**: often conflated in product marketing.
- **Identity vs. Personal data**: not all identifiers are personal data in all
contexts.
- **Deletion vs. Revocation**: erasure may require more than assertion revocation.
- **Subject**: GDPR data subject vs. OIDC/SAML subject.
## Candidate Canonical Mappings
TODO: Map source-specific concepts to identity-canon candidate concepts.
| GDPR concept | Candidate canonical concept |
| --- | --- |
| Data subject | Natural Person (privacy overlay) |
| Pseudonymization | Processing pattern on Identifier / Profile |
| Pseudonymous identifier | Scoped Identifier / Pseudonymous Identifier |
| Additional information | Separately secured Evidence Source / key |
| Purpose limitation | Scope + policy metadata on processing |
| Cross-system link | Synonymity Assertion (privacy classification required) |
| Erasure request | Lifecycle State + assertion revocation |
| Identifiability risk | Privacy classification on links |
| Controller | Organization actor (downstream legal role) |
| Anonymized dataset | Out of scope for personal identity linking |
## Open Questions
TODO:
- Question 1
- Question 2
- Should canon include a standard `privacy_classification` enum for assertions?
- How should erasure of one account affect Synonymity Assertions touching other
accounts (S02)?
- Does pseudonymization key storage warrant a canonical secured Scope type?
- Should identifiability review be documented as operator workflow in downstream
recommendations only?
## References
TODO: Add canonical URLs, RFC/spec identifiers, documentation links, and citation notes.
- GDPR Article 4(5) pseudonymization — https://gdpr-info.eu/art-4-gdpr/
- GDPR Recital 26 on identifiability — https://gdpr-info.eu/recitals-novo/26/
- EDPB Guidelines on identifiability (various) — https://edpb.europa.eu/
- ISO/IEC 20889 privacy enhancing data de-identification terminology

88
research/entity-resolution-privacy/synonymity-assertions.md
View File

@@ -2,49 +2,105 @@
## Source Type
TODO: Identify whether this is a standard, specification, product documentation, academic concept, architecture pattern, or implementation reference.
Concept synthesis from identity-canon ResearchSeed, entity resolution practice,
federation account linking, and semantic web `sameAs` patterns.
## Domain
TODO: Classify the source domain.
Identity linking, scoped equivalence, account linking, and non-destructive record
association.
## Why This Source Matters
Weak, strong, scoped, operational, legal, and privacy-preserving synonymity assertions.
Synonymity assertions are the identity-canon-native model for linking records
without merge — synthesizing federation binding, entity resolution, and
semantic equivalence patterns.
## Key Concepts
TODO:
- concept 1
- concept 2
- concept 3
- **Synonymity assertion**: scoped, evidenced claim that two or more identifiers,
records, or actors refer to the same target for a stated purpose.
- **Relation type**: `same_as`, `probably_same_as`, `linked_to`, `represents`,
`controls`, `acts_for` (from ResearchSeed).
- **Strength**: weak, medium, strong, authoritative bands.
- **Scope**: namespace, tenant, relying party, or purpose boundary limiting
assertion validity.
- **Evidence**: verification event, issuer signature, operator review, import
job output.
- **Source system**: system that created or maintains the assertion.
- **Lifecycle**: proposed, active, revoked, expired, superseded.
- **Privacy classification**: controls visibility and correlation risk.
- **Non-merge invariant**: linked records retain independent identity and provenance.
- **Supersession chain**: new assertion replaces old when identifiers change.
## Relevant Terminology
TODO: Extract terms used by the source and note their source-specific meaning.
| Term | Source meaning |
| --- | --- |
| Synonymity | Sameness or equivalence under conditions. |
| Assertion | Explicit modeled statement, not implicit merge. |
| same_as | High-confidence equivalence. |
| probably_same_as | Probabilistic equivalence. |
| linked_to | Operational convenience link. |
| represents | One record represents another (controller, profile). |
| Scope | Boundary limiting assertion meaning. |
| Strength | Confidence band. |
| Revocation | Assertion no longer valid. |
| Supersession | New assertion replaces prior. |
## Modeling Assumptions
TODO: Capture assumptions made by the source about users, accounts, organizations, tenants, groups, roles, identities, relationships, or credentials.
- **Equivalence is contextual**, not universal.
- **Multiple assertions can coexist** for different scopes and purposes.
- **Conflicting assertions possible**; require review workflow.
- **Downstream systems consume assertions** according to their assurance needs.
- **Privacy-limited assertions** must not leak across scopes (S14).
- **Federation bindings are synonymity assertions** (`iss`+`sub` → local account).
- **sameAs on web is weak by default** unless corroborated.
## Identity-Canon Implications
TODO: Explain what this source suggests for the canonical identity model.
- Synonymity Assertion is a **first-class Relationship** class in canon.
- Recommended fields align with ResearchSeed and ConceptualModel invariants.
- OIDC RP binding, SAML persistent NameID mapping, SCIM `externalId`
correlation, and entity-resolution matches all project into Synonymity
Assertions with appropriate strength.
- Supports all linking scenarios: S12 (weak), S13 (strong), S14 (privacy-limited).
- **P7** is the governing principle; merge is downstream exception only.
- Revocation from RISC/SSF events should update assertion Lifecycle State.
## Terminology Conflicts
TODO: Record where this source uses terms differently from other sources.
- **Link vs. Merge**: products say "linked accounts" after merge.
- **sameAs vs. same_as**: semantic web informal vs. canon typed relation.
- **Account linking vs. Identity linking**: may target accounts or identifiers.
- **Alias vs. Synonymity**: alias is presentation; synonymity is assertion.
- **Duplicate resolution vs. Synonymity**: MDM duplicate implies survivor selection.
## Candidate Canonical Mappings
TODO: Map source-specific concepts to identity-canon candidate concepts.
| Practice / source pattern | Candidate canonical concept |
| --- | --- |
| OIDC iss+sub → local user | Strong Synonymity Assertion (scoped) |
| SAML persistent NameID map | Strong Synonymity Assertion |
| Probabilistic duplicate score | Weak Synonymity Assertion (`probably_same_as`) |
| Operator-verified link | Strong Synonymity Assertion (authoritative) |
| Pairwise sub RP binding | Privacy-limited Synonymity Assertion |
| SCIM externalId correlation | Identifier Binding / medium Synonymity |
| schema.org sameAs | Weak Synonymity Assertion (caution) |
| DID equivalentId | Method-dependent Synonymity |
| VC subject DID binding | Strong Synonymity Assertion with cryptographic evidence |
## Open Questions
TODO:
- Question 1
- Question 2
- Should `linked_to` remain distinct from `same_as` for operational vs. semantic links?
- What minimum field set is mandatory for all Synonymity Assertions (see OpenQuestions)?
- How should conflicting assertions be represented (priority, review state)?
- Should privacy classification be enum or policy reference?
## References
TODO: Add canonical URLs, RFC/spec identifiers, documentation links, and citation notes.
- identity-canon ResearchSeed.md — synonymity assertion fields
- identity-canon ConceptualModel.md — identity linking model
- OIDC account linking practice — https://openid.net/specs/openid-connect-core-1_0.html
- W3C VC subject identification — https://www.w3.org/TR/vc-data-model-2.0/