Deduplication

Ledly’s deduplication engine prevents duplicate leads from entering your system. Using advanced fuzzy matching algorithms, it can detect duplicates even when data isn’t exactly the same.

How Deduplication Works

When a new lead arrives, Ledly checks it against existing leads using your configured matching criteria. The system calculates a similarity score (0-100%) and takes action based on your threshold settings.

New Lead → Match Check → Score Calculation → Threshold Comparison → Action

Matching Methods

Exact Match

Fields must be identical (case-insensitive).

Fuzzy Match

Uses similarity algorithms to detect near-matches. Perfect for catching typos and variations.

Supported Algorithms

Configuration

Duplicate Actions

When a duplicate is detected, Ledly can take one of several actions:

Reject

Block the duplicate lead from entering the system.

{
  "success": false,
  "error": "Duplicate lead detected",
  "duplicate_of": "lead_abc123",
  "similarity_score": 92
}

Merge

Combine the new data with the existing lead, updating empty or older fields.

Link

Accept the new lead but link it to the existing lead for tracking.

Flag for Review

Accept the lead but mark it for manual review.

Allow

Accept the lead as a new record (useful for intentional re-submissions).

Match Field Configuration

Email Matching

Email matching includes intelligent normalization:

Phone Matching

Phone numbers are normalized to digits only:

Name Matching

Names use fuzzy matching with configurable algorithms:

Duplicate Window

The duplicate window determines how far back to check for duplicates.

Multi-Field Matching

Combine multiple fields for more accurate matching:

AND Logic

All fields must match to be considered a duplicate.

Match if:
  email similarity > 95%
  AND first_name similarity > 80%
  AND last_name similarity > 80%

OR Logic

Any field match counts as a duplicate.

Match if:
  email similarity > 95%
  OR phone similarity > 95%

Weighted Scoring

Assign weights to different fields.

Final score = (email_score × 0.5) + (name_score × 0.3) + (phone_score × 0.2)

Deduplication Log

Every duplicate check is logged for auditing:

Viewing the Log

1. Go to Leads → Deduplication Log
2. Filter by date range, action taken, or similarity score
3. Click any entry to see match details

Log Entry Details

{
  "timestamp": "2024-12-25T10:30:00Z",
  "incoming_lead": {
    "email": "[email protected]",
    "first_name": "John",
    "last_name": "Smith"
  },
  "matched_lead_id": "lead_abc123",
  "match_fields": {
    "email": { "score": 100, "method": "exact" },
    "name": { "score": 96, "method": "jaro_winkler" }
  },
  "final_score": 98,
  "threshold": 85,
  "action_taken": "reject",
  "reason": "Duplicate detected"
}

Vendor-Specific Deduplication

Configure different deduplication rules per vendor:

Setting Vendor Rules

1. Go to Vendors → Select vendor → Deduplication
2. Override the organization defaults
3. Configure vendor-specific settings
4. Save changes

Cross-Vendor Deduplication

By default, deduplication checks across all vendors. You can configure:

Within Vendor Only

Only check for duplicates from the same vendor.

Use case: Different vendors may legitimately send the same lead.

Cross-Vendor (Default)

Check for duplicates across all vendors.

Use case: Prevent paying multiple vendors for the same lead.

Vendor Groups

Check within defined vendor groups.

Use case: Group vendors by lead type and only dedup within groups.

Performance Considerations

High-Volume Optimization

For organizations processing thousands of leads per hour:

1. Reduce match fields - Use only essential fields (email, phone)
2. Shorten window - Use 1-24 hour windows
3. Use exact match first - Fuzzy matching is slower
4. Index custom fields - Contact support for custom field indexing

Estimated Processing Times

API Reference

Check for Duplicates

Before submitting a lead, you can check if it’s a duplicate:

POST /api/dedup/check-duplicate
Content-Type: application/json
 
{
  "email": "[email protected]",
  "first_name": "John",
  "last_name": "Smith",
  "phone": "555-123-4567"
}

Response (duplicate found):

{
  "is_duplicate": true,
  "matched_lead_id": "lead_abc123",
  "similarity_score": 94,
  "match_details": {
    "email": 100,
    "name": 88
  }
}

Response (no duplicate):

{
  "is_duplicate": false
}

Get Deduplication Config

GET /api/dedup

Update Deduplication Config

POST /api/dedup
 
{
  "enabled": true,
  "threshold": 85,
  "window_hours": 24,
  "action": "reject",
  "match_fields": ["email", "phone"]
}

Best Practices

1. Start with email-only matching - It’s the most reliable identifier

2. Use 85-90% threshold - Lower catches more duplicates but may have false positives

3. Monitor false positives - Review the dedup log regularly to tune settings

4. Combine with validation - Validate email/phone format before dedup for better matching

5. Consider your use case - Marketing leads may tolerate duplicates; sales leads may not

6. Document vendor expectations - Communicate dedup policies to vendors

Troubleshooting

False Positives (Good Leads Rejected)

• Increase the similarity threshold (e.g., 85% → 92%)
• Use more match fields with AND logic
• Review if name matching is too aggressive
• Check if email normalization is causing issues

False Negatives (Duplicates Getting Through)

• Lower the similarity threshold
• Add more match fields
• Extend the duplicate window
• Switch to fuzzy matching for names

Performance Issues

• Reduce the duplicate window
• Use fewer match fields
• Switch to exact matching for high-volume fields
• Contact support for custom indexing