PRPickrack

Remove Duplicate Lines

Dedupe lines while preserving order. Configurable case-sensitivity, whitespace trimming, keep-first vs keep-last.

Dedupe lines while preserving order. Case-sensitive option, whitespace trim, keep-first vs keep-last.

Remove Duplicate Lines scans your text line-by-line and outputs each unique line once. Unlike Excel's 'Remove duplicates' or a SQL DISTINCT, Pickrack preserves the original order — first (or last) occurrence stays where it was. Configurable: case sensitivity, whitespace trimming, keep-first vs keep-last, and optional blank-line removal.

Use it for: cleaning up email lists, deduping URL lists, removing repeated log entries, simplifying tag dumps, cleaning copy-pasted spreadsheet rows, or any list that's been concatenated from multiple sources.

Free, no signup, browser-side. Up to ~100,000 lines processed instantly. Privacy-friendly: your list never leaves the browser tab.

Key features

  • Preserves order — Unlike sort | uniq in shell, the output keeps the original order — first occurrence's position is the kept one.
  • Keep-first vs keep-last — Toggle which duplicate to retain. Keep-last is rare but useful when later occurrences have more recent data.
  • Case-insensitive option — Apple, apple, and APPLE count as the same line. Toggle off for strict case matching.
  • Whitespace trim — hello and hello count as the same (leading/trailing spaces ignored for comparison).
  • Drop blank lines — Also remove empty lines as part of the dedupe pass.
  • Live stats — Input lines · unique count · removed count — see at a glance how many dupes were in your data.

How to use

  1. Step 1: Paste your list — Anything one-per-line: emails, URLs, names, log lines, IDs.
  2. Step 2: Toggle case-sensitivity + whitespace — Defaults (case-insensitive ON, trim ON) match most use cases. Turn off for exact-string dedup.
  3. Step 3: Pick keep-first or keep-last — Almost always keep-first (default). Keep-last for log replay or recent-wins data.
  4. Step 4: Copy result — Output updates live. Click Copy or 'Swap in' for further operations.

When to use

  • Email list cleanup — combine multiple subscriber CSVs → dedup before import
  • URL list cleanup — dedupe a bookmark export or scraped link list
  • Log entry analysis — see unique error messages from a noisy log
  • Tag list — dedupe a CSV of post tags
  • Mailing list import — required by most ESPs (Mailchimp, etc.) before upload
  • Spreadsheet data prep — clean a column before pivot table analysis
  • SQL data prep — paste a column of values, dedupe, then paste back as IN (...) clause

Frequently asked questions

How is this different from Sort Lines with dedup?

Sort Lines with dedup REORDERS your data (by alpha, numeric, etc.). Remove Duplicates PRESERVES the original order. Use Remove Duplicates when order matters (e.g., chronological log), Sort with dedup when you want alphabetized + unique.

What if I want to dedupe by a specific column (CSV)?

Not in v1 — Pickrack dedupes by exact line match. For CSV column dedup: (1) extract the column with Find & Replace, dedupe, then merge back. Or use a CLI tool like awk -F, '!seen[$2]++'. CSV column awareness is on the roadmap.

Why is whitespace trim a separate option from case-insensitive?

They're independent. You might want case-sensitive but whitespace-insensitive (e.g., dedup IDs that may have stray spaces). Toggle each per your data.

What's the difference between keep-first and keep-last?

If your data has 5 duplicate rows of 'Alice', keep-first keeps row #1 of those 5 (and the position of #1 in the output). Keep-last keeps row #5 (at row #5's original position). The position of the kept row is preserved either way.

Are blank lines treated as duplicates?

Yes by default — multiple consecutive blank lines collapse to one. Toggle 'Drop blank lines' to remove all blanks entirely. Untoggle to keep blanks 1:1.

Does this scale to millions of lines?

v1 uses an in-memory Set — fine up to ~500,000 lines on a modern browser. For larger datasets use sort -u on the command line or a database DISTINCT.

Related tools

Sort Lines

Sort text lines alphabetically, numerically, by length, naturally, or shuffle. Optional dedup + blank filter.

Find & Replace

Find and replace text with regex + capture groups. Live preview, match count, browser-side.

Word Frequency Counter

Count word, character, or line occurrences. Ranked top N, filter by length, drop stopwords, CSV export.