Removing duplicates is a one-click job with three hidden questions inside it: what counts as “the same”, which copy survives, and whether you want the repeated items or the rare ones. Answer those and any list, emails, SKUs, attendees, keywords, cleans up in seconds with our free deduplicator, in your browser, with nothing uploaded.
In this guide
Question one: what counts as the same?
Exact matching is the default and it is stricter than human eyes: “apple”, “Apple”, and ” apple” are three different strings and zero duplicates to a naive comparison. The three matching options that close the gap:
- Trim whitespace, so ” apple” matches “apple”. Trailing spaces from spreadsheet copies are the single most common reason a dedupe “misses” obvious duplicates.
- Ignore case, so “Apple” matches “apple”, the right call for emails and names, the wrong one when case is data.
- Unicode normalization, the deep cut: the same accented character can be stored in two encodings, visually identical and byte-different, so café can fail to match café. Normalization folds them together, and the deduplicator offers it precisely because this failure is otherwise undiagnosable by looking.
The honest order of operations from the cleanup pillar applies: normalize first or enable matching options, then dedupe, never the reverse.
Question two: which copy survives?
When five copies become one, which one? First occurrence is the standard: it preserves the list’s original order and keeps the earliest entry, which matters when later copies carry typos or the order encodes priority. Last occurrence earns its place when the list is chronological and the newest version of each item is the truth, a re-registered attendee with a corrected email, for example. The unique items tool offers both, and the choice is invisible on toy lists and decisive on real ones, which is exactly the kind of option worth picking consciously rather than by default.
Question three: duplicates out, or duplicates only?
Deduplication has three useful outputs, not one:
- Each item once, the classic dedupe: every distinct value, repeats collapsed.
- Only the duplicates: just the values that appeared more than once, which answers the investigative question, who registered twice, which SKUs collide, what got pasted into both halves.
- Only the singletons: values appearing exactly once, the mirror image, useful for spotting the unmatched leftovers after a merge.
All three modes live in the unique items tool, and the second one is the sleeper: “show me the duplicates” is often the actual question hiding behind “remove the duplicates”. For comparing two separate lists rather than one list against itself, shared and missing items, the list comparator is the right shape of tool.
Verifying the result
Counts make deduplication auditable: items before, minus items after, equals duplicates removed, and the item counter gives both numbers in seconds. The habit pays off in two directions. If the difference is zero on a list you were sure had duplicates, the matching options are too strict, almost always whitespace or case. If the difference is suspiciously large, the options are too loose for this data, ignore-case folding distinct SKUs like “a1B” and “A1b” together. Counting turns both mistakes from silent to visible, which is the entire point of verification.
The consecutive-repeats special case
Sometimes repeats are only noise when adjacent: log lines repeating while a state persists, sensor readings, stuttered pastes. Removing all duplicates would destroy real data (the same reading legitimately recurs later); what you want is collapsing consecutive repeats only, and that is a different operation with its own tool, the repeating items remover, which also offers run-length capping, keep at most N consecutive copies, for the in-between cases. Choosing between global and consecutive dedupe is just asking whether distance matters: emails, no; time-ordered logs, yes.
Frequently asked questions
Does deduplication change the order of my list?
Keep-first deduplication preserves the original order of the survivors; sorting is a separate, optional step. Tools that combine them are sorting by choice, not necessity, so look for the order option if it matters.
How do I dedupe by one column, like email, when lines have several fields?
A line deduplicator compares whole lines, so pull the relevant column out first, with a column extractor or a spreadsheet, dedupe, and rejoin. Whole-line matching across multi-field rows is usually too strict to catch what you mean.
Can deduping fix “same person, different spelling”?
No: “Jon Smith” and “John Smith” are different strings, and merging them is fuzzy matching, a judgment-laden job beyond exact tools. Exact dedupe handles exact repeats; humans or specialized software handle near-matches.
Why does my count differ from the spreadsheet’s “remove duplicates”?
Different matching defaults, usually case or whitespace. Apply the same options on both sides and the counts reconcile; when they do not, the difference list itself, via the duplicates-only mode, shows exactly which items the two tools disagree about.