Remove Duplicate Lines
Paste a list or block of text and the tool drops repeated lines while keeping the original order. Trim whitespace, sort alphabetically, or mark duplicates for review instead of removing them.
How to use the duplicate line remover
1. Paste the text
Drop your list or text block into the Input box. The tool treats every newline as a record boundary. Each line is compared as a complete unit, edge to edge, against every other line.
2. Set the toggles
Three toggles cover the common edge cases. Case-sensitive defaults off, so "Apple" and "apple" collide. Trim whitespace before comparing defaults on, so accidental leading or trailing spaces do not block matches. Sort output alphabetically defaults off (original order preserved); turn it on for a canonical A-to-Z output.
3. Pick the keep mode
Three modes control what happens to duplicates. Keep first (default) drops every copy after the first time a line appears. Keep last drops every earlier copy and only the final occurrence survives. Keep all, mark duplicates does not remove anything; it appends a marker to every line that has at least one duplicate so you can audit before cutting.
4. Clean and copy
Click Clean. The deduped text appears on the right with a count of removed lines and a breakdown of which lines were repeated. Use the Copy button to grab the result, or Download to save a .txt file.
When you need to remove duplicate lines
Line-level duplication is one of the most common cleanup problems in everyday text work. Here is where it shows up.
Line dedup vs word dedup
This tool dedupes at the line level. The whole line, every character up to the newline, is the unit. Two lines must match end-to-end to be considered duplicates. If you only need to drop repeated words inside a paragraph, use the word-level tool instead.
| Tool | Unit | Best for |
|---|---|---|
| Remove Duplicate Lines | Whole line (newline-delimited) | Lists, address books, CSV rows, URL exports, log entries. |
| Remove Duplicate Words | Single token (space-delimited) | Prose stutters ("the the"), vocab extraction, tag cleanup. |
A common mistake is to use line dedup on a paragraph of prose and find that nothing changes (because no two full lines match). If that happens, switch to word-level dedup.
Deduplication patterns: input to output examples
The dedup result depends on how the comparison is made. Eight common scenarios people run into, and how each setting changes what survives:
| Scenario | Input (one per line) | Output |
|---|---|---|
| Email list, mixed case (lowercase first, then dedup) | Jose@cliqpod.io, jose@cliqpod.io, JOSE@CLIQPOD.IO | jose@cliqpod.io (one row) |
| URL list, trailing-slash variants | word.tips/, word.tips, word.tips/ | Verbatim: 2 rows survive. Slash-normalized: 1 row. |
| Keyword list with stray spaces | word counter, word counter (trailing space), word counter | Verbatim: 3 rows. Trim-first: 1 row. |
| Keep first vs keep last occurrence | apple (line 1), banana (line 2), apple (line 5) | Keep first: apple, banana. Keep last: banana, apple. |
| Case-sensitive vs case-insensitive comparison | Apple, apple, APPLE | Case-sensitive: 3 rows. Case-insensitive: 1 row (the first form is kept). |
| Whitespace-normalized vs verbatim | hello world (double space), hello world (single space) | Verbatim: 2 rows. Whitespace-collapsed: 1 row. |
| JSON-lines, comparing as strings | {"id":1,"name":"a"}, {"name":"a","id":1} | Both survive (string-equal comparison sees different key orders as different lines). |
| Word-frequency-capped dedup (keep first N per token) | 5 lines containing foo across a 20-line file, cap = 2 | First 2 foo lines kept, the other 3 dropped. |
Rule of thumb: normalize before you dedupe. If jose@cliqpod.io and Jose@Cliqpod.IO are the same person to you, lowercase the column first, then run the line dedup. The tool compares strings exactly as given.
Related tools
Frequently asked questions
How do I remove duplicate lines from a list?
Paste the list (one item per line) into the input box. Leave Trim whitespace on so accidental indentation does not block matches. Keep the default Keep first behavior. Click Clean. The unique lines come out in their original order, with a count of how many duplicates were dropped.
What does Trim whitespace before comparing actually do?
When this toggle is on (the default), the tool strips leading and trailing whitespace from each line before comparing. So "apple", " apple", and "apple " are all treated as the same line. The surviving copy is trimmed in the output. Turn this off only if whitespace at the edges is meaningful to your data (rare).
How is this different from removing duplicate words?
This tool dedupes at the line level: an entire line (every character up to the newline) is the unit. Two lines must match end-to-end to be considered duplicates. The Remove Duplicate Words tool dedupes at the word level (single tokens). Pick line-level for lists, address books, CSV-style rows. Pick word-level for prose stutters or vocab extraction.
What happens with empty lines?
Empty lines are counted as a line. If you have three blank lines in a row, two of them are duplicates and will be dropped. If you need to keep blank lines as paragraph separators, dedupe first and then re-format, or use the Remove Empty Lines tool instead.
Will sorting change the order of unique lines?
Yes. With Sort output alphabetically OFF (the default), the original order is preserved (the first occurrence of each unique line stays where it was). With it ON, the output is re-ordered A to Z after dedup. Use sort when you want a canonical list (for example, before importing into a spreadsheet column).
What does Keep all but mark duplicates do?
This mode does not remove anything. Instead, each line that has at least one duplicate gets a marker appended (for example "[DUPLICATE]") so you can review the matches manually before deciding what to remove. Useful when the dedup is irreversible and you want to audit first.
Does case sensitivity matter for line dedup?
With case-sensitive OFF (the default), "Apple" and "apple" on different lines are treated as duplicates. With it ON, they are different lines. Most prose and casual list cleanup wants OFF. Code identifier lists, URLs with case-sensitive paths, or filename lists usually want ON.
I'm pasting email lists and customer records to remove duplicate rows. Are those contacts going to your database?
No. Row dedup runs in your browser. Email addresses, customer records, and any other rows you paste stay in the textarea and are not transmitted to any database on our end. See also: Text to One Line. See also: All removers.
More wordcounter.ai tools
Other tools you might find useful.