Clean up messy lists in seconds
Duplicate lines creep into everything: merged spreadsheets, keyword lists, email lists, inventory exports and log files. Paste your list here and this tool removes every repeated line, with options to ignore case, trim stray spaces, drop empty lines, sort the result A to Z, or reverse it. The counter shows exactly how many duplicates were removed.
Everyday jobs for this tool
- Merge two contact or keyword lists without double entries.
- Turn a shuffled export into a clean alphabetical list.
- Find how many unique values a column really contains.
- Prepare lists for import into spreadsheets, CRMs and email tools.
How the deduplication works under the hood
The tool splits your pasted text into separate lines and then walks through them from top to bottom, keeping a set of every line it has already seen. The first time a line appears, it is added to the set and copied into the output. Any later line whose comparison key is already in the set is skipped, which is why the first version of a repeated entry is always the one that survives. The options change how that comparison key is built. With trimming on, spaces at the start and end are removed before comparing, so a line with stray spaces still matches its clean twin, and the trimmed version is what appears in the output. With case-sensitive matching off, the key is lowercased, so "Apple" and "apple" collide and count as one line, though the output keeps the capitalization of whichever version came first. Empty lines are dropped before the comparison step when that option is on. Sorting and reversing happen only after deduplication, so they arrange the unique lines and never bring a duplicate back. The counters describe the non-empty input lines, the unique lines produced, and the difference between the two.
A worked example
Paste this eight line list: apple, banana, Apple, cherry, banana, one blank line, date, apple. With the default options, meaning case-insensitive matching, trimming on and empty lines removed, the output is four lines: apple, banana, cherry, date. The statistics report 7 lines in, because the blank line is not counted, 4 unique lines out, and 3 duplicates removed: the capital "Apple", the second "banana" and the second "apple". Now tick case-sensitive matching and run the same list again. This time "Apple" is treated as its own entry, the output grows to five lines, and only 2 duplicates are removed. That one checkbox is the difference between a clean fruit list and a correct list of case-sensitive product codes.
Tips for getting better results
- Keep a copy of the original: deduplication cannot be undone once you paste over your source, so save the raw list first, especially when the removed count turns out larger than you expected.
- Normalize before you dedupe: if entries differ only in capitalization, run the list through the Case Converter set to lowercase first, so visually identical entries match even with case-sensitive matching on.
- Merging two lists is just pasting: paste the second list directly underneath the first and dedupe once; every overlap between the two lists disappears in a single pass.
- Use the removed count as a sanity check: if you expected about ten overlaps and the tool reports two hundred, your source data probably has a formatting quirk worth investigating before you import the result.
Common mistakes to avoid
- Expecting near duplicates to match: lines that differ by an internal extra space, a trailing word or a punctuation mark are different lines to any exact matching tool, so tidy obvious variants in your source or by hand.
- Turning case sensitivity off for codes: in SKUs, usernames and voucher codes, upper and lower case often identify genuinely different items, and merging them silently loses real entries.
- Forgetting the first version wins: when two variants of a line exist, the one nearer the top survives, so put your preferred spelling or capitalization first if the list mixes both.
- Assuming A to Z sorting understands numbers: sorting compares lines as text, so "Item 10" sorts before "Item 2"; pad numbers to the same width, such as "Item 02", if numeric order matters.