Remove Duplicate Lines & Sort

Deduplicate, sort, shuffle, and clean up any list of lines.

100% in your browser — nothing uploaded

Remove duplicate lines from a list

Input lines

Case-insensitive compare Trim lines before comparing Remove empty lines

0Total lines

0Unique lines

0Duplicates removed

Result

About this duplicate line remover

Paste any list and clean it up in one click: remove duplicate lines while keeping the first occurrence, sort alphabetically in either direction, sort by line length, apply a numeric-aware natural sort (so file2 comes before file10), reverse the order, or shuffle it randomly. Comparison options let you ignore letter case, trim surrounding spaces, and drop empty lines, and a counter shows total lines, unique lines, and how many duplicates were removed.

It’s the fast way to clean email lists, log extracts, SQL query results, keyword exports, or any column copied from a spreadsheet. Each button applies a single operation to your input, so results stay predictable, and everything runs locally in your browser with JavaScript — your list is never uploaded, which makes it safe for customer emails and other data you wouldn’t paste into a random website.

How to remove duplicate lines in Excel, Notepad++, VS Code and the shell

In Excel the command sits on the Data tab, in the Data Tools group: Remove Duplicates. The trap is that it works on rows, not on lines of text. It compares each row across the columns you tick and deletes the whole row permanently, and Microsoft warns that data "will be removed from all columns, even if you do not select all the columns at this step". It also compares what a cell displays rather than what it stores, so one date shown as 3/8/2006 and another as Mar 8, 2006 count as unique.

Where you are	What to run	The catch
Excel	Data > Data Tools > Remove Duplicates	Deletes entire rows, permanently. Will not run on outlined or subtotaled data until you remove the outline and subtotals.
Excel, non-destructive	Data > Sort & Filter > Advanced > "Unique records only"	Copies the unique records elsewhere or hides duplicates in place instead of deleting them.
Notepad++	Edit > Line Operations > Remove Duplicate Lines	Built in, no plugin needed. Its neighbour, Remove Consecutive Duplicate Lines, only drops copies sitting immediately below the first one, so that one needs sorted text to catch everything.
VS Code	Command Palette > Delete Duplicate Lines	Built in, but there is no default keyboard shortcut, so it is invisible until you search for it. With nothing multi-line selected it applies to the whole file.
Terminal	sort -u list.txt	Outputs only the first of lines with equal keys, but hands back a sorted file, and the ordering depends on your locale.
Terminal, order kept	awk '!seen[$0]++' list.txt	Keeps the first occurrence exactly where it was. This is the one to reach for when the order carries meaning.

sort -u list.txt > unique.txt        # sorted output
awk '!seen[$0]++' list.txt > unique.txt  # original order kept
LC_ALL=C sort -u list.txt            # plain byte order, no locale rules

Why the awk line works: seen[$0] does not exist the first time a line appears, and an uninitialized awk value counts as 0, so !0 is true and the line prints. The ++ then sets it to 1, so every later copy tests false. sort -u is shorter but hands back sorted output, and its ordering follows the environment locale unless you set LC_ALL=C.

What none of them show you is the damage: Excel deletes the rows outright, the editors just make the file shorter, and the shell prints only the survivors. The box above keeps the first occurrence in place and reports how many lines went in, how many were unique and how many duplicates it dropped, with switches to ignore case, trim spaces or skip blank lines before it compares.

FAQ

Which line is kept when duplicates are removed?

The first occurrence, in its original position and with its original formatting. Even if you compare case-insensitively or with trimming enabled, the line that appears in the output is exactly the first one you pasted.

What is natural sort?

A numeric-aware ordering that compares digits by value instead of character by character: file2 sorts before file10, and item9 before item100. A plain alphabetical sort would put file10 first because the character 1 comes before 2.

Is my list uploaded to a server?

No. Splitting, comparing, and sorting all happen in your browser with JavaScript. Nothing is sent anywhere, so it’s safe for email lists, customer records, and internal data.