Write a function to remove all duplicate elements from a list, keeping only the first occurrence. Implement both approaches: (a) Preserving order, (b) Without preserving order.

Overview

Removing duplicates from a list is a common data processing task that tests your understanding of Set operations, order preservation, and algorithm efficiency. The challenge varies based on whether the original order must be maintained.

Example:

[1, 2, 3, 2, 4, 1, 5]

[1, 2, 3, 4, 5]

Approach 1: Set Conversion (Order Not Preserved)

Simplest approach using Set for automatic deduplication.

dartCopy
List<int> removeDuplicatesUnordered(List<int> list) {
  return list.toSet().toList();
}

void main() {
  print(removeDuplicatesUnordered([1, 2, 3, 2, 4, 1, 5]));
  // Output: [1, 2, 3, 4, 5] (order may vary)

  print(removeDuplicatesUnordered([5, 5, 5, 1, 2, 1]));
  // Output: [5, 1, 2] (order not guaranteed)
}

Time Complexity: O(n) Space Complexity: O(n) - Set storage

Pros:

• One-liner solution
• Very efficient
• Simple and readable

Cons:

• Does NOT preserve original order
• Order depends on hash implementation

Approach 2: LinkedHashSet (Order Preserved)

Use LinkedHashSet to maintain insertion order.

dartCopy
List<int> removeDuplicatesOrdered(List<int> list) {
  return list.toSet().toList();  // LinkedHashSet preserves order in Dart
}

// More explicit version
List<int> removeDuplicatesOrderedExplicit(List<int> list) {
  return LinkedHashSet<int>.from(list).toList();
}

void main() {
  print(removeDuplicatesOrdered([1, 2, 3, 2, 4, 1, 5]));
  // Output: [1, 2, 3, 4, 5] (order preserved)

  print(removeDuplicatesOrdered([3, 1, 2, 3, 1]));
  // Output: [3, 1, 2] (first occurrence kept)
}

Time Complexity: O(n) Space Complexity: O(n)

Note: In Dart,

toSet()

LinkedHashSet

Approach 3: Manual Tracking with Set

Manually track seen elements for more control.

dartCopy
List<int> removeDuplicatesManual(List<int> list) {
  Set<int> seen = {};
  List<int> result = [];

  for (int item in list) {
    if (seen.add(item)) {  // add() returns true if element was added
      result.add(item);
    }
  }

  return result;
}

void main() {
  print(removeDuplicatesManual([1, 2, 3, 2, 4, 1, 5]));
  // Output: [1, 2, 3, 4, 5]
}

Time Complexity: O(n) Space Complexity: O(n)

Pros:

• Full control over logic
• Can add custom filtering
• Order preserved

Approach 4: Where with Tracking

Functional approach using

where()

dartCopy
List<int> removeDuplicatesFunctional(List<int> list) {
  Set<int> seen = {};
  return list.where((item) => seen.add(item)).toList();
}

void main() {
  print(removeDuplicatesFunctional([1, 2, 3, 2, 4, 1, 5]));
  // Output: [1, 2, 3, 4, 5]
}

Time Complexity: O(n) Space Complexity: O(n)

Pros:

• Concise functional style
• Order preserved
• Elegant one-liner

Approach 5: For Custom Objects

Remove duplicates from objects based on a property.

dartCopy
class Person {
  final int id;
  final String name;

  Person(this.id, this.name);

  @override
  String toString() => 'Person($id, $name)';

  @override
  bool operator ==(Object other) =>
      other is Person && other.id == id;

  @override
  int get hashCode => id.hashCode;
}

List<Person> removeDuplicatePersons(List<Person> persons) {
  return persons.toSet().toList();
}

// Alternative: Based on specific property
List<Person> removeDuplicatesByProperty(
  List<Person> persons,
  dynamic Function(Person) keySelector,
) {
  Set<dynamic> seen = {};
  return persons.where((p) => seen.add(keySelector(p))).toList();
}

void main() {
  var people = [
    Person(1, 'Alice'),
    Person(2, 'Bob'),
    Person(1, 'Alice Clone'),  // Duplicate ID
  ];

  print(removeDuplicatePersons(people));
  // Output: [Person(1, Alice), Person(2, Bob)]

  print(removeDuplicatesByProperty(people, (p) => p.id));
  // Same output, more flexible
}

Complete Solution Class

dartCopy
class DuplicateRemover {
  // Order preserved (default in Dart)
  static List<T> remove<T>(List<T> list) {
    return list.toSet().toList();
  }

  // Explicit order preservation
  static List<T> removePreservingOrder<T>(List<T> list) {
    return LinkedHashSet<T>.from(list).toList();
  }

  // Order not preserved (use regular Set)
  static List<T> removeUnordered<T>(List<T> list) {
    return HashSet<T>.from(list).toList();
  }

  // With custom key selector
  static List<T> removeByKey<T, K>(
    List<T> list,
    K Function(T) keySelector,
  ) {
    Set<K> seen = {};
    return list.where((item) => seen.add(keySelector(item))).toList();
  }

  // Keep last occurrence instead of first
  static List<T> removeKeepLast<T>(List<T> list) {
    return list.reversed.toSet().toList().reversed.toList();
  }

  // Count removed duplicates
  static Map<String, dynamic> removeWithStats<T>(List<T> list) {
    Set<T> unique = list.toSet();
    return {
      'result': unique.toList(),
      'original': list.length,
      'unique': unique.length,
      'removed': list.length - unique.length,
    };
  }
}

void main() {
  var nums = [1, 2, 3, 2, 4, 1, 5];

  print(DuplicateRemover.remove(nums));
  // [1, 2, 3, 4, 5]

  print(DuplicateRemover.removeKeepLast(nums));
  // [3, 2, 4, 1, 5] (last occurrences)

  print(DuplicateRemover.removeWithStats(nums));
  // {result: [1, 2, 3, 4, 5], original: 7, unique: 5, removed: 2}

  var people = [
    Person(1, 'Alice'),
    Person(2, 'Bob'),
    Person(1, 'Alice Clone'),
  ];

  print(DuplicateRemover.removeByKey(people, (p) => p.id));
  // [Person(1, Alice), Person(2, Bob)]
}

Comparison

toSet()

LinkedHashSet

HashSet

Edge Cases

dartCopy
void testEdgeCases() {
  // Empty list
  assert(removeDuplicatesOrdered([]).isEmpty);

  // No duplicates
  assert(removeDuplicatesOrdered([1, 2, 3]) == [1, 2, 3]);

  // All duplicates
  assert(removeDuplicatesOrdered([5, 5, 5, 5]) == [5]);

  // Single element
  assert(removeDuplicatesOrdered([7]) == [7]);

  // Null values (if nullable)
  assert(removeDuplicatesOrdered([null, 1, null, 2]) == [null, 1, 2]);

  // Negative numbers
  assert(removeDuplicatesOrdered([-1, -2, -1]) == [-1, -2]);

  // Mixed types (with dynamic)
  // assert(removeDuplicatesOrdered([1, '1', 1]) == [1, '1']);
}

Best Practices

1. Use toSet() for Simple Cases

dartCopy
// ✅ Simplest and most readable
List<int> unique(List<int> list) => list.toSet().toList();

2. Be Explicit About Order Requirements

dartCopy
// ✅ Clear intention
List<int> uniqueOrdered(List<int> list) {
  return LinkedHashSet<int>.from(list).toList();
}

List<int> uniqueUnordered(List<int> list) {
  return HashSet<int>.from(list).toList();
}

3. Use Functional Style for Conciseness

dartCopy
// ✅ One-liner with order preservation
List<T> unique<T>(List<T> list) {
  Set<T> seen = {};
  return list.where(seen.add).toList();
}

4. Implement equals/hashCode for Custom Objects

dartCopy
// ✅ Required for Set deduplication
class User {
  final int id;
  final String name;

  User(this.id, this.name);

  @override
  bool operator ==(Object other) =>
      other is User && other.id == id;

  @override
  int get hashCode => id.hashCode;
}

Real-World Applications

1. Remove Duplicate Tags

dartCopy
List<String> uniqueTags(List<String> tags) {
  return tags.map((t) => t.toLowerCase()).toSet().toList();
}

2. Deduplicate API Results

dartCopy
List<User> deduplicateUsers(List<User> users) {
  return DuplicateRemover.removeByKey(users, (u) => u.id);
}

3. Clean User Input

dartCopy
List<String> cleanInputList(String input) {
  return input
      .split(',')
      .map((s) => s.trim())
      .where((s) => s.isNotEmpty)
      .toSet()
      .toList();
}

print(cleanInputList('apple, banana, apple, , cherry'));
// [apple, banana, cherry]

Performance Tip

For very large lists (millions of elements):

dartCopy
import 'dart:collection';

// ✅ More efficient for huge datasets
List<int> removeDuplicatesEfficient(List<int> list) {
  return LinkedHashSet<int>.from(list).toList();
}

Resources

• Set Class - Dart API
• LinkedHashSet - Dart API
• HashSet - Dart API