Stepping through Collections

Lesson purpose

Lesson 7 is about stepping through, or iterating, all sorts of collections. It widens the idea of a collection beyond arrays and lists: a directory is a collection of files, and a file is a collection of bytes, lines, or records. Students then look at the three C# approaches to iteration: the enumerator (iterator) protocol that powers foreach, indexers that give an object array-style access, and iterator blocks that use yield return to produce a sequence one value at a time. The organizing idea: iteration separates how a sequence is produced from how its values are consumed.

Learning objectives

• Treat a directory as a sequence of FileInfo objects and a file as a sequence of bytes.
• Trace the LoopThroughFiles program, including command-line arguments, DirectoryInfo.GetFiles, BinaryReader, and X2 hexadecimal formatting.
• Read hexadecimal numbers and explain why a hex dump uses two digits per byte.
• Explain why different collections need different access schemes.
• Compare List<T>.ForEach, the foreach statement, and a manual enumerator.
• Describe IEnumerable<T>, IEnumerator<T>, GetEnumerator, MoveNext, Current, and Reset.
• Write and use an indexer declared with this[...].
• Write iterator blocks with yield return and yield break, including named iterators and iterator properties.
• Explain lazy evaluation and the risk of an infinite iterator.

1. Collections beyond arrays

Lesson 6 introduced collection classes: .NET constructs that hold groups of items. This lesson stretches the idea:

• A directory can produce a sequence of files.
• A file can produce a sequence of bytes, lines, or records.
• A custom object can expose values through an enumerator or iterator block.
• The consumer, usually foreach, only needs an agreed traversal protocol; it does not care how the values are stored or produced.

Quick check: what sequence does a directory naturally produce?

2. Iterating through a directory of files

The LoopThroughFiles program visits every file in a directory and “hex-dumps” each one, printing its bytes as hexadecimal numbers. It works for binary files as well as text files. The program is a four-stage pipeline:

1. Choose a directory: use the first command-line argument, or the current directory when none is supplied.
2. Create a file list: DirectoryInfo.GetFiles() returns a FileInfo[].
3. Read bytes: FileInfo.OpenRead() and a BinaryReader supply blocks of bytes.
4. Format output: {0:X2} prints each byte as two hexadecimal digits.

Warnings: a directory with many or large files takes a long time; press Ctrl+C to stop any console program. An invalid directory prints a message such as Directory "x" invalid followed by the system’s “Could not find a part of the path” message and No files left.

Hexadecimal refresher

Hexadecimal (base 16) uses digits 0–9 and then A–F, where A = 10 and F = 15. The 0x prefix marks a hex literal:

Hex	Decimal	Why
0xD	13	D = 13
0x10	16	1 × 16 + 0
0x2A	42	2 × 16 + 10

Letter case does not matter: C equals c. A byte ranges from 0 to 255 (0xFF), so every byte fits in exactly two hex digits.

Program setup and Main

File classes live in System.IO, so the program needs using System.IO; in addition to using System;.

public static void Main(string[] args)
{
    string directoryName;
    if (args.Length == 0)
    {
        directoryName = Directory.GetCurrentDirectory();
    }
    else
    {
        directoryName = args[0];
    }
    Console.WriteLine(directoryName);

    FileInfo[] files = GetFileList(directoryName);

    foreach (FileInfo file in files)
    {
        Console.WriteLine("\n\nhex dump of file {0}:", file.FullName);
        DumpHex(file);
        Console.WriteLine("\nPress Enter to continue to next file");
        Console.ReadLine();
    }

    Console.WriteLine("\nNo files left");
}

• args holds the command-line arguments. An empty array (args.Length == 0) means none were typed.
• Directory.GetCurrentDirectory() returns the working directory. Run from Visual Studio, that is the project’s build output folder.
• The Directory class manipulates directories; the FileInfo class describes a file (its FullName with path, its Name without path, creation and modified dates) and can move, copy, or delete it.
• foreach visits each FileInfo once and delegates the contents to DumpHex, then pauses so the user can read the output.

Trace the program at two levels: files in the outer loop, and bytes inside DumpHex.

Creating the file list

public static FileInfo[] GetFileList(string directoryName)
{
    FileInfo[] files = new FileInfo[0];   // start with an empty list
    try
    {
        DirectoryInfo di = new DirectoryInfo(directoryName);
        files = di.GetFiles();
    }
    catch (Exception e)
    {
        Console.WriteLine("Directory \"{0}\" invalid", directoryName);
        Console.WriteLine(e.Message);
    }
    return files;
}

• A DirectoryInfo object describes a directory the way a FileInfo describes a file, and it can list its files as a FileInfo[].
• File-system access can fail because of a bad path or missing permission. The try/catch block (taught fully in Lesson 9) keeps that failure inside this method.
• Starting with an empty array means the caller’s foreach still works when no files are available; it simply runs zero times. Modern code often writes Array.Empty<FileInfo>().
• Caution from the book: be careful about returning a reference to an object’s internal data. Handing out a reference to a class’s private storage lets callers bypass the class’s rules. Returning this freshly built array exposes nothing private, so it is safe.

Reading and formatting bytes

public static void DumpHex(FileInfo file)
{
    FileStream fs;
    BinaryReader reader;
    try
    {
        fs = file.OpenRead();
        reader = new BinaryReader(fs);
    }
    catch (Exception e)
    {
        Console.WriteLine("\ncan't read from \"{0}\"", file.FullName);
        Console.WriteLine(e.Message);
        return;
    }

    for (int line = 1; true; line++)
    {
        byte[] buffer = new byte[10];
        int numBytes = reader.Read(buffer, 0, buffer.Length);
        if (numBytes == 0)
        {
            return;
        }

        Console.Write("{0:D3} - ", line);
        DumpBuffer(buffer, numBytes);

        if (line % 20 == 0)
        {
            Console.WriteLine("Press Enter to continue another 20 lines or type Q to go to the next file.");
            string input = Console.ReadLine() ?? "";
            if (input.ToUpper() == "Q")
            {
                break;
            }
        }
    }
}

public static void DumpBuffer(byte[] buffer, int numBytes)
{
    for (int index = 0; index < numBytes; index++)
    {
        Console.Write("{0:X2}, ", buffer[index]);
    }
    Console.WriteLine();
}

Talking points:

• A FileInfo only describes a file; OpenRead() actually opens it as a read-only FileStream, and BinaryReader reads bytes from that stream.
• Open-ended loop: for (int line = 1; true; line++) has no fixed count. The reader decides when iteration ends: Read returns 0 when no data remains.
• Buffer: at most ten bytes are requested for each output line.
• Actual count: the last read may return fewer than ten bytes, so DumpBuffer loops to numBytes, not buffer.Length, to avoid printing unused zeros.
• Format strings: {0:D3} pads the line number to three digits (001); {0:X2} prints a byte as two uppercase hex digits (13 becomes 0D).
• Modulo pause: a console window shows about 25 lines, so line % 20 == 0 pauses at lines 20, 40, 60, and so on. This “every Nth time” trick is useful in many loops.

Reading a dump: 0x61 is the letter a, 0x65 is e, 0x20 is a space, and 0D, 0A is a carriage return followed by a line feed. The book’s sample first line (53, 74, 72, 65, 61, 6D, 20, 28, 70, 72) decodes to “Stream (pr”. The book’s own decoding of that line is incorrect; have students decode it with an ASCII table. The same codes are valid for the lower part of Unicode, which C# uses for text.

Running with a command-line argument

Three ways to run LoopThroughFiles:

• From a Developer Command Prompt for VS 2022: loopthroughfiles c:\temp.
• In Visual Studio with a command-line argument: open the project’s Properties, choose Debug, enter a path such as C:\Temp in Command Line Arguments (newer versions place this under a launch profile), save, and start the program.
• With no argument, so the current directory is used.

Quick check: what output format turns byte 13 into 0D? Why does DumpBuffer receive numBytes?

3. The general problem of accessing a collection

Different collection types have different access schemes. An array supports fast indexing, but a linked list holds only a reference to the next item and is meant to be walked in order. This method cannot work for every collection:

void MyClearMethod(Collection aColl, int index)
{
    aColl[index] = 0;   // indexing does not work for all collections
}

The CollectionMoveNext example shows three ways to visit a List<string>:

List<string> colors = new List<string> { "Red", "Yellow", "Green", "Blue" };

// 1. The List<T>.ForEach method with a delegate
colors.ForEach(delegate (string value)
{
    Console.WriteLine(value);
});

// 2. The foreach statement
foreach (string col in colors)
{
    Console.WriteLine(col);
}

// 3. A manual enumerator
var colEnum = colors.GetEnumerator();
while (colEnum.MoveNext())
{
    Console.WriteLine(colEnum.Current);
}

Approach	Strength	Weakness
List<T>.ForEach method	Flexible; accepts a delegate or lambda (Lesson 18)	Harder to read; it is a method call, not the foreach statement
foreach statement	Clear intent: visit each value; the compiler handles the enumerator and disposal	Less control over position
Manual enumerator	Full control over when Current changes; can start or pause at a chosen state	More code

Classroom convention: prefer foreach unless the enumerator itself is part of the problem.

4. The enumerator (iterator) protocol

The book treats iterator and enumerator as synonyms: an object that hands out a collection’s values one at a time. (Indexers and iterator blocks, later in the lesson, are other answers to the same problem.)

Advantages of the enumerator approach:

• Each collection class can define its own iteration class; because it implements the standard IEnumerator interface, it is usually straightforward to code.
• Application code does not need to know how the collection works internally. That is good encapsulation.
• Application code can create several independent iterators over the same collection. Each iterator knows where it is, so several traversals can run at different positions at once.

Because the protocol must work for everything from arrays to linked lists, it is deliberately minimal and does not provide random access. The nongeneric IEnumerator has three members:

• Reset() sets the enumerator back to the beginning of the collection. The generic IEnumerator<T> omits Reset; start a new traversal by calling MoveNext on a fresh enumerator.
• MoveNext() moves to the next object and returns false when the collection is exhausted.
• Current is a property, not a method, that returns the object at the current position.

IEnumerator iterator = new IteratorMyCollection(myColl);
while (iterator.MoveNext())
{
    ContainedDataObject contained = (ContainedDataObject)iterator.Current;
    // … use contained …
}

• The first MoveNext moves to the first element; each later call moves one position.
• Reading Current is invalid before the first MoveNext or after MoveNext returns false.
• The nongeneric Current returns object, so a cast is needed. The generic version avoids the cast.

How foreach uses the protocol

foreach works with any class that implements IEnumerable or IEnumerable<T>:

interface IEnumerable<T>
{
    IEnumerator<T> GetEnumerator();
}

interface IEnumerator<T>
{
    bool MoveNext();
    T Current { get; }
}

• Sequence role: IEnumerable<T> can create a fresh traversal through GetEnumerator().
• Cursor role: IEnumerator<T> represents one traversal in progress.
• Under the hood, foreach calls GetEnumerator(), then repeatedly calls MoveNext() and reads Current, and disposes the enumerator when done.
• IEnumerable<T> and IEnumerator<T> are different but related interfaces. Prefer the generic versions for type safety. IEnumerator<T> inherits from the nongeneric IEnumerator (interface inheritance, Lesson 17).
• Arrays and all .NET collection classes already implement these interfaces. Students need to implement them only when writing a custom collection.

Quick check: which member changes the enumerator’s position? Translate foreach (int n in values) into the protocol steps.

5. Indexers: accessing a collection the array way

container[n] uses the subscript operator [] with an index n. An indexer lets a class define that bracket syntax for itself, including for collections that would not otherwise support it, and with index types other than int, such as container["Joe"].

An indexer looks like a property with this and a parameter in brackets in place of the property name:

public string this[int index]
{
    get { return _items[index]; }
    set { _items[index] = value; }
}

• this[...]: plays the role of the property name.
• get accessor: runs for s = obj[i];.
• set accessor: runs for obj[i] = "text";, receiving the assigned value as value.
• The book’s short form (get => …; set => … = value;) is valid when the accessors do nothing else; its sample refers to the class name instead of a backing array, so show it with a private field as above.

The KeyedArray example

KeyedArray behaves like an array indexed by strings. It keeps two parallel arrays: _keys holds the string identifiers, and _arrayElements holds the data at the matching positions. A non-integer index is called a key.

public class KeyedArray
{
    private string[] _keys;
    private object[] _arrayElements;

    public KeyedArray(int size)          // constructor (Lesson 14)
    {
        _keys = new string[size];
        _arrayElements = new object[size];
    }

    private int Find(string targetKey)
    {
        for (int i = 0; i < _keys.Length; i++)
        {
            if (String.Compare(_keys[i], targetKey) == 0)
            {
                return i;
            }
        }
        return -1;
    }

    private int FindEmpty()
    {
        for (int i = 0; i < _keys.Length; i++)
        {
            if (_keys[i] == null)
            {
                return i;
            }
        }
        throw new Exception("Array is full");
    }

    public object this[string key]
    {
        set
        {
            int index = Find(key);
            if (index < 0)
            {
                index = FindEmpty();
                _keys[index] = key;
            }
            _arrayElements[index] = value;
        }
        get
        {
            int index = Find(key);
            if (index < 0)
            {
                return null;
            }
            return _arrayElements[index];
        }
    }
}

• Find searches the keys and returns the matching index or -1.
• FindEmpty returns the first slot with no key, or throws when the array is full.
• The setter reuses an existing key’s slot or claims an empty one, then stores the value.
• The getter returns the stored value, or null when the key is absent.
• Encapsulation: the parallel arrays stay private behind one access rule. An indexer defines access syntax; it does not require the storage to be an array.

KeyedArray ma = new KeyedArray(100);
ma["Bart"] = 10;
ma["Lisa"] = 8;
ma["Maggie"] = 2;

int age = (int)ma["Lisa"];
Console.WriteLine("Lisa is {0}", age);   // Lisa is 8

The cast (int) is required because KeyedArray stores object. A generic version (Lesson 8) would remove the cast. In production code, Dictionary<TKey, TValue> already provides this behavior; KeyedArray exists to teach the indexer mechanism. An indexer can also be overloaded with several index types on the same class.

Quick check: what plays the role of the method name in an indexer? What does ma["Homer"] return?

6. Iterator blocks

Writing a full enumerator class for every custom collection is tedious. Since C# 2.0, an iterator block lets the compiler write the enumerator. Users of the class simply write foreach, and nobody calls GetEnumerator, MoveNext, or Current by hand or casts the results.

class MonthDays
{
    string[] months =
        { "January 31", "February 28", "March 31", "April 30",
          "May 31", "June 30", "July 31", "August 31",
          "September 30", "October 31", "November 30", "December 31" };

    public System.Collections.IEnumerator GetEnumerator()
    {
        foreach (string month in months)
        {
            yield return month;
        }
    }
}

MonthDays md = new MonthDays();
foreach (string month in md)
{
    Console.WriteLine(month);
}

• A method containing yield return becomes resumable traversal code. The compiler generates the enumerator class and the state that remembers where execution paused.
• The class no longer implements IEnumerator itself; it only provides GetEnumerator() with an iterator block.
• Iterator blocks often contain a loop that yields each stored item, so an inner loop serves items to an outer foreach.

What a collection really is

In the most general sense, a collection stores or produces multiple objects and lets a consumer retrieve them one at a time. It may not store anything in the usual sense:

class StringChunks
{
    public System.Collections.IEnumerator GetEnumerator()
    {
        yield return "Using iterator ";
        yield return "blocks ";
        yield return "isn't all ";
        yield return "that hard";
        yield return ".";
    }
}

StringChunks has no array at all; its “collection” is the sequence of yield return statements. Iterated with WriteLine, each chunk prints on its own line; iterated with Write, the chunks form one sentence.

yield return and yield break

	yield return	yield break
Effect	Produce one value and pause the iterator	End the sequence immediately
Next request	Resume right after the statement	MoveNext reports false
Resembles	MoveNext() plus Current	The break statement

The first request returns the first value, the next request the second, and so on. Between requests the iterator remembers its position.

class YieldBreakEx
{
    int[] primes = { 2, 3, 5, 7, 11, 13, 17, 19, 23 };

    public System.Collections.IEnumerator GetEnumerator()
    {
        foreach (int prime in primes)
        {
            if (prime > 13) yield break;
            yield return prime;
        }
    }
}
// Produces 2, 3, 5, 7, 11, 13

yield break ends the iterator even though source data remains.

Iterator blocks do not have to live in a formal collection class; they can be static methods beside Main, or extension methods (Part 2).

7. Iterator shapes: named iterators and properties

Besides GetEnumerator(), an iterator block can take two other forms.

Named iterators

A named iterator is a method with any name that returns IEnumerable (not IEnumerator):

class EvenNumbers
{
    public System.Collections.IEnumerable DescendingEvens(int top, int stop)
    {
        if (top % 2 != 0)
        {
            top -= 1;
        }
        for (int i = top; i >= stop; i -= 2)
        {
            yield return i;
        }
    }
}

EvenNumbers en = new EvenNumbers();
foreach (int even in en.DescendingEvens(11, 3))
{
    Console.WriteLine(even);   // 10, 8, 6, 4
}

• Normalize input: an odd top moves down to the nearest even number.
• Computed values: nothing is stored; each loop pass computes and yields one number. Iteration can go down, or in steps of two, or any way the method chooses.
• The odd stop value 3 is never yielded because only even values are generated.
• The book’s version also tests if (i < stop) yield break; inside the loop; that test can never be true because the loop condition already guarantees i >= stop.
• If the method were static, it would be called through the class: EvenNumbers.DescendingEvens(11, 3).

A modern generic form, IEnumerable<int> DescendingEvens(int top, int stop), removes the need for the nongeneric interface and gives foreach typed values.

Infinite iterators and laziness

public System.Collections.IEnumerable PositiveIntegers()
{
    for (int i = 0; ; i++)
    {
        yield return i;
    }
}

Defining this iterator does not build every integer. Values are produced lazily, only as the consumer requests them. The consumer is responsible for stopping, for example with break, or the method should accept a stop value. Streams of powers of two, squares, primes, Fibonacci numbers, or random values are natural iterator ideas.

Iterator properties

A property’s get accessor can be an iterator block:

class PropertyIterator
{
    double[] doubles = { 1.0, 2.0, 3.5, 4.67 };

    public System.Collections.IEnumerable DoubleProp
    {
        get
        {
            foreach (double db in doubles)
            {
                yield return db;
            }
        }
    }
}

PropertyIterator prop = new PropertyIterator();
foreach (double db in prop.DoubleProp)
{
    Console.WriteLine(db);   // 1, 2, 3.5, 4.67
}

The property returns IEnumerable, has no parentheses, and has a get accessor but no set.

Shape	Returns	Used as
GetEnumerator() iterator	IEnumerator	foreach (var x in obj)
Named iterator	IEnumerable	foreach (var x in obj.Method(args))
Iterator property	IEnumerable	foreach (var x in obj.Property)

Quick check: who must stop an infinite iterator? Which shape lets the caller pass arguments?

Classroom application

Case study: a scheduling tool must expose future class dates without building a giant list first, and callers should use foreach and stop after the dates they need.

1. Decide whether an indexer or an iterator is the clearer public surface. (An iterator fits lazy, sequential access; an indexer fits lookup by a key such as schedule["Monday"].)
2. Write IEnumerable<DateTime> ClassDates(DateTime start) that yields every Tuesday and Thursday, and have Main stop after ten dates.
3. Decide on termination: should the iterator yield break at the end of the semester, or should the consumer stop it?
4. Extension: write a Schedule class with a string indexer that stores a room for each day name.

Identify the mechanism before revealing:

• A foreach loop needs a fresh object that tracks its position. (Enumerator from GetEnumerator.)
• A custom object should support schedule["Monday"]. (Indexer.)
• A sequence should compute one value only when requested. (yield return.)
• A prime sequence must stop before values greater than 13. (yield break.)

Common misconceptions

• A collection does not have to store its items; it only has to produce them one at a time.
• List<T>.ForEach is a method that takes a delegate; foreach is a language statement.
• Current is invalid before the first MoveNext and after MoveNext returns false.
• IEnumerator<T> has no Reset; start over with a new enumerator.
• IEnumerable<T> is the sequence that creates enumerators; IEnumerator<T> is one traversal in progress.
• Several enumerators over one collection keep independent positions.
• An indexer is declared with this[...]; it defines bracket syntax, not a storage format.
• A FileInfo describes a file; it does not open it.
• The last buffer read may be partly full; format only the bytes actually read.
• yield return pauses the method; it does not end it. yield break ends it.
• Iterator blocks are lazy, so an infinite iterator is safe only if the consumer stops.

Lesson summary

Directories, files, arrays, and custom objects can all expose values as sequences. foreach relies on a small protocol: an IEnumerable<T> supplies an IEnumerator<T> that advances with MoveNext and exposes Current. Indexers give a class keyed, array-like access with this[...], and iterator blocks let the compiler build the enumerator from yield return and yield break, producing values lazily through GetEnumerator, named iterators, or properties. Lesson 8 shows how generics keep one implementation type-safe for many element types.