In the previous tutorial, you learned database access with sqlx. Now let us work with files. File I/O is one of the most common tasks in Go — reading configuration files, processing logs, exporting data, and more.

Go makes file I/O simple with the os and bufio packages. And the io.Reader and io.Writer interfaces make everything composable.

Reading an Entire File

The simplest way to read a file is os.ReadFile:

package main

import (
    "fmt"
    "log"
    "os"
)

func main() {
    data, err := os.ReadFile("hello.txt")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(data))
}

os.ReadFile reads the entire file into memory as a byte slice. This is fine for small files. For large files, use a scanner or reader instead.

Writing a File

Write a file with os.WriteFile:

package main

import (
    "log"
    "os"
)

func main() {
    content := []byte("Hello, Go!\nThis is a test file.\n")

    err := os.WriteFile("output.txt", content, 0644)
    if err != nil {
        log.Fatal(err)
    }
}

0644 is the file permission — owner can read and write, others can only read. This is the standard permission for most files.

Opening and Closing Files

For more control, use os.Open (read) and os.Create (write):

package main

import (
    "fmt"
    "log"
    "os"
)

func main() {
    // Create and write
    file, err := os.Create("data.txt")
    if err != nil {
        log.Fatal(err)
    }

    _, err = file.WriteString("Line 1\nLine 2\nLine 3\n")
    if err != nil {
        file.Close()
        log.Fatal(err)
    }
    file.Close()

    // Open and read
    file, err = os.Open("data.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    buf := make([]byte, 1024)
    n, err := file.Read(buf)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Read %d bytes:\n%s", n, buf[:n])
}

Important: Always close files when you are done. Use defer file.Close() right after opening. This ensures the file is closed even if the function returns early due to an error.

Reading Line by Line with bufio.Scanner

For large files, read line by line instead of loading everything into memory:

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    file, err := os.Open("data.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)

    lineNum := 0
    for scanner.Scan() {
        lineNum++
        fmt.Printf("%d: %s\n", lineNum, scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        log.Fatal(err)
    }
}

scanner.Scan() returns true for each line and false at the end of the file. scanner.Text() returns the current line as a string (without the newline character).

The default maximum line length is 64 KB. For longer lines, increase the buffer:

scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 1024*1024), 1024*1024) // 1 MB buffer

Buffered Writing with bufio.Writer

Writing many small pieces is slow because each write goes to disk. Use bufio.Writer to buffer writes:

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    file, err := os.Create("buffered.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    writer := bufio.NewWriter(file)

    for i := 1; i <= 1000; i++ {
        fmt.Fprintf(writer, "Line %d: This is some content\n", i)
    }

    // Flush remaining data to disk
    err = writer.Flush()
    if err != nil {
        log.Fatal(err)
    }
}

Important: Always call writer.Flush() before closing the file. Otherwise, data in the buffer will be lost.

io.Reader and io.Writer — Go’s I/O Foundation

These two interfaces are the foundation of all I/O in Go:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Files, network connections, HTTP bodies, buffers, and many other types implement these interfaces. This makes them composable — you can connect any reader to any writer.

io.Copy — Connect a Reader to a Writer

package main

import (
    "io"
    "log"
    "os"
)

func main() {
    // Copy file
    src, err := os.Open("source.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer src.Close()

    dst, err := os.Create("copy.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer dst.Close()

    bytes, err := io.Copy(dst, src)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Copied %d bytes", bytes)
}

io.Copy reads from the reader and writes to the writer until the reader returns io.EOF. It handles buffering internally.

io.MultiWriter — Write to Multiple Destinations

package main

import (
    "io"
    "log"
    "os"
)

func main() {
    file, err := os.Create("log.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // Write to both file and stdout
    multi := io.MultiWriter(os.Stdout, file)

    io.WriteString(multi, "This goes to the screen AND the file\n")
    io.WriteString(multi, "So does this\n")
}

io.TeeReader — Read and Copy at the Same Time

package main

import (
    "io"
    "log"
    "os"
    "strings"
)

func main() {
    reader := strings.NewReader("Hello, this is some data")

    file, err := os.Create("tee.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // Read from reader, copy everything to file
    tee := io.TeeReader(reader, file)

    // Reading from tee also writes to file
    data, err := io.ReadAll(tee)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Read: %s", data)
    // file now contains the same data
}

Working with Directories

Create a Directory

// Create a single directory
err := os.Mkdir("mydir", 0755)

// Create nested directories (like mkdir -p)
err := os.MkdirAll("path/to/nested/dir", 0755)

List Directory Contents

package main

import (
    "fmt"
    "log"
    "os"
)

func main() {
    entries, err := os.ReadDir(".")
    if err != nil {
        log.Fatal(err)
    }

    for _, entry := range entries {
        info, _ := entry.Info()
        if entry.IsDir() {
            fmt.Printf("[DIR]  %s\n", entry.Name())
        } else {
            fmt.Printf("[FILE] %s (%d bytes)\n", entry.Name(), info.Size())
        }
    }
}

Walk a Directory Tree

Use filepath.WalkDir to recursively visit all files and directories:

package main

import (
    "fmt"
    "io/fs"
    "log"
    "path/filepath"
)

func main() {
    err := filepath.WalkDir(".", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if d.IsDir() {
            fmt.Printf("[DIR]  %s\n", path)
        } else {
            fmt.Printf("[FILE] %s\n", path)
        }

        return nil
    })

    if err != nil {
        log.Fatal(err)
    }
}

To skip a directory, return filepath.SkipDir from the callback.

Temporary Files

Use os.CreateTemp for temporary files that you do not need to keep:

package main

import (
    "fmt"
    "log"
    "os"
)

func main() {
    // Create temp file in the default temp directory
    tmpFile, err := os.CreateTemp("", "myapp-*.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer os.Remove(tmpFile.Name()) // Clean up when done
    defer tmpFile.Close()

    fmt.Println("Temp file:", tmpFile.Name())

    tmpFile.WriteString("temporary data")
}

The * in the pattern is replaced with a random string. This prevents name collisions.

JSON File Processing

Read and write JSON files:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "os"
)

type Config struct {
    Host     string `json:"host"`
    Port     int    `json:"port"`
    Database string `json:"database"`
    Debug    bool   `json:"debug"`
}

func main() {
    // Write JSON file
    config := Config{
        Host:     "localhost",
        Port:     8080,
        Database: "tutorial",
        Debug:    true,
    }

    data, err := json.MarshalIndent(config, "", "  ")
    if err != nil {
        log.Fatal(err)
    }

    err = os.WriteFile("config.json", data, 0644)
    if err != nil {
        log.Fatal(err)
    }

    // Read JSON file
    fileData, err := os.ReadFile("config.json")
    if err != nil {
        log.Fatal(err)
    }

    var loaded Config
    err = json.Unmarshal(fileData, &loaded)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Host: %s, Port: %d\n", loaded.Host, loaded.Port)
}

CSV File Processing

Read and write CSV files with the encoding/csv package:

package main

import (
    "encoding/csv"
    "fmt"
    "log"
    "os"
)

func main() {
    // Write CSV file
    file, err := os.Create("users.csv")
    if err != nil {
        log.Fatal(err)
    }

    writer := csv.NewWriter(file)
    writer.Write([]string{"Name", "Email", "Age"})
    writer.Write([]string{"Alex", "alex@example.com", "30"})
    writer.Write([]string{"Sam", "sam@example.com", "25"})
    writer.Write([]string{"Jordan", "jordan@example.com", "28"})
    writer.Flush()
    file.Close()

    // Read CSV file
    file, err = os.Open("users.csv")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    reader := csv.NewReader(file)
    records, err := reader.ReadAll()
    if err != nil {
        log.Fatal(err)
    }

    for i, record := range records {
        if i == 0 {
            continue // Skip header
        }
        fmt.Printf("Name: %s, Email: %s, Age: %s\n", record[0], record[1], record[2])
    }
}

For large CSV files, read one record at a time instead of using ReadAll:

reader := csv.NewReader(file)

for {
    record, err := reader.Read()
    if err != nil {
        break // EOF or error
    }
    fmt.Println(record)
}

Error Handling Patterns for File Operations

Always handle file errors properly:

func processFile(path string) error {
    file, err := os.Open(path)
    if err != nil {
        if os.IsNotExist(err) {
            return fmt.Errorf("file does not exist: %s", path)
        }
        if os.IsPermission(err) {
            return fmt.Errorf("permission denied: %s", path)
        }
        return fmt.Errorf("failed to open file: %w", err)
    }
    defer file.Close()

    // Check if it is a directory
    info, err := file.Stat()
    if err != nil {
        return fmt.Errorf("failed to stat file: %w", err)
    }
    if info.IsDir() {
        return fmt.Errorf("%s is a directory, not a file", path)
    }

    // Process the file...
    return nil
}

Check for specific error types:

  • os.IsNotExist(err) — file does not exist
  • os.IsPermission(err) — no permission to access
  • os.IsExist(err) — file already exists (when creating)

Common Mistakes

1. Forgetting to close files.

// Wrong — file is never closed
file, _ := os.Open("data.txt")
data, _ := io.ReadAll(file)

// Correct — use defer
file, _ := os.Open("data.txt")
defer file.Close()
data, _ := io.ReadAll(file)

Unclosed files leak file descriptors. Your program can run out of file descriptors and crash.

2. Not flushing buffered writers.

writer := bufio.NewWriter(file)
writer.WriteString("data")
// Missing writer.Flush() — data may be lost!

Always flush before closing.

3. Reading large files into memory.

// Bad for large files — loads everything into memory
data, _ := os.ReadFile("huge.log")

// Better — read line by line
scanner := bufio.NewScanner(file)
for scanner.Scan() {
    processLine(scanner.Text())
}

Use bufio.Scanner or io.Reader for large files.

Source Code

You can find the complete source code for this tutorial on GitHub:

GO-20 Source Code on GitHub

What’s Next?

  • Struct validation with the validator library
  • Pagination and rate limiting
  • Graceful shutdown
  • Structured logging with slog

This is part 20 of the Go Tutorial series. Follow along to learn Go from scratch.