Julia Evans

Key Insight: Contributing examples to official man pages is a high-leverage way to improve documentation accuracy and accessibility, because the review process guarantees correctness in a way blog posts never can.

Julia Evans contributed examples sections to the official man pages for tcpdump and dig, motivated by her earlier writing about how examples make man pages more useful. Her goal was to provide the most basic, beginner-friendly examples for infrequent users who don't remember how the tools work. She found the process rewarding and collaborative, learning new things from maintainers along the way. The experience shifted her perspective on official documentation, making her cautiously optimistic that it could be as useful as blog posts while being more accurate. She also wrote a custom markdown-to-roff converter to avoid learning the roff language directly.

Key Insight: Man pages can be dramatically improved through design techniques like options summaries, categorical organization, embedded examples, and cheat sheets — all within the existing constrained format.

Julia Evans explores what makes man pages effective and easier to navigate, drawing on examples from rsync, strace, curl, Perl, and the ASCII man page. She crowdsourced favorite man pages on Mastodon and catalogs specific techniques: options summaries, category-based organization, embedded cheat sheets, tables of contents with hyperlinks, per-option examples, and table formatting. She also touches on the GNU project's preference for info manuals over man pages and notes tools like tldr and Dash that supplement man pages. The post frames man page improvement as a design challenge within a constrained format.

Key Insight: Mature, 'boring' frameworks like Django reward beginners with explicitness and batteries-included features that let you get things done without fighting the tooling.

Julia Evans shares her experience starting to learn Django after years of building websites with Go binaries and static sites. She compares it favorably to Rails, appreciating Django's explicitness which makes it easier to return to projects after months away. She highlights several features she's enjoying: the built-in admin interface, the ORM's elegant JOIN syntax using double underscores, automatic migrations, and the batteries-included approach to email, CSRF, and other web features. She's running Django with SQLite in production and finding the documentation excellent, though she remains slightly uneasy about the settings file's lack of typo protection.

Key Insight: The most effective way to improve open source documentation is not expert debate about clarity, but gathering evidence from real users about what actually confuses them.

Julia Evans describes her effort to improve Git's official documentation, including writing a new 'data model' document explaining core concepts like objects, references, and the index. Rather than relying on expert intuition about what's clear, she recruited ~80 test readers on Mastodon to identify confusing terminology and sentences in existing man pages. This evidence-based approach helped her update man pages for git add, checkout, push, and pull. She reflects on the difficulty of writing documentation that is both accurate and clear, and shares notes on Git's contribution process including using GitGitGadget to avoid learning email-patch workflows.

Key Insight: The best tool configuration is the one you don't have to maintain — built-in sensible defaults can outweigh decades of customized muscle memory.

Julia Evans shares her experience switching from Vim to the Helix text editor after 20 years of Vim usage. The primary motivation was getting language server support without extensive configuration. She finds Helix's built-in LSP support, search with full context, and inline keybinding help significantly better than her Vim setup. While she notes several annoyances including poor Markdown list handling and occasional crashes, the switch was easier than expected. Her Helix configuration is remarkably minimal compared to her hundreds-of-lines Neovim config.

Key Insight: Even after 20 years of daily terminal use, the terminal's behavior remains poorly understood because it's not one system but many independent pieces of software with unwritten rules, and selectively learning its internals transforms vague intuition into genuine debugging ability.

Julia Evans announces her new zine 'The Secret Rules of the Terminal,' the product of months of deep-dive research into how terminals actually work. Despite using the terminal daily for 20 years, she discovered she had many misunderstandings about its inner workings. The zine explains how the four pieces of the terminal (shell, terminal emulator, programs, and TTY driver) fit together, along with core conventions and practical tips. She argues that while terminal internals are a mess of decades-old decisions, understanding select parts can meaningfully improve your daily experience. The project involved extensive collaboration with technical reviewers and was accompanied by numerous blog posts exploring specific terminal topics.

Key Insight: Compiling C programs from source is demystified once you understand that most build errors are dependency problems solvable by passing the right flags to the compiler and linker via make's environment variables.

Julia Evans explains how non-C-programmers can compile C programs from source using make, walking through the full process from installing a compiler to dealing with dependency errors. She uses real examples (paperjam, sqlite, qf) to demonstrate common problems and solutions, focusing on how environment variables like CPPFLAGS and LDLIBS pass flags to the compiler and linker. The post demystifies the configure/make workflow and offers practical tips like looking at how Linux distributions build the same package to find clues for fixing errors. She concludes that understanding C compilation basics is valuable even for people who never plan to write C.

Key Insight: The terminal's escape code ecosystem suffers from the same fragmentation that plagued early web browsers, and could benefit from the same shift toward clear, shared standards rather than per-terminal detection.

Julia Evans surveys the landscape of ANSI escape code standards, from the 1976 ECMA-48 specification through xterm control sequences and the terminfo database. She identifies two competing approaches programs take: using terminfo to adapt to different terminals, or hardcoding a 'common set' of widely-supported escape codes. She draws a parallel between terminfo's TERM variable and browser user agent detection, noting both led to similar workaround patterns. The post concludes that clearer standards could make terminals feel less 'outdated' and enable richer features, drawing hope from how web standards eventually improved from a similarly fragmented state.

Key Insight: Adding a directory to your PATH involves multiple shell-specific decisions and subtle gotchas that are rarely documented in one place, making a seemingly simple task genuinely confusing for newcomers.

Julia Evans writes a comprehensive guide to adding a directory to your PATH, a task that seems obvious to experienced terminal users but has many undocumented gotchas. She walks through identifying your shell, finding the correct config file, determining which directory to add, and restarting the shell. The post also covers common problems like running the wrong program version, duplicate PATH entries, and losing shell history when reloading config. She includes shell-specific syntax for bash, zsh, and fish, and discusses tools like cargo and Homebrew that use source scripts instead of explicit PATH additions.

Key Insight: Terminal frustrations aren't beginner problems — even decades-long users struggle with syntax recall, system inconsistencies, and poor discoverability, revealing fundamental UX failures baked into the terminal's legacy design.

Julia Evans surveyed 1600 terminal users about their biggest frustrations and spent days categorizing the qualitative responses. The top complaints include remembering syntax, difficulties switching between systems, color configuration problems, keyboard shortcut inconsistencies, and copy/paste issues. Notably, 95% of respondents had 4+ years of terminal experience and 40% had 21+ years, meaning these frustrations persist even for experts. The results reveal that discoverability, steep learning curves, and legacy design decisions continue to plague the terminal experience decades into people's careers. Evans connects several categories as related problems: syntax recall ties to history search, discoverability drives the learning curve, and the outdated feel of terminals is precisely what makes them ubiquitous.

Key Insight: Getting a 'modern' terminal experience is genuinely hard because it requires coordinating many interdependent components, each with its own configuration, and experienced users underestimate this because their own setup evolved incrementally over decades.

Julia Evans explores what it actually takes to achieve a 'modern' terminal experience, cataloging the specific features she considers essential and which system components are responsible for each. She starts by thinking it's simple, then realizes the list of requirements keeps growing. The post identifies three main obstacles: shell configuration (bash/zsh defaults are inadequate), terminal text editor setup (vim/emacs require years of investment), and individual application quirks. She emphasizes that terminal configuration is an interconnected system where changing one thing can break others, and recommends making changes slowly over time. Her honest conclusion is that getting a modern terminal experience is genuinely hard, despite what experienced users might assume.

Key Insight: Pipes don't get stuck — programs silently buffer their output when they detect they're not writing to a terminal, and understanding the isatty check is the key to diagnosing and fixing the problem.

The post explains why piping commands like 'tail -f file | grep thing1 | grep thing2' often appear to produce no output. The root cause is that programs buffer their output (typically 8KB) when writing to a pipe rather than a terminal, so data never actually reaches the next command in the pipeline. Programs use the isatty function to decide between line buffering (for terminals) and block buffering (for pipes). Evans catalogues which common commands buffer, how various programming languages handle it, and presents five practical solutions including stdbuf, unbuffer, and using line-buffered flags. She also notes that pressing Ctrl-C on a buffered pipeline loses the buffer contents entirely.

Key Insight: Terminal programs follow a consistent set of unwritten conventions that feel like laws of nature but are actually explicitly implemented by each program, and making these implicit rules explicit accelerates learning for newcomers.

Julia Evans catalogs the unwritten rules that terminal programs follow, despite there being no formal standards beyond POSIX and a few guidelines. She identifies seven conventions: Ctrl-C quits noninteractive programs, q quits TUIs, Ctrl-D quits REPLs, programs stick to 16 ANSI colors, readline keybindings are loosely supported, colors are disabled when piping, and - means stdin/stdout. She emphasizes these are descriptive observations from 20 years of terminal use, not prescriptions, and notes that understanding which rules are the program's responsibility helps explain why implementations vary slightly.

Key Insight: Importing JavaScript libraries without a build system requires understanding the three types of JS files (classic, ES module, CommonJS) and knowing how to identify and work with each, which is learnable detective work despite poor documentation.

Julia Evans explains how to import frontend JavaScript libraries without using a build system, a task she finds repeatedly frustrating because most library documentation assumes a build system. She identifies three types of JS files libraries provide: 'classic' global variable files, ES modules, and CommonJS modules. She walks through real examples (Chart.js, @atcute/oauth-browser-client, @atproto/oauth-client-browser) showing how to identify each type and use them in the browser. The post covers practical techniques including UMD files with script tags, ES modules with importmaps, and using esm.sh to convert CommonJS modules. She concludes that ES modules being a web standard gives her confidence in their longevity, and frames the post as understanding the JS landscape rather than complaining about it.

Key Insight: Not everything needs to be a full blog post or permanently archived — creating a lightweight TIL section fills the gap between ephemeral social media and polished blog posts.

Julia Evans describes adding a new TIL (Today I Learned) section to her blog for saving interesting tools and facts she'd otherwise only post on social media. The motivation was that useful links and tips she shared on Mastodon/Bluesky were getting lost because she couldn't remember them later. She set up a lightweight system with custom styling, a Rake task for quick posting, and a separate RSS feed. She also reflects on the POSSE philosophy, concluding she prefers letting most social media posts be ephemeral while selectively archiving specific categories on her own site.

Key Insight: The 33 ASCII control characters are a messy historical artifact where telegraph-era codes, OS signals, readline shortcuts, and application bindings all collide in the same tiny namespace with no logical organization.

Julia Evans explores the 33 ASCII control characters and how they actually work in modern terminals. She categorizes them into OS-handled codes (like Ctrl-C for SIGINT), literal keypresses (Enter, Tab), readline shortcuts, and application-specific bindings. The post reveals surprising details like how there are only 33 possible control codes, how Ctrl-M is literally identical to Enter, and how the original ASCII names bear little relationship to modern terminal behavior. She discusses the complexity of canonical vs noncanonical mode, the configurable nature of these codes via stty, and the surprisingly deep history behind something as simple as the Backspace key.

Key Insight: Simple data structure optimizations like deduplicating strings and using properly-sized types can dramatically reduce memory usage without sacrificing the simplicity and speed of a basic binary search approach.

Julia Evans describes her journey to reduce memory usage in Mess With DNS, which was getting OOM killed on a 465MB VM. The main culprit was a 117MB in-memory IP-to-ASN lookup database. She tried three approaches: SQLite (worked but was 500x slower), a trie (used even more memory), and optimizing her existing array-based binary search. The winning solution combined deduplicating ASN name/country strings and switching from net.IP to Go's newer netip.Addr type, cutting memory from 117MB to 46MB. She also honestly describes how the real debugging process was far messier than the linear narrative suggests.

Key Insight: Upgrading a static site generator after years of stability is mostly a markdown renderer migration problem, where the hardest part is finding silent rendering breakages that produce no errors.

Julia Evans documents her experience upgrading Hugo from version 0.40 (2018) to v0.135 (2024), detailing every breaking change she encountered along the way. The most painful change was switching markdown renderers from Blackfriday to Goldmark, which required manually updating 80 out of 700 markdown files. She developed a systematic process of diffing old and new HTML output to find silent rendering differences. Despite the tedium, she considers the upgrade worthwhile because the new CommonMark-compliant renderer is more futureproof and actually fixed some long-standing rendering bugs she hadn't noticed. She also reflects on static site generator backwards compatibility, noting Jekyll and 11ty are reportedly better in this regard.

Key Insight: Terminal colors are a mess of unstandardized defaults, competing color modes, and colorscheme remapping conflicts, but modern features like minimum contrast and 24-bit color support are finally making the situation manageable.

Julia Evans explores the surprisingly complex world of terminal color configuration, cataloging 11 distinct problems users face from unreadable blue-on-black text to vim colorscheme synchronization issues. She explains how the 16 ANSI colors lack standardization across terminal emulators, how 256-color and 24-bit color modes create additional complexity, and how colorscheme remapping can cause unexpected clashes with programs like ngrok. She shares her personal setup using base16-shell with Solarized Light and discusses the evolution of vim's color support from ANSI-only to 24-bit hex colors in 2016. Her biggest practical takeaway is enabling the 'minimum contrast' feature available in modern terminals like iTerm2, which automatically adjusts colors to ensure readability.

Key Insight: Go's explicit, boilerplate-heavy approach to web development pays off over time because code you can read and understand after a two-year break is more valuable than code that was magical to write initially.

Julia Evans shares practical lessons from building a web application in Go, covering routing improvements in Go 1.22, SQL code generation with sqlc, SQLite configuration tips, and garbage collector tuning. She highlights a trailing-slash gotcha in Go's new router that silently breaks POST requests by redirecting them to GET. She explains why she prefers Go for small web projects: single binary deployment, minimal toolchain setup, and code that's easy to return to after long breaks. She contrasts this with Rails, where the magic that makes initial development fast becomes an obstacle when returning to abandoned projects.

Key Insight: The best developer tools are the ones that just work without configuration, and fish proves that smart defaults can eliminate entire categories of terminal frustrations most people have learned to tolerate.

Julia Evans explains why she still loves the fish shell after 10 years of daily use, expanding on a 2017 post with new reasons. She highlights fish's zero-configuration philosophy, smart history autosuggestions that validate against existing files, safe multiline pasting, terminal repair on every prompt, and sensible history defaults. She acknowledges downsides like occasional missing tool integrations and confusing PATH management with fish_add_path. The post concludes that fish is ideal for people who want great defaults without configuration overhead, noting its growing popularity at 16% in her Mastodon poll despite never being a system default.

Key Insight: When building on top of an existing API for internal use, pragmatism beats purity — tailor your API to your frontend's needs, duplicate information freely, and translate error messages for your actual users.

Julia Evans describes migrating Mess With DNS, her DNS learning playground, from a custom DNS implementation to PowerDNS after users reported various RFC compliance issues. The migration involved eight distinct challenges including intercepting DNS queries, redesigning the API, managing record IDs, translating error messages, switching from Postgres to SQLite, upgrading Vue.js, improving frontend state management, and sequencing the rollout. She shares practical lessons about API design, particularly that internal APIs should be tailored to the frontend's needs rather than following abstract purity principles. The project was split into phases and deployed incrementally to keep changes manageable.

Key Insight: Years of casual experience with a programming language can leave fundamental misconceptions undetected, and structured 'common mistakes' resources are an efficient way to surface and fill those gaps.

Evans describes discovering that Go structs are copied on assignment, a basic language behavior she'd missed despite years of casual Go usage. A bug in a range loop where she returned a pointer to a copied struct rather than the original led her to the '100 Go Mistakes' book, which revealed additional gaps in her understanding. She catalogs several Go pitfalls including slice append side effects and the distinction between value and pointer receivers. The post serves as both a practical Go debugging story and an endorsement of the '100 common mistakes' learning format.

Key Insight: Terminal text input is confusing because programs silently use four different input systems with varying capabilities, but learning to identify which one you're in transforms the experience from chaotic to predictable.

The post categorizes terminal text input into four modes: baseline (no input library), readline-based, alternative libraries like libedit, and fully custom systems. Evans explains why entering and editing text at the command line is surprisingly complicated, primarily due to inconsistency between programs. She traces readline's keybindings back to Emacs, explains why some programs don't use readline (licensing, simplicity, minimal interactivity), and recommends rlwrap as a way to add readline support to any program. The core message is that understanding which input mode you're in makes the terminal feel predictable rather than chaotic.

Key Insight: Shell job control is an underused but practical tool with specific scenarios where it outperforms the common habit of opening new terminal tabs, especially for killing unresponsive processes and correlating interleaved output from background tools like tcpdump.

Evans explores the use cases for shell job control (fg, bg, Ctrl+Z) after polling her Mastodon audience for their habits. She explains the three process states (foreground, background, stopped) and the commands for moving between them. The post catalogs nine practical reasons people use job control, from killing unresponsive processes to suspending CPU-hungry programs. Evans admits she mostly prefers opening new terminal tabs but found several use cases compelling, particularly killing Ctrl+C-resistant processes and running tcpdump in the background alongside other commands.

Key Insight: Git's real difficulty isn't its internals but the gap between its confusing UI messages and the straightforward logic underneath — once you bridge that gap, the fear disappears.

Julia Evans announces her new zine 'How Git Works,' designed for people who've used git for years but still find it intimidating. She explains that git's confusing terminology, misleading messages, and inconsistent behavior make it genuinely hard to learn from usage alone, but once you understand the internal logic, dealing with its quirks becomes routine. The zine focuses on git's observable behavior rather than deep internals, connecting what users see in the UI to what's happening under the hood. She worked with collaborators for 8 months and incorporated feedback from 66 beta readers to refine the explanations.

Key Insight: Git's error messages are confusing not because users lack knowledge, but because the messages omit context, use inconsistent terminology, and overload users with options instead of guiding them toward the most likely fix.

Julia Evans examines several confusing Git error messages, explaining why they're misleading and what she does to work around them. She covers messages from git push, git pull, git checkout, git switch, git rebase, and git clean, noting patterns like ambiguous references to HEAD and 'them' that change meaning depending on context. She acknowledges that improving error messages is genuinely difficult due to translation requirements, funding challenges, and the difficulty of knowing whether a new message is actually better. The post reveals that even experienced Git users develop workarounds rather than truly understanding every message, and that much confusion stems from Git's dual-purpose commands and inconsistent terminology.

Key Insight: Approaching a new hobby with low expectations, embracing mistakes, and resisting the urge to optimize or buy more stuff makes learning more enjoyable and sustainable.

Julia Evans shares her experience learning to crochet tiny cacti and other amigurumi as a non-computer hobby. She describes her deliberately low-pressure, chaotic approach: modifying patterns on the fly, skipping recommended tools like safety eyes and stitch markers, and embracing mistakes as learning opportunities. The post covers practical notes on yarn types, hook sizes, stitches learned, and strategies for dealing with crochet's counting requirements. She emphasizes keeping the hobby compact and low-investment, repurposing old materials instead of buying new ones. The post is part of April Cools Day, where tech bloggers write about topics outside their usual domain.

Key Insight: Even experienced Git users have significant gaps in their understanding of core terminology, which means technical writers should never assume familiarity with terms like HEAD, fast-forward, or reference.

Julia Evans shares results from dozens of Mastodon polls about how people use, understand, and think about Git. The polls cover workflows (merge vs rebase), mental models (commits as diffs vs snapshots), terminology confidence (HEAD, fast-forward, reference), and common problems (merge conflicts, lost work). She acknowledges the polls are unscientific but finds them useful for checking her assumptions against how others actually use Git. The results consistently reveal that many experienced developers have significant gaps in their understanding of core Git terminology.

Key Insight: Git's 'current branch' is not a single well-defined concept but at least four different things that usually agree but meaningfully diverge during rebases, tag checkouts, and other edge cases.

Evans explores what 'current branch' actually means in git by testing four different definitions across six scenarios. She compares what .git/HEAD contains, what git status reports, what was most recently checked out, and what the shell prompt shows. Through scenarios like detached HEAD, mid-rebase, bare repositories, and fresh git init, she demonstrates that these definitions frequently disagree with each other. The post concludes that 'current branch' is not as well-defined a concept as the git glossary implies, and that git does significant work behind the scenes to track context beyond what HEAD alone captures.

Key Insight: HEAD in git is more confusing than it appears because it conflates three related concepts and its meaning changes depending on context, particularly flipping between merge and rebase operations.

Julia Evans investigates why so many git users feel unconfident about understanding HEAD, discovering it's more complicated than she initially thought. She breaks HEAD into three distinct concepts: the .git/HEAD file, HEAD as a revision parameter in commands, and HEAD as it appears in various git output formats. She explains detached HEAD state, how to interpret git log's parenthetical references, and why HEAD in merge conflicts is particularly confusing because its meaning flips between merge and rebase operations. She concludes that git's inconsistent terminology around HEAD makes it harder to learn than it needs to be.

Key Insight: Crowdsourcing which git config options real developers actually use is far more valuable than reading documentation, because it surfaces the 20 options that matter from the 600 that exist.

Julia Evans surveyed her Mastodon followers about their favorite git config options and compiled the most popular ones into a ranked guide. The post covers options ranging from pull behavior (pull.rebase, pull.ff) and merge conflict display (zdiff3) to diff algorithms (histogram) and push defaults. She explains each option with practical examples, showing how merge.conflictstyle diff3 adds original code to conflict markers and how diff.algorithm histogram produces clearer diffs when functions are reordered. Evans shares her own minimal config and the three options she added after writing the post, while noting she's learned to be cautious about changing too many settings at once.

Key Insight: Diverged branches aren't a single problem with a single fix—recognizing which of three situations you're in (keep both, overwrite remote, or overwrite local) determines the right resolution strategy.

The post explains what it means for local and remote git branches to diverge, how to recognize the situation, and what to do about it. Evans walks through the four possible states of local vs remote branches and focuses on the diverged case. She shows three ways to detect divergence (git status, git push errors, git pull errors) and notes that git's error messages can be confusing for newcomers. She then presents three main resolution strategies: rebasing to keep both sets of changes, force-pushing to overwrite remote changes, or hard resetting to overwrite local changes. The key message is that there's no single best solution—it depends on your workflow and why the divergence happened.

Key Insight: The .git directory is mostly just small text files containing commit IDs, and understanding this structure is useful but insufficient — git's complexity lives in the operations and workflows built on top of it.

Julia Evans provides a guided tour of the .git directory, explaining each component with concrete examples. She covers the core data model (HEAD, branches, commits, trees, blobs) and supporting structures (reflog, remote-tracking branches, tags, stash, config, hooks, staging area). Each section shows the actual file contents using cat or git cat-file commands, demystifying what are often treated as opaque internals. She emphasizes that while understanding .git internals is valuable, it doesn't explain everything about git — merges, rebases, collaboration workflows, and conflict resolution require separate understanding.

Key Insight: Wrong mental models about Git can be more useful than correct ones for everyday work, and the near-even split between people who think of commits as diffs versus snapshots suggests there's no single 'right' way to conceptualize them.

Evans explores how people mentally model Git commits, finding through a Mastodon poll that opinions are nearly evenly split between thinking of commits as diffs (51%) versus snapshots (42%). She explains that Git internally stores commits as snapshots but uses packfiles for compression, making both mental models partially correct. She argues that 'wrong' mental models (like thinking commits are diffs) can be extremely useful for everyday Git usage. The post examines the practical advantages of each perspective and why the common advice to 'just learn Git internals' often fails as a teaching strategy.

Key Insight: NixOS offers more reliable server configuration than Ansible by controlling the entire OS state rather than just the parts you remember to manage, but the learning curve and debugging experience remain genuinely painful.

Julia Evans documents her experience setting up NixOS on a personal server to replace her chaotic Ansible-managed setup. She walks through the process step by step, from installing NixOS via nixos-infect on Hetzner, to creating a flake, deploying configuration, and setting up a Go service with Caddy. She finds NixOS more reliable than Ansible because it controls the entire OS rather than just parts of it, but acknowledges the Nix language syntax and debugging experience remain confusing. The post is a practical guide mixed with honest reflections on the tradeoffs of using Nix.

Key Insight: Creating clear educational material is far harder than building the underlying implementations, because understanding why something is hard requires losing your own expertise bias.

Julia Evans reflects on her fourth year running Wizard Zines full-time, covering a productive 2023 that included publishing a zine on integers and floats, creating interactive playgrounds, releasing 'Implement DNS in a Weekend,' giving a Strange Loop keynote, and writing extensively about Git. She hired an operations manager to handle business logistics, migrated her social presence to Mastodon, and started research for a new Git zine. The year's recurring theme is that deep educational projects take longer than expected, and she's learning to accept that pace.

Key Insight: Git commits are conceptually just folders containing snapshots of your code, and mounting them as an actual filesystem makes this mental model tangible while providing a surprisingly practical way to explore repository history.

Julia Evans built git-commit-folders, a tool that mounts a git repository as a filesystem where every commit appears as a folder, branches appear as symlinks, and you can explore old code by navigating directories. She experimented with FUSE, NFS, and WebDav implementations on Mac OS, ultimately focusing on NFS since FUSE requires kernel extensions that Mac is making harder to install. The project aims to make the concept that 'every git commit is like a folder' feel concrete and intuitive. Along the way she encountered numerous practical problems including handling repositories with millions of commits, stale NFS file handles, inode numbering, and mysterious error messages. She found the tool genuinely useful for tasks like searching for deleted functions and quickly viewing files on other branches without setting up worktrees.

Key Insight: People's intuitive model of git branches as offshoots from a parent branch is actually very close to how merges, rebases, and pull requests work in practice—git just refuses to track that relationship, forcing users to re-specify it every time.

Evans explores the gap between how people intuitively think about git branches (as offshoots from a parent branch) and how git actually implements them (as pointers to commits with full history). She argues that the intuitive model is actually quite useful because it closely matches how rebases, merges, and pull requests work in practice. The key disconnect is that git has no concept of branch hierarchy or parent-child relationships between branches, forcing users to specify these relationships explicitly every time. She also highlights confusing UI choices like the different meanings of '..' and '...' in git log versus git diff.

Key Insight: Nix flakes offer stronger reproducibility than alternatives but come with significant papercuts and confusing error messages, making a deliberately simple configuration that you actually understand more valuable than a sophisticated one you don't.

Julia Evans documents her experience migrating from basic nix-env usage to nix flakes after 9 months of using nix. She walks through creating a flake that consolidates all her installed packages into a single configuration file, drawing an analogy between flakes and Docker containers to make the concept accessible. Along the way, she encounters six distinct problems including git tracking quirks, unfree packages, relative path bugs, and mysterious daemon errors. Despite the friction, she arrives at a simple setup using a flake.nix file and a 6-line bash script, deliberately avoiding more complex tools like home-manager in favor of a configuration she can actually understand.

Key Insight: Git cherry-pick, revert, and rebase all use 3-way merge rather than simple patch application, using the parent commit as a common ancestor to enable intelligent conflict resolution.

Julia Evans investigates how git cherry-pick actually works under the hood, discovering it's not simply applying a patch as she previously assumed. She demonstrates that naive patch application fails without merge conflict resolution, while cherry-pick uses a 3-way merge internally. The post explains 3-way merge by showing how two changed files plus their common ancestor allow git to intelligently combine changes. She maps this to cherry-pick (using the parent commit as base) and revert (same thing but with commits flipped), and coins the term '3-way patch' for this technique of specifying patches as before/after file pairs rather than diffs.

Key Insight: The widespread frustration with git rebase stems not from rebase being inherently bad, but from the accumulated hidden knowledge needed to use it safely—knowledge that experienced users have internalized but rarely share with newcomers.

Julia Evans investigates why many developers hate git rebase by crowdsourcing specific bad experiences from Mastodon. She catalogs fifteen distinct problems people encounter, from repeatedly fixing the same merge conflict to accidentally losing work through force pushes on shared branches. She distinguishes between two kinds of rebase—rebasing on an ancestor (safer) and rebasing onto a diverged branch (riskier)—and identifies three levels of 'commit discipline' that explain why people argue about rebase. She concludes that while she'll keep using rebase, the number of pitfalls makes her more cautious about recommending it to newcomers without proper safety guidelines.

Key Insight: Git's terminology is confusing not because individual terms are hard, but because the same words are reused for different concepts and different words are used for the same concept, creating a web of ambiguity that even experienced users stop noticing.

Julia Evans catalogs the most confusing terminology in git, drawing from responses to a Mastodon thread asking what git jargon people find bewildering. She works through dozens of terms including HEAD vs heads, detached HEAD state, ours/theirs during merges and rebases, the difference between .. and ..., and the overloaded meanings of 'track' and 'index/staged/cached'. The post reveals that virtually every core git concept has some terminological confusion associated with it, whether from overloaded words, inconsistent naming, or misleading implications. Evans acknowledges that after 15 years of using git, she had never noticed many of these ambiguities herself. She concludes by picking her top three most confusing terms and noting that writing the post gave her a better understanding of why people find git so bewildering.

Key Insight: Many confusing aspects of Git come from the same concept having multiple names or from default display behavior hiding what's actually happening under the hood.

Evans shares four lesser-known Git facts she discovered while working on explaining how Git works. She clarifies that 'index', 'staging area', and '--cached' all refer to the same .git/index file. She explains that git stash works by creating actual commits labeled with a stash reference, that not all Git references are branches or tags, and that merge commits appear empty in diffs because git show only displays conflicts, not the full changes.

Key Insight: Technical topics feel impossibly hard not because of personal inadequacy but because of systemic reasons like hidden internals, confusing tools, and accumulated trivia — and once you identify those reasons, you can build tools and share knowledge to fix them.

Julia Evans gave a keynote at the final Strange Loop conference about why seemingly basic technical topics like bash, HTTP, SQL, and DNS take years to learn. She argues that the difficulty isn't personal failure but stems from systemic reasons: hidden system internals, enormous underlying complexity, confusing tool output, and piles of obscure trivia. She presents concrete strategies for making hard things easier, including sharing useful tools like shellcheck, turning overwhelming lists into small curated ones, telling chronological stories of what computers do, and building tools that reveal hidden system behavior. The talk celebrates the collaborative nature of learning, identifying archetypes like the 'grumpy old-timer' and 'loud newbie' who each contribute to collective understanding.

Key Insight: Git stores every file version as a complete snapshot in content-addressed storage, and finding old versions requires traversing the same commit-tree-blob hierarchy that git log walks behind the scenes.

This post explores exactly where git stores your files inside the .git/objects directory, using short Python scripts to demonstrate the mechanics. Evans shows how git uses content-addressed storage, where each file is stored at a path derived from its SHA1 hash, compressed with zlib. She walks through the full process of tracing an old version of a file by navigating from commit to tree to blob objects. The post also explains packfiles, which git uses to compress many objects together, and demonstrates that git log must traverse every commit's tree to track a file's history. Evans ends by counting the historical file versions in her repository and noting git automatically ran gc when objects got too numerous.

Key Insight: Running a single-person Mastodon server gives you identity control and moderation autonomy, but Mastodon's distributed architecture means you'll see an incomplete view of conversations, profiles, and engagement — tradeoffs that stem directly from how federation works.

Julia Evans shares her experience running a single-person Mastodon server after migrating from mastodon.social. She explains the tradeoffs of self-hosting: you get identity control via your own domain, but suffer from poor reply visibility, incomplete profiles, and lack of discovery features due to how Mastodon's distributed architecture works. She covers practical details like managed hosting costs, account migration gotchas, the surprisingly good API, and quirks around DMs, search limitations, and defederation. Despite the rough edges, she finds Mastodon worth it for the quality of technical conversations.

Key Insight: Getting comfortable on the command line is less about memorizing commands and more about reducing the fear of breaking things, finding a compelling reason to be there, and making the environment feel personal.

Evans crowdsourced advice from people who recently overcame command line intimidation, organizing their responses into three categories: reducing risks, finding motivation, and discovering helpful resources. The risk reduction strategies focus on making destructive mistakes reversible through backups, git, safe-rm aliases, and avoiding wildcards. Motivation often comes from discovering a 'killer app' like ripgrep or fzf that makes the command line genuinely more powerful than GUI alternatives. Practical comfort boosters include fancy shell setups like fish or oh-my-zsh, cheat sheet tools like tldr, and the realization that you don't need to memorize everything upfront. The post emphasizes that there's no single path to comfort—different people find different entry points.

Key Insight: You can shape internet discussion toward useful, constructive feedback by writing about facts and experiences rather than opinions, asking genuine questions, and strategically avoiding flamebait topics.

Julia Evans shares her strategies for getting useful feedback from internet comments on her blog posts, rather than simply avoiding comments altogether. She advocates writing about facts and personal stories rather than opinions, which naturally steers discussion toward constructive territory. She describes tactics like asking technical questions she doesn't know the answer to, preempting common suggestions, and avoiding flamebait topics. She also discusses setting boundaries on social media, analyzing negative comments for useful insights, and refusing to engage in arguments.

Key Insight: A simple 'Hello World' program triggers a complex chain of OS operations — PATH resolution, forking, ELF parsing, dynamic linking — all of which you can observe yourself using Linux debugging tools like strace, readelf, and debugfs.

Evans traces the full chain of events that occur when running a simple 'Hello World' Python program on Linux, from shell parsing through PATH resolution, inode lookups, forking, execve, ELF parsing, dynamic linking, and finally the write system call. Rather than focusing on Python-specific behavior, she examines what happens for any dynamically linked executable. The post emphasizes hands-on exploration, showing readers how to use tools like strace, readelf, debugfs, dd, ldd, and ltrace to observe each step themselves. She digs surprisingly deep, reading raw inodes from disk and decoding ELF permission bits by hand, while honestly noting the parts she still doesn't fully understand.

Key Insight: DNS is hard to learn not because the concepts are complex, but because the system hides its internals, the tools have poor UX, and critical gotchas are only discovered through painful personal experience rather than being systematically documented.

Evans explores why DNS remains difficult to learn despite being a 35-year-old, relatively simple technology. She identifies several structural reasons: much of the system is hidden from users (resolver caches, library implementations, server conversations), the primary debugging tool dig has confusing output, DNS has weird gotchas that take years to discover (like negative caching), people interact with DNS too infrequently to build expertise, and it's scary to experiment with. She proposes solutions including better education about hidden systems, friendlier tooling, community-maintained gotcha collections, and tools like Mess With DNS that make safe experimentation possible.

Key Insight: Lima offers a refreshingly simple way to run Linux VMs on Mac, and sometimes a single chaotic global VM with sensible defaults beats carefully configured per-project environments.

Julia Evans introduces Lima, a tool for running Linux VMs on Mac, which she finds much simpler than Vagrant. She needed Linux VMs to run projects that depend on Linux namespaces and bubblewrap. Lima's setup is just three commands and provides sensible defaults like mounting your home directory read-only. She encounters some issues with read-write mounts and networking quirks but finds the tool useful enough. She also mentions OrbStack as an alternative with better networking, and explains why she prefers VMs over containers for her use case.

Key Insight: The best way to open source imperfect projects is to stop pretending you'll fix them and instead be upfront about their limitations.

Julia Evans announces she's open sourcing her nginx playground project, a small tool for testing nginx configurations she built in 2021. She explains why she hesitated to open source it before—she struggles with maintaining open source projects, feels bad about ignoring issues, and worried about messy code. Her new approach is to simply be honest that the project is unmaintained rather than pretending she'll improve it later. She documents the caveats, tests the build instructions, and releases it as-is.

Key Insight: Binary data becomes far less intimidating once you realize most binary file formats are just integers, and floating point can be explained clearly without heavy mathematical notation.

Julia Evans announces her new zine 'How Integers and Floats Work,' explaining her motivations for creating it. She wanted to demystify binary data by showing that most binary file formats are just integers, and to provide a clearer explanation of floating point without heavy math jargon. The zine comes with an interactive playground called Memory Spy that lets users inspect how variables are represented in memory. She highlights collaborators, beta readers, and tools like float.exposed that influenced the project.

Key Insight: The barriers people perceive to blogging — needing originality, expertise, correctness, and comprehensiveness — are mostly self-imposed myths, and the real value comes from sharing personal experiences and just publishing.

Julia Evans debunks eight common myths that discourage people from blogging about technical topics. She argues you don't need to be original, an expert, or perfectly correct to write valuable posts. The post draws from her own extensive blogging experience to show that personal stories, short posts, and 'I just learned this' content are genuinely valuable. She advocates writing for one specific person rather than trying to cover every audience, and notes that boring posts simply get ignored while interesting ones get boosted. She closes by acknowledging that blogging isn't for everyone, pushing back against the 'everyone should blog' pressure.

Key Insight: Making low-level memory representation visible and interactive through simple C programs helps demystify how computers actually store data, regardless of what programming language you use.

Julia Evans announces Memory Spy, a new interactive playground that lets users run C programs and visualize how variables are represented in memory. The tool is designed as a companion to her upcoming zine about integers and floats, making low-level memory representation accessible even to people who don't know C. The playground compiles code with clang, runs it through lldb, and returns byte-level variable representations. She discusses the design decisions, including a click-on-a-line navigation model instead of traditional step-through debugging, and her pragmatic approach to security for running untrusted code.

Key Insight: Building a toy implementation of a protocol from scratch, even in a high-level language, gives you a deep and lasting confidence in your understanding that reading documentation alone cannot provide.

Julia Evans announces a free guide for implementing a DNS resolver from scratch in Python over a weekend. The project is about 200 lines of code and teaches both binary protocol parsing and how DNS actually works behind the scenes. She chose Python for approachability and uses only standard library modules to show the implementation truly from scratch. The guide is built as Jupyter notebooks converted to a website, and includes bonus exercises to extend the toy resolver toward a more realistic implementation.

Key Insight: Deep technical understanding comes not from marathon study sessions but from a series of tiny, curiosity-driven deep dives — spying on systems, noticing confusion, reading specs, experimenting, and building your own terrible version — spread over years.

Julia Evans shares her keynote from RubyConf Mini about strategies for learning hard technical topics, using DNS as a running example. She describes her approach of 'tiny deep dives' — doing nothing for months, then intensely learning for a short burst before returning to dormancy. The talk walks through five concrete strategies: spying on systems with tools like dig and Wireshark, noticing when you're confused, reading specifications like RFCs, doing experiments with tools like Mess With DNS, and implementing your own terrible version of the thing. She demonstrates each strategy with DNS examples, including debugging a negative caching mystery and building a DNS client from scratch in 120 lines of Ruby. Her core message is that taking years to learn something is normal, and small focused investigations compound into deep understanding.

Key Insight: Interactive playgrounds that let people manipulate bits directly build better intuition about integer representation than any amount of abstract explanation.

Julia Evans announces integer.exposed, a new interactive playground for exploring how integers work at the bit level. Inspired by Bartosz Ciechanowski's float.exposed, the tool lets users visualize and manipulate integer representations including signed/unsigned values, bitwise operations, and endianness. The playground accompanies her upcoming zine about integers and floating point numbers. She walks through several exercises demonstrating signed integer representation, arithmetic vs logical right shift, binary counting, and byte order swapping. The tool is deliberately simple, built with Vue.js in just two files.

Key Insight: The breadth of available programming playgrounds — from regex testers to DNS tools to language REPLs — makes interactive experimentation accessible for nearly every technical concept, and curating them into a single reference is itself a valuable contribution.

Julia Evans compiles a comprehensive list of programming playgrounds — interactive browser-based tools for experimenting with code, data formats, and tools without any local setup. The list was crowdsourced from Mastodon and covers everything from compilers and shaders to DNS, SQL, regex, and dozens of programming languages. She organizes them into categories including general tools, data formats, and programming languages (both official and unofficial playgrounds). The post is primarily a curated reference resource rather than an argumentative piece, reflecting her enthusiasm for interactive learning tools.

Key Insight: Structuring feedback into predefined categories makes it both easier to give and easier to act on, turning an unscalable manual process into a systematic one.

Julia Evans describes the custom feedback site she built with Marie Flanagan to collect structured beta reader feedback on her zines. The original process of emailing PDFs and manually collating replies in Trello didn't scale beyond 10 reviewers. Inspired by the 'Help This Book' platform but needing image support, they built a site with five feedback categories: 'I learned something,' 'I love this,' 'I have a question,' 'I'm confused,' and 'I have a suggestion.' The categorized emoji system lets her quickly scan pages to see what's working and what needs rewriting, and she prioritizes feedback from learners over experts. The site has scaled feedback from 3-4 people to dozens and significantly improved zine quality.

Key Insight: The 8-bit byte emerged from a convergence of practical constraints — English text needs more than 6 bits, BCD needs multiples of 4, powers of 2 simplify hardware, and smaller bytes waste less space — making it a natural choice rather than pure accident.

The post investigates why x86 computers use 8-bit bytes, exploring whether it's a historical accident or an objectively good choice. Evans traces the history from IBM System/360's 1964 decision through the Intel 8008 to modern x86, examining multiple contributing factors. Key reasons include fitting the English alphabet in one byte (2^6=64 isn't enough), compatibility with binary-coded decimal (which needs multiples of 4 bits), the advantages of power-of-2 sizes for hardware design, and the principle that smaller addressable units waste less space. She concludes it's a natural convergence of practical constraints rather than pure accident.

Key Insight: Nix builds are ultimately just bash scripts receiving environment variables—the Nix language exists primarily to generate these variables and ensure all dependencies are explicitly declared.

Evans explores how Nix builds work under the hood by compiling a real C program (paperjam) from scratch, deliberately avoiding Nix's standard helpers. She starts with the core `derivation` function and an almost-empty build script, then iteratively fixes errors—missing PATH binaries, compiler setup, header files, linker flags, and code signing—to understand each layer. The post demystifies derivations as essentially environment variables passed to bash scripts, and shows how stdenv automates the tedious parts she did manually. She concludes with a clear three-step mental model: .nix files compile to .drv files (environment variables and inputs), which get passed to build scripts, with stdenv's bash code eval'd along the way.

Key Insight: You can use nix as a simple package installer without buying into its full ecosystem, and its real value lies in isolated builds with explicit dependencies — though the complexity cost may not be worth it for casual use.

Julia Evans shares her experience adopting the nix package manager on Mac as an alternative to Homebrew, deliberately keeping her usage as simple as possible. She explains how to install packages, build custom nix packages for a C++ program called paperjam, and install a 5-year-old version of Hugo. While she finds nix confusing and overly complex, she appreciates its ability to compile software with correct dependencies and install multiple library versions simultaneously. She explicitly avoids nix's more advanced features like flakes, NixOS, and home-manager. A later update reveals she eventually switched back to Homebrew, finding nix not worth the complexity.

Key Insight: For small web projects, the long-term ability to make quick edits without fighting build tool rot is more valuable than the marginal benefits a build system provides.

Evans explains why she avoids JavaScript build systems for her small web projects, despite acknowledging their usefulness for complex applications. She demonstrates the problem with a concrete example: trying to rebuild a 2-year-old Vue project that fails due to dependency rot and Node.js incompatibilities. Her core argument is that for small ~500-line projects, the long-term maintenance cost of a build system outweighs its marginal benefits. She advocates for loading libraries via script tags, vendoring dependencies, and keeping projects simple enough to edit years later on a new machine.

Key Insight: Self-publishing a physical zine involves a surprisingly long chain of manual, iterative steps — from Python scripts to margin measurements to print proofs — that mirrors the kind of debugging process the zine itself teaches.

Julia Evans announces that print copies of The Pocket Guide to Debugging have arrived at the warehouse and are shipping to customers. She shares photos of the finished product and provides a detailed 22-step breakdown of what was involved in producing the print version, from creating a printable PDF to dealing with binding margins to coordinating with a print company. The post reveals the surprisingly manual and iterative nature of self-publishing a physical zine, including agonizing over binding style, discovering margin problems only after receiving a proof, and painstakingly readjusting every page. She printed 1500 copies and had about 700 preorders.

Key Insight: The famous 0.1 + 0.2 result isn't just generic floating point imprecision — the exact sum lands precisely between two representable floats, and the IEEE 754 round-to-even tiebreaker rule picks 0.30000000000000004.

Evans digs into the exact mechanics of why 0.1 + 0.2 equals 0.30000000000000004 in floating point arithmetic, rather than settling for the vague explanation that 'floating point is inexact.' She walks through the calculation step by step, showing the exact decimal values of 0.1 and 0.2 as 64-bit floats, adding them together, and demonstrating that the result lands exactly between two representable floating point numbers. The tiebreaker rule (round to even significand) produces 0.30000000000000004 rather than 0.3. She also explores how floating point display works, noting that computers print the shortest decimal that rounds back to the correct float, and wonders whether showing exact values might actually be less confusing.

Key Insight: Integer bugs lurk in every programming language — not just C — and the same bit pattern can mean completely different things depending on signedness, byte order, and language semantics.

Julia Evans catalogs nine common problems developers encounter when working with integers across programming languages. She covers database primary key overflow, unsigned integer underflow in Go, two's complement representation quirks, Java's signed-only byte shifting, byte order confusion, modulo behavior differences across languages, C compilers optimizing away overflow checks due to undefined behavior, and the subtle && vs & typo bug. The post uses concrete code examples in Go, Java, C, and JavaScript to demonstrate that integer problems aren't limited to low-level C programming. She explains two's complement representation as foundational context for understanding why these bugs occur.

Key Insight: Floating point problems aren't random or unpredictable—they follow directly from how numbers are represented, and understanding the gaps between representable values explains most real-world bugs.

Julia Evans collects real-world examples of floating point bugs to make abstract warnings concrete. She presents eight specific cases: an odometer that stops counting, corrupted tweet IDs in JavaScript, a variance calculation that goes negative, cross-language calculation discrepancies, the Kerbal Space Program 'Deep Space Kraken,' inaccurate timestamps, page layout errors, and collision checking failures. Throughout, she emphasizes that floating point isn't 'bad' or random—it's standardized and predictable—but developers need to understand when gaps between representable numbers cause problems. She provides runnable code for most examples and suggests practical fixes like using doubles, integers, or better algorithms.

Key Insight: Transparent sharing of indie business metrics reveals that simple decisions like offering product bundles can dramatically reshape revenue, and that a niche technical education business can sustain itself globally with customers in over 120 countries.

Julia Evans shares detailed business metrics and graphs from her Wizard Zines business for 2022. She breaks down print vs digital sales (42%/58%), corporate vs personal revenue (16% corporate), purchasing power parity program usage across 81 countries, and the impact of bundling zines (65% of revenue from bundles, up from 3% in 2018). She also covers her migration from Gumroad to Shopify, revenue patterns tied to zine releases, and geographic distribution of sales across 126 countries. The post demonstrates a refreshingly transparent approach to sharing indie business data.

Key Insight: The gap in debugging education isn't a lack of high-level principles but a lack of specific, actionable strategies that developers can reach for when stuck in the middle of a frustrating bug.

Julia Evans announces her new zine, The Pocket Guide to Debugging, containing 47 strategies for solving tricky bugs. She wrote it because debugging is a huge part of programming yet rarely taught formally. The zine took 3.5 years to develop and includes choose-your-own-adventure debugging mysteries. She describes how the hardest part was turning high-level advice into specific, actionable strategies, working daily with a collaborator for 6 months to achieve this. She notes that even as an experienced debugger, she finds the table of contents useful as a reference when stuck on frustrating bugs.

Key Insight: Effective debugging requires resisting the urge to fix immediately and instead investing in understanding the root cause, which pays dividends as lasting knowledge.

Julia Evans presents a debugging manifesto developed for her debugging zine, outlining eight principles for approaching bugs with the right attitude. The manifesto emphasizes understanding bugs before fixing them, maintaining confidence when stuck, being open to surprising root causes while still suspecting your own code first, and collaborating with others. She frames debugging not as a chore but as an investment in learning, arguing that bugs often reveal gaps in mental models that are worth exploring.

Key Insight: Effective log analysis in distributed systems requires disciplined techniques — searching by request ID, verifying errors are actually new, and finding the first error in a cascade — rather than staring at dashboards hoping patterns emerge.

Julia Evans shares practical tips for analyzing logs in distributed systems debugging, drawn from community suggestions on Mastodon. The post covers techniques ranging from searching by request ID and correlating across services to building timelines and reformatting logs into tables. She emphasizes the importance of verifying that suspicious errors are actually new, finding the first error in a cascade, and being aware of time synchronization issues. The tips span both command-line tools and dedicated log analysis platforms, with a brief section on generating better logs in the first place.

Key Insight: Taking the time to reproduce and deeply understand compiler errors—rather than just fixing them—reveals how language features like borrowing and lazy evaluation interact in non-obvious ways.

Julia Evans shares her experience learning Rust through Advent of Code, focusing on understanding compiler errors rather than just fixing them. She walks through two specific errors: a borrowing issue caused by temporary values being freed too early, and a confusing trait implementation error with Sum. For each error, she creates minimal reproductions and works through the underlying language concepts. She finds that variables are semantically meaningful in Rust (affecting when memory is freed) and that lazy iterator evaluation creates non-obvious dependencies on borrowed data. The post demonstrates her methodical approach to learning: fix, reproduce, explain, and ask for help.

Key Insight: DNS query generation and parsing, which sounds intimidating, can be implemented from scratch in 120 lines of Ruby by starting with a captured packet from Wireshark and progressively replacing hardcoded bytes with proper encoding logic.

Julia Evans walks through building a minimal DNS query tool in Ruby from scratch, demonstrating how to generate DNS queries and parse DNS responses in about 120 lines of code. She starts by capturing a real DNS query from Wireshark, then progressively replaces hardcoded bytes with proper encoding functions. The post covers the DNS packet format including headers, domain name encoding, and the tricky DNS compression scheme. She builds a complete working clone of `dig` that can resolve domain names to IP addresses, explaining each step of the binary protocol along the way.

Key Insight: The trailing dot in domain names exists because both zone files and search domains can silently modify unqualified names, so the dot explicitly signals that the domain is complete and nothing should be appended.

The post explains why domain names sometimes end with a trailing dot, tracing the answer through DNS internals, zone files, dig output, and search domains. Evans starts by noting that the simple answer ('it means fully qualified') doesn't explain why it matters. She shows that DNS wire format doesn't use dots at all, then explains how zone files use the trailing dot to distinguish absolute from relative names. The key practical insight is that search domains (used in local networks and Kubernetes) can silently append suffixes to domain names, so the trailing dot explicitly signals 'nothing else gets added.' She concludes with practical advice: use trailing dots when configuring DNS, but not in browsers where it often breaks.

Key Insight: Tun/tap virtual network devices let you bypass the kernel's TCP stack and send raw packets from userspace programs, making it possible to experiment with your own network protocol implementations.

Julia Evans explains how to send raw network packets in Python using tun/tap virtual network devices. The core problem is that if you try to implement TCP yourself on a normal network interface, the Linux kernel will interfere by closing connections it didn't initiate. Tun/tap solves this by creating a virtual network device that your program controls directly. The post walks through creating a tun interface, setting up NAT with iptables, and writing a minimal Python program that sends a hardcoded SYN packet to example.com and receives a SYN ACK reply. She also demonstrates using tcpdump to observe the packets on both the virtual and real network interfaces.

Key Insight: Debugging expertise comes from five distinct types of knowledge — codebase, system, tools, strategies, and experience — and recognizing which you're missing helps you improve more deliberately.

Evans shares a framework for categorizing debugging skills, drawn from academic research on troubleshooting. She reframes five categories from a paper into actionable ways to improve: learn the codebase, learn the broader system, learn your tools, learn strategies, and get experience. Each category represents a different kind of knowledge needed for effective debugging, from specific codebase familiarity to general heuristics. She notes that experience is perhaps the most straightforward to acquire since programmers encounter bugs constantly throughout their careers.

Key Insight: Building a toy remote login server reveals why pseudoterminals exist: the Linux kernel needs a real terminal device to handle signal routing, process groups, and terminal-aware programs, which raw TCP sockets simply cannot provide.

Julia Evans builds a toy remote login server (like a tiny SSH without security) to explain how pseudoterminals work in Linux. She walks through the server implementation in Go, showing how to create a pseudoterminal, connect it to bash, and ferry data between a TCP connection and the terminal. Along the way, she demonstrates why you can't just use a raw TCP socket as bash's stdin/stdout (Ctrl+C breaks, top doesn't work), and dives into Linux sessions and process groups to explain how the kernel routes signals like SIGINT. The post includes a working client implementation and a public demo running tetris.

Key Insight: Terminal communication is fundamentally just bytes sent back and forth — keystrokes encoded as simple byte values and display instructions as escape sequences — a protocol largely unchanged since the 1970s VT100.

Evans explores what actually happens at the byte level when you press keys in a terminal, using a browser-based terminal (xterm.js with goterm) to log bytes sent and received over a websocket. She demonstrates that the client is essentially 'dumb' — it sends raw keystrokes and relies on the server to echo them back. She maps out how Ctrl combinations, special keys, and escape sequences are encoded as bytes, revealing that Alt is literally the same as Escape, and Ctrl+letter simply maps to that letter's position in the alphabet. The post explains how ANSI escape sequences control terminal display, originating from the VT100 era. She acknowledges the encoding scheme is weird by modern standards but exists for decades of backwards compatibility.

Key Insight: Monitoring tiny web services doesn't require sophisticated infrastructure — a cheap uptime checker, an end-to-end healthcheck, and automatic restarts can achieve 99.95% uptime with near-zero operational effort.

Julia Evans describes her approach to monitoring small, low-stakes web services she runs as side projects. She explains how she went from no monitoring (and embarrassing downtime discoveries) to a simple three-step system: uptime checkers that email on failure, end-to-end healthchecks that verify the service actually works, and automatic restarts when healthchecks fail. The key insight is that monitoring doesn't have to be complex or expensive — her entire setup costs about $5/year and achieves 99.95% uptime. She emphasizes that this approach is specifically for tiny services where occasional downtime is acceptable.

Key Insight: Bubblewrap can start containers 35x faster than Docker by operating at a lower level, but this exposes Linux namespace complexities that Docker deliberately abstracts away.

Julia Evans explores bubblewrap as a fast alternative to Docker for running containers, motivated by her need for sub-5ms container startup times for an nginx playground project. She benchmarks Docker/podman at ~300ms startup versus bubblewrap at ~8ms, explaining that bubblewrap achieves this by being a much lower-level tool that skips features like overlay filesystems and networking bridges. The post documents several surprising pitfalls she encountered: processes appearing in ps but being unkillable due to mismatched PID namespaces, capabilities issues with privileged ports, and UID mapping problems that break tools like apt. Despite these sharp edges, she concludes that bubblewrap works well for her constrained use cases and is worth the tradeoffs.

Key Insight: sqlite-utils removes the friction of getting data into SQLite by auto-generating schemas and handling real-world import problems like evolving fields and duplicates, making ad-hoc data analysis much more accessible.

Julia Evans describes sqlite-utils, a tool that makes importing data into SQLite easy for ad-hoc data analysis. She walks through a practical example of importing Shopify order data, showing how the tool auto-generates schemas from JSON, handles schema changes with alter, and deduplicates with upsert. She also covers the command line interface and pairs it with Metabase for visualization. The post highlights how this setup replaced a more complex Flask/plotly/pandas workflow.

Key Insight: Even seemingly simple technical topics like DNS can turn out to be far more complex than expected once you try to explain them thoroughly.

Julia Evans shares draft pages that were cut from her 'How DNS Works' zine. She originally planned a broader zine covering everything needed to own a domain, including domain registration, TLS, and email, but discovered DNS alone required all 28 pages. Rather than let the drafts sit unused, she published them as bonus content on her blog, covering domain registration, email basics, and TLS fundamentals. She acknowledges the pages aren't as polished or cohesive as her usual zine work.

Key Insight: DNS appears magical and incomprehensible until you learn how it actually works, at which point even the weirdest problems become totally understandable.

Julia Evans announces her new zine about DNS, explaining why she chose the topic: DNS is ubiquitous, has an elegant decentralized design, and is frustratingly confusing without proper understanding. She highlights companion tools she built including Mess With DNS (a playground for safe DNS experimentation) and a simple DNS lookup tool. The zine was a collaborative effort with Marie Claire LeBlanc Flanagan over four months, featuring an innovative custom feedback system that collected ~1200 feedback items from ~60 beta readers. DNS security topics like DoH, DoT, and DNSSEC were deliberately excluded because the landscape is still evolving.

Key Insight: The Unix command line is experiencing a renaissance of modern replacements and novel tools that improve on decades-old defaults, and crowd-sourcing recommendations is an effective way to discover them.

Julia Evans compiled a crowd-sourced list of modern command line tools after asking on Twitter about newer alternatives to standard Unix utilities. The list covers direct replacements for classic tools like grep, find, ls, and du, as well as entirely new inventions that solve problems the original Unix toolkit didn't address. She organizes them into categories: replacements for standard tools, new inventions, JSON/YAML/CSV tools, grep variants, and lesser-known older utilities. The post serves as a practical reference for anyone looking to modernize their terminal workflow, and she closes by sharing her personal favorites and the tools she's most excited to try.

Key Insight: Implementing a deliberately simplified toy version of a real protocol like TLS 1.3 reveals surprising complexity in key derivation and encryption layering that you'd never appreciate from reading documentation alone.

Julia Evans documents her experience implementing a toy version of TLS 1.3 in Go, motivated by her habit of learning networking protocols by building working implementations. She dramatically simplifies the task by only supporting one cipher suite, skipping certificate verification, and targeting a single website (her own blog). Using the invaluable resource tls13.ulfheim.net, she walks through each step: the client/server hello exchange, elliptic curve Diffie-Hellman key exchange, the surprisingly complex key derivation process, authenticated encryption with AES-GCM, and finally downloading her blog's homepage. The project reinforced her belief that you should never roll your own crypto while giving her concrete understanding of TLS internals.

Key Insight: Tracking and celebrating small, naturally-occurring learning milestones is more motivating and realistic than setting big goals that may never match your actual needs.

Julia Evans argues against traditional goal-setting for learning programming, finding it stressful and unhelpful when goals go unmet. She proposes instead celebrating 'tiny milestones' — small learning achievements that happen naturally through projects and debugging. Using her own experience of setting C/C++ learning goals nine years ago and achieving none of them, she shows how she still made enormous progress in systems programming through a different path. The post reframes 'failed' goals as simply redirected learning, and encourages tracking and celebrating small wins to stay motivated.

Key Insight: Browsers aren't magic — any request your browser makes is just HTTP that anyone can replicate, which is both a useful hack for personal scripts and a critical security lesson for API developers.

Evans walks through a practical step-by-step process for using undocumented web APIs by intercepting browser requests with developer tools. She demonstrates copying a request as cURL from the network tab, stripping unnecessary headers through trial and error, and translating the minimal request into a Python script. She discusses practical limitations like expiring session cookies and the importance of being respectful with request volume. The post concludes with the critical security insight that this technique applies to your own APIs too — anyone can inspect and manipulate the requests your frontend makes.

Key Insight: The most rewarding personal programming projects are low-stakes, completable in a single session, and solve some small real problem in your life — even if the solution isn't strictly necessary.

Julia Evans shares a collection of small personal programming projects she's written to solve minor problems in her life, from scraping festival calendars to generating envelope labels. She frames these tiny projects not as practical problem-solving (many could have been solved faster without code) but as a joyful way to spend an evening or weekend programming. The projects range from API polling for vaccine appointments to reverse engineering file formats, all united by being low-stakes, fun, and personally useful. She recommends this approach especially for people learning to program, emphasizing that the best part is the combination of creative fun and tangible real-world output.

Key Insight: The widely-used getaddrinfo function behaves differently across platforms and implementations, which is the hidden cause of many confusing DNS behaviors developers encounter.

The post explores the surprising behaviors of getaddrinfo, the libc function responsible for most DNS lookups. Evans explains that not all programs use it (dig and Go have their own resolvers), that its implementation differs significantly across platforms (glibc, musl, Mac OS), and that Mac's version doesn't even use /etc/resolv.conf, instead routing through mDNSResponder. These differences explain common DNS mysteries like why dig doesn't check /etc/hosts, why switching Linux distributions can break DNS, and why Macs have DNS caching but Linux doesn't always.

Key Insight: Many programming tasks that once required deep expertise have become dramatically easier thanks to modern tools and services, making it worth periodically revisiting things you previously considered too hard.

Evans shares a crowdsourced list of programming tasks that used to be difficult but have become significantly easier thanks to modern tools and services. The list spans infrastructure (Let's Encrypt for SSL, Terraform for cloud), languages (Go for concurrency, Rust for memory safety), hosted services (GitHub Actions for CI/CD, Firebase for realtime apps), and developer experience improvements (Docker for dev environments, LSP for IDE tooling). The post is intentionally brief and list-focused, serving as a catalog of progress rather than deep analysis. Evans frames it as inspiration for things worth learning now that the barrier to entry has dropped.

Key Insight: DNS terminology is genuinely ambiguous because the most commonly used terms ('nameserver' and 'DNS resolver') each have multiple meanings, and the precise alternatives are rarely used in practice, so learners need to rely on context clues to disambiguate.

Evans examines how the common DNS terms 'nameserver' and 'DNS resolver' each have multiple meanings depending on context, which creates confusion for people learning DNS. 'Nameserver' can mean either an authoritative nameserver (storing domain records) or a recursive nameserver (caching and resolving queries). 'DNS resolver' can mean a stub resolver library like getaddrinfo, a stub resolver server like dnsmasq, or a full recursive nameserver like 8.8.8.8. She provides real-world examples for each meaning and a Google results survey showing the ambiguous generic terms vastly outnumber the precise ones. Rather than inventing clearer terminology, she argues it's better to explain the existing messy terminology people actually encounter.

Key Insight: The entire DNS resolution algorithm is simple enough to fit in 20 lines of code — start at a root nameserver and iteratively follow referrals through the Answer, Additional, and Authority sections until you get your answer.

Julia Evans demonstrates how DNS resolution works by building a toy DNS resolver in 80 lines of Go (with the core function being just 20 lines). She explains the four sections of a DNS response (Question, Answer, Authority, Additional) and shows how a resolver iteratively queries nameservers starting from root servers, following referrals through glue records and authority records until it finds an answer. She compares her toy resolver's behavior to the real resolver 'unbound' using tcpdump, revealing that production resolvers make additional queries to cache nameserver addresses. She also wrote a bash version that, despite being janky, actually helped simplify the Go version's design.

Key Insight: IPv6 adoption is as much a public commons problem as a technical one — supporting it reduces strain on shared IPv4 infrastructure and improves the internet for users in regions with scarce IPv4 addresses.

Evans explores why web servers should support IPv6 despite major sites like Twitter and GitHub not doing so. She explains how ISPs without enough IPv4 addresses use CGNAT to share addresses among customers, which creates performance bottlenecks. The post catalogs reasons gathered from Twitter responses, ranging from reducing CGNAT load and improving performance to enabling IPv6-only servers and making home server networking easier. She also honestly notes that adding IPv6 support can sometimes cause problems when broken network equipment fails IPv6 connections. Evans frames IPv6 support partly as a public commons issue where broad adoption benefits everyone, especially users in regions with limited IPv4 access.

Key Insight: Self-hosting static sites on a simple nginx server can be faster, cheaper, and more transparent than managed services, with less maintenance overhead than expected.

Julia Evans describes her experience migrating all her static sites from Netlify to a self-hosted $5/month DigitalOcean droplet running nginx. She walks through her decisions: building a tiny Docker-based build system that clones clean Git repos, using rsync for deployment, Ansible for configuration management, and rewriting a small Lambda function in Go. The result is faster builds (7 seconds vs. a minute on Netlify), cheaper bandwidth (20x less than Netlify's overage pricing), and better visibility into 404 errors through nginx logs. She notes that the setup is simpler than expected and acknowledges she doesn't yet know what long-term maintenance will look like.

Key Insight: DNS failures in production are extraordinarily diverse and often masquerade as unrelated problems, making the 'it's always DNS' meme a reflection of genuine operational pain rather than just a joke.

Julia Evans catalogs the many ways DNS can break systems in practice, noting that textbook knowledge of DNS doesn't prepare you for real-world failures. She collected examples from Twitter of DNS problems that didn't initially appear to be DNS problems. The post covers issues ranging from slow resolvers and timeouts to Kubernetes ndots configuration, negative caching, nginx and Java caching DNS forever, musl's lack of TCP DNS support, and round-robin DNS breaking with getaddrinfo. The key theme is that DNS failures are diverse, often subtle, and frequently masquerade as other problems.

Key Insight: The only guaranteed way to find a domain's authoritative nameserver is to walk the DNS hierarchy from the root, because this uses the exact same information source that real DNS resolvers use.

This post explains how to reliably find a domain's authoritative nameserver, a key debugging step when DNS updates don't seem to be working. Evans walks through a two-step process: querying a root nameserver, then querying the TLD nameserver to get the definitive answer. She explains why this method is guaranteed to be correct—it mirrors exactly what DNS resolvers do behind the scenes. The post also evaluates several alternative methods (dig ns, dig soa, whois, dig +trace) and explains why each can give incorrect results in edge cases.

Key Insight: DNS's decentralized complexity isn't just an obstacle to understanding — it's what gives you the power to control security, privacy, performance, and custom behavior for your domains and networks.

Julia Evans explores the reasons why someone might want to run their own DNS server, covering both authoritative nameservers and resolvers. She first establishes that DNS's decentralized nature gives users control over how their DNS works, then catalogs specific motivations gathered from Twitter responses. For authoritative nameservers, reasons include security, familiar tooling, custom behavior, cost savings, and resilience against shared-infrastructure attacks. For resolvers, reasons include privacy, blocking malicious sites, resolving internal domains, and avoiding ISP manipulation. The post frames DNS's complexity as a feature rather than a bug, arguing that understanding the 'why' behind the system makes learning it more motivating.

Key Insight: Understanding the simple foundations of how computers work gives you the confidence to handle any complex new technology, and the best way to teach that is through hands-on experimentation rather than reading.

Julia Evans reflects on her second full year working independently on Wizard Zines, articulating a mission focused on explaining established technologies and helping people build foundational confidence with computers. She shifted energy toward building educational websites rather than finishing zines, creating several DNS-related tools and playgrounds. She found success collaborating with Marie on Mess With DNS and opened a print zine store. Despite releasing more free content than paid products, the business remains sustainable through zine sales. She also began relaxing her rule against writing anything negative, finding that slightly more critical posts turned out better.

Key Insight: The best way to make a complex system like DNS learnable is to give people a safe, real environment to break things in, with visibility into what's happening behind the scenes.

Julia Evans announces Mess With DNS, an interactive educational tool that gives users a real subdomain and DNS server to experiment with. The tool addresses three barriers to learning DNS: lack of a safe domain to test with, invisible behind-the-scenes queries, and not knowing what experiments to try. She details the design process, including user testing that significantly improved the UI, and the backend implementation in Go with a custom authoritative DNS server, websocket-based live streaming, and pragmatic security decisions. The post is a thorough technical walkthrough that doubles as a reflection on building educational tools.

Key Insight: DNS 'propagation' is actually cache expiration — DNS is a pull-based system where resolvers cache records, and understanding this distinction lets you predict and minimize update delays.

The post argues that the common term 'DNS propagation' is misleading because it implies DNS records are actively pushed out to resolvers, when in reality DNS is a pull-based system where resolvers cache records and you're simply waiting for those caches to expire. Evans demonstrates that new DNS records are actually available instantly, and the delays people experience come from cached records at resolvers, browsers, and operating systems. She walks through four specific reasons DNS updates can be slow: resolvers ignoring TTLs, negative DNS caching, browser/OS caching, and long-lived nameserver record caches. She concludes that while the terminology is likely here to stay, recognizing its misleading nature can help people build more accurate mental models.

Key Insight: dig's verbose default output becomes manageable with a few formatting flags and a .digrc file, and that same verbosity is actually dig's advantage over simpler DNS tools when you need full response details.

Julia Evans provides a practical guide to using the dig command-line tool for DNS queries. She breaks down dig's arguments into two categories: query options (name, type, server) and formatting options. She shows how to tame dig's verbose default output using +noall +answer, +short, and a .digrc configuration file. She also covers reverse DNS lookups with -x and the +trace option for understanding DNS resolution. The post concludes by explaining why she prefers dig over simpler alternatives like dog and host: dig shows all sections of DNS responses when needed.

Key Insight: A 'file not found' error when executing a binary that clearly exists usually means the binary's ELF dynamic linker interpreter is missing from the system, not the binary itself.

Julia Evans debugs a confusing 'file not found' error when running a Go binary inside an Alpine Docker container, even though the binary clearly exists on disk. She traces the problem through strace, readelf, and Google searches to discover that the ELF binary's dynamic linker interpreter (/lib64/ld-linux-x86-64.so.2) doesn't exist in Alpine Linux. The root cause is that Go unexpectedly produced a dynamically linked binary due to cgo being enabled by default. She finds two fixes: using the golang:alpine build container, or setting CGO_ENABLED=0 to produce a statically linked binary.

Key Insight: You don't need to adopt a complex build pipeline you don't understand just to use npm imports — simpler tools like esbuild can bridge the gap between 'pretend it's 2005' and modern JavaScript packaging.

Julia Evans describes her journey from a simple 'pretend it's 2005' frontend approach (plain HTML and script tags) to adopting esbuild as a build tool. She explains how she was frustrated by npm libraries that require imports and complex build pipelines she didn't understand. After discovering esbuild, she hit a problem where Vue's runtime-only build was loaded instead of the full build with template compiler. She traces through the issue using strace and learns about npm build artifacts, eventually fixing it by importing the specific Vue distribution file she needed.

Key Insight: Diagnosing DNS problems requires going beyond browser and application error messages to actively probe with dig, compare multiple DNS servers, and watch actual DNS traffic with tcpdump.

Julia Evans walks through practical tools and techniques for determining whether a server or networking problem is caused by DNS. She covers browser error messages (which are unhelpfully vague), programming language error messages, using dig to confirm DNS resolution, querying multiple DNS servers to catch caching issues, and using tcpdump to spy on DNS traffic. She illustrates each technique with real debugging stories, including a negative caching gotcha and an nginx failure caused by a DNS server bug.

Key Insight: Asking yes/no questions and stating your current understanding are the most effective tactics for getting useful answers, because they constrain the responder and require confidence that most question-asking advice actively undermines.

Julia Evans expands on her earlier 'how to ask good questions' post with practical tactics for getting useful answers when asking technical questions. She identifies three common failure modes: getting explanations of things you already know, irrelevant information, or jargon-filled responses. Her primary tactics are asking yes/no questions to constrain answers, stating your current understanding so others can correct it, and being willing to interrupt or redirect when explanations go off track. She argues that extracting information from people is a learnable skill that requires confidence, and that most 'how to ask questions' advice unhelpfully undermines the asker's confidence.

Key Insight: BGP may be a protocol you can't directly experiment with, but there are surprisingly accessible public tools that let anyone observe and understand internet routing, which is practically useful when upstream routing problems affect your network.

Motivated by the 2021 Facebook outage, Julia Evans explores tools that let anyone investigate BGP routing information, despite BGP being a protocol most people can't directly experiment with. She walks through BGP terminology (autonomous systems, routes, announcements) and demonstrates four practical tools: BGPView for looking up AS information, traceroute/mtr with ASN flags, looking glass services like PCH that let you query real routing tables, and BGPlay for visualizing historical BGP events. She uses the Facebook outage as a concrete example throughout, showing how routes disappeared and returned. The post concludes that understanding BGP tools is practically useful even for people who can't publish routes, because it helps diagnose network issues caused by upstream routing problems.

Key Insight: Working with local small businesses for printing and fulfillment trades slightly higher costs for dramatically better turnaround, flexibility, and fewer absurd logistical problems.

Julia Evans announces that all her technical zines are now available in print through a preorder system, with shipping in time for Christmas. She explains the logistics of the preorder model, pricing with discounts for PDF buyers and bulk orders, and free shipping tiers. The post highlights her decision to work with small, local businesses near Seattle for both printing and fulfillment rather than cheaper overseas options. She shares testimonials from happy customers about the print quality and explains how the arrangement with nearby small companies provides faster turnaround and more flexibility.

Key Insight: Building useful developer tools doesn't require complex architecture—a static frontend, a single API endpoint, and pragmatic security tradeoffs can produce something genuinely valuable for learning.

Julia Evans built an nginx playground website that lets users paste in nginx configurations and test them with curl commands, getting back either results or error logs. She walks through the technical architecture: a static Vue.js frontend on a CDN paired with a single-endpoint Go backend that runs each nginx config in an isolated network namespace. The security approach combines practical isolation measures (network namespaces, no database, separate frontend hosting) with an honest acknowledgment of remaining risks. She discusses performance tradeoffs of running on a tiny fly.io instance and design decisions borrowed from JSFiddle and CodePen. The project came together quickly because she kept the scope minimal, and she's considering building similar playgrounds for other tools.

Key Insight: The most effective technical teaching doesn't start from the beginning — it meets people where they are, identifies the specific gap in their existing knowledge, and fills in just that one piece.

Julia Evans articulates her teaching philosophy: instead of explaining topics from the ground up, she targets specific knowledge gaps that experienced practitioners already have. She argues that most programmers learn 'backwards' — building things first with abstractions, then filling in foundational knowledge later. This mirrors how people actually work: you set up a website before learning HTTP, learn Rails before understanding SQL. Her approach assumes readers already know a lot, identifies the one or two things they're missing, and explains just those gaps. She traces this style to her experience as a drop-in math tutor, where she had 20 minutes to diagnose what someone didn't know and fill in the minimum useful information.

Key Insight: Dropping into a REPL at a breakpoint combines the simplicity of print-statement debugging with the power of a full debugger, and its low cognitive overhead makes it a technique you'll actually use.

Evans advocates for debugging by dropping into a REPL at a breakpoint, showing how this works in Python (via ipdb), Ruby (via pry), and even partially in C (via gdb). She demonstrates the technique with concrete code examples, showing how you get access to all in-scope variables and functions at the breakpoint. She contrasts this with traditional debuggers, noting that REPL debugging is easier to remember because you just set a breakpoint and start running code interactively. The post positions this as a middle ground between print-statement debugging and full debugger usage.

Key Insight: Quadratic algorithms sneak up on you because they work fine on small inputs, but replacing a list scan with a hashmap lookup is often a simple fix that delivers dramatic speedups at scale.

Evans explains why quadratic-time algorithms are dangerously slow and how hashmaps can convert them to linear time. Using a concrete example of intersecting two lists, she shows that nested loops over 100,000 elements take nearly 3 minutes while a hashset-based approach handles 10 million elements in under 2 seconds. She demonstrates this with runnable Python code and actual timing results, making the performance difference visceral rather than theoretical. The post also notes that 'accidentally quadratic' code is a real-world problem that can sneak up on programmers because it works fine on small inputs.

Key Insight: The best programming examples come from grepping your own real code for patterns and then distilling those patterns down to minimal, self-contained demonstrations rather than inventing contrived scenarios from scratch.

Julia Evans explains her approach to writing good programming examples: start with real code you've actually written, then distill it down to something minimal and self-contained. She demonstrates this by showing how a contrived Python lambda example (squaring numbers) is less effective than one derived from real usage patterns (sorting events by time). She argues that realistic examples help 'sell' concepts by showing readers tasks they might actually encounter, and acknowledges that distilling real code into clean examples can take hours of work but is worth the effort when many people will read them.

Key Insight: Most 'impossible' bugs aren't genuinely complex — they feel impossible because of compounding factors like wrong assumptions, lack of system knowledge, and difficulty observing program state.

Julia Evans categorizes the reasons why bugs can feel 'impossible' to understand, based on about 400 Twitter responses. She identifies five main categories: the bug is hard to reproduce, you don't understand the overall system, it's hard to get data about the program's internal state, one of your assumptions is wrong, and the bug is genuinely complex. She emphasizes that bugs always have logical explanations but can feel impossible due to combinations of these factors. The post is groundwork for a debugging zine she's writing, and she finds it reassuring that many developers encounter the same types of 'impossible' bugs.

Key Insight: Building a quality physical product for technical education requires rejecting convenient shortcuts like print-on-demand in favor of dedicated partners who can match your quality standards.

Julia Evans announces the launch of a print zine store where readers can buy physical copies of her 'Bite Size' technical zines. She details her decision to work with a quality print company rather than print-on-demand services, and has partnered with a small fulfillment company for shipping. The store includes stickers, offers discounts for existing PDF buyers, and bundles PDF versions with print purchases. She started with a limited run of 350 copies each to avoid waste, with plans to expand the catalog and add bulk rates.

Key Insight: Your personal struggles are the best compass for finding blog topics that will genuinely help others, but the post should focus on what you learned, not the struggle itself.

Julia Evans argues that the best blogging strategy is to write about things you've personally struggled with, because your struggles likely mirror others' experiences. The key distinction is that posts should focus not on the struggle itself but on what you eventually learned from it. She explains that struggles naturally guide you toward specific, useful topics rather than generic overviews. The process can take years—sometimes you need distance before you can articulate the lesson. She also notes that teaching others who are struggling can reveal gaps in your own understanding, and that vulnerability on the internet can be managed by focusing on technical problems and being selective about what you share.

Key Insight: Actually examining raw memory with gdb transforms abstract concepts like the stack, heap, and buffer overflows into something concrete and understandable.

This post walks through using gdb to inspect the stack of a simple C program, showing exactly how variables are laid out in memory. Evans demonstrates how to find variable addresses, examine raw bytes, and understand the difference between stack-allocated and heap-allocated strings. The walkthrough deliberately uses the unsafe `gets` function to demonstrate a buffer overflow, showing how writing past a buffer's bounds overwrites adjacent memory. The post also covers how stack smashing protection detects overflows using canary values, and how /proc/PID/maps reveals where the stack and heap live in memory. Evans argues that hands-on exploration with gdb makes abstract concepts like the stack, heap, and pointers much more concrete.

Key Insight: The OSI model was designed for protocols that lost to TCP/IP, so learners should map layers 2, 3, 4, and 7 to their TCP/IP equivalents and safely ignore the rest.

Julia Evans explains why the OSI model is a poor fit for understanding TCP/IP networking, despite being widely referenced. She maps which OSI layers do correspond to TCP/IP (layers 2, 3, 4, and 7) and highlights that layers 5 and 6 have no clear equivalent. She traces the OSI model's origins to the Protocol Wars of the 70s and 80s, where OSI protocols lost to TCP/IP, explaining the mismatch. She recommends talking about specific protocols like Ethernet or HTTP instead of abstract layer numbers.

Key Insight: The real barrier to making content freely available is often fear of lost revenue, but operating with a spirit of abundance rather than scarcity can be both more generous and better for business.

Julia Evans announces that she has published all 273 of her programming comics in a searchable online archive at wizardzines.com/comics. Previously, the comics were scattered across Twitter posts, making them hard to find, link to, and update. She built a simple search using list.js with custom word-boundary matching. She explains her initial hesitation about potentially cannibalizing zine sales, but decided to embrace abundance over scarcity since her zine business is going well.

Key Insight: The best way to teach debugging is not through flowcharts or documentation but through interactive puzzles that show real tool output and let learners safely make mistakes, mirroring how debugging actually works.

Julia Evans describes her new project of building choose-your-own-adventure-style debugging puzzles using Twine, where players diagnose specific networking bugs using real tools like tcpdump, dig, and curl. She explains key design decisions: showing realistic command output rather than pre-interpreted results, letting players collect knowledge like an inventory, using free-form text boxes without penalizing wrong answers, and including red herrings to mirror real debugging. The puzzles reproduce actual bugs in VMs to ensure realistic output. Early feedback was overwhelmingly positive, with the main critique being that players wanted more tool options than were available. She frames the experience as simulating debugging alongside a knowledgeable coworker.

Key Insight: Strace's power comes from the fact that programs can document their behavior in many ways, but they can only interact with the operating system through system calls—making strace a universal debugging tool.

Evans crowdsourced 200 responses from Twitter about real-world strace use cases and categorized them into 11 problem types. The problems fall into three broad categories: finding files a program depends on, figuring out why a program is stuck or slow, and finding out why a program is failing. The most popular use case is simply finding where a program's config file lives. She highlights that strace's power comes from its simplicity—it just traces system calls—yet this is sufficient to debug an enormous range of problems from permissions errors to network failures to reverse engineering closed-source software. The post serves as both a practical reference and motivation for practice challenges she's building.

Key Insight: Building simpler, friendlier interfaces to existing low-level information — like matching DNS queries to responses automatically — can make debugging accessible to everyone instead of just power users.

Julia Evans introduces dnspeep, a 250-line Rust tool she built that shows DNS queries and responses from your computer in a human-readable format. The tool uses libpcap to capture packets and matches queries with their responses automatically, solving the main usability problem with tcpdump's DNS output. She explains the motivation: DNS seems mysterious to people who can't easily observe it, and existing tools like tcpdump require too much expertise to parse. She walks through several implementation challenges including libpcap compatibility issues, static linking difficulties, and the surprisingly hard problem of naming a DNS tool when every spy-related word is taken.

Key Insight: Most DNS tools expose the full complexity of the protocol when most people just need to see what records exist for a domain, so stripping away unnecessary details makes DNS far more approachable.

Julia Evans built a simple web tool for making DNS queries because existing tools like dig and Google's DNS lookup tool produce output that's overwhelming for most people. She wanted something approachable for a zine she's writing about owning a domain name. The tool presents DNS responses in a clean table format, strips out unnecessary technical details like flags and the 'IN' class, and includes a 'GET ME ALL THE RECORDS' button that queries every record type. The backend is a small Go service running on Netlify functions, with a Vue.js frontend. She also explains why ANY queries don't reliably work and why record types in responses aren't actually redundant.

Key Insight: Setting up CI always follows the same deceptively optimistic arc—it seems simple until you're deep in debugging, and the shared frustration is universal among developers.

Julia Evans describes her experience trying to migrate rbspy's CI from Travis to GitHub Actions, which turned into a frustrating debugging session. She also mentions an exciting pull request that adds C function identification to rbspy, a feature she'd wanted for years. The CI migration followed a familiar pattern: assuming it would be easy, then spending excessive time debugging. She found some hope in a tmate-based debugging action that allows SSH access to the CI environment.

Key Insight: Getting started with Rust and WebAssembly is more approachable than expected, requiring only standard npm tooling to compile and serve a working example.

Julia Evans documents a brief foray into WebAssembly during her Recurse Center batch. She paired with others to explore creating art in Rust compiled to WebAssembly and found the setup process surprisingly simple. The post notes the initially confusing use of npm to build a Rust program, and mentions compilation times on different machines.

Key Insight: Impressive VM density numbers in demos are often just a function of having more RAM — the per-VM memory cost stays roughly the same regardless of scale.

Julia tests how many Firecracker VMs she can run on a small 1GB DigitalOcean droplet and discovers the limit is about 4 VMs before the kernel starts OOM killing them. Each VM uses roughly 100MB of memory, which she considers reasonable given that the kernel alone is 30MB and VMs can't share memory. She reconciles this with Firecracker's impressive demo of 4000 VMs by noting it ran on a machine with 512GB of RAM — simple arithmetic shows the per-VM memory usage is consistent. She concludes by considering fly.io as an alternative.

Key Insight: Deploying real applications reveals surprising default behaviors in familiar tools—like nginx caching DNS forever—that you only discover by running into them.

Julia documents a day spent trying to deploy a puzzle application on fly.io, covering three practical discoveries along the way. She learns about IPv6 address notation and when square brackets are needed, discovers that nginx resolves DNS only once at boot unless you use a variable in proxy_pass, and finds that setting up Wireguard is surprisingly simple. The post is a candid learning log where not everything works perfectly—her nginx DNS fix still isn't functioning. It captures the messy reality of deploying applications and the incremental knowledge gained from wrestling with infrastructure.

Key Insight: Centralizing logs into a single system like journald eliminates the cognitive overhead of remembering where to look when debugging, and strace is a natural teaching tool for understanding program I/O.

Julia describes consolidating her server's scattered logging setup—Rails, a Go proxy, and a Firecracker manager were each logging to different places. She unified them through journald and Docker Compose configuration, then wrote a small bash helper to query logs easily. She also created two beginner strace puzzles for her teaching project, both centered on discovering where programs write their output.

Key Insight: Sometimes the pragmatic approach is to leverage existing tools and their internals directly rather than reimplementing them or using official but inadequate interfaces.

Julia documents a day of progress on her Firecracker VM puzzle platform, tackling three problems: speeding up kernel boot times, building puzzle tarballs using Docker's internal overlayfs, and loading those tarballs into VMs. She found that a kernel boot delay was caused by i8042 AUX port detection and fixed it with a boot option, though a second delay remained unsolved. She built a Python script that exploits Docker's overlay filesystem internals to create tarballs, and a Go function to extract them into VM images.

Key Insight: Optimizing the wrong layer first (init system) led to discovering the real bottleneck (kernel boot time), illustrating how performance debugging often requires peeling back assumptions one at a time.

Julia documents her attempts to make Firecracker VMs boot faster, initially planning to replace systemd with a lighter init system. Along the way she encounters loop device allocation bugs, learns about the devpts filesystem for pseudoterminals, and ultimately discovers she can keep systemd by disabling most services. The real revelation comes at the end: after optimizing userspace, she realizes the kernel itself is taking 1 second to boot — 7x slower than Firecracker's benchmark of 150ms — suggesting the kernel compilation is the actual bottleneck.

Key Insight: Editor features that improve code navigation (like go-to-definition) provide more value than ones that merely shortcut terminal commands you already know.

Julia describes a low-energy day where she translated a bash script for device mapper setup into Go. She also experimented with the vim-go plugin and found three useful features: auto-formatting, building, and go-to-definition. The go-to-definition feature stood out as genuinely useful compared to her usual approach of grepping through code.

Key Insight: Linux device mapper provides a surprisingly simple way to implement copy-on-write for VM images, avoiding expensive full copies by overlaying writes at the block level on top of a shared read-only base image.

Julia documents her exploration of Linux device mapper as a way to efficiently manage Firecracker VM filesystem images. Rather than copying base images every time a VM launches, she uses device mapper's copy-on-write snapshot feature to overlay writes on top of a read-only base image. She walks through a bash script implementation that uses losetup and dmsetup, following patterns from the Ignite project. The post is written as rough working notes, acknowledging it's not polished but meant as a personal reference.

Key Insight: When container networking works locally but fails on a server, the culprit is often a host firewall like ufw silently dropping packets rather than a fundamental networking misconfiguration.

Julia Evans documents debugging a networking issue where pinging the host IP from inside a container worked on her laptop but not on her server. After initially suspecting bridge configuration issues, she discovered through iptables verbose output that Ubuntu's ufw firewall was silently dropping packets. She disabled ufw entirely since the server already had DigitalOcean's external firewall. The post also briefly covers her experiments with Ignite for managing Firecracker VMs.

Key Insight: Firecracker is accessible enough for individual developers to use directly for personal projects, not just cloud providers, and its sub-second VM boot times enable interactive experiences that were previously impractical.

Julia Evans documents her experience learning to use Firecracker, AWS's open-source microVM technology, for a personal project: a CTF-like game where each player gets their own virtual machine. She explains why she chose VMs over containers (needing full root access for sysctls, iptables, perf), why Firecracker's sub-second boot time solves her problem compared to DigitalOcean VMs (60s) or qemu (20s), and walks through practical setup including hello-world scripts, Docker bridge networking, and building custom images. She shares her Go HTTP service for managing VMs and discusses running Firecracker with nested virtualization on DigitalOcean.

Key Insight: Docker's container image format serves as a surprisingly effective and simple build system for Firecracker VM filesystems, bypassing the complexity and slowness of cloud-init.

Evans documents her progress building VMs with Firecracker, pivoting from cloud-init to Docker for building VM images after cloud-init proved too slow. She converts Docker container filesystems to ext4 images that boot in Firecracker, which is both simpler and faster than her previous approach. She also figures out Docker Compose networking to put VMs and containers on the same bridge. After getting frustrated building her own VM service, she discovers fly.io, which does essentially the same Docker-to-Firecracker conversion she built, and gets VMs running there using their undocumented Go API.

Key Insight: Building Linux kernels from source is surprisingly fast and accessible, but assembling a working VM image from existing Ubuntu artifacts is harder than expected because cloud images and Docker images serve different purposes than a Firecracker VM.

Julia documents her struggles building Ubuntu VM images for Firecracker, learning about disk image manipulation, kernel compilation, and base image selection along the way. She discovered that compiling the Linux kernel is surprisingly fast, taking only about 5 minutes. She tried both Ubuntu cloud images and Docker images as bases for her VMs, finding problems with each approach. The cloud image wouldn't boot due to a UEFI-related issue, while the Docker image boots but lacks essential system components.

Key Insight: Common technical explanations like 'a bridge is like a switch' often fail because they omit crucial implementation details that would actually help people understand the concept.

Julia Evans documents her progress building an SSH game project using Firecracker VMs. She shares three accomplishments: understanding why Linux bridges are confusing (they're actually two things bundled together), fixing slow Ubuntu VM boot times caused by systemd entropy generation, and writing a Go HTTP server to manage Firecracker VM lifecycles. The post captures a day of hands-on systems work with honest accounts of confusion and pragmatic workarounds.

Key Insight: Container and VM networking relies on bridges that simulate physical network behavior using layer 2 devices, ARP, route tables, and NAT rules—and understanding these components is necessary when copy-pasting configurations inevitably fails.

Julia Evans documents her journey to understand what a network bridge is, motivated by needing to configure Firecracker VM networking. She explains that a bridge is a layer 2 ethernet device that routes packets to attached network interfaces based on MAC addresses, and that containers use ARP to discover MAC addresses just like physical networks. She walks through the specific steps needed to get internet access from inside a VM: creating a tap device, attaching it to a bridge, configuring route tables on both host and guest, and adding SNAT/iptables rules. Despite not fully understanding every detail, she successfully gets her Firecracker VM connected to the internet.

Key Insight: Debugging a physical discomfort problem follows the same systematic approach as debugging software: gather information, form hypotheses, collect evidence, and iterate when results are ambiguous.

Julia investigates whether her new ThinkPad laptop's screen is causing daily headaches, learning about pulse width modulation (PWM) as a backlight dimming technique that some people are sensitive to. She discovers the 'panel lottery' phenomenon where Lenovo ships different display panels in the same laptop model without telling buyers. She takes slow-motion videos of three different laptops to compare flickering behavior and identifies her panel model using Linux commands. Despite finding her panel has a 980 Hz PWM frequency (considered high enough that most sensitive people shouldn't notice), she remains unsure if PWM is the actual cause of her headaches.

Key Insight: Customizing web-based terminal tools often requires digging into their JavaScript internals, and the results may not perfectly match native terminal rendering due to subtle differences in how shells handle colors.

Julia documents her process of customizing the terminal colors in gotty, a web-based terminal tool, by finding and integrating HTerm themes from Blink Shell's open source theme collection. She modified gotty's JavaScript to inject Solarized theme preferences, though the result didn't perfectly match her local terminal. She also improved her deployment workflow by switching from building Docker images on the server to pulling pre-built images from GitHub's container registry.

Key Insight: Complex-looking problems often have simple solutions, especially when you pair with someone else and look at the actual code instead of assuming it will be hard.

Julia describes modifying gotty, a Go webserver that puts terminals on websites, to handle multiple websocket connections simultaneously instead of running separate processes per session. Her previous setup required coordinating three different pieces (multiple gotty processes, a proxy server, and a Rails server), which was fragile and flaky. By pairing with another Recurser, they forked gotty and wrote a simple HTTP handler that routes requests by session ID to the right command, replacing the complex multi-process architecture with a single Go program. The solution turned out to be much simpler than expected.

Key Insight: A simple, repeatable OS migration strategy—tar the home directory, reinstall fresh, manually curate packages—beats more sophisticated approaches by being reliable and understandable.

Julia Evans documents setting up a new ThinkPad T14s laptop, sharing her pragmatic approach to OS migrations: tar the entire home directory, reinstall Ubuntu from scratch, and manually curate packages. She also switched from stimulus.js to Vue.js for frontend work and describes ongoing issues with her Go proxy server setup for SSH connections to VMs.

Key Insight: Server-sent events offer a dramatically simpler alternative to websockets for server-to-client streaming, though real-world deployment behind proxies like nginx introduces subtle bugs from protocol version mismatches and middleware buffering.

Julia Evans discovers server-sent events (SSE) as a simpler alternative to websockets for streaming updates from a server to a client. She explains that SSE is just regular HTTP with a text/event-stream content type and a dead-simple wire protocol. While implementing SSE with Rails behind nginx, she hits two surprising bugs: nginx defaulting to HTTP/1.0 for upstream connections (which breaks long-lived connections), and Rack's ETag middleware buffering the entire response before sending. Both bugs were solved with help from Stack Overflow, reinforcing that weird computer interactions are often well-documented by others who hit the same problems.

Key Insight: Intermittent infrastructure failures often come from assuming one service being ready means all initialization is complete, when in reality you need explicit completion signals.

Julia describes debugging an unreliable VM launch process for her programming puzzles. The root cause was that SSH becoming available didn't mean cloud-init had finished running, so her setup scripts would sometimes fail due to missing dependencies. She fixed this by waiting for cloud-init's result.json file before proceeding. She also wrote a local testing script using qemu that made iteration much faster.

Key Insight: Sometimes the direct, lower-level tool is easier to use than the abstraction layers built on top of it, because the abstractions introduce their own complexity and failure modes.

Julia documents her day spent learning qemu to automatically test cloud-init.yaml files locally instead of spinning up DigitalOcean VMs. She first tried virt-install and Vagrant's experimental cloud-init support, both of which failed. She eventually succeeded by using qemu directly, finding it surprisingly simple once she bypassed the abstraction layers. She shares a complete, working script for starting an Ubuntu image with cloud-init and qemu, while noting several things she still doesn't understand about the underlying mechanics.

Key Insight: Separating exploratory, unpolished writing from a curated feed lets you share learning-in-progress without diluting your main body of work.

Julia Evans posts a brief note explaining that she has been writing daily blog posts about her time at the Recurse Center but has removed them from her main RSS feed. She feels the daily posts are more stream-of-consciousness than her typical polished writing. She provides links to find the posts in a dedicated category and a separate RSS feed for those interested in following along.

Key Insight: For small, frequently-edited datasets, storing model data in files rather than a database can dramatically simplify the development workflow while preserving the option to switch back later.

Evans describes two improvements she made to her puzzle game project during a Recurse Center batch. First, she deployed her Docker Compose setup to production on a DigitalOcean droplet, finding it easier than expected and more robust than her previous systemd-based approach. Second, she switched her Puzzle model from database-backed ActiveRecord to ActiveHash, which stores data in a file/hashmap instead of a database table, making it much easier to edit her small set of puzzles.

Key Insight: Pair programming is a superpower for debugging because sharing context with another person can collapse hours of frustration into minutes of productive problem-solving.

Julia recounts a day at the Recurse Center where she debugged a Rails login issue through pair programming and discovered several useful git tricks along the way. A missing database column was quickly tracked down using 'git log -S', which searches for commits that changed the count of a string. She also discovered the 'delta' tool for beautiful terminal diffs and learned about 'git log --patch'. The post also covers using Python's pathlib module and a working directory context manager for a cloud-init YAML generation script.

Key Insight: Docker Compose provides a surprisingly simple way to set up multi-service dev environments, with built-in DNS-based networking that makes container-to-container communication trivial.

Julia Evans shares her positive experience using Docker Compose to set up a local development environment for a Ruby on Rails project with multiple services. She walks through her docker-compose.yml configuration, explains how Docker Compose networking works (including digging into the internal DNS server and iptables rules), and covers practical tips like persisting database files and Rails console history. She contrasts the simplicity of Docker Compose with her previous failed attempt using Puppet and Vagrant, and briefly discusses its viability for small-scale production use.

Key Insight: The most effective learning tools reward discovery and experimentation rather than testing existing knowledge, and creative output scales surprisingly linearly with effort invested.

Julia Evans reflects on her 2020 accomplishments, which included publishing four zines (CSS, Bash, SQL, and Containers), experimenting with interactive learning tools, shipping physical zines, running !!Con remotely, and growing her business. Despite the pandemic, she found her revenue grew 2.3x, largely because she doubled her zine output. She experimented with companion websites and a questions-based learning tool, discovering that framing matters—rewarding learning rather than correct answers changed how people engaged. She also reflects on what worked well, including beta readers, collaboration, and adopting organizational tools like Trello and Focusmate.

Key Insight: Rails' built-in testing infrastructure and massive community documentation make automated testing surprisingly easy to adopt, even for someone who habitually forgets to write tests.

Julia Evans shares her experience discovering Rails testing tools while building an app to manage virtual machines for programming puzzles. She describes a common pattern of starting without tests, struggling with manual testing, then finally adding automated tests. She highlights several Rails testing features she found helpful: auto-generated test scaffolds, fixtures for test data, Devise authentication helpers, WebMock for HTTP request mocking, and easy integration tests. She appreciates Rails' mature ecosystem and extensive documentation.

Key Insight: When stuck on a library error, adding print statements directly to the library's source code is a simple but powerful way to understand what's actually happening.

Julia Evans shares slides from a 10-minute talk at Recurse Center about common blogging myths that discourage people from writing, including beliefs that you need to be original, expert, or always correct. She also describes a debugging technique of adding print statements directly to library code to understand confusing errors. The post is brief and practical, combining encouragement for aspiring bloggers with a concrete debugging tip.

Key Insight: A simple, hacked-together deployment using git hooks and systemd on a plain VM can be more fun and practical than wrestling with complex cloud platforms.

Julia Evans documents her experience setting up a Rails deployment on a Digital Ocean VM after getting frustrated with GCP's IAM complexity. She explores Digital Ocean's app platform and marketplace VM images, sets up push-to-deploy using git post-receive hooks, and learns about systemd's ExecStartPre for running pre-deployment scripts. She encounters a confusing journald logging issue that was fixed by restarting the service. The post captures the pragmatic joy of hacking together a simple deployment system rather than over-engineering it.

Key Insight: Following tutorials without understanding them works until you need to customize, at which point the bugs you encounter become the actual learning moments.

Julia Evans documents her experience setting up OAuth login in a Rails project using Devise and OmniAuth. She followed tutorials without fully understanding them, which mostly worked for GitHub login but broke when implementing a custom OAuth provider. She discovered two key bugs: misunderstanding that access_token.get('/me') makes an HTTP request rather than a hashmap lookup, and needing the X-FORWARDED-PROTO header set to https for correct redirect URIs. She ends by questioning whether Devise is even necessary.

Key Insight: Clustering quality from autoencoder embeddings is directly bounded by what the autoencoder actually learns to reconstruct, so if it only captures coarse features like size, the clusters will only reflect coarse features.

Evans documents her experiment using an RNN autoencoder to cluster hand-drawn faces from the Google Quick Draw dataset. She encodes each face (represented as a sequence of vectors) into a 50-dimensional vector, then decodes it back, training the network to minimize reconstruction error. The autoencoder learns to reproduce face outlines but fails at interior features. When she clusters the encoded vectors with DBSCAN, the clusters separate mostly by face size rather than facial features, which she notes logically follows from what the autoencoder learned to encode.

Key Insight: GCP IAM's resource-specific permission model is fundamentally different from AWS's unified approach, requiring different commands and Terraform resources for each type of resource you want to grant access to.

Julia Evans documents her frustrating experience trying to understand Google Cloud IAM after years of working with AWS IAM. She discovers that GCP's IAM terminology maps confusingly to AWS concepts — a GCP 'IAM role' is like an AWS 'permission,' while a GCP 'service account' is like an AWS 'IAM role.' She finds that granting permissions in GCP requires different approaches depending on the resource type, unlike AWS's unified model. After hitting a wall trying to grant SQL database access at 11:30 PM, she resorts to granting overly broad editor permissions just to get things working.

Key Insight: The real skill behind effective programming comics isn't drawing but teaching — identifying what people find genuinely surprising and distilling complex topics down to their most important facts.

Julia Evans breaks down her process for creating programming comics into three categories: surprising/hidden facts, lists of important things, and relatable stories. She argues that drawing skill is not the hard part — the real challenge is identifying what's genuinely surprising or useful to explain. She describes two sources for surprising facts: things she recently learned herself, and things she observes others finding surprising. She emphasizes that she only writes comics about topics she knows well, because distilling a topic to its 3 most important facts requires deep understanding, not surface familiarity.

Key Insight: Having millions of tutorials available doesn't substitute for clearly articulating your specific question and searching the right venue for the answer.

Evans shares her experience debugging a specific question about autoencoders during her Recurse Center batch. She was confused about whether the output or hidden vector from a PyTorch Encoder class represents the encoding. After struggling to find answers in the many deep learning blog posts available, she found clarity by searching PyTorch forums directly. The answer—that the hidden vector is the encoding and you discard the output—unblocked her work significantly.

Key Insight: When learning a new technique, simplifying your dataset dramatically is more productive than struggling with a complex one you don't yet understand.

Julia documents her attempt to use autoencoders to cluster neural network-generated face sketches. She wants to group Quick Draw faces into clusters so she can identify which groups she likes best and train models on those. She discovers autoencoders as a way to handle clustering sequences of vectors (where k-means won't work), learns the basic encoder-decoder pattern, but ends the day still confused about implementation details. She plans to try a simpler toy example next.

Key Insight: You can learn neural networks by implementing them from scratch, and even imperfect results that generate vaguely English-like nonsense are genuinely impressive when you understand the model learned everything from individual characters.

Julia Evans documents her attempt to implement char-rnn (character-level recurrent neural network) using PyTorch, following Karpathy's famous blog post on RNN effectiveness. She walks through the entire process: preparing training data from Hans Christian Andersen fairy tales, defining an LSTM model, writing training code, and generating text output. While her results don't match Karpathy's quality, she produces text that resembles English and shares the complete ~75 lines of code. She reflects on lessons learned, including that letting PyTorch's built-in LSTM handle backpropagation through time worked better than her manual implementation.

Key Insight: Debugging ML training failures is uniquely frustrating because the feedback loops are slow, the failure modes are unintuitive, and even basic diagnostics like gradient norms are hard to interpret without experience.

Julia Evans documents her struggles debugging why an RNN model's loss function climbs instead of converging, while experimenting with back-propagation through time (BPTT). She compares training one character at a time versus doing gradient descent periodically, finding that sequence length significantly affects whether the model trains successfully. The post captures the frustration of slow feedback loops in ML experimentation, where each hypothesis takes 30 minutes to test. She catalogs open questions about gradient clipping, exploding/vanishing gradients, and numerical stability, showing the messy reality of hands-on machine learning work.

Key Insight: Debugging machine learning training often comes down to mundane configuration issues like learning rate tuning and accidentally resetting optimizer state, not algorithmic breakthroughs.

Evans documents her experience debugging why her RNN model couldn't learn to generate Shakespeare-like text after successfully training it to count to three. She discovered two problems: the learning rate was set too high (0.01 instead of 0.002), and she was accidentally resetting the Adam optimizer's state in her training loop. After fixing both issues, the model progressed from generating random characters to producing vaguely English-looking text. Along the way, she provides a concise mathematical explanation of how deep learning training and learning rates work.

Key Insight: CSS isn't arbitrary or impossible — it's counterintuitive, and learning a small number of key fundamentals can transform years of frustrated guessing into confident, systematic problem-solving.

Julia Evans announces her new zine 'Hell Yes! CSS!' and shares her personal journey from finding CSS impossibly frustrating to developing real confidence with it. After watching a professional designer work with CSS efficiently and without guesswork, she realized CSS isn't arbitrary — it just has counterintuitive rules that can be learned. She uses the example of position: absolute (which positions relative to the closest positioned parent, not the page) as a case where one key fact unlocked understanding. The zine covers CSS fundamentals and emphasizes that attitude matters as much as knowledge. She also built an interactive CSS examples site as a companion learning resource.

Key Insight: When a complex problem isn't working, simplifying to a trivial case you fully understand is a powerful debugging and learning strategy.

Julia describes her progress learning RNNs at the Recurse Center by dramatically simplifying her task. After failing to train an RNN to generate Shakespeare-like English, she stepped back to train the same RNN on a trivially simple task: generating repeating sequences of numbers like 0 1 2. This worked, proving her RNN implementation was functional. She shares her PyTorch code and notes that longer sequences take longer to train.

Key Insight: Learning neural networks by building something broken and debugging it piece by piece — sharing failures openly — is more honest and instructive than pretending to understand everything from the start.

Julia Evans documents her early attempts to train a character-level RNN to generate text, following Karpathy's char-rnn approach. She describes the learning curve of working with tensors, debugging dimension mismatches, and discovering that PyTorch's cross entropy loss isn't what she expected. Her model currently produces gibberish instead of coherent text, and she shares the broken code openly. The post captures the messy, iterative reality of learning neural networks from scratch by just trying things and getting confused.

Key Insight: Hard programming projects become manageable when you start with minimal, working code and incrementally transform it, rather than trying to build from scratch or adapt broken examples.

Julia Evans describes her approach to tackling programming projects outside her comfort zone: find existing code that works, then incrementally modify it until it does what you want. She emphasizes that the starting code must actually run on your machine, and that the technology must support incremental changes. She contrasts her positive experience with Rails (easy to modify step by step) against neural network code (hard to modify without breaking). The post frames good starting points as magical accelerators for learning new technologies.

Key Insight: Pursuing a specific creative question — like why some generated faces are more interesting than others — is an effective way to bootstrap understanding of a complex ML model you know nothing about.

Julia Evans documents her exploration of sketch-rnn, a neural network trained on Google's Quickdraw dataset, to generate drawings of faces during generative art week at the Recurse Center. She walks through getting the model working, refactoring callback-heavy code to use async/await, and building a simple gallery website that draws faces in real time. While the generated faces aren't particularly appealing yet, she identifies the next challenge: teaching the model to distinguish between 'interesting' and 'boring' faces. The post captures her iterative, curiosity-driven approach to learning about RNNs through a concrete creative project.

Key Insight: The best simple explanations work because the writer believes the idea isn't fundamentally complicated, and presents it at exactly the level of complexity it deserves — with enough jargon for searchability and enough accuracy to be genuinely useful.

Evans explores the challenge of writing simple explanations of complex topics without coming across as condescending. She argues that good simple explanations should include some jargon to give readers search terms, strive for accuracy rather than oversimplification, and use stories and fun visual elements only when the underlying content is genuinely clear. She emphasizes having a specific audience in mind and notes that extreme simplification can become entertainment for experts rather than education for learners. Her core philosophy is that simple explanations work because the ideas themselves often aren't fundamentally complicated.

Key Insight: Whimsical creative constraints like fitting a website onto a fridge image can lead to elegant CSS solutions using simple image-repeating techniques.

Julia Evans shares a quick update from Day 3 at the Recurse Center about building a toy refrigerator poetry forum website. She needed to display all content on a fridge image but the content wouldn't fit, so she devised a CSS technique to create an infinitely tall fridge using three images: a top, a repeatable 1px middle line, and a bottom. The solution is simple HTML/CSS shared via CodePen.

Key Insight: Rails' conventions like associations, scaffolding, and migrations dramatically reduce boilerplate by trading configuration for sensible defaults, making it appealing for small web projects.

Julia Evans shares her second day of building a refrigerator poetry forum in Rails, a toy project designed to help her evaluate Rails for small web projects. She describes using jQuery UI's draggable library to let users drag words around, and explores Rails' ORM associations feature for efficiently loading related data. She also highlights Rails scaffolding for code generation and database migrations for schema changes, finding both features surprisingly straightforward and somewhat magical.

Key Insight: Rails' convention-over-configuration magic is powerful but produces opaque error messages that point far from the actual problem, making debugging require understanding the framework's implicit behaviors.

Julia Evans documents her first day at a Recurse Center batch, where she started a Rails project and encountered a confusing error message. She iterated on her Rails setup seven times before settling on a minimal configuration with Postgres. The core debugging story involves a typo in a validates declaration ('user' instead of 'username') that produced a misleading error about an undefined method on save. She reflects on Rails' 'magic' — both its power in providing built-in session management and authentication primitives, and its downside of producing opaque error messages that obscure root causes.

Key Insight: The most valuable feedback comes from asking readers to report what they personally found confusing, not what they think others might struggle with, because people are bad at simulating confusion they don't actually feel.

Julia Evans describes her new process for getting beta reader feedback on her zines before release. She recruits 10-20 readers from her mailing list, sends them detailed instructions about what kind of feedback she wants (things they personally found confusing, questions they have, things they learned), then consolidates and acts on the feedback. The process helped her identify patterns in confusion, make targeted improvements, and feel more confident about the final product. She emphasizes the importance of being specific about what feedback you want and targeting readers who match your intended audience.

Key Insight: Growth as a programmer means recognizing when your comfort zone has expanded and deliberately seeking new challenges in unfamiliar territory.

Julia Evans announces she's starting another virtual batch at the Recurse Center, a self-directed programming retreat. She explains what RC is—a free, self-directed retreat where you choose what to learn and work on fun projects alongside other programmers. She shares her tentative plans including learning Rails, experimenting with generative neural networks, and possibly building a game, while acknowledging she'll probably abandon these plans as she did last time. She also mentions upcoming zine releases on CSS and bash.

Key Insight: Email marketing works best when you treat it as genuinely helping people rather than selling to them, because building trust through useful content naturally leads to happy customers.

Julia Evans shares what she's learned about email marketing after overcoming her initial aversion to it. She debunks several myths she held: that selling is adversarial, that marketing email is inherently unwanted, and that you need to trick people into subscribing. Her core realization is that marketing works best when it's about building trust by genuinely helping people, rather than pushing products. She credits the 30x500 business course for reshaping her thinking and describes her 'Saturday Comics' newsletter as an example of marketing that people actually love receiving.

Key Insight: Ninja solves the build system complexity problem by making its language so simple that you generate build files with code you already know, rather than learning yet another arcane build DSL.

Julia Evans introduces ninja, a build system she finds refreshingly simple compared to make or bazel. She explains that ninja's entire syntax boils down to just two concepts: rules and builds. The key insight is that ninja files are designed to be generated programmatically, so instead of learning a complex build language, you write a Python script to generate the ninja file. She demonstrates her workflow of converting SVG files to PDFs for her zines, showing how ninja handles incremental builds without the complexity of traditional build systems.

Key Insight: Starting a physical product business works best when you resist the urge to overengineer and instead begin by doing everything yourself to learn the process firsthand.

Julia Evans announces the first print run of her Wizard Zines business, shipping 400 copies of 'Help! I have a Manager!' from her home in Montreal. She explains her decision to handle printing and shipping herself rather than using a fulfillment company, embracing the concept of 'flintstoning' — starting simple rather than overengineering. The post covers practical details like pricing, shipping logistics, and her experience finding a quality print shop. She frames this as an experiment that could scale up if successful.

Key Insight: JMAP makes it surprisingly easy for anyone to build custom email UI experiments as pure frontend apps, lowering the barrier that IMAP's complexity previously imposed.

Evans describes building a 'focus and reply' email feature for Fastmail using their JMAP protocol, inspired by a similar feature in Hey.com. She walks through the process of authenticating with JMAP, querying for emails in a 'Reply Later' folder, and rendering them in a simple Vue.js frontend. The post serves as a practical quickstart guide for JMAP since she found few examples when learning the API. She highlights JMAP's request chaining as a key design feature that reduces latency. The whole project came out to 170 lines of JavaScript running entirely in the browser with no server code.

Key Insight: The best technical writing comes from topics you once struggled with but eventually came to love, because that journey from confusion to understanding is exactly what makes the explanation valuable to others.

Julia Evans reflects on her process for choosing zine topics, identifying four key criteria: the topic must be fundamental, useful for programming work, stable over time, and possible to learn quickly. She realizes that beyond these objective criteria, she can only write about topics she genuinely finds exciting and wants to share. She notes that her best topics are things she once struggled with but eventually came to love, like tcpdump and containers. The post lists potential future zines and topics she doesn't feel ready to tackle yet.

Key Insight: Accepting that browser bugs are a normal part of CSS development, rather than a source of frustration, leads to a healthier and more productive debugging mindset.

Julia Evans shares six new CSS comics and reflects on her evolving relationship with CSS debugging. She discusses how handling browser bugs is a normal part of CSS development, citing the flexbugs repository and a specific Safari bug she encountered. She categorizes three distinct ways CSS can go wrong: wrong styles applied, correct styles with unexpected behavior due to spec misunderstanding, and correct styles broken by browser bugs. The post advocates for a calmer, more accepting mindset toward browser quirks rather than frustration.

Key Insight: OpenType substitution rules and fonttools make it surprisingly easy to programmatically modify font behavior, though making a font truly mimic handwriting remains an unsolved challenge.

Julia Evans documents her experiment using OpenType font features to make her handwritten font look more like actual handwriting by introducing character variation. She used iFontMaker to create the original font, then learned from Tristan Hume's !!Con talk that OpenType rules can replace characters based on context. Using the fonttools Python library, she wrote about 33 lines of code to merge multiple handwriting fonts and apply substitution rules. While she found the process surprisingly easy and fun, she wasn't fully satisfied with the results, describing them as 'uncanny valley' — still obviously a font but with slightly more variation.

Key Insight: CSS, like systems programming, has counterintuitive rules that become manageable once you commit to learning them systematically rather than avoiding them.

Julia Evans shares a series of CSS comics she's been creating while deliberately learning CSS. She draws a parallel between CSS and systems programming, arguing both have counterintuitive facts that become manageable once learned. She advocates reading the CSS specs directly as an authoritative source, finding that intimidating topics like selector specificity are simpler than expected. The post includes links to several comics covering the box model, units, selectors, position absolute, and inline vs block elements.

Key Insight: Telling someone's manager about their great work is a simple, high-impact practice anyone can do to help colleagues get recognized—but always ask first, because even positive feedback can backfire.

Evans advocates for telling your coworkers' managers when they're doing great work, framing it as a concrete way to support colleagues—especially those from underrepresented groups. She emphasizes always asking the person first, since well-intentioned praise can backfire if it conflicts with someone's goals or focus areas. The post details when this matters most: highlighting invisible work like documentation, cross-team help, or problem prevention that managers may not see. She explains the mechanics—be specific, explain impact, highlight when someone exceeds expectations for their level. The practice directly supports promotion cases by providing third-party evidence of valuable contributions. Evans positions this as something anyone at any level can do to lift up their coworkers.

Key Insight: Command line tools like scanimage can be simpler and more effective than GUI alternatives for tasks like scanning, turning a dreaded Linux hardware interaction into a quick shell script.

Julia Evans discovers scanimage, a command line tool for scanning documents on Linux. After struggling with a GUI scanning application, she finds that scanimage from the sane-utils package works easily from the terminal. She walks through discovering her scanner, listing its options, and writing a simple 5-line shell script to scan documents to PDF. The post captures her surprise that Linux scanning turned out to be straightforward rather than the nightmare she expected.

Key Insight: A sustainable newsletter doesn't require constant new content — curating and sequencing your best existing work can be more valuable to subscribers and more maintainable for the creator.

Julia Evans describes her 'Saturday Comics' mailing list, which sends subscribers one programming comic per week drawn from her best past work rather than her latest output. She created it to solve three problems: not everyone follows Twitter, she couldn't maintain a traditional weekly newsletter, and her best comics were hard to discover. The list uses ConvertKit's sequence feature so each subscriber starts from the beginning regardless of when they join. After a year, 5000 people have subscribed and the response has been positive. She frames the list as a trust-building marketing tool rather than a conversion-optimization exercise.

Key Insight: Documenting your existing interview process for candidates is a low-effort, low-controversy change that levels the playing field and makes interviews less wasteful for everyone.

Julia Evans advocates for companies to write documents explaining their interview process to candidates. She describes how at her previous job, candidates were sometimes caught off guard by unexpected interview formats, wasting everyone's time. The solution was simple: write a short document covering what to expect, how to prepare, and what candidates would be evaluated on. She emphasizes that documenting an existing process is far easier than changing it, and that keeping the document updated over time is even more important than the initial writing. The post frames this as a small, achievable culture change that levels the playing field for all candidates.

Key Insight: entr gives any custom build process the same instant-feedback file-watching experience that frameworks provide built-in, with just one line of bash.

Julia Evans introduces entr, a command line tool that reruns arbitrary commands whenever specified files change. She explains how it fills a gap when custom build processes lack built-in file watching, unlike frameworks like Hugo or Flask. The post covers key flags like -r for restarting servers, -c for clearing the screen, and -d for detecting new files. She demonstrates practical patterns combining entr with git ls-files to watch tracked and untracked files. The post briefly explains that entr uses inotify on Linux to watch for filesystem events.

Key Insight: Modern browser APIs have made plain JavaScript surprisingly capable for small projects, often eliminating the need for frameworks that add complexity without proportional benefit.

Julia Evans shares her experience building a small interactive website using plain JavaScript without any frameworks or libraries. After years of relying on jQuery and Vue.js for side projects, she discovered that modern browser APIs like classList, querySelectorAll, scrollIntoView, and innerHTML made it surprisingly easy to build what she needed in about 50 lines of code. The post walks through specific patterns she used, like toggling CSS classes for UI state changes and using CSS transitions for animations. She positions the advice as practical tips for fellow non-professional frontend developers who want to build small, functional websites without framework overhead.

Key Insight: How you frame a learning tool matters as much as its content — shifting from 'what don't you know' to 'what did you learn' fundamentally changes whether people engage productively with their own knowledge gaps.

Julia Evans describes her iterative process of building a formative assessment tool to help self-directed learners identify what to learn next. She started with flashcards, discovered problems with how questions interact with readers' mental models, and evolved toward a side-by-side question/answer format inspired by The Little Schemer. A key design insight was that framing matters enormously: being told what you don't know feels bad, but being helped to learn something new feels good. She added an 'I learned something' button to shift focus from scoring to discovery.

Key Insight: Technical documentation like Unix man pages is permeated with cognitive metaphors — processes are people, data is food, resources are wealth — revealing that even our most precise technical language is fundamentally figurative.

Inspired by Maggie Appleton's talk on metaphors and the book 'Metaphors We Live By,' Julia Evans explores the pervasive use of cognitive metaphors in Unix man pages. She greps through man pages on her computer looking for examples of common metaphor categories like processes as people, data as food, and limits as buildings. The post catalogs dozens of examples organized by metaphor type, revealing that technical documentation is saturated with figurative language we rarely notice. She finds metaphors treating processes as living beings (death, birth, orphaned), data as physical objects or food, resources as consumable goods, and system limits as architectural features like floors and ceilings.

Key Insight: Docker's --cap-add flag secretly does double duty, adding both Linux capabilities and seccomp-bpf whitelist entries, which is why it fixes strace even though the root cause is a seccomp rule blocking ptrace, not a missing capability.

Julia Evans investigates why strace fails inside Docker containers with a 'ptrace not permitted' error. She systematically tests and rejects two hypotheses — missing CAP_SYS_PTRACE capability and user namespace differences — before discovering the real cause: Docker's default seccomp-bpf profile blocks the ptrace system call entirely. She then digs into Docker's source code to explain why --cap-add=SYS_PTRACE fixes it, finding that the flag secretly also whitelists related system calls in the seccomp profile. The post demonstrates how containers are built from multiple interacting security mechanisms that can produce surprising behavior.

Key Insight: Containers stop being confusing once you understand they're just six Linux kernel features glued together, and that understanding of fundamentals is what makes debugging production container issues tractable.

Julia Evans announces her new zine 'How Containers Work!' which explains the Linux kernel features that make containers possible. She describes how containers initially confused her despite her Linux knowledge, and explains they're actually six different features glued together. The zine covers cgroups, namespaces, pivot_root, seccomp-bpf, and related fundamentals. She demonstrates that containers can be implemented in just 15 lines of bash, demystifying them. The zine is deliberately container-runtime-agnostic, focusing on kernel primitals rather than Docker or Kubernetes. She also announces a redesign of wizardzines.com done with designer Melody Starling.

Key Insight: Signed distance functions make shader programming surprisingly approachable by replacing complex conditional geometry with simple, composable mathematical functions that define shapes by distance.

Julia Evans walks through her process of learning to write GPU shaders, starting from the default Shadertoy example and building up to a shiny spinning octahedron animation. She introduces the core concept of signed distance functions (SDFs) as a powerful alternative to hardcoded if-statements for defining shapes, explaining how they're simple to define, easy to compose, and easy to transform. The post demonstrates a step-by-step learning approach: start with the basics, iterate quickly with better tooling, draw simple shapes with math, then adopt SDFs and raytracing for 3D rendering. She emphasizes that shaders are more approachable than they appear, especially when you copy working tutorial code and modify it experimentally.

Key Insight: SQL is a high-leverage skill worth learning well because it unlocks the ability to query massive datasets across the many powerful SQL databases that exist, but its non-intuitive behaviors need clear, example-driven explanation.

Julia Evans announces her new zine about SQL called 'Become a SELECT Star!' She explains her motivation for writing it, drawing on her experience writing SQL queries for machine learning at her previous job. The zine focuses on SQL basics with carefully crafted examples showing queries, input tables, and outputs. She also built an interactive SQL playground using SQLite compiled to WebAssembly so readers can run the examples in their browser. She teases that her next zine will likely cover containers.

Key Insight: The best tools are ones that deliver what you want out of the box, without requiring you to invest time in configuration.

Julia Evans describes her journey from lightweight window managers through tiling window managers like xmonad back to Gnome, and then discovering PaperWM, a Gnome extension that brings tiling window management to the Gnome desktop. PaperWM arranges all windows in a line rather than arbitrary tiles, which she finds appealingly simple. She highlights that she only uses a handful of keyboard shortcuts. The post emphasizes her growing appreciation for tools that work well out of the box without requiring configuration.

Key Insight: The word 'server' is genuinely ambiguous — it can mean a program, a VM, a container, a physical machine, or a distributed system — and this ambiguity is usually a useful abstraction rather than a problem.

Evans explores why the seemingly simple question 'what's a server?' turns out to be surprisingly hard to answer. She walks through multiple valid meanings: a server can be a program listening on a port, a virtual machine you SSH into, a container with cgroup limits, a physical 1U rack-mounted computer, or an entire distributed system of thousands of machines. Using Gmail search as a concrete example, she shows how 'it happens on the server' can mean a request touches 20 computers across 3 countries. The conclusion is that 'server' is context-dependent and genuinely ambiguous, which is fine until something goes wrong.

Key Insight: X batches window management events by default, and understanding this through tracing tools like xtrace reveals how debugging at the protocol level can quickly resolve seemingly mysterious behavior.

Evans shares the results and her experience from a programming challenge to write a tiny window manager that bounces windows around the screen. She explains what an X window manager is and how it communicates with the X server via messages. The core technical lesson involves discovering that X batches XMoveWindow calls, which prevented sequential moves from working as expected. The solution was using XSync to force X to process events immediately. She also showcases several community solutions that went beyond the basic challenge requirements.

Key Insight: Intimidating-sounding systems programming tasks like writing a window manager become approachable when you define a deliberately minimal scope and build on existing tiny implementations.

Julia Evans presents a programming challenge to build a deliberately silly Linux window manager that bounces windows around the screen. She recommends starting from tinywm, a 50-line C window manager, and provides tooling instructions using Xephyr for safe development. She completed the challenge herself in about 5 hours with 50 lines of code, despite having never looked at window manager source code before. The post frames what sounds intimidating—writing a window manager—as surprisingly approachable when you strip away all the real requirements.

Key Insight: Good programming exercises require rigorous testing and thoughtful design — clear success criteria, appropriate challenge, minimal setup friction, and room to grow — because learners invest far more time and expect to develop real skills, not just absorb facts.

Julia Evans explores what qualities make programming exercises effective for learning. She identifies criteria including being fun, teaching relevant skills, providing appropriate challenge, having clear success criteria, and being completable in a reasonable time. She draws an analogy to Julia Child's cookbook testing process, noting that exercises require more rigorous testing than blog posts because learners invest more time and expect to develop actual skills. The post also emphasizes practical concerns like avoiding setup friction, making help accessible, and ensuring learners won't get stuck on irrelevant details. She concludes with a consolidated list of nine criteria that good exercises should meet.

Key Insight: Overlay filesystems are a surprisingly simple mechanism at the heart of containers — you can understand them completely by running a single mount command and experimenting with reads, writes, and deletes.

Julia Evans explains how overlay filesystems (overlayfs) work as a key building block of container technology. She demonstrates that containers use overlays instead of copying entire images to save disk space and time, with a read-only lower directory and a writable upper directory. Through hands-on terminal examples, she shows how reads, writes, and deletes work in an overlay, including the surprising character device trick used to represent deleted files. She also explains how Docker layers map to multiple lower directories and briefly covers alternative storage drivers like btrfs snapshots.

Key Insight: Growing beyond a simple tool's limitations is a sign of progress, and the right upgrade is one complex enough that you won't outgrow it.

Julia Evans shares her experience switching from Notability to Affinity Designer for drawing her programming zines on iPad. After making four zines with Notability's simple interface, she hit limitations around mixing text boxes with pencil drawings. Affinity Designer offers dramatically more features including custom paper sizes, color palettes, object grouping, and flexible text handling. She notes that importing existing work via PDF is surprisingly smooth. The post also reflects on how iPad creative apps offer great value and capability, even for someone who primarily uses Linux.

Key Insight: Because shipping costs are based on weight tiers rather than per-item, bundling multiple lightweight zines into packs of 5-10 makes print distribution economically viable where single-zine orders would not be.

Julia Evans shares her research into logistics for shipping printed zines to customers worldwide. She evaluates several options including Amazon's various programs (KDP, Advantage, Fulfilled By Amazon), and third-party fulfillment services like Blackbox, Shipbob, and Whiplash. She discovers that letter mail won't work for items with monetary value, and that shipping costs are charged in weight increments that make single-zine orders impractical. She concludes that bundling 5-10 zines per shipment is the most economically viable approach, since a pack of lightweight zines costs roughly the same to ship as a single one.

Key Insight: SQLite's single-file amalgamation build is so simple that compiling from source is actually easier than hunting for compatible pre-built binaries.

Julia Evans needed a newer version of SQLite to support window functions for a SQL examples website, but upgrading turned into a surprisingly educational adventure. After failing to use a pre-built binary (wrong loader for the build server), breaking her system trying to install a mismatched Debian package, and hitting shared library version mismatches, she finally tried compiling from source. SQLite's compilation process turned out to be remarkably simple: just download the amalgamation tarball and run a single gcc command. The experience highlighted how SQLite's design for embedded contexts makes it uniquely easy to build from source.

Key Insight: Sometimes the best reason to pursue a publishing deal isn't the money but the learning opportunity, especially when you can negotiate to keep your work freely accessible.

Julia Evans announces a print box set of her first seven free Linux zines, published through No Starch Press. The collection includes zines on strace, networking, tcpdump, perf, debugging tools, and Bite Size Linux, with new color covers. She negotiated to keep all the zines freely available online despite the print deal. The project was motivated more by curiosity about print publishing than profit, as she earns only 10% per sale compared to 97% from her online store.

Key Insight: Understanding that SQL queries semantically execute in the order FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT demystifies why certain queries are valid and others aren't, even though database engines optimize the actual execution order.

Julia Evans explains that SQL SELECT queries don't conceptually execute starting with SELECT — they follow a specific semantic order: FROM/JOIN, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT. This ordering explains why certain operations are valid or invalid, like why you can't filter on window functions in WHERE. While database engines optimize and reorder operations for performance, understanding the semantic order helps reason about query correctness. She notes that other query tools like LINQ and pandas naturally follow this more logical order, and expresses surprise that this foundational concept isn't more widely taught.

Key Insight: Teaching developers technical concepts through self-published zines can generate nearly six figures in annual revenue, proving that quality educational content can be monetized directly without ads or sponsorships.

Julia Evans shares revenue data from her zine business (wizardzines.com) for 2019, revealing $87,858 in revenue through September, on track to match 2018's $101,558. Revenue spikes dramatically in months when new zines are released, with September's $30,000 driven mostly by individual sales of the HTTP zine. She breaks down expenses at roughly 10% of revenue and highlights her buy-one-give-one program that has distributed 1,358 free copies. The post aims to demonstrate that making money by teaching technical topics directly to developers is viable without relying on ads or sponsorships.

Key Insight: Effective programming exercises should start with a challenge that tests the target skill, then offer easier exercises only if needed, because many learners already have partial knowledge and need to be pushed rather than walked through basics.

Julia Evans shares her process of building interactive SQL exercises designed to help people improve their practical SQL skills. She explains that SQL is exciting because distributed SQL engines let you query billions of rows quickly, and that exercises are needed because knowledge differs from skills. Her key design insight was to start each section with a challenge rather than easy exercises, so experienced users aren't bored. She tracks anonymous query data to understand where people get stuck and iterates on the exercises based on real usage patterns.

Key Insight: After years of explaining computer things as a side project, turning it into a sustainable solo business feels like a natural next step when the work is something you feel compelled to do.

Julia Evans announces she's leaving her job to spend a full year working on explaining computer things. After six years of blogging, 11 zines, 370,000 words, and 20 talks, she's turning her passion for technical education into a full-time endeavor. She plans to write more zines and build interactive learning tools. She'll run it as a solo business, which is already generating enough revenue to live on, without plans to hire employees or take on consulting work.

Key Insight: HTTP is fundamentally simple, and understanding its basics transforms caching, cookies, and web security from impenetrable magic into straightforward concepts you can debug and reason about.

Julia Evans announces her new zine 'HTTP: Learn your browser's language!' which teaches web developers the fundamentals of the HTTP protocol. She explains that understanding HTTP requests, responses, and headers demystifies caching, cookies, and web security. The zine was inspired by a conversation with Marco Rogers about how new developers would benefit from understanding HTTP basics. Evans describes her iterative creative process of posting individual pages to Twitter and incorporating feedback before assembling the final zine. She also introduces examplecat.com, a whimsical domain she registered for the zine's HTTP examples.

Key Insight: The simplest path to learning web development is publishing a single HTML file on the internet, stripping away all the tooling and complexity that usually obscures what's actually happening.

Julia Evans explains the simplest possible way to put a static HTML page on the internet, walking through creating a single index.html file and hosting it on Neocities. She advocates for keeping web projects simple, noting that all her own sites are just static HTML and CSS. She mentions alternatives like GitHub Pages, GitLab Pages, and Netlify, but recommends Neocities as the lowest-friction starting point for beginners. The post serves as an encouraging entry point for people who want to learn HTML/CSS by actually publishing something.

Key Insight: Zine-making is fundamentally low-tech and accessible—the barrier to entry is paper and a pen, not expensive software or hardware.

Julia Evans explains practical, accessible ways to create zines without expensive tools. She realized that when people asked about her tools, they really wanted to know how to get started themselves, and 'buy a $500 iPad' wasn't a helpful answer. She walks through five methods for creating zine content—paper, Google Docs, iPad, single-sheet folding, and Adobe Illustrator—then covers how to assemble pages into booklets using either a photocopier or computer-based methods like pdflatex. The core message is that zine-making is fundamentally a low-tech, accessible activity.

Key Insight: Git becomes much less intimidating when you start by learning to navigate and read a repository using just five commands, rather than trying to learn everything at once.

Julia Evans presents a set of 18 hands-on Git exercises focused on navigating a real repository (Ruby's source code) rather than a toy example. The exercises use only five git subcommands (checkout, log, diff, show, status) to build comfort with SHAs, tags, branches, and viewing file history. She emphasizes that git SHAs always refer to the same code, making them a reliable anchor for understanding repository state. The post includes complete solutions for each exercise, making it self-contained for self-study. The approach breaks down Git learning into a manageable chunk by focusing purely on reading and navigating rather than writing or modifying.

Key Insight: curl is a micro-skill that can be learned through deliberate practice in a few focused sessions, and the resulting fluency pays off every time you need to quickly test an API.

Julia Evans presents 21 curl exercises designed as a deliberate practice experiment for learning HTTP requests from the command line. Inspired by Kathy Sierra's concept of micro-skills that can be learned in three 45-minute sessions, she identifies curl fluency as one such skill. The exercises use httpbin.org as a mirror to reflect requests back, progressively covering GET, POST, PUT, DELETE, headers, JSON bodies, authentication, and content negotiation. The post argues that fluency with command-line tools like curl is genuinely valuable for day-to-day development work like testing APIs.

Key Insight: Debugging is fundamentally a cycle of questioning your own assumptions, designing experiments to test them, and building better mental models of the system — and the hardest part to teach is knowing where you're stuck.

Julia Evans aggregates community wisdom on debugging into a practical framework after realizing there aren't many good debugging guides. She breaks down the process into reproducible steps: reproduce the bug, reproduce it quickly, check your assumptions, and experiment methodically. The post emphasizes that good error handling and informative error messages are as important as debugging technique itself. She highlights creative debugging approaches like using sounds and LEDs when traditional tools aren't available. The post acknowledges that debugging skill is hard to teach because the hardest parts vary by experience level, and she's still trying to understand where newer programmers actually get stuck.

Key Insight: Tensor categories formalize the intuition that you can prove and define mathematical concepts through pictures, where spatial arrangement in diagrams carries precise algebraic meaning.

Evans explains why monoidal (tensor) categories are interesting, focusing on how they let you prove and define mathematical concepts using pictures/diagrams. She walks through examples from the paper 'Seven Sketches in Compositionality,' showing how inequalities can be proved by arranging boxes and lines in diagrams, and how the trace of a matrix can be defined by simply drawing a circle around a function. She connects this to her own master's thesis on topological quantum computing and acknowledges she's not sure it relates to programming, though others have drawn connections to software design.

Key Insight: Networking tools aren't inherently hard — the real barrier is that documentation doesn't distinguish the essential from the obscure, making practical knowledge feel more inaccessible than it actually is.

Julia Evans announces her third 'Bite Size' zine focused on networking, explaining that it covers practical skills like inspecting SSL certificates, making DNS queries, and capturing network traffic. She frames the zine as the resource she wished she had when she started doing networking work, noting that these tasks aren't inherently hard but are obscured by unhelpful man pages. She also reflects on the positive experience of switching from free to paid zines, discusses wanting better feedback from readers, and teases potential future topics.

Key Insight: Blog organization should evolve with the size of the archive — reverse chronology works early on, but at scale, emergent categorization helps readers understand the trains of thought behind a body of work.

Julia Evans describes reorganizing her blog's front page from a chronological list into 28 categories, after six years and 390 posts. She traces the evolution from a standard 'show 5 recent posts' layout to a full chronological list inspired by Dan Luu, and finally to categorized sections. Her method involved dumping all post titles into a text file and using a Marie Kondo-style approach to discover natural groupings, then implementing them as a Hugo taxonomy. She argues that while reverse chronology works well for years, adding structure eventually becomes necessary for readers to navigate a large body of work.

Key Insight: Anonymous review and a focus on joy over prestige creates a conference where the best talks come from unexpected people sharing genuine excitement about computing.

Julia Evans promotes !!Con 2019, a two-day conference in NYC celebrating the joy, excitement, and surprise of computing. She explains the conference's inclusive philosophy, including open captioning, paid speakers, anonymous talk review, and active diversity outreach. The post encourages submissions by showcasing past talk titles that range from Islamic geometry to making mushrooms glow. New this year, non-traditional formats like plays and songs are welcome. She emphasizes that anonymous review helps surface great first-time speakers and that detailed outlines matter more than reputation.

Key Insight: The most useful networking tool knowledge is typically passed between coworkers informally, not found in man pages, which is exactly the gap a good zine can fill.

Julia Evans shares work-in-progress comics from her upcoming zine about Linux networking tools. She explains that while she loves networking, getting started with the tools is tricky because their man pages are enormous and the most useful tricks are often learned from coworkers. The zine aims to be that helpful coworker who shares practical tips about tools like curl, ssh, openssl, netcat, nmap, and ethtool. She advocates mixing basic and advanced tools so beginners learn what exists.

Key Insight: The best marketing strategy for independent creators is making genuinely good work, sharing it freely, and simply telling people where they can buy more — no aggressive optimization needed.

Julia Evans shares her early thinking about marketing for her zines business, exploring how to promote her work in a way that feels authentic and not annoying. She examines common internet marketing techniques like mailing lists, social proof, and advertising, evaluating each against her personal comfort level. Her current approach relies heavily on Twitter, where she posts free comics that naturally attract followers who then buy her zines. She considers whether to invest in email marketing and conversion tracking but remains skeptical of aggressive optimization, preferring to focus on making good things and letting people know where to find them.

Key Insight: The guarantees of nonparametric methods like the bootstrap rest on surprisingly deep mathematics, and having more data points provides exponentially — not linearly — better statistical confidence.

Julia Evans works through the early chapters of 'All of Nonparametric Statistics' by Larry Wasserman, trying to build intuition for nonparametric methods and the bootstrap. She explores how nonparametric statistics avoids strong distributional assumptions, walks through Hoeffding's inequality to show how confidence scales with dataset size, and attempts to understand the formal conditions under which bootstrap estimators are consistent. She finds that the math behind bootstrap consistency is surprisingly difficult and that even the textbook doesn't fully prove general consistency. The post is explicitly a work-in-progress learning journal rather than a polished explanation.

Key Insight: Building innovative tools that inspire others, teaching through honest business practices, and making small workplace improvements are all deeply satisfying forms of impact that compound over time.

Julia Evans reflects on her 2018 side projects and professional growth. She built rbspy, a Ruby profiler that inspired similar tools for Python and PHP. She launched a surprisingly successful zine business at wizardzines.com that could have sustained her financially. She shifted her conference talks from 'cool tools' to substantive work topics, gave her first podcasts, and wrote 54 blog posts. She also describes small but meaningful workplace inclusion projects and her ongoing struggle to define what a senior engineer's job actually is.

Key Insight: When operating high-reliability infrastructure, deliberately ignoring most of the ecosystem to focus on a few core, well-understood components is a valid and often superior strategy to chasing every new tool.

Julia Evans shares her KubeCon talk about making major infrastructure changes while maintaining high availability (99.99%+ uptime), drawing on her experience with Kubernetes and Envoy at Stripe. She provides a toolbox for derisking migrations and making infrastructure changes with confidence. She advocates for ignoring most Kubernetes ecosystem software to focus on core components, and notes that many companies are still figuring out how to use this new software reliably. The talk represents a personal milestone as her first work talk she's genuinely happy with, reflecting two years of learning how to make big infrastructure changes safely.

Key Insight: Tech comics succeed or fail based on the clarity of the information they convey, not the quality of the art—most people underinvest in the explanation and overinvest in the illustration.

Julia Evans addresses the frequent requests she gets to create comics for open source projects, and instead offers a practical guide for anyone wanting to document tech projects with comics. She clarifies that a zine is a printed booklet, not a single drawing, and emphasizes that the information matters far more than the art. She outlines three formats—a single graphic, a series of comics, or a printed zine—and explains when each is appropriate. She also covers three approaches to hiring: finding a tech-savvy illustrator, collaborating with an illustrator while providing clear explanations, or drawing it yourself with stick figures. The core message is that most people underinvest in the explanation and overinvest in the art.

Key Insight: RAII isn't just for freeing memory — tying any cleanup action to object lifetime via destructors makes resource management automatic and nearly impossible to forget, even for non-obvious resources like reference counters.

Julia Evans explains how she came to understand RAII (Resource Acquisition Is Initialization) through working with Envoy's C++ codebase. She walks through a concrete example: Envoy's circuit breakers need to track open TCP connection counts accurately, and destructors solve this by automatically decrementing the counter when a connection object is destroyed. The post demonstrates how tying resource cleanup to object lifetime prevents bugs that would arise from manually tracking cleanup across many code paths. She notes this pattern works because C++ guarantees destructors run when objects leave scope, are removed from collections, or when exceptions occur.

Key Insight: Running new software in production inevitably causes incidents, so the strategy is to encounter them in non-critical contexts first, ensure each incident only happens once through thorough investigation, and categorize failure modes to focus attention where it matters.

Evans discusses the significant investment required to run new software in production safely, particularly at high availability targets like 99.99% uptime. She presents a practical playbook: start with non-critical use, let it bake, run into problems, and fix them iteratively. Using Envoy and Kubernetes as concrete examples, she catalogs the kinds of issues teams encounter—mostly configuration surprises rather than bugs—and argues for treating each incident as a one-time learning opportunity. She emphasizes categorizing components into 'ok to break' and 'not ok to break' lists to focus attention where it matters most.

Key Insight: Constrained utility-class CSS frameworks like Tailwind can be more effective than traditional CSS for occasional web developers because limited options and inline-style composition eliminate the cognitive overhead of naming classes and managing stylesheets.

Julia Evans describes her experience using Tailwind CSS to build a website for her zines, despite having self-described mediocre CSS skills. She contrasts the old CSS Zen Garden philosophy of semantic HTML with Tailwind's utility-class approach, where you add many small CSS classes directly to HTML elements instead of writing custom CSS. She finds Tailwind's approach to media queries particularly useful, as prefixes like 'md:' make responsive design trivially easy. The post argues that this utility-first approach works well because HTML templates reduce duplication, and the constrained set of options actually helps non-experts make better design decisions.

Key Insight: The best teaching diagrams and comics are opinionated — they highlight the few key architectural decisions or scenarios that matter most, rather than trying to comprehensively map every component.

Julia Evans explores the question of when teaching with comics is most effective, pushing back on simplistic 'learning styles' explanations while acknowledging that learning preferences still matter for self-directed learners. She identifies four specific situations where comics excel: creating focused architectural diagrams that highlight key decisions rather than exhaustively mapping components, illustrating real-world scenarios from complex systems, summarizing structured lists like command-line arguments, and explaining simple ideas where mixing text and small diagrams aids comprehension. Her central insight is that good teaching diagrams should be opinionated, emphasizing the most important architectural decisions rather than trying to be comprehensive.

Key Insight: Envoy's real power over traditional proxies like nginx and HAProxy lies in its dynamic configuration via discovery services, which enable centralized control of distributed network infrastructure without application code changes, though this power comes with significant operational complexity.

This post introduces Envoy, a network proxy in the same space as HAProxy and nginx, focusing on practical configuration basics. Evans walks through a minimal working example, explaining Envoy's core concepts: listeners (bind to ports), filters (process requests), routes (match and direct traffic), and clusters (backend server groups). She then covers Envoy's distinctive feature of dynamic configuration via discovery services (CDS, EDS, RDS, LDS), which allow centralized control of network routing without config file restarts. The post concludes that discovery services are what make Envoy genuinely exciting, enabling circuit breaking, traffic shaping, and encrypted communication without application code changes, while acknowledging the significant operational complexity involved.

Key Insight: Expertise can paradoxically make it harder to teach a topic because you forget what it was like to be confused, and collaboration with someone who remembers that confusion can bridge the gap.

Julia Evans announces a new zine collaboration with Katie Sylor-Miller called 'Oh shit, Git!' that helps people fix common git mistakes. The zine was inspired by Katie's website ohshitgit.com, which explains how to recover from git mishaps. About 50% of the content is new, including fundamental explanations of how git works internally. Julia reflects on the challenge of writing about git because she had forgotten what it felt like to be confused by it, and notes this is her fifth zine of 2018.

Key Insight: Writing down a broad inventory of possible career goals — then honestly asking which ones would actually feel meaningful — is more useful than defaulting to whatever your company's career ladder rewards.

Evans catalogs a wide range of possible career goals for software developers, organized into categories like technical, job, entrepreneurship, product, people/leadership, communication, and work environment. The list was crowdsourced from Twitter and intentionally mixes big and small goals pointing in very different directions. She emphasizes that no one can tell you what your goals should be, and that achieving a goal doesn't always feel as satisfying as expected. The post was inspired by the idea that company career ladders don't necessarily align with personal goals, and encourages readers to reflect on which milestones would actually feel meaningful to them.

Key Insight: Selling creative educational work rather than giving it away can actually increase genuine readership while funding higher quality output, because people engage more seriously with things they pay for.

Julia Evans explains her decision to switch from giving away her programming zines for free to selling them for $10 each. She weighs the tradeoff between free distribution (which maximizes reach) and paid distribution (which funds higher quality work and signals seriousness). Using her own data, she argues that 3,000 paid sales may result in more actual readers than 5,000 free downloads, since buyers are more likely to read what they paid for. She discusses corporate pricing experiments, the advantages of posting drafts on Twitter as a free complement, and various friction points she's encountered with selling digital products.

Key Insight: Working effectively with your manager is a learnable skill built on clear communication, not something that just happens naturally even with a great manager.

Julia Evans announces her new zine 'Help! I have a manager!' which covers how to work effectively with your manager. The zine focuses on communication skills she wishes she'd known at the start of her career. She emphasizes that even with a great manager, there are proactive steps you can take to improve the working relationship. The zine covers practical topics like what to discuss in 1:1 meetings and how to ask for feedback.

Key Insight: The myths we believe about what makes programming projects 'impossible' often turn out not to be true, and ambitious work is more accessible than we assume.

This is a brief announcement post sharing Julia Evans' talk 'Build Impossible Programs' from the Deconstruct 2018 conference. The talk draws on her experience building a Ruby profiler (rbspy) and focuses on debunking myths about doing ambitious programming work. Unlike her previous technical deep-dive on the same project, this talk takes a 'squishier' approach examining beliefs that hold people back from attempting difficult projects. She also mentions her hand-drawn slide style and notes that 2300 zines were printed and distributed at the conference.

Key Insight: Rust-based sampling profilers like py-spy and rbspy are converging on a shared ecosystem of code and consistent interfaces that could make profiling accessible across many programming languages.

Julia Evans highlights py-spy, a new Python profiler written in Rust by Ben Frederickson that takes a similar approach to her own rbspy project. She explains how py-spy can profile any running Python program using process_vm_readv to read memory, and praises its top-like default UI and pip-based installation. She notes that py-spy is likely more stable long-term than rbspy because it relies on Python's public header files rather than unstable internal interpreter structures. Evans expresses excitement about an emerging ecosystem of Rust-based sampling profilers that share code and could eventually cover many programming languages with consistent interfaces.

Key Insight: When you can't find or fix the root cause of an intermittent infrastructure bug, a well-placed workaround that patches the symptom at the right layer can be the most practical solution.

Julia Evans describes a six-month intermittent bug where her blog would display raw HTML instead of rendered pages due to a missing Content-Type HTTP header. Despite extensive debugging attempts, she couldn't reliably reproduce the issue or determine whether her web host or Cloudflare's CDN was dropping the header. She ultimately solved it using Cloudflare Workers, a paid feature that lets you run custom JavaScript on every HTTP request, to detect and add the missing Content-Type header. She acknowledges it's a hack but finds it preferable to migrating web hosts.

Key Insight: Learning hard things requires the concrete skill of decomposing vague confusion into specific, answerable questions—a skill that itself improves with deliberate practice.

Evans breaks down the meta-skill of learning into four concrete, practicable sub-skills: identifying what you don't understand, having confidence in what you do know, asking good questions, and doing effective research. She argues that a growth mindset alone isn't sufficient—learning is itself a skill that requires deliberate practice. Drawing on her experience with self-directed math education, a pure math degree, and the Recurse Center, she shows how translating vague confusion into specific questions is the most critical skill. She reframes confusion as a positive signal that you're about to learn something new.

Key Insight: Most free developer education is funded by misaligned incentives—reputation-building or product marketing—which raises the question of whether developers should invest more in paid learning materials that are directly incentivized to be excellent.

Julia Evans examines the economics of developer education, mapping out who actually pays for the learning resources that professional developers rely on. She identifies several categories: companies with products to sell (developer advocates), individuals investing in exposure (bloggers and speakers), recruiting-funded education (Recurse Center), companies and individuals selling education directly, and corporate training. She reflects on the fact that most of what she reads is free, funded either by people building their reputation or companies marketing products, and questions whether those incentive structures produce the best learning materials. The post is prompted partly by her own transition from giving everything away for free to selling her zines.

Key Insight: When a creative side project unexpectedly becomes commercially successful, the resulting business complexity is worth embracing because it enables better collaborations and higher-quality work—but it also forces hard questions about accessibility and incentives.

Julia Evans announces she started a Canadian corporation to manage the business side of her increasingly successful zine sales. After Bite Size Linux sold 2,600 copies—far exceeding expectations—she needed proper infrastructure for taxes, expenses, and paying collaborators. The corporation makes it easier to hire illustrators, collaborate with other creators, and potentially expand into print shipping. She also introduced corporate pricing tiers for workplace use. The post ends with thoughtful open questions about the tension between selling creative work and keeping it accessible.

Key Insight: Podcasts offer a lower-friction alternative to conference talks for sharing technical knowledge, though they trade away the editing control that makes writing feel safer.

Julia Evans shares four recent podcast appearances covering topics from rbspy and profiling to zines, teaching, and computer networking. She reflects on the differences between podcasting and her primary medium of writing, noting that podcasts don't allow the same editing control. She finds podcasts easier to organize than conference talks since they require no travel, and notes the unexpected importance of recording equipment quality. She sees podcasting as useful practice for extemporaneous speaking and mentions aspirations for future recorded content like animated videos.

Key Insight: Command line mastery is often gatekept by verbal tradition rather than documentation, and curated guides that highlight what actually matters can democratize that knowledge.

Julia Evans announces her new zine 'Bite Size Command Line', a sequel to 'Bite Size Linux', which explains the basics of Unix command line tools in an approachable format. Despite having used Linux for 15 years, she learned several new tricks while writing it. The zine aims to help beginners by curating the most important command line arguments and options, since man pages often overwhelm with too many options. She positions the zine as a substitute for the verbal knowledge transfer that typically happens between experienced and newer command line users.

Key Insight: IP routing on Linux involves three layered systems — route tables, iptables NAT, and IPsec — and understanding how they intercept and rewrite packets is essential for anyone working with containers or cloud networking.

Julia Evans breaks down how IP addresses and routing work on Linux, covering three systems that affect packet delivery: route tables, iptables, and IPsec. She explains subnets and CIDR notation, then walks through how Linux decides where to send packets using the route table. The bulk of the post covers iptables NAT — both destination NAT (used to redirect packets to different addresses, as in container networking tools like kube2iam) and source NAT (used to masquerade container IPs as host IPs). She closes by mapping these Linux networking concepts to their AWS VPC equivalents, showing how the same principles apply at cloud infrastructure scale.

Key Insight: Counter-intuitively, sending network packets more slowly at the bottleneck rate produces better performance than sending as fast as possible, and a small change to Linux's packet data structure could enable this across much of the internet.

Julia Evans summarizes a keynote by Van Jacobson at netdev about the future of congestion control on the internet. The talk argued that the internet has fundamentally changed since 1988: switches are no longer dramatically faster than servers, so clients can easily saturate links in the middle of the network. Counter-intuitively, sending packets more slowly at the bottleneck rate actually improves performance because it avoids tail-of-stream packet drops that cause expensive timeouts. The proposed solution involves adding an 'earliest departure time' timestamp to Linux kernel packet data structures and replacing queues with timing wheels for scheduling. Since Linux runs on Android phones, wifi routers, and most servers, changing the Linux networking stack alone could meaningfully improve internet congestion.

Key Insight: IPsec's split architecture—userspace for infrequent key exchange, kernel for packet encryption—is what makes it practical to implement in the Linux kernel, unlike TLS which requires constant per-connection negotiation.

Julia Evans shares her notes from the first day of the netdev conference, where she attended an IPsec workshop. She explains that IPsec is a kernel-level protocol for encrypting IP packets, commonly used for VPNs but not exclusively. The post breaks down how IPsec works with its userspace key exchange (IKE) and kernel encryption components, and contrasts it with TLS to explain why IPsec lives in the kernel. She covers the two modes of IPsec (transport and tunnel), the concept of opportunistic IPsec using DNS records, and notes that IPsec and TLS have historically influenced each other's design.

Key Insight: You only need a small number of Wireshark techniques to effectively debug most networking problems, and the tool's real power lies in making abstract protocols concrete and helping you quickly determine which side of a connection is misbehaving.

Julia Evans shares her practical Wireshark workflow for debugging networking problems, focusing on the small number of features she actually uses regularly. She describes her typical process of capturing packets with tcpdump in production, then analyzing them locally in Wireshark. The post covers five main techniques: analyzing pcap files, isolating TCP connections, decoding protocols on unusual ports, inspecting packet contents, and searching with Wireshark's query language. She emphasizes that knowing just a few Wireshark features covers 95% of her use cases, and that the tool doubles as a way to learn networking protocols by making them concrete and inspectable.

Key Insight: The ed text editor, often dismissed as an obsolete Unix relic, becomes genuinely useful when you realize that any ed session is a replayable script — making it a simple way to batch-apply Vim-macro-like edits across many files.

Julia Evans discovered a practical use for the ed text editor when she needed to batch edit 200 files at work. After reading Michael W Lucas's 'Ed Mastery' book (published as a semi-joke on April Fools' Day), she learned that any ed session can be replayed as a script, making it useful for applying vim-macro-like edits to multiple files. She walks through how she translated a simple Vim workflow into an ed script to insert lines after a pattern match. The post reframes ed from a terrifying relic into an occasionally useful tool for batch file editing.

Key Insight: Getting a useful stack trace from a segfault on Linux is straightforward once you know the two settings that silently prevent core dumps: ulimit -c and kernel.core_pattern.

Evans walks through the practical steps of debugging a segfault on Linux, from getting a core dump to extracting a useful stack trace with gdb. She explains the two key settings that often prevent core dumps from being written: ulimit and kernel.core_pattern. The post covers how Ubuntu's Apport system intercepts core dumps by default and how to override it. She also briefly mentions valgrind and ASAN as alternative approaches. The post distills what took her a full week of frustration into a clear, reproducible five-step process.

Key Insight: A timed-release model — paid early access followed by free release — can sustainably fund educational technical content where pure donations fail.

Julia Evans announces the release of her new zine about profiling and tracing with perf on Linux. She highlights that perf is an underappreciated tool that many developers don't know about, and the zine aims to help people get started with it. She also explains her funding model: selling early access for $10, then releasing for free after a few weeks. 250 people bought early access to this zine.

Key Insight: Building a Ruby profiler requires reading memory directly from the Ruby interpreter across multiple versions without stopping the target process.

This is a brief announcement post about a talk Julia Evans gave at Localhost, the Recurse Center's monthly talk series. The talk covers the technical core of rbspy, her Ruby profiler tool, focusing on how it reads memory from the Ruby interpreter to determine what function a process is running. She highlights the challenges of supporting multiple Ruby versions and whether the Ruby process needs to be stopped during profiling. The post is essentially a pointer to the 30-minute video talk followed by 30 minutes of Q&A.

Key Insight: Reverse engineering proprietary file formats is more approachable than it seems because developers reuse standard formats like zip, plist, and simple float arrays rather than inventing custom encodings.

Julia reverse engineered the Notability app's proprietary .note file format to convert her existing SVG drawings from another app. She discovered the format is a zip file containing Apple plist files with drawing data stored as arrays of 32-bit floats. The entire process took about a day and resulted in less than 200 lines of Python. She walks through each step from unzipping the file to decoding curve points, demonstrating that reverse engineering is more approachable than it seems because developers tend to reuse standard formats rather than inventing custom ones.

Key Insight: Sabbaticals and fellowships can unlock meaningful open source work that wouldn't happen otherwise, but they require flexibility from sponsors and employers, a prototype to build confidence, and the willingness to apply before you have all the logistics figured out.

Julia Evans reflects on her experience taking a 3-month open source sabbatical funded by Segment's Open Fellowship to build rbspy, a Ruby profiler. She catalogs various open source funding models and explains how flexibility from both Segment and her employer Stripe made the sabbatical possible. She emphasizes that having a working prototype beforehand gave her confidence, and that applying even without a clear plan led to a great outcome. The post encourages others to consider sabbaticals and to apply for opportunities even when logistics seem uncertain.

Key Insight: Software documentation should go beyond basic usage to include conceptual guides and tool comparisons that help users understand the broader domain, not just the specific tool.

Julia Evans announces the documentation site for rbspy, her Ruby profiler. She emphasizes her belief that undocumented software might as well not exist, and describes the extra effort she put into making the docs useful. Beyond standard usage docs, she included a comparison with stackprof and a general guide to profiling concepts like flamegraphs and benchmarking. She built the site with Hugo using a minimalist theme and added a responsive hamburger menu for mobile.

Key Insight: Rust's greatest potential is democratizing systems programming, and the community benefits most from talks by intermediate learners who can bridge the gap between beginners and experts.

Julia Evans shares her wishlist for RustConf talks as a program committee member, focusing on talks that help people become better systems programmers. She suggests topics ranging from Rust compiler internals and library deep-dives to systems concepts like databases and filesystems, C interop best practices, and emerging Rust patterns. She emphasizes that speakers don't need to be expert Rust programmers, noting that intermediate-level speakers often give the most useful talks because they remember what learning felt like.

Key Insight: Computer networking fundamentals like TCP are both more approachable and more universally relevant than most programmers realize, and there's no reason to wait years before learning them.

Julia Evans appeared on the CodeNewbie podcast to discuss how the internet works, covering TCP, UDP, netcat, and other networking fundamentals. She notes that many programmers go years without learning about computer networking, despite these concepts being both interesting and approachable. The post promotes making systems-level knowledge accessible to beginners and shares related resources including her own zines and a recommended networking book.

Key Insight: Broad networking competence comes not from mastering a textbook but from accumulating dozens of small, concrete skills over time through curiosity and on-the-job problem solving.

Julia Evans catalogs the specific networking concepts and skills she's learned over roughly eight years, from basic HTTP requests to container networking. Rather than prescribing a learning path, she presents her knowledge as an accumulation of small, concrete skills picked up one at a time, mostly on the job. She lists tools she's found useful for networking tasks and describes her learning process as identifying small gaps, filling them, and repeating. The post argues that broad competence in networking comes from patient, incremental learning rather than any single course or resource.

Key Insight: eBPF uprobes combined with ELF parsing make it possible to build a working ltrace alternative in about 200 lines of Rust that solves the original tool's inability to attach to running processes and its performance overhead.

Julia Evans documents her weekend project building an ltrace clone using eBPF and bcc, written in Rust. The original ltrace tool traces library calls but has problems: it can't reliably attach to running processes and introduces significant performance overhead via ptrace. Her prototype solves both issues by using eBPF uprobes instead, parsing ELF files to find dynamically linked functions, then attaching eBPF probes to trace them. The whole thing is about 200 lines of Rust and successfully traces library calls on a running Firefox process.

Key Insight: A blog's real success is measured not by page views but by the meaningful conversations, genuine help, and human connections it creates between writer and readers.

Julia Evans argues that blog success should be measured by meaningful conversations and relationships rather than page views. She explains that blogging has been rewarding because it connects her with people working on similar problems who share tips and insights. She advocates for writing posts that genuinely help people, treating readers as real humans rather than an 'audience' consuming 'content.' She recommends tracking subscriber counts as a more meaningful metric than page views, and emphasizes that even small blogs with engaged readers can be deeply successful.

Key Insight: The transition from 'will this ever work?' to 'people are actually using this' is a surprising and gratifying milestone in building developer tools.

Julia Evans reports on weeks 6 and 7 of her sabbatical building rbspy, a Ruby profiler. The major milestone is that rbspy appears to actually work reliably, with no new bug reports in weeks. She added several new features including a 'report' subcommand for generating visualizations from raw data, a live summary display during profiling, and C function tracking in Ruby stacks. Her next goals are releasing a proper beta version and creating documentation.

Key Insight: Remote work succeeds not through isolation management but through deliberate overcommunication and intentional relationship-building within an organization that has genuine remote-first infrastructure.

Julia Evans reflects on four years of working remotely for a San Francisco company (Stripe) while living in Montreal. She addresses common concerns about remote work across three categories: mental health/productivity, basic job effectiveness, and career growth. She's been promoted multiple times, led major projects, and learned extensively from colleagues, attributing her success to working in a remote-friendly organization and being intentional about communication. She emphasizes that remote work requires deliberate relationship-building and overcommunication, but that it hasn't limited her career growth or ability to do meaningful work.

Key Insight: The most useful performance reviews replace generic praise with specific positive feedback about strengths, impact, and forward-looking goals — making invisible contributions visible and giving people actionable information about what they're doing well.

Julia Evans shares her approach to writing useful performance reviews, focusing on positive constructive feedback rather than generic praise. She outlines three main strategies: highlighting specific strengths, articulating the person's impact on the organization, and discussing forward-looking goals. She emphasizes that positive feedback should be specific and actionable, not just complimentary. She also argues that negative feedback should never appear for the first time in a performance review, and that investing significant time in writing thoughtful reviews is worthwhile.

Key Insight: Profiling tools that work from outside a process—requiring only a PID—can reveal surprising allocation patterns like implicit Range object creation that would otherwise go unnoticed.

Julia Evans reports on week 5 of building rbspy, a sampling CPU profiler for Ruby. The week's highlights include completing Mac support and experimenting with memory allocation profiling. She demonstrates a tool that tracks every Ruby memory allocation by function, filename, and line number, showing how it can reveal surprising allocation sources like Range object creation. She outlines upcoming work including Debian packaging, UI improvements beyond flamegraphs, and continued memory profiling experiments.

Key Insight: Rust's type system and performance make it a natural fit for building complex BPF tracing tools that outgrow Python's capabilities, while remaining just as approachable for simple cases.

Julia Evans announces an experimental Rust library (rust-bcc) for writing BPF tracing tools, porting the Go BCC library to Rust. She explains how the BPF Compiler Collection works — compiling C to eBPF bytecode, attaching it to kernel/userspace functions, and communicating results back via data structures like hashmaps and perf buffers. She argues that while Python BCC bindings work well for simple tools, Rust is better suited for more complex tracing experiments like streaming Ruby memory allocations. The post walks through two working examples (strlen counting and opensnoop) and demonstrates practical Ruby allocation tracking.

Key Insight: eBPF uprobes enable building magical debugging tools that can observe a running process's internals from the outside, without any code changes to the target program.

Julia Evans describes a one-day experiment building a tool that can spy on a running Ruby process's memory allocations using eBPF and uprobes, without modifying the target process. She attaches a small BPF program to Ruby's internal newobj_slowpath function to count allocations per class, using bcc's Python/C toolkit. To resolve class pointers into human-readable names, she writes a Rust program that copies the target process's memory maps into its own address space and calls rb_class2name directly. The result is a live-updating view of which Ruby classes are being allocated most frequently, achieved in a single day of hacking.

Key Insight: Building developer tools in Rust can lower the barrier to contribution enough that non-Rust programmers can meaningfully contribute, while the real complexity lies in platform-specific systems programming.

Julia Evans shares a week 4 progress update on rbspy, her Ruby profiler written in Rust. The project received external contributions including new flags and callgrind output format support, and she added container profiling support. Mac support proved harder than expected due to kernel bugs and memory map difficulties. She's considering building a reusable Rust visualization library and is excited about prototyping memory profiling capabilities.

Key Insight: A race condition in macOS High Sierra's task_for_pid can be triggered by a trivial 17-line C program, freezing system tools — demonstrating how subtle kernel bugs lurk in common system call sequences.

While building Mac support for rbspy (her Ruby profiler), Julia Evans accidentally discovered what appears to be a Mac kernel bug in High Sierra. Early adopters reported that rbspy was freezing their computers, which was mystifying since the program wasn't running as root. With help from the community, she narrowed it down to a race condition in the `task_for_pid` Mach function and reproduced it with a 17-line C program. The bug causes `ps` and Activity Monitor to stop working when `task_for_pid` is called on a process immediately after exec'ing it. The same bug was independently affecting htop users. Her workaround is simply to sleep for a few milliseconds before calling `task_for_pid`.

Key Insight: Linux's /proc filesystem approach to exposing process information, while seemingly janky, is dramatically more accessible than Mac's Mach kernel APIs for the same task—a difference measured in hours versus days of development effort.

Julia Evans documents her multi-day journey to read memory maps from a Mac process, something she needs for her Ruby profiler rbspy. On Linux this is trivial via /proc/PID/maps, but on Mac it requires diving into Mach kernel APIs. She first tried the vmmap command-line tool but found it unacceptably slow (2 seconds). She then wrote her own partial vmmap clone in Rust using the mach crate, achieving 15x faster performance in about 100 lines of code. She's still working on handling dynamically linked libraries stored in the 'dyld shared cache' and shares useful resources for others tackling the same problem.

Key Insight: Container support often reduces to understanding Linux namespaces — once you know the mount namespace is the issue, the fix is a few system calls, no container runtime interaction required.

Julia Evans explains how she added Linux container support to rbspy, her Ruby profiler, so it could profile Ruby programs running inside containers from the host machine. The core problem was that containerized processes have different mount namespaces, meaning the Ruby binaries visible in /proc/PID/maps don't exist on the host filesystem. The fix involved using setns() to temporarily switch to the target process's mount namespace to read the binaries, then switching back. After publishing, someone pointed out an even simpler approach: reading files through /proc/PID/root/ which provides the process's filesystem view directly. The whole change was about 50 lines of code, illustrating how understanding container fundamentals makes container support straightforward.

Key Insight: Writing an architecture document for others is one of the most effective refactoring strategies because explaining your code to an outsider forces you to confront the parts that don't make sense.

Julia Evans reports on week 3 of building rbspy, a Ruby profiler written in Rust. She released an alpha version, refactored the codebase from a single 1000-line file into well-organized modules, and implemented a clever testing strategy using core dumps. The refactoring was guided by writing an architecture document, which helped identify parts of the code that didn't make sense. She highlights contributions from collaborator Kamal and early community engagement, with plans for Mac support and container support next.

Key Insight: Building a profiler that works across many Ruby versions reveals surprising insights about the tools you profile, like how Rubocop's call stacks mirror the AST structure of the code it analyzes.

Julia Evans reports on the second week of building a Ruby profiler in Rust. She expanded version support to cover Ruby 1.9.1 through 2.5.0, added line number tracking, and improved error handling using Rust's failure crate. The highlight was successfully profiling Rubocop, a popular Ruby linter, which produced interesting flamegraph visualizations showing how Rubocop's call stacks mirror the AST structure of the code it analyzes. She plans to release a first experimental version in the coming week.

Key Insight: Even pausing a process with ptrace doesn't guarantee valid stack traces because the interpreter's internal stack can be in a transitional state, but for a sampling profiler, dropping the rare invalid trace is perfectly acceptable.

Evans explores whether her Ruby profiler should use ptrace to pause a Ruby process before collecting stack traces. She finds that not pausing yields about 1 error per 1000 traces, while pausing reduces errors to about 1 per 10,000 — but doesn't eliminate them entirely. The remaining errors occur when the Ruby interpreter is paused mid-execution of internal functions like vm_call_iseq_setup, where the stack is temporarily in an invalid state. She demonstrates how easy ptrace is to use from Rust, leveraging RAII to ensure processes are always restarted. She concludes that since it's a sampling profiler, dropping a few invalid traces is acceptable.

Key Insight: Rust's deliberate investment in compiler ergonomics has dramatically lowered the barrier to entry without compromising its core guarantees of safety and performance, but the community needs to actively market this improvement to overcome its reputation as a hard language.

Julia Evans reflects on how Rust has become significantly easier to use since she first tried it in 2013, attributing the improvement to deliberate compiler ergonomics work rather than her own skill growth. She demonstrates this with real compiler error messages that now suggest fixes directly, and highlights features like .clone() as pragmatic escape hatches for beginners. She praises Rust's crate ecosystem, Cargo, bindgen, and macros as tools that enabled her to build a Ruby profiler supporting 35 Ruby versions. For Rust's 2018 goals, she proposes a marketing push to tell people Rust is easier now, and clearer messaging on rust-lang.org about who Rust is for.

Key Insight: Resolving a symbol's runtime memory address requires navigating three layers of indirection — ASLR base addresses, dynamically loaded libraries, and ELF program header offsets — turning a seemingly simple lookup into a surprisingly complex task.

Julia Evans documents three specific problems she encountered when trying to resolve symbol addresses in memory while building a Ruby profiler. She explains how ASLR randomizes where binaries are loaded, how symbols may live in dynamically linked libraries rather than the main binary, and how the ELF program header's vaddr offset must be accounted for. She walks through each problem using concrete examples with readelf, /proc/PID/maps, and musl's dlsym source code. The post illustrates how a seemingly simple task — finding a symbol's memory address — requires navigating multiple layers of OS and binary format complexity.

Key Insight: Building a reliable profiler requires testing across a combinatorial explosion of environments, and the hardest part is making it work on stripped binaries where key symbols are missing.

Julia Evans shares her first weekly progress update on building a Ruby profiler during her open source sabbatical. The main challenges involve reliably extracting stack traces from Ruby processes across different OS and Ruby version combinations, especially when debug symbols are stripped. She set up Docker-based integration tests across multiple Linux distributions and made partial progress on profiling system Ruby without debug symbols by using educated guessing to find key memory addresses. She also discusses adopting the C4 open source contribution model after advice from a more experienced maintainer.

Key Insight: Calling C functions from gdb is far more complex than just jumping to an address — it requires carefully orchestrating stack frames, registers, instruction pointers, and trap instructions via ptrace.

Julia Evans investigates how gdb can call C functions in a running program, a capability she found surprisingly non-obvious. She walks through the mechanics: finding function addresses via the symbol table, creating stack frames, changing the instruction pointer, and setting trap instructions. Through experiments with strace and register inspection, she pieces together that gdb builds new stack frames on top of the existing stack, modifies the instruction pointer via ptrace, and places trap instructions in libc functions like longjmp to regain control. She concludes that the process is more complicated than it first appears and embraces having unanswered questions.

Key Insight: Meaningful inclusion work often isn't about inventing new ideas but about finding the time and willingness to execute on changes everyone already agrees should happen.

Julia Evans reflects on her 2017 professional year, covering her work shipping Kubernetes at Stripe, her growth as a tech lead, and her inclusion-focused organizational projects. She details revamping Stripe's engineering career ladder to be more objective and promoting 'Brag Documents' for tracking accomplishments. She also reviews her prolific output of 4 zines, 77 blog posts, and multiple keynotes, while planning a sabbatical to build Ruby profiling tools in 2018.

Key Insight: Automating a blog newsletter via RSS-to-email removes all ongoing effort while serving the large audience of readers who prefer email over RSS.

Julia Evans describes how she set up an automatic weekly email digest of her blog using Mailchimp's RSS-to-email feature. After initially dismissing the idea as too time-consuming, she realized it could be fully automated. She walks through the simple setup process, shares her custom HTML template, and discusses bugs she encountered. The newsletter grew to over 1,050 subscribers in about six months with zero ongoing effort from her.

Key Insight: A well-designed one-page cheat sheet can make a powerful but complex tool like perf immediately accessible, revealing capabilities that even experienced users might not know about.

Julia Evans shares a one-page perf cheat sheet she created as part of her upcoming perf zine. After struggling to explain perf concisely, she decided to make a centerfold reference covering basic and advanced command line arguments. The examples are sourced from Brendan Gregg's perf reference page. She highlights surprising capabilities like counting system calls by process with live updates and tracking sent network packets by process.

Key Insight: Rust's macro_rules! system is more approachable than its reputation suggests, and even simple textual substitution macros can solve real code generation problems like compiling the same logic against dozens of different type definitions.

Julia Evans describes writing her first Rust macro using macro_rules! to solve a practical problem: generating the same get_stack_trace function for 30 different Ruby versions in her Ruby profiling tool. Each Ruby version had structurally similar but distinct bindings, so she needed the same code compiled against different type definitions. She found the macro system surprisingly easy to use, going from zero knowledge to a working solution in under an hour. The macro let her avoid massive code duplication by parameterizing the Ruby version identifier.

Key Insight: Debugging unfamiliar bug classes systematically — narrowing scope, creating minimal reproductions, and writing up the problem for others — often leads you to the answer yourself while teaching you about the underlying systems.

Evans documents debugging a segfault in her Rust program that grabbed stack traces from Ruby processes. The bug stemmed from unsafe code that incorrectly used Vec::from_raw_parts, passing the wrong capacity and violating alignment requirements when casting a byte vector into a vector of structs. The segfault only occurred with jemalloc (not the system allocator), which made debugging confusing but ultimately revealed that jemalloc's stricter memory management was actually catching the bug earlier. She fixed it by creating a slice view of the byte vector instead of trying to transmute the vector itself, letting Rust handle deallocation of the original bytes.

Key Insight: Rust's bindgen tool makes C interop nearly effortless by auto-generating bindings from header files, enabling practical use cases like supporting dozens of Ruby versions in a profiler.

Julia Evans introduces bindgen, a Rust tool that automatically generates Rust bindings from C header files, framing it as essentially '#include for Rust.' She explains how bindgen fits into her vision of Rust as offering all the power of C with better tooling, and demonstrates its practical use by generating bindings for 33 different Ruby versions' internal header files for her Ruby profiler project. She highlights useful features like type whitelisting and automatic Debug derive annotations, and notes that while committing generated bindings works, the recommended approach is generating them at build time via build.rs.

Key Insight: Python's C API exposes far more internal VM structure than Ruby's, which relies on opaque types — making it fundamentally harder to build external profiling tools for Ruby.

Evans compares the public C API header files exposed by Ruby and Python, motivated by her work building a Ruby profiler similar to pyflame. She discovers that Python exposes far more structs and detailed type information than Ruby, whose API relies heavily on an opaque VALUE type. Both languages are technically embeddable, though Ruby's embedding story is less documented. The key finding is that Ruby's public headers don't expose the internal VM structs needed for external process profiling, unlike Python's headers which include things like PyThreadState.

Key Insight: Ruby's internal struct layouts change with every major version, making compile-time profiler approaches much harder than equivalent Python profilers and potentially requiring either DWARF parsing or pre-generated bindings for dozens of Ruby versions.

Evans investigates whether Ruby's internal struct definitions change frequently enough to complicate building a sampling profiler. She compares two approaches: compiling against Ruby header files at build time (like pyflame does for Python) versus parsing DWARF debug symbols at runtime (like gdb). After examining struct layouts across Ruby 2.1 through 2.5, she finds that unlike Python which only has three variations, Ruby's internal structs change with every major version release. She concludes that either DWARF-based decoding or committing headers for all ~40 minor Ruby versions are the viable paths forward.

Key Insight: CPU profilers are built from a surprisingly small set of OS and interpreter primitives—setitimer for sampling, interpreter hooks for tracing, and ptrace for out-of-process observation—and understanding these building blocks demystifies how any profiler works.

Julia Evans surveys the internals of 14 Ruby and Python CPU profilers to understand how they actually work under the hood. She identifies two fundamental categories—tracing profilers that record every function call via interpreter hooks, and sampling profilers that periodically snapshot the stack using setitimer signals. She demonstrates that all these profilers are built from a surprisingly small set of fundamental capabilities (setitimer, PyEval_SetProfile, rb_add_event_hook, ptrace). The post includes code examples showing how simple stack sampling can be, highlights tradeoffs like tracing overhead and setitimer's signal interference with syscalls, and notes her particular interest in pyflame's out-of-process ptrace approach for her own Ruby profiler project.

Key Insight: The biggest barrier to fixing performance problems isn't the complexity of the fix—it's that profiling tools are too hard to set up, so people never get the data they need to diagnose the problem.

Julia Evans announces a three-month sabbatical from Stripe to build better Ruby profiling tools through the Segment Open Source Fellowship. She's frustrated that Ruby and Python profiling requires choosing libraries, adding gems, and writing setup code, unlike C and Java where you can just attach to a running process. Her goal is to build a tool where you simply point at a PID and get useful profiling data immediately. She acknowledges this is new territory for her—she's used many profilers but never built one—and is both excited and nervous about the challenge.

Key Insight: Getting really good at programming is less about mastering specific technologies and more about developing meta-skills like asking great questions, reading source code, and systematic debugging.

Julia Evans announces her new free zine 'So you want to be a wizard,' which covers strategies for getting really good at programming. The zine includes topics like asking great questions, reading source code, debugging skills, and software design. She thanks the 200 people who paid for early access and notes that people have been giving it to newer developers. She also mentions redesigning her zines page and reflecting that she's now published 8 zines total, with 4 made in 2017 alone.

Key Insight: Understanding Rust references requires knowing that memory can live in three places, and the distinction between owned heap types and their reference counterparts is the key mental model for writing Rust structs.

Julia Evans explores what references mean in Rust by starting from a simple struct that won't compile due to a missing lifetime parameter. She explains that Rust needs to know at compile time where memory lives (heap, stack, or data segment) to determine when to free it. The post walks through two solutions: using Box<T> for heap-allocated data, or adding lifetime parameters for references to existing memory. She identifies the relationship between owned heap types (Vec<T>, String, Box<T>) and their reference counterparts (&[T], &str, &T) as a key insight for understanding Rust. The post concludes by observing that most real-world Rust structs don't use lifetimes, based on examining ripgrep's source code.

Key Insight: Releasing imperfect work into the world is better than keeping it polished but unseen on your laptop.

Julia Evans announces a small zine compiling 24 Linux comics she drew the previous November. She formatted them into a printable booklet after people kept asking her to write a book, deciding to release something imperfect rather than keep it unpublished. She also promotes her new 'So You Want to Be a Wizard' zine about learning hard things, and shares the Python script she uses to format zines into printable booklets.

Key Insight: Linux group permissions are checked against group IDs baked into each process at creation time, not looked up dynamically from /etc/group, which is why adding a user to a group doesn't affect already-running processes.

Julia Evans discovers that Linux group membership checks don't work the way most people assume. Rather than checking /etc/group at runtime, Linux stores group IDs as attributes directly on each process. This means adding a user to a group doesn't affect already-running processes — you need to start a new login session for the changes to take effect. The post walks through the discovery, demonstrates the behavior with concrete examples, and explains how the login process initializes groups via initgroups().

Key Insight: The biggest barrier to building fun small web projects isn't coding skill but deployment friction, and tools that eliminate setup make programming feel joyful and democratic again.

Julia Evans enthusiastically introduces Glitch as a platform for building small JavaScript web projects with zero setup. The platform auto-reloads your backend code as you type, making it feel like the old days of editing PHP directly on a server. She highlights the 'remix' feature that lets you fork and modify example apps in clicks, and shares her own app for turning off Twitter retweets. The post argues that removing friction from deployment makes programming feel fun again, even if it's not a 'serious development practice.'

Key Insight: The biggest barrier to learning programming tools isn't the concepts—it's the installation, and Binder eliminates that friction entirely by making notebooks instantly interactive in the browser.

Julia Evans discovers Binder, a free tool that lets anyone host interactive Jupyter notebooks online. She explains the progression from static notebook sharing (GitHub, nbviewer) to fully interactive environments, drawing on her experience running Python data science workshops where installation issues consumed valuable time. She highlights how Binder, built on JupyterHub and Kubernetes, eliminates the setup barrier entirely by letting users run notebooks directly in the browser.

Key Insight: Kubernetes networking is operationally complex with many failure modes, and operators should minimize moving parts and think critically about each component rather than adopting the full stack uncritically.

Julia Evans explores the operational challenges of running Kubernetes networking in production, arguing that while there's plenty of guidance on setup, there's far less on ongoing operation. She walks through the key networking components—overlay networks, kube-proxy, kube-dns, and ingress—detailing specific failure modes she's encountered or anticipates. She emphasizes that networking software is inherently hard to operate because it depends heavily on Linux kernel configuration, requirements change over time, and debugging requires deep expertise. Her core advice is to minimize moving parts, think critically about architecture rather than blindly following what Google does, and invest heavily in understanding each component.

Key Insight: Kubernetes' real power comes from its stateless, etcd-backed architecture that provides a unified API for all cluster state, making it both a useful platform and a foundation for building custom distributed systems — but operating it well requires understanding the layers beneath its abstractions.

Julia Evans explains why Kubernetes is interesting after three months of working with it full-time, deliberately avoiding buzzwords. She highlights that Kubernetes lets developers deploy services without setting up servers, provides unified visibility into all running applications through etcd, and continues running existing code even when its own components fail. She argues that Kubernetes' stateless design makes it resilient to bugs in its own core components, and that its constrained model makes it a good platform for building custom distributed systems. She concludes that while Kubernetes enables amazing things, it requires deep understanding of underlying systems like Linux networking and isn't easy to operate.

Key Insight: Understanding what a pprof file actually contains — stack traces stored as protobuf with metadata — demystifies the tool and makes it easier to reason about what profiling data means and how to use it effectively.

Evans explores Go's pprof profiling tool, focusing on heap profiling for debugging memory problems. She walks through the basics of setting up pprof, collecting profiles, and analyzing them with go tool pprof. The core of the post deconstructs the actual protobuf format of a pprof file to understand what data it contains. She explains key distinctions like alloc_space vs inuse_space and how the heap profiler tracks both allocations and frees. She also discovers through strace that pprof resolves function names by reading the original binary from disk, not from the profile file itself.

Key Insight: Deliberately pursuing topics that feel hard or scary, combined with daily writing and demoable projects, can produce lasting learning and career momentum in a compressed timeframe.

Julia Evans revisits how she spent her 46 days at the Recurse Center four years earlier, breaking down her time across projects like a TCP stack in Python, a mini OS in Rust, and a gzip implementation in Julia. She averaged 50-100 lines of code per day and deliberately avoided comfortable topics like web development, focusing instead on things that seemed hard or scary like networking, operating systems, and security. She spent 1-2 hours daily blogging as a deliberate 'media strategy' to land a good job afterward, which worked. The post reflects on what made the experience valuable: the people, the debugging culture, and discovering unknown-unknowns like strace and netcat that still excite her years later.

Key Insight: Vim's built-in :mksession command lets you save and restore your entire editing session without any plugins.

Julia Evans shares her discovery of vim's built-in session feature, which lets you save and restore all open files and editor state. The feature requires no plugins—just :mksession to save and :source to restore. She drew a comic explaining the workflow and links to three plugins that extend session functionality.

Key Insight: Treaps offer the same performance as red-black trees for balanced BST operations but with dramatically simpler algorithms for insertion and deletion.

Julia Evans introduces the treap, a randomized data structure that combines a binary search tree with a heap to create a balanced BST. She explains it's useful for implementing ordered maps (like C++'s std::map) and notes that while std::map typically uses red-black trees, treaps perform equivalently in expected value with much simpler insert/delete algorithms. The post is a quick visual sketch format with hand-drawn illustrations explaining how treaps work.

Key Insight: Linux provides layered tools for packet drop diagnosis: `netstat -i` tells you if drops are happening, `dropwatch` tells you which kernel function is responsible, and `perf` gives you full stack traces.

Julia Evans explores how to detect and diagnose packet drops on a Linux server. She starts with the basic diagnostic of using `netstat -i` to see if packets are being dropped on any interface, then introduces `dropwatch`, a tool that identifies which kernel function is responsible for dropping packets. She also covers using `perf` to monitor the `kfree_skb` kernel event for packet drop debugging. The post is framed as personal notes for future reference rather than a battle-tested guide.

Key Insight: Debugging opaque kernel networking problems becomes tractable when you combine strace to capture raw netlink messages with pyroute2 to decode them into human-readable route attributes.

Evans walks through debugging a Kubernetes networking problem where a container network backend couldn't delete routes. She discovers netlink, the Linux kernel interface for managing routes and network configuration, and demonstrates how to use strace to capture netlink messages, pyroute2 to decode them, and tcpdump/wireshark as alternatives. The root cause turned out to be an invalid RTA_OIF field set to 0 (not a valid network interface ID) in the netlink message. She also covers other debugging tools like ip monitor and nltrace, evaluating their usefulness for this specific problem.

Key Insight: Network interfaces are the gateway into the Linux network stack — without one, packets never enter the stack at all, and the route table determines which interface each packet gets assigned to.

Julia Evans explores what network interfaces actually are by running hands-on experiments with Linux network namespaces. She creates a namespace with no interfaces to show that servers can bind to ports but can't receive packets without at least one interface. She then investigates why packets appear multiple times in tcpdump when containers are involved, tracing the answer to how routing assigns packets to specific network devices. The post builds up a working mental model of interfaces through experimentation rather than definitions.

Key Insight: Sharing your work plans openly is scary because failure becomes public, but the coordination benefits and feedback you get far outweigh the comfort of hiding until you're done.

Evans argues that publicly announcing work plans and projects, while scary, is far more effective than hiding until things are finished. She identifies the fear of failure and changing plans as reasons people avoid sharing, but counters that transparency helps teams coordinate, surfaces problems early, and creates organizational momentum. She acknowledges legitimate reasons to keep early-stage ideas private, particularly in cultures with excessive 'stop energy.' The post concludes that the benefits of open communication about plans far outweigh the risks of occasional failure or plan changes.

Key Insight: Contributing to open source isn't technically harder than regular work—it's the social and contextual challenges that make it difficult, and those can be addressed with specific tactics like smaller PRs, detailed descriptions, and extensive code reading.

Julia Evans shares her evolving approach to contributing to open source projects at work, moving from passively accepting bugs to actively fixing them. She breaks down why open source contributions feel harder than regular work despite requiring the same core engineering skills—it's the social dynamics, unfamiliar conventions, and lack of context that create friction. She offers practical tactics like starting with tiny PRs, writing detailed descriptions, reading lots of code, and breaking changes into smaller pieces. She concludes that open source contribution is a valuable work skill but not something she'd do strictly for fun in her spare time.

Key Insight: The most effective professional learning happens during work hours through deliberate choices — picking projects for their learning potential, reading source code and pull requests, and treating time spent understanding systems as a core part of the job rather than extra work.

Julia Evans explores how programmers can invest in learning without spending significant personal time outside of work. She argues that great programmers don't necessarily code after hours, and shares practical strategies gathered from Twitter for learning on the job. The strategies range from choosing projects for their learning potential, to reading pull requests and source code, to watching how senior colleagues operate. She makes a strong case that learning during work hours is not only acceptable but is actually part of the job, and that spending time understanding things deeply pays dividends over time.

Key Insight: Kubernetes' certificate infrastructure is more flexible than documentation suggests—its many CA settings are independent of each other, so you can map your own PKI requirements onto a cluster rather than being forced into a single-root-CA model.

This post demystifies the overwhelming number of certificate and key command line arguments in Kubernetes by breaking them down into distinct, independent certificate authorities. Evans explains that contrary to common documentation, a Kubernetes cluster does not require a single root CA—you can use multiple independent CAs for different purposes. She walks through five main CA categories: the API server's TLS certificate, client certificate authentication, authenticating proxy certificates, service account keys (which uniquely don't use CAs at all), and kubelet certificates. The post aims to help readers map their own PKI requirements onto Kubernetes' flexible certificate infrastructure.

Key Insight: Practices from large tech companies can contain genuinely useful ideas, but they need to be evaluated critically against your own team's scale rather than adopted as gospel.

Julia Evans discusses three software development practices she's encountered while working with Kubernetes that originate from Google's internal culture. She covers cherry-picking commits for release management, writing shell scripts in Go instead of bash, and using services instead of scripts for infrastructure automation. While she finds the first two practices genuinely useful at any scale, she's more skeptical about the service-oriented approach for smaller teams. She concludes with a broader point about being critical of practices from large companies, noting that what works for 50,000 engineers may not suit a team of 20.

Key Insight: Optimizing distributed systems for performance often introduces subtle failure modes that simpler but slower designs would avoid, and understanding these tradeoffs requires reading the actual source code.

Evans investigates how the Kubernetes scheduler works internally after encountering a bug where pods got stuck in Pending state forever. She traces through the scheduler's source code to discover it uses an event-driven queue system rather than a simple polling loop, which is an optimization for performance at scale. The bug turned out to be a missing error handler call that prevented failed pods from being re-queued. She explores the concept of Kubernetes 'informers' and the deliberate design decision not to resync the scheduler, since maintainers prefer correctness bugs to be surfaced rather than hidden. The post doubles as a demonstration of how to reverse-engineer a complex system's internals by reading source code with specific questions in mind.

Key Insight: Complex technical topics like Linux tracing can be distilled into short, visual zine formats to make them more accessible.

Julia Evans announces a new zine about Linux tracing tools, created as a companion to a previous blog post on the topic. The zine is 12 pages and available in both a digital reading version and a print version. It's a characteristically quick, visual format for making Linux tracing concepts accessible.

Key Insight: The confusing Linux tracing ecosystem becomes navigable once you understand its three layers: data sources (where tracing data originates), collection mechanisms (how data leaves the kernel), and frontends (the tools you actually use).

Julia Evans breaks down the confusing landscape of Linux tracing systems into three clear categories: data sources (kprobes, uprobes, tracepoints, USDT probes), mechanisms for collecting data (ftrace, perf_events, eBPF, systemtap, LTTng), and user-facing frontends (perf, bcc, trace-cmd). She explains how probes dynamically modify assembly at runtime while tracepoints are compiled into programs, and walks through how each mechanism extracts data from the kernel. The post concludes that eBPF is the most promising direction for newer kernels, while perf trace remains the simplest tool for tracing system calls.

Key Insight: eBPF transforms BPF from a packet filtering mechanism into a general-purpose framework for safely running custom programs in the kernel, enabling tracing and monitoring of virtually any kernel or userspace function.

Julia Evans shares notes from a Papers We Love talk about the original BPF paper and modern eBPF. She explains how BPF solved packet filtering by moving filtering logic into the kernel and using control flow graphs instead of inefficient tree evaluation. The post traces BPF's evolution from packet filtering to eBPF, which can attach programs to kernel functions, userspace functions, network cards, and more. She walks through how tcpdump converts filter expressions into BPF bytecode that gets JIT-compiled to native code. The piece highlights eBPF's ability to hook into virtually any kernel or userspace function through kprobes and uprobes, making it a powerful tracing and monitoring tool.

Key Insight: You don't need travel or production polish to share technical knowledge — quick, unedited screencasts from home can be just as effective as conference talks.

Julia Evans experiments with recording short screencasts as an alternative to traveling for conference talks. She records three 2-minute demos covering /proc, strace (applied to Python's asyncio library), and tcpdump (for DNS queries), all done in a single take with no editing. The post is a brief announcement sharing the results and noting how surprisingly easy the process was. She hints she may make more in the future.

Key Insight: Reactive frameworks like Vue.js simplify small interactive programs by letting you focus on data structures rather than manually wiring up DOM updates.

Julia Evans shares a tiny whack-a-mole game she built while learning Vue.js. The game uses a simple data structure (a grid of 1s and 0s) with Vue.js handling the DOM updates automatically. She contrasts this with her usual jQuery approach, which she describes as producing disastrous callback-heavy code. She finds Vue.js appealing for small interactive programs because it's simpler than React for her needs.

Key Insight: Being on call creates an irreplaceable feedback loop between writing code and seeing how it fails, building engineering intuition about real failure modes, monitoring needs, and system design that you can't get any other way.

Evans reframes on-call duty from a dreaded obligation into a powerful learning opportunity for software engineers. Drawing from a Charity Majors Twitter thread, she catalogs the specific skills being on call teaches: understanding real failure modes, building monitoring instincts, learning system limitations, developing judgment about severity, and designing for reliability. She defines 'on call' broadly as being responsible for your code when it breaks, not necessarily midnight pages. The post argues that the feedback loop of writing code and watching it fail in production is uniquely valuable for building engineering intuition that carries forward to every future system you design.

Key Insight: Text generation, parsing, and word vectors are three accessible building blocks that make it easy to start experimenting with natural language programming.

Julia Evans shares her excitement about NLP tutorials written by Allison Parrish, a creator of computer-generated language art. The tutorials cover text generation with Tracery, text parsing with spaCy, and understanding word vectors. Evans walks through each tutorial, highlighting practical examples like identifying characters in Pride and Prejudice and averaging word vectors. She frames these three tools as accessible building blocks for anyone wanting to experiment with text programming.

Key Insight: Database storage engines split into two fundamental approaches — log-structured (fast sequential writes) and page-oriented (fast random reads) — and understanding this distinction unlocks comprehension of real-world database design tradeoffs.

Evans shares her excitement about reading Martin Kleppmann's 'Designing Data-Intensive Applications' and explains the concept of log-structured storage she learned from it. She breaks down two fundamental approaches to database storage: log-structured engines (which append writes sequentially for faster writes) and page-oriented engines like B-trees (which update data in place). She illustrates the concept with a two-line bash script that functions as the 'world's simplest database.' She also clarifies the distinction between log-structured storage and write-ahead logs, explaining how WALs protect against data loss during crashes.

Key Insight: iptables is less intimidating once you know how to dump all rules with iptables-save and find better documentation than the man page for understanding concepts like MASQUERADE.

Julia Evans breaks down iptables basics after being motivated to learn them through Kubernetes work. She explains how to view all iptables rules (requiring four separate commands or using iptables-save), walks through the nat and filter tables, and dissects a specific MASQUERADE rule from the Kubernetes source code. She highlights that the iptables man page is incomplete and points to better resources like iptables.info for understanding concepts like SNAT and MASQUERADE. The post demystifies how network address translation works for container networking, drawing an analogy to home networks.

Key Insight: Kubernetes becomes comprehensible once you understand that every component is stateless and all coordination happens through watching etcd for work, doing it, and writing the result back.

Julia Evans shares her journey learning Kubernetes over six months, moving from confused reader to someone actually setting up a cluster. She explains the architecture by breaking it down into components: the kubelet runs containers on nodes, the API server writes to etcd, the scheduler assigns pods to machines, and the controller manager handles higher-level abstractions. The central insight is that etcd is Kubernetes' brain — every component watches etcd for work, does it, and writes state back. She emphasizes that understanding fundamental networking concepts is essential for debugging Kubernetes issues.

Key Insight: epoll outperforms select and poll because it asks the kernel to persistently track file descriptors rather than re-specifying them on every call, making it the foundation of modern event-driven servers on Linux.

Julia Evans explores the Linux system calls select, poll, and epoll for monitoring multiple file descriptors, drawing from The Linux Programming Interface book. She explains that select and poll share the same underlying kernel code but differ in their interfaces, with poll performing better for sparse file descriptor sets. The key insight is that epoll dramatically outperforms both by having the kernel remember which file descriptors to monitor, avoiding redundant work on each call. She walks through epoll's three-step API (create, ctl, wait) and shows real strace output from a Go program using epoll for DNS queries. The post also covers level-triggered vs edge-triggered notifications and links to libraries like libuv, libev, and Go's runtime that use epoll under the hood.

Key Insight: Handwritten books have a rich tradition across many fields because the format naturally encourages the seamless integration of visual elements like diagrams and illustrations that would be difficult to achieve with conventional typesetting.

Julia Evans compiled a collection of handwritten books after asking Twitter whether it would be too weird to write a fully handwritten book. She gathered examples spanning technical topics like electronics, computing, and math, as well as cookbooks, self-help, and hiking guides. The examples share a common thread: handwriting enables seamless integration of diagrams, illustrations, and visual elements without fighting typesetting tools. The collection ranges from Scott McCloud's Understanding Comics to Dijkstra's EWD manuscripts to the Moosewood Cookbook, demonstrating that handwritten books have a rich history across many genres.

Key Insight: Charging for early access to educational technical content can validate its value and fund future creative work, while still making it freely available to everyone.

Julia Evans announces the public release of her 'Let's learn tcpdump' zine, which was previously available as a $10 early access experiment. She shares the results of the paid early access model: 190 people bought it for a total of $1,970, including one enterprise license at $100. The positive response and revenue validated that people value her educational content enough to pay for it. She credits collaborators who helped with lettering, review, and illustration.

Key Insight: Small configuration decisions made years apart—pre-gzipping files for bandwidth savings, adding Cloudflare, enabling HSTS—can interact in unexpected ways that create debugging puzzles where no single component is clearly at fault.

Julia Evans debugs why her website was displaying garbled content by tracing the issue through HTTP headers. She discovers the response was gzipped but missing the Content-Encoding: gzip header, which meant browsers couldn't decode it. The root cause was an old Apache .htaccess rule she'd written years ago to always serve gzipped content regardless of what the client requested, which interacted badly with Cloudflare's caching. She fixed it by removing the .htaccess rule and clearing the Cloudflare cache, though she never fully determined why the Content-Encoding header was being stripped.

Key Insight: Complex computer science concepts don't need to be dumbed down for young or non-traditional audiences — they just need to be explained clearly and presented in an inclusive, welcoming way.

Julia Evans recommends Bubblesort Zines by Amy Wibowo, a collection of computer science zines aimed at teenage girls that cover topics like sorting algorithms, networking, cryptography, and image processing. She highlights the 'How does the internet' zine for its impressive range of topics and the newly released 'Pixel Perfect' zine on image processing. Evans emphasizes that these zines explain complex concepts clearly without dumbing them down, making CS accessible to audiences who might not see themselves in traditional CS materials. She draws from her own experience of wanting to learn programming as a teenager but not knowing where to start.

Key Insight: HSTS and the preload list exist because redirecting HTTP to HTTPS still leaves a window where the first request is insecure, and closing that window requires a permanent commitment to serving HTTPS.

Julia Evans explains the Strict-Transport-Security (HSTS) header after enabling it on her own site. She walks through why simply redirecting HTTP to HTTPS isn't sufficient—browsers can still leak cookies or be tricked by malicious networks on that first HTTP request. She demonstrates how GitHub uses the HSTS header and the browser preload list to ensure HTTPS is always used. She also covers the practical tradeoff: once you enable HSTS, you're committing to serving HTTPS indefinitely, since browsers will refuse to load the HTTP version.

Key Insight: tcpdump seems overwhelming at first, but the core command line arguments you actually need fit on a couple of pages, making it far more approachable than its man page suggests.

Julia Evans announces a new 12-page zine about tcpdump, a network packet analysis tool she once found intimidating but now uses comfortably. She describes her journey from being confused by tcpdump's output and man page to feeling confident using it for networking problems. The zine is offered as an early-access purchase for $10, with a free release planned later. She also cross-promotes her earlier networking fundamentals zine and shares preview pages showing that only a few command line arguments are really essential.

Key Insight: Acting surprised when someone doesn't know something—whether the surprise is genuine or performed—shuts down learning, but redirecting that energy into excitement creates a moment of connection and teaching.

Evans champions the 'no feigning surprise' social rule from the Recurse Center, which says you shouldn't act shocked when someone doesn't know something. She argues this applies even when you're genuinely surprised, because the negative effect on the other person is the same regardless of intent. She suggests transmuting surprise into excitement as a positive alternative. The post uses personal anecdotes to illustrate both sides—being the teacher who doesn't judge and the learner who isn't embarrassed by gaps in knowledge.

Key Insight: The best developer tools are the ones that work brilliantly out of the box without requiring configuration, and fish's context-aware autosuggestions from shell history are a killer feature worth switching shells for.

Julia Evans explains why she switched to the fish shell three years ago and why she'll never go back to bash. The post highlights fish's killer feature: context-aware autosuggestions from shell history that adapt based on your current directory. She also praises fish's syntax highlighting for catching typos, its more readable loop syntax, and its simpler history search. The core appeal is that fish provides excellent defaults with zero configuration, unlike zsh which requires extensive setup to match. She acknowledges fish isn't POSIX compliant but considers it a minor inconvenience since you can just type 'bash' when needed.

Key Insight: Nonparametric statistical methods like bootstrapping and shuffling are both more practical and more intuitive for programmers because they replace assumption-laden formulas with computation.

Julia Evans argues that traditional statistics education focused on formulas and tests like chi-squared is less useful for programmers than computational, nonparametric methods. She points out that classical statistical tests make assumptions (like normal distribution) that often don't hold for real-world data. Instead, methods like bootstrapping and shuffling let you get statistically valid answers by writing programs rather than applying formulas. The post curates a list of recommended resources for learning statistics through programming, including talks, free textbooks, and books by Allen Downey and Larry Wasserman.

Key Insight: The most effective way for individual engineers to change workplace culture is to start by doing the thing themselves — creating a template, writing a document, setting a standard — which makes it far easier for others to follow.

Julia Evans argues that individual software engineers can make meaningful culture changes at work, even without management authority. She shares several examples from her own experience: creating interview rubrics that became company-wide standards, publishing 'what to expect' guides for candidates, writing more inclusive job descriptions, and improving team documentation. Her partner's experience shows how writing one detailed postmortem can seed an entire postmortem culture. She acknowledges that some changes require top-down authority, but maintains that leading by example on focused, small efforts can compound into significant improvements.

Key Insight: Deep specialization at a conference — where every attendee knows far more than you about the topic — creates an exceptionally valuable learning environment precisely because it exposes you to things you've never heard of.

Julia Evans recaps the final day of the netdev 2.1 conference, covering nftables (the replacement for iptables), IoT mesh networking, and a networking daemon for data center reliability. She highlights nftables' unified approach to packet classification and its nearly 2x performance improvement over iptables. The post also explores low-power networking protocols like 6LoWPAN and the surprising constraints of devices that can only send 100 bytes per day. She closes with reflections on the conference culture, appreciating its deep specialization and friendly community while noting the lack of women's t-shirt sizes.

Key Insight: Modern Linux networking is evolving rapidly with tools like BBR, XDP, and tc that challenge fundamental assumptions about congestion control, packet processing, and hardware programmability, but benchmarking these advances reliably remains surprisingly difficult.

Julia Evans shares her notes from the second day of the netdev conference, covering several networking topics including Google's BBR TCP congestion control algorithm, running the Linux kernel networking stack in userspace, XDP/eBPF capabilities and myths, and the surprisingly powerful tc tool. She explains how BBR differs from traditional TCP congestion algorithms by estimating bandwidth rather than just reacting to packet loss, and documents interesting benchmarking challenges encountered when running BBR in the Linux Kernel Library project. The post also covers practical details about programming network cards with tc and the importance of rigorous network benchmarking.

Key Insight: XDP and eBPF make custom kernel-level packet filtering both fast and accessible, with a complete DDoS filter implementable in just 270 lines of C that outperforms iptables by more than 2x.

Evans recaps a netdev conference tutorial on building a DDoS prevention filter using eBPF and XDP. The tutorial walked through a complete working example: an XDP program running in the kernel that filters packets by IP address, a userspace command-line tool for managing the blacklist, and per-CPU counters tracking blocked packets. She explains the key concepts—eBPF as a kernel virtual machine without loops, XDP as an early-stage packet filter running at the network driver level, and BPF maps as the communication mechanism between kernel and userspace. The XDP approach achieved 9.7 million packets/second compared to iptables' 4.5 million, demonstrating both the performance and flexibility advantages.

Key Insight: XDP represents a fundamental shift in Linux networking by allowing packet filtering to happen as early as possible in the stack, and companies like Cloudflare and Facebook are already using it in production for massive performance gains in DDoS mitigation and load balancing.

Julia Evans shares her stream-of-consciousness notes from day 1 of netdev, a Linux networking conference. The talks covered XDP (eXpress Data Path) usage at Cloudflare and Facebook for DDoS mitigation and load balancing, showing how XDP provides dramatic performance improvements over iptables and IPVS. She documents her learning process openly, including things she didn't understand, conversations at lunch about RX queues and tc capabilities, and a networking performance workshop covering memory allocation bottlenecks and AF_PACKETv4. The post captures how much new material she absorbed despite much of it going over her head.

Key Insight: Deliberately adding network latency with tc reveals that HTTPS with HTTP/2 handles high-latency conditions dramatically better than plain HTTP, making a compelling performance case for HTTPS adoption.

Julia Evans introduces tc (traffic control), a Linux tool for deliberately adding network latency to test how applications behave under slow conditions. She demonstrates adding 500ms delay to her laptop's wireless interface and discovers something surprising: HTTPS pages degraded far less than HTTP pages under high latency, because HTTP/2 (which requires HTTPS) enables parallel requests. She also notes tc's usefulness for testing server behavior, such as observing how a DNS client handles a slow DNS server, while warning about the side effects of slowing down all packets on a machine you're SSHed into.

Key Insight: Bash is full of non-obvious gotchas that bite even experienced users, but a few defensive habits—quoting variables, using set -eu, and running shellcheck—make it manageable without needing to master every quirk.

Julia Evans catalogs the quirks and safety tips every bash scripter should know, from variable assignment pitfalls to the critical importance of `set -eu` flags. She covers basic constructs like for loops, if statements, and functions, but focuses on the non-obvious gotchas that trip people up even after years of use. The post emphasizes defensive practices: always quoting variables, using `set -e` to stop on errors, `set -u` to catch unset variables, and running shellcheck to lint scripts. She advocates for keeping bash scripts simple rather than fighting the language's weirdness.

Key Insight: Being honest about what you don't know lets you write outside your comfort zone and paradoxically builds more trust with readers than projecting false expertise.

Julia Evans shares the blogging principles that help her consistently publish content on the internet. She advocates for honesty about knowledge gaps, writing for a past version of yourself, staying positive, sticking to personal experience rather than prescriptive advice, and writing about things right after learning them. She argues that admitting what you don't know actually builds trust rather than undermining credibility. The post emphasizes that not every post needs to be popular and that frequently posting small interesting things matters more than optimizing for page views.

Key Insight: ftrace lets you trace Linux kernel function calls with zero overhead when disabled, and trace-cmd makes it practical to use for understanding what the kernel is actually doing when debugging system-level problems.

The post introduces ftrace, a Linux kernel feature for tracing kernel function calls that has been available since kernel 2.6. Evans walks through practical examples using trace-cmd, a friendlier frontend to ftrace's filesystem-based interface, demonstrating how to trace individual functions like do_page_fault, all functions for a specific process, and kernel events like scheduling switches. She explains that ftrace works by dynamically inserting tracing instructions into kernel functions at runtime using pre-compiled NOP slots, meaning zero overhead when not in use. The post also points readers to Brendan Gregg's perf-tools and kernelshark for more accessible ways to leverage ftrace.

Key Insight: Writing a fact-based inventory of your career accomplishments reveals growth patterns, exposes gaps between values and actions, and combats the feeling that you haven't done enough.

Julia Evans describes her experiment of writing down everything she accomplished over three years at her job. She used a structured format covering projects, design work, collaboration, and non-job tasks, spending half a day producing 13 pages of details. The exercise revealed patterns in her growth, like the shift from guided work to independence and the late adoption of design documents. She found the fact-based approach more useful than self-evaluation, and recommends the practice for career reflection.

Key Insight: Cgroup memory limits create per-container memory boundaries that can cause swapping and OOM kills even when the host machine has free memory, breaking the intuitive assumption that available RAM prevents memory pressure.

Julia investigates a puzzling production issue where machines with plenty of free memory were swapping and processes were being OOM killed. After trying swappiness tuning and disabling swap without success, she discovers that cgroup memory limits were the root cause — processes were constrained to less memory than the machine had available. The post walks through her debugging process of elimination and the key insight from reading dmesg output. She reflects on whether swap makes sense on production servers at all, given its unpredictable interactions with cgroups.

Key Insight: The most effective recruiting isn't about finding 'the best' people — it's about finding people for whom your specific job represents a genuinely exciting opportunity to grow.

Julia Evans reflects on what makes job opportunities exciting to candidates, arguing that the best recruiting strategy is finding people for whom your specific job would be a genuinely exciting opportunity. She observes that generic recruiting emails about roles similar to what she's already done are far less compelling than opportunities to do something new and different. She challenges the notion that hiring is about finding 'the best engineers' for 'the best companies,' arguing instead that great hiring is about matching people with roles that represent genuine growth opportunities. She references Dan Luu's observation that competing for the same talent pool as everyone else is expensive and ineffective.

Key Insight: Neural style transfer works by mathematically defining 'content' and 'style' as different representations within an image recognition network, then using gradient descent to find an image matching both targets.

Evans breaks down the paper 'A Neural Algorithm of Artistic Style' to understand how neural network style transfer actually works mathematically. She explains that the technique repurposes an object recognition network (VGG) by defining 'content' as the activation vector at a deep layer and 'style' as the Gram matrix of feature maps at earlier layers. The process then uses gradient descent to find an image that minimizes the difference between the target content and target style. She's honest about not fully understanding the Gram matrix intuition but finds the approach exciting because it suggests other weird art ideas might also work if formulated mathematically.

Key Insight: The POSIX specification deliberately treats paths with exactly two leading slashes as a special case with implementation-defined behavior, creating a surprising inconsistency where 'cd //' and 'cd ///' behave differently.

Julia Evans explores a quirky Unix behavior where 'cd //' preserves the double slash in the path while 'cd ///' collapses to a single slash. She discovers through the POSIX specification that three or more leading slashes are collapsed to one, but two leading slashes are explicitly left as implementation-defined. The post uses stat to confirm that / and // point to the same inode, deepening the mystery of why the distinction exists in the spec at all. She ends with an open question about the historical reason, later updated with a Stack Overflow link explaining the behavior.

Key Insight: While UDP theoretically allows packets up to 65535 bytes, Ethernet's 1500-byte MTU is the practical limit on the internet, and packets exceeding it will be dropped or fragmented.

The post explores the question of maximum packet sizes on the internet, starting from UDP's theoretical 65535-byte limit and explaining why real-world packets are constrained to 1500 bytes by Ethernet's Maximum Transmission Unit (MTU). Evans walks through what happens when packets exceed the MTU (they get dropped or fragmented), how Path MTU Discovery works using ICMP packets and the 'don't fragment' flag, and notes that jumbo frames allow larger packets in controlled environments like datacenters.

Key Insight: The most compelling ML art tools are ones that augment your creative process while preserving your sense of authorship, letting you make things that still feel like yours.

Julia Evans shares her excitement about PaintsChainer, a machine learning tool that automatically colors black and white images. She experiments with it on her zine artwork and traces photos, exploring how ML art tools can augment creative ability. The post reflects on what makes certain AI art tools compelling versus others, concluding that the best tools are ones where the output still feels like something you personally made. She contrasts PaintsChainer favorably with deepart.io, noting that creative control over the process matters more than raw impressiveness.

Key Insight: SSL certificates become far less mysterious when you download one, parse it with openssl, and understand the chain of trust from root CAs to the domain you're visiting.

Julia Evans dissects an SSL/TLS certificate by downloading one from mail.google.com using openssl and walking through its components. She explains the key fields including domain names, expiry dates, public keys, and signatures, then covers how certificate signing chains work and why certificate authorities matter. The post also introduces certificate transparency as a mechanism to detect rogue certificates. She concludes by acknowledging that SSL configuration is complex but that tools like Let's Encrypt make it accessible.

Key Insight: Instead of searching for the theoretically best tool, evaluate whether your current system is too expensive, too hard to operate, or failing at its job — if it passes all three, move on to real problems.

Evans argues that obsessing over choosing the 'best' software tool is paralyzing and largely unproductive. She proposes an alternative framework borrowed from capacity planning: only change something if it's too expensive, too hard to operate, or not doing its job. If a system passes those three tests, it doesn't matter whether something theoretically better exists. She suggests that after confirming something will work, you should just pick whatever the team feels best about, since building something that works well enough is already hard enough.

Key Insight: Physical zines can replace slides as a talk format, taking comparable preparation time while creating a more memorable and enjoyable experience for both speaker and audience.

Julia Evans describes an experiment in giving a tech talk about machine learning in production using a printed zine instead of slides. She created an 8-page zine based on Google's 'Rules of Machine Learning' document, printed 70 copies at home, and distributed them at a local meetup. She found the format worked well and was comparable in preparation time to slides. The post shares both digital and printable versions of the zine and reflects on the experience of combining physical media with technical presentations.

Key Insight: A programming book doesn't have to be a comprehensive textbook — it can be a modular collection of self-contained spreads designed to spark curiosity and point readers toward deeper exploration.

Julia Evans shares an idea for a programming book that would combine her comic-style drawings with short written explanations. Each pair of pages would focus on one concept, with a comic on one side and signposting to further resources on the other. She wants something that inspires curiosity rather than serving as a comprehensive textbook, more like 'Thing Explainer' than a technical reference. The book would likely focus on Linux topics.

Key Insight: You don't need to be a manager to make your team better — deliberate practices like public communication, open questions, and genuine willingness to be wrong give every team member power to shape team dynamics.

Julia Evans shares insights from a conversation with her friend Amy Hanlon about how individual team members (not just managers) can actively make their teams better. The key practices include communicating work publicly in shared channels, asking questions openly, scheduling small brainstorming meetings, and approaching discussions with genuine openness to being wrong. Evans emphasizes that teams aren't static — members have real power to shape team dynamics. The post reframes team quality as something every individual can influence through deliberate communication habits.

Key Insight: The most sustainable way to learn through programming is to treat code as disposable—freeing yourself from quality and maintenance obligations lets you focus entirely on understanding.

Julia Evans defines her three rules for programming experiments: they don't have to be good, they don't have to work, and you have to learn something. She emphasizes that these experiments are typically short (a few hours), require no maintenance, and are driven by curiosity rather than producing usable software. This makes them far more tractable than contributing to open source projects. The program itself is incidental—what matters is exploring questions and understanding new concepts.

Key Insight: The best systems talks make unfamiliar topics immediately understandable by clearly laying out the assumptions and design decisions behind the systems we use every day.

Julia Evans recaps her experience at Systems We Love 2016, highlighting her favorite talks from the conference. She shares brief reviews of five talks covering topics from Go's race detector to NTP assumptions to DNS protocol design. Each mini-review focuses on what she learned or found interesting, emphasizing clarity of explanation as what makes a talk great. The post serves as a curated guide to the conference's recorded talks.