AskAiml
Sign in Get started

Project layout — src/, tests/, config, and secrets

A layout that survives the jump from script to package to service, plus config and secret patterns.

🧰 Module 0 9 min read Not started

Why this matters

The first script you write fits in one file. The fifth grows hairy. The fiftieth requires structure — or you can't onboard anyone, run tests, or publish to PyPI. Senior engineers organise code so it scales from prototype to package without rewrites. This lesson installs that muscle memory: src/ layout, modules, packages, __init__.py, relative imports, and how sys.path actually works.

Learning objectives

  1. Distinguish modules and packages.
  2. Apply the src/ layout for installable projects.
  3. Use absolute and relative imports correctly.
  4. Understand __init__.py, namespace packages, and import side-effects.
  5. Use __main__.py for CLI entry points.

1. The four levels of organisation

script.py                      single file
    ↓
module (a .py file)            grouped functions/classes
    ↓
package (a directory)          group of modules
    ↓
distribution (a PyPI release)  installable; has pyproject.toml

You can stop at any level. A 100-line tool stays a single script. A 10k-line project becomes a distribution.

2. The canonical src/ layout

my-project/
├── pyproject.toml
├── README.md
├── .gitignore
├── .python-version
├── uv.lock
├── src/
│   └── my_project/
│       ├── __init__.py
│       ├── core.py
│       ├── utils.py
│       └── cli.py
└── tests/
    ├── __init__.py
    ├── test_core.py
    └── test_cli.py

Why src/?

  • Forces installation (uv pip install -e .) before importing — catches "works in repo, breaks when installed" bugs.
  • Keeps tests from accidentally importing the un-installed local copy.
  • Standard since ~2019; recommended by the Python Packaging Authority.

The other layout (top-level package, no src/) is acceptable for tiny tools but less safe.

3. Modules — a .py is a module

python
# utils.py
def slugify(s: str) -> str:
    return s.lower().replace(" ", "-")

Use it:

python
# main.py
import utils                              # whole module
from utils import slugify                 # specific name
from utils import slugify as slug         # rename

Every module has dunder attributes:

  • __name__: the import name; equals "__main__" when the file is run directly.
  • __file__: path to source.
  • __doc__: the docstring (first triple-quoted string).
  • __package__: containing package name.

The if __name__ == "__main__": idiom

python
def main() -> None:
    print("running as script")

if __name__ == "__main__":
    main()

When imported, __name__ is "utils". When run via python utils.py, it's "__main__". The idiom lets a module be both library and script.

4. Packages — a directory with __init__.py

my_project/
├── __init__.py         <- makes the directory a package
├── core.py
└── utils.py

Import:

python
import my_project
from my_project import core
from my_project.utils import slugify

The package itself is a module — the code in __init__.py runs once on first import.

What goes in __init__.py?

Three common patterns:

Empty (most files):

python
# (empty)

Re-export public API:

python
# __init__.py
from my_project.core import Engine, Item
from my_project.utils import slugify

__all__ = ["Engine", "Item", "slugify"]

So callers can do from my_project import Engine instead of from my_project.core import Engine.

Version constant:

python
__version__ = "1.2.3"

Subpackages

my_project/
├── __init__.py
├── api/
│   ├── __init__.py
│   ├── auth.py
│   └── users.py
└── core/
    ├── __init__.py
    └── engine.py
python
from my_project.api.auth import login

Namespace packages (no __init__.py)

Python 3.3+ allows namespace packages that span multiple directories without an __init__.py. Useful for plugin systems. For application code, prefer regular packages with explicit __init__.py.

5. Absolute vs relative imports

Absolute (preferred):

python
# my_project/api/users.py
from my_project.core.engine import Engine

Relative (also fine inside a package):

python
# my_project/api/users.py
from ..core.engine import Engine     # two dots = parent package
from .auth import login              # one dot = same package

When use which

  • Public packages → absolute (clear, robust to refactoring).
  • Tightly-coupled subpackages → relative (one fewer thing to rename when the top-level renames).

Don't mix from foo.bar import baz with from .bar import baz in the same package without a reason; pick a style.

Forbidden: relative import outside a package

python
# random_script.py at the top of repo
from .utils import slugify           # ImportError: attempted relative import with no known parent package

Relative imports only work inside installed/imported packages.

6. How sys.path and import work

When you write import foo, Python:

  1. Checks sys.modules (cache).
  2. Searches sys.path directories for foo.py or foo/__init__.py.
  3. Executes the module top-to-bottom.
  4. Stores the resulting module object in sys.modules.
  5. Binds the name foo in the importer's namespace.
python
import sys
print(sys.path)
print(sys.modules.keys())

sys.path order (rough):

  1. The script's directory (or '' for the cwd in REPL).
  2. PYTHONPATH environment variable.
  3. Installed site-packages (the venv).

The "I have two utils.py" bug

Your script imports utils. Python finds ./utils.py (top of sys.path) instead of the package's my_project/utils.py. Result: bizarre behaviour. Fix: install your project (uv pip install -e .) and import via the package name.

Reload during development

If you change a module mid-REPL session, import returns the cached version. Use:

python
import importlib
importlib.reload(mymod)

Or just restart the REPL. (Jupyter has %autoreload.)

7. Entry points and python -m

You can run any module / package as a script:

powershell
uv run python -m my_project          # runs my_project/__main__.py
uv run python -m my_project.cli      # runs my_project/cli.py

__main__.py is the convention for "what this package does when run as a script":

python
# my_project/__main__.py
from my_project.cli import main
main()

Pros over a top-level script:

  • Works regardless of cwd.
  • Doesn't pollute sys.path[0] with the project's parent dir.
  • -m resolves the venv correctly.

For installed CLIs, declare a console script entry point in pyproject.toml:

toml
[project.scripts]
my-cli = "my_project.cli:main"

After uv pip install -e ., the command my-cli is on PATH and calls my_project.cli.main().

8. Best-practice file structure for different sizes

Tiny tool (≤ 100 lines)

tool.py
README.md
pyproject.toml

Small library (one package, no subpackages)

my-lib/
├── pyproject.toml
├── README.md
├── src/my_lib/
│   ├── __init__.py
│   └── core.py
└── tests/
    └── test_core.py

Medium app (subpackages, CLI, config)

my-app/
├── pyproject.toml
├── README.md
├── src/my_app/
│   ├── __init__.py
│   ├── __main__.py
│   ├── cli.py
│   ├── config.py
│   ├── api/
│   │   ├── __init__.py
│   │   ├── routes.py
│   │   └── auth.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── engine.py
│   └── data/
│       ├── __init__.py
│       └── models.py
├── tests/
│   ├── unit/
│   └── integration/
├── scripts/
│   └── seed.py
├── docs/
└── docker/
    └── Dockerfile

Big monorepo

Multiple packages under packages/, shared pyproject.toml, workspaces.

9. Imports etiquette

PEP 8 standard import order, grouped with blank lines:

python
# 1. Standard library
import os
import sys
from pathlib import Path

# 2. Third-party
import httpx
import pydantic
from rich.console import Console

# 3. First-party (your own packages)
from my_project.core import Engine
from my_project.utils import slugify

# 4. Local (relative)
from .helpers import build_request

ruff (with I rule) sorts these automatically.

Avoid:

  • from module import * (pollutes namespace, breaks tooling).
  • Importing inside functions (only when needed to break circular imports or defer slow imports).
  • Re-exporting half a package's internals from __init__.py. Pick a deliberate public API.

10. Circular imports — and how to break them

python
# a.py
from b import B
class A: pass

# b.py
from a import A
class B: pass

Boom: ImportError.

Fixes (in order of preference):

  1. Restructure — move the shared dependency to a third module.
  2. Move the import inside the function that needs it.
  3. Use TYPE_CHECKING:
    python
    from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from b import B            # only seen by type-checker, not at runtime
class A:
    def use(self, b: "B") -> None: ...

Restructuring is usually the right answer. Circular imports often signal a design issue.

Hands-on lab (1 hour)

  1. Convert a single-file tool.py into a src/-layout package.
  2. Add a cli.py exposing a main(); wire [project.scripts] so my-cli works after uv pip install -e ..
  3. Add __main__.py so python -m my_project works.
  4. Split into core/ and utils/ subpackages; update imports to be absolute.
  5. Intentionally introduce a circular import; fix it with TYPE_CHECKING.
  6. Add __all__ to one module; verify from x import * only exposes those names.
  7. Bonus: add a namespace package extension (a separate repo that contributes more modules under the same top-level name).

Common pitfalls

  1. Forgetting __init__.py and wondering why imports fail.
  2. Running python my_project/cli.py (often breaks imports) instead of python -m my_project.cli.
  3. Putting business logic at module top-level (runs on import); only definitions and lightweight constants belong there.
  4. Catching ImportError to "make it work both ways" — usually masks real bugs.
  5. Importing the test package from production code (sneaks past CI, fails on PyPI install).

Self-check

  1. Module vs package vs distribution.
  2. What does if __name__ == "__main__": do?
  3. Difference between absolute and relative imports.
  4. Why use the src/ layout?
  5. How does Python find a module on import?

References

Sign in to save your progress and earn badges.