Python Project with Packages
Organizing Code
When working on Python projects, I ogranize my code into Python that I import in scripts, notebooks or applications. This helps in keeping the codebase clean and modular.
Working with serveral loose Python files can quickly add some friction: import from other files, relative paths, etc. That’s why I prefer to build installable package from the beginning.
At the design phase I may even create different packages to try out different designs. Code is clearly separated. Importing one package or another is a matter of changing one line in the script or notebook.
In this post I show how I setup my python project with multiple packages.
That makes it possible to import code from different packages like this:
from package1 import some_function
from package2 import another_function
This helps organize code and separate concerns.
Project Structure
As you can see from the project structure from below, the project is split into three python projects:
- One root project (workspace) with scripts, applications or notebooks
- Two packages in
src/
that are installable packages
my-project/ # Workspace root
├── pyproject.toml # Workspace configuration
├── uv.lock # Dependency lock file
├── .venv/ # Shared virtual environment
├── README.md
├── scripts/ # Utility scripts (beware to make root project a package)
│ ├── deploy.py
│ └── migrate.py
└── src/ # All packages live here
├── package_1/ # First package
│ ├── pyproject.toml # Package 1 config
│ ├── README.md
│ └── package_1/ # Python code
│ ├── __init__.py
│ ├── main.py
│ └── utils.py
└── package_2/ # Second package
├── pyproject.toml # Package 2 config
├── README.md
└── package_2/ # Python code
├── __init__.py
├── api.py
└── models.py
To avoid confusion between the overall project /
and the three python projects, let me call the later pyproject’s.
Each pyproject has its own pyproject.toml
file to manage project meta data, dependencies, development tool configurations, build settings and entry points for applications.
In general, I like to organize my projects with:
- One root project (workspace) with scripts, applications or notebooks
- Workspace import packages from
src/
- Packages in
src/
are installable packages - Packages in
src/
maintain their own dependencies - Packages in
src
are installed as editable packages in the workspace - One shared virtual environment (venv)
- Shared dependencies for the workspace and the packages
Multi package setup
Let’s start building the pyproject.toml
files for the project.
The workspace pyproject.toml
file contains package1
and package2
as dependencies.
Workspace setup
# /pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build_meta"
[project]
name = "my-project-workspace"
version = "0.1.0"
description = "Workspace for multiple related packages"
authors = [
{name = "Your Name", email = "your.email@example.com"},
]
readme = "README.md"
requires-python = ">=3.9"
[tool.uv]
package = false # If workspace is not a package
### Option 1: loose packages ####
# This makes both packages editable-installable
dependencies = [
# Common dependencies used by multiple packages
"requests>=2.28.0",
"pydantic>=2.0.0",
"package-1",
"package-2",
]
[tool.uv.sources]
package-1 = { path = "src/package_1", editable = true }
package-2 = { path = "src/package_2", editable = true }
##################################
### Option 2: uv workspace ####
[tool.uv.workspace]
members = ["src/package_1", "src/package_2"]
# Or with glob
members = ["src/*"]
################################
[project.optional-dependencies]
# Development tools for the entire workspace
dev = [
"pytest>=7.0.0",
"black>=23.0.0",
"ruff>=0.0.290",
"mypy>=1.0.0",
]
[tool.black]
line-length = 88
target-version = ['py39']
[tool.ruff]
line-length = 88
target-version = "py39"
There are two options to setup the workspace:
- Loose packages: list each package as a dependency and make them editable-installable
- uv workspace: use the
uv
workspace feature to automatically include all packages insrc/
Note that I am using uv
as the package manager and hatchling
as the build backend.
Individual package setup
Now let’s look at the individual package pyproject.toml
files. They are pretty standard.
Package 1:
# /src/package_1/pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build_meta"
[project]
name = "package-1"
version = "0.1.0"
description = "First package in the workspace"
authors = [
{name = "Your Name", email = "your.email@example.com"},
]
readme = "README.md"
requires-python = ">=3.9"
# Package-specific dependencies
dependencies = [
"fastapi>=0.100.0", # Only needed by package_1
"uvicorn>=0.23.0",
]
[project.optional-dependencies]
dev = [
"pytest-asyncio>=0.21.0", # Specific to this package's testing needs
]
# If this package provides CLI commands
[project.scripts]
package1-cli = "package_1.cli:main"
Package 2:
# /src/package_1/pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build_meta"
[project]
name = "package-2"
version = "0.1.0"
description = "Second package in the workspace"
authors = [
{name = "Your Name", email = "your.email@example.com"},
]
readme = "README.md"
requires-python = ">=3.9"
dependencies = [
"package-1", # References the other workspace package!
"pandas>=2.0.0", # Only needed by package_2
"matplotlib>=3.7.0",
]
[project.scripts]
package2-process = "package_2.processor:main"
Dependencies between packages
In the manual setup we need to do some extra work. For example to import package_1
in package_2
, we need to add a brittle path reference in package_2
’s pyproject.toml
file:
# In package_2/pyproject.toml - referencing package_1
dependencies = [
"package-1 @ file:///../../src/package_1", # Brittle path reference
]toml
With the uv workspace setup, uv automatically resolves the package by name:
# In package_2/pyproject.toml - referencing package_1
dependencies = [
"package-1", # Just the package name!
]
Working with the multi-package project
In this setupuv
maintains a single shared virtual environment for the entire workspace, ensuring consistency across all packages.
Installing dependencies
To install all dependencies for the workspace and its packages, simply run:
uv sync
This will set up one single shared virtual environment for the entire workspace.
Running commands
To run a command in the shared virtual environment, use uv run
:
uv run python scripts/deploy.py
uv run pytest
uv run package1-cli --help
uv run package2-process
Editable Install
To make sure that changes in the packages are immediately reflected in the workspace, I used editable installs.
uv run ipython
With the IPYthon auto reload module functions are automatically reloaded upon edits:
%load_ext autoreload
%autoreload 2
from package_1 import some_function
Adding dependencies
In pyproject.toml
I included two options: manual package setup or uv workspace setup. The two setups require slightly different usage for adding dependencies.
To add a new dependency to a package from the command line with uv
:
uv add requests # Add package only to root
cd src/package_1 && uv add fastapi # Must navigate to each package
With the uv workspace setup, you can add dependencies to any package from the root:
# Adding dependencies
uv add requests # To workspace root
uv add --package package-1 fastapi # To specific package
uv add --dev pytest-asyncio # Dev dependency to workspace
Conclusion
Setting up a Python project with multiple packages helps in organizing code, separating concerns, and managing dependencies effectively.
The advances are:
- Single shared virtual environment for the entire workspace
- Consistent dependency versions across all packages
- Simplified dependency management
- Easy to add or remove packages from the workspace
The setup that I have shown uses uv
as the package manager and hatchling
as the build backend, but similar principles apply to other tools like poetry
or pipenv
. The workspace mode of uv
is particularly useful for managing multiple related packages in a single project.