Project Structure
Explains the default layout of an Acme project, including key files like acme.yml, the pipelines and transforms directories, and environment configuration.
Project Structure
A Acme project is a directory containing configuration files, transform definitions, and optional test fixtures.
Default layout
my-project/
โโโ acme.yml # Main pipeline configuration
โโโ pipelines/ # Additional pipeline definitions
โ โโโ users.yml
โ โโโ orders.yml
โ โโโ analytics.yml
โโโ transforms/ # Custom transform functions
โ โโโ __init__.py
โ โโโ normalize.py
โ โโโ enrich.py
โโโ tests/ # Pipeline tests
โ โโโ test_users.yml
โ โโโ fixtures/
โ โโโ sample_users.csv
โโโ .env # Environment variables (not committed)
โโโ .env.example # Example environment file
โโโ .acme/ # Local state (auto-generated)
โโโ runs/ # Pipeline run history
โโโ cache/ # Connector cache
Key files
acme.yml
The main configuration file. Defines sources, transforms, destinations, and scheduling.
name: my-project
version: "1.0"
defaults:
batch_size: 5000
retry_count: 3
timeout: 300
sources:
- type: postgres
name: main_db
connection: ${DATABASE_URL}
See Configuration Reference for all available options.
pipelines/ directory
For projects with multiple pipelines, you can split them into separate files:
Each pipeline file inherits the defaults from acme.yml but can override any setting.
transforms/ directory
Custom Python functions for data transformations. See Transforms for details.
# transforms/normalize.py
def normalize_email(row):
"""Lowercase and strip whitespace from email addresses."""
row["email"] = row["email"].lower().strip()
return row
.env file
Never commit
.env files
The .env file contains secrets like database passwords and API keys. Always add it to .gitignore.
DATABASE_URL=postgresql://user:pass@localhost:5432/mydb
API_KEY=sk-abc123...
SLACK_WEBHOOK=https://hooks.slack.com/services/...
Next steps
- Your First Pipeline โ a hands-on tutorial
- Environment Variables โ managing secrets and configuration