Published anonymously with FlowershowReport abuse

Project Structure

A Acme project is a directory containing configuration files, transform definitions, and optional test fixtures.

Default layout

my-project/
โ”œโ”€โ”€ acme.yml           # Main pipeline configuration
โ”œโ”€โ”€ pipelines/             # Additional pipeline definitions
โ”‚   โ”œโ”€โ”€ users.yml
โ”‚   โ”œโ”€โ”€ orders.yml
โ”‚   โ””โ”€โ”€ analytics.yml
โ”œโ”€โ”€ transforms/            # Custom transform functions
โ”‚   โ”œโ”€โ”€ __init__.py
โ”‚   โ”œโ”€โ”€ normalize.py
โ”‚   โ””โ”€โ”€ enrich.py
โ”œโ”€โ”€ tests/                 # Pipeline tests
โ”‚   โ”œโ”€โ”€ test_users.yml
โ”‚   โ””โ”€โ”€ fixtures/
โ”‚       โ””โ”€โ”€ sample_users.csv
โ”œโ”€โ”€ .env                   # Environment variables (not committed)
โ”œโ”€โ”€ .env.example           # Example environment file
โ””โ”€โ”€ .acme/             # Local state (auto-generated)
    โ”œโ”€โ”€ runs/              # Pipeline run history
    โ””โ”€โ”€ cache/             # Connector cache

Key files

acme.yml

The main configuration file. Defines sources, transforms, destinations, and scheduling.

name: my-project
version: "1.0"

defaults:
  batch_size: 5000
  retry_count: 3
  timeout: 300

sources:
  - type: postgres
    name: main_db
    connection: ${DATABASE_URL}

See Configuration Reference for all available options.

pipelines/ directory

For projects with multiple pipelines, you can split them into separate files:

Each pipeline file inherits the defaults from acme.yml but can override any setting.

transforms/ directory

Custom Python functions for data transformations. See Transforms for details.

# transforms/normalize.py
def normalize_email(row):
    """Lowercase and strip whitespace from email addresses."""
    row["email"] = row["email"].lower().strip()
    return row

.env file

Never commit .env files

The .env file contains secrets like database passwords and API keys. Always add it to .gitignore.

DATABASE_URL=postgresql://user:pass@localhost:5432/mydb
API_KEY=sk-abc123...
SLACK_WEBHOOK=https://hooks.slack.com/services/...

Next steps

Built with LogoFlowershow