Published anonymously with FlowershowReport abuse

Configuration File Reference

The acme.yml file is the primary configuration file for a Acme project. This page documents every available setting.

Minimal configuration

name: my-project
version: "1.0"

sources:
  - type: csv
    path: ./data/input.csv

destinations:
  - type: json
    path: ./output/result.json

Complete reference

# โ”€โ”€โ”€ Project โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
name: my-project # Required. Project name.
version: "1.0" # Required. Configuration version.
description: "My data pipeline" # Optional. Human-readable description.

# โ”€โ”€โ”€ Defaults โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
defaults:
  batch_size: 5000 # Rows per batch (default: 5000)
  workers: 4 # Parallel transform workers (default: 4)
  timeout: 300 # Pipeline timeout in seconds (default: 300)
  retry_count: 3 # Max retries on failure (default: 3)
  retry_delay: 10 # Seconds between retries (default: 10)
  buffer_memory: 256mb # Max memory for buffering (default: 256mb)

# โ”€โ”€โ”€ Sources โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
sources:
  - type: postgres # Connector type
    name: main_db # Unique name for this source
    connection: ${DATABASE_URL} # Connection string (use env vars!)
    query: "SELECT * FROM users" # SQL query
    timeout: 60 # Query timeout in seconds
    batch_size: 10000 # Override default batch size
    incremental:
      column: updated_at # Column for incremental extraction
      initial_value: "2025-01-01" # Starting value for first run

# โ”€โ”€โ”€ Transforms โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
transforms:
  - type: filter
    condition: "status = 'active'"
  - type: map
    fields:
      full_name: "first_name || ' ' || last_name"
  - type: python
    function: transforms.my_function
  - type: select
    columns: [id, full_name, email]

# โ”€โ”€โ”€ Destinations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
destinations:
  - type: bigquery
    dataset: analytics
    table: users
    write_mode: upsert # append | replace | upsert | merge
    key: id # Key column for upsert/merge

# โ”€โ”€โ”€ Scheduling โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
schedule: "0 */6 * * *" # Cron expression
timezone: "UTC" # Timezone for schedule (default: UTC)

# โ”€โ”€โ”€ Dependencies โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
depends_on:
  - load-orders # Wait for these pipelines first
  - load-exchange-rates

# โ”€โ”€โ”€ Error Handling โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
error_handling:
  strategy: retry # fail | retry | skip
  max_retries: 3
  retry_delay: 10s
  backoff: exponential # linear | exponential | constant
  dead_letter:
    type: json
    path: ./errors/

# โ”€โ”€โ”€ Schema Validation โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
schema:
  id: { type: integer, required: true }
  email: { type: string, format: email }
  age: { type: integer, min: 0, max: 150 }

# โ”€โ”€โ”€ Notifications โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
notifications:
  on_failure:
    - type: slack
      webhook: ${SLACK_WEBHOOK}
  on_success:
    - type: slack
      webhook: ${SLACK_WEBHOOK}
      only_after_failure: true

# โ”€โ”€โ”€ Monitoring โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
monitoring:
  metrics:
    enabled: true
    port: 9090 # Prometheus metrics port
  alerts:
    - name: high-latency
      condition: "duration_seconds > 300"
      channels: [slack]

# โ”€โ”€โ”€ Logging โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
logging:
  level: info # debug | info | warning | error
  format: json # json | text
  output: stdout # stdout | file
  file: /var/log/acme.log # Log file path (when output: file)

# โ”€โ”€โ”€ Plugins โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
plugins:
  connectors:
    - path: ./connectors/my_source.py
      type: my_source
  transforms:
    - path: ./transforms/custom.py

# โ”€โ”€โ”€ Auth (API server) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
auth:
  provider: api_key # api_key | oauth2
  # For OAuth 2.0:
  # issuer: https://auth.example.com
  # client_id: ${OAUTH_CLIENT_ID}

[!warning] YAML indentation YAML is sensitive to indentation. Use 2 spaces (not tabs). Most errors in acme.yml are caused by incorrect indentation.

Validating your configuration

acme validate
โœ“ acme.yml is valid
โœ“ All sources configured correctly
โœ“ All transforms found
โœ“ All destinations configured correctly
โœ“ Schedule expression valid
Built with LogoFlowershow