Skip to content
Weik.ioDocs
2026.1
Try Weik.io

You're reading a page of the Weik.io 2026.1 docs. The current release is 2026.2. Read the 2026.2 page

MFT Configuration

Weik.io 2026.1

Configure managed file transfers between storage systems.

On this page

Defines a managed file transfer (MFT) operation that moves files between storage systems with optional processing and scheduling.

Schema properties#

Property Type Required Default Description
source MftConfigurationSpecStorage Yes - Source storage configuration
destination MftConfigurationSpecStorage Yes - Destination storage configuration
command string No - Python command to execute during transfer (for processing files)
schedule string No - Cron expression for scheduled transfers
requirements Dictionary<string, string> No {} Python package requirements for command execution
parameters Dictionary<string, string> No {} Additional transfer parameters
additionalPackages string No - Additional Python packages to install
args string No - Additional command arguments
filters string No - File filtering patterns (glob patterns)
project string No "" Project name for organizing transfers

Storage properties#

Property Type Required Default Description
name string Yes - Name of the storage system
path string Yes - Path within the storage system

YAML examples#

Basic file transfer#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: daily-backup-transfer
spec:
  source:
    name: local-storage
    path: /data/exports
  destination:
    name: azure-backup
    path: /backups/daily
  schedule: "0 0 2 * * ?"

Transfer with file filtering#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: csv-data-sync
spec:
  source:
    name: sftp-server
    path: /incoming/data
  destination:
    name: data-lake
    path: /raw/customer-data
  filters: "*.csv"
  schedule: "0 0/30 * * * ?"

Transfer with processing#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: process-and-transfer
spec:
  source:
    name: source-storage
    path: /raw/files
  destination:
    name: processed-storage
    path: /processed/files
  command: |
    import pandas as pd
    import sys

    def process_file(file_path):
        # Read CSV file
        df = pd.read_csv(file_path)

        # Clean and transform data
        df = df.dropna()
        df['processed_date'] = pd.Timestamp.now()

        # Save processed file
        output_path = file_path.replace('.csv', '_processed.csv')
        df.to_csv(output_path, index=False)

        return output_path
  requirements:
    pandas: ">=2.0.0"
  schedule: "0 0 0/6 * * ?"

Complex transfer with parameters#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: encrypted-backup-transfer
spec:
  source:
    name: production-database-exports
    path: /exports/encrypted
  destination:
    name: offsite-backup
    path: /backups/${date}
  command: decrypt_and_compress.py
  args: "--compression-level 9 --verify-integrity"
  requirements:
    cryptography: ">=41.0.0"
    python-dateutil: ">=2.8.0"
  parameters:
    EncryptionKey: "{{sys:backup-encryption-key}}"
    RetryCount: "3"
    Timeout: "3600"
  filters: "*.enc"
  schedule: "0 0 1 * * ?"
  project: production-backups

Multiple MFT configurations#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: orders-sync
spec:
  source:
    name: erp-system
    path: /exports/orders
  destination:
    name: data-warehouse
    path: /staging/orders
  filters: "orders_*.json"
  schedule: "0 0 * * * ?"
  project: data-integration
---
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: customer-sync
spec:
  source:
    name: crm-system
    path: /exports/customers
  destination:
    name: data-warehouse
    path: /staging/customers
  filters: "customers_*.json"
  schedule: "0 30 * * * ?"
  project: data-integration
---
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: products-sync
spec:
  source:
    name: pim-system
    path: /exports/products
  destination:
    name: data-warehouse
    path: /staging/products
  filters: "products_*.json"
  schedule: "0 15 * * * ?"
  project: data-integration

Transfer with complex processing#

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: image-optimization-transfer
spec:
  source:
    name: uploads-storage
    path: /raw/images
  destination:
    name: cdn-storage
    path: /optimized/images
  command: |
    from PIL import Image
    import os

    def optimize_image(file_path):
        """Optimize and resize images"""
        img = Image.open(file_path)

        # Resize if larger than max dimensions
        max_size = (1920, 1080)
        img.thumbnail(max_size, Image.Resampling.LANCZOS)

        # Save optimized image
        output_path = file_path.replace('.jpg', '_optimized.jpg')
        img.save(output_path, 'JPEG', quality=85, optimize=True)

        return output_path
  requirements:
    Pillow: ">=10.0.0"
  filters: "*.jpg,*.jpeg,*.png"
  schedule: "0 0/15 * * * ?"
  parameters:
    MaxFileSize: "5242880"
    DeleteSource: "false"

Usage notes#

MFT configurations automate file transfers between storage systems with optional processing, filtering, and scheduling.

Storage configuration#

The source and destination objects define storage endpoints:

  • name references a configured storage system (must exist)
  • path specifies the location within that storage

Storage systems must be configured separately before creating MFT configurations.

File processing#

The command field can contain:

  • Inline Python code for simple transformations
  • Path to a Python script file
  • Name of an installed Python package/module

Processing happens during transfer:

  1. File is retrieved from source
  2. Command processes the file
  3. Processed file is transferred to destination

Scheduling#

The schedule field uses cron expressions to automate transfers:

  • Run transfers at specific times
  • Coordinate multiple related transfers
  • Balance load across different schedules

Omit schedule for manual/on-demand transfers.

File filtering#

The filters field accepts:

  • Single pattern: *.csv
  • Multiple patterns: *.csv,*.json,*.xml
  • Glob patterns: data_*.csv, 202?-??-??.json

Only files matching the filter patterns are transferred.

Parameters#

Common parameters:

  • RetryCount: Number of retry attempts on failure
  • Timeout: Transfer timeout in seconds
  • DeleteSource: Whether to delete source files after successful transfer
  • OverwriteExisting: Whether to overwrite existing destination files
  • PreserveTimestamps: Whether to preserve file timestamps

Project organization#

Use the project field to group related MFT configurations:

  • Enables batch operations on related transfers
  • Improves organization and filtering
  • Useful for managing complex transfer workflows

Best practices#

  • Test transfers manually before scheduling
  • Use descriptive names indicating source, destination, and purpose
  • Implement error handling in processing commands
  • Choose appropriate schedules to avoid conflicts
  • Monitor transfer success rates and failures
  • Use filters to transfer only necessary files
  • Consider storage costs when designing transfer patterns
  • Log processing results for troubleshooting
  • Set appropriate timeouts for large file transfers
  • Clean up old files to manage storage usage
  • Use variables for sensitive information (credentials, keys)
  • Document processing logic for maintenance

Error handling#

Transfers may fail due to:

  • Network connectivity issues
  • Authentication failures
  • Insufficient permissions
  • Storage capacity limits
  • Processing errors

Include error handling in processing commands and configure appropriate retry behavior in parameters.

Monitoring#

Monitor MFT operations:

Terminal
weikio integration mft ls
weikio integration mft status <mft-id>
weikio integration mft history <mft-id>

Sign in to the live demo

Sign in with the shared viewer account. It can open every screen of the example projects but can't change anything.

E-mail
viewer@demo.weik.io
Password
try-weikio-live-demo
Open the live demo