Managing ML Pipelines with Datamon

Introduction

Datamon helps build ML pipelines by adding versioning, auditing and lineage tracking to cloud storage tools (e.g. Google GCS, AWS S3). This is not a replacement for these tools, but rather a way to manage their inputs and outputs.

Datamon works by providing a git-like interface to manage data efficiently: your data buckets are organized in repositories of versioned and tagged bundles of files.

Installation and Setup

Version Information

# Check Datamon version
d2 version
# Output:
# Version: 2.3.0
# BuildDate: 2020-11-10T11:29:17Z
# Commit: 17ce7d6
# Working tree: clean

Configuration

# Set config for different environments
d2 config set --config global-onec-co-datamon-config --context dev
d2 config set --config global-onec-co-datamon-config --context staging
d2 config set --config workshop-config

Context Management

# List all contexts
d2 context list
# Output:
# [dev prod staging]

# Get current context
d2 context
# or
d2 get context

# Get specific context
d2 context get --context dev
# Output:
# Model Version: 0
# Name: dev
# WAL: dev-onec-co-datamon-metadata-wal
# ReadLog: dev-onec-co-datamon-readlog
# Blob: global-onec-co-datamon-blob
# Metadata: dev-onec-co-datamon-metadata
# Version Metadata: dev-onec-co-datamon-vmetadata

# Create new context
d2 context create test

Repository Management

# Get repository details
d2 repo get zenrin-estat-residential
# or
d2 repo get --repo zenrin-estat-residential

# List repositories
d2 repo list | grep zenrin-estat-residential

# Create new repositories
d2 repo create --repo ntd-road-source-dev --description "raw download of ntd road data"
d2 repo list | grep ntd

# Create repository in specific context
d2 repo create --context staging --repo ntd-road-source-staging --description "the original ntd road data"
d2 repo list --context staging | grep ntd

Bundle Management

Uploading Bundles

# Upload a bundle
d2 bundle upload --path folder-to-upload --repo mkang-test-repo --message "my first upload"

# Upload to specific repository
d2 bundle upload --path ~/occ/prod/01-built-object-service/tmp/ntd/RI --repo ntd-road-source-dev
d2 bundle upload --path ~/occ/prod/01-built-object-service/tmp/ntd/RI --repo ntd-road-source-dev --message "upload RI"

Listing Bundles

# List all bundles
d2 bundle list

# List bundles in a repository
d2 bundle list --repo mkang-test-repo
d2 bundle list --repo ntd-road-source-dev
d2 bundle list --repo zenrin-estat-residential
d2 bundle list --repo resilience-japan-hazard-maps
d2 bundle list --repo Seattle-Sample-Corelogic-Run-Data

# List bundles in different contexts
d2 bundle list --repo ntd-road-source-staging --context staging
d2 bundle list --repo ntd-road-source-dev --context dev

Bundle Operations

# List files in a bundle
d2 bundle list files --repo mkang-test-repo --bundle 1fySBuavEhqWAXnYnZEiDCNm8TC
d2 bundle list files --repo mkang-test-repo --bundle 1fySBuavEhq 2>/dev/null | grep file

# Mount bundle
d2 bundle mount --repo mkang-test-repo --mount ~/mnt --daemonize
d2 bundle mount --repo mkang-test-repo --label 1fySBuavEhqWAXnYnZEiDCNm8TC --mount ~/mnt --daemonize

# Download bundle
d2 bundle download --repo mkang-test-repo --destination .