datamon
- version info
> d2 version
Version: 2.3.0
BuildDate: 2020-11-10T11:29:17Z
Commit: 17ce7d6
Working tree: clean
> datamon2
Datamon helps build ML pipelines by adding versioning, auditing and lineage tracking to cloud storage tools
(e.g. Google GCS, AWS S3).
This is not a replacement for these tools, but rather a way to manage their inputs and outputs.
Datamon works by providing a git like interface to manage data efficiently:
your data buckets are organized in repositories of versioned and tagged bundles of files.
Usage:
datamon [command]
Available Commands:
bundle Commands to manage bundles for a repo
config Commands to manage the config file
context Commands to manage contexts.
diamond Commands to manage diamonds
help Help about any command
label Commands to manage labels for a repo
repo Commands to manage repos
upgrade Upgrades datamon to the latest release
usage Generates documentation
version prints the version of datamon
web Webserver
Flags:
--config string Set the config backend store to use (bucket name: do not set the scheme, e.g. 'gs://')
--context string Set the context for datamon (default "dev")
--force Forces upgrade even if the current version is not a released version
-h, --help help for datamon
--loglevel string The logging level. Levels by increasing order of verbosity: none, error, warn, info, debug (default "info")
--metrics Toggle telemetry and metrics collection
--metrics-password string Password to connect to the metrics collector backend. Overrides any password set in URL
--metrics-url string Fully qualified URL to an influxdb metrics collector, with optional user and password
--metrics-user string User to connect to the metrics collector backend. Overrides any user set in URL
--upgrade Upgrades the current version then carries on with the specified command
Use "datamon [command] --help" for more information about a command.
- config
d2 config set --config global-onec-co-datamon-config --context dev
**d2 config set --co**nfig global-onec-co-datamon-config --context staging
d2 config set --config workshop-config
- context
d2 context list
d2 context
d2 get context
d2 context
d2 context get --context dev
d2 context get dev
d2 context get dev
d2 context
d2 context create test
d2 context
d2 context list
d2 context get staging
d2 context
d2 context get dev
d2 context get
d2 context list
➜ ~ d2 context list
[dev prod staging]
➜ ~ d2 context get --context dev
Model Version: 0, Name: dev, WAL: dev-onec-co-datamon-metadata-wal, ReadLog: dev-onec-co-datamon-readlog, Blob: global-onec-co-datamon-blob, Metadata: dev-onec-co-datamon-metadata, Version Metadata: dev-onec-co-datamon-vmetadata
➜ ~ d2 context get --context staging
Model Version: 0, Name: staging, WAL: staging-onec-co-datamon-metadata-wal, ReadLog: staging-onec-co-datamon-readlog, Blob: global-onec-co-datamon-blob, Metadata: staging-onec-co-datamon-metadata, Version Metadata: staging-onec-co-datamon-vmetadata
➜ ~ d2 context get --context prod
Model Version: 0, Name: prod, WAL: prod-onec-co-datamon-metadata-wal, ReadLog: prod-onec-co-datamon-readlog, Blob: global-onec-co-datamon-blob, Metadata: prod-onec-co-datamon-metadata, Version Metadata: prod-onec-co-datamon-vmetadata
- repo
d2 repo get zenrin-estat-residential
d2 repo get --repo zenrin-estat-residential
d2 repo list | grep zenrin-estat-residential
d2 repo create --repo ntd-road-source-dev --description "raw download of ntd road data"
d2 repo list | grep ntd
d2 repo create --context staging --repo ntd-road-source-staging --description "the original ntd road data"
d2 repo list --context staging | grep ntd
d2 repo
d2 repo get zenrin-estat-residential
d2 repo get --repo zenrin-estat-residential
d2 repo list | grep zenrin-estat-residential
d2 repo get --repo resilience-japan-hazard-maps
d2 repo get --repo Seattle-Sample-Corelogic-Run-Data
- bundle
d2 bundle upload --path folder-to-upload --repo mkang-test-repo --message "my first upload"
d2 bundle list --repo mkang-test-repo
d2 bundle list files --repo mkang-test-repo --bundle 1fySBuavEhqWAXnYnZEiDCNm8TC
d2 bundle list files --repo mkang-test-repo --bundle 1fySBuavEhq 2>dev/null | grep file
d2 bundle list files --repo mkang-test-repo --bundle 1fySBuavEhq 2>/dev/null | grep file
d2 bundle mount --repo mkang-test-repo --mount ~/mnt -demonize
d2 bundle mount --repo mkang-test-repo --label 1fySBuavEhqWAXnYnZEiDCNm8TC --mount ~/mnt -demonize
d2 bundle mount --repo mkang-test-repo --mount ~/mnt --demonize
d2 bundle mount --repo mkang-test-repo --mount ~/mnt --daemonize
d2 bundle download --repo mkang-test-repo --destination .
d2 bundle list --repo ntd-road-source-staging --context staging
d2 bundle list --repo ntd-road-source-dev --context dev
d2 bundle upload --path ~/occ/prod/01-built-object-service/tmp/ntd/RI --repo ntd-road-source-dev
d2 bundle upload --path ~/occ/prod/01-built-object-service/tmp/ntd/RI --repo ntd-road-source-dev --message "upload RI"
d2 bundle list
d2 bundle list --repo ntd-road-source-dev
d2 bundle
d2 bundle mount --repo ntd-road-source-dev
d2 bundle list --repo zenrin-estat-residential
d2 bundle list --repo resilience-japan-hazard-maps
d2 bundle list --repo Seattle-Sample-Corelogic-Run-Data
- label
d2 label set --repo mkang-test-repo --bundle 1fySBuavEhqWAXnYnZEiDCNm8TC --label my-first-upload
d2 label set --repo mkang-test-repo --bundle 1fy --label the-second-label
d2 label set --repo mkang-test-repo --bundle 1fySBu --label the-second-label
d2 label set --repo mkang-test-repo --bundle 1fySBuavEhqWAXnYnZEiDCNm8TC --label the-second-label
d2 label list --repo mkang-test-repo
d2 label list --repo mkang-test-repo --prefix my
d2 *label list --repo mkang-test-repo --prefix the*
d2 label list --repo zenrin-estat-residential
- webserver
d2 web
http://0.0.0.0:65242
