digdag
https://tech.gamevilcom2us.com/blog/3922
Google 클라우드 플랫폼 기반 빅데이터 시스템 구축기
HIVE 애널리틱스 HIVE는 인증, 멤버십, 프로모션, 빌링, 노티피케이션, 고객센터, 애널리틱스 등으로 구성된 모바일 게임 플랫폼입니다. HIVE 플랫폼을 이용해 게임을 개발하
tech.gamevilcom2us.com
https://docs.google.com/presentation/d/11C_BKio0DZIop_ZjJk7ogxQtWV5qHIr-hHjw277z64k/htmlpresent
그들이 AWS 위에서 데이터 파이프라인을 운영하는법 (2019)
그들이 AWS 위에서 데이터 파이프 라인을 운영하는 법 Devops Korea Jun 8, 2019 1ambda @ yanolja bit.ly/2VKMAZv
docs.google.com
https://qiita.com/shiozaki/items/f79eecf8e1878aa64a40
DigdagとEmbulkを利用してBigQueryにRDS（Aurora、MySQL）のマスタデータを同期する - Qiita
この記事はVASILY DEVELOPERS BLOGにも同じ内容で投稿しています。よろしければ他の記事もご覧ください。 こんにちは、バックエンドエンジニアの塩崎です。 さて、VASILYではData Warehouseとして...
qiita.com
https://gist.github.com/hiroyuki-sato/48f1119d8e7436a1014c1958916cf0b8
digdag embulk example
digdag embulk example. GitHub Gist: instantly share code, notes, and snippets.
gist.github.com
Digdag – Open Source Workflow Engine for the Multi-Cloud Era
Simple, Open Source, Multi-Cloud Workflow Engine Digdag is a simple tool that helps you to build, run, schedule, and monitor complex pipelines of tasks. It handles dependency resolution so that tasks run in series or in parallel. Digdag replaces cron, faci
www.digdag.io
https://github.com/suaaa7/digdag-server
suaaa7/digdag-server
Digdag server with Docker Compose. Contribute to suaaa7/digdag-server development by creating an account on GitHub.
github.com
https://docs.digdag.io/operators/bq.html
bq>: Running Google BigQuery queries — Digdag 0.9.41 documentation
bq>: Running Google BigQuery queries bq> operator runs a query on Google BigQuery. _export: bq: dataset: my_dataset +step1: bq>: queries/step1.sql +step2: bq>: queries/step2.sql destination_table: result_table +step3: bq>: queries/step3.sql destination_tab
docs.digdag.io
https://wapa5pow.com/build-analytics-environment-on-eks/
EKS(Kubernetes)上にDigdag・Embulk・Redashで分析環境を構築する - Koichi Ishida blog
Koichi Ishida blog
wapa5pow.com
https://github.com/septeni-original/k8s_job
septeni-original/k8s_job
kubernetes job operator for Digdag. Contribute to septeni-original/k8s_job development by creating an account on GitHub.
github.com
** digdag docker-compose
version: '3'
services:
digdag:
container_name: digdag-server
build:
context: .
dockerfile: digdag/Dockerfile
depends_on:
- postgresql
command: --task-log ./task_log
ports:
- "65432:65432"
- "65433:65433"
volumes:
- /tmp:/tmp
- ./digdag/projects:/opt
- ./digdag/server.properties:/etc/digdag/server.properties
postgresql:
container_name: postgresql
image: postgres:9.6.17
environment:
- POSTGRES_USER=digdag
- POSTGRES_PASSWORD=digdag
- POSTGRES_DB=digdag
volumes:
- ./postgres-data:/var/lib/postgresql/data
** digdag Dockerfile
FROM openjdk:8-jdk
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
RUN apt-get update && apt-get install -y --no-install-recommends \
curl gettext-base postgresql-client \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get install -y git
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN chmod +x kubectl
RUN mv ./kubectl /usr/local/bin/kubectl
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
ENV AWS_ACCESS_KEY_ID=***
ENV AWS_SECRET_ACCESS_KEY=***
RUN aws eks update-kubeconfig --name dhk-david --region us-east-1
# docker client
ENV DOCKER_CLIENT_VERSION=19.03.2 \
DOCKER_API_VERSION=1.40
RUN curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_CLIENT_VERSION}.tgz \
| tar -xzC /usr/local/bin --strip=1 docker/docker
# digdag server
ENV DIGDAG_VERSION=0.9.41
RUN curl -o /usr/local/bin/digdag --create-dirs -L https://dl.digdag.io/digdag-${DIGDAG_VERSION} && \
chmod +x /usr/local/bin/digdag
COPY digdag/server.properties /etc/digdag/server.properties
COPY digdag/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
EXPOSE 65432 65433
ENTRYPOINT ["/usr/local/bin/entrypoint.sh","/usr/local/bin/digdag","server","--config","/etc/digdag/server.properties"]
** digdag sever.properties
server.bind = 0.0.0.0
server.port = 65432
server.admin.bind = 0.0.0.0
server.admin.port = 65433
server.access-log.pattern = json
database.type = postgresql
database.user = digdag
database.password = digdag
database.host = postgresql
database.port = 5432
database.database = digdag
digdag.secret-encryption-key = MDEyMzQ1Njc4OTAxMjM0NQ==