digdag

 

 

https://tech.gamevilcom2us.com/blog/3922

Google 클라우드 플랫폼 기반 빅데이터 시스템 구축기

 

 

 

https://docs.google.com/presentation/d/11C_BKio0DZIop_ZjJk7ogxQtWV5qHIr-hHjw277z64k/htmlpresent

그들이 AWS 위에서 데이터 파이프라인을 운영하는법 (2019)

 

 

 

 

 

https://qiita.com/shiozaki/items/f79eecf8e1878aa64a40

DigdagとEmbulkを利用してBigQueryにRDS（Aurora、MySQL）のマスタデータを同期する - Qiita

 

 

 

 

https://gist.github.com/hiroyuki-sato/48f1119d8e7436a1014c1958916cf0b8

digdag embulk example

 

 

 

https://www.digdag.io/

Digdag – Open Source Workflow Engine for the Multi-Cloud Era

 

 

https://github.com/suaaa7/digdag-server

suaaa7/digdag-server

 

 

https://docs.digdag.io/operators/bq.html

bq>: Running Google BigQuery queries — Digdag 0.9.41 documentation

 

 

https://wapa5pow.com/build-analytics-environment-on-eks/

EKS(Kubernetes)上にDigdag・Embulk・Redashで分析環境を構築する - Koichi Ishida blog

 

 

https://github.com/septeni-original/k8s_job

septeni-original/k8s_job

 

 

** digdag docker-compose

version: '3'

services:
  digdag:
    container_name: digdag-server
    build:
      context: .
      dockerfile: digdag/Dockerfile
    depends_on:
      - postgresql
    command: --task-log ./task_log
    ports:
      - "65432:65432"
      - "65433:65433"
    volumes:
      - /tmp:/tmp
      - ./digdag/projects:/opt
      - ./digdag/server.properties:/etc/digdag/server.properties
  postgresql:
    container_name: postgresql
    image: postgres:9.6.17
    environment:
      - POSTGRES_USER=digdag
      - POSTGRES_PASSWORD=digdag
      - POSTGRES_DB=digdag
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

 

** digdag Dockerfile

FROM openjdk:8-jdk

ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

RUN apt-get update && apt-get install -y --no-install-recommends \
  curl gettext-base postgresql-client \
  && rm -rf /var/lib/apt/lists/*

RUN apt-get install -y git
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN chmod +x kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

ENV AWS_ACCESS_KEY_ID=***
ENV AWS_SECRET_ACCESS_KEY=***
RUN aws eks update-kubeconfig --name dhk-david --region us-east-1

# docker client
ENV DOCKER_CLIENT_VERSION=19.03.2 \
    DOCKER_API_VERSION=1.40
RUN curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_CLIENT_VERSION}.tgz \
  | tar -xzC /usr/local/bin --strip=1 docker/docker

# digdag server
ENV DIGDAG_VERSION=0.9.41
RUN curl -o /usr/local/bin/digdag --create-dirs -L https://dl.digdag.io/digdag-${DIGDAG_VERSION} && \
  chmod +x /usr/local/bin/digdag

COPY digdag/server.properties /etc/digdag/server.properties
COPY digdag/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

EXPOSE 65432 65433

ENTRYPOINT ["/usr/local/bin/entrypoint.sh","/usr/local/bin/digdag","server","--config","/etc/digdag/server.properties"]

 

** digdag sever.properties

server.bind = 0.0.0.0
server.port = 65432
server.admin.bind = 0.0.0.0
server.admin.port = 65433
server.access-log.pattern = json

database.type = postgresql
database.user = digdag
database.password = digdag
database.host = postgresql
database.port = 5432
database.database = digdag

digdag.secret-encryption-key = MDEyMzQ1Njc4OTAxMjM0NQ==