쉼터

https://towardsdatascience.com/big-data-managing-the-flow-of-data-with-apache-nifi-and-apache-kafka-af674cd8f926 Apache NiFi And Kafka Docker Example An example of how to publish data to kafka docker container using a nifi processor. towardsdatascience.com

https://eyeballs.tistory.com/410 [NiFi] standalone NiFi 에 security 적용하는 방법(OIDC) google OAuth2 를 이용하여 NiFi 를 암호화하는 방법에 대해 설명한다. NiFi 에서는 OIDC(OpenID Connect) 라고 부르는가 봄. Apache NiFi 가 기본 설정값으로 설치되었다면, NiFi WebUI 에는 누구나 접근이 가능.. eyeballs.tistory.com

https://github.com/m-hogue/nifi-k8s m-hogue/nifi-k8s Simple Kubernetes YAMLs for deploying Apache NiFi and supporting components in k8s - m-hogue/nifi-k8s github.com https://grafana.com/grafana/dashboards/11387 NiFi Monitoring dashboard for Grafana General view of Nifi grafana.com ## 1.13.2 에는 spark-receiver 가 있다. docker run --name nifi \ -p 9090:9090 \ -d \ -e NIFI_WEB_HTTP_PORT='9090' \ apache..

https://github.com/dagster-io/dagster dagster-io/dagster A data orchestrator for machine learning, analytics, and ETL. - dagster-io/dagster github.com https://docs.dagster.io/deployment/guides/docker Dagster Docs A data orchestrator for machine learning, analytics, and ETL docs.dagster.io

https://aws.amazon.com/ko/blogs/security/mitigate-data-leakage-through-the-use-of-appstream-2-0-and-end-to-end-auditing/ Mitigate data leakage through the use of AppStream 2.0 and end-to-end auditing | Amazon Web Services Customers want to use AWS services to operate on their most sensitive data, but they want to make sure that only the right people have access to that data. Even when the right ..

from airflow import DAG from airflow.operators.python_operator import PythonOperator from datetime import datetime, timedelta import boto3, json, time glue_client = boto3.client('glue', region_name='us-east-1') default_args = { 'owner': 'SAM', 'start_date': datetime(2019, 5, 27) # days_ago(n=1) } default_schedule_interval = "0 15 * * *" # '@daily' def run_customer_job(random_base): job = glue_cl..

engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/ Playing around with Apache Airflow & BigQuery My Confession I have a confession….my crontab is a mess and it’s keeping me up at night….don’t worry, it’s not really keeping me up….but you might know what i me… engineering.pmc.com

gist.github.com/ddelange/6e33f8f0df3a97d4a371d055aa2d58ac Airflow Slack notifications Airflow Slack notifications. GitHub Gist: instantly share code, notes, and snippets. gist.github.com

laptrinhx.com/using-aws-cdk-to-deploy-your-amazon-managed-workflows-for-apache-airflow-environment-2530058942/ Using AWS CDK to deploy your Amazon Managed Workflows for Apache Airflow environment Using AWS CDK to deploy your Amazon Managed Workflows for Apache Airflow environment What better way to celebrate CDK Day tomorrow, than to return to a previous blog where I... laptrinhx.com github.com/..