* 카프카 + 주피커
https://miiingo.tistory.com/196
[Apache Kafka] Docker를 이용한 zookeeper, kafka 실행
Docker를 이용한 zookeeper, kafka 실행 기본 개념 Apache Kafka Apache Kafka란 LinkedIn에서 개발된 분산 메시징 시스템으로, 대용량의 실시간 로그 처리에 특화된 아키텍처 설계를 통해 기존 메시징 시스템보다..
miiingo.tistory.com
* 카프카(클러스터링) + 매니저 + 주피커 + 도커
https://gist.github.com/dkurzaj/2a899de8cb5ae698919f0a9bbf7685f0
Docker compose Kafka, Zookeeper and Kafka manager
Docker compose Kafka, Zookeeper and Kafka manager. GitHub Gist: instantly share code, notes, and snippets.
gist.github.com
* Test
netstat -an |grep '2181\|9092\|9000'
./kafka-topics.sh --zookeeper 121.252.119.196:2181 --create --topic scott --partitions 1 --replication-factor 1
./kafka-topics.sh --zookeeper 121.252.119.196:2181 --list --topic scott
./kafka-topics.sh --zookeeper 121.252.119.196:2181 --describe
./kafka-console-producer.sh --broker-list 121.252.119.196:9092 --topic scott
./kafka-console-consumer.sh --bootstrap-server 121.252.119.196:9092 --topic scott --from-beginning
* Manager
- Access Kafka Manager : http://localhost:9000/.
- Add a new cluster.
- Name it as you like (**Localtest** for example).
- IP : localhost:2181 or ${IP}:2181 ex) 13.209.87.141:2181
- Kafka version (corresponding here to the Kafka version inside the image : **wurstmeister/kafka:1.0.0**). But as I write this Gist, the latest Kafka version available in Kafka Manager is **0.11.0.0** so I select this one, but it's sufficiently compatible with the 1.0.0 version of Kafka according to this topic: https://github.com/yahoo/kafka-manager/issues/451
- Tick **Enable JMX Polling** in order to see the metrics of the topics
- Tick **Poll consumer information** to know the consumer of a topic (it may not work)
- Tick **Enable Active OffsetCache** to see the offsets
* 카프카-드루이드-슈퍼셋
https://blog.godatadriven.com/divolte-kafka-druid-superset
GoDataDrivenBlog
Real time analytics: Divolte + Kafka + Druid + Superset In today's world you want to learn from your customers as fast as possible. This blog gives an introduction to setting up streaming analytics using open source technologies. We'll use Divolte, Kafka,
blog.godatadriven.com
* 카프카 파이선
https://github.com/dpkp/kafka-python/
dpkp/kafka-python
Python client for Apache Kafka. Contribute to dpkp/kafka-python development by creating an account on GitHub.
github.com
* Kafka Cluster 구축하고 python으로 테스트
http://hellowuniverse.com/2018/01/21/
2018년 January 21일 | H3ll0 Un1v3rse
Kafka® 는 대용량 데이터를 처리하기에 좋은 도구 입니다. horizontally scalable, fault-tolerant, wicked fast하며 수 천개의 회사에서 사용중이죠! 이번 테스트는 zookeeper server를 별도로 구성하지 않고 kafka에 built-in된 zookeeper를 사용하였습니다. 1> Base Environment Ubuntu 16.04 Server – Kafka # ip setting Kafka 1: 192.1
hellowuniverse.com
* docker-kafka-ssl
https://github.com/orefalo/docker-kafka-ssl
orefalo/docker-kafka-ssl
Kafka 2 way SSL setup. Contribute to orefalo/docker-kafka-ssl development by creating an account on GitHub.
github.com
* 카프카 S3 연동
https://swalloow.github.io/kafka-connect
Swalloow Blog
{% if page.logo %}{% endif %} {% if page.navigation %} Menu {% endif %} {{ site.name }} {{ site.description }}...
swalloow.github.io
* 드루이드
http://icednut.github.io/2018/03/05/20180305-start-druid/
Icednut's Note
출처: 스칼라로 배우는 함수형 프로그래밍 5장 Chapter 05. 엄격성과 나태성이번 챕터에서 다룰 내용 List의 한계 Stream (Lazy List) 스트림 순회의 비엄격성과 나태성 List의 한계스칼라에서 다음 코드는 어떻게 수행될까? List(1,2,3,4).map(_ + 10).filter(_ % 2 == 0).map(_ * 3) 스칼라 List의 map과 filter 함수는 임시적인 목록을 만들어서 그 결과를 반환한... Read More
icednut.github.io
* 카프카 드루이드 연동
https://cleanprogrammer.net/loading-data-from-apache-kafka-to-druid/
Loading Data From Apache Kafka to Druid - Clean Programmer
This tutorial demonstrates how to load data streams from a Kafka topic to Druid, using the Kafka Indexing Service to build an realtime analytics stack.
cleanprogrammer.net
* 드루이드 S3 연동
http://druid.io/docs/latest/design/index.html
Druid | Apache Druid (incubating) Design
Table of Contents
* 드루이드 장고
https://github.com/nalabelle/druid-django
nalabelle/druid-django
Druid Backend for Django REST Framework. Contribute to nalabelle/druid-django development by creating an account on GitHub.
github.com
* 아호 코라식
https://pypi.org/project/pyahocorasick/
pyahocorasick
pyahocorasick is a fast and memory efficient library for exact or approximate multi-pattern string search. With the ahocorasick.Automaton class, you can find multiple key strings occurrences at once in some input text. You can use it as a plain dict-like T
pypi.org
'BigData' 카테고리의 다른 글
| Mysql 운용 (0) | 2019.05.30 |
|---|---|
| Docker 운용 (0) | 2019.05.29 |
| GitKraken 사용법 (0) | 2019.05.17 |
| Apache Kafka 가이드 (0) | 2019.05.13 |
| Apache Cassandra 가이드 (0) | 2019.05.13 |