쉼터

OpenLens 설치

Kyle79 — Mon, 1 Apr 2024 15:58:34 +0900

OpenLens Desktop는 hombrew로 설치가능하다.

brew install --cask openlens

기존의 node shell, pod shell 기능은 다음과 같이 확장하여 설치할 수 있다.

1. OpenLens - Extensions 메뉴

2. @alebcay/openlens-node-pod-menu 입력 후 설치

3. node shell, pod shell 설치 확인

메트릭) lens-metrics

- https://raw.githubusercontent.com/shilazi/openlens-lens-metrics/main/build/shilazi-openlens-lens-metrics-0.1.0.tgz

파일을 받은 후 파일 경로르 추가해서 Install

한글 인코딩 깨지는 경우

Kyle79 — Thu, 2 Mar 2023 10:10:41 +0900

# python common 
# -*- coding: utf-8 -*-

#python 2.x
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

#python <=3.3
import imp
imp.reload(module)
sys.setdefaultencoding('utf-8')

# For >= Python3.4:
import importlib
importlib.reload(module)
sys.setdefaultencoding('utf-8')

BrowserMob Proxy

Kyle79 — Sun, 11 Dec 2022 14:50:53 +0900

https://browsermob-proxy-py.readthedocs.io/en/stable/

Welcome to BrowserMob Proxy’s documentation! — BrowserMob Proxy 0.6.0 documentation

Docs » Welcome to BrowserMob Proxy’s documentation! Edit on GitHub Welcome to BrowserMob Proxy’s documentation! Python client for the BrowserMob Proxy 2.0 REST API. How to install BrowserMob Proxy is available on PyPI, so you can install it with pip:

browsermob-proxy-py.readthedocs.io

https://github.com/lightbody/browsermob-proxy

GitHub - lightbody/browsermob-proxy: A free utility to help web developers watch and manipulate network traffic from their AJAX

A free utility to help web developers watch and manipulate network traffic from their AJAX applications. - GitHub - lightbody/browsermob-proxy: A free utility to help web developers watch and manip...

github.com

import settings

from browsermobproxy import Server
from selenium.webdriver import DesiredCapabilities

config = settings.Config

server = Server(config.BROWSERMOB_PATH)
server.start()
proxy = server.create_proxy()

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % proxy.proxy)
chrome_options.add_argument('--headless')

capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True
capabilities['acceptInsecureCerts'] = True

driver = webdriver.Chrome(options=chrome_options,
    desired_capabilities=capabilities,
   executable_path=config.CHROME_PATH)

proxy.new_har("sahibinden", options={'captureHeaders': True})
driver.get("https://www.sahibinden.com/en")

entries = proxy.har['log']["entries"]
for entry in entries:
    if 'request' in entry.keys():
        print(entry['request']['url'])
        print(entry['request']['headers'])
        print('\n')

proxy.close()
driver.quit()

Selenium Wire

Kyle79 — Sun, 11 Dec 2022 14:43:18 +0900

https://pypi.org/project/selenium-wire/

selenium-wire

Extends Selenium to give you the ability to inspect requests made by the browser.

pypi.org

for s in chrome.get_requests():
    if s.response and s.response.status_code == 200
        if 'gzip' in s.response.headers.get('Content-Encoding', '').lower():
            api_response_body = gzip.decompress(s.response.body)
            set_api_headers = dict(s.headers)
        else:
            api_response_body = s.response.body

User-agent generator

Kyle79 — Sun, 11 Dec 2022 14:24:41 +0900

pip install user-agent

https://domdom.tistory.com/entry/%ED%81%AC%EB%A1%A4%EB%A7%81-%ED%8C%8C%EC%9D%B4%EC%8D%AC-%ED%81%AC%EB%A1%A4%EB%A7%81-%EC%8B%9C-%EA%B8%B0%EA%B8%B0-%EC%B0%A8%EB%8B%A8-%EC%9A%B0%ED%9A%8C%ED%95%98%EA%B8%B0-fake-useragent-%EC%82%AC%EC%9A%A9%EB%B2%95user-agent-%EC%82%AC%EC%9A%A9%EB%B2%95

[크롤링] 파이썬 크롤링 시 기기 차단 우회하기 (fake-useragent 사용법/user-agent 사용법)

개요 파이썬에서 웹 크롤러를 만들 때 가장 큰 어려움은 역시 웹 사이트에서의 크롤러 또는 자동화 도구의 차단인 것 같습니다. 이런 경우 보통 time.sleep(3) 과 같이 몇 초간 크롤러를 쉬도록 하는

domdom.tistory.com

Selenium Wire Option

Kyle79 — Sun, 11 Dec 2022 14:22:06 +0900

d = DesiredCapabilities.CHROME
d['loggingPrefs'] = {'browser': 'ALL'}
d['acceptSslCerts'] = True

# Chrome mini
chromium = '/home/usr/chrome/headless-chromium'
if os.path.exists(chromium):
    chrome_options.binary_location = chromium
    
if not os.path.exists(chrome_path):
    chrome_path = '/usr/local/bin/chromedriver'
    
if wired:
    sw_options = { 
        # 'request_storage': 'memory',
        # 'request_storage_max_size': 100,
        # 'ignore_http_methods': [],
    }
    
    if len(proxy_list) > 0:
        proxy = random.choice(proxy_list)
        ip = proxy['ip']
        port = proxy['port']
        sw_options['proxy'] = {
            'http': f'http://{ip}:{port}', 
            'https': f'http://{ip}:{port}', 
            'no_proxy': 'localhost,127.0.0.1'
        }
        
    chrome = wired_webdriver.Chrome(chrome_path, options=chrome_options, desired_capabilities=d, seleniumwire_options=sw_options)    
else:
    chrome = webdriver.Chrome(chrome_path, options=chrome_options, desired_capabilities=d)    

chrome.implicitly_wait(timeout)

Selenium Chrome Option

Kyle79 — Sun, 11 Dec 2022 14:19:42 +0900

options = webdriver.ChromeOptions()
options.add_argument('--kiosk')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument('--hide-scrollbars')
# options.add_argument('--single-process')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--auto-open-devtools-for-tabs')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--dns-prefetch-disable')
options.add_argument("enable-automation")
options.add_argument("--disable-browser-side-navigation")
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--ignore-ssl-errors')
options.add_argument("--ignore_ssl")

options.add_argument("--disable-application-cache")
options.add_argument("--disable-crash-reporter")
options.add_argument("--disable-in-process-stack-traces")
options.add_argument("--output=/dev/null")
options.add_argument('--disable-features=IsolateOrigins,site-per-process')

options.add_argument("--lang=en_US.UTF-8")
options.add_argument("--language=en_US:en")
options.add_argument("--lc_all=en_US.UTF-8")

options.add_argument("--homedir=/tmp")
options.add_argument("--disk-cache-dir=/tmp/cache-dir")
options.add_argument("--data-path=/tmp/data-path")
options.add_argument("--user-data-dir=/tmp/user-data")
options.add_argument("--disable-software-rasterizer")
options.add_argument('--disable-logging')
options.add_argument('--log-level=3')
options.add_argument('--v=99')

options.add_argument('--no-zygote')
options.add_argument('--no-startup-window')
options.add_argument("--disable-session-crashed-bubble")
options.add_argument("--disable-ipv6")
options.add_argument("--allow-http-screen-capture")
options.add_argument('--vmodule')
# options.add_argument('--remote-debugging-port=9222')
options.add_argument("--disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--incognito")
options.add_argument("--disable-popup-blocking")
options.add_argument("--excludeSwitches")
options.add_argument("--disable-default-app")
# options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--allow-cross-origin-auth-prompt")
options.add_argument("--allow-running-insecure-content")
options.add_argument('--allow-insecure-localhost')

options.add_argument("--allow-file-access")
options.add_argument("--allow-file-access-from-files")

options.add_argument('--load-images=false')
options.add_argument('--proxy-type=http')
options.add_argument('--ssl-protocol=any')

options.add_argument("start-fullscreen")
options.add_argument("start-maximized")

prefs = {
"intl.accept_languages": "en,en_US",
"profile.managed_default_content_settings.images":2,
"profile.default_content_setting_values.notifications":2,
"profile.managed_default_content_settings.stylesheets":2,
"profile.managed_default_content_settings.cookies":2,
"profile.managed_default_content_settings.javascript":1,
"profile.managed_default_content_settings.plugins":1,
"profile.managed_default_content_settings.popups":2,
"profile.managed_default_content_settings.geolocation":2,
"profile.managed_default_content_settings.media_stream":2,
"credentials_enable_service": False,
"profile.password_manager_enabled": False,
}
options.add_experimental_option("prefs", prefs)
options.add_experimental_option("detach", True)
options.add_experimental_option('excludeSwitches', ['enable-automation'])

ChromeDriver & Selenium Wire setting

Kyle79 — Sun, 11 Dec 2022 13:52:46 +0900

Mac

curl -SL https://chromedriver.storage.googleapis.com/107.0.5304.62/chromedriver_mac64.zip > chromedriver.zip

/bin/echo "y" | unzip chromedriver.zip -d macbin/

chmod 755 macbin/chromedriver

cp macbin/chromedriver /usr/local/bin/chromedriver

rm chromedriver.zip

rm -rf macbin

IDC

# Step1

sudo apt-get install -y libfontconfig1 unzip libnss3 libx11-6 libx11-xcb1

sudo apt-get -f install

# Step2

curl -kLo google-chrome-stable_current_amd64.deb -x amazon.proxy.crawling.coupang.net:8080 https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \

&& sudo dpkg -i ./google-chrome-stable_current_amd64.deb \

&& rm ./google-chrome-stable_current_amd64.deb

mkdir -p /home/usr/chrome/ \

&& curl -kLo chromedriver.zip https://chromedriver.storage.googleapis.com/108.0.5359.22/chromedriver_linux64.zip -x amazon.proxy.crawling.coupang.net:8080 \

&& unzip chromedriver.zip \

&& mv chromedriver /home/usr/chrome/chromedriver \

&& chmod 0755 /home/usr/chrome/chromedriver \

&& rm chromedriver.zip

# Step3

pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org setuptools-rust

pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip

pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org pyOpenSSL==22.0.0

pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org selenium==3.141.0

pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org selenium-wire==4.6.5

Docker

FROM ubuntu:18.04

RUN apt clean && apt-get update

RUN apt-get install -y apt-utils

RUN apt-get -y upgrade

RUN apt-get install -y apt-transport-https \

ca-certificates \

gnupg2 \

wget \

software-properties-common \

python-pip \

python-dev \

python3-dev \

python3-pip \

python-setuptools \

apt-transport-https \

build-essential \

default-libmysqlclient-dev \

apt-utils \

sudo \

lxc \

curl \

vim \

ca-certificates \

redis-tools \

ENV PYTHONIOENCODING utf-8

RUN apt-get update && apt-get install -y libfontconfig1 unzip libnss3 libx11-6 libx11-xcb1

RUN mkdir -p /home/usr/chrome

RUN curl -kLo chromedriver.zip https://chromedriver.storage.googleapis.com/86.0.4240.22/chromedriver_linux64.zip \

&& unzip chromedriver.zip \

&& mv chromedriver /home/usr/chrome/chromedriver \

&& chmod 0755 /home/usr/chrome/chromedriver \

&& rm chromedriver.zip

RUN curl -kLo headless-chromium.zip https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-57/stable-headless-chromium-amazonlinux-2.zip \

&& unzip headless-chromium.zip \

&& mv headless-chromium /home/usr/chrome/headless-chromium \

&& chmod 0755 /home/usr/chrome/headless-chromium \

&& rm headless-chromium.zip

RUN pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org setuptools-rust

RUN pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip

RUN pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org pyOpenSSL==22.0.0

RUN pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org selenium==3.141.0

RUN pip3 install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org selenium-wire==4.6.5

osx - brew 디스크 정리

Kyle79 — Sun, 24 Apr 2022 12:44:57 +0900

1> Cleanup

brew cleanup

2> Bundle

brew bundle dump --global --force
rm -rf /usr/local/Homebrew/Cellar
brew bundle --global

https://velog.io/@iamchanii/Brewfile%EC%9D%84-%EC%9D%B4%EC%9A%A9%ED%95%B4%EC%84%9C-%ED%8C%80-%EA%B0%9C%EB%B0%9C-%ED%99%98%EA%B2%BD-%EB%A7%8C%EB%93%A4%EA%B8%B0

Brewfile을 이용해서 팀 개발 환경 만들기

제 블로그와 회사 기술 블로그에 올렸던 글을 가져왔습니다. 플러스티브이 개발팀은 도커를 이용해서 개발 환경을 맞춘 다음 로컬 환경에서 개발합니다. 원할한 협업을 위해 개발 환경을 설정

velog.io

Hive Sql (Join 및 최적화)

Kyle79 — Sat, 16 Apr 2022 15:58:06 +0900

https://kimeunyeol.tistory.com/34

HIVE SQL - JOIN 이해(공식문서)

Hive Joins Hive Joins Join Syntax Examples MapJoin Restrictions Join Optimization Predicate Pushdown in Outer Joins Join Syntax Hive에서 제공하는 Join구문 join_table: table_reference [INNER] JOIN ta..

kimeunyeol.tistory.com

http://fantasio.life/2021/06/16/hive-join-tip/

(hive) 조인 관련 tip 정리 · Keep going..

힌트 존재 - 어떤 테이블을 마지막으로 흘려보낼야 하는지 지정해준다. /*+ STREAMTABLE(alias) */

fantasio.life

https://sungwookkang.com/m/1456

Hive JOIN 및 조인 최적화

Hive JOIN 및 조인 최적화 · Version : Hive Hive는 RDBMS와 같은 JOIN(조인) 구문을 제공하며 기능도 거의 유사하지만 동등조인(EQUI-JOIN)만 제공한다. 내부조인(INNER JOIN)의 경우 조인하는 모든..

sungwookkang.com

https://royzero.tistory.com/m/50

[SQL] WITH절 사용하기

SQL문을 작성하다 보면 같은 Query구문을 반복해서 작성하는 경우가 발생합니다. 이런 경우 보통 1~2번의 반복은 개발자 입장에서 작성하지만, 잦은 반복 또는 동일 구문에 대한 조건 등의 추가로

royzero.tistory.com