俺的BLOG - Page 2 of 44 - 足迹，留给未来的自己

虚拟机VirtualBox中安装Centos7并配置Nginx

需要注意的点：

让虚拟机能访问外网，需要配置网络，使用NAT配置
centos7 中查看IP的命令： ip addr
centos7 中默认的防火墙不是iptables，而是firewalle，关闭方式： systemctl mask firewalld
安装Nginx前，要先安装EPEL库： sudo yum install epel-release

【参考】

虚拟机网络配置：https://www.jianshu.com/p/3585fecbfd4b
关闭防火墙：https://www.cnblogs.com/kreo/p/4368811.html
安装Nginx：https://blog.csdn.net/u011138533/article/details/53162499

python 通过SSL加密方式发送邮件

def send_email(mail_user, mail_password, mail_recipient, subject, body):
	FROM = mail_user
	TO = mail_recipient if type(mail_recipient) is list else [mail_recipient]
	SUBJECT = subject
	TEXT = body

	# Prepare actual message
	message = """From: %s\nTo: %s\nSubject: %s\n\n%s
	""" % (FROM, ", ".join(TO), SUBJECT, TEXT)
	try:
		server = smtplib.SMTP_SSL("mail.openedoo.org", 465)
		server.ehlo()
		#server.starttls()
		server.login(mail_user, mail_password)
		server.sendmail(FROM, TO, message)
		server.close()
		return 'successfully sent the mail'
	except Exception as e:
		return "failed to send mail"

def send_email(mail_user, mail_password, mail_recipient, subject, body):

FROM = mail_user

TO = mail_recipient if type(mail_recipient) is list else [mail_recipient]

SUBJECT = subject

TEXT = body

# Prepare actual message

message = """From: %s\nTo: %s\nSubject: %s\n\n%s

""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

try:

server = smtplib.SMTP_SSL("mail.openedoo.org", 465)

server.ehlo()

#server.starttls()

server.login(mail_user, mail_password)

server.sendmail(FROM, TO, message)

server.close()

return 'successfully sent the mail'

except Exception as e:

return "failed to send mail"

【摘自：】https://www.programcreek.com/python/example/6443/smtplib.SMTP_SSL

webdriver的close,quit,dispose

webDriver.Close() – Close the browser window that the driver has focus of
webDriver.Quit() – Calls Dispose()
webDriver.Dispose() Closes all browser windows and safely ends the session

【摘自：】https://stackoverflow.com/questions/15067107/difference-between-webdriver-dispose-close-and-quit

nginx——展示目录中的文件列表

在配置nginx的vhosts时，在http模块下添加以下配置，如

http{
    listen 80;
    
    autoindex on; #自动显示目录
    autoindex_exact_size off; #人性化方式显示文件大小否则以byte显示
    autoindex_localtime on; #按服务器时间显示，否则以gmt时间显示
}

http{

listen 80;

autoindex on; #自动显示目录

autoindex_exact_size off; #人性化方式显示文件大小否则以byte显示

autoindex_localtime on; #按服务器时间显示，否则以gmt时间显示

}

显示效果：

【参考】http://www.cnblogs.com/siqi/archive/2012/10/07/2713732.html

Python3中如何得到Unicode码对应的中文？

先检查text是什么类型

如果type(text) is bytes，那么

text.decode('unicode_escape')

				1
2

						text.decode('unicode_escape')

如果type(text) is str，那么

text.encode('latin-1').decode('unicode_escape')

				1

						text.encode('latin-1').decode('unicode_escape')

作者：mailto1587
链接：https://www.zhihu.com/question/26921730/answer/49625649
来源：知乎

python-pip3安装pycurl

若安装时遇到报ssl相关的错误(如下)

pip3 install pycurl
Collecting pycurl
  Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
    Complete output from command python setup.py egg_info:
    Using curl-config (libcurl 7.54.0)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 913, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 582, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 99, in __init__
        self.configure()
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 316, in configure_unix
        specify the SSL backend manually.''')
    __main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/

pip3 install pycurl

Collecting pycurl

Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz

Complete output from command python setup.py egg_info:

Using curl-config (libcurl 7.54.0)

Traceback (most recent call last):

File "<string>", line 1, in <module>

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 913, in <module>

ext = get_extension(sys.argv, split_extension_source=split_extension_source)

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 582, in get_extension

ext_config = ExtensionConfiguration(argv)

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 99, in __init__

self.configure()

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 316, in configure_unix

specify the SSL backend manually.''')

__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/

解决方法参考：

新版MAC：

pip3 uninstall pycurl# 卸載庫
export PYCURL_SSL_LIBRARY=openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include# openssl相關頭文檔路徑
pip3 install pycurl --compile --no-cache-dir # 重新編譯安裝

pip3 uninstall pycurl# 卸載庫

export PYCURL_SSL_LIBRARY=openssl

export LDFLAGS=-L/usr/local/opt/openssl/lib

export CPPFLAGS=-I/usr/local/opt/openssl/include# openssl相關頭文檔路徑

pip3 install pycurl --compile --no-cache-dir # 重新編譯安裝

参考链接：https://hk.saowen.com/a/431b34187c78c994d278c1a1a10d480bab83b28085468aca67ce9171ea8307fc

windows上考虑安装anaconda，通过anaconda来解决：

conda install -c anaconda pycurl

1	conda install -c anaconda pycurl

参考链接：https://anaconda.org/anaconda/pycurl

Python爬虫模拟人为操作

有些大厂，像BAT可能对爬虫这类检测做的很厉害，为了模拟人为操作，需要让脚步运行的慢一些。

以下python代码用于登录支付宝账号，为了更像真人操作，

录入账号密码时，每次录入一个字符都暂停若干时间;
且暂停时间是随机时间，不固定;
同时也在一些按钮上悬停若干时间后再执行点击事件。

import random
import time

from selenium import webdriver
from selenium.webdriver import ActionChains

url = "https://auth.alipay.com/login/index.htm"
name = "你的支付宝账号"
password = "支付宝登录密码"


# 缓慢输入内容
def send_keys_delay_random(controller, keys, min_delay=0.5, max_delay=1.5):
    for key in keys:
        controller.send_keys(key)
        time.sleep(random.uniform(min_delay, max_delay))

# 延迟随机时间
def delay_time(min_delay=0.5, max_delay=1):
    time.sleep(random.uniform(min_delay, max_delay))

print("启动浏览器，打开页面")
driver = webdriver.Chrome()
driver.get(url)
delay_time()

print("选择账号密码登录")
chose_btn = driver.find_element_by_xpath('//*[@id="J-loginMethod-tabs"]/li[2]')
ActionChains(driver).move_to_element(chose_btn).click().perform()
delay_time()

print("输入账号")
username_input = driver.find_element_by_id("J-input-user")
username_input.clear()
send_keys_delay_random(username_input, name)
delay_time(1, 2)

print("输入密码")
password_input = driver.find_element_by_id("password_rsainput")
password_input.clear()
send_keys_delay_random(password_input, password)
delay_time()

print("点击登录")
btn_login = driver.find_element_by_id("J-login-btn")
ActionChains(driver).move_to_element(btn_login).click().perform()


driver.quit()
print("正常退出")

import random

import time

from selenium import webdriver

from selenium.webdriver import ActionChains

url = "https://auth.alipay.com/login/index.htm"

name = "你的支付宝账号"

password = "支付宝登录密码"

# 缓慢输入内容

def send_keys_delay_random(controller, keys, min_delay=0.5, max_delay=1.5):

for key in keys:

controller.send_keys(key)

time.sleep(random.uniform(min_delay, max_delay))

# 延迟随机时间

def delay_time(min_delay=0.5, max_delay=1):

time.sleep(random.uniform(min_delay, max_delay))

print("启动浏览器，打开页面")

driver = webdriver.Chrome()

driver.get(url)

delay_time()

print("选择账号密码登录")

chose_btn = driver.find_element_by_xpath('//*[@id="J-loginMethod-tabs"]/li[2]')

ActionChains(driver).move_to_element(chose_btn).click().perform()

delay_time()

print("输入账号")

username_input = driver.find_element_by_id("J-input-user")

username_input.clear()

send_keys_delay_random(username_input, name)

delay_time(1, 2)

print("输入密码")

password_input = driver.find_element_by_id("password_rsainput")

password_input.clear()

send_keys_delay_random(password_input, password)

delay_time()

print("点击登录")

btn_login = driver.find_element_by_id("J-login-btn")

ActionChains(driver).move_to_element(btn_login).click().perform()

driver.quit()

print("正常退出")

生产环境工具推荐

脱敏查数据库：https://redash.io/

代码部署：http://www.walle-web.io/

SQL提交审核平台：http://yearning.io/

让Mac的终端使用代理

查看终端使用的IP：

curl ip.cn

1	curl ip.cn

让终端使用已安装的shadowsocks的http代理服务：

1. 临时使用

启动SS后，在终端输入
export ALL_PROXY=socks5://127.0.0.1:1080

1 2	启动SS后，在终端输入 export ALL_PROXY=socks5://127.0.0.1:1080

2. 永久使用

在~/.bash_profile中配置：
export ALL_PROXY=socks5://127.0.0.1:1080

1 2	在~/.bash_profile中配置： export ALL_PROXY=socks5://127.0.0.1:1080

备注，shadowsocks默认代理的是本地的1080端口，直接执行以上命令即可。

【摘自：】http://blog.xiaozan.me/1179.html