python Archives - 俺的BLOG

Scrapy爬取内容去重

使用以下pipeline即可：

from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

    def __init__(self):
        self.ids_seen = set()

    def process_item(self, item, spider):
        if item['id'] in self.ids_seen:
            raise DropItem("Duplicate item found: %s" % item)
        else:
            self.ids_seen.add(item['id'])
            return item

from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

def __init__(self):

self.ids_seen = set()

def process_item(self, item, spider):

if item['id'] in self.ids_seen:

raise DropItem("Duplicate item found: %s" % item)

else:

self.ids_seen.add(item['id'])

return item

【摘自：】https://docs.scrapy.org/en/latest/topics/item-pipeline.html#duplicates-filter

python 通过SSL加密方式发送邮件

def send_email(mail_user, mail_password, mail_recipient, subject, body):
	FROM = mail_user
	TO = mail_recipient if type(mail_recipient) is list else [mail_recipient]
	SUBJECT = subject
	TEXT = body

	# Prepare actual message
	message = """From: %s\nTo: %s\nSubject: %s\n\n%s
	""" % (FROM, ", ".join(TO), SUBJECT, TEXT)
	try:
		server = smtplib.SMTP_SSL("mail.openedoo.org", 465)
		server.ehlo()
		#server.starttls()
		server.login(mail_user, mail_password)
		server.sendmail(FROM, TO, message)
		server.close()
		return 'successfully sent the mail'
	except Exception as e:
		return "failed to send mail"

def send_email(mail_user, mail_password, mail_recipient, subject, body):

FROM = mail_user

TO = mail_recipient if type(mail_recipient) is list else [mail_recipient]

SUBJECT = subject

TEXT = body

# Prepare actual message

message = """From: %s\nTo: %s\nSubject: %s\n\n%s

""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

try:

server = smtplib.SMTP_SSL("mail.openedoo.org", 465)

server.ehlo()

#server.starttls()

server.login(mail_user, mail_password)

server.sendmail(FROM, TO, message)

server.close()

return 'successfully sent the mail'

except Exception as e:

return "failed to send mail"

【摘自：】https://www.programcreek.com/python/example/6443/smtplib.SMTP_SSL

webDriver.Close() – Close the browser window that the driver has focus of webDriver.Quit() – Calls Dispose() webDriver.Dispose() Closes all browser windows and safely ends the session 【摘自：】https://stackoverflow.com/questio …

Python3中如何得到Unicode码对应的中文？

先检查text是什么类型如果type(text) is bytes，那么

text.decode('unicode_escape')

1 2	text.decode('unicode_escape')

如果type(text) is str，那么

text.encode('latin-1').decode('unicode_escape')

1	text.encode('latin-1').decode('unicode_escape')

作者：mailto1587 链接：https://www.zhihu.com/question/26921730/answer/496 …

python-pip3安装pycurl

若安装时遇到报ssl相关的错误(如下)

pip3 install pycurl
Collecting pycurl
  Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
    Complete output from command python setup.py egg_info:
    Using curl-config (libcurl 7.54.0)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 913, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 582, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 99, in __init__
        self.configure()
      File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 316, in configure_unix
        specify the SSL backend manually.''')
    __main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/

pip3 install pycurl

Collecting pycurl

Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz

Complete output from command python setup.py egg_info:

Using curl-config (libcurl 7.54.0)

Traceback (most recent call last):

File "<string>", line 1, in <module>

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 913, in <module>

ext = get_extension(sys.argv, split_extension_source=split_extension_source)

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 582, in get_extension

ext_config = ExtensionConfiguration(argv)

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 99, in __init__

self.configure()

File "/private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/setup.py", line 316, in configure_unix

specify the SSL backend manually.''')

__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/rt/r5cq9kls5qngd41nrn9bx6w40000gn/T/pip-install-01_xgi2x/pycurl/

解决方法参考：新版MAC：

pip3 uninstall pycurl# 卸載庫
export PYCURL_SSL_LIBRARY=openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include# openssl相關頭文檔路徑
pip3 install pycurl --compile --no-cache-dir # 重新編譯安裝

pip3 uninstall pycurl# 卸載庫

export PYCURL_SSL_LIBRARY=openssl

export LDFLAGS=-L/usr/local/opt/openssl/lib

export CPPFLAGS=-I/usr/local/opt/openssl/include# openssl相關頭文檔路徑

pip3 install pycurl --compile --no-cache-dir # 重新編譯安裝

参考链接：https://hk.saowen.com/a/431b34187c78c994d278c1a1a10d480bab83b28085468aca67ce9171ea8307fc &n …

分类存档：python

Scrapy爬取内容去重

python 通过SSL加密方式发送邮件

webdriver的close,quit,dispose

Python3中如何得到Unicode码对应的中文？

python-pip3安装pycurl

Python爬虫模拟人为操作

python——flask提供WebService时实现多种返回类型

Python——如何迁移（部署）项目

使用Python制作爬虫并提供WebService接口