俺的BLOG - Page 2 of 45 - 足迹，留给未来的自己

Scrapy爬取内容去重

使用以下pipeline即可：

from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

    def __init__(self):
        self.ids_seen = set()

    def process_item(self, item, spider):
        if item['id'] in self.ids_seen:
            raise DropItem("Duplicate item found: %s" % item)
        else:
            self.ids_seen.add(item['id'])
            return item

from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

def __init__(self):

self.ids_seen = set()

def process_item(self, item, spider):

if item['id'] in self.ids_seen:

raise DropItem("Duplicate item found: %s" % item)

else:

self.ids_seen.add(item['id'])

return item

【摘自：】https://docs.scrapy.org/en/latest/topics/item-pipeline.html#duplicates-filter

当你写爬虫抓不到APP请求包的时候该怎么办？

原文链接：https://zhuanlan.zhihu.com/p/46433599

一步步教你Android反编译

原文链接：https://juejin.im/post/5aed9694518825673e358436

虚拟机的centos系统磁盘占用满解决方法

主要是清理占用空间较多的文件夹，如log目录；若磁盘空间分配上不合理，也可以考虑调整磁盘分配。

参考：

https://www.linuxidc.com/Linux/2018-01/150515.htm

https://blog.csdn.net/e_wsq/article/details/79531493

安装swoole

若使用pecl或编译安装swoole时，出现如下类似错误，

undefined symbol: __builtin_saddl_overflow

1	undefined symbol: __builtin_saddl_overflow

则需要升级下gcc(达到5.0以上版本)：

sudo yum install -y yum-utils  centos-release-scl
sudo yum -y --enablerepo=centos-sclo-rh-testing install devtoolset-7-gcc
echo "source /opt/rh/devtoolset-7/enable" | sudo  tee -a /etc/profile
source /opt/rh/devtoolset-7/enable
gcc --version

sudo yum install -y yum-utils centos-release-scl

sudo yum -y --enablerepo=centos-sclo-rh-testing install devtoolset-7-gcc

echo "source /opt/rh/devtoolset-7/enable" | sudo tee -a /etc/profile

source /opt/rh/devtoolset-7/enable

gcc --version

【摘自：】https://forum.phalconphp.com/discussion/17595/problem-with-compiled-phalcon-with-php-721

安装swoole时可能出现的错误

错误内容类似如下：

define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)

1	define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)

解决方式：

sudo yum install centos-release-scl
sudo yum install devtoolset-7
scl enable devtoolset-7 bash

sudo yum install centos-release-scl

sudo yum install devtoolset-7

scl enable devtoolset-7 bash

【摘自：】https://github.com/swoole/swoole-src/issues/2119

PS, 用 pecl install swoole 时，使用默认安装配置项比较容易成功

CentOS 7安装Shadowsocks客户端实现联网

链接：

1. https://i.jakeyu.top/2017/03/16/centos使用SS翻墙/

2. https://xeylon.com/server/140.html

备注：

1. 如果删除了privoxy，提示无法联网，可执行以下命令取消网络代理设置

unset http_proxy;
unset https_proxy;

1 2	unset http_proxy; unset https_proxy;

【摘自】：http://blog.sina.com.cn/s/blog_5f54f0be0102v4tu.html

2. 运行以下命令查看SS服务器的IP地址：

curl --socks5 127.0.0.1:1080 http://httpbin.org/ip

1	curl --socks5 127.0.0.1:1080 http://httpbin.org/ip

【摘自：】https://xeylon.com/server/140.html

3. What’s the difference between /usr/lib/systemd/system and /etc/systemd/system?

4. 使用 service privoxy start 或 service sslocal start 失败或无法开机自启：https://www.cnblogs.com/liuxuzzz/p/5329536.html

yum安装PHP7

The CentOS 7 official software repositories have PHP 5.4 which has reached the end of life and no longer actively maintained by the developers.

To keep up with the latest features and security updates, you need a newer (probably the latest) version of PHP on your CentOS 7 system.

For the purpose of this guide, we will be operating the system as root, if that is not the case for you, make use of the sudo command to acquire root privileges.

Installing PHP 7 on CentOS 7

1. To install PHP 7, you have to install and enable EPEL and Remi repository on your CentOS 7 system with the commands below.

# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm

1 2	# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm

2. Next, you need to install yum-utils, a collection of useful programs for managing yum repositories and packages. It has tools that basically extend yum’s default features.

It can be used for managing (enabling or disabling) yum repositories as well as packages without any manual configuration and so much more.

# yum install yum-utils

1	# yum install yum-utils

3. One of the programs provided by yum-utils is yum-config-manager, which you can use to enable Remirepository as the default repository for installing different PHP versions as shown.

# yum-config-manager --enable remi-php70   [<strong>Install PHP 7.0</strong>]

1	# yum-config-manager --enable remi-php70 [<strong>Install PHP 7.0</strong>]

If you want to install PHP 7.1, PHP 7.2 or PHP 7.3 on CentOS 7, just enable it as shown.

# yum-config-manager --enable remi-php71   [<strong>Install PHP 7.1</strong>]
# yum-config-manager --enable remi-php72   [<strong>Install PHP 7.2</strong>]
# yum-config-manager --enable remi-php73   [<strong>Install PHP 7.3</strong>]

# yum-config-manager --enable remi-php71 [Install PHP 7.1]

# yum-config-manager --enable remi-php72 [Install PHP 7.2]

# yum-config-manager --enable remi-php73 [Install PHP 7.3]

4. Now install PHP 7 with all necessary modules with the command below.

# yum install php php-mcrypt php-cli php-gd php-curl php-mysql php-ldap php-zip php-fileinfo

1	# yum install php php-mcrypt php-cli php-gd php-curl php-mysql php-ldap php-zip php-fileinfo

Afterwards, double check the installed version of PHP on your system.

# php -v

# php -v

Check PHP Version in CentOS 7

【原文：】https://www.tecmint.com/install-php-7-in-centos-7/

PhpStorm中terminal修改为git bash后git log中文显示问题修复

找到GitBash安装目录，找到Git\etc\bash.bashrc文件（如：D:\Program Files\Git\etc\bash.bashrc），在文件末尾添加：

<span class="hljs-comment"># support chinese</span>
<span class="hljs-built_in">export</span> LANG=<span class="hljs-string">"zh_CN.UTF-8"</span>
<span class="hljs-built_in">export</span> LC_ALL=<span class="hljs-string">"zh_CN.UTF-8"</span>

# support chinese

export LANG="zh_CN.UTF-8"

export LC_ALL="zh_CN.UTF-8"

在PhpStorm设置 Tools->Terminal->Shell path 为如： D:\Program Files\Git\bin\bash.exe 这样的路径。
重新打开Terminal即可生效。

【原文：】https://xeylon.com/tools/136.html

YII2中 ActiveForm里的field怎么可以输出时间而不是时间戳？

class XXX extends \yii\db\ActiveRecord
{
    public function rules()
    {
        return [
        
            ...other rules...
            
            ['time', function($attr, $params) {
                if ($this->hasErrors()) return false;

                $datetime = $this->{$attr};
                
                $time = strtotime($datetime);
                // 验证时间格式是否正确
                if ($time === false) {
                    $this->addError($attr, '时间格式错误.');
                    return false;
                }
                // 将转换为时间戳后的时间赋值给time属性
                $this->{$attr} = $time;
                return true;
            }],
        ];
    }

    ...others...
    
    /**
     * 从数据库中 find() 数据后将 time 格式化。
     */
    public function afterFind()
    {
        parent::afterFind();
        $this->time = date('Y-m-d', $this->time);
    }

class XXX extends \yii\db\ActiveRecord

{

public function rules()

{

return [

...other rules...

['time', function($attr, $params) {

if ($this->hasErrors()) return false;

$datetime = $this->{$attr};

$time = strtotime($datetime);

// 验证时间格式是否正确

if ($time === false) {

$this->addError($attr, '时间格式错误.');

return false;

}

// 将转换为时间戳后的时间赋值给time属性

$this->{$attr} = $time;

return true;

}],

];

}

...others...

/**

* 从数据库中 find() 数据后将 time 格式化。

public function afterFind()

{

parent::afterFind();

$this->time = date('Y-m-d', $this->time);

}

【摘自：https://segmentfault.com/q/1010000009408189】