Home > News content

Google open source robots.txt parser

via:博客园     time:2019/7/2 22:31:30     readed:1423

Google said the past 25 yearsRobots Exclusion Protocol (REP)The protocol has always been a standard of agreement, bringing a lot of uncertainty to webmasters and crawler tool developers.

Google nowAnnounceIt will be the first to focus on making REP an industry standard, and as part of this effort, it has open sourced its own robots.txt parser.Source codeHosted on GitHub with an Apache License 2.0 license. The robots.txt parser is a C++ library for parsing and matching rules in a robots.txt file. It has been around for 20 years and contains code written in the 1990s.

China IT News APP

Download China IT News APP

Please rate this news

The average score will be displayed after you score.

Post comment

Do not see clearly? Click for a new code.

User comments