Skip to main content

New open source robots.txt projects

 Last year we released the robots.txt parser and matcher that we use in our production systems to the open source world. Since then, we've seen people build new tools with it, contribute to the open source library (effectively improving our production systems- thanks!), and release new language versions like golang and rust, which make it easier for developers to build new tools.

With the intern season ending here at Google, we wanted to highlight two new releases related to robots.txt that were made possible by two interns working on the Search Open Sourcing team, Andreea Dutulescu and Ian Dolzhanskii. 

Robots.txt Specification Test

First, we are releasing a testing framework for robots.txt parser developers, created by Andreea. The project provides a testing tool that can validate whether a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. Currently there is no official and thorough way to assess the correctness of a parser, so Andreea built a tool that can be used to create robots.txt parsers that are following the protocol.

Java robots.txt parser and matcher

Second, we are releasing an official Java port of the C++ robots.txt parser, created by Ian. Java is the 3rd most popular programming language on GitHub and it's extensively used at Google as well, so no wonder it's been the most requested language port. The parser is a 1-to-1 translation of the C++ parser in terms of functions and behavior, and it's been thoroughly tested for parity against a large corpora of robots.txt rules. Teams are already planning to use the Java robots.txt parser in Google production systems, and we hope that you'll find it useful, too. 

As usual, we welcome your contributions to these projects. If you built something with the C++ robots.txt parser or with these new releases, let us know so we can potentially help you spread the word! If you found a bug, help us fix it by opening an issue on GitHub or directly contributing with a pull request. If you have questions or comments about these projects, catch us on Twitter!

It was our genuine pleasure to host Andreea and Ian, and we're sad that their internship is ending. Their contributions help make the Internet a better place and we hope that we can welcome them back to Google in the future.

Original Post:- https://webmasters.googleblog.com/2020/09/new-open-source-robotstxt-projects.html?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+blogspot%2FamDG+%28Official+Google+Webmaster+Central+Blog%29

Popular posts from this blog

Reliance Jio's new call charges, voucher, free data offer: 5 things to know

For the first time, Reliance Jio customers will have to pay for making voice calls For every ₹10 you spent on call charges, Jio will give you 1 GB free data Not all voice calls made from a Reliance Jio mobile phone connection will now be free but the telecom operator has promised to refund the charges in the form of free internet data. Even though the charges for making voice calls to other mobile phone networks are as low as 6 paise a minute, it will be for the first time that Jio customers will have to pay for voice calls, free since its launch in 2016. All voice calls made to Non-Jio mobile numbers will be charged at 6 paise per minute from your next recharge. Reliance Jio's new call charges, voucher, free data offer 5 things to know about Jio's call charges, data offer: 1) Jio said due to the undergoing regulatory policy changes for mobile call related charges, it is compelled to recover Interconnect Usage Charge or IUC for all Non-Jio mobile voice calls. It me...
Read more

The 4-step Creative Writing Process for Professional Bloggers [2020]

The 4-step Creative Writing Process for Professional Bloggers I've mentored 1000s of new bloggers in the past few years (through my 100-day blogging course), and do you know what is the most frequently asked question? "How do I write better blog posts?" This is the most important question to ask if you want to become a blogger. From my 10-years of experience in blogging, and inputs from other experienced bloggers, I have come up with a 4-step process for writing.  You can read it here: The 4-step Creative Writing Process for Professional Bloggers In this extensive article I've covered the 4-step process in depth... 1. Gain Knowledge by Learning 2. Add Experience from Past & Present 3. Go into Deep Thought 4. Manifest Your Ideas into Writing This is one of the most important and the longest blog post I've written on my blog.  Read it here:  The 4-step Creative Writing Process for Professional Bloggers I've also explained about t...
Read more

Sharing what we learned on the first Virtual Webmaster Unconference

  The first Virtual Webmaster Unconference successfully took place on August 26th and, as promised, we’d like to share the main findings and conclusions here. How did the event go? As communicated before, this event was a pilot, in which we wanted to test a) if there was an appetite for a very different type of event, and b) whether the community would actively engage in the discussions. To the first question, we were overwhelmed with the interest to participate; it definitely exceeded our expectations and it gives us fuel to try out future iterations. Despite the frustration of many, who did not receive an invitation, we purposefully kept the event small. This brings us to our second point: it is by creating smaller venues that discussions can happen comfortably. Larger audiences are perfect for more conventional conferences, with keynotes and panels. The Virtual Webmaster Unconference, however, was created to hear the attendees’ voices. And we did. What did we learn in the ses...
Read more