The BOE robot.txt is a long list of commands that brings together hundreds of URLs body within the Ministry of the Presidency, which can not be found on search engines. Some of these sites date back to 1967, but after an arduous review of most of them can be easily glimpse a common denominator: most are pardons.
For example, if we choose randomly corresponding to June 12, 2008 we will meet three pardons. The first check is a person whose name and surname are the same as those of a candidate of CiU municipal elections Municipi of L'Hospitalet de Llobregat. She was sentenced by the Supreme Court and author of a continuing offense of computer fraud, and sentenced to three years and six months in prison. But it was not to jail because the Ministry of Justice decided to pardon her. After the BOE include the robot.txt, this information has been virtually buried in the network.
It is not alone. Many links from the 70s contain pardons to different personalities. Each and every one of these addresses no longer appear in search engines but still on the Internet. As he explained to Teknautas Pablo Fernández Burgueño, lawyer's office Abanlex, "the BOE may be withdrawn from the search engines this information on its own initiative, if it sees fit to harm anyone in particular, or after receiving a request of law to oblivion. keep in mind that many users are calling for the withdrawal of some data on the internet following the judgment against Google. "
The inclusion of specific links in the list of BOE robots.txt is done by the State Agency of the Official State Gazette and may be requested by the affected, by the Spanish Data Protection Agency (on its own initiative or at the request of the affected), by the agency ordered to perform publication (ex officio or at the request of the affected), or, finally, the BOE itself.
The law provides that you can de-index of office when it has been established the unlawfulness of the publication or the data is no longer useful for the purpose for which it was collected. From Teknautas we have asked the BOE what is the procedure followed and especially the criteria, but at the time of writing still not been delivered.
A TOOL THAT HAS MANY DISADVANTAGES
For the lawyer Luis Gervas of Pisa, responsible for the Code of Right to be Forgotten, published in the Official Gazette, and a member of the website Salirdeinternet, "the inclusion of links in the listing robots.txt by the BOE creates obvious problems such as they are open and with minimal technical knowledge can be consulted by anyone. in this list you can find up judgments of the Constitutional Court itself. in addition, the risk that these links are re-indexed by Google after a long run, but so even more damaging. "
"In any case, the BOE acts following the recommendations of the AEPD, which in 2011 refused to twice their complaints. Over the years, the question arises resort to other techniques deindexation, especially considering that even Google recommends other systems, "added the Pisa.
The BOE is not the only organization that uses the robots exclusion standard. The Royal House also used it to remove from your website all traces of Inaki Urdangarin. But without much success as some pages, such as web.archive, still retain the original URL.
Anyway not forget that this file does not guarantee the isolation, as may be the case that a search engine remove their information searches, but not others. Specifically Google offers its users a practical guide to block URLs with robot.txt files.
Digital Newspaper El Confidencial