The hacker's custom script navigates the complex web of council portals, revealing a treasure trove of planning data. The scraped data will be made available to the public, offering unparalleled insight into the UK's planning system.
UK PLANNING DATA UNLOCKED: 2.6M DECISIONS SCRAPED FROM 241 COUNCIL PORTALS
_A lone hacker has cracked open the notoriously opaque world of UK planning data, revealing a treasure trove of 2.6 million decisions across England, Scotland, and Wales. The move has significant implications for transparency and accountability in the UK's planning system. As the hacker continues to scrape the remaining council portals, the true extent of the data's value is only beginning to emerge._
A lone hacker has made a groundbreaking achievement in unlocking the UK's planning data, scraping 2.6 million decisions from 241 council portals. The move has sent shockwaves through the planning community, with many hailing it as a major victory for transparency and accountability. The UK's planning system has long been criticized for its opacity, with decisions often made behind closed doors and without adequate public scrutiny.
The Scraping Problem
The UK's planning data is technically public, but in practice, it's locked behind over 400 different council portals, each with its own unique schema and security measures. The hacker has spent four months scraping 241 of these portals, using custom scripts to navigate the complex web of ASP.NET and AWS WAF protections. The resulting dataset is a staggering 2.6 million decisions, offering unparalleled insight into the UK's planning system.
Implications for Transparency
The release of this data has significant implications for transparency and accountability in the UK's planning system. For the first time, citizens and researchers will be able to access and analyze the decisions made by local councils, revealing patterns and trends that were previously invisible. This could lead to a more informed and engaged public, better equipped to hold their elected officials to account.
Technical Challenges
The hacker's achievement is all the more impressive given the technical challenges involved. Many of the council portals are still running outdated software, with some dating back to 2004. The use of bespoke ASP.NET and AWS WAF protections added an extra layer of complexity, requiring the hacker to develop custom scripts to navigate these systems. The success of this project demonstrates the power of determination and technical expertise in unlocking even the most secure systems.
Future Directions
As the hacker continues to scrape the remaining council portals, the true extent of the data's value is only beginning to emerge. The dataset could be used to identify areas of inefficiency in the planning system, or to reveal patterns of corruption and cronyism. The potential applications are vast, and it will be exciting to see how researchers and citizens choose to use this new resource.
The release of this data is a significant step forward for transparency and accountability in the UK's planning system. As the hacker continues to scrape the remaining council portals, the true extent of the data's value will only continue to grow. One thing is certain: the UK's planning system will never be the same again.
Sources: Hacker News, UK Council Planning Portals