If the intention was to build an automated "scraping" application to retrieve all the various information available on Open Georgia, then a rigorous, production-ready, ELT (not ETL) solution would need to be developed, tested, and deployed to ensure all data was presented and that it met a high degree of data quality. For the Open Georgia Analysis effort, we just want to focus in on the salary data presented for teachers and staff across several local boards of education.
Drilling down into the Salaries & Travel Reimbursements area of the site much information is quickly at your fingertips. For this analysis effort, only the following Organization data was retrieved.
| Fiscal Year | Organization Type | Organization | Records |
|---|---|---|---|
| 2010 | Local Boards of Education | ATLANTA INDEPENDENT SCHOOL SYSTEM | 9,201 |
| 2010 | Local Boards of Education | COBB COUNTY SCHOOL DISTRICT | 20,377 |
| 2010 | State Agencies, Boards, Authorities and Commissions | PUBLIC SAFETY, DEPARTMENT OF | 1,914 |
| 2010 | Local Boards of Education | FULTON COUNTY BOARD OF EDUCATION | 15,408 |
| 2011 | Local Boards of Education | FULTON COUNTY BOARD OF EDUCATION | 15,200 |
| 2012 | Local Boards of Education | FULTON COUNTY BOARD OF EDUCATION | 14,843 |
Quite obviously these 76,943 (very well-formed) records do not mee the 3V's of "big data", but they do allow us to exercise the Hadoop tooling to perform some level of data analysis.