Development of an automated tool to consolidate information about Portuguese civil parishes

Abstract

The remarkable growth of the Internet accounts for a substantial creation of knowledge, an important asset responsible for the creation of value for nations. Centralized databases have thus increased in size and complexity. Maintaining and updating the information stored in the databases cannot be done efficiently by humans alone; automated tools have been used for quite some time with various degrees of success. One of the first software tools to emerge was the "web crawler", which is the basis of how search engines work. Another important class of tools, called "internet bots" or simply "bots" (from the word "robot"), is used to help humans manage large quantities of data. This work describes the development of an automated tool to gather information from various sources (both online and offline) about Portuguese civil parishes ("freguesias" in Portuguese) that can be used, for instance, by marketing companies or by Wikipedia editors to update their respective web pages. Even though Wikipedia has used bots for over 10 years, the web pages of Portuguese civil parishes are frequently outdated or have insufficient information. In addition, the information that can be used to update these web pages is scattered in various sources and in a format that does not allow an easy comparison between two or more parishes. For instance, an organization may need to compare the distribution of population from various parishes according to the number of people per family, age group or marital status. The development of the application followed the main steps of Software Engineering namely, requirement specification, application design and implementation, and testing. The program is able to receive an updated file containing all Portuguese civil parishes and allows the user to select those desired. Furthermore, the user can select offline databases in the form of a spreadsheet according to stipulated parameters. After this the user can still customize an output text, inserting variables that are replaced later by the application in the form of information from the databases, according to each civil parish. Following the compilation of databases in text form, the output for each civil parish is saved as a text file. In order to demonstrate the potential capability of automatic editing of pages in Wikipedia, the application has the capability to preview the text in a Wikipedia test page. The result of this particular work for a particular case demonstrates the construction of an easy-to-use and practical tool that both basic and advanced users can use to extract information about Portuguese civil parishes.