Data Mining and Extracting Historical Data
What We did
As part of the effort to design and code a software solution to meet the client's objective we performed a study their historical data and created a database design to house the millions of lines of data. The database also had to be designed to ensure that data could be efficiently queried. To ensure the availability of data, we setup a backup solution
Once a relational database design had been created and provisioned on a server, we now had the task of extracting the historical data and loading it into our new database. To do this we built a script to scrape the data line by line from existing logs and automated the data transfer.
Once the data was loaded into the database it was time to construct a means for our client to access and search the data. Our client was not well versed in database queries so a Graphical User Interface was the right course of action. We created a user interface where the client could select multiple columns to display, the ability to create queries through a search box, and return the results in a table.</div> </div>