Data Mining - Retrieving Information From Data

Data Mining - Retrieving Information From Data

Data mining definition is the process of retrieving information from data. It has become very important now days because data that is processed is usually kept for future reference and mainly for security purposes in a company. Data transforms is processed into information and it is mostly used in different ways depending on what information one is extracting and from where the person is extracting the information.

It is commonly used in marketing, scientific information and research work, fraud detection and surveillance and many more and most of this work is done using a computer. This definition can come in different terms data snooping, data fishing and data dredging all this refer to data mining but it depends in which department one is. One must know data mining definition so that he can be in a position to make data.

The method of data mining has been there for so many centuries and it is used up to date. There were early methods which were used to identify data mining there are mainly two: regression analysis and bayes theorem. These methods are never used now days because a lot of people have advanced and technology has really changed the entire system.

With the coming up or with the introduction of computers and technology, it becomes very fast and easy to save information. Computers have made work easier and one can be able to expand more knowledge about data crawling and learn on how data is stored and processed through computer science.

Computer science is a course that sharpens one skill and expands more about data crawling and the definition of what data mining means. By studying computer science one can be in a position to know: clustering, support vector machines and decision trees there are some of the units that are found on computer science.

It's all about all this and this knowledge must be applied here. Government institutions, small scale business and supermarkets use data.

The main reason most companies use data mining is because data assist in the collection of information and observations that a company goes through in their daily activity. Such information is very vital in any companies profile and needs to be checked and updated for future reference just in case something happens.

Businesses which use data crawling focus mainly on return of investments, and they are able to know whether they are making a profit or a loss within a very short period. If the company or the business is making a profit they can be in a position to give customers an offer on the product in which they are selling so that the business can be a position to make more profit in an organization, this is very vital in human resource departments it helps in identifying the character traits of a person in terms of job performance.

Most people who use this method believe that is ethically neutral. The way it is being used nowadays raises a lot of questions about security and privacy of its members. Data mining needs good data preparation which can be in a position to uncover different types of information especially those that require privacy.

A very common way in this occurs is through data aggregation.

Data aggregation is when information is retrieved from different sources and is usually put together so that one can be in a position to be analyze one by one and this helps information to be very secure. So if one is collecting data it is vital for one to know the following:

    How will one use the data that he is collecting?
    Who will mine the data and use the data.
    Is the data very secure when am out can someone come and access it.
    How can one update the data when information is needed
    If the computer crashes do I have any backup somewhere.

It is important for one to be very careful with documents which deal with company's personal information so that information cannot easily be manipulated.

source : http://ezinearticles.com/?Data-Mining---Retrieving-Information-From-Data&id=5054887

One of the Main Differences Between Statistical Analysis and Data Mining

One of the Main Differences Between Statistical Analysis and Data Mining

Two methods of analyzing data that are common in both academic and commercial fields are statistical analysis and data mining. While statistical analysis has a long scientific history, data mining is a more recent method of data analysis that has arisen from Computer Science. In this article I want to give an introduction to these methods and outline what I believe is one of the main differences between the two fields of analysis.

Statistical analysis commonly involves an analyst formulating a hypothesis and then testing the validity of this hypothesis by running statistical tests on data that may have been collected for the purpose. For example, if an analyst was studying the relationship between income level and the ability to get a loan, the analyst may hypothesis that there will be a correlation between income level and the amount of credit someone may qualify for.

The analyst could then test this hypothesis with the use of a data set that contains a number of people along with their income levels and the credit available to them. A test could be run that indicates for example that there may be a high degree of confidence that there is indeed a correlation between income and available credit. The main point here is that the analyst has formulated a hypothesis and then used a statistical test along with a data set to provide evidence in support or against that hypothesis.

Data mining is another area of data analysis that has arisen more recently from computer science that has a number of differences to traditional statistical analysis. Firstly, many data mining techniques are designed to be applied to very large data sets, while statistical analysis techniques are often designed to form evidence in support or against a hypothesis from a more limited set of data.

Probably the mist significant difference here, however, is that data mining techniques are not used so much to form confidence in a hypothesis, but rather extract unknown relationships may be present in the data set. This is probably best illustrated with an example. Rather than in the above case where a statistician may form a hypothesis between income levels and an applicants ability to get a loan, in data mining, there is not typically an initial hypothesis. A data mining analyst may have a large data set on loans that have been given to people along with demographic information of these people such as their income level, their age, any existing debts they have and if they have ever defaulted on a loan before.

A data mining technique may then search through this large data set and extract a previously unknown relationship between income levels, peoples existing debt and their ability to get a loan.

While there are quite a few differences between statistical analysis and data mining, I believe this difference is at the heart of the issue. A lot of statistical analysis is about analyzing data to either form confidence for or against a stated hypothesis while data mining is often more about applying an algorithm to a data set to extract previously unforeseen relationships.

Source:http://ezinearticles.com/?One-of-the-Main-Differences-Between-Statistical-Analysis-and-Data-Mining&id=4578250

Data Mining vs Screen-Scraping

Data Mining vs Screen-Scraping

Data mining isn't screen-scraping. I know that some people in the room may disagree with that statement, but they're actually two almost completely different concepts.

In a nutshell, you might state it this way: screen-scraping allows you to get information, where data mining allows you to analyze information. That's a pretty big simplification, so I'll elaborate a bit.

The term "screen-scraping" comes from the old mainframe terminal days where people worked on computers with green and black screens containing only text. Screen-scraping was used to extract characters from the screens so that they could be analyzed. Fast-forwarding to the web world of today, screen-scraping now most commonly refers to extracting information from web sites. That is, computer programs can "crawl" or "spider" through web sites, pulling out data. People often do this to build things like comparison shopping engines, archive web pages, or simply download text to a spreadsheet so that it can be filtered and analyzed.

Data mining, on the other hand, is defined by Wikipedia as the "practice of automatically searching large stores of data for patterns." In other words, you already have the data, and you're now analyzing it to learn useful things about it. Data mining often involves lots of complex algorithms based on statistical methods. It has nothing to do with how you got the data in the first place. In data mining you only care about analyzing what's already there.

The difficulty is that people who don't know the term "screen-scraping" will try Googling for anything that resembles it. We include a number of these terms on our web site to help such folks; for example, we created pages entitled Text Data Mining, Automated Data Collection, Web Site Data Extraction, and even Web Site Ripper (I suppose "scraping" is sort of like "ripping"). So it presents a bit of a problem-we don't necessarily want to perpetuate a misconception (i.e., screen-scraping = data mining), but we also have to use terminology that people will actually use.

Source: http://ezinearticles.com/?Data-Mining-vs-Screen-Scraping&id=146813

Assuring Scraping Success with Proxy Data Scraping

Assuring Scraping Success with Proxy Data Scraping

Have you ever heard of "Data Scraping?" Data Scraping is the process of collecting useful data that has been placed in the public domain of the internet (private areas too if conditions are met) and storing it in databases or spreadsheets for later use in various applications. Data Scraping technology is not new and many a successful businessman has made his fortune by taking advantage of data scraping technology.

Sometimes website owners may not derive much pleasure from automated harvesting of their data. Webmasters have learned to disallow web scrapers access to their websites by using tools or methods that block certain ip addresses from retrieving website content. Data scrapers are left with the choice to either target a different website, or to move the harvesting script from computer to computer using a different IP address each time and extract as much data as possible until all of the scraper's computers are eventually blocked.

Thankfully there is a modern solution to this problem. Proxy Data Scraping technology solves the problem by using proxy IP addresses. Every time your data scraping program executes an extraction from a website, the website thinks it is coming from a different IP address. To the website owner, proxy data scraping simply looks like a short period of increased traffic from all around the world. They have very limited and tedious ways of blocking such a script but more importantly -- most of the time, they simply won't know they are being scraped.

You may now be asking yourself, "Where can I get Proxy Data Scraping Technology for my project?" The "do-it-yourself" solution is, rather unfortunately, not simple at all. Setting up a proxy data scraping network takes a lot of time and requires that you either own a bunch of IP addresses and suitable servers to be used as proxies, not to mention the IT guru you need to get everything configured properly. You could consider renting proxy servers from select hosting providers, but that option tends to be quite pricey but arguably better than the alternative: dangerous and unreliable (but free) public proxy servers.

There are literally thousands of free proxy servers located around the globe that are simple enough to use. The trick however is finding them. Many sites list hundreds of servers, but locating one that is working, open, and supports the type of protocols you need can be a lesson in persistence, trial, and error. However if you do succeed in discovering a pool of working public proxies, there are still inherent dangers of using them. First off, you don't know who the server belongs to or what activities are going on elsewhere on the server. Sending sensitive requests or data through a public proxy is a bad idea. It is fairly easy for a proxy server to capture any information you send through it or that it sends back to you. If you choose the public proxy method, make sure you never send any transaction through that might compromise you or anyone else in case disreputable people are made aware of the data.

A less risky scenario for proxy data scraping is to rent a rotating proxy connection that cycles through a large number of private IP addresses. There are several of these companies available that claim to delete all web traffic logs which allows you to anonymously harvest the web with minimal threat of reprisal. Companies such as offer large scale anonymous proxy solutions, but often carry a fairly hefty setup fee to get you going.

Source:http://ezinearticles.com/?Assuring-Scraping-Success-with-Proxy-Data-Scraping&id=248993