Data Mining and Data Warehouse Essay

ABSTRACT information archeological site, the extraction of hidden figureive information from large selective informationbases, is a powerful impertinently technology with great potential to help companies focus on the most important information in their information wargon houses. Data exploit tools predict hereafter trends and sorts, allowing businesses to make proactive, knowledge- driven decisions bodys. Data storage store is a computer system designed to give business decision-makers instant approach to information. The warehouse copies its entropy from existing systems like order entry, general ledger, and human resources and stores it for use by executives rather than programmers. Data warehouse substance abusers use special software that enables them to create and rag information when they need it, as opposed to a reporting schedule defined by the information systems (IS) department. This paper describes the meaning of information warehouse and entropy mining ba sic architecture of info wareho utilise and information mining, functions and working of data mining. It in addition presents data mining from data warehouseINTRODUCTION current organizations are under enormous pressure with recent development of the technology. Clearly we need a rapid access to all kinds of information. To assist this we need to accept the past and to target relevant trend analysis. So to perform any trend analysis we essential have a database. In most organizations you result find really large databases in operation for normal daily traffics. These references of databases are known as operational databases in most cases they have non been design to store historical data or to respond to queries but simply to support all the applications for day to day legal proceeding.The second type of database run aground in organizations is the data warehouse. This is designed for strategic decision support and is largely built up from the databases that make up the operational database. The basic characteristic of a data warehouse is that it contains vast amount of data which drop mean billions of records. Smaller, local data warehouse are called data marts. A data warehouse is designed especially for decision support queries therefore only data that is needed for decision support is extracted from the operational data and stored in the data warehouse along with the duration when it was retrieved from operational databases.DEFINITIONDATA WAREHOUSINGA data warehouse is a subject-oriented, integrated, time-variant and non-volatile line of battle of data in support of concerns decision making process. Subject-Oriented A data warehouse ignore be use to analyze a particular subject heavens. For example, sales can be a particular subject. Integrated A data warehouse integ pass judgment data from multiple data sources. For example, source A and source B may have different ports of identifying a product, but in a data warehouse, there will be only a single way of identifying a product.Time-Variant Historical data is kept in a data warehouse. For example, one can retrieve data from 3 months, 6 months, 12 months, or horizontal older data from a data warehouse. This contrasts with a transactions system, where often only the most recent data is kept. For example, a transaction system may hold the most recent address of a customer, where a data warehouse can hold all addresses associated with a customer. Non-volatile Once data is in the data warehouse, it will not change. So, historical data in a data warehouse should never be altered.The following are the typical steps involved in the data wareho utilize project cycle.* Requirement Gathering* Physical Environment Setup* Data Modeling* ETL* OLAP Cube construct* Front End Development* Report Development* Performance Tuning* Query Optimization* Quality Assurance* Rolling out to intersection* Production Maintenance* Incremental EnhancementsBenefits of a data warehouseA data warehouse maintains a copy of information from the source transaction systems. This architectural complexity provides the opportunity to * Maintain data history, even if the source transaction systems do not. * Integrate data from multiple source systems, enabling a central diorama across the enterprise. This benefit is always valuable, but particularly so when the organization has grown by merger. * Improve data quality, by providing consistent codes and descriptions, flagging or even fixing bad data. * Present the organizations information consistently.* Provide a single common data model for all data of interest regardless of the datas source. * Restructure the data so that it makes sense to the business users. * Restructure the data so that it delivers excellent query performance, even for complex analytic queries, without impacting the operational systems. * Add value to operational business applications, notably customer relationship management (CRM) systems.Data minelayi ng (DM)Data mining, likewise known as knowledge discovery, refers to computer-assisted tools and techniques for sifting with and analyzing these vast data stores in order to find trends, patterns, and correlations that can guide decision making and profit checking. Data mining covers a wide variety of uses, from analyzing customer purchases to discovering galaxies.In essence, data mining is the equivalent of finding gold nuggets in a mountain of data. The massive task of finding hidden gold depends heavily upon the power of computers The purpose of DM is to analyze and understand past trends and predict future trends.By predicting future trends, business organizations can better position their products and services for financial gain. Nonprofit organizations have also achieved significant benefits from data mining, such as in the area of scientific progress. The concept of data mining is simple yet powerful. The simplicity of the concept is deceiving, however. Traditional metho ds of analyzing data, involving query-and-report approaches, cannot handle tasks of such magnitude and complexity. Data mining consists of volt major elements* Extract, transform, and load transaction data onto the data warehouse system. * Store and manage the data in a multidimensional database system. * Provide data access to business analysts and information technology professionals. * Analyze the data by application software.* Present the data in a reclaimable format, such as a graph or table.Data mining services can be utilise for the following functions * Research and surveys Data mining can be used for product research, surveys, market research and analysis. Information can be gathered that is quite useful in driving peeled marketing campaigns and promotions. * Information collection Through the web scraping process it is practicable to collect information regarding investors, investments and funds by scraping through related websites and databases. * Customer opinions C ustomer views and suggestions revivify an important role in the way a company operates. The information can be readily be found on forums, blogs and other resources where customers freely provide their views. * Data scanning Data collected and stored will be not be important unless scanned. Scanning is important to identify patterns and similarities contained in the data.* stemma of information This is the processing of identifying the useful patterns in data that can be used in decision making process. This is so because decision making must be based on sound information and facts. * Pre-processing of data Usually the data collected is stored in the data warehouse. This data needs to be pre-processed.by pre-processing it center some data that may be deemed unimportant may therefore re removed manually be data mining experts.* wind vane data Web data usually poses many challenges in mining. This is so because of its nature. For instance, web data can be deemed as dynamic meaning it keeps changing from time to time. Therefore it means the process of data mining should be repeated in regular intervals. * Competitor analysis There is a need to understand how your competitors are fairing on in the business market. You need to know both their weaknesses and strengths. Their methods of marketing and distribution can be mined. How they reduce their overall costs is also quite important.* Online research The internet is highly regarded for its huge information. It is evident that it is the largest source of information. It is possible to gather a lot of information regarding different companies, customers and your business clients. It is possible to detect frauds through online means. * News Nowadays with almost all major newspapers and news sources posting their news online it is possible to gather information regarding trends and other minute areas. In this way, it is possible to be in the better position of competing in the market. * Updating data This is quit e important. Data collected will be baseless unless it is updated. This is to ensure that the information is relevant so as to make decisions from it.How does data mining work?While large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are available statistical, machine learning, and neural networks. Generally, any of four types of relationships are sought * Classes Stored data is used to locate data in predetermined groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order.This information could be used to increase traffic by having daily specials. * Clusters Data items are grouped according to logical relationships or consumerpreferences. For example, data can be mined to identify market segments or consumer affinities. * Associations Data can be mined to identify associations. The beer-diaper example is an example of associative mining. * Sequential patterns Data is mined to anticipate behavior patterns and trends. For example, an outdoor equipment retailer could predict the likelihood of a backpack being purchased based on a consumers purchase of sleeping bags and hiking shoes.Industries/fields where data mining is currently applied are as follows 1. Data Mining in the Banking Sector Worldwide, banking sector is ahead of many other industries in using mining techniques for their vast customer database. Although banks have employed statistical analysis tools with some success for several years, previously unseen patterns of customer behavior are now coming into clear focus with the aid of new data mining tools. These statistical tools and even the OLAP find out the answers, but more innovative data mining tools provide insight to the a nswer. or so of the applications of data mining in this industry are (i)Predict customer reaction to the change of interest rates (ii)Identify customers who will be most receptive to new product offers (iii)Identify loyal customers(iv) Pin point which clients are at the highest essay for defaulting on a give (v)Find out persons or groups who will opt for each type of loan in the following year (vi)Detect fraudulent activities in credit card transactions (vii)Predict clients who are likely to change their credit card affiliation in the next quarter (viii)Determine customer preference of the different modes of transaction namely through teller or through credit cards, etc.2. Data Mining in the Insurance SectorInsurance companies can benefit from modern data mining methodologies, which help companies to reduce costs, increase profits, retain current customers, acquire new customers, and develop new products .This can be done through (1)Evaluating the risk of the assets being insured taking into account the characteristics of the asset as well as the owner of the asset. (2)Formulating Statistical Modeling of Insurance Risks(3)Using the Joint Poisson/Log-Normal Model of mining to optimize insurance policies (4)And finally finding the actuarial Credibility of the risk groups among insurers3. Data Mining in TelecommunicationAs on this date, every activeness in telecommunication has used data mining technique.(1)Analysis of telecom service purchases(2)Prediction of telephone calling patterns(3)Management of resources and network traffic(4)Automation of network management and maintenance using artificial intelligence to diagnose and repair network transmission problems, etc4. Data Mining in Fraud DetectionData dredging has found wide and useful application in various fraud staining processes like (1)Credit card fraud detection using a combined parallel approach (2)Fraud detection in the voters list using neural networks in combination with symbolic and analog data mining. (3)Fraud detection in passport applications by designing a specific online learning diagnostic system. (4)Rule and analog based detection of false medical claims and so on.An Architecture for Data MiningTo best apply these pass on techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. more data mining tools currently operate outside of the warehouse, requiring extra steps for extracting, importing, and analyzing the data. Furthermore, when new insights require operational implementation, integration with the warehouse simplifies the application of results from data mining. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign management, fraud detection, new product rollout, and so on. Figure 1 illustrates an architecture for advanced analysis in a large data warehouse.Figure 2 Integrated Data Mining Architectur eFROM DATA WAREHOUSE TO DATA MININGDM is a set of methods for data analysis, created with the aim to find out specific dependence, relations and rules related to data and making them out in the new, higher-level quality information. As distinguished from the data warehouse, which has unique data approach, DM gives results that show relations and mutuality of data. Mentioned dependences are mostly based on various mathematical and statistic relations.Figure 3 Process of knowledge data discoveryEMERGING TRENDS IN DATA MININGWeb mining is the application of data mining techniques to discover patterns from the Web. According to analysis targets, web mining can be shared into three different types, which are Web practise mining, Web content mining and Web structure mining. Web usage miningWeb usage mining is the process of extracting useful information from server logs i.e. users history. Web usage mining is the process of finding out what users are looking for on Internet. Some user s might be looking at only textual data, whereas some others might be interested in multimedia data. Web structure miningWeb structure mining is the process of using graph theory to analyze the node and connection structure of a web site. According to the type of web structural data, web structure mining can be divided into two kinds 1. Extracting patterns from hyperlinks in the web a hyperlink is a structural agent that connects the web page to a different location. 2. Mining the document structure analysis of the tree-like structure of page structures to describe HTML or XML tag usage. Web content miningWeb content mining is the mining, extraction and integration of useful data, information and knowledge from Web page contents. Data Stream Mining is the process of extracting knowledge structures from continuous, rapid data records. A data stream is an ordered sequence of instances that in many applications of data stream mining can be read only once or a small number of times usi ng limited computing and storage capabilities. Examples of data streams embarrass computer network traffic, phone conversations, ATM transactions, web searches, and sensor data.