Map Reduce

Increasing performance and decreasing cost of microprocessors are making it feasible to move more processing power to the data source. This allows us to investigate new methods of storage delivery and storage management that were not plausible in the past. Our architecture, inspired by agent-based techniques and active disk technology, promotes an open storage management framework that embeds functionality into storage devices. We use local agents to implement self-control with automated capability that can be dynamically adapted to meet the storage management through improved search capabilities in the virtual environment and also retain capabilities like: security, performance and availability requirements.

Advancement in information and communication technology (ICT) has given rise to explosion of data in every field of operations. Working with the enormous volume of data (or Big Data, as it is popularly known as) for extraction of useful information to support decision making is one of the sources of competitive advantage for organizations today. Enterprises are leveraging the power of analytics in formulating business strategy in every facet of their operations to mitigate business risk. Volatile global market scenario has compelled the organizations to redefine their supply chain management (SCM). In this paper, we have delineated the relevance of Big Data and its importance in managing end to end supply chains for achieving business excellence. A Big Data-centric architecture for SCM has been proposed that exploits the current state of the art technology of data management, analytics and visualization. The security and privacy requirements of a Big Data system have also been highlighted and several mechanisms have been discussed to implement these features in a real world Big Data system deployment in the context of SCM. Some future scope of work has also been pointed out.

The last days, the data and internet are become increasingly growing which occurring the problems in big-data. For these problems, there are many software frameworks used to increase the performance of the distributed system. This software is used for available of large data storage. One of the most beneficial software frameworks used to utilize data in distributed systems is Hadoop. This software creates machine clustering and formatting the work between them. The Hadoop consists of two major components which are Hadoop Distributed File System (HDFS) and Map Reduce (MR). By Hadoop, we can process, count and distribute of each word in a large file and know the number of affecting for each of them. In this paper, we will explain what is Hadoop and its architectures, how it works and its performance analysis in a distributed system according to many authors. In addition, assessing each paper and compare with each other.

Map-Reduce is a programming model and an associated implementation for processing and generating large data sets. This model has a single point of failure: the master, who coordinates the work in a cluster. On the contrary, wireless sensor networks (WSNs) are distributed systems that scale and feature large numbers of small, computationally limited, low-power, unreliable nodes. In this article, we provide a top-down approach explaining the architecture, implementation and rationale of a distributed fault-tolerant IoT middleware. Specifically, this middleware consists of multiple mini-computing devices (Raspberry Pi) connected in a WSN which implement the Map-Reduce algorithm. First, we explain the tools used to develop this system. Second, we focus on the Map-Reduce algorithm implemented to overcome common network connectivity issues, as well as to enhance operation availability and reliability. Lastly, we provide benchmarks for our middleware as a crowd tracking application for a preserved building in Greece (i.e., M. Hatzidakis' residence). The results of this study show that IoT middleware with low-power and low-cost components are viable solutions for medium-sized cloud computing distributed and parallel computing centres. Potential uses of this middleware apply for monitoring buildings and indoor structures, in addition to crowd tracking to prevent the spread of COVID-19.

Data mining is the process of analyzing data from different perspectives and summarizing it into useful information the patterns, associations, or relationships among all this data can provide information. Spatial Data Mining (SDM) is the part of data mining. It is mainly used for finding the figure in data that is related to space. In spatial data mining, to get the result analyst use geographical or spatial information which require some special technique to get geographical data in the appropriate formats. SDM is mainly used in earth science data, crime mapping, census data, traffic data, and cancer cluster (i.e. to investigate environment health hazards). For real time processing, Stream data mining is used for the prediction of storm using spatial dataset with the help of stream data mining strategy i.e. CMR (Cloud Map-Reduce). In this, Stream data mining is presented to detect the storm disaster of coastal area which is located in Central America country and then by taking the dataset of coastal area which are having various regions. There are various regions i.e. Panama, Greater Antilles, Mexico golf which is used to detect the storm. It also detects that, is the region is affected or not, if affected then which area of that region is affected and from this it is helpful to predict the storm before the disaster occur. In this, two parameters are used to test the technique i.e. processing time and computational load and using this parameter compare the previous and proposed technique. The main aim of this is to predict storm disaster before it happen with the help of directionality and velocity of storm.

Cloud computing has emerged as a model that harnesses massive capacities of data centers to host services in a cost-effective manner. MapReduce has been widely used as a Big Data processing platform, proposed by Google in 2004 and has become a popular parallel computing framework for large-scale data processing since then. It is best suited for embarrassingly parallel and data-intensive tasks. It is designed to read large amount of data stored in a distributed file system such as Google File System (GFS), process the data in parallel, aggregate and store the results back to the distributed file system. Scheduling is one of the most critical aspects of MapReduce. Also three important scheduling issues in MapReduce such as locality, synchronization and fairness exist. This paper tries to illustrate and analyze the overview of thirteen different aware scheduling algorithms with different techniques and approaches for MapReduce in Hadoop and their scheduling issues and problems. At the end, Advantages and disadvantages of these algorithms are identified.

Data-intensive    systems    encompass
terabytes  to  petabytes  of  data.  Such  systems
require  massive  storage  and  intensive  computa-
tional power in order to execute complex queries
and generate timely results. Further, the rate at
which  this  data  is  being  generated  induces
extensive challenges of data storage, linking, and
processing.  A  data-intensive  cloud  provides  an
abstraction of high availability, usability, and effi-
ciency    to    users.    However,    underlying    this
abstraction, there are stringent requirements and
challenges  to  facilitate  scalable  and  resourceful
services through effective physical infrastructure,
smart  networking  solutions,  intelligent  software
tools, and useful software approaches. This paper
analyzes    the    extensive    requirements    which
exist  in  data-intensive  clouds,  describes  various
challenges related to the paradigm, and assess nu-
merous solutions in meeting these requirements
and challenges. It provides a detailed study of the
solutions and analyzes their capabilities in meet-
ing emerging needs of widespread applications.

Opinion Mining (OM) is a field of Natural Language Processing (NLP) that aims to capture human sentiment in the given text. With the ever-spreading of online purchasing websites, micro-blogging sites, and social media platforms, OM in online social media platforms has picked the interest of thousands of scientific researchers. Because the reviews, tweets and blogs acquired from these social media networks, act as a significant source for enhancing the decision making process. The obtained textual data (reviews, tweets, or blogs) are classified into three different class labels which are negative, neutral and positive for analyzing and extracting relevant information from the given dataset. In this contribution, we introduce an innovative MapReduce improved weighted ID3 decision tree classification approach for OM, which consists mainly of three aspects: Firstly We have used several feature extractors to efficiently detect and capture the relevant data from the given tweets, including N-grams or character-level, Bag-Of-Words, word embedding (GloVe, Word2Vec), FastText, and TF-IDF. Secondly, we have applied a multiple feature selector to reduce the high feature's dimensionality, including Chi-square, Gain Ratio, Information Gain, and Gini Index. Finally, we have employed the obtained features to carry out the classification task using an improved ID3 decision tree classifier, which aims to calculate the weighted information gain instead of information gain used in traditional ID3. In other words, to measure the weighted information gain for the current conditioned feature, we follow two steps: First, we compute the weighted correlation function of the current conditioned feature. Second, we multiply the obtained weighted correlation function by the information gain of this current conditioned feature. This work is implemented in a distributed environment using the Hadoop framework, with its programming framework MapReduce and its distributed file system HDFS. Its primary goal is to enhance the performance of a well-known ID3 classifier in terms of accuracy, execution time, and ability to handle the massive datasets. We have carried out several experiences that aims to assess the effectiveness of our suggested classifier compared to some other contributions chosen from the literature. The experimental results demonstrated that our ID3 classifier works better on COVID-19_Sentiments dataset than other classifiers in terms of Recall (85.72 %), specificity (86.51 %), error rate (11.18 %), false-positive rate (13.49 %), execution time (15.95s), kappa statistic (87.69 %), F1-score (85.54 %), classification rate (88.82 %), false-negative rate (14.28 %), precision rate (86.67 %), convergence (it convergent towards the iteration 90), stability (it is more stable with mean deviation standard equal to 0.12 %), and complexity (it requires much lower time and space computational complexity). INDEX TERMS ID3 decision tree, opinion mining, Hadoop, HDFS, MapReduce, feature extractors, feature selectors, DataMining, big data, information gain.

Increasing performance and decreasing cost of microprocessors are making it feasible to move more processing power to the data source. This allows us to investigate new methods of storage delivery and storage management that were not plausible in the past. Our architecture, inspired by agent-based techniques and active disk technology, promotes an open storage management framework that embeds functionality into storage devices. We use local agents to implement self-control with automated capability that can be dynamically adapted to meet the storage management through improved search capabilities in the virtual environment and also retain capabilities like: security, performance and availability requirements.

Log Analysis is a critical procedure in most framework and system exercises where log information is utilized for different reasons, for example, for execution checking, security examining or notwithstanding for revealing and profiling. Nonetheless, as years cruised by, the volume of log information increments alongside the span of the framework just as the quantity of clients included. Customary or existing log analyser instruments are not ready to deal with the huge measure of information. Thusly, Big Data is the answer for defeated this issue. The principle motivation behind this paper is to introduce a survey of log document investigation in Big Data condition dependent on past research works. This paper likewise features the qualities of Big Data just as Hadoop Framework that has been generally utilized as Big Data application. Results from the papers assessed demonstrate that dominant part analysts connected MapReduce as the principle segment of Hadoop for investigating the log records and HDFS as the information stockpiling. Past analysts have likewise utilized different instruments and calculations together with the Hadoop Framework for investigation purposes. The discoveries of this paper will give an intelligible audit of Hadoop use execution in breaking down various kinds of log records and prescribe justifiable outcomes for end clients to use in future work.

The Big Data is the new technology in the field of research in recent years and is not only big in amount , but also produced at speed and variety, which endeavors the research upsurge in multidisciplinary fields
like Decision making , Healthcare industry and business analysis. Due to the basic features (Volume, speed, Velocity, and Variety) of Data when it is difficult to store and analyse with existing tools and techniques. It
explains distinctive tasks in scalability, storage, computational complexity, analytical, security issues. Hence here the salient features of big data and how this shakes the storage tools and processing technique. It shows the taxonomy of Big Data application areas and explanation of data characteristics, privacy and security issues. Furthermore, it also discovers research issues and challenges in big data storage, privacy and security of data and data processing.