Back to Table of contents

Primeur weekly 2017-02-13

Focus

Funding agencies and academia need to rethink reward structure for computational tool developers to tackle big scientific challenges ...

OpenAire to support more technical e-Infrastructures with Open Research Data guidelines, publication and training ...

Exascale supercomputing

ISC High Performance keynote forecasts future role of HPC in weather and climage prediction ...

Crowd computing

Towards equal access to digital coins ...

Quantum computing

Large groups of photons on demand - an equivalent of photonic 'integrated circuit' ...

Protecting quantum computing networks against hacking threats ...

Sorting machine for atoms ...

Focus on Europe

HPC-Europa3 provides access to European HPC systems ...

Peter the Great Saint Petersburg Polytechnic University is first university to introduce computational computer code for Airbus ...

Middleware

Introducing Hazelcast Jet: a new lightweight, distributed data processing engine ...

Bright Computing announces strategic alliance with Curtiss-Wright ...

Hardware

NVIDIA powers new class of supercomputing workstations with breakthrough capabilities for design and engineering ...

Van Andel research institute optimises HPC pipeline to drive research discoveries and new drug therapies with end-to-end DDN solution ...

CoolIT Systems issued U.S. patent for modular heat-transfer solutions ...

ORNL researchers break data transfer efficiency record ...

Cray reports 2016 full year and fourth quarter financial results ...

Supermicro deploys 30,000+ MicroBlade servers to enable one of the world's highest Efficiency (1.06 PUE) data centres ...

Mellanox ships more than 100,000 cables for next generation 100 Gb/s networks ...

UMass Amherst boosts deep learning research with powerful new GPU cluster ...

Oak Ridge National Laboratory enhances data integrity and accessibility with Active Archive Solutions ...

Applications

ANSYS spurs pervasive engineering simulation with release 18 ...

Computing sciences students: Get your team together for the SC17 Student Cluster Competition in Denver ...

Latest Allinea update advances code optimization across platforms ...

Con Edison selects C3 IoT for Big Data and predictive analytics platform and applications ...

New algorithms may revolutionize drug discoveries - and our understanding of life ...

When data's deep, dark places need to be illuminated ...

Computer trained to predict which AML patients will go into remission and which will relapse ...

The Cloud

Cycle Computing collaborates with ANSYS on its Enterprise Cloud HPC offering ...

IBM launches "Digital - Nation Africa" and invests $70 million to bring digital skills to Africa with free, Watson-powered skills platform for 25 million people ...

Introducing Hazelcast Jet: a new lightweight, distributed data processing engine

7 Feb 2017 Palo Alto - Hazelcast, an open source in-memory data grid (IMDG) with hundreds of thousands of installed clusters and over 17 million server starts per month, has launched Hazelcast Jet - a distributed processing engine for Big Data streams. With Hazelcast's IMDG providing storage functionality, Hazelcast Jet is a new Apache 2 licensed open source project that performs parallel execution to enable data-intensive applications to operate in near real-time. Using directed acyclic graphs (DAG) to model relationships between individual steps in the data processing pipeline, Hazelcast Jet is simple to deploy and can execute both batch and stream-based data processing applications. Hazelcast Jet is appropriate for applications that require a near real-time experience such as sensor updates in IoT architectures (house thermostats, lighting systems), in-store e-commerce systems and social media platforms.

IoT devices churn out massive amounts of data which flows to computers for analysis. In some cases, analyses must be done in near real-time - log analysis, monitoring, fraud detection, dashboard data extraction, and placement. However, there are challenges to handling large amounts of data:

  • Ingest high volumes of data, while keeping up-to-date with what's coming in;
  • Anomaly detection, recognize and validate the correctness of data entering the system;
  • Analyze, aggregate, extrapolate and react to a situation in near real-time;
  • Work with extremely low latency access to data; and
  • Reporting, whether running on-demand or via scheduled batch processing.

Hazelcast Jet is built on top of a one-record-per-time architecture - sometimes known as continuous operators. This means that it processes incoming records as soon as possible, opposed to accumulating records into micro-batches, consequently lowering latency for applications. Jet ingests data at high-velocity, via socket, file, HDFS or Kafka interfaces, and processes the business logic or complex computation on incoming data. A pure in-memory approach, Jet is 20x faster than Hadoop, enabling users to meet service-level requirements. It is extremely simple to program and to deploy - in particular Jet can be fully embedded for OEMs and for Microservices - making it is easier for manufacturers to build and maintain next generation systems.

Important components include:

  • Events-based architecture for low latency Transaction Processing System (TPS) applications
  • Uses directed acyclic graphs (DAG) to model data flow
  • High level java.util.stream API for great simplicity
  • Low level Core (DAG) API for maximum flexibility allowing direct manipulation of vertices representing data source readers, joiners, sorters, aggregators and data sinks
  • Distributed connectors for reading and writing: Hazelcast IMDG Map and List, HDFS, Kafka, File, Socket

Greg Luck, CEO of Hazelcast, stated: "Hazelcast Jet is a super fast, low latency, next generation DAG Engine for Big Data processing. We believe that the Hadoop and Spark ecosystems are too complex to program and to deploy and have set out to bring Hazelcast's legendary simplicity to Big Data. We have designed it as a general purpose engine for the intersect of Big Data programmers and Java programmers. But if you are already a Hazelcast user or have data in Hazelcast it will be the easiest way to solve your Big Data problems."

Hazelcast will be providing 24×7 enterprise support subscriptions for Hazelcast Jet.

Source: Hazelcast

Back to Table of contents

Primeur weekly 2017-02-13

Focus

Funding agencies and academia need to rethink reward structure for computational tool developers to tackle big scientific challenges ...

OpenAire to support more technical e-Infrastructures with Open Research Data guidelines, publication and training ...

Exascale supercomputing

ISC High Performance keynote forecasts future role of HPC in weather and climage prediction ...

Crowd computing

Towards equal access to digital coins ...

Quantum computing

Large groups of photons on demand - an equivalent of photonic 'integrated circuit' ...

Protecting quantum computing networks against hacking threats ...

Sorting machine for atoms ...

Focus on Europe

HPC-Europa3 provides access to European HPC systems ...

Peter the Great Saint Petersburg Polytechnic University is first university to introduce computational computer code for Airbus ...

Middleware

Introducing Hazelcast Jet: a new lightweight, distributed data processing engine ...

Bright Computing announces strategic alliance with Curtiss-Wright ...

Hardware

NVIDIA powers new class of supercomputing workstations with breakthrough capabilities for design and engineering ...

Van Andel research institute optimises HPC pipeline to drive research discoveries and new drug therapies with end-to-end DDN solution ...

CoolIT Systems issued U.S. patent for modular heat-transfer solutions ...

ORNL researchers break data transfer efficiency record ...

Cray reports 2016 full year and fourth quarter financial results ...

Supermicro deploys 30,000+ MicroBlade servers to enable one of the world's highest Efficiency (1.06 PUE) data centres ...

Mellanox ships more than 100,000 cables for next generation 100 Gb/s networks ...

UMass Amherst boosts deep learning research with powerful new GPU cluster ...

Oak Ridge National Laboratory enhances data integrity and accessibility with Active Archive Solutions ...

Applications

ANSYS spurs pervasive engineering simulation with release 18 ...

Computing sciences students: Get your team together for the SC17 Student Cluster Competition in Denver ...

Latest Allinea update advances code optimization across platforms ...

Con Edison selects C3 IoT for Big Data and predictive analytics platform and applications ...

New algorithms may revolutionize drug discoveries - and our understanding of life ...

When data's deep, dark places need to be illuminated ...

Computer trained to predict which AML patients will go into remission and which will relapse ...

The Cloud

Cycle Computing collaborates with ANSYS on its Enterprise Cloud HPC offering ...

IBM launches "Digital - Nation Africa" and invests $70 million to bring digital skills to Africa with free, Watson-powered skills platform for 25 million people ...