Psydex - Technology - AIM (All In Memory) Indexes


	TECHNOLOGY • AG (Analytics Grid) • AIM Indexes • Massive Parallel Processing • Universal Modeling • Predictive Analytics • PEP (Event Processor) • Partner Integration • Sources • Query Language • AG APIs BIG DATA - BIG THINKING The World will produce a ZetaByte of new data this year with 80% in the form of UnStructured emails, documents, text messages and social media conversations. Traditional approaches to analyzing and processing data do not work for the Real Time World. White Papers

AIM (All In Memory) Indexes

Indexes in Psydex AG are Source Specific representations of every possible way data might be queried. Indexes are constructed as AIM (All-In-Memory) dimensional graphs of words, attributes and values organized around TIME. Each Source is partitioned across Psydex AG as n Index Instances each representing a Time Slice of a source. A typical Source Index might have hundreds of Instances functioning seamlessly across many nodes. An instance resides in a single 4+ Gig process space.

Psydex Text Analytics Grid Decision Support

Elastic and Adaptable
Source-specific Index Loaders handle both streaming, real-time and historical data loading. Indexes can be easily updated and deployed with no impact to historical or real time data ingest. Failover, redundancy and network latency considerations are inherent to Psydex Index Loaders. Psydex has a comprehensive list of standard Index Loaders for common source types, as well as a framework for quickly developing custom loaders/indexers. The Grid Manager utilizes a parallel-job loading process when a Source is initialized. Sources can be initialzed in just minutes, and are able to process sustained, real-time inflows of new data by instantiating new Index Instances in an intelligent and manageable way.

Terabytes of indexed data can be brought online in minutes...

Segmentation and Threaded Execution
Each Index Instance processes an inbound request by breaking apart query expressions into many segments (the smallest executable unit) and then simultaneously executes each segment in a separate thread. When processing a request, Index Instance level threading occurs in parallel across all targeted Instances in the grid.

Separate the Knowledge
Since knowledge is consantly evolving, a key advantage of Psydex's approach to indexing is to make no assumptions about the queries that will be executed. When indexing, only tokens (e.g. words) and their positions are associated to time withno categorization or Taxonomic structure being associated with the index. Source-Specific Metadata is associated in the index with the token and time dimensioms. This assume-nothing approach to indexing allows models to evolve independently and facilitates real-time backtesting and the application of new "Knowledge" to historical data.