Negentropy

noSQL (Document-based) or Distributed Cache - a Practitioner's tale.

2011-03-19T20:59:00.000-07:00

As numerous discussions at Shutterfly and on various online posts and this blog by Greg Luck indicates, there is definitely some interest in the market in terms of understandings where Distributed Cache and NoSQL Movement use-cases intersect and where they don't.

Of course, both aim to offload the RDBMS strong-hold in the enterprise by either

Deflection (i.e. minimizing direct access to the RDBMS) - i.e. a proxy to the RDBMS or
Avoidance (i.e. allowing for some types of data to not be stored in the RDBMS to begin with).

PURE-L2 Architecture:

Usage of L1-L2 analogies in distributed-cache literature stems from multi-level cache hierarchies described in CPU architectures - basically the L1 cache is local to your client application process and the L2 is an off-host/off-process manager of cache-data.

See this animated GIF -

As can be seen, whether the off-host process that manages the cache-data is MongoD or MemcacheD or Terracotta-Server, architecturally they all look equivalent - i.e. a pure-L2 with no-L1 - so that all data needs to be retrieved from over the network and then massaged into a POJO for consumption by the application. (FYI - I used these specific technology implementations as proxies for NoSQL and Distributed Cache since I have good familiarity with Mongo (Document Based noSQL), Terracotta-Ehcache and Memcache - given that we use these 3 technologies in our stack at Shutterfly).

Given the architectural equivalence here and our runtime-observations, one could argue that in a pure-L2 architecture, using No-SQL technologies seems like the way to go (especially if there is a requirement to view the data outside of the confines of the application's Domain Model - e.g MongoDB Shell). Or at least, drop them into the same pool of candidate technologies and map features as they relate to your use-case.

For us definitely, with Mongo-DB as L2, in certain use cases
- we observed good response times (see attached image)
- we experienced good query expressability
- we could validate data manipulations expected off the application easily from the MongoDB Shell.

Although there was definitely the domain model/ BSON impedance mismatch to be overcome with Morphia annotations and APIs. For a comparison along some common dimensions, see the table below:

Dimension	MongoD	MemcacheD	Terracotta Ehcache
Retrieval	BSON from MongoD that can be mapped via Morphia into a POJO	Serialized version from MemcacheD - that needs to be de-serialized via Java-Serializaton as a POJO	Object DNA from Terracotta Server - available at the application transparently as an Ehcache-Element.
Latency (as seen by app)	1ms - more	1ms - more	1ms - more
Horizontal Scaling	Mongo Replica Sets (available in open-source)	Multiple MemcacheD (available in open-source)	Terracotta Server Cluster Array (not Available in open-source, in enterprise-kit
High Availability	Mongo Replica Sets (available in open-source)	Not a good story - no HA available for MemcacheD (change to MemcacheDB but that has its own costs)	Terracotta Server Cluster Array (not Available in open-source, in enterprise-kit
Query Expressability	Very good - see Mongo Querying and Advanced Querying	Key-Value lookup only	Key-Value although Ehcache 2.5 and onward now offer some search capability via searchability API

Of course, there are use cases where cache access in the order of milli-seconds is a non-starter - you want cache-access in the order of mirco-seconds and hence if Network-hops and Impedance-mismatches can be avoided, you want to avoid them. A cache is about local data - and assuming your application has good locality characteristics, you want to exploit L1 caching and in such a case a L1-L2 architecture offered by some of the distributed cache implementations (Terracotta-Ehcache, Oracle-Coherence) or a bloated L1 architecture (Terracotta-BigMemory) could well be the way to go.

If you have experiences or comments on MongoDB (or other noSQL) and distributed cache usage comparisons, feel free to share them or comment.
Cheers,

Terracotta - Comparisons with RDBMS/IMDB/ORDBMS/ORM/OODBMS/NoSQLDB

2010-01-28T19:11:00.000-08:00

OMG, with all these acronyms, Borat would have observed that the software industry is run by teenage girls, unlike the great nation of Kazakhstan!

(The attached animated Gif summarizes the text below at a high-level. Click on the image to see the animation.)

Anyways, Relational Databases have been around for 30+ years (the predecessors – Networked/Hierarchical databases are extinct now). They matured over years and fit the bill in the early 90s, until Internet Scale and data-intensive applications broke their back. Basic issues have always been the same:

A> IMPEDANCE-MISMATCH: Domain Model to Relational Model Conversion – The impedance mis-match between your object-oriented application domain model and the relational schema that must support it results in a lot of tedious co-ordination between middle-ware/database developers, code and time (Time to arrive at a fixed/inflexible Schema Design, Stored Procedures and Interfaces to I/O against the schema and tuning/data-archival/backup etc.). This manifests as:

o Complexity

o Greater Time-To-Market

B> RDBMS IMPLEMENTATION: Data managed by the RDBMS is in Memory and/on Disk. Arriving at a QEP (Query Execution Plan) and retrieving/updating data off Memory and/or Disk with Transactional ACID Guarantees involves complex machinery in terms of transaction-logs/redo-logs/locking/guaranteeing isolation-levels - i.e. costs in terms of processing resource and latency. Typically the RDBMS, being an "enterprise" resource also gets abused, since any application desiring persistence tends to utilize it – thereby leading to bottle-necks on the Database/Database-Machine. One’s options then are then to Scale Up the Database Infrastructure and Pay More in terms of Database Licenses (typically per CPU Licensing) and/or Invest in Clustered Database Technology (which is also expensive and a little immature to boot) - and both of these are very expensive options. Or Shard – up in the application tier. All this manifests as:

o Poor Latency with regards to Query Execution.

o Poor Scalability due to a Database-Tier Bottle-neck.

o Additional layer of complexity.

o Excessive Costs.

C> DATA-REMOTENESS: Data is always one/many hops away from the Application where the Business logic is often codified and executed. And where definitely the Presentation Layer sits. This manifests itself as:

o Increased Latency to the Application.

The business is rife with sharp engineers, greedy business-men and dedicated workaholics – and status quo does not remain so, for more than an attosecond (10¯¹⁸)! So what’s everyone been up to – to solve these problems?

A> To resolve Impedance-Mismatch, the industry’s efforts have revolved around the following approaches:

o OODBMS – Object Oriented DBMS: (Examples è Db4o, Objectivity/DB, Intersystems-Caché, Versant, Progress)

o ORDBMS – Object Relational DBMS /Universal Servers: (Examples è PostgreSQL, Cubrid, OpenLink Virtuoso)

o ORM – Object Relational Mapping Software: (Examples è Hibernate, Oracle-TopLink, IBatis, EclipseLink, Django).

B> To resolve RDBMS Implementation Issues:

o Issue of Latency due to Data Being On Disk and requiring complex QEPs, IMDB - In Memory Database: (Examples è Oracle Times-Ten, IBM Solid-DB, Sybase-ASE 15.5, MySQL Cluster, eXtremeDB)

o Brand New No-SQL Databases: (Examples è Apache HBase, SimpleDB, Project Voldemort, Facebook Cassandra, Amazon Dynamo (closed), Google BigTable (closed)).

C> To resolve Data-Remoteness, the industry’s efforts have revolved around, expectedly, solving the problem in the middle-ware and/or client-TIER with:

o Caching and Distributed Caching Technologies: (Examples è Terracotta Ehcache, Oracle Coherence)

o IMDG (In Memory Data-Grid): (Examples è Terracotta Platform (with TIM-Messaging), Hadoop, Oracle Coherence)

o Distributed Java Heap: (Examples è Terracotta Platform and special editions of the Terracotta Platform such as Terracotta Sessions, Terracotta Quartz; and a couple of other wanna-bes).

Solving the problem at the middle-tier is what folks have discovered works better and more cost effectively in that one can reduce Complexity, improve Latency and Scale/Scalability at once (See Table-A for Details) unlike the other approaches that solve one or more, but not the entire set of problems.

Table-A:

Problem	RDBMS	IMDB	No-SQLDB	OODBMS	ORDBMS	OR-Mapper	Terracotta
Solution to impedance mismatch between Relational and Object Oriented Domain Model	X	X	X	√ Reachability Persistence.	√	√ Mapping Configuration files	√ Reachability Persistence.
Latency Improvement	X	√ (Since data is in memory, Query Execution Times are improved, although Data-Remoteness problem remains).	√ (Fast since several do not attempt to provide ACID guarantees and do not require fixed table schemas and avoid join operations).	X (For some navigational and associative lookups, object graphs can be pulled up as is)	X	X YES - (if complemented with Distributed 2^nd Level Cache e.g. Terracotta Ehcache (with H2LC plugin for Hibernate)	√ (Close to in-memory speeds possible)
Improve Scale and Solve Scalability Issues	X Yes (if Ready to Spend $$$$$ on Scaling UP + DB-Cluster s/w like Oracle RAC)	X Yes (to a certain extent, but limited by amount of memory and the need for Non-volatile RAM) or compromising with Save-Points.	√ Yes (assuming eventual consistency good enough)	X	X	X YES - (with Distributed 2^nd Level Cache e.g. Terracotta Ehcache (with H2LC plugin for Hibernate)	√ (since I/O is against HA Memory)
Easy Integration to RDBMS (entrenched source of truth)	N/a	√ Typically yes (e.g. Times Ten with Oracle)	X	X	n/a	n/a	√ (Write-Behind, Write-Through (with/without JTA))
Better Support for Fast Changing Domain Models (Agile)	X	X	X	√	X	X	√
Heterogeneous Application/ Stack Support.	√	√	√	X (Typically you store strongly-typed objects in the database, which makes sharing across Apps/Heterogenous stacks difficult)	√ (Data is ultimately in relational form)	√ (Data is always in relational form – and ORM merely bridges the impedance)	√ (Java Solution – multiple apps can access the cache. Terracotta Ehcache supports EhcacheRESTful so heterogeneous Stacks can access the same cache service)
Implementation Maturity	√	?	?	?	X	√	√ (Vetted by 1000s of OSS and Commerical Deployments over 6 years)
Common Standard	√	X	X	X	X	X	X/√ (Terracotta Ehcache is JSR 107 Compliant).

So, as the argument above indicates, if you are considering any of the following:

o more expensive RDBMS upgrade,

o ORDBMS,

o IMDB,

o OODBMS,

o No-SQL DB,

look to instead solve it with middleware, whilst still leveraging your existing investments.Of the Middle/Client-Tier based solutions, Terracotta is the most compelling, since along with agreeable runtime characteristics and visibility, you get the most number of choices with regards to application expressability and deployment choices to fit your needs/budget and to trade off correctness v/s scale – see http://www.terracotta.org/products

If the data does eventually reside in a RDBMS, look at

o Terracotta Ehcache (with the Hibernate 2^nd Level Cache Plugin) if using an ORM (Hibernate)

o Terracotta Ehcache (with the ability to write Behind - see http://www.terracotta.org/beta/darwin) & join Greg Luck's upcoming Webinar at http://gregluck.com/blog/archives/2010/01/sign-up-for-a-webcast-i-am-giving-3-february-boost-application-performance-and-monitoring-with-terracotta-ehcache/).

There is plenty of quantitative evidence to support the above, in the form of white-papers and case-studies and an impressive client roster at http://www.terracotta.org.

What can Terracotta do for you in 2010.

2010-01-12T18:47:00.000-08:00

2009 was a great year for Terracotta in many ways. In terms of product releases, each release was packed with features that raised the bar in terms of value and simplicity, at least an order of magnitude. Here's a brief list:

Release of the Terracotta Server Array - i.e multiple active Terracotta Servers with the Terracotta Platform 3.0 release, to afford virtually unlimited scale-out characteristics.

Amazingly simple yet powerful, Hibernate 2nd Level Cache Product released with Terracotta Platform 3.1, offers immediate database offload for Applications that ORM (Hibernate) to the Database.

Acquisition of Ehcache and the tighter integration between this ubiquitous caching framework and the underlying Terracotta smart replication technology with 3.1.1 - in the form of Terracotta Ehcache Product Line.

Acquisition of Quartz and the tighter integration between this ubiquitous job-scheduling framework and the underlying Terracotta smart replication technology with 3.2 - in the form of the Terracotta Quartz Product Line.

And with each release, the Terracotta Platform and its smart replication technology (DSO) was solidified and improved upon in terms of Lock Management, Fault Path Optimization, Heart Beating features and Disk I/O on the Terracotta Server.

Several Visualization tools were added to the Platform and Products, mentioned above (Ehcache Panel, Hibernate Cache Region Panel, Improvements to the Terracotta Developer Console and Terracotta Operations Console).

A picture is worth a thousand words - so look at the attached animated GIF which delineates the entire suite of Terracotta Products available and where they apply within the stack.

(Seems like you need to click on the Animated image to get the animation going).

So use a specific product to solve a point problem or with the Enterprise Suite, ensure that each element within the stack benefits from simpler code-base, lesser RDBMS dependency and lower operating costs. The DX, EX, FX are basically versions along a scale continuum - so for standalone, non-distributed versions use DX, for medium scale use EX, for high-end scale use FX (since that allows for an unlimited number of active Terracotta Servers).

The power that is depicted in the animated graphic is real:

For Terracotta Ehcache look at - http://www.terracotta.org/ehcache/ and http://www.terracotta.org/ehcache/distributed-cache/ehcache-ex-hibernate.
For Terracotta Quartz look at - http://dsoguy.blogspot.com/2009/12/clustered-quartz-scheduler-in-5-minutes.html and http://www.terracotta.org/quartz/
For Terracotta Sessions look at - http://www.terracotta.org/web-sessions/
For Terracotta Platform, look at - http://www.terracotta.org/platform/

2010 Roadmap is packed with features as well, which should further simplify your consumption of Terracotta even further, with further improvements in Scaling and Latency. Enjoy....

So... you want a Terracotta-based Distributed Cache

2009-06-12T13:02:00.000-07:00

INTRODUCTION:

There are many parallels between the Eastern parable of the 6 blind monks and an elephant and Terracotta’s capabilities and usage – in that Terracotta on the one-hand ships a platform based on its flagship DSO (Distributed Shared Objects) technology, which can be exploited in a multitude of ways to concoct distributed systems manufactured with POJOs from the J2SE JDK, but then also ships many ready-made solutions (Terracotta Integration Modules – TIMs in short) that solve specific problems in the enterprise-computing landscape e.g. TIMs exist for Http-Session Clustering, Master-Worker Frameworks (i.e. Map Reduce via tim-messaging), Distributed Job Scheduling (via tim-quartz), distributed Caching (via tim-concurrent-collections, tim-EhCache) and so on and so forth. An ever increasing compendium of these "half-truths" of what Terracotta actually is can be found at http://forge.terracotta.org.

This article is focused on Distributed Cache implementations (i.e. Java-based implementations of in-memory Data-Structures – which are backed by a Data-Source such as a Relational Database) with Terracotta and builds up on prior musings on the topic by Alex Miller and self. It includes both POJO-based solutions and TIM-based solutions, with the latter affording you faster time to market as the description of TIMs above might lead you to conclude. It attempts to open the hood and describes how each distributed-cache feature is satisfied by Terracotta, describes the typical process around some common implementations within the community and also highlights what different interfaces are available to the application developer and why to choose one over the other. Would also recommend reading The clustered data structures guide in conjunction with this blog to get a thorough understanding of how you can skin this cat.

TYPICAL REQUIREMENTS OF A DISTRIBUTED CACHE:

When looking for a comprehensive Distributed Cache Solution, customers often express the following requirements consistently:

- Cache Specific Features: such as Correctness, Eviction, Disk Spill-over, Cache-Hydration, Querying, Eventing, Transactional Updates and Partitioning.

- Non Functional Features: such as High Availability, Scalability/Latency, Security, Visibility/Management and Replication capabilities with Geographically Dispersed Active or DR (Disaster Recovery) Data-centers. And in some rare cases, Cross Platform usability (e.g. Dot-Net, C++ and Java based cache-consumers)

The table below describes how Terracotta delivers these requirements, based on the several 100 production-deployments of this usage of Terracotta:

CACHE FEATURES		CORE DSO PLATFORM	TERRACOTTA TIMs	EXTERNAL (i.e.in App/Infrastucture)
Cache Specific Features:
Large Cache Sizes	Disk Spillover	Terracotta Virtual Memory ensures that you can accommodate large caches that exceed the size of client-JVM/TC-Server-JVM Heaps. Hot references are materialized in your local JVM heap, “warm” references in your Terracotta Server JVM heap and everything is on disk (assuming persistent-disk option is on).
	Cache Eviction		TTL, TTI schemes are delivered via tim-map-evictor. Fixed-Time expirations (i.e. empty cache at 5am) and custom schemes can be hand-built. Note that TIMs: EHCache, H2LC & DistributedMap, tccache TIMs already have eviction built in.
	Cache Partitioning	Terracotta Server Striping available in FX (Paid Only version). So the striped layer is basically multiple Mirror-Groups which in turn contain 1 Active Terracotta Server JVM and any number of Mirror (i.e. hot standby) Terracotta JVMs. See VIDEO.		One can also partition up at the application tier – but then the partitioning-scheme and routing is to be implemented by the application. Terracotta cluster events greatly ease partition management (Refer to Terracotta Consulting, if unsure)

Cache Correctness	Across Threads in the same JVM and across JVM Process Boundaries.	Java Memory Model is extended by DSO from Amongst Threads in a single JVM Process to Amongst Threads in a single and across JVM Process Boundaries.
	Across JVM->DB		Delivered via Tim-async that allows for asynchronous Write Behind to the database. In case of H2LC – Hibernate writes to 2^nd Level Cache on Commit. Terracotta is a transactional 2nd Level cache-provider.
	Across DB->JVM			Typically implemented externally, where database triggers keep track of DML changes and publish to a JMS-Topic and one client JVM is a subscriber to this JMS-Topic and upon consumption of the message, updates the cache. The replication of this change is done by Terracotta (assuming the cache is clustered via Terracotta DSO). Use Terracotta cluster events to innoculate against subscriber-JVM failures.

Cache Eventing		Could be implemented via – Distributed Wait/Notify or DMI and Cluster Events (Refer to Terracotta Consultants)		Or via a custom event publication mechanism (Terracotta gives you JVM level clustering, after all - so Lists, Queues etc. can all be clustered just as easily as a Map).

Cache Visibility and Management		Implemented via: - Terracotta Developer Console - Terracotta Operations Center(FX/Paid only) - JMX API - Cluster Events. -Locality API	H2LC Tim features a specialized Management Console based on Hibernate Statistics (TC 3.1 onwards) – that pinpoints queries being offloaded off the database, cache-hit-ratios etc. and is very useful in determining the efficacy of the 2ndLevel Cache in Hibernate based Apps.

Cache Indexing/ Querying			Searchable Map exports an enhanced Map interfaces and supports searches via a local Lucene Data-Store, whose disk-based-Index is kept in synch across JVMs with Terracotta DSO.

Transactional Cache Updates			Terracotta Hibernate 2nd Level Cache Implementation supports (shortly) a Transactional Cache. Similarly, tim-async's implementation provides guarantees around once-and-only-once write to the database with no transaction loss.	Also, often implemented in the App, currently. e.g. in case of a write-through cache - Clustered cache is updated only after the transaction is committed to the database. Or in case of queues to be flushed to the db - double-headed-queues are popular - i.e. "txn-begin = pop-work-off-queue1", process and "txn-end = pop-work-off-queue2" (refer to Terracotta Consultants)

Cache Preloading		Achieved via Fault-count tuning. It is a global setting though (Consultants know how to reflectively change this at run-time). More features are on the way to make cache-pre Loading across client-JVM/TCServer-JVMs virtually painless.	Alternatively one could implement Maps with non-partial behaviors – so that the entire cache can be faulted in one go into the client JVM from the Terracotta Server JVM (or from disk into the Terracotta Server JVM).	The Solutions described to the left were about pre-loading between Terracotta Server and Client JVMs. To pre-load from the data-source, typically cache warmers have to be written currently at the application tier. Note that you should be careful about coarsening your locks within your application code in case of bulk-loads. Expect more TIM offerings here soon. (Refer to Terracotta Consultants for now)
Cache Non Functional Features:
High Availability	Un-Planned Down-Time	Terracotta DSO + Network Active Passive technology ensures that you have no single point of failure in your architecture and that you are protected against some double/ multiple failures.In persistent-mode, all state received by the Terracotta server is written to disk as well - giving you much higher levels of HA than traditional, pure-N-memory-locations-based approaches.
	Planned Down-Time	Terracotta helps you with several planned downtime scenarios in that you can execute rolling upgrades across the cluster, for those scenarios. More detailed discussions are documented in Terracotta run-books (Documentation included in Paid Versions).

Security				Follow Terracotta Security Best Practices

Disaster Recovery		Could be implemented via Terracotta Backup (available in the FX – Paid Version) or via an additional Hot-Standby Node in the DR (assuming it is ok to impact every live-transaction by a marginal amount). See notes…on this topic		Could be implemented in the application via client-JVMs publishing via JMS to a receiving JVM in the DR Site.

Geographically Dispersed Data Centers				Recommend local clusters and application implemented JMS-based (apacheMQ for example) mechanisms if synchronization is needed (or refer to Terracotta Consultants)

Hetergogenous Platform Support (.Net, C#, C++, Java based client JVMs etc.)				Terracotta is Java only solution - however non-Java Clients can talk to a Java-based Soap Service that shares state with the Java-based cluster. Ofcourse object identity contracts would be broken. (or refer to Terracotta Consultants)

Note that Scalability/Latency has deliberately not been included as a non-functional feature, since IMHO that is an emergent property of the architecture/implementation.

-First off, given that application threads are executing I/O against shared in-memory Data-structures as against the RDBMS (other data-store) automatically gives you significantly better latencies and scale.

-Additionally, Terracotta replicates very smart with only field-level fine-grained changes darting around the network to only those JVMs that have those references materialized.

- Thirdly Terracotta implements Greedy-Locking (just like the JVM’s BiasedLocking), LockGC and Lock Lease Policies – which ensure no lock-starvation and significantly reduce distributed lock overheads.

- Fourthly, if the cache-size is too large to fit onto a single Terracotta Server JVM’s heap and you are faced with demanding latency SLAs, you can stripe the Terracotta Server tier to ensure that all clustered data fits onto Heap on the Terracotta Server JVMs.

Terracotta thus offers you an extremely scalable implementation, where you can scale out the number of client JVMs and the number of Terracotta-Server-JVM Mirror-Groups based on the application load and the amount of clustered-I/O it executes – all on commodity hardware with the commodity Hotspot JVM you have grown to love.

DEVELOPER OPTIONS:

As a developer you have many options in terms of what cache-interface you choose - POJOs, Spring and TIMs (Modules) …If anything the trick with Terracotta is to choose the right implementation for your use-case out of the several available. The section below tries to answer that specific question.

Note that –

- Non-functional features such as

o Cache-Correctness (intra and inter-JVM – Terracotta provides the highest levels of correctness out of the box - since no other product tries to honor the JMM across process-boundaries),

o HA (very high-levels of HA given Disk Persistence and NAP),

o Scale

- And some cache-specific features such as:

o Cache-Pre-Load

o Disk-Spillover

o Partitioning

- Are provided at the core-platform layer – i.e. via Terracotta flagship technology of JVM-level clustering (DSO or Distributed Shared Objects).

- Some features are to be implemented as Best Practices, externally in the application and/or infrastructure:

o WAN/DR Support,

o Security

So they are not depicted in the table below. Note that you can use an existing TIM or package your integration as a TIM.

APPLICATION INTERFACES		WHEN TO USE THIS INTERFACE	What TIMs do I need.	CACHE FEATURES
				Eviction (TTL, TTI)	Cache- specific Visibility/ Mgmt	Querying	Eventing

DSO Platform provides				Locality API	Tools such as TDC, Ops-Center etc.		Cluster Events and JMX API
Terracotta Modules/ TIMs
	H2LC	Application ORMs to the Database via Hibernate and is in need of a highly scalable, available, Distributed 2^nd Level Cache - i.e. a Hibernate Accelerator (3.1)	tim-hibernate tim-hibernate-cache tim-map-evictor tim-concurrent-collections	√	√ (via hibernate statistics)	x	x
	EHCACHE	Application is agnostic to Cache Store Implementation or is already using EHCache as a Cache Store.	tim-ehcache-* tim-ehcache-commons	√	x	x	x
	DISTRIBUTED MAP BUILDER	If Cache-keys are Strings and you want a built-in Evictor.	tim-map-evictor, tim-concurrent-collections	√	x	x	x
	TCCACHE	Do not care about identity and want a distributed store with a get/put API – This will free you from having to deal with Correct Locking (which is needed for Object Identity preservation) and Instrumentation sections in tc-config.xml (since what is in Terracotta is a byte-array)	tim-tccache, tim-concurrent-collections	√	x	x	x
	CSM	If Cache-Keys are Strings and you plan on implementing your own evictor – i.e. you just need a performant distributed data-store with identity.	tim-concurrent-collections	x	x	x	x
	SEARCHABLE MAP	If you want to be able to query your Map and get returned a sub-set and otherwise maintain a Map-Like interface	tim-searchable	√ (coming soon)	x	√	√
	ESOTERIC MAP Variations	If you want a Non Partial HashMap or a , Non-Partial/Non- Clearable (Datastructures that are clearable by the Terracotta Virtual Memory Manager implement the Clearable interface) HashMap	tim-maps, or tim-tree-map-cache.	x	x	x	x

JDK	CHM	High concurrency (given striped locks) and Key is not a String (If Key is a string, use CSM).	None tc-config.xml	x	x	x	x
	HashMap	If you might be doing multi-segment operations on a CHM or may need coarser locking that Collection-Level APIs such as get/put – then use a HashMap since it is not implicitly locked by TC like a CHM is.	None tc-config.xml	x	x	x	x
	Map (HashMaps)	Terracotta does not implement partial keys. Therefore use this if the #keys are so large that the keys alone won't ever fit into a JVM	Tim-maps	x	x	x	x

SPRING		Spring just injects the framework /POJO and you cluster the underlying frameworks/ POJOs – see Examples of injecting Terracotta-clustered EHcache, Hibernate, Quartz etc. via Spring.

COMMON TERRACOTTA-BASED DISTRIBUTED CACHE IMPLEMENTATION:

If one samples the hundreds of Production Deployments of Distributed Caches with Terracotta, 2-3 winners out of the 10 odd possible options (described in the above section) easily emerge. The rest are more esoteric and specific users have chosen those options/data-structures given some specific requirements in their use-cases. The 2-3 "market-share-based-winners" include:

- EHCache via tim-ehcache

- Hibernate 2^nd Level Cache via tim-hibernate-cache/tim-hibernate (with EHCache as the 2^nd level Cache provider in pre Terracotta 3.x versions and with Terracotta ConcurrentStringMap (CSM) as the 2^nd Level Cache Provider), if the ORM is in the application’s data-access path.

-Distributed-Map via tim-concurrent-collections.

The approach is to marry the data-structure and interfaces (i.e. the cache) with the smart (fine-grained), transparent (no java-serialization/ externalization), correct (in conformance with the JMM) replication capabilities afforded by the Terracotta-DSO-platform to arrive at a scalable, highly-available Distributed Cache.

Here is a description of the typical SDLC for a EHCache-Based Distributed-Cache Implementation, for example.

- INTEGRATION:

o Packaged as a TIM – so you just need tim-ehcache-commons and tim-ehcache (and possibly tim-async) …Modules. Based on what keys/values are being dumped into the EHCache Store, you may need tweaks to the tc-config.xml. Refer to this EHCache integration manual.

o Usage of tim-async is optional since it would depend on what the application can tolerate. Use it if you can write-behind to the database. If you need to write-through to the database – then the Hibernate integration would accommodate you, but in applications without Hibernate, recommend writing to the database and on-db-Transaction-commit writing to the clustered-cache (of course, it depends on the particulars of your application).

o Other best practices are to be aware of:

§ Application TTL and TTI – setting it too low can result in too much garbage on the Terracotta Server and setting it too high increases cache-size needlessly (not to mention data-freshness concerns, if the database receives DML from underneath the application).

§ Read/ Write Ratio – (Terracotta does exceedingly well in all these scenarios, but the value you derive is different)

· All Read / No Write (except to hydrate): You deflect N-1 Queries off the database assuming N is the size of your cluster.

· Mostly Read / Some Write (update): Database Deflection + Correctness across JVMs.

· Mostly Write/ Some Read: Database Deflection + Correctness across JVMs. Might need transaction tuning as well.

- TESTING/TUNING:

o Post Functional Testing, the next step is to usually execute stress tests - to simulate production-level cache-sizes, TTL/TTI and Read/Write ratios and other Cache I/O – tuning here to meet SLAs would typically involve:

§ Instrumentation Tuning – may require you to reduce the scope of the instrumentation in your tc-config.xml. This depends on the object graphs that comprise the keys and values of the elements that are within the Cache.

§ Lock tuning -

· EhCache.concurrency may need to be changed from the default of 127. Higher concurrency allows multiple threads to perform simultaneous operations against EhCache.

· Lock Levels: Set ehcache.lock.readLevel = READ to something more optimistic if the application allows it. If application can afford to read stale data, set ehcache.lock.readLevel = NO_LOCK. This allows read to be performed without acquiring any lock, though the application runs the risk of not reading latest data/changes (and exceptions in some cases) – so not recommended unless the cache is more-or-less read-only and even then with caution.

§ Eviction tuning -

· Tune ehcache.global.eviction* properties and

· Based on application needs try setting 'timeToIdleSeconds and/or timeToLiveSeconds' inside ehcache.xml to reasonable values.

§ Memory Tuning -

· Garbage Collection Tuning.

· Distributed Garbage Collection Tuning.

· Fault/Flush Tuning if needed, based on cache-sizes and available Heap on Client and Terracotta Server JVMs.

§ Transaction Tuning, if needed, to exploit the core-platform’s batching, windowing and transaction-folding optimizations and based on whether the application tends to write in a burst or creates a lot of cache-objects.

- HARDENING and ROLLOUT:

o Once done – you may or may not wish to execute Failure Tests in your specific environment (Terracotta does execute close to 25+ HA tests to guarantee no Single Points of Failure in the architecture) and/or Long Running Tests. Of course, we recommend these, especially if the default heartbeats and tolerances and failover timings do not meet your SLA.

o Getting ready for production additional involves ensuring Monitoring is in place and the technology is socialized amongst the Operational Staff who run and maintain the cluster. Various Terracotta run-books are very useful here (available with the Terracotta subscription). Your implementation may end up then looking like what is depicted in Figure1:

SUMMARY:

In summary you have many options in terms of Interfaces and the core-platform provides for the HA, Scale, Visibility and other non-functional features that a distributed Cache demands - You can read more at http://www.terracotta.org. to get to an implementation with Terracotta looks something like what is depicted here:

Hope this was useful both in terms of an overview and in terms of useful pointers to specific issues in your march across the SDLC.

Terracotta Usage Amongst Content Portals

2009-01-21T21:14:00.000-08:00

TYPICAL REQUIREMENTS:

Content Creation: A combination of
- User-Generated
- Feeds from Content Providers (e.g. Movies, Weather from Yahoo etc.)
- Editorially Generated Content

Content Publishing: On demand, once approved.

Content Portal sites often feature pay-for-content services that imply:
- User Subscription/Registration.
- User Authentication and Authorization

Traditional Distribution of Caches that query the DB for User Profile Data and/or Content-Attributes/Specific Content

TYPICAL IMPLEMENTATION:

Typically Content Creation/Publication has been implemented:

Against the database in that the CMS publishes to the database and any concomitant caches are thereafter cleared. This has 2 issues:
- Database can get overloaded depending on the amount of content streaming in and based on the fact that caches get cleared once the new content gets published, which then results in a spike in database queries.
- To hedge against spikes, often the caches are cleared only at graveyard-hours on the site. So, often there is a delay between content publication and availability for consumption.

With Terracotta:
- You solve both problems. THE CMS application would publish against in memory Java data-structures which are replicated to JVMs via Terracotta that serve up the content to users of the Content Portal.
- One thus obviates the usage of the database and cache clearing and hence content is available as soon as possible.

Example:
- Portal consists of multiple portlets. One portlet displays weather. Content for the portlet streams in through a Yahoo Feed. Weather for 5 zip-codes change - publication is done in-memory to Java data-structures. (e.g. to specific elements in a CHM where key is ZipCode, Value is Weather-Forecast Object and the CHM is shared between Publication JVMs and Consumption JVMs).

Terracotta Solution Considerations (see figure):
- Publishing Application and Consumption Application are implemented as two different applications.
- The applications MUST be factored in such a way that both share common data structures that represents the content.
- Class Loader issues: Since these are 2 different applications, they feature different class loaders - so a common classloader is needed
- EXAMPLE:
  - Tomcat appserver. CMS app deployed as CMSApp Context Root and Consuming app deployed as UserApp ContextRoot.
  - A class loader is created for each web application. The CMSApp ClassLoader loads all Unpacked-Classes+Resources in the CMSApp/WEB-INF/classes directory & Classes+Jars in the CMSApp/WEB-INF/lib directory of your web application archive, and are made visible to the containing web application, but to no others. The UserApp ClassLoader loads all Unpacked-Classes+Resources in UserApp/WEB-INF/classes and Classes+Jars in UserApp/WEB-INF/lib directory of your web application
  - Given this visibility problem now, you would need to have classes loaded off the Shared ClassLoader, so that all unpacked classes and resources in $CATALINA_BASE/shared/classes, as well as classes and resources in JAR files under $CATALINA_BASE/shared/lib, are made visible to all (and thus CMSApp and UserApp) Web Applications and the Classloader name across CMSApp/ UserApp remain identical (Terracotta requirement)

Subscription and User Authentication and Authorization:
- Typically authentication workflow and authorization managed via Http-session.Terracotta gives you multiple options here
Traditional Distributed Caching: See http://www.terracotta.org/web/display/orgsite/Data+Caching and http://javamuse.blogspot.com/2008/11/distributed-cachingdatabase-offload.html for caching JDBC/ORM querying of the database for Specific Content

Other examples of Terracotta being used in Content Portals include Message Board Clustering (e.g JForum and others) and special case usages.

Hope those of you in the Portal business find this useful.

Terracotta Usage in Ecommerce

2009-01-13T14:56:00.000-08:00

As this Computer-world article indicates, even as late as 2008, most E-Commerce sites have not yet solved their scale and availability issues, resulting in serious business losses, at a time they can ill afford to do so. Most ECommerce sites are implemented in Java and causes of outage of course vary widely, but typically usual suspects include (but are not limited to) the following:

Malicious DOS Attacks/ Unintentional but excessive "Spidering"
Poor Capacity Planning at the network tier:

Poor or Non-Existent CDN strategy (so no traffic is deflected off Source site)
Load Balancer being overwhelmed.
Network Bandwidth saturation

Poor Systems infrastructure
Misconfigured HTTP Settings (i.e. the several parameters in httpd.conf, if Apache)
Misconfigured App-Server Settings (i.e. the several parameters in web.xml/server.xml if Tomcat, Connector related settings etc.)
Poor Garbage Collection tuning on JVMs (e.g. large # GCs, long Full GCs).
Database/Application MisConfiguration (e.g. sizing of SGA, DB Connection Pool etc.)
Software Development Issues:

Database overwhelmed in terms of Reads/Writes; Quality of PL/SQL Algorithms/Code (e.g. DB Locks, Full table scans etc.)
Poor Application Architecture/ Design/ Implementation/ Configuration.

The last one - i.e. poor design and implementation of servlets that constitute the Commerce Site and Backend Systems is certainly something that consumes the bulk of the Development team's effort/time. A common theme here amongst front-end Java Developers is the over-dependence on a already heavily used database - given that it is there and provides persistence. This seriously limits scale, so that during traffic spikes, when the business needs the Systems to be most available (given the number of prospective customers) is exactly when the system blacks-out or browns-out.

Terracotta can help reduce and obviate RDBMS usage in many cases and allow you to safely operate "in-memory", that is durable across JVM life-cycles - so that you get scale and HA while allowing you to maintain a POJO-based programming environment - see http://www.terracotta.org and especially, http://www.terracotta.org/web/display/orgsite/Kill+Your+Database

LET US SEE HOW, Terracotta would add value specifically in E-Commerce Applications. Typically, the E-commerce business involves both FRONT-END (Websites, SOAP services etc.) and BACK-END systems (Supply Chain Planning/Fulfillment/Warehousing) systems. i.e.

A> ORDER ACQUISITION SYSTEMS to enable:

SUB SYSTEM	PURPOSE	COMMON TERMINOLOGY/ PROBLEMS	COMMON SOLUTIONS/ PROBLEMS with the SOLUTIONS	TERRACOTTA VALUE-ADD
ELECTRONIC CATALOG CREATION/ MODIFICATION	MERCHANTS decide on what to sell. Procure inventory through BUYERS. Product goes through large workflow(akin to CMS) before ready for publication to catalog (e.g. copy, price, description, photography etc.) as a saleable item.	ITEM CREATION/CATALOG MODIFICATION: Typically, each these state changes via this CMS-like Workflow are typically stored in a RDBMS.	Employ a home-grown or off-shelf CMS or Document Management System before final publication to the catalog - database.	You could obviate the need for the RDBMS and maintain workflow state in Terracotta, given that Terracotta provides durable, Network Attached Memory for your Java Applications . See: http://www.terracotta.org/web/display/orgsite/What+Is+Terracotta and http://orionl.blogspot.com/2008/03/terracotta-is-not-object-cache.html HOWEVER, given the volumes and the fact that this is an internal company process, usually scale/HA is not a serious problem, so there may not be much ROI to be gained with Terracotta here.
ELECTRONIC CATALOG PRESENTATION	Users browse through a pre-determined product classification hierarchy - Departments/Categories/Sub-Categories/Shelves	BROWSE: Not caching Browsing activity leads to RDBMS saturation especially under high volume.Most sites typically report that 90% of all activitiy on the site is BROWSE/SEARCH.	CATALOG CACHE - i.e. local Cache on each JVM of Catalog Database queries	Distribute the CATALAOG-CACHE via Terracotta with minimal / no work. This implies that the database sees the query for a given Key, just once and otherwise results are served from cache. Terracotta is about as efficient as it can get with regards to minimizing replication payload. CATALOG-CACHE is presumably mostly read, but with Terracotta distributing it and providing correctness guarantees across JVMs in the cluster, you could allow intra-day updates. See: https://www.terracotta.org/web/display/orgsite/Data+Caching and http://javamuse.blogspot.com/2008/11/distributed-cachingdatabase-offload.html
	Users search for specific products.	SEARCH: Not caching Search activity leads to Search Engine saturation especially under high volume	Implement SEARCH CACHE on each local JVM	Distribute the SEARCH CACHE via Terracotta. Search Engine sees a query for a given key just once and therafter results are served from cache. Terracotta is about as efficient as it can get with regards to minimizing replication payload. Mostly read. See https://www.terracotta.org/web/display/orgsite/Data+Caching and http://javamuse.blogspot.com/2008/11/distributed-cachingdatabase-offload.html
	Inventory position (i.e. Availability Status) being up to date is of utmost importance. If Out-Of-Stock displayed (if inventory currently exists) then lost sales imply opportunity cost. If In-Stock displayed and item out of stock, there is a fulfillment issue	INVENTORY CACHE: If not cached, Database saturated. If cached - typically, Inventory cache not up to date vis-à-vis back-end systems and inconsistent across JVMs	INVENTORY CACHE is locally cached and when it changes within the database, the change is distributed via JMS. Alternatively, there is no change propagation but one could keep the INVENTORY CACHE TTL very low (e.g. a few minutes). In either case, there is a risk of the CACHE position being incorrectly reflected across JVMs at any point in time.	Distribute the Inventory Cache via Terracotta - you are guaranteed inter-JVM correctness and you simplify code by reducing dependency on JMS and other mechanisms of state replication. High writes/reads (Comparison Shopping WebSites may involve frequent price changes as well) See https://www.terracotta.org/web/display/orgsite/Data+Caching and http://javamuse.blogspot.com/2008/11/distributed-cachingdatabase-offload.html
USER AUTHENTICATION, INTEREST and PURCHASE	Mechanisms to identify the user, and allowing the user to express Desire for a product and Enabling the retail Transaction.	AUTHENTICATION/ AUTHORIZATION information - needs to be preserved across requests since HTTP is a stateless protocol. Typically state preserved in a HTTP SESSION that is keyed off session-id and session-id is written as a cookie to the user's browser.	If HTTP Session is not replicated and persisted elsewhere- losing a JVM would imply a poor user experience, since the user would have to re-authenticate and/or re-establish their position in a workflow.	Cluster HTTP SESSIONS with Terracotta for High Availability. See: http://www.terracotta.org/web/display/orgsite/HTTP+Session+Clustering
	Allow the user to express interest in a basket of products	SHOPPING CART/ WISH LISTS	Stored in HTTP Session or Cache keyed off customer-id (if registered customer) / visitor-id (if unregistered customer). If HTTP Session is not replicated and persisted elsewhere- losing a JVM would imply a poor user experience, since the user would lose his/her cart	Cluster HTTP Session with Terracotta for high availability of HTTP Sessions. See http://www.terracotta.org/web/display/orgsite/HTTP+Session+Clustering and https://www.terracotta.org/web/display/orgsite/User+Session
	Enable execution of the buying transaction	CHECKOUT - typically implemented as a workflow since it is a multi-stage process across several HTTP Requests involving Credit-Card information, Shipping information etc.	Users position in the workflow and pre-validated input is typically stored in the HTTP Session.If HTTP Session is not replicated and persisted elsewhere- losing a JVM would imply a poor user experience, since the user would be thrown out of the checkout process and have to restart all over	HTTP SESSIONS clustered with Terracotta for High Availability. One could even hedge against DB unavailability by collecting the orders in a LinkedBlockQueue for example (made durable by Terracotta) and later draining it asynchronously to the database See http://www.terracotta.org/web/display/orgsite/HTTP+Session+Clustering and https://www.terracotta.org/web/display/orgsite/User+Session and http://forge.terracotta.org/releases/projects/tim-async

B> ORDER FULFILLMENT SYSTEMS:

SUB SYSTEM	PURPOSE	COMMON TERMINOLOGY/ PROBLEMS	COMMON SOLUTIONS/ PROBLEMS with the SOLUTIONS	TERRACOTTA VALUE-ADD
OMS (ORDER MANAGEMENT SYSTEMS)	Decomposition of the order into order line-items (e.g. an order may include flowers and basket balls each of which is fulfilled by a different distributor/warehouse)	OMS typically requires a fair bit of complex algorithmic knowledge to arrive at the right decomposition of Orders into Order-line items	Typically executed as a DBMS job	The database is the right place to do this work given the algorithmic complexity and the amount of data to be referenced.
FULFILLMENT SYSTEMS	Now that orders have been collected, there is a whole machinery needed to actually deliver the product to the customer.	FULFILLMENT SYSTEMS: Must handle B2B work with 3rd party distributors and warehouses along with order line-item lifecycle	Typically state-changes to the order-line-item lifecycle are reflected within a Fulfillment and Order Management database, which then results in backlogs from "Order Confirmation" Emails to updates of the status of the order line-item as it proceeds through the warehouse.	By doing some of the computations around how order-line-items get divvied up and sequenced amongst distributors and absorbing state changes in memory and later asynchronously flushing these to the database, you can scale-out your fulfillment systems. Sending out emails to customers to keep them notified of fulfillment movement can be scaled out across multiple "Worker" JVMs. See Terracotta Master Worker framework at http://www.terracotta.org/web/display/orgsite/Master+Worker and http://www.terracotta.org/web/display/orgsite/Batch+Processing which allows for a useful way to partition work amongst a set of participating JVMs and also see http://forge.terracotta.org/releases/projects/tim-async/
CUSTOMER SERVICE	Deal with customers who have had issues either during placement of the order or in terms of the order fulfillment i.e. returns, etc.	CUSTOMER SERVICE call spike could impact the Customer and Order Database in terms of read/write activity.	A complex set of entities (Customers, Order History, Fulfillment status of specific orders etc.) need to be cached. Problems with caching this data per customer is that cache-hit ratio is very low. (i.e. the customer profile is typically needed for just the one contact the customer may have with customer service).	One could pre-hydrate certain portions of the cache (e.g. if a spike of calls around a particular product) and/or Distribute a Reference Cache of Products to cut database latency to a certain extent.

IN SUMMARY: You could use Terracotta for:

HTTP SESSION CLUSTERING:
- To provide high availability to your HTTP SESSION(which likely contains user-authentication information, shopping cart, position within checkout).
- Terracotta's Session Clustering implementation is the best in the market in terms of minimizing session replication overhead, transparency and visibility.
CACHE DISTRIBUTION:
- To distribute CATALOG CACHE, SEARCH CACHE, INVENTORY CACHE and elements of the family of Caches needed for CUSTOMER SERVICE.
- Terracotta supports a variety of distributed caches (from EHCache to Hibernate to POJOs implemented as Hashmaps, CHMs or any arbitrary collection grouping etc.)
INTER-JVM CO-ORDINATION/BATCH PROCESSING
- To co-ordinate amongst participating JVMS for Customer notification, Work allocation amongst warehouses, Update of Fulfillment life-cycle changes (asyncrhonously drained to the database), etc.
- Terracotta master-worker framework encapsulates the distribution of work amongst "workers" and aggregation of the output of individual "workers" and exposes some interfaces for you to implement any application specific logic.
In addition, there are several other systems that comprise a E-Commerce business:
- WAREHOUSE MANAGEMENT SOFTWARE: Planning for work for each shift of the warehouse and Enabling the execution of the Pick, Cartonization, Shipment of the packaged product. (e.g. Retek)
- WAREHOUSE PLANNING: Linear Programming Exercise to minimize shipping costs given demand forecast and Warehouse Locations and Fulfillment portfolio (e.g. Manugistics, ILog).
- DEMAND FORECASTING: Forecasting demand based on history and new trends. Grouping the demand temporally and geographically (e.g. Manugistics, i2 etc.)
- BUSINESS METRICS MANAGEMENT: Recording user click-trail and make marketing/merchandising decisions (e.g. Coremetrics, Omniture etc.) and SEVERAL OTHERS
- However, these are often procured as shrink-wrapped software from other vendors. So you may have more luck persuading the vendor to partner with Terracotta if scale and HA of the solution are unsatisfactory. Alternatively, if these are home grown Java based systems, you could investigate what data-structures need to be DSO (Distributed Shared Objects), so that they become durable, Since Terracotta is a general purpose platform that clusters at the JVM-Level - it is applicable in any POJO based app. See http://www.terracotta.org/web/display/orgsite/JVM+Level+Clustering

See http://www.terracotta.org and http://www.terracottatech.com for more detail.

Distributed Caching/Database Offload with Terracotta DSO

2008-11-30T15:17:00.000-08:00

You've probably heard about how Terracotta DSO (Terracotta's flagship general purpose JVM-level clustering technology) can help you offload your database. You have a variety of choices depending on your implementation and your goals - This article tries to bring them all together in 1 place so you could more easily determine what Terracotta offering better suits your needs.

See the attached diagram for a decision matrix of sorts. The Green Lozenges lead you to products on terracotta.org that might best fit your stack.

In a nutshell:

At a high-level, the first step is to acknowlede the problem with the implementation.
This problem will typically manifest itself as UNACCEPTABLE LATENCY associated with certain application transactions OR as DATABASE RESOURCES Being PEGGED (e.g. CPU running uncomfortably high, for the DBA/SysAdmin's comfort).
Now, it follows that if the data the App is mutating is "close" to the application then that is much better from a latency perspective, than if the data has to be fetched off the network from your SGA/Disk that comprises your Oracle Database Instance. This data may rightfully live on the database but it will still greatly improve latency if all operations were done in memory against pre-fabricated Domain Model Objects. Enter CACHING. Now your application presumably runs on a cluster of JVMs - so just caching locally on a given JVM, would render the caches inconsistent amongst the JVMs, if these caches and or the Database are also receiving updates to the underlying data. Enter DISTRIBUTED CACHING - where all Read and Write access now occurs in Memory across all the cluster JVMs (and is kept in synch with the database - by "writing-through" or "writing-behind"). i.e. Decide if you just need Caching (e.g. read only data set), or you need Distributed Caching ( a data-set that is being modified by the application as well. A Distributed Cache for a read-only data-set only helps in reducing query load to the database by 1/N where N is the number of client-JVMs in the cluster).
Once you've established that Distributed Caching is the solution - decide on whether changes need to be persisted to the Database, and if so, need to be written synchronously (write through) or asynchronously (write behind).
Decide on whether the application can tolerate any incorrectness amongst threads on different JVMs reading/writing to the cache. Terracotta is really the only solution, in my opinion, if the application cannot tolerate ANY incoherent concurrent access, whilst still delivering scale/performance.

In summary:
A> There are several technologies that enable "DISTRIBUTED CACHES". So in that usage - yes, Terracotta is one of many such "distributed cache" products in the market - but Terracotta's Networked Attached Memory innovation and implementation offer some unique and powerful advantages:

Preservation of your Java Programming Model - you write natural Java - no new API to program to. You could thus cluster your own custom caches be they implemented as Hashmaps, Hastables, ConcurrentHashMaps, LinkedHashMaps etc.
Correctness guarantees (as defined by the Java Memory Model) of access across the cluster of participating JVMs.
Ready Made clustering solutions (Terracotta Integration Modules - TIMs) for popular OSS caches (e.g. EHcache etc.).
Ready Made support for ORMs (Terracotta Integration Modules - TIMs): Hibernate (Detached-Mode, 2nd Level Cache); Ibatis etc.
"Intelligent replication" to minimize Network Chatter and Network Replication payloads (byte code injection gurantees only mutated Object Fields are shipped to at least 1 other element in the network - the Terracotta Server (and more than 1 if other elements in the cluster reference that field)).
Abilitity for the cache to exceed the size of HEAP on a given application JVM without the need for partitioning (assuming somewhat Gaussian access patterns across key-space).
In depth Monitoring and Visibility via the Developer Console and the Terracotta Operations Center.
L1 (component in client JVM), L2 (dedicated Terracotta JVM) design with persistence affords higher Level of HA - e.g. cache preserved even if all JVMs are lost. HA is in the box - in that no Shared Disk be provisioned.
JMX Events and Hooks to respond programatically to cluster situations such as Client JVM failure or new Client JVM introduction.
OSS Model of Distribution and many others.....

In addition,
B> Evaluate if certain data sets in the database really needs to be persisted in the database or if the persistence afforded by Terracotta suffices. Typically - data should be in your RDBMS/System of Record

If it is long-lived and business-critical. See http://tech.puredanger.com/2008/08/01/thinking-about-data-lifetimes/
If the Data need to be Reported on?
If the Data need to be queried extensively along many diverse criteria - SQL is much more expressive than OQL or any such option? (e.g. if you have an entity X with 3 attributes and Y with 4 attributes where 1 attribute is common across them then you can query X alone in 3C1 + 3C2 + 3C3 = 7 ways and Y alone in 4C1 + 4C2 + 4C3 + 4C4 = 15 ways and X & Y together in multiple other ways).

In cases other than the ones mentioned above - there is a strong possibility that you could just work off memory, assuming some technology like Terracotta gave you HA for that chunk of memory. Advantages apart from DB Offload is the simpler code base - since one simply works off the Java Domain Model, where Terracotta delivers HA and correctness of access for those data structures, without the need to involve a Database and the marshalling/unmarshalling code to overcome the impedance mismatch between the object world and the relational world. This has been proven out on several occasions now - you can get more at http://www.terracotta.org/web/display/orgsite/Common+Use+Patterns

So hopefully after reading this, it is easier for you to determine which Terracotta offering best solves your data-base offload problem....

Tuning, Testing, Deploying Terracotta Implementations - how does my application time-to-market change?

2008-06-25T20:56:00.000-07:00

I work at Terracotta with the Field-Engineering Team. What we have noticed over 100s of implementations, is that a lot of technology-adaptors thus far, are pretty adept at getting to Dev Completion with Terracotta-DSO on their own. This coupled with the Terracotta Integration Module strategy (see http://www.terracotta.org/confluence/display/integrations/Home ) makes me believe that Terracotta’s promise of maintaining natural Java Semantics and hence reducing the level of effort needed by the Developer to cluster his/her app is, by and large, very real.

Another observation though is that, self-adopters while successful at getting to Integration-Complete phases of their projects, have needed, relatively speaking, more hand-holding with regards to tuning, testing and deploying their applications into production. Reasons are:

For the most part, programming languages feature no specification of behavioral characteristics outside of the contractual interface specification of inputs and resultant outputs. As an example, let's take the notion of sizing collections. We all understand that one should pre-size collections, so as to avoid expensive rehashing at run-time. At a default of 16 and load facotr of .75, as soon as the size of the Hashtable gets to 12, we are already paying the cost of a rehash (to roughly 32).But very few developers actually size their collections due to whatever reason (ignorance, negligence, or genuine inability to predict the collection-size). It becomes a later mop-up tuning task, as against being accounted for during implementation....Now a Hashtable/Hashmap with some self-correcting size related heuristics or some eventing on rehash (so one can react to it programmatically) would be useful - but in the absence of one, there is no substitute to competence when implementing.
The above argument is to highlight that even with technologies (such as Terracotta) that modify byte-code to give you desirable characteristics of add-a-node-predictive-scale and High Availability, there is really no substitute for thought and discipline when engineering, rigor during testing and scientific methodologies when stress-testing and deploying.
Of course, Terracotta is a byte-code enhancing technology and is working on providing the end-user more tooling, more visibility and more rules/inference-engines to suggest possible root-causes – and hence ease the process across the entire SDLC. You already get the gist of this approach via the Snapshot Visualization Tool and there is a lot more, coming down the pipe.

So then assuming that you have had to make some delta-changes to your application to get to Dev-Completion (e.g. initialize Terracotta transients , modify the coarse-ness of locks etc.) what is the typical time spent around Tuning, Testing, Deploying ? Perhaps these comparisons between any Java Application and a Terracotta-integrated Java Application can highlight the delta involved - this way you could have a clearer understanding of how time-to-market for your application changes, when you integrate Terracotta from POC-Production.
(Note that java Performance tuning is such a vast topic – and so we aren’t talking about all possible ways to enhance performance/scale of your Java app itself, so no NIO or AWT improvements for example - just identifying what one typically does during Terracotta tuning and how really a fair bit of it is similar to what one would do, without Terracotta in the picture, in the first place.)

Tuning Terracotta: See here for several common practices:

Programmer Consideration		Plain Old Java Application (POJA)	Terracotta integrated Plain Old Java application (TI-POJA)

Data structures	Size	Validate right choices of Data structure based on expected size of the structure.	Similar to a POJA. Additional Considerations: · Whether a collection is “partial” or not (Hashmap, CHM, Hashtable, LinkedHashMap, Arrays are partial – others are not as of 2.6). · Does the object-graph play well with the Terracotta Virtual Memory Manager
	Pattern of Access	If lookup based on key use associative array (e.g. Hashmap) instead of a Vector etc.	Identical to a POJA
	Extent of I/O against the collection and Concurrency of Access	Example: · Hashtable or Hashmap based on correctness needs? · Hashmap or ConcurrentHashMap based on how concurrent the access needs to be?	Same concept how concurrent is the access across the cluster.
	Data-structure implementation	Myriad other considerations based on how the data-structure is implemented and the domain-model under consideration. See here for more discussion	Identical to a POJA

Synchronization (given a requirement of a certain level of correctness)	Contention on monitor that you are synchronizing on	Be careful to lock on the right monitor. May need to stripe Lock.	Identical
	Scope of lock	Coarse grained versus Fine grained (is it method level or can you just keep it local to a specific block of code).The more fine-grained the better.	Identical - just note that in most cases there maybe value to making the lock more coarse-grained to benefit from batching.
	Type of lock	Pessimistic Synchronization or lower level synchronization semantics (volatile or Atomic or ReentrantReadWriteLock etc.)	Volatiles are not supported as of 2.6. In addition one has to look at lock-types e.g. Synchronous-Write, Write, Read, Concurrent, None. You can get more details on distributed locking here .

Memory Management	Garbage Collector: Choice of Collector. Xms, Xmx, Ratio of Eden to Old etc.	CMS or Parallel or both. Appropriate sizing of Heap, Young, Old etc.	Identical. See here In addition one needs to consider DGC (Distributed Garbage Collection) on the Terracotta Server.
	Soft References	Soft Reference Policy.	Tune Virtual Memory Manager on L1, L2.
	Virtual Memory Usage.	Avoid thrash with virtual memory monitor via vmstat	Identical – some thrash is allowable as long as within Latency SLA– between L1<->L2<->L2Disk. Tune faulting, fault-flush, committing-Terracotta transactions

Traffic Patterns and Distribution. Locality of Reference		Based on data size - Partition Data across JVMs or not.Locality of reference is a good thing.	Identical (one can over come data-size limitation assuming trade-off on latency SLA is possible). Delta: In several use-cases (but not all) Terracotta VMM can mean that you do not need to partition (hence simpler app). In some cases, given the data-volume and performance SLA, there may be no way to achieve Locality of Reference without partitioning. See here.

Other Considerations			Instrumentation Scope. See here

Testing and Deployment: See here for several thoughts around how testing a distributed application is different from a non-distributed application.

Consideration		Any Plain Old Java Application (POJA)	Any Terracotta integrated Java application (TI-POJA)
Testing	Functional	Code-Coverage is important to determine thoroughness of testing.	Identical - to ensure that there are no possible Terracotta runtime exceptions (UnlockedSharedException and NonPortableExceptions) lurking around.
	Scale/Performance Testing	Need Test Script that mirrors production load. Production Monitoring should also be on. Need quantitative measures (average, median, max) of latencies for a basket of transactions and overall TPS measure.	Testing a distributed app is a different ball-game. Ideally a framework to spin-up/tear-down JVMs is useful. You will t to consider throughput in the presence of various failover scenarios (e.g. when a client-JVM fails, when the Primary Terracotta-server fails etc.), state of distributed components (e.g. a client JVM is at 100% cpu utilization, network switch is in the middle of a re-boot sequence etc.), manifestation of the object graph across client JVMs , Terracotta Server JVMs and disk (given that Terracotta has a Virtual Memory Feature which allows you to exceed physical limits of RAM on your client JVM) etc. You probably also want to run your monitoring scripts at the same time. See here
	Availability	Need to identify Single Points of Failures in the application deployment	Terracotta will provide you an implementation with no single points of failure. Some effort needed towards tuning parameters that determine failover times /behaviors of various components based on the type-of failure. See here. The Network Active/Passive Link also talks about all the availability tests you could execute to reconfirm no SpoF in you specific environment.
Deployment
	Sizing of hosts on which JVM resides	Based on capacity planning exercises	Identical. Additionally need to size Terracotta servers and choose storage strategy for cluster state. You may need to determine how many client JVMs as well to support Application-TPS and based on latency SLAs (i.e. if faulting is not OK)
	Monitoring	CPU, Memory, Disk, Network, GC, TPS of each client-JVM etc.	Identical. In addition probably 10-20 more Terracotta specific parameters to monitor and of course the Terracotta Servers/Disk they write to.
	Run-book	Protocol for Recovery and Troubleshooting in case of situations.	Identical. Given that the architecture will not have single points of failure, the run-book would probably feature more about what to do to restore HA to the environment.

As is evident from the above arguments, there is a fair bit of parallelism between tuning and deploying any Java Application and a Terracotta Integrated App...although distributing your app, necessarily requires a few changes around the entire software-manufacturing process. Hopefully based on the above list and your Dev-exercise, you can predict better the additional time-to-market (I would guesstimate typical ranges average around 3-12 weeks for mid-high complexity apps from start to finish and it will just get better with more TIMs, Visibility and specific Documentation). You can read more about specifics at http://www.terracotta.org/confluence/display/wiki/TechnicalFAQ . We are in the process of cleaning up this documentation with links to supporting product features etc.– so keep an eye out for new content at http://www.terracotta.org

Why is RAT a better mouse-trap?

2007-10-11T18:13:00.002-07:00

What is RAT and what is it better at?

RAT = Remotely Attached Tenured (generation). You might have also heard of it as NAM (Network Attached Memory/Heap) for your Java applications. Bear with me for a moment as we go through a broad argument as to why RAT makes sense.

Let’s assume you reference a Cache in your Java Application – as this cache fills up, your Java Heap is under pressure. So, what are your options to avoid an OOME (Out Of Memory Exception):

1. Implement an evictor ⇒

Reapers remove Cache-Entries based on some LRU/LFU algorithm.
Problem is that these Cache-Entries were expensive-to construct in the first place. I suppose you have no choice if reaping is being done for cache-freshness but then doing it for getting around HEAP constraints – that hurts…

2. Rely on Java’s SoftReference capabilities i.e.

Create Cache-Entries as SoftReferences and
Then specify a SoftRefLRUPolicyMSPerMB (where essentially the JVM nulls your "carefully-constructed, expensive" data-structures) e.g. if set to 100 and if 200 MB is available on heap then => all SoftReferences that were created over 20,000 (200x100) ms ago – i.e. 20s ago, become eligible for collection...Again, hurts.

3. Rely on Java Serialization and overflow to disk.

Overflow leastRecentlyUsed/leastFrequentlyUsed Cache-Entries to disk based on some upper-watermark in terms of Heap utilization and pull them off disk, if the reference is not live on local Heap.
Cache-Entries and the object-graph associated with them need to implement java.io.Serializable and you need to marshall /unmarshall your object graph (at the very least Cache-Entry would feature 2 methods to serialize, deserialize itself).
So dosen't hurt as bad, although we had to “pollute” application code (with several classes having to implement java.io.Serializable and having to invest in “Marshalling /UnMarshalling from Domain Object Model <-> Disk Persistable Representation" code) and Java serialization isn’t cheap and local disk can’t be shared with other JVMs on other boxes.

So now what if, instead of all these gyrations around losing "carefully-constructed, expensive" Cache-Entries, or polluting your code - your object references (Cache-Entries and the graph attached to them) were able to just automatically become “Semi SoftReferences” when you encounter HEAP pressures – i.e.

LeastRecently/LeastFrequently used References just get nulled in the local JVM (so they become eligible for GC) and thus prevent the local JVM from throwing an OOME (similar to how Soft Reference would have worked).
But, in fact you don’t permanently lose the “carefully-constructed, expensive” Cache-Entry – since unbeknownst to the application-developer, these object references were also being maintained on a remote JVM – and when the local JVM does need such a “nulled” reference, it gets transparently pulled in from the remote JVM.

Sounds neat – dosen’t it? Your local JVM does not OOME and you did not have to invest any effort in your application to get this behavior of not losing the Cache-Entry from within your Java Domain Model.

That is precisely what is implemented as Terracotta’s virtual heap feature. The local JVM is your application JVM and the remote JVM is the “Terracotta Server” JVM and you get this behavior with just configuration. See:
http://www.terracotta.org/confluence/display/docs1/Concept+and+Architecture+Guide#ConceptandArchitectureGuide-VirtualHeap
and
http://www.terracotta.org/confluence/display/orgsite/Virtual+Heap+for+Large+Datasets
and
http://www.terracotta.org/confluence/display/orgsite/How+Terracotta+Works

In fact the remote JVM (the Terracotta server) can additionally persist the object references it maintains to disk. You thus get sections of Tenured (Old) Generation to become

Virtually unbounded (limited by only the size of disk you can attach to the remote JVM).
To become durable.
i.e. your Heap begins to look like what is attached in the picture (please ignore, Survivor, Perm etc. for the sake of this argument).

Now, Assuming ON AVERAGE:

World population for the past 50 years has averaged 5 billion at any one point in time &&
That 5% of the population on average have been programmers &&
That each programmer worked 300*8 = 2400 hours/year (I kid - the real no is probably more like 4800) &&
That 20% of all that time was devoted to managing memory and dealing with resultant issues:

Back in Mainframe days, it was about preserving every single byte you consumed;
In C/C++ days, it was about debugging “malloc”s without “free”s and pointer arithmetic gone haywire and
In these Java days, it is about tuning GC and avoiding OOMEs.

Therefore, time spent by humankind on Memory related issues = 5 * 50 * .05 * 2400 * .2 = 6 Trillion Man-hours – clearly not a trivial amount of time ☺. Now that you can use Terracotta Virtual heap and save yourself all the bother of writing all kinds of mechanisms to avoid OOMEs due to Heap constraints, hopefully we can make a dent on human-hours spent on dealing with this - 20% coming down to 19% itself is a saving today of 1 billion human-hours/year ;-) .

Of course, persist to a database a certain class of data (business critical data, data you need to query/report on) – databases are unmatched in terms of recovery, backup, check-pointing features and accessibility via SQL.

But, now that you have this sliver of Tenured that can expand and expand (at the rate and cost of disk) and still stay persistent across JVM lifecycles, without you having modify Java code – you have this new weapon in your armory. Check it out at http://www.terracotta.org

(Will post more details on how to control virtual memory and a little bit about its inner workings on the next blog)....

Clustered Java - Operational Needs

2007-07-11T16:31:00.000-07:00

Businesses are in constant flux and mutate in response to environmental stimuli - the Java application you deployed in the past has to change - new functionality, capacity, availability, monitorability requirements get tagged on.

Typically, the operations management and team (sysadmins, network engineers, DBAs etc.) who get measured on uptime/scale and the overall smooth functioning of the data-center. But ofcourse, they rely very heavily on the application/ application-infrastructure developer to help them get there. What is the expected application behavior:
1. When it gets deployed off 1 node to multiple nodes.
2. When 1 of the JVMs crashes.
3. When state gets out of synch across these nodes.
4. Under load etc.

Any stream-lining here will simplify the Development <-> Operations handoff i.e. fewer calls to you, the developer and lower maintenance costs to the organization as a whole. So, then:

Where does the application developer’s job end and the software-infrastructure engineer’s job begin?
Where does the software-infrastructure engineer’s job end and the operational-infrastructure engineer’s job begin ?

Now whether you are in an IT culture where

Roles are clearly demarcated across silos – as in the attached figure OR
You are in a culture/organization, where the above described roles don’t have clean boundaries…

there is no denying that while there is an argument for “silo-ization” – concerns around application functionality, latency, scalability, availability and manageability cut across these boundaries. The characteristics of a successful, highly efficient IT organization arguably are:

Core-competencies within each silo.
Clearly defined artifacts and interfaces across each boundary resulting in minimal friction /iterative loops across each boundary AND
A good understanding the up-stream/down-stream consequence of decisions within each silo.

As an example, lets focus on a big, hard-to-tame beast of Availability and Scalability for Java Applications. Of course, as an application-developer you are bound by what requirements flow-downstream - but assuming reasonable business requirements/project- management ;-), you would impact the final infrastructure/management foot-print of your application, based on decisions you execute around:

Task / Decision	Downstream Impact
Modelling of real-world entities as Database-schema and as a OO hierarchy in the Java-tier. And implementation of these entitites in the Database and at the Java tier (Data-structures employed, package/class organization, "garbage" created)	Latency, Scale
Marshalling/Unmarshalling across the OO-DB representations (ORM).	Latency, Scale
Computational algorithms, control-flow	Latency, Scale
Dependency management.	Maintainability, Manageability
Management of state across JVMs and JVMs and DB/external systems.	Availability, Scale

Wrong choices anywhere (e.g. an inappropriate data-structure, needless sql within a tight-loop, massive amounts of new objects instantiated/request etc.) can adversely impact “down-stream” concerns such as latency/scale/manageability. In today’s environment, the first-4 bullet points on the list above are clearly better-contained within the application developer’s world and classic application trouble-shooting can get rid of reported problems.
Arguably however, the last bullet point on the list above around state-management across JVMs and external systems is the fuzziest – and routinely crosses the Development / Operations chasm. The consequence of addressing this “downstream-concern” much later is that the current state-of-the-art is intrusive/labor-intensive, expensive, inconsistent and features frequent round-trips across the Dev/Ops interface – leading to inefficiencies and high support-overhead costs. i.e. one is better off taking a long-term view during the SDLC as against when maintaining the application (i.e. during SMLC – Software Maintenance LifeCycle).

Focussing further on an application on a single-JVM now running on multiple JVMs, the interfaces between Ops and Dev (and at times between application-infrastructure-development and application-software development) are weakly defined - resulting in frequent traversals across the chasm and inefficiencies. So, what are some of these operational concerns/needs when running a Java application on a cluster of JVMs.

Load Balancing strategy:

How to ensure that the load gets appropriately partitioned across the JVMs in the cluster?
Does the application need Locality of Reference - i.e. do requests need to be stickily routed to the same JVM?
How to re-route and what is user-experience when a JVM goes down?

Application Correctness: not synching state (e.g. caches) or any optimistic way of sending updates over implies

Reacting to potentially incoherent application/state (cache) across the cluster.
Non-guaranteed delivery of messaging across JVMs can easily lead to the brain being split across the participants in the cluster. (e.g. if the cache reflected price on an item in an electronic catalog and is inconsistent on 2 different JVMs, which JVM then has the right cache value - how does operations recover, do they need to restart the cluster ?)

Capacity Planning:

Operations needs a deterministic way to scale. Today there is a lack of a deterministic way to scale (e.g. add-a-node on demand).
Few low clustering overhead solutions are available in the market (or if home-grown) - since the majority use expensive Java-serialization. i.e. The requirement of more scale per operational-dollar in a clustered environment is still largely unmet by today's solutions.
Dynamic Infrastructure Provisioning based on usage characteristics.

Breach of Availability SLA (Unplanned downtime - i.e the fewer pages go off, the better):

Due to an Application Server Failure (Industry Java-app availability ~ 98%)
Cascading Failure (1 JVM goes down and the others go down as well).
Unpalatable Recoverability (long restart, application/cache warming, infrastructure abuse)
Cluster work-load balance/re-balance in response to outage/recovery.

Breach of Availability SLA (Planned downtime) - i.e. can i continue to maintain business continuity:

Application Release Processes: Rolling upgrades/Incompatible patches (e.g. apps that depend on Java Serialization)
Content Management (database/schema-related downtime).

Manageability / Monitorability

Few/No Intrusions into Release/Patch process
No comprehensive cluster visibility - Cluster management today is per node.
Proliferation of Tools/Mgmt Interfaces and complexity for the NOC.
Running Stateless with Stateful applications (arbitrary servers out-of/in-rotation).

Dev <-> Ops HandOff

High Overhead/Friction in Dev<->Ops Hand-Off
Operational Training costs
NOC and Production-Support Complexity/Errors

Others : Peculiar to a given IT/Organization’s culture.

Terracotta (the company behind the open-source clustering technology - DSO/Distributed Shared Objects) attempts (successfully, imho) to address all of the above operational concerns (with the exception of Load Balancing, Dynamic Infrastructure Provisioning 3-b) via a non-intrusive (simpilicity for Dev by clustering at the JVM), cost-effective (open-source) and consistent mechanism to cluster state across JVMs:

The application architect identifies what is stateful and codifies the clustering behavior in a XML file. Application code is cleanly separated from the clustering concern.
Operations gets a standard terminology and a standard mechanism (i.e a clean interface from Dev) to cluster Java applications, efficiently (given the implementation) and cost-effectively (highly scalable and open-source).

You can read more about it on the numerous blogs and the website at http://www.terracotta.org.

Clustering GlassBox

2006-12-29T22:53:00.000-08:00

Anyone who has maintained an application, would agree that measuring and managing an application's runtime characteristics and diagnosing performance, scale issues is part art, part science. A fair bit of data needs to be collected but then aggregated at the appropriate level so as to be useful and actionable, especially when you are under the gun to solve a production issue.

GlassBox is a cool open-source tool that promises exactly that for your Java applications in that it can give unprecedented visibility and adds on a diagnostic layer above this to summarize any potential issue succintly (see figure). Added bonus - it does this via AOP i.e. non-intrusive as far as your application is concerned.

However all of the measurement that GlassBox provides are for a given JVM - so coming back to our earlier point around aggregating measurements, wouldn't it be cool if there was some way (with minimal code changes) to collect these stats across the entire cluster of JVMs - so that way when a problem occurs, you can correlate data across all JVMs and on the cluster as a whole, so as to easily identify if a problem is cluster-wide or is isolated to a single JVM(s).

I sat with Ron Bodkin, CTO of Glassbox to see if we could cluster Glassbox with Terracotta. Here is what we had to do :

We installed GlassBox and then Terracotta for Spring (since glassBox uses Spring internally)

Modified the catalina.sh script and passed in 4 additional JAVA OPTIONS (so Terracotta could get hooked in). Container was Tomcat 5.5.x.

Ron then identified the Spring Bean that had all of the state (i.e. the per-JVM statistics).

We ran into a few configuration issues, which had to get sorted. Upgraded Spring to 1.2.8 (1.2.8 and upwards is supported by Terracotta). Even so, we ran into some exceptions and then realized that there was another old spring.jar in the classpath that needed to get removed.

Filled out the terracotta config file (tc-config.xml) which is where you state what bean needs to be clustered. Added wild-card characters to the application name and the resources entry, so Terracotta could find the Spring application context file, which captures the bean definitions. tc-config.xml entries look like

<jee-application name ="*glassbox*">
< paths><path>*beans.xml</path><paths>

Also the requirements were such that cluster-wide stats needed to be updated on a timed basis and not in real-time (i.e. as soon as a given statistic changed on any JVM in the cluster). So, Ron then wrote a method that essentially made an intelligent deep clone of the object graph that needed to be clustered - this method would fire off every few seconds. And we then clustered the bean that wrapped this deep clone instead. Config entries were:

<beans>
<bean name = "distributedRegistryHolder"></bean>
<beans>
<locks><autolock>
<method-expression> *glassbox.track.api.StatisticsRegistryHolderImpl.copy(..)</method-expression>
<lock-level>write</lock-level>
</autolock></locks>

Terracotta for Spring also has a feature where with a one-line config entry, you could cluster HttpSessions. So we turned that on. Something like adding a
<session-support>true</session-support>
element to the tc-config.xml. A GlassBox User can used a Web-based client to monitor your App - not a whole lot of state is in HttpSession, except for things such as GUI preferences - so those would now get preserved in case of loss of a given app-server. Not much but 1 less minor inconvenience in the world.

And it took us all of 4 hours to complete all of this. A satisfying 4 hours I might add. Now we get cluster wide monitoring by just integrating 2 open source products without modifying a line of application code. And you as the application architect can breathe a little easier, in that should things go south, you have a dashboard that helps you zero in on the issue instead of having to grep through logs across disparate machines and doing a ton of correlation work...