Data-Intensive Applications
What data intensive applications need?
- Databases : Store data for future reference
- Caches : Remember the result of an expensive operation
- Search Indexes : Search data by keyword or filter
- Stream Processing : Send a message to another process
- Batch Processing : Periodically crunch a large amount of accumulated data
If you have an application-managed caching layer (using Memcached or similar), or a full-text search server (such as Elasticsearch or Solr) separate from your main database, it is normally the application code’s responsibility to keep those caches and indexes in sync with the main database.
Questions
- How do you ensure that the data remains correct and complete, even when things go wrong internally?
- How do you provide consistently good performance to clients, even when parts of your system are degraded?
- How do you scale to handle an increase in load?
- What does a good API for the service look like?
To answer the above questions we need to focus on three concerns that are important in most software systems.
- Reliability
- Scalability
- Maintainability