Master Class : Run your Big data and analytics workloads at Azure Cloud
NetCom Learning
When: November 22, 2022 at 10am
I learned that data is stored in a database that is optimized for online transactional processing (OLTP) operations that support applications. Data is then stored using “ACID” based transactions:
Atomicity – each transaction is treated as a single unit of work, which either succeeds or fails completely
Consistency – transactions can only take the data in the database from one valid state to another
Isolation – concurrent transactions cannot interfere with one another
Durability – when a transaction has succeeded, the data changes are persistent in the database
There are two different data processing types: batch processing and stream processing. Batch processing deals with processing high-volume data in batches or groups simultaneously or continuously. An example of this would be at the end of a business cycle for end of month payrolls. This type of processing makes calculations on a large dataset which is then reduced to a smaller set in the results. On the other hand, stream processing is the process of analyzing and managing data in real-time. It evaluates events as they happen and unlike batch jobs, this processing does not need to wait until all the data has been collected before starting the analysis/getting any results.
This relates to my career goals since a career in IT may entail the need to sort through lots of data. Batch and Stream processing could be very useful for different sets of data that you receive. It is important to know which specific process to use in order to gain the most successful and accurate report.