Big Data Applications and Challenges

AQA 4.11.2 — Big Data applications and challenges

Retrieval: the previous lesson established why Big Data is a genuinely different problem from a large ordinary database. This lesson evaluates what organisations actually do with it, and what it genuinely costs them — technically, ethically, and organisationally.

Key vocabulary

• Distributed processing — splitting a computation across many machines working in parallel, rather than one machine working alone.
• MapReduce — a distributed processing pattern with two phases: Map (each machine independently processes its own portion of the data, producing intermediate results) and Reduce (those intermediate results are combined into the final answer).
• Data governance — the policies and processes an organisation uses to control who can access data, how it's used, and how long it's kept.

Understand — MapReduce, conceptually

Counting how many times each product category appears across billions of order records on a single machine would take an enormous amount of time sequentially. MapReduce splits the records across many machines: each machine (the Map phase) independently counts categories within its own slice of the data, producing partial counts; a final step (the Reduce phase) combines every machine's partial counts into the true, overall totals. This is directly analogous to Sequence 13's divide-and-conquer idea (merge sort) — split the problem, solve the pieces independently, combine the results — but distributed across separate machines working genuinely in parallel, not just separate recursive calls on one machine.

Evaluate — genuine opportunities

• Personalisation — recommending products or content based on patterns across millions of users' behaviour, far beyond what a single organisation's own small customer base could reveal.
• Fraud detection — spotting an unusual transaction pattern only becomes possible by comparing it against a genuinely large history of normal behaviour.
• Medical research — identifying rare disease correlations requires data from far more patients than any single hospital sees.

Evaluate — genuine ethical issues, not just "privacy is a concern"

• Consent — data is often collected as a side effect of using a service (browsing history, location data), not through a genuine, informed opt-in decision.
• Bias — a model trained on historical data can encode and amplify existing unfairness (e.g. a hiring-recommendation system trained on a workforce that was historically unrepresentative reinforces that imbalance, not corrects it).
• Surveillance — aggregating enough data about an individual, even from sources that each seem harmless alone, can reconstruct a detailed profile of their behaviour without them ever having provided it directly for that purpose.
• Security — a Big Data system is a uniquely valuable target; a single breach can expose far more individuals' data at once than a breach of any one organisation's smaller database ever could.

Exam-style worked example

Question: A retailer wants to use customers' purchase history to predict what they're likely to buy next, and shares this predictive data with third-party advertisers without explicitly telling customers this specific use. Evaluate the ethical issues this raises. (4 marks)

Model answer: This raises a genuine consent issue: customers agreed to their data being used to run the retailer's own service, but sharing derived predictions with a third party for advertising is a different, unstated use they were never asked to explicitly agree to. It also raises a transparency issue — customers cannot meaningfully object to, or correct, a use of their data they don't know is happening. Even if no individual piece of data shared is itself highly sensitive, the aggregation of purchase patterns can reveal genuinely private information (health conditions inferred from purchases, for example) that customers would reasonably expect to remain private.

Evaluate — practical organisational implications

Adopting Big Data approaches is not purely a technical decision: it requires specialist skills (staff who can build and maintain distributed systems) that are expensive and in high demand; genuine infrastructure cost (storage and processing at scale is never free, even using cloud providers); and data governance — clear policies about who can access what data, for how long, and under what justification, which becomes significantly harder to enforce consistently once data is distributed across many systems and teams.

Common mistake

Treating "Big Data" as an unambiguous positive for any organisation that can afford it. As the evaluation above shows, genuine costs (skills, infrastructure, governance) and genuine ethical risks (bias, consent, security) are real trade-offs against the opportunities — not a footnote to an otherwise straightforwardly good decision.

Check your understanding

A city council proposes using Big Data analysis of public transport smart-card usage to improve bus routes, but the same data could also reveal individual residents' detailed daily movement patterns. Evaluate whether this proposal should proceed, and justify what safeguard(s) would make it more acceptable. (4 marks)

(The core proposal (improving bus routes from aggregate usage patterns) is a genuine, defensible opportunity - but using data that can reveal an INDIVIDUAL's detailed movements raises a real privacy risk beyond what's needed for the stated purpose. A genuine safeguard would be aggregating or anonymising the data specifically for the route-planning use case (e.g., only ever working with counts of journeys between stops, never data traceable back to one card/person), and a clear data governance policy restricting who can access the underlying individual-level data and why - proceeding without such safeguards would use data disproportionate to the actual stated purpose.)

Challenge

A hospital trust wants to use Big Data techniques to predict patient readmission risk from historical records. Evaluate one genuine opportunity and one genuine ethical risk this creates, and justify one safeguard that would help address the risk without abandoning the opportunity.

Looking ahead: the final lesson of this sequence brings SQL and Big Data together in one realistic, synoptic organisational scenario.