Member-only story
What is Big Data? How to deal with Big Data Set?
Working with Popular Python Libraries
Big data refers to the type of data that is characterized by 5 Vs (refer Knowledge Discovery in Big Data from Astronomy and Earth Observation):
- Volume —data generated in massive amount.
- Velocity — data generated in fast rate.
- Variety — data generated in diversified ways.
- Validity — data requires effort to determine their actual quality or trustworthiness.
- Value — data requires effort to determine their actual meaning in a prescribed context .
Big data comes from many kinds of sources (transaction processing systems, customer databases, documents, emails, medical records, internet clickstream logs, mobile apps and social networks) and they include many types of data formats such as machine-generated data (network and server log files and data from sensors on manufacturing machines), industrial equipment and internet of things devices (refer TechTarget- examples of big data).
Archive Org keeps a large collection of digital media found in the Internet comprising of web pages, audio, video etc. Kaggle provides over 50,000 public datasets and 400,000 public notebooks. Forbes lists 33 global data sources . AwesomeData provides 36 topic-centric data…
