The AI-Powered Evolution of Big Data
Source abstract
Abstract
The rapid advancement of artificial intelligence (AI), coupled with the global rollout of 4G and 5G networks, has fundamentally transformed the Big Data landscape, redefining data management and analysis methodologies. The ability to manage and analyze such vast and varied datasets has exceeded the capacity of any individual or organization. This study introduces an enhanced framework that expands upon the traditional four Vs of Big Data—volume, velocity, volatility, and veracity—by incorporating six additional dimensions: value, validity, visualization, variability, volatility, and vulnerability. This comprehensive framework offers a novel and straightforward approach to understanding and addressing the complexities of Big Data in the AI era. This article further explores the use of ‘Big D’, an AI-driven, RAG-based Big Data analytical bot powered by the ChatGPT-4o model (ChatGPT version 4.0). This article’s innovation represents a significant advance in the field, accelerating and deepening the extraction and analysis of insights from large-scale datasets. This will enable us to develop a more nuanced and comprehensive understanding of intricate data landscapes. In addition, we proposed a framework and analytical tools that contribute to the evolution of Big Data analytics, particularly in the context of AI-driven processes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Big Data and Business Intelligence · Blockchain Technology Applications and Security · Big Data Technologies and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Technology, Kean University, Union, NJ 07083, USA
Department of Computer Science, Cihan University Sulaimaniya, Sulaymaniyah City 46001, Iraq
Department of Computer Science, Wenzhou-Kean University, Wenzhou 325060, China
Funding
Internal Research Support Program
References (74)
Show source-provided references
1. 3D data management: Controlling data volume, velocity and variety
2. Computers: From Eniac to Univac: As inventors, Eckert and Mauchly were clear successes, but as entrepreneurs they had some difficult times10.1109/MSPEC.1981.6369705 ↗
3. Fragmented future
4. O’Reilly, T. (2024, November 03). What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software. Available online: https://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html.
5. Users of the world, unite! The challenges and opportunities of social media10.1016/j.bushor.2009.09.003 ↗
6. A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBench10.1186/s40537-020-00388-5 ↗
7. Diebold, F.X. (2024, November 03). On the Origin(s) and Development of the Term ’Big Data’. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2152421.
8. That ‘Internet of Things’ Thing
9. Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv.
10. Sutskever, I., Martens, J., and Hinton, G.E. (July, January 28). Generating text with recurrent neural networks. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WT, USA.
11. Applications and challenges of AI-based algorithms in the COVID-19 pandemic10.1136/bmjinnov-2020-000648 ↗
12. David, E. (2024, October 12). Walmart Bets on Multiple AI Models with New Wallaby LLM. Available online: https://venturebeat.com/ai/walmart-bets-on-multiple-ai-models-with-new-wallaby-llm/.
13. Sifted Team (2024, October 12). How Amazon Is Using AI To Become the Fastest Supply Chain in the World. Available online: https://sifted.com/resources/how-amazon-is-using-ai-to-become-the-fastest-supply-chain-in-the-world/.
14. Cleveland Clinic (2024, October 12). How AI Is Being Used to Benefit Your Healthcare. Available online: https://health.clevelandclinic.org/ai-in-healthcare.
15. Daley, S. (2024, October 12). AI in Healthcare: Uses, Examples and Benefits. Available online: https://builtin.com/artificial-intelligence/artificial-intelligence-healthcare.
16. Mayo Clinic Press Editors (2024, October 12). AI in Healthcare: The Future of Patient Care and Health Management. Available online: https://mcpress.mayoclinic.org/healthy-aging/ai-in-healthcare-the-future-of-patient-care-and-health-management/.
17. Sergiienko, B. (2024, October 12). Why Generative AI in Banking Is a Secret Weapon: Your Blueprint for Implementation. Available online: https://masterofcode.com/blog/generative-ai-in-banking.
18. Reynolds, K. (2024, October 12). COVID-19 Increased the Use of AI. Here’s Why It’s Here to Stay. Available online: https://www.weforum.org/agenda/2021/02/covid-19-increased-use-of-ai-here-s-why-its-here-to-stay/.
19. Appen (2024, October 12). The 2020 State of AI and Machine Learning Report. Available online: https://www.appen.com/whitepapers/the-state-of-ai-and-machine-learning-report.
20. Batra, R. (2018). Database Management Systems and Tools. SQL Primer, Apress.10.1007/978-1-4842-3576-8 ↗
21. Federated learning: A comprehensive review of recent advances and applications10.1007/s11042-023-17737-0 ↗
22. Artificial Intelligence and Blockchain Integration in Business: Trends from a Bibliometric-Content Analysis
23. Archives of quantum computing: Research progress and challenges10.1007/s11831-023-09973-2 ↗
24. A brief survey on big data: Technologies, terminologies and data-intensive applications10.1186/s40537-022-00659-3 ↗
25. A study of big data evolution and research challenges10.1177/0165551518789880 ↗
26. Big data: Dimensions, evolution, impacts, and challenges10.1016/j.bushor.2017.01.004 ↗
27. A survey on federated learning: Challenges and applications10.1007/s13042-022-01647-y ↗
28. Preskill, J. (2023). Quantum computing 40 years later. Feynman Lectures on Computation, CRC Press.10.1201/9781003358817-7 ↗
29. Rayhan, A., and Shahana, R. (2023). Quantum Computing and AI: A Quantum Leap in Intelligence. AI Odyssey: Unraveling the Past, Mastering the Present, and Charting the Future of Artificial Intelligence, NotunKhabar. Available online: https://www.amazon.com/Odyssey-Unraveling-Mastering-Artificial-Intelligence/dp/B0CCXLCGDM.
30. Impact of Big Data Analytics on Digital Marketing: Academic Review10.52783/jes.2327 ↗
31. Sargiotis, D. (2024, October 12). Integrating AI and Big Data in Virtual Infrastructures: Transforming Educational Landscapes for the Future. Available online: https://discovery.researcher.life/article/integrating-ai-and-big-data-in-virtual-infrastructures-transforming-educational-landscapes-for-the-future/980542e1f19838588443702eac62087f.10.2139/ssrn.4789850 ↗
32. 15 years of Big Data: A systematic literature review10.1186/s40537-024-00914-9 ↗
33. Advances in large language models: ChatGPT expands the horizons of neuroscience10.3934/steme.2023016 ↗
34. Cui, Y., Yang, Z., and Yao, X. (2023). Efficient and effective text encoding for chinese llama and alpaca. arXiv.
35. Multimodal artificial intelligence foundation models: Unleashing the power of remote sensing big data in earth observation
36. ChatGPT and Big Data: Enhancing Text-to-Speech Conversion10.58496/MJBD/2023/005 ↗
37. Yeasir Fahim, J. (2024, October 12). Mastering the Art of AI Language: An In-Depth Exploration of Prompting Techniques and Their Influence on Model Performance. Available online: https://digital.kenyon.edu/cgi/viewcontent.cgi?article=1031&context=dh_iphs_ss.
38. Rashid, A., Baloch, N., Rasheed, R., and Ngah, A.H. (J. Sci. Technol. Policy Manag., 2024). Big data analytics-artificial intelligence and sustainable performance through green supply chain practices in manufacturing firms of a developing country, J. Sci. Technol. Policy Manag., ahead of print.10.1108/JSTPM-04-2023-0050 ↗
39. Big data and performance measurement research: Trends, evolution and future opportunities10.1108/MBE-06-2019-0053 ↗
40. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019). ERNIE: Enhanced language representation with informative entities. arXiv.10.18653/v1/P19-1139 ↗
41. Artificial intelligence AI-based Chatbot study of ChatGPT, Google AI Bard and Baidu AI10.30574/wjaets.2023.8.1.0045 ↗
42. Sundu, M., Yasar, O., and Findikli, M.A. (2022). Data-driven innovation: Digital tools, artificial intelligence, and big data. Organizational Innovation in the Digital Age, Springer International Publishing.10.1007/978-3-030-98183-9_6 ↗
43. Bormida, M.D. (2021). The big data world: Benefits, threats and ethical challenges. Ethical Issues in Covert, Security and Surveillance Research, Emerald Publishing Limited.10.1108/S2398-601820210000008007 ↗
44. Raubenheimer, J. (2021). Big data in academic research: Challenges, pitfalls, and opportunities. Big Data in Education: Pedagogy and Research, Springer.10.1007/978-3-030-76841-6_1 ↗
45. The evolution of data science and big data research: A bibliometric analysis10.1007/s11192-020-03371-2 ↗
46. A General framework for studying the evolution of the digital innovation ecosystem: The case of big data10.1016/j.ijinfomgt.2018.10.023 ↗
47. An integration-oriented ontology to govern evolution in big data ecosystems10.1016/j.is.2018.01.006 ↗
48. Bonner, S., Kureshi, I., Brennan, J., and Theodoropoulos, G. (2017). Exploring the evolution of big data technologies. Software Architecture for Big Data and the Cloud, Morgan Kaufmann.10.1016/B978-0-12-805467-3.00014-4 ↗
49. Visualizing the knowledge structure and evolution of big data research in healthcare informatics10.1016/j.ijmedinf.2016.11.006 ↗
50. Salminen, V., Ruohomaa, H., and Kantola, J. (2017). Digitalization and big data supporting responsible business co-evolution. Advances in Human Factors, Business Management, Training and Education, Proceedings of the AHFE 2016 International Conference on Human Factors, Business Management and Society, Walt Disney World, FL, USA, 27–31 July 2016, Springer International Publishing.10.1007/978-3-319-42070-7_96 ↗
51. The evolution of big data as a research and scientific topic: Overview of the literature
52. Camacho, J., Macia-Fernandez, G., Diaz-Verdejo, J., and Garcia-Teodoro, P. (May, January 27). Tackling the big data 4 vs for anomaly detection. Proceedings of the 2014 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Toronto, ON, Canada.10.1109/INFCOMW.2014.6849282 ↗
53. A brief introduction on Big Data 5Vs characteristics and Hadoop technology10.1016/j.procs.2015.04.188 ↗
54. Generative AI service implementation using LLM application architecture: Based on RAG model and LangChain framework
55. OpenAI (2024, August 09). GPT-4o System Card. Available online: https://cdn.openai.com/gpt-4o-system-card.pdf.
56. How leading organizations use big data and analytics to innovate10.1108/SL-06-2015-0054 ↗
57. Alaskar, T.H., Alsadi, A.K., Aloulou, W.J., and Ayadi, F.M. (2024). Big Data Analytics, Strategic Capabilities, and Innovation Performance: Mediation Approach of Organizational Ambidexterity. Sustainability, 16.10.3390/su16125111 ↗
58. Democratizing artificial intelligence: How no-code AI can leverage machine learning operations10.1016/j.bushor.2023.04.003 ↗
59. Widad, E., Alaoui, I.E., and Gahi, Y. (2022). Data quality in the era of big data: A global review. Big Data Intelligence for Smart Applications, Springer.
60. Integrated omics: Tools, advances and future approaches10.1530/JME-18-0055 ↗
61. Unstructured data in marketing10.1007/s11747-018-0581-x ↗
62. Intersoft Consulting (2024, August 10). General Data Protection Regulation (GDPR). Available online: https://gdpr-info.eu/.
63. (2024, August 10). California Consumer Privacy Act (CCPA), Available online: https://www.oag.ca.gov/privacy/ccpa.
64. IBM (2024, August 10). IBM Watson Health Introduces New Opportunities for Imaging AI Adoption. Available online: https://newsroom.ibm.com/2021-11-30-IBM-Watson-Health-Introduces-New-Opportunities-for-Imaging-AI-Adoption.
65. Microsoft Research (2024, August 10). AI For Good Lab. Available online: https://www.microsoft.com/en-us/research/group/ai-for-good-research-lab/.
66. (2024, August 10). BigQuery Overview. Available online: https://cloud.google.com/bigquery/docs/introduction.
67. (2024, August 10). Get Started with TensorFlow. Available online: https://www.tensorflow.org/.
68. (2024, August 10). IBM InfoSphere Information Server. Available online: https://www.ibm.com/information-server.
69. (2024, August 10). Apache NiFi Documentation. Available online: https://nifi.apache.org/documentation/.
70. (2024, August 10). Data Quality Solutions. Available online: https://www.talend.com/products/data-quality/.
71. Abdalla, H.B., Awlla, A.H., Kumar, Y., and Cheraghy, M. (2024, January 26–27). Big Data: Past, Present, and Future Insights. Proceedings of the 2024 Asia Pacific Conference on Computing Technologies, Communications and Networking, Chengdu, China.10.1145/3685767.3685777 ↗
72. NSF (2024, August 12). NSF 24-589: Computer and Information Science and Engineering: Core Programs, Available online: https://new.nsf.gov/funding/opportunities/computer-information-science-engineering-core-programs/nsf24-589/solicitation.
73. (2024, August 10). SearchGPT Prototype. Available online: https://openai.com/index/searchgpt-prototype/.
74. Deshmukh, A., Hegde, G., Lathi, R., and Govikarn, S. (February, January 30). A literature survey on latent semantic indexing. Proceedings of the International Conference on Computing 2012, Maui, HI, USA.