Presenting a predictive model for sentiment analysis of urban multi-modal data with deep learning Case study: Iranian formal megacities

Document Type : Original Article

Author

Associate Professor, Department of Urban Design, Faculty of Architecture and Urban Planning, Iran University of Art, Tehran, Iran

Abstract

Urban planning and design issues are highly complex, making it challenging to achieve a comprehensive understanding of these issues, especially on a large scale, using traditional urban planning methods alone. Therefore, the integration of data science and modeling presents a promising approach for comprehending urban issues. Extensive research and reports on real urban projects implemented across various countries suggest that machine learning and deep learning techniques have the potential to effectively address urban problems and offer additional benefits such as reducing economic costs associated with inefficiencies, improving operational speed, and enhancing predictability. The integration of machine learning and deep learning can be utilized in the evaluation, design, and post-implementation stages of urban planning and design processes. User experience, including environmental stress, happiness, and fear, significantly influences the character and activities within urban spaces. Hence, it is crucial for urban designers to create new spaces or revitalize existing ones in a way that evokes a positive emotional response and a sense of belonging among users. Numerous studies have explored this subject, primarily focusing on evaluating people's emotions within physical environments and occasionally within virtual settings, using AI-based methodologies. Sentiment analysis or opinion mining, previously employed before the rise of social media, is now being investigated and analyzed using computational methods with the aid of vast amounts of data published on social media platforms. This research domain, widely used in economic, marketing, political, and social research, has increasingly permeated the realm of urban studies. However, it is relatively new within the field of urban studies, with only a limited number of research studies conducted thus far. The main objective of this research is to develop a predictive model for understanding users' emotional responses towards urban spaces through an analysis of multi-modal data, including images and textual content, published in virtual spaces. This study incorporates two notable innovations:
The use of a machine learning approach in sentiment analysis of multi-modal data on an urban scale, which has been explored in data science research but remains insufficiently explored in the realm of urban planning and management in Iran.
The creation of a dataset comprising textual and visual data relevant to urban environments.
It aims to present a predictive model that effectively analyzes sentiment in textual and visual media data derived from Twitter. The research employs a quantitative methodology and utilizes a supervised deep learning technique. The case study encompasses multi-modal urban data from eight megacities in Iran, collected over a ten-year period until December 2021. An algorithmic hybridization based on the Convolutional Neural Network (CNN) architecture has been developed and employed to train the textual data, resulting in an accuracy rate of 77%. Additionally, the visual data has been trained using three distinct algorithms, with VGG19 demonstrating superior performance and achieving an accuracy rate of 88%. A hybrid algorithm has been devised to train the multi-modal data, yielding an accuracy of 66%. Subsequently, a combination model leveraging pre-trained VGG19 models alongside the developed text combination model has been implemented, achieving an elevated accuracy of 81%. These outcomes demonstrate the machine's capacity to predict sentiment polarity in textual-visual data related to cities.

Keywords

Main Subjects

References

راد، رؤیا، موسوی، محمد، و وردی، فاطمه (1399). استفاده از یادگیری عمیق در تشخیص خودکار بیماری گیاهان براساس پردازش تصویر برگ. تحقیقات سامانه‌ها و مکانیزاسیون کشاورزی، 21(76)، ۴۹-۶۸.
راستگو، راضیه، و کیانی، کوروش (1398). شناسایی چهره با استفاده از تنظیم دقیق شبکه‌های کانولوشنی عمیق و رویکرد یادگیری انتقالی. مجله مدل‌سازی در مهندسی، 17 (58)، ۱۰۳-۱۱۱.
زاهدی حقیقی، سیده سعیده، سخایی، سید محمود، و دلیری، محمدرضا (1398). تشخیص حالت‌های احساسی مبتنی‌بر EEG با استفاده از شبکه‌ یادگیری عمیق. مجله‌ مهندسی پزشکی زیستی، 13 (2)، ۹۵-۱۰۴.
فولادی، صابر، فرسی، حسن، و محمدزاده، سجاد (1399). تشخیص و طبقه‌بندی سرطان پوست با استفاده از یادگیری عمیق. مجله علمی دانشگاه علوم پزشکی بیرجند، 26 (1)، ۴۴-۵۳.
موسوی، سیدمهدی، عبادی، حمید، و کیانی، عباس (1398). ارائه‌ روشی بهینه مبتنی‌بر یادگیری عمیق به‌منظور طبقه‌بندی طیفی مکانی تصاویر با قدرت تفکیک مکانی بالا در مناطق نیمه‌شهری. نشریه علوم و فنون نقشه‌برداری، 9 (2)، 170-151.
وب‌سایت خانه‌ بیگ دیتای ایران (1397). پردازش متن با Jhazm  نسخه جاوا کتابخانه هضم برای پردازش زبان فارسی.  بازیابی شده در 28 بهمن 1401، از https://bigdata-ir.com
یوسفی متقاعد، مریم، و صبوحی، هادی (1398). مروری بر تحلیل احساسات شبکه‌های اجتماعی درحوزه‌ قطبیت. کنفرانس بین‌المللی وب‌پژوهی، 3 و 5 اردیبهشت ماه، تهران، ایران.
 Baly, R., Hobeica, R., Hajj, H., El-Hajj, W., Shaban, K.B., & Al-Sallab, A. (2016). A Meta-Framework for Modeling the Human Reading Process in Sentiment Analysis. ACM Transactions on Information Systems (TOIS), 35(1), 1-21. https://doi.org/10.1145/2950050.
Benevenuto, F., Araújo, M., & Ribeiro, F. (2015). Sentiment Analysis Methods for Social Media. In Proceedings of Brazilian Symposium on Multimedia and the Web (WebMedia). ACM, Manaus, Brazil, 11–11. https: //doi.org/10.1145/2820426.2820642.
Bonasoli de Oliveira, W., Dorini, L.B., Minetto, R., & Silva, T.H. (2020). OutdooeSent: Setiment analysis of urban outdoor images by using semantic and deep features. ACM transaction on information systems, 1(1), 1-29.
Campos, V., Jou, B., & Giró i Nieto, X. (2017). From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction. Image and Vision Computing, 65, 15 -22. https://doi.org/10.1016/j.imavis.2017.01.011.
Chapman, L., Resch, B., Sadler, J., Zimmer, S., Roberts, H., & Petutsching, A. (2018). Investigating the emotional responses of individuals to urban green space using Twitter data: a critical comparison of tree different methods of sentiment analysis. Urban Planning, 3 (1), 21-33.
Chen, F., Gao, Y., Cao, D., & Ji, R. (2015). Multimodal hypergraph learning for microblog sentiment prediction. In IEEE International Conference on Multimedia and Expo (ICME). IEEE, Turin, Italy, 1–6. https://doi.org/10.1109/ICME.2015.7177477.
Chen, NC., Nagakura, T., & Larson, K. (2016). Social media as Complementary Tool to Evaluate Cities -Data Mining Innovation Districts in Boston, Herneoja, Aulikki; Toni Österlund & Piia Markkanen (eds.), Complexity & Simplicity - Proceedings of the 34th eCAADe Conference - Volume 2, University of Oulu, Oulu, Finland, 22-26 August 2016, 447-456.
Chen, T. (2014). Deep SentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks. CoRR abs/1410.8586, 1–7. arXiv:1410.8586. http://arxiv.org/abs/1410.8586.
Chen, T., Xu, R., He, Y., & Wang, X. (2017). Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Systems with Applications, 72, 221-230.
Dhall, A., Goecke, R., & Gedeon, T. (2015). Automatic Group Happiness Intensity Analysis. IEEE Transactions on Affective Computing, 6(1), 13–26. https://doi.org/10.1109/TAFFC.2015.2397456.
Eboli, L., & Mazzulla, G. (2011). A methodology for evaluating transit service quality based on subjective and objective measures from the passenger’s point of view. Transport Policy, 18(1), 172–181. http://dx.doi.org/10.1016/j.tranpol.2010.07.007.
EURISY. (2017). Good City Life: crowdsourcing satellite data and emotions to map our urban landscape. Retrieved 2022, Feb. 18, from https://goodcitylife.org/
Fathullah, A., & S.Willis, K. (2018). Engaging the Senses: The Potential of Emotional Data for Participation in Urban Planning. Urban Science, 2(4), 1-21. doi:10.3390/urbansci2040098.
He, W., Zha, S., & Li, L. (2013). Social media competitive analysis and text mining: A case study in the pizza industry. International Journal of Information Management, 33, 464–472.
Hollander, J. B., & Renski, H. (2015). Measuring urban attitudes using Twitter: an exploratory study. Lincoln Institute of Land Policy.
Huang, H., Gartner, G., Klettner, S., & Schmidt, M. (2014). Considering affective responses towards environments for enhancing location based services. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 93-96. doi:10.5194/isprsarchives-XL-4-93-2014.
Hutto, C.J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model foe sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216-225. https://doi.org/10.1609/icwsm.v8i1.14550.
Ji, R., Cao, D., & Lin, D. (2015). Cross-Modality Sentiment Analysis for Social Multimedia. IEEE International Conference on Multimedia Big Data.
Kenyon, S., & Lyons, G. (2003). The value of integrated multimodal traveler information and its potential contribution to modal change. Transport Research Part F: Traffic Psychology Behavior, 6(1), 1–21. http://dx.doi.org/10.1016/s1369-8478(02)00035-9.
Kim, E., Rosenwasser, D., & Castillo Lopez, J.L.G. (2020). Urban emotion: The interrogation of social media and its implication within urban context. Proceedings of the 38th eCAADe Conference - Volume 2, TU Berlin, Berlin, Germany, 16-18 September, 475-482.
Liu, B., & Zhang, L. (2012). A survey of opinion mining and sentiment analysis. Mining Text Data, 415-463.
Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis lectures on human language technologies. 1-167.
Liu, L., Silva, E.A., Wu, C., & Wang, H. (2017). A machine learning-based method for the large-scale evaluation of qualities of the urban environment. Computers, environment and urban systems, 65, 113-125.
Mazzulla, G., & Forciniti, C. (2012). Spatial association techniques for analyzing trip distribution in an urban area. European Transport Research Review, 4(4), 217–233.http://dx.doi.org/10.1007/s12544-012-0082-9.
Oteros-Rozas, E., Martin-Lpez, B., Fagerholm, N., Bieling, C., & Plieninger, T. (2018). Using social media photos to explore the relation between cultural ecosystem services and landscape features across five European sites. Ecol. Indic. 94, 74–86.
Parrett, G. (2016). 3.5 million photos shared every minute in 2016. Deloitte. https://goo.gl/uwF81P.
Poria, S., Peng, H., Hussain, A., Howard, N., & Cambria, E. (2017). Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing, 261, 217-230.
Resch, B., Summa, A., Sagl, G., Zeile, P., & Exner, J. (2014). Urban emotions-Geo-semantic emotion extraction from technical sensors, human sensors and crowd sourced data. In G. Gartner & H. Huang (Eds.), Progress in location-based service, Springer.
Ribeiro, F.N., Araujo, M., Goncalves, P., Goncalves, M.A., & Benevenuto, F. (2016). SentiBench -A benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1), 1–29. https://doi.org/10.1140/epjds/s13688-016-0085-1.
Rossi, L., Boscaro, E., & Torsello, A. (2018). Venice through the Lens of Instagram: A Visual Narrative of Tourism in Venice, 18 Companion of the Web Conference, 1190-1197.
Schwartz, A.J., Dodds, P.S., O`Neil-Dunne, J.P.M., Danforth, C.M., & Ricketts, T. (2019). Visitors to urban greenspace have higher sentiment and lower negativity on Twitter. People and Nature, 476-485.
Sdoukopoulos, A., Nikolaidou, A., Pitsiava-Latinopoulou, M., & Papaioannou, P. (2018). Use of social media for assessing sustainable urban mobility indicators. International Journal of sustainable development and planning, 13(2), 338-348.
Setiawan, E.I., Juwiantho, H., Santoso, J., Sumpeno, S., Fujisawa, K., & Purnomo, M.H. (2021). Multiview sentiment analysis with Image-Text-Concept features of Indonesian social media posts, International Journal of Intelligent Engineering & Systems, 14(2), 521-535.
Silva, T.H., Vaz de Melo, P.O.S., Almedia, J.M., Salles, J., & Loureiro, A.A.F. (2013). A comparison of Foursquare and Instagram to the study of city dynamics and urban social behavior, Proc. ACM SIGKDD Int. Workshop on Urban Computing (UrbComp’13).
Sim, J., Miller, P., & Swarup, S. (2020). Tweeting the high line life: a social media lens on urban green spaces. Sustainability, 12, 1-18.
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S., & Pantic, M. (2017). A survey of multimodal sentiment analysis. Image and Vision Computing, 65, 3–14.
Tebyanian, N. (2020). Application of machine learning for urban landscape design: a primer for landscape architect. Journal of digital landscape architecture, 5, 217-226.
Tieskens, K.F., Zantenm B, T.V., Schulp, C.J.E., & Verburg, P.H. (2018). Aesthetic appreciation of the cultural landscape through social media: An analysis of revealed preference in the Dutch river landscape. Landscape and Urban Planning, 177, 128–137.
Tu, W., Cao, J., Yue, Y., Shaw, S., Zhou, M., Wang, Z., Chang, X., Xu, Y., & Li, Q. (2017). Coupling mobile phone and social media data: A new approach to understanding urban functions and diurnal patterns. International Journal of Geographical Information Science, 31, 2331–2358.
Wu, J., Lin, Z., & Zha, H. (2016). Multi-view common space learning for emotion recognition in the wild. In ACM International Conference on Multimodal Interaction. ACM, Tokyo, Japan, 464–471. https://doi.org/10.1145/2993148.2997631.
You, L., & Tuncer, B. (2016). Exploring public sentiments for livable places based on a crowd-calibrated sentiment analysis mechanism. IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM). August 18-21, USA: San Francisco.
Zhan, X., Wang, Y., Rao, Y., & Li, Q. (2019). Learning from Multi-annotator Data: A Noise-aware Classification Framework. ACM Transactions on Information Systems (TOIS), 37(2), 1 -28. https://doi.org/10.1145/3309543.
Zhou, G-Y & Huang, J. X. (2017). Modeling and Mining Domain Shared Knowledge for Sentiment Analysis. ACM Transactions on Information Systems (TOIS), 36(2), 1-36. https://doi.org/10.1145/3091995.
Journal of Architecture and Urban Planning
Volume 17, Issue 45
December 2024
Pages 79-103

Files

Share

How to cite

Statistics