Optimization of Web Crawler Based on Automated Machine Learning

Chenyu Liao

doi:10.47297/taposatWSP2633-456918.20250604

PDF
导出
分享
收藏
专辑

Optimization of Web Crawler Based on Automated Machine Learning

6卷第4期

浏览量: 0 Downloads: 0

Optimization of Web Crawler Based on Automated Machine Learning
Optimization of Web Crawler Based on Automated Machine Learning
South China Normal University, Guangzhou, Guangdong, 510440, China
2025年6卷第4期页码：109-113
纸质出版日期：2025-04-22
DOI：10.47297/taposatWSP2633-456918.20250604
Accepted：

Scan QR Code

(2025). Optimization of Web Crawler Based on Automated Machine Learning. 科技理论与实践（英文版）, 6(4), 109-113.

Liao Chenyu.Optimization of Web Crawler Based on Automated Machine Learning[J].Theory and Practice of Science and Technology,2025,06(04):109-113.
(2025). Optimization of Web Crawler Based on Automated Machine Learning. 科技理论与实践（英文版）, 6(4), 109-113. DOI： 10.47297/taposatWSP2633-456918.20250604.

Liao Chenyu.Optimization of Web Crawler Based on Automated Machine Learning[J].Theory and Practice of Science and Technology,2025,06(04):109-113. DOI： 10.47297/taposatWSP2633-456918.20250604.

摘要

Abstract

With the rapid growth of web data

traditional web crawlers face challenges such as low efficiency and poor accuracy in processing dynamic web pages and complex data structures. This paper proposes an AutoML-based optimization method for web crawlers

leveraging adaptive feature extraction

dynamic learning updates

and multi-modal data fusion to enhance performance and adaptability. The approach employs deep learning models to automatically extract text and image features

integrates reinforcement learning for intelligent classification

and utilizes online learning to adapt to real-time web changes. Experimental results demonstrate significant improvements in crawling efficiency and accuracy. The proposed method also supports cross-domain applications

offering an advanced solution for data collection in fields like information retrieval and data mining.

关键词

Keywords

references

Peng, T., Zuo, W., & Liu, Y. (2005). Characterization of Evaluation Metrics in Topical Web Crawling Based on Genetic Algorithm. In Advances in Natural Computation (pp. 123-134). Springer.

Akilandeswari, J., & Gopalan, N.P. (2007). A Novel Design of Hidden Web Crawler Using Reinforcement Learning Based Agents. In Advanced Parallel Processing Technologies (pp. 527-534). Springer.

Shrivastava, G.K., Pateriya, R.K., & Kaushik, P. (2023). An efficient focused crawler using LSTM-CNN based deep learning. International Journal of System Assurance Engineering and Management, 14, 391-407.

Zhang, Y., Zhang, Y., & Zhang, M. (2019). Deep learning for web data extraction: A survey. Journal of Web Engineering, 18(1-2), 1-30.

Barbosa, L., & Freire, J. (2007). An Adaptive Crawler for Locating Hidden-Web Entry Points. In Proceedings of the International World Wide Web Conference (pp. 441-450).

Chakrabarti, S., van den Berg, M., & Dom, B. (1999). Focused crawling: A New Approach to Topic-specific Web Resource Discovery. Computer Networks, 31(11-16), 1623-1640.

Pant, G., & Srinivasan, P. (2005). Learning to crawl comparing classification schemes. ACM Transactions on Information Systems, 23(4), 430-462.

Raghavan, S., & Garcia-Molina, H. (2001). Crawling the Hidden Web. In Proceedings of the 27th VLDB Conference (pp. 129-138).

Han, J., Pei, J., & Yin, Y. (2000). Mining Frequent Patterns without Candidate Generation. In SIGMOD Conference (pp. 1-12).

Menczer, F., Pant, G., & Srinivasan, P. (2004). Topical web crawlers: Evaluating adaptive algorithms. ACM Transactions on Internet Technologies, 4(4), 378-419.

文章被引用时，请邮件提醒。

Submit

No data

Related Author

No data

Related Institution

No data

⁰