Transforming Hand Drawn Wireframes into Front-End Code with Deep Learning
Source abstract
Abstract
The way towards generating a website front end involves a designer settling on an idea for what kind of layout they want the website to have, then proceeding to plan and implement each aspect one by one until they have converted what they initially laid out into its Html front end form, this process can take a considerable time, especially considering the first draft of the design is traditionally never the final one. This process can take up a large amount of resource real estate, and as we have laid out in this paper, by using a Model consisting of various Neural Networks trained on a custom dataset. It can be automated into assisting designers, allowing them to focus on the other more complicated parts of the system they are designing by quickly generating what would rather be straightforward busywork. Over the past 20 years, the boom in how much the internet is used and the sheer volume of pages on it demands a high level of work and time to create them. For the efficiency of the process, we proposed a multi-model-based architecture on image captioning, consisting of Convolutional neural network (CNN) and Long short-term memory (LSTM) models. Our proposed approach trained on our custom-made database can be automated into assisting designers, allowing them to focus on the other more complicated part of the system. We trained our model in several batches over a custom-made dataset consisting of over 6300 files and were finally able to achieve a Bilingual Evaluation Understudy (BLEU) score for a batch of 50 hand-drawn images at 87.86%
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (46)
Show source-provided references
1. A. Robinson, “Sketch2code: Generating a website from a paper mockup,” ArXiv, 2019. [Online]. Available: https://arxiv.org/abs/1905.13750.
2. Image processing, analysis, and machine vision
3. Research on image processing technology of computer vision algorithm
4. T. Beltramelli, “Pix2code: Generating code from a graphical user interface screenshot,” ArXiv, 2017. [Online]. Available: https://arxiv.org/abs/1705.07962.10.1145/3220134.3220135 ↗
5. K. N. Haque, M. A. Yousuf and R. Rana, “Image denoising and restoration with CNN-LSTM encoder decoder with direction attention,” ArXiv, 2018. [Online]. Available: https://arxiv.org/abs/1801.05141.
6. J. Ferreira, A. Restivo and H. S. Ferreira, “Automatic generation of synthetic website wireframe datasets from source code,” M.S. Dissertation, University in Porto, Portugal, 2020.
7. A. Garcia, “A review on deep learning techniques applied to semantic segmentation,” ArXiv, 2017. [Online]. Available: https://arxiv.org/abs/1704.06857.
8. Advanced deep-learning techniques for salient and category-specific object detection: A survey10.1109/MSP.2017.2749125 ↗
9. Computer vision approaches based on deep learning and neural networks: Deep neural networks for video analysis of human pose estimation
10. P. Mayhew and C. Worsley, “Software prototyping: The management implications”, P.h.D. dissertation, 2019.
11. S. Yacoob, “Paving the way for software prototyping,” [Online]. Available: https://www.justinmind.com/blog/ui-sketching/ seen June 2021.
12. Pros and cons of tangible and digital wireframes
13. “Refsnes Data, “Introduction to html,” [Online]. Available: https://www.w3schools.com/html/ seen June 2021.
14. “Winnie Nguyen, “Why is Low-fidelity wireframe important in product design?” Available: https://uxdesign.cc/why-low-fidelity-wireframe-curious-in-product-design-c7bea87bc23d seen June 2021.
15. J. Erikson, “Smart UX: High fidelity wireframes,” Available: https://usabilitygeek.com/smart-ux-high-fidelity-wireframes/ seen June 2021.
16. Y. Lecun, Y. Bengio and G. Hint, “Deep learning review,” Available: http://www.personal.psu.edu/lxx6/DeepLearning.pdf seen June 2021.
17. Refsnes Data, “Introduction to css,” Available: https://www.w3schools.com/css/ last seen June 2021.
18. Introduction to javascript,” Available: https://www.w3schools.com/js/ seen June 2021.
19. Misko Hevery, “What is angularjs?” Available: https://docs.angularjs.org/guide/introduction seen June 2021.
20. J. Alammar, “The illustratedword2vec,” [Online]. Available: https://jalammar.github.io/illustrated-word2vec/ seen July 2021.
21. A gentle introduction to calculating the bleu score for text in python
22. Deep learning vs. traditional computer vision”
23. Reverse engineering mobile application user interfaces with remaui (t)
24. A complete optical character recognition methodology for historical documents
25. M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin and D. Tarlow, “Deepcoder: Learning to write programs,” ArXiv, 2017.
26. Sketching interfaces: Toward more human interface design10.1109/2.910894 ↗
27. Quantum image edge extraction based on improved sobel operator10.1007/s10773-019-04177-6 ↗
28. A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” ArXiv, 2021.10.1016/j.physd.2019.132306 ↗
29. Y. Liu, M. M. Cheng, X. Hu, J. W. Bian, L. Zhang et al., “Richer convolutional features for edge detection,” ArXiv, 2013. [Online]. Available: https://arxiv.org/pdf/1612.02103.pdf.
30. O. Ronneberger, P. Fischer and T. Brox “U-Net: Convolutional networks for biomedical image segmentation,” ArXiv, 2015 [Online]. Available: https://arxiv.org/pdf/1505.04597.pdf.10.1007/978-3-319-24574-4_28 ↗
31. Color based image segmentation using different versions of Kmean in two spaces
32. R. Rana, J. Epps, R. Jurdak, X. Li, R. Goecke, M. Breretonk and J. Soar et al., “Gated recurrent unit (gru) for emotion classification from noisy speech,” ArXiv, 2016 [Online]. Available: https://arxiv.pdf/1org/801.06105.pdf.
33. Why one-hot encode data in machine learning?
34. Effective modeling of encoder-decoder architecture for joint entity and relation extraction10.1609/aaai.v34i05.6374 ↗
35. T. Beltramelli, “Github,” [Online]. Available: https://github.com/tonybeltramelli/pix2code seen July 2021.
36. Thomas Wood, “What is the softmax function?” Available: https://deepai.org/machine-learning-glossary-and-terms/softmax-layer/seen July 2021.
37. I. Sriram and A. K. Hosseini “Research agenda in cloud technologies,” ArXiv, 2010. [Online]. Available: https://arxiv.org/ftp/arxiv/papers/1001/1001.3259.pdf.
38. Y. Hu, A. Huber, J. Anumula and S. Liu, “Overcoming the vanishing gradient problem in plain recurrent networks,” ArXiv, 2018 [Online]. Available: https://arxiv.org/abs/1801.06105.
39. Twitter bootstrap, [Online]. Available: https://getbootstrap.com/2.0.2/ seen July 2021.
40. Research on improved canny edge detection algorithm
41. R. C. Staudemeyer and E. R. Morris, “Understanding LSTM–a tutorial into long short-term memory recurrent neural networks,” ArXiv, 2019. [Online]. Available: https://arxiv.org/pdf/1909.09586.pdf.
42. S. Yuheng and Y. Hao “Image segmentation algorithms overview,” ArXiv, 2017. [Online]. Available: https://arxiv.org/ftp/arxiv/papers/1707/1707.02051.pdf seen July 2021.
43. Research on the visualization of spatio-temporal data
44. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis10.1038/srep26286 ↗
45. Bisong · Building Machine Learning and Deep Learning Models on Google Cloud Platform · 2019 · 110.1007/978-1-4842-4470-8 ↗
46. Understanding data augmentation for classification: When to warp?