About me

I am currently a research scientist at CSIRO’s Data61. Before joining Data61, I was a Ph.D. student in computer science at Australian National University. My supervisors are Zhenchang Xing and Chunyang Chen.

My work lies in the fields of software engineering, deep learning, and human computer interaction. By understanding the semantics of user interfaces, my work aims to improve designers’, developers’, and end-users’ efficiency when designing, developing and using the mobile applications. I am currently working on android code generation from User Interface (UI) design, UI design search and generation, and mobile application accessibility enhancement.

News

• [2022-12-01] Selected as a Junior PC Member at MSR2023
• [2022-08-31] Gave a talk titled “Understanding Fairness Issues in Machine Unlearning and Dark Pattern” at NIST Workshop on Mitigating AI Bias in Context
• [2022-07-08] Released a preprint version of our work “Extracting Replayable Interactions from Videos of Mobile App Usage”
• [2022-01-10] Joined CSIRO’s Data61 as a research scientist
• [2021-11-16] Our paper “Towards Complete Icon Labeling in Mobile Applications” accepted at CHI 2022
• [2021-05-26] Presented our paper “Wireframe-based UI Design Search through Image Autoencoder” at ICSE 2021

Publications

[Preprint] Extracting Replayable Interactions from Videos of Mobile App Usage
Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols and Xiaoyi Zhang
July 2022 · PDF

[CHI’22] Towards Complete Icon Labeling in Mobile Applications
Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols and Xiaoyi Zhang.
PDF · Talk · Acceptance Rate 12.5% (324/2,597)

[ESEC/FSE’20] Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu and Guoqiang Li
PDF · Tool Demo · Talk · Code · Acceptance Rate 28% (101/360)

[TOSEM’20] Wireframe-based UI Design Search through Image Autoencoder.
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang.
PDF · Talk · Code

[ICSE’20] Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning.
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang.
🏆 ACM SIGSOFT Distinguished Paper Award
PDF · Talk · Code · Acceptance Rate 20.9% (129/617)

[IJCNN’17] Ensemble Application of Convolutional and Recurrent Neural Networks for Multi-label Text Categorization.
Guibin Chen, Deheng Ye, Zhenchang Xing, Jieshan Chen, and Erik Cambria.
PDF

Service

Program Committee
· The Mining Software Repositories (MSR-23-Junior PC, MSR-22-Shadow PC)
· International Conference on Software and System Processes (ICSSP 2023 - PC Member)
· IEEE International Symposium on Software Reliability Engineering (ISSRE-22-Industry Track)
· ACM CHI Conference on Human Factors in Computing Systems (CHI-22-Computational UI Workshop)

Reviewer
· IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR-23)
· ACM CHI Conference on Human Factors in Computing Systems (CHI-22, CHI-23)
· ACM international joint conference on pervasive and ubiquitous computing (UbiComp-22)
· International Journal of Human-Computer Interaction (IJHCI-22)
· IEEE International Symposium on Software Reliability Engineering (ISSRE-22)
· ACM Symposium on User Interface Software and Technology (UIST-22)
· IEEE Transactions on Software Engineering (TSE)
· Information and Software Technology (IST Journal-2021)
· The Mining Software Repositories (MSR-2021)