HCI PhD candidate @dgpToronto @UofT | Student Researcher @GoogleDeepMind | prev @Meta @RealityLabs @allen_ai @SemanticScholar
- Tweets 122
- Following 433
- Followers 630
- Likes 566
🔬Research ideation is hard: After the spark of a brilliant initial idea, much work is still needed to further develop it into a well-thoughtout project by iteratively expanding and refining the initial idea and grounding it to relevant literature. How can we better support this?
11
81
637
93,172
🛠️Introducing IdeaSynth, a system that decomposes research ideas into facets (e.g. RQ, method, impact) and uses literature-grounded feedback to help scientists evolve, refine, and compose idea facets into concrete, literature-grounded research briefs. (Cond. accpt. to #CHI2025 !)
2
3
47
3,337
How can users create web automation that executes the right UI logic for different input data and web content? DiLogics semantically breaks down tasks, maps user demonstrations, and generalizes to new cases.
I'll be presenting DiLogics at #UIST2023!
P:arxiv.org/abs/2308.05828
1
8
45
8,149
🤖👩🏻💻Proactive AI tools like @GitHubCopilot @cursor_ai @allhands_ai promise to assist developers by anticipating their needs and automating engineering processes—but do they truly help?
We evaluated three design probes to explore the trade-offs of proactive AI programming support
1
13
57
7,749
A great line of research works! Feel free to check out my two #CHI2025 papers:
IdeaSynth: Iterative Research Idea Development
arxiv.org/abs/2410.04025
Assistance or Disruption? Evaluating Designs of Proactive AI Programming
arxiv.org/abs/2502.18658
More details on this one soon!😉
That’s the lineup! 🚀 Full details here:
tovigrossman.com
🌟 Shoutout to the incredible grad students and post-docs involved this work! Give them a follow to stay updated on their research:
@LepingQ, @joaleong, @kevpjk, @VioletHanyinuo, @ItsHaiNotHi, @MajeedKazemi, @sangho_suh, @ludwigsidenmark, @BlaineLewis15, @hciresearcher, @karthikm0
(13/13)
5
3
37
7,841
#UIST2023
Super excited to be at UIST again! This time I’m serving as a student volunteer.
I’ll also be presenting my paper DiLogics, a neuro-symbolic PBD system that creates generalizable UI automation based on semantic understanding.
P: arxiv.org/abs/2308.05828
1
1
33
2,805
🎉 Huge thanks to my collaborators @siangliulue @ToviGrossman @kjfeng_ @josephcc @turingmusician Matt Latzke @Hoper_Tom @bhavana_dalvi ! This work was done during an internship @SemanticScholar @ai2_s2research @allen_ai.
Paper: arxiv.org/abs/2410.04025
2
2
26
2,013
Super excited to receive the Honorable Mention Award at #UIST2022!
Shoutout to my mentors and coauthors for the great collaboration!
Feel free to check out the SemanticOn paper and demo in the links below!
Honored to receive #UIST2022 Honorable Mention!
SemanticOn helps specify visual & textual semantic conditions in web automation programs by NL desc. & content highlighting.
Kudos to my mentee Kevin Pu, & coauthors.
P: bit.ly/3Vc2agO
D: bit.ly/3Mhz9w0
4
1
25
At #UIST2024 again as an SV! Also gonna take photos for @XiaohangTang who's gonna present our work VizGroup, a visual analytics system for real-time classroom activities. Be sure to check it out!
I'll be at #UIST2024 presenting VizGroup! Join us for the paper (AI as Copilot; 4:40 PM - 4:55 PM) and poster (5:40 PM - 5:55 PM) sessions on Tue =]
1
2
23
2,371
Had a chance to visit the great Crater Lake before attending #uist2022! Even more excited to present our honorable mention paper SemanticOn done with @ToviGrossman @xwangsd and @yanchenum !
programs.sigchi.org/uist/202…
P: bit.ly/3Vc2agO
D: bit.ly/3Mhz9w0
3
5
18
How can we collaborate with AI agents on complex tasks (like doing research 👀) that require careful planning and execution?
Definitely check out Cocoa where we introduce a novel co-plan and co-execute interaction pattern!
🤔Giving complex tasks to AI agents is easy—getting them to do exactly what you want isn’t. How can human-AI collaboration give us more reliable & steerable agents?
🍫Introducing Cocoa, our new interaction paradigm to balance human & AI agency in complex human-AI workflows. 🧵
6
18
3,094
📝 IdeaSynth’s Core Features 1️⃣Idea Facets
Researchers can break down their initial ideas and iterate on different aspects (e.g., problems, methods, contributions). Using a canvas-based UI, IdeaSynth allows the user to connect, extend, and compose different facets.
1
13
2,132
🎉 Shoutout to my collaborators @dnlzro @IanArawjo @HaijunXia @ZiangXiao @ToviGrossman @HCI_Prof_YC!
The paper is conditionally accepted to #CHI2025, link to preprint: arxiv.org/abs/2502.18658
Share your thoughts and let’s discuss how to design better AI tools for developers!
11
454
🧐Why does IdeaSynth matter?
Most ideation tools generate broad initial ideas. But much manual effort is spent to specify and refine those ideas into detailed, actionable projects. Our work aims to support researchers in the follow-up iterative process after an initial spark.
1
10
2,417
Jiannan is an awesome researcher, mentor, and friend, highly recommend! 🚀
Working in a lively HCI community at the vibrant heart of Singapore! SMU HCI is offering multiple fully-funded PhD positions in the fields of accessibility, visualization, human-AI interaction, AR/VR, health, and HRI. 🌐 smuhci.com/ More info see 🧵. RT appreciated.
1
10
2,678
📝 IdeaSynth’s Core Features 2️⃣Literature-grounded Feedback
IdeaSynth provides actionable suggestions at facet- and canvas-level. The suggestions are grounded in a set of user-collected papers on the topic to identify novelty and contextualize ideas.
1
9
1,644
📝 IdeaSynth’s Core Features 3️⃣Turning Idea Facets into Research briefs
After exploring variations of idea facets, the researcher can select a path of idea facets to dynamically compose their ideas into cohesive research briefs that can be shared for feedback
1
9
1,450
🤖🔬 AI in Research:
IdeaSynth shows the potential for human-AI collaboration in research workflows. We outline design implications for preserving the researcher’s agency, and adapting to diverse research needs while ensuring ethical AI use in ideation.
1
1
8
1,862
📊 Key Study Findings:
- Literature-based suggestions and facet generation led to more exploration
- Linking facets to create a research brief boosted confidence in the resulting ideas
- The deployed IdeaSynth was used across ideation stages for new and existing projects
1
6
1,322
📋 Practical takeaways:
- Expanding and iterating on initial research ideas is critical in research ideation
- Literature context enhances the relevance and actionability of AI feedback
- Flexible interfaces (like IdeaSynth’s canvas) can inspire idea exploration and refinement.
1
4
1,261
👩🏻🎨Implications:
We share 5 design implications to enhance human-AI collaboration in programming, including facilitating code understanding, adapting AI salience, and adjusting proactivity for each sub-process.
We hope these insights can help guide future AI programming tools!
1
1
356
Someone told me dreaming is like an LLM that’s trained on your past experiences, so that the new randomly generated dreams must be somehow related to what has happened in the past but could just be amalgamated nonsense and I can’t stop thinking about that
1
2
285
📊 Quantitative findings:
Efficiency: Users spent less time interpreting proactive AI responses than prompted ones
Awareness: Visual context cues improved users' understanding of AI actions
Trade-offs: Proactivity boosted the sense of collaboration but also disrupted workflows
1
1
1
2,108
📋 Takeaways:
Users desired more proactivity for specific processes like code scaffolding or debugging
Balancing proactivity with flexible user control and agency is key
Proactivity reduces low-level efforts but also leads to over-reliance, citing concerns over code understanding
1
265
🛠️ Design Probe Dimensions:
- Proactive timing: AI intervenes during task boundaries or signals of user need
- Visual presence: Represent AI actions and context to increase awareness
- Flexible scope: Supports global and local interactions via chat and breakout threads
1
1
287
