TLDR: CodeWatcher is a novel client-server system implemented as a VS Code plugin that captures detailed, real-time telemetry data on how developers interact with code generation tools (CGTs). It logs specific events like code insertions, deletions, copy-paste actions, and focus shifts, providing fine-grained insights into coding behavior. This data enables researchers to understand CGT usage, evaluate developer productivity, and even classify code as AI-generated, AI-modified, or user-written. The tool aims to support research in responsible AI, improve educational feedback, and enhance industry practices by offering unprecedented visibility into the human-AI coding process.
Understanding how developers write code, especially when using modern AI-powered code generation tools (CGTs), has always been a complex challenge. Traditional methods often focus on the final product, like whether the code works, but they miss the crucial details of the coding process itself. This lack of insight makes it difficult to improve tools, boost productivity, or even understand where developers might be struggling within their Integrated Development Environments (IDEs).
This is where a new tool called CodeWatcher comes in. It’s a lightweight and unobtrusive client-server system designed to capture detailed interaction events directly from within the Visual Studio Code (VS Code) editor. Think of it as a silent observer that logs meaningful actions without getting in the way of a developer’s workflow. CodeWatcher records events such as code insertions made by CGTs, deletions, copy-paste actions, and even when a developer shifts their focus away from or back to the IDE. This continuous monitoring provides a rich stream of data about developer activity.
The system is built with three main components: a VS Code plugin that runs locally on the developer’s machine, a Python-based RESTful API that acts as the server, and a MongoDB database for storing all the collected data. All these components are designed to be easily deployed and scaled using containers, making it practical for various research and industry settings. By meticulously structuring and timestamping each event, CodeWatcher allows researchers to reconstruct entire coding sessions, offering deep insights into how and when CGTs are utilized during development. This capability is vital for advancing research in responsible AI, understanding developer productivity, and evaluating CGTs from a human-centered perspective.
How CodeWatcher Captures Interactions
CodeWatcher’s core strength lies in its ability to capture fine-grained programming activity without being disruptive. It specifically focuses on user interactions with CGTs, rather than just general coding behavior. The client plugin monitors eight distinct event types:
- Start: When a new file is opened and becomes active.
- End: When a file is no longer in focus.
- Insertion: When code is pasted from outside the IDE or when code suggestions (more than 3 characters) are accepted.
- Deletion: When any content is removed.
- Unfocus: When the user switches away from the IDE.
- Focus: When the IDE regains attention.
- Copy: When a copy command is executed.
- Paste: When a paste command is executed (the actual content is captured by a subsequent Insertion event).
Each event is recorded with details like its type, a precise timestamp, the relevant code snippet or user input, and the full line of code where the event occurred. To keep the data clean and relevant, CodeWatcher filters out single-character inputs and groups small edits into larger insertion and deletion events. Crucially, it distinguishes between typed input and accepted code generation suggestions, providing specific data on how AI tools are used.
Validating the Tool’s Accuracy
To ensure CodeWatcher works as intended, the developers rigorously validated its accuracy and completeness. They conducted 150 trials with two users, simulating realistic developer workflows. The validation confirmed that the tool correctly identifies and logs each interaction, that the captured metadata (like timestamps and text content) is precise, and that data is transmitted to the server without loss. The system proved robust under various scenarios, including rapid editing, accepting multi-line suggestions, working with multiple files, and handling concurrent users.
Identifying AI-Generated Code: A Practical Example
One compelling use case for CodeWatcher is its ability to help researchers analyze the authorship of code. The paper describes a post-processing script that uses CodeWatcher’s logs to determine whether each line in a final code submission was AI-generated, AI-modified by the user, or entirely user-written. This is achieved by comparing the final code with historical logs using a fuzzy string matching algorithm. Based on similarity scores, lines are labeled accordingly.
In an empirical example with three developers completing tasks, the results showed that, on average, 68% of the code was directly generated by an AI tool, 7% was AI-generated but modified by the user, and 25% was entirely user-written. This demonstrates CodeWatcher’s potential to reliably distinguish between human and AI contributions, offering a foundation for deeper analysis.
Also Read:
- AI’s Role in Software Teams: Boosting Collaboration and Productivity
- BIGCODEARENA: Elevating Code Generation Evaluation Through Execution
Broad Applications and Future Impact
The potential applications of CodeWatcher are extensive:
- Educational Feedback: Instructors can use the data to understand student engagement, identify over-reliance on AI, and refine curricula.
- Research on Programming Behavior: Researchers can study how programmers interact with code, IDEs, and CGTs, comparing behaviors across different tools or demographics.
- CGT Evaluation: The tool can assess the utility and limitations of CGTs, informing improvements in AI suggestion engines.
- Industry Applications: Development teams can evaluate productivity, streamline workflows, monitor onboarding, and even use logs for audit trails.
- Ethics and AI Accountability: CodeWatcher enhances transparency in AI-authored code, crucial for fair grading, authorship attribution, and identifying potential biases in CGTs.
- IDE Personalization: Real-time data can pave the way for intelligent, personalized IDEs that offer context-sensitive hints, documentation, or debugging suggestions when a user is struggling.
While CodeWatcher offers significant advancements, it does have some limitations, such as potential compatibility issues with different IDEs or CGTs, and the current version requires a server connection (though local setup is possible). It also intentionally doesn’t capture non-code text like comments, which might limit its use for certain types of analysis.
In conclusion, CodeWatcher provides a practical and robust foundation for studying the intricate ways developers interact with code generation tools, offering invaluable data for educators, researchers, and industry professionals alike. You can find more details about this research paper here: CodeWatcher Research Paper.
