In a Remote IDE question, you can let the candidate work with an AI assistant inside the development environment, the way they would on the job. The conversation with the assistant is part of the assessment: it is recorded and shows up in the result, next to the code.
This way you assess the code that was delivered and also how the person used AI to get there.
Who can use it
The Coodesh assistant in the IDE is available on paid plans, in workspaces where Coodesh has turned the feature on. To turn it on in yours, talk to the Coodesh team. On the free plan, the session always starts without the assistant.
Turn the assistant on in the question
In the Library, open the Remote IDE question and go to Environment settings. Under AI policy, choose Coodesh credential. The default is No AI assistant, so that no question turns AI on by accident.
Once the Coodesh credential is selected, the Assistant level field appears:
Explanation and code (default): the assistant reads the file the person points to and proposes small, targeted changes, which the person accepts or not. When asked how to do something, it answers with a code snippet. It does not hand over the full solution to the question.
Agent: the assistant goes through the project, edits files and runs commands on its own.
Use Explanation and code when most of the code should be the person's own. Use Agent when what you want to see is how they steer an agent: what they ask for, what they check and what they fix.
The other options in the AI policy list cannot be selected yet.
Change it for one assessment only
In the assessment's Sections tab, open the question with the arrow next to its name. In the AI policy and Assistant level fields, you change what applies to this assessment only. With no choice, the question's setting applies, and the Inherit from question option shows what it is.
The usage limit
Each question has a limit on assistant usage, agreed between your company and Coodesh. It can be counted in two ways:
By messages: counts the messages the person sends to the assistant.
Same budget for everyone: each person gets the same usage budget per question, and the IDE shows how much of it has been used, as a percentage.
On top of the question's limit, there is the workspace's monthly limit. If it has already run out when the person opens the IDE, the session starts without the assistant.
What the candidate sees
Before starting the assessment, on the terms step, the candidate confirms that they will not use external AI tools and that they are aware the assistant will be available and that their interactions with it are recorded for the company.
Inside the IDE, the assistant sits in the chat panel. In the status bar and at the top of the chat, an indicator shows how much of the limit has been used. Near the end, the indicator changes color. When the limit runs out, it says so, and the person carries on with the editor and the terminal.
When reviewing
Open the person's result and click the question. Right after Automatic review comes the AI assistant usage section, built from the conversation with the assistant and the final code. It shows:
Summary: how the person used the assistant, in a few lines.
Conversation figures: messages sent, requests about the task, requests in which the assistant changed the code and how many new lines came from the assistant.
Who wrote the final code: lines written by the assistant, copied from the chat by the person and written by the person, compared with the starter project.
AI usage criteria: each criterion marked as met, partial, not met or no evidence, with the requests that back it up.
Signals: positive points and points of attention in how the assistant was used.
Timeline: each request and reply, with the minute it happened and its effect on the code (change applied, change proposed, snippet in the chat or no effect), alongside the person's commits.
Suggested interview questions, to dig into what came up in the conversation.
The section does not change the score. It is there to support the reviewer.
If the assistant was not available in the session, the result says why, for example when the usage limit had already been reached.
AI usage criteria
The criteria describe how the person should work with the assistant on that question. Every question starts with the platform's default criteria. To set your own, open the question in the Library and, under AI usage criteria, click Customize criteria. You can have up to 8 criteria, with names of up to 80 characters and descriptions of up to 300. Restore default goes back to the platform's criteria.
Frequently asked questions
Does using the assistant change the score?
No. The score still comes from grading the code. The assistant usage section is extra information for the reviewer.
What happens when the limit runs out in the middle of the question?
The assistant stops answering and the indicator shows that the limit was reached. The environment keeps working, and the person finishes the question with the editor and the terminal.
Can I allow the assistant in one question and block it in another?
Yes. The policy is set per question, and each assessment can still adjust its own.
Learn more: How to use the Remote IDE question type.