Code LLMs in Software Development: Three Lessons from Practice

Lesson 1: Copilots Only Create Value If You Truly Master Them

Much has already been written about the programming capabilities of code LLMs. Success stories about efficiency gains are everywhere. But one legitimate question remains: do these tools also work in our specific context?

We didn’t want to rely on assumptions, we wanted to measure this in practice. That’s why we organized targeted information sessions and activated GitHub Copilot licenses for multiple teams and projects, always with our customers’ approval. This gave developers controlled access to relevant functionality and encouraged them to consciously integrate it into their workflows. After several weeks, we evaluated the results.

The conclusion was clear: effective adoption remained limited to a few basic features. A survey across teams showed that, in practice, LLMs were mainly used as:

• advanced auto-completion

• a convenient chat function integrated into the code editor

These features are useful, but they tap into only a fraction of what code LLMs are capable of. This usage largely corresponds to the most basic forms of the four standard agents, or specialized LLM profiles, offered by GitHub Copilot:

• Ask: the standard question-and-answer profile

• Edit: to apply targeted code changes

• Plan: to think through structure and approach

• Agent: to execute tasks autonomously

Especially Plan and Agent go a step further: they don’t just support the development process, they automate parts of it.

To deepen adoption structurally (less Ask, more Plan and Agent), we launched an internal expertise program led by experienced users, our champions. Through targeted training sessions and workshops, they share practical knowledge across teams. This allows us to build competence in a controlled way, rather than relying on isolated individual experiments.

Without focused guidance, example use cases, and structured knowledge sharing, most of the added value remains untapped. Anyone expecting real returns must therefore actively invest in training, coaching, and clear working frameworks.

Lesson 2: The Right Context Is Crucial for an LLM

Our champions learned a second key lesson: set the right expectations. Today’s models are best compared to a “book-smart” junior developer. They have an impressive amount of knowledge, but they need clear and extensive context to deliver exactly what you expect.

When we provide that context explicitly, code LLMs can independently generate medium to larger chunks of code. A thorough code review remains essential, however—the responsibility for quality always stays with the developer.

Fortunately, standardized ways now exist to capture this context explicitly and make it reusable, independent of any specific LLM platform. An additional advantage is that these context files are also easy for humans to understand. Teams use them to explicitly document agreements, standards, and architectural decisions.

This delivers three concrete benefits:

• Lower onboarding threshold: Less implicit knowledge is required, reducing dependency on specific team members.

• Visible intent: Not just the how, but also the why behind the code becomes clear.

• Consistency and auditability: Agreements around coding style, commits, and reviews remain verifiable.

Good context files therefore serve simultaneously as working agreements, onboarding material, and input for LLMs.

Lesson 3: Asynchronous Work Already Saves Time Today

These context files also enable new forms of efficiency. Asynchronous work is a concrete example. An LLM can generate code while a developer is attending a meeting.

In a specific, measured experiment, it took less than five minutes to formulate a focused task, start Copilot before the meeting, and review the result afterward. In this way, an LLM can convert up to 75% of meeting time into productive development time.

This is especially valuable for developers who spend a lot of time in meetings—often senior profiles. But the same rule applies here: asynchronous gains only materialize when the necessary context is provided clearly and completely upfront.

Toward Further Automation

With the right context and targeted use, code LLMs can already automate simple to moderately complex development tasks today. Human expertise remains essential at the strategic level: developers set direction, solve new and unpredictable problems, and make well-founded decisions.

By offloading repetitive work to AI, space is created for deeper analysis and creative thinking. Achieving a solid return from code LLMs therefore requires focused investment. Anyone looking to deploy them structurally should start small and controlled:

• begin with well-defined use cases in a safe experimental environment

• learn from those experiments and translate insights into reusable infrastructure building blocks

• scale only once the approach has been sufficiently validated

Only then can solutions be scaled and industrialized in a controlled way across projects.

AI Enablement in the Cegeka Software Factory

Within the Cegeka Software Factory, a way of organizing software development across teams, countries, and projects with consistent quality and scalability, we are now structurally embedding our successful experiments into our infrastructure.

We are investing in more specialized agents and in the Model Context Protocol (MCP). Specialized agents autonomously execute clearly defined subtasks, from brainstorming and implementation to testing and code reviews, creating a multi-agent system that works more consistently and efficiently. MCP extends these agents with secure access to external tools such as code repositories, build servers, and notification platforms, allowing them to go far beyond simple text or code generation.

Would you like your teams to work in an AI-enabled way across every step of the software development lifecycle? We’re happy to support you with targeted adoption programs, training, and hands-on workshops that help your teams get started quickly and safely. You can get in touch via the form below.