Using LLMs to Develop Software: Ethics, Risks, and Responsible Practice

A living framework for NGOs, civil society organizations, and mission-driven teams navigating AI adoption in software development.

Second draft - due to update Nov 2026


Collaborators

This framework was built in collaboration and is adopted by the following partners:

Introduction

AI coding tools have moved from novelty to daily workflow in under two years. Andrej Karpathy coined the term "vibe coding" in early 2025 - describing developers who prompt AI, accept all suggestions, and barely read the output. By early 2026, he had already moved on, calling the practice outdated and advocating instead for "agentic engineering": careful, supervised AI-assisted development with full human oversight 1. An early-2025 study found AI tools made experienced open-source developers slower 21, but the same team's early-2026 follow-up found weak evidence of a speedup instead, as models improved 38. In response to the general adoption of AI tools open-source communities have made growing efforts to establish appropriate governance and usage policies to reflect their use.

This trajectory tells us something important: the tools are real and improving rapidly, but the hype cycle consistently outpaces responsible adoption. For any organization working in the public interest and given the prevalence of LLM use in software in this moment, a governance approach that provides deliberate attention to the ethics of LLM use in software is required.

The gap between LLM use and governance is apparent: a 2026 survey found 93% of aid workers had used AI tools, but only 22% worked under a formal AI policy 54, and local organizations in crisis regions show the highest daily usage with the least governance support 55.

This document provides a framework in three parts: the ethical concerns AI adoption raises and how to mitigate them; the specific responsibilities that arise when AI intersects with open-source practice; and the human dimensions - learning, craft, and cognition - that must be protected as these tools become pervasive.


1. Ethical Concerns

AI adoption is not just a tooling decision. It is a values decision. Below are the primary ethical risks, alongside practical mitigation strategies.

This framework aims at harm reduction, not blame. Shaming individuals for using AI, or for refusing to, is counterproductive: it drives usage underground and makes honest assessment of risk impossible. Nobody should be forced to use AI tools, and nobody should be shamed for declining to work with AI output. What follows is intended to support honest dialogue about risks, practical mitigations, and better technical literacy.

1.1 Data Privacy and Security

Prompts sent to proprietary AI services may be stored or reused. Pasting sensitive data - beneficiary records, donor information, strategy documents, personnel details - into a commercial AI tool creates privacy and security exposure. In software development, this mostly surfaces as leaked credentials and logs.

Research published in Nature Scientific Reports highlights the cybersecurity risks inherent in AI-generated code, including injection vulnerabilities, insecure templates, and insufficient input validation 2. Agents themselves are also now a target: prompt injection through connected tooling is increasingly common 31, and coding agents account for the majority of reported agentic AI security incidents 32.

LLMs are exceptional tools for discovering security vulnerabilities in code 33, but be especially careful using them while investigating an undisclosed vulnerability, since prompts describing it may be stored, exposed, or later discoverable, creating a disclosure risk for the organizations maintaining the affected software. Chat logs carry no legal privilege and can be obtained in litigation, as shown when a US court ordered OpenAI to produce 20 million user conversations 30.

Mitigation approaches:

1.2 Bias and Discrimination

AI models are trained on internet-scale data that reflects existing societal biases. Research has shown LLMs associating specific ethnic groups with violence, reproducing gender stereotypes, and skewing outputs toward Western perspectives 3. For organizations serving marginalized communities globally, this is not an abstract concern - it is an operational risk. AI-generated content, code, or analysis may silently encode assumptions that undermine the very populations an organization exists to serve.

In coding, bias can surface in less obvious ways: culturally narrow test data, dataset assumptions, internationalization blind spots, and biased evaluation criteria in synthetic datasets.

Mitigation approaches:

1.3 Environmental Cost

Training large language models requires enormous computational resources. Each query consumes energy. For organizations with environmental or sustainability principles, adoption of AI tools creates a tension between productivity gains and ecological impact.

The energy used by each LLM prompt does not tell the full story. When considering this point, the wider implications on society should be considered. The electricity consumption may be small per-prompt (see calculations appendix), but usage of proprietary LLM services overall sends a signal to corporations running them that they are desirable. We end up with a feedback loop: people/businesses use AI → utilization/revenue expectations rise → labs demand compute → cloud providers commit capacity → financing becomes available → GPUs/servers get ordered → component production expands → data centers get constructed → utilities add generation/transmission → decades-long physical assets exist 37. This entire chain has huge repercussions on long term environmental impact.

Mitigation approaches:

1.4 Labour and Exploitation

The refinement of AI models often relies on low-paid human labor for data labeling and content moderation, frequently in low- and middle-income economies. The training data itself was often collected without consent from its creators. Using these tools means participating in a supply chain with unresolved ethical questions about consent, compensation, and intellectual property 20.

As investigative journalism and studies catch up, data supporting this is growing. Investigations found Kenyan workers paid under $2 per hour to filter traumatic content 43, with similar conditions reported in India 44. Fairwork's 2025 ratings of cloudwork platforms show that, of the data-work suppliers assessed, none paid a living wage and only two guaranteed minimum wage 45. Big tech firms also route this work through at least 30 intermediary platforms, fragmenting accountability 46. Amnesty International concluded in 2026 that generative AI systems built on unlawfully scraped data are incompatible with international human rights law 47.

Mitigation approaches:

1.5 Intellectual Property and Copyright

Current AI models raise unresolved questions about copyright. The LLVM Project's AI policy states it clearly: using AI tools to regenerate copyrighted material does not remove the copyright, and contributors remain responsible for ensuring nothing infringing enters their work 4. The risk includes inadvertently incorporating copyrighted code or text into publicly released outputs.

The law is getting clearer on three points. You only own AI-generated code if you meaningfully shaped it; prompting alone does not make you the author 56, and code nobody owns cannot be covered by your project's license. The main live lawsuit against GitHub Copilot is not about copying code, but about license and credit notices being stripped out 59, so keep those notices intact. Legal trouble has so far landed on the companies that trained on pirated material, not on the people using the tools 58. In the EU, providers must now publish a summary of their training data 57, which is worth checking when picking a tool.

There is no way to fully avoid the underlying ethical problem - training material taken without consent - short of not using the models. Most models called 'open' only release their weights, and their training data carries the same problems as proprietary models. A small number are trained only on openly licensed material - see recommended open models.

Mitigation approaches:

1.6 Digital Divide and Equity

AI coding tools are already more accessible to people in wealthy countries, and as the technology industry attempts to recoup its enormous capital investments, prices are likely to rise. At the same time, AI tools are eroding the equitable commons of free and open-source knowledge and universally accessible knowledge bases like Stack Exchange. There is a real risk of a two-tier system developing: massively powerful tools running in corporate data centers for the well-resourced, much less capable local instances for everyone else, and a diminished shared commons between the two 5.

Connectivity and access are not automatically benefits. Scholars of digital colonialism describe a recurring discourse of benevolence, where projects framed as bridging divides extend extraction and dependency instead 50. Support for AI adoption should follow community consultation and demonstrated need, not the assumption that access to these tools is inherently good.

Mitigation approaches:

1.7 Economic and Financial Risk

Depending on AI tools is also a financial bet. Proprietary LLM pricing is volatile, models are deprecated with little notice, and current prices are widely believed to be subsidised by investor capital. The Bank for International Settlements warns that an AI investment bust, opaque circular financing between AI firms, and record debt levels are interlocking risks to the wider financial system 48. The externalities are already visible: AI datacenter demand has driven severe global memory price rises, with DRAM contract prices up around 60% in a single quarter of 2026 49, raising hardware costs for organizations and communities that never touch an LLM.

Mitigation approaches:


2. AI in Open Source: Responsibility, Pressure, and Maintenance

AI affects not just how we code - but how we participate in the commons.

2.1 Asymmetric Pressure and Extractive Contributions

Dries Buytaert, lead of the Drupal project, describes the core problem precisely: AI makes it cheaper to contribute, but it does not make it cheaper to review 6. More contributions are flowing into open-source projects, but the burden of evaluating them still falls on the same small group of maintainers. This creates asymmetric pressure that risks burning out the people who hold projects together 51.

The LLVM Project introduced the concept of an "extractive contribution" - one where the cost to maintainers of reviewing it exceeds the benefit to the project 4. Before AI, posting a change for review signalled genuine interest from a potential long-term contributor. AI has decoupled effort from intent. A drive-by contributor can now generate a large patch in minutes and shift hours of review work onto volunteers.

Daniel Stenberg, maintainer of curl, canceled the project's bug bounty program after AI-generated reports flooded his seven-person security team - fewer than one in twenty turned out to be real bugs. Yet in the same period, an AI security startup used AI well and found all 12 zero-day vulnerabilities in a recent OpenSSL security release, some hiding for over 25 years 7. The difference was not whether AI was used. It was expertise and intent behind the contributions. Further to this point, a study of agent-assisted pull requests on Github showed hand-written PRs are more frequently trusted and merged, with agent-assited PRs having a near 50% chance of needing reviewer revisions 52.

AI-generated code also frequently reinvents the wheel - producing custom implementations rather than leveraging well-tested community libraries. This creates fragmentation and shifts maintenance burden onto the ecosystem 8. Repository data bears this out: an analysis of 623 million code changes found code reuse falling by a third, refactoring dropping from 21% to under 4% of changed lines, and duplicated code blocks up 81% since AI assistance became widespread 53.

Mitigation: review discipline and contribution hygiene. Good engineering practice matters more than ever. Organizations should formalize policies addressing AI in contributions. For practical guidance, see Working with AI Tools as a Developer and Repo Checklist.

Regardless of whether AI is used:

2.2 Long-Term Maintenance

While AI can be effective for quickly getting something up and running, it creates a significant pain point when it comes to maintaining or upgrading that code. If the people responsible for a codebase do not understand how it was built, they will eventually hit a wall - making maintenance, debugging, and upgrades extremely difficult. This can ultimately restrict an organization's ability to build anything new, because it is trapped by code it cannot confidently modify 9.

AI tools are demonstrably helpful when assisting someone who already understands the codebase and the broader technical landscape, but they are far less reliable as a substitute for that understanding.

See AI-Assisted Coding Guide for details on appropriate usage of LLMs.

2.3 What Leading Projects Are Doing

Project responses range from cautious acceptance to outright bans. The landscape is moving fast, but the following represent the most significant approaches as of 2026. Notably, the platforms hosting open-source projects have been slow to provide maintainer tooling for filtering or flagging AI-generated contributions - several projects cite this as a direct driver of their restrictive policies 10. OSS foundations, meanwhile, have largely focused on licensing questions rather than the quality and burnout crisis maintainers are facing now 10.

For an ongoing view, RedMonk maintains a policy landscape covering 86 organizations 60, and a community-maintained table tracks over 300 project policies 61.

Disclosure and accountability:

Still navigating:

Restrictive approaches or bans:

The common thread: human accountability, transparent AI use, respect for maintainer time, and protection of the commons. Notably, even projects that ban external AI contributions often use AI internally - the issue is not the tool itself but the absence of understanding, accountability, and genuine engagement behind the contribution.


3. Sustaining Human Skill, Judgment, and Craft

AI tools are powerful, but they interact with human cognition in ways that require deliberate management.

3.1 Cognitive Risks

Research confirms what many developers suspect: how AI is used matters as much as whether it is used. In a randomized study, participants who relied solely on generated code scored just 24–39% on follow-up comprehension tests, while those who asked for explanations scored 65–86% 22. The delegation group finished fastest - but retained the least.

Several patterns can erode skill if left unchecked:

For small teams, this is a serious risk. If developers stop deeply understanding the systems they maintain, there is no safety net - and the people least equipped to debug AI-written code may be those whose skills were eroded by relying on it.

Mitigation approaches:

3.2 Preserving the Craft of Engineering

AI can generate syntactically correct code quickly. But framing the right problem, designing architecture, evaluating trade-offs, aligning with stakeholder needs, and mentoring others - these remain deeply human tasks. As AI handles more of the mechanical work of coding, it becomes more important, not less, for human interaction to focus on problem framing, approach discussion, and alignment before setting AI to do the implementation work.

This applies with particular force to junior developers. The errors and dead ends that feel frustrating during independent work are also where the deepest learning happens. Skipping that struggle in favor of AI-generated solutions can create a gap between apparent productivity and actual competence - one that may not become visible until something breaks in production.

Practical commitments:


Guiding Principles

  1. Human accountability is non-negotiable. AI assists; humans decide, review, and own the output.
  2. Transparency is mandatory. When AI is used, it should be disclosed - in code commits, in documents, in reports. Disclosure must never be punished: research shows that when disclosing AI use attracts stigma, people simply hide it 39.
  3. Protect your maintainers. Never allow AI to increase the burden on those who review and maintain code without providing corresponding relief.
  4. Prioritize learning over speed. An organization's greatest asset is its people. If AI adoption undermines their ability to learn and grow, the short-term productivity gain is not worth it.
  5. Never input sensitive data into commercial AI tools. Beneficiary data, personnel information, strategic documents, and donor details must not enter commercial AI systems without clear data governance.
  6. Interrogate bias actively. Every AI output that touches the communities you serve should be critically evaluated for embedded assumptions.
  7. Respect the open-source commons. Ensure AI-assisted contributions are high quality, transparent, and do not extract more from maintainers than they give back.
  8. Champion equitable access. Advocate for and invest in open-source models that can run locally, ensuring communities are not left behind.
  9. Use fit-for-purpose models. Match the tool to the task; do not default to the largest available model.
  10. Always favor small, reviewable changes. Good engineering discipline is the best defense against AI-generated complexity.

Living Document Commitment

AI capabilities, norms, and risks evolve rapidly. This document should be reviewed and updated at least every three months. Responsible AI adoption is not about maximizing automation - it is about responsibly augmenting human capacity while protecting beneficiaries, contributors, the open-source ecosystem, and the long-term capability of teams.

This framework is intended as a starting point for consultation among NGOs, civil society organizations, and mission-driven teams. Contributions, critique, and adaptation are welcome.


References

  1. Karpathy, A. (2025–2026). From "vibe coding" to "agentic engineering."
  2. Nature Scientific Reports (2026). Cybersecurity risks in AI-generated code.
  3. Queen Margaret University Library. Generative AI: Ethics.
  4. LLVM Project. AI Tool Policy.
  5. Stack Overflow Blog (2025). Whether AI is a bubble or revolution, how does software survive?
  6. Buytaert, D. (2025). AI creates asymmetric pressure on open source.
  7. AI finds 12 of 12 OpenSSL zero-days while curl cancelled its bug bounty.
  8. Mapscaping Podcast. Vibe coding and the fragmentation of open source.
  9. Caimito (2025). The recurring dream of replacing developers.
  10. Holterhoff, K. (2026). AI Slopageddon and the OSS Maintainers. RedMonk.
  11. Nair, K. (2026). AI usage in popular open source projects.
  12. Linux kernel mailing list AI Policy.
  13. cURL contribution policy: On AI use in curl.
  14. QGIS Enhancement Proposal. AI Tool Policy.
  15. GDAL AI Tool Policy.
  16. OpenDroneMap AI contribution policy discussion.
  17. Debian General Resolution: LLM usage in Debian.
  18. Hicks, C. Cognitive helmets for the AI bicycle.
  19. Cloud Native PG AI Usage Policy
  20. Regilme, S.S.F. (2024). Artificial Intelligence Colonialism
  21. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
  22. Shen, H.S & Tamkin, A. (2026). How AI Impacts Skill Formation
  23. Epoch AI. How much energy does ChatGPT use?
  24. OpenAI. Data Residency Docs
  25. International Energy Agency. Global Emissions Report
  26. Founders Pledge. Climate And Lifestyle Report
  27. Effective Environmentalism. Climate Charity Recommendations
  28. GeoServer Project. AI Policy.
  29. Hannon, K. (2026). Open Source Maintainership in the Age of AI. Kubernetes Blog.
  30. Data Privacy + Security Insider (2026). When Chats Become Evidence: Court Affirms Order Requiring OpenAI to Produce 20 Million De-Identified ChatGPT Logs.
  31. Cloud Security Alliance (2026). Agentjacking: MCP Injection Hijacks AI Coding Agents.
  32. Help Net Security (2026). Prompt Injection Still Drives Most Agentic AI Security Failures in Production. Reporting on OWASP GenAI Security Project, State of Agentic AI Security and Governance v2.01.
  33. Anthropic (2026). Project Glasswing: An Initial Update.
  34. Oviedo, F. et al. (2026). Energy Use of AI Inference: Efficiency Pathways and Test-Time Scaling. Joule.
  35. Jegham, N. et al. (2025). How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference.
  36. Masley, A. (2025). A Cheat Sheet for Conversations About AI's Environmental Impact. Argues individual AI use is climate-negligible; cited here as a counterpoint.
  37. Zitron, E. (2026). The More You Buy, The More You Lose. Where's Your Ed At. Documents hyperscaler capacity commitments, circular financing between AI labs and their suppliers, and the resulting component demand loop.
  38. METR (2026). We Are Changing Our Developer Productivity Experiment Design.
  39. Schilke, O. & Reimann, M. (2025). The Transparency Dilemma: How AI Disclosure Erodes Trust. Organizational Behavior and Human Decision Processes.
  40. Data Center Watch (2026). $64 Billion of Data Center Projects Blocked or Delayed.
  41. Food & Water Watch (2026). How to Stop a Data Center Near You.
  42. Brookings (2026). Data Center Moratoriums Are Not a Substitute for Oversight.
  43. Perrigo, B. (2023). OpenAI Used Kenyan Workers on Less Than $2 Per Hour. Time.
  44. The Guardian (2026). "In the End You Feel Blank": India's Female Workers Watching Hours of Abusive Content to Train AI.
  45. Fairwork (2025). Cloudwork Ratings 2025. Oxford Internet Institute.
  46. SOMO (2026). Big Tech Sets Unfair Terms and Conditions for AI Data Workers Globally.
  47. Amnesty International (2026). Violations in the Shell: Exposing the Human Rights Costs of Generative AI.
  48. Bank for International Settlements (2026). Annual Economic Report 2026.
  49. Tom's Hardware (2026). Memory Price Surge Begins to Cool as Consumers Hit Affordability Limit.
  50. Nothias, T. (2025). An Intellectual History of Digital Colonialism. Journal of Communication.
  51. Baltes, S., Cheong, H. & Treude, C. (2026). "An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development.
  52. Watanabe, Y. et al. (2026). On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub.
  53. GitClear (2026). The AI Code Quality Maintainability Gap.
  54. Nhando, D. (2026). Uncovering the Humanitarian and Nonprofit Sectors' AI Governance Crisis. TechPolicy.Press.
  55. Parkinson, K.M. (2026). How Are Humanitarians Using AI in 2026? Humanitarian Leadership Academy.
  56. US Copyright Office (2025). Copyright and Artificial Intelligence, Part 2: Copyrightability.
  57. EU AI Act. Article 53: Obligations for Providers of General-Purpose AI Models.
  58. Authors Guild (2025). What Authors Need to Know About the Anthropic Settlement.
  59. Joseph Saveri Law Firm. GitHub Copilot Litigation.
  60. Holterhoff, K. (2026). Generative AI Policy Landscape in Open Source. RedMonk.
  61. Open Source AI Contribution Policies (community-maintained).

Additional Sources

The following sources informed the development of this framework but are not directly cited above.


Disclaimer: Initial content summarized by Claude Opus 4.6 from the sources listed above, then manually reviewed and edited. This document is released for consultation and collaborative refinement.


Appendix A: Methodology for Estimating LLM Energy & CO₂ Emissions and Donation Proxy

There is no easy way to estimate energy usage of LLM queries.

Below are some simple 'back-of-the-envelope' calculations to give a rough estimation of the potential magnitude of energy consumption.

1. Approximate LLM Usage

The most accurate approach would be to average token use per team member on a given provider.

However, as we do not have a prescriptive usage policy, and developers can use open models, we need approximations:

6400 prompts per month

2. Convert Queries to Electricity Usage

Energy per query: ~0.008 kWh

51.2 kWh usage per month (for a 5 person team)

3. Convert Electricity to CO₂ Emissions

tCO₂e = kWh_total × (gCO₂/kWh / 1000 / 1000)

0.023 tonnes CO₂ equivalent produced per month

4. Convert Emissions to Donation Proxy

Recommendation: ~23 USD donation per month, for a team of 5 devs using LLM assistance.