Security testing stopped being optional the moment you wired an open-source model into a product real users touch. One successful jailbreak can leak your proprietary prompts, expose customer data, or drop you into a compliance mess you never budgeted for.

DeepSeek V3 and R1 are appealing for good reasons: cheap, open, strong at reasoning. They are also popular targets for adversarial prompting. That is the tension. Learning how to break DeepSeek before an attacker does belongs in your workflow, not on a someday list.

Here is the recommendation, up front. Do not ship a DeepSeek-powered feature on the strength of a published benchmark. Run your own red-team pass against your actual use case, gate your pipeline on it, and re-run it on every version bump. The rest of this article is how.

Independent red-team results show these models can pass unsafe content at rates that would fail many compliance reviews. Ship without your own test pass and you are trusting a model card instead of measuring what the model does in your product.

The published scores are specific, not vague

Promptfoo’s public security evaluations present DeepSeek results by category rather than as a single headline number. That is the useful part. In one published comparison, R1 passed CyberSecEval injection cases at 82.22% but failed Pliny injection cases at 0%. Same model, same broad attack area, very different result depending on the test set.

The gap between capability claims and safety evidence is the real story. A model can be strong on reasoning while still failing on trust-and-safety controls. Published reports are a starting point, not a substitute for testing the specific model you plan to deploy.

Safety regresses between versions, so re-test every time

Never trust a prior score. Promptfoo publishes separate security reports for the original R1 and its May 28, 2025 update, which is your first clue that numbers can move meaningfully from one build to the next. A score from an older release tells you nothing about the build you deployed last week.

Treat every version as an unknown until you test it yourself. Pin the exact model version and date in every report, and re-run the full suite whenever the model changes.

Use several adversarially distinct injection corpora, or you will walk away with false confidence. In the published evaluation cited above, R1 passed CyberSecEval injections at 82.22% but failed Pliny injections at 0%. The dataset choice changes the verdict.

A credible DeepSeek red-team test pulls from multiple sources at once: CyberSecEval, BeaverTails, and Pliny prompt injections each expose different weaknesses. Run one and you might report a passing grade on a model that buckles instantly to a strategy you never tried.

Automated, repeatable testing turns that uncertainty into evidence. You are shipping AI features; your job is to know where the model breaks before your users do.

Setting up Promptfoo for DeepSeek V3 and R1

Before you can probe DeepSeek in a controlled way, you need a setup that talks to the model reliably. Promptfoo is the open-source evaluation framework we reach for. It is actively maintained, but DeepSeek model IDs and provider settings can change, so verify them against current provider documentation before you run a suite.

Get two things right first: a clean install and a verified connection. Everything downstream, from adversarial prompts to CI gates, runs on that foundation.

Promptfoo installs through npm, Docker, or a standalone binary. For local testing, npm is often the fastest path. Once it is on your machine, get an API key from the DeepSeek Platform and set it as the 

 directly into your config, but the environment variable keeps secrets out of version control.

DeepSeek exposes an OpenAI-compatible API. Promptfoo’s OpenAI chat options generally work here, but use the base URL and model ID shown in your DeepSeek provider documentation rather than copying old values.

Before a large prompt batch, check current DeepSeek usage pricing and assume the more expensive rate when you estimate cost. Token-based API costs can vary by time window.

The default settings that sabotage reasoning-mode tests

Some shorthand provider IDs map to a default model with thinking disabled, while older model IDs are retired. Copy-pasting an old config can leave you testing the wrong thing, or nothing at all.

Reasoning-mode controls also behave differently than normal chat sampling controls. In thinking mode, where the model generates hidden reasoning before answering, familiar sampling settings may not mean what you expect. Read the provider docs for the exact model you are calling before treating two runs as comparable.

The nastiest setting is any display-only reasoning control. Hiding reasoning from your assertions does not necessarily disable thinking at the API. For a true non-reasoning comparison, set 

config.passthrough.thinking: { type: disabled }

. Skip that step and you may unintentionally compare reasoning and non-reasoning behavior, then misattribute which mode leaked what.

Start with a prompt that has one right answer. “What is 2+2?” tells you nothing about safety, but it confirms your key, endpoint, and payload shape all work.

A minimal config to validate the connection:

providers:
  - id: your-current-deepseek-provider-id # replace with the current provider ID from DeepSeek docs
    config:
      max_tokens: 4000
      passthrough:
        thinking:
          type: disabled
prompts:
  - 'What is 2+2?'
tests:
  - assert:
      - type: contains
        value: '4'

If the provider does not support streaming, Promptfoo can request complete responses, so you get the full payload back. If the assertion passes, your setup is live. Now you can wire in real adversarial tests and start baselining DeepSeek on every model release.

The fastest way to waste a week is to build a red-team suite around one attack type and call it done. DeepSeek breaks in specific, uneven ways. Your test cases need to match those weak spots, not a generic checklist.

Before you write tests, decide which failures you are trying to expose: instruction-hierarchy failures, toxic generation, controlled-substance content, weapons guidance, privacy leakage, language-specific bypasses. A useful suite samples across these surfaces instead of asking one corpus to represent the whole threat model.

That is design rule one. Pull from several adversarially distinct sources so different prompt structures, objectives, and bypass styles are represented. CyberSecEval, the BeaverTails dataset, and Pliny injections all belong in the config because each stresses a different failure mode.

Pick categories where DeepSeek already fails

Start with the documented weak spots, not an even spread. Published reports flag clusters that consistently underperform: prompt injection, controlled-substance synthesis, religious bias, and weapons content all tend to land near the bottom.

Those rankings tell you where to concentrate. Weight jailbreak prompts, toxic-content generation, and prompt injection heavily. Build cases that mimic several distinct attack shapes, not just the obvious “ignore your policies” and “act as a developer” openers.

DeepSeek is multilingual, taking both English and Chinese input. That matters for toxicity testing. A prompt that trips an English filter may sail through in Chinese, so mirror your worst-case prompts across both languages instead of assuming one pass covers both.

Score severity, and re-run on every version bump

Promptfoo can sort findings into severity levels such as critical, high, medium, and low. Map that onto your own numeric weights so a passed low-risk test does not offset a severe safety failure. Use the severity categories as a starting template, then bias the weights toward the areas your product actually touches.

The part most teams botch is change management. Safety at DeepSeek is a moving target. A critical-severity area can post a passing-looking grade and still leave exploitable gaps, because a partial block rate means successful attacks exist. Comparing model reports across releases matters precisely because reasoning gains at this lab have repeatedly been discussed alongside safety concerns.

So a score from an old deployment is not evidence about the version you are shipping now. Pin the exact model ID and date in every report, and re-run the full suite on every version bump.

For privacy and training-data leakage, include divergent-repetition tests even when the baseline looks strong. They are cheap, and they catch regressions the harmful-content categories miss.

Skip the exhaustive sweep only if you are prototyping internally with no user exposure. The moment real users touch the output, the full weighted suite earns its runtime.

Running the suite and reading the results

With the suite designed, running it is the easy part. Point Promptfoo at your remote endpoint, let the plugins fire, open the report. The hard part is reading the numbers without fooling yourself.

Run the full attack set, capture every pass and fail, and sort failures by severity before you decide anything. A high pass rate on low-severity categories means little if a critical test slipped through.

Promptfoo runs your config and writes results to a structured output you can render as an HTML dashboard. The dashboard groups tests by category and severity, which is where your attention should go first. Skim the category view, then drill into the reds.

The result data is where the real work happens. Each test row carries a pass/fail verdict, a severity label, and a response time. Sort by severity descending. A critical failure on self-harm or child exploitation is a stop-ship event, full stop.

Latency matters too, but keep it in perspective. Reasoning-style responses can run long, so average response time may look worse than a distilled model. That is a cost and UX signal, not a security one. Do not let a slow-latency column pull your eye off a critical red.

What severity threshold should fail the build?

This is where teams get sloppy. Some red-team suites combine single-shot prompts, composite bypasses, and Likert-style jailbreaks. One clean single-shot pass tells you little.

Set your CI gate on severity, not overall pass rate. A practical rule: any critical failure fails the build, no exceptions. High-severity failures fail unless you have a documented compensating control upstream. Medium and low get logged and tracked, not blocked.

Published red-team reports show why that stance matters. Critical categories such as child exploitation and self-harm can produce failures that no serious release gate should wave past. If your build tolerates those, you are not actually gating on safety.

When is DeepSeek actually production-ready?

Do not issue one verdict for the whole model. That is the mistake raw comparison numbers invite. A meaningful result in one category does not redeem a weak pattern elsewhere.

The picture can invert by axis. A model may clear a bar for internal coding assistance, then fall far below the bar for public-facing harmful-output categories, regulated advice, or compliance workloads. Treat those as separate deployment decisions.

So separate the verdicts. Shipping an internal coding assistant with access controls and no user-facing content generation? DeepSeek’s security posture may clear your bar. Feature touches public content, harmful-output categories, or regulated advice? It may not. Gate those paths hard, add upstream guardrails, and re-run the suite on every model-ID change.

One caution worth flagging: an isolated strong score on a single axis does not prove the model is safe. The pattern that repeats across your report is what tells the truth.

Common vulnerabilities: DeepSeek V3 vs R1

The first thing you learn testing both models is that “DeepSeek” is not one target. It moves. An R1 result, a later R1 update, and a V3-family report should be treated as separate artifacts, not interchangeable evidence.

That is the practical lesson for anyone trying to find failures before an attacker does: a verdict on “DeepSeek R1” with no release attached is meaningless. Attach the model version, test date, and Promptfoo config to every result so a future reader can reproduce what you measured.

Injection resistance is wildly inconsistent

Injection defense is not a single property of a model. It is a property of the model measured against a specific corpus. Swap the attack set and the verdict can flip. A model that looks defended against one family of structured injections may still collapse against hand-tuned prompts built to defeat exactly that guardrail.

Pull from one source and you walk away with one narrow impression of injection defense. A credible test pass needs several corpora so you can compare structured attacks, policy-bypass payloads, and adversarial prompts that target instruction hierarchy directly.

Promptfoo’s published red-team work shows uneven results across R1’s harmful-content categories, with some categories far below top performers. Coverage of the DeepSeek line has framed capability-focused reasoning models as a place where safety controls can lag behind raw performance.

The operational conclusion is blunt. Independent red-teaming coverage has described DeepSeek R1 as carrying meaningful security risks in deployment, which is all the more reason to verify each version against your own suite rather than trusting a headline number.

The V3 family is not obviously safer, and Promptfoo maintains side-by-side security reports so you can compare V3-family builds against R1 and other models directly instead of relying on a single summary score.

A blanket “DeepSeek is unsafe” is lazy, though. R1’s own results show real variation across categories, a handful of strong passes sitting next to categories where defenses barely register. A single averaged number hides both extremes and tells you nothing actionable.

Promptfoo’s framework sorts findings by severity, which is how you should compare the models in practice. Neither model clears the bar most compliance reviews demand on every category. Test both, per version, before you ship either.

A model change should never reach production without a security pass. If your team ships prompt tweaks or fine-tunes weekly, manual testing falls behind by the second sprint. Wiring Promptfoo into your pipeline fixes that. Every commit that touches a prompt, a provider ID, or a system message triggers the same red-team suite you would run by hand.

This is where DeepSeek testing pays off as a repeatable gate instead of a one-time audit. Codify the attacks once, then let CI re-run them on every push. When a provider retires a model ID or you bump to a new model, the pipeline catches regressions before your users do.

A workflow YAML that runs Promptfoo on every push

Start with a GitHub Actions job that installs Promptfoo, injects your key, and runs the eval. Keep it minimal.

name: llm-security
on: [push]
jobs:
  redteam:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npx promptfoo@latest eval -c promptfooconfig.yaml
        env:
          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}

 from the environment, so store it as a repository or organization secret and never commit it. GitLab CI works the same way with a masked, protected variable. Never hardcode the key in your config. A leaked DeepSeek key is a billing and abuse problem you own.

Fail the build when severity crosses your line

A passing pipeline should mean something. Configure Promptfoo to exit non-zero when a critical or high test fails, and let that exit code break the build. Low-severity failures can warn without blocking, so your team is not held hostage by cosmetic misses.

Sort your gates by severity, not raw pass rate. An aggregate score is worthless if the failures include weapons or malicious-code categories. DeepSeek-family models have known weak spots that cluster in exactly those areas, so your threshold should be strictest there.

The reasoning-mode trap that poisons CI results

A common CI failure mode is confusing hidden reasoning with disabled reasoning. Suppress the chain-of-thought display but leave the model in thinking mode, and your assertions may see a clean response while the underlying behavior follows a different execution path.

To pin behavior across CI runs, disable thinking explicitly with 

. Otherwise a diff may reflect configuration drift rather than a real safety regression.

Add one more layer. Defense-in-depth matters here. Chain a toxicity or profanity filter after the model response, strip dangerous instructions before they reach the prompt, and cap your token budget to shrink the injection surface. The gate catches regressions. The filters catch what slips through.

You have read the reasoning, wired up the config, and gated your pipeline. The open question is whether you can start testing DeepSeek today without losing a week to setup. You can. The trick to knowing before someone else does is treating it as a repeatable loop, not a one-off audit.

Use the workflow below as a compact operating plan.

AUTHOR

Dr. Dipen

Security testing stopped being optional the moment you wired an open-source model into a product real users touch. One successful jailbreak can leak your proprietary prompts, expose customer data, or drop you into a compliance mess you never budgeted for. DeepSeek V3 and R1 are appealing for good…

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

Free Tools

How to Security-Test DeepSeek V3 and R1 with Promptfoo

Tags

Responses (0)

Free AI Career Tools

AI Job Listings

ATS Resume Checker

Startup Perks

Masterclasses

Tutorials

Fullstack React with TypeScript