scientific-critical-thinking

Scientific Critical Thinking

Overview

Critical thinking is a systematic process for evaluating scientific rigor. Assess methodology, experimental design, statistical validity, biases, confounding, and evidence quality using GRADE and Cochrane ROB frameworks. Apply this skill for critical analysis of scientific claims.

When to Use This Skill

This skill should be used when:

• Evaluating research methodology and experimental design
• Assessing statistical validity and evidence quality
• Identifying biases and confounding in studies
• Reviewing scientific claims and conclusions
• Conducting systematic reviews or meta-analyses
• Applying GRADE or Cochrane risk of bias assessments
• Providing critical analysis of research papers

Visual Aids (Optional)

Only add figures when the user explicitly requests a diagram (for example, a GRADE flowchart, bias decision tree, or evidence-quality framework).

When figures help:

• Critical thinking framework diagrams
• Bias identification decision trees
• Evidence quality assessment flowcharts
• GRADE or risk-of-bias evaluation frameworks

How to create figures:

• Preferred: Use the scientific-schematics skill for AI-generated diagrams from a natural-language description
• Alternative: Build figures in your usual tools (draw.io, PowerPoint, matplotlib, etc.)

Run from the repository root, with OPENROUTER_API_KEY set:

python skills/scientific-schematics/scripts/generate_schematic.py "GRADE evidence assessment flowchart with downgrade and upgrade factors" -o figures/grade_flowchart.png --doc-type report

Disclosure: AI schematic generation sends your prompt to OpenRouter (a third-party API). Do not include unpublished sensitive details unless that transmission is appropriate for your project.

---

Core Capabilities

Seven capability areas, each with the questions to ask and what the answers imply, are in references/core_capabilities.md:

1. Methodology critique — design, controls, confounding, and whether the method can answer the question asked.
2. Bias detection — selection, measurement, publication, and cognitive biases.
3. Statistical analysis evaluation — power, multiplicity, p-value misuse, effect sizes.
4. Evidence quality assessment — study hierarchy, replication, and strength of inference.
5. Logical fallacy identification — the fallacies that recur in scientific argument.
6. Research design guidance — how to strengthen a design before data collection.
7. Claim evaluation — separating what was shown from what is being asserted.

Per-topic detail is in references/scientific_method.md, references/common_biases.md, references/statistical_pitfalls.md, references/evidence_hierarchy.md, references/logical_fallacies.md, and references/experimental_design.md.

Application Guidelines

General Approach

1. Be Constructive

   • Identify strengths as well as weaknesses
   • Suggest improvements rather than just criticizing
   • Distinguish between fatal flaws and minor limitations
   • Recognize that all research has limitations

2. Be Specific

   • Point to specific instances (e.g., "Table 2 shows..." or "In the Methods section...")
   • Quote problematic statements
   • Provide concrete examples of issues
   • Reference specific principles or standards violated

3. Be Proportionate

   • Match criticism severity to issue importance
   • Distinguish between major threats to validity and minor concerns
   • Consider whether issues affect primary conclusions
   • Acknowledge uncertainty in your own assessments

4. Apply Consistent Standards

   • Use same criteria across all studies
   • Don't apply stricter standards to findings you dislike
   • Acknowledge your own potential biases
   • Base judgments on methodology, not results

5. Consider Context

   • Acknowledge practical and ethical constraints
   • Consider field-specific norms for effect sizes and methods
   • Recognize exploratory vs. confirmatory contexts
   • Account for resource limitations in evaluating studies

When Providing Critique

Structure feedback as:

1. Summary: Brief overview of what was evaluated
2. Strengths: What was done well (important for credibility and learning)
3. Concerns: Issues organized by severity
   • Critical issues (threaten validity of main conclusions)
   • Important issues (affect interpretation but not fatally)
   • Minor issues (worth noting but don't change conclusions)
4. Specific Recommendations: Actionable suggestions for improvement
5. Overall Assessment: Balanced conclusion about evidence quality and what can be concluded

Use precise terminology:

• Name specific biases, fallacies, and methodological issues
• Reference established standards and guidelines
• Cite principles from scientific methodology
• Use technical terms accurately

When Uncertain

• Acknowledge uncertainty: "This could be X or Y; additional information needed is Z"
• Ask clarifying questions: "Was [methodological detail] done? This affects interpretation."
• Provide conditional assessments: "If X was done, then Y follows; if not, then Z is concern"
• Note what additional information would resolve uncertainty

Reference Materials

This skill includes comprehensive reference materials that provide detailed frameworks for critical evaluation:

• references/scientific_method.md - Core principles of scientific methodology, the scientific process, critical evaluation criteria, red flags in scientific claims, causal inference standards, peer review, and open science principles

• references/common_biases.md - Comprehensive taxonomy of cognitive, experimental, methodological, statistical, and analysis biases with detection and mitigation strategies

• references/statistical_pitfalls.md - Common statistical errors and misinterpretations including p-value misunderstandings, multiple comparisons problems, sample size issues, effect size mistakes, correlation/causation confusion, regression pitfalls, and meta-analysis issues

• references/evidence_hierarchy.md - Traditional evidence hierarchy, GRADE system, study quality assessment criteria, domain-specific considerations, evidence synthesis principles, and practical decision frameworks

• references/logical_fallacies.md - Logical fallacies common in scientific discourse organized by type (causation, generalization, authority, relevance, structure, statistical) with examples and detection strategies

• references/experimental_design.md - Comprehensive experimental design checklist covering research questions, hypotheses, study design selection, variables, sampling, blinding, randomization, control groups, procedures, measurement, bias minimization, data management, statistical planning, ethical considerations, validity threats, and reporting standards

When to consult references:

• Load references into context when detailed frameworks are needed
• Use grep to search references for specific topics: grep -r "pattern" references/
• References provide depth; SKILL.md provides procedural guidance
• Consult references for comprehensive lists, detailed criteria, and specific examples

Remember

Scientific critical thinking is about:

• Systematic evaluation using established principles
• Constructive critique that improves science
• Proportional confidence to evidence strength
• Transparency about uncertainty and limitations
• Consistent application of standards
• Recognition that all research has limitations
• Balance between skepticism and openness to evidence

Always distinguish between:

• Data (what was observed) and interpretation (what it means)
• Correlation and causation
• Statistical significance and practical importance
• Exploratory and confirmatory findings
• What is known and what is uncertain
• Evidence against a claim and evidence for the null

Goals of critical thinking:

1. Identify strengths and weaknesses accurately
2. Determine what conclusions are supported
3. Recognize limitations and uncertainties
4. Suggest improvements for future work
5. Advance scientific understanding

Agent operating procedure

1. Check the environment. Collect the user's draft, data, figures and target venue before writing anything.
2. Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
3. Run a small version first. Draft one section or paragraph and confirm tone, structure and terminology with the user.
4. Execute the full task using the instructions and references above.
5. Validate the result. Every claim traces to the user's data or a verified source; numbers match tables and figures; citations are real (citation-verification).
6. Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.

If this happens	Do this
A claim has no supporting data or citation	Flag it in the text (e.g. [citation needed]) instead of inventing support.
A function, flag or endpoint in these instructions is missing in the installed version	Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API.
A required input, identifier or parameter is ambiguous	Ask the user, or state the assumption explicitly before running.

Integrity rules

• Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
• The user is the author: never add results, references or claims they did not provide or approve.
• Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
• Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.

Related skills

• bgpt-paper-search: Search scientific papers and retrieve structured experimental data extracted from full-text studies via the BGPT MCP server.
• systematic-review-prisma: Plan, run and report systematic reviews and meta-analyses to PRISMA 2020 standards.
• peer-review: Prepare evidence-bounded, constructive peer-review drafts and structured manuscript assessments.