Testing in CI/CD Is Not the Same as Testing on Your Machine

"It Works on My Machine" Is a Test Design Problem

Your test suite passes locally. You push. CI is red. The failure is in a test that passed 20 consecutive times locally this morning. You check the logs: java.net.ConnectException: Connection refused or AssertionError: expected <42> but was <41> or a timeout that never triggers locally.

This pattern isn't bad luck. It's a test that encodes assumptions about the environment where it runs — assumptions that are true on your development machine and false on a CI runner. Identifying and eliminating those assumptions is what makes a test suite portable, which is a prerequisite for a trustworthy CI pipeline.

The Most Common Environment Assumptions

File system assumptions are subtle and prevalent. Tests that construct file paths with hardcoded separators ("src/test/resources/data.json" on Unix, fails on Windows CI runners), tests that write to /tmp and assume it's writable, tests that assume a specific working directory when the CI runner may set a different one.

// Fragile: assumes Unix path separator and specific working directory
File testData = new File("src/test/resources/fixtures/payments.json");

// Portable: use classpath loading, which works regardless of working directory
InputStream stream = getClass().getResourceAsStream("/fixtures/payments.json");
// Or with Spring:
Resource resource = new ClassPathResource("fixtures/payments.json");

Timing assumptions are the most common source of tests that pass locally (fast developer machine, low load) and fail in CI (slower runner, higher load, JVM startup overhead). Thread.sleep-based synchronization, @Timeout values that are too tight, and assertions that assume an async event has completed without proper waiting.

// Fragile: assumes the event is processed in under 100ms
orderService.placeOrder(order);
Thread.sleep(100);
assertThat(eventStore.findByOrderId(order.id())).isNotEmpty();

// Portable: wait explicitly with a reasonable timeout
orderService.placeOrder(order);
await().atMost(5, SECONDS)
       .until(() -> eventStore.findByOrderId(order.id()).size() > 0);
// Awaitility library: designed for this exact pattern

Port assumptions break when a port is already in use on the CI runner, or when two parallel test jobs try to bind the same port. Never hardcode ports in tests.

// Fragile: port 8080 might already be in use
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
class ApiTest {}

// Portable: let the OS assign a free port
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@LocalServerPort
int port;
class ApiTest {}

Locale and timezone assumptions cause date formatting and parsing tests to fail on CI runners configured with different defaults. Always be explicit:

// Fragile: depends on system default locale
String formatted = dateFormat.format(date);

// Portable: specify locale and timezone explicitly
DateTimeFormatter formatter = DateTimeFormatter
    .ofPattern("dd/MM/yyyy HH:mm")
    .withLocale(Locale.UK)
    .withZone(ZoneId.of("UTC"));

Container Environment Differences

CI runners are typically Linux containers with constrained resources — 2 CPUs, 7 GB RAM, no GPU, slower disk I/O than a developer's SSD. Tests that make assumptions about available resources fail under these constraints:

• Tests allocating large in-memory structures may OOM on CI runners
• Testcontainer-based tests that rely on fast container startup may time out
• Tests with CPU-intensive work (encryption, compression) may hit time limits

The fix is defensive configuration: explicit timeouts that account for slower environments, resource-bounded data generation in tests, and Testcontainer startup checks that wait for actual readiness rather than port availability.

Making the CI Environment Local-Reproducible

The better long-term solution is making the environments match more closely. This means:

• Running local development in a container that matches the CI image (Docker Compose for local dev, same base image as CI)
• Using .env.test files for environment-specific configuration that can be checked in (non-sensitive values only) and replicated in CI via environment variables
• Pinning the JDK and toolchain version to the same version in both environments

# .github/workflows/ci.yml: pin the same JDK version used locally
- uses: actions/setup-java@v4
  with:
    java-version: '21.0.3'     # Pin the patch version, not just major
    distribution: 'temurin'

If the CI runner uses JDK 21.0.3 and your developers use JDK 21.0.1, behavior differences in edge cases (particularly around string handling, GC behavior, and cryptographic defaults) can cause hard-to-diagnose inconsistencies.

The Diagnostic Workflow

When a test fails in CI but not locally, the investigation order is: first check timing (add logging to see if the test is slow in CI), then check ports (look for bind errors), then check file paths (add working directory logging), then check locale/timezone (print system properties at test start). Most local-vs-CI failures fall into one of these four categories. Work through them in order rather than guessing.